Salesforce/Axis MessageElement Out Of Memory Error resolved !

If you are a java developer who’s using Apache Axis for developing web service client applications/mashups and facing “Out of Memory” issues because of “org.apache.axis.message.MessageElement”, then this post can help you in fixing this memory error. This post if also of relevance to developers creating application using Salesforce Web Services via Axis client stubs, its quite easy to go Out of memory when using salesforce partner wsdl to create a bunch of records.

“MessageElement” a nasty, badly documented class !

This class is the one that most of the developers use or even its used by Axis under the hoods at a couple of places. The sad part about this class is its very badly documented and the way developer needs to keep its state correct is not clear at all. Over that, this class eats insane amount of memory if used for bulk operations. So developers should be very careful to operate not in bulk, but chunks with this class.

Using MessageElement class in a risky way

MesageElement class mostly becomes memory hog when creating new instances of it. I faced out of memory issues for this class during one of my recent assignments, we were trying to create many records in salesforce and operation failed always because of out of memory error. We were using the salesforce sample code to create MessageElement instances like this below

	public static MessageElement newMessageElement(String name, Object value) throws Exception {
		MessageElement me = new MessageElement("", name);	
		me.setObjectValue(value);
		Element e = me.getAsDOM();
		e.removeAttribute("xsi:type");
		e.removeAttribute("xmlns:ns1");
		e.removeAttribute("xmlns:xsd");
		e.removeAttribute("xmlns:xsi");
		me = new MessageElement(e);
		return me;
	}

This is a strange but was working way for us to get the job done, we are not Axis experts unfortunately 😉

But this stuff become unusable then we tried to research around possible solutions, so ideas that came out during that research was

• To create a single message element and clone it rather creating from scratch.

  • This didn’t worked, MessageElement maintains a complex XML structure that is not easily to clone and then transform to new values.

• To reduce the number of records we are working on, call Garbage collector and sleep 🙂

Finally one of my peer Amit, figured a way that is similar to cloning MessageElement, but not really a clone and it worked for us. He suggested to cache a Template MessageElement, and use that for creating new MessageElements. Its explained in detail below.

Alternate Solution – A Template Element

Alternate solution was to cache the Xml “org.w3c.dom.Element” used internally by MessageElement class. MessageElement also gives constructors that accept org.w3c.dom.Element. So we can cache a single Element instance as Template to create new MessageElement instances. As shown below.

private static MessageElement TEMPLATE_MESSAGE_ELEMENT = new MessageElement(
			"", "temp");
// The Template org.w3c.dom.Element instance
private static Element TEMPLATE_XML_ELEMENT;

static {
	try {
		// Create and cache this org.w3c.dom.Element instance for once here.
		TEMPLATE_XML_ELEMENT = TEMPLATE_MESSAGE_ELEMENT.getAsDOM();
	} catch (Exception e) {
		throw new RunTimeException(e);
	}
	TEMPLATE_XML_ELEMENT.removeAttribute("xsi:type");
	TEMPLATE_XML_ELEMENT.removeAttribute("xmlns:ns1");
	TEMPLATE_XML_ELEMENT.removeAttribute("xmlns:xsd");
	TEMPLATE_XML_ELEMENT.removeAttribute("xmlns:xsi");
}


public static MessageElement fromTemplateElement(String name, Object value)
		throws SOAPException {
	// Use the TEMPLATE org.w3c.dom.Element to create new Message Elements
	MessageElement me = new MessageElement(TEMPLATE_XML_ELEMENT);
	me.setObjectValue(value);
	me.setName(name);
	return me;
}

So when we started using the new fromTemplateElement() method, we never got OutOfMemory error, even for big operations that involve handling too many records.

Benchmarks – Why alternate solution “Template Element” is really better ?

For benchmarking I just setup a simple fixture, that you guys can also try quickly to get confidence. In this fixture I just iterated a 100,000 times to create 100,000 MessageElement instances using both approaches.

Here is the fixture (My Machine Core2Duo, 4 Gigs of RAM)

public class MessageElementTest {
// NOTE: Copy the above code and static block for both methods here

	public static void main(String[] args) throws SOAPException {
		List<MessageElement> elems = new ArrayList<MessageElement>();
		for (int i = 0; i < 100000; i++) {
			// Create fake name using current millis
			String name = "name" + System.currentTimeMillis();
			Object value = "val" + System.currentTimeMillis();
			// Print iteration and free memory
			System.out.println("Iteration " + i + " Free Memory :"
					+ +Runtime.getRuntime().freeMemory());
			
			// First approach
			MessageElement e = newMessageElement(name, value);			
			elems.add(e);
		}
		System.out.println(elems.size());
	}

Here are the results, this approach never completed 100,000 iterations and failed near 35,000. Here is the sample output

Iteration 0 Free Memory :15874400
Iteration 1 Free Memory :14616912
.......
.........
.......
Iteration 35642 Free Memory :5256
Iteration 35643 Free Memory :1344
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space

Now to try alternate approach just comment the single line and add this line as shown below

// First approach
// MessageElement e = newMessageElement(name, value);			
// Alternate Approach
MessageElement e  = fromTemplateElement(name, value);

Now I was able to complete all 100,000 iterations without any Out of memory issues etc. Even I ended with decent memory in hand. Here are the results

Iteration 0 Free Memory :15874400
Iteration 1 Free Memory :14252944
Iteration 2 Free Memory :14252944
...
....
....
Iteration 99998 Free Memory :17744408
Iteration 99999 Free Memory :17744408
100000

So its clearly visible that the original approach is just good for sample codes, if you are planning to put your app in production then must try the Alternative approach.

All the best !