Memory Leak

[Jim Reed]

We have noticed that after several redeploys of the petclinic sample application we get an OutOfMemory error. We've tryed using Spring 1.1rc1 and 1.1rc2 with Tomcat 4.1.30.

I have seen several threads in the mailinglist (i.e. http://article.gmane.org/gmane.comp.java.springframework.devel/5107) that indicates this was a problem with the pre-1.0.2 versions of Spring.

I have attempted to write my own simple application using Spring and I see the same issue with it running on Tomcat 4.1.30 and OC4J 9.0.3.

Is anyone else still seeing this problem with the 1.1 versions of Spring?

[gpoirier]

I don't think the PetClinic sample have a ContextListener that call java.beans.Introspector.flushCaches() when its destroyed, did you try adding that?

[Jim Reed]

Yes, or at least I think I added it correctly. I added the folllowing entry into my web.xml....

<listener>
<listener-class>mystuff.DummyListener</listener-class>
</listener>

and I add the following class to the WAR....

package mystuff;

import java.beans.Introspector;
import javax.servlet.ServletContextEvent;
import javax.servlet.ServletContextListener;

public class DummyListener implements ServletContextListener
{

public void contextInitialized(ServletContextEvent event)
{
System.out.println( "Init was called..." );
}

public void contextDestroyed(ServletContextEvent event)
{
Introspector.flushCaches();
System.out.println( "Destroy was called..." );
}

}

When I deploy and redeploy the application, I see my System.out.println s just as one would expect.

[Juergen Hoeller]

IIRC, there's also a memory leak in CGLIB, regarding its implicitly created subclasses. In general, I'm confident that that Spring itself does everything it can to avoid memory leaks now.

With an Introspector.flushCaches call, you're also cleaning up behind other tools that use JavaBeans introspection. So if there's still a leak, I bet that it is in one of the other tools used in the application, and not caused by JavaBeans introspection.

Juergen

[Maarten]

Hi,

How do I know if my webapp needs a ContextListener that calls

java.beans.Introspector.flushCaches()

Do I only need it when using Spring AOP features and/or Hibernate ?

I guess there's never any harm in calling it when the servlet context is about to be shut down ?

Maarten

[gpoirier]

Maarten,

There's indeed no harm in doing so, the only thing it could cause it a possible slow down of the Introspector calls while it build its cache again. Also, note that Spring offers a listener out-of-the-box for this, it's: org.springframework.web.util.IntrospectorCleanupLi stener.

As far as when doing it, it's needed for anything that call "Introspector.getBeanInfo" a class that isn't in the system class loader, but doesn't itself clear the reference in the cache. I believe Hibernate does that, and many libraries such as Jakarta's commons-beanutils.


Juergen,

Any idea how to reproduce the memory leak in CGLIB? I tried creating a CGLIB Proxy throught spring's ProxyFactoryBean with proxyTargetClass at true, but I couldn't reproduce any memory leak this way.

I was able to reproduce the OutOfMemoryError when reloading the petclinic, but I'm trying to isolate the problem, to see what is preventing the ClassLoader to be garbage collected. I haven't experimented much yet, but I haven't found anything so far.

[Colin Sampaleanu]

Actually, the CGLIB related proxy code _had_ a leak, part of which was fixed for 1.0.2, and part of which was fixed shortly after. I don't know that there remain any issues in the current codebase.

Regards,

[gpoirier]

I had time recently to do further investigation trying to find the cause of the memory leak for the PetClinic. Turns out there's sereral causes, I found one certain cause, one potential, and there's still a leak so there's at least one more cause.

The first thing is DriverManager. The Drivers registrer themself to DriverManager, if they are in a child ClassLoader that is thrown away, they will just hang there, not possible to be collected. Application code must deregistrer them explicitely. The drivers are kept in a Vector with a strong reference.

The potential memory leak I found is the Runtime.getRuntime().addShutdownHook(Thread) method. There's at least the JDK logger that uses it, and the Cleaner they register seems to have an indirect reference on child ClassLoaders. The shutdown hooks are kept in a HashSet with a strong reference.

I'll update if I manage to figure out other things that prevent the Webapp ClassLoader to be garbage collected.

Guillaume

[gpoirier]

I found another potential cause that prevent a ClassLoader to be collected, that's ThreadLocal variables. Spring's use doesn't seem to be possible to cause that, because anytime a ThreadLocal variable is used, it seems to be only for the scope of a request/method, then removed. However, Dom4j, used by the PetClinic, seems to use ThreadLocal variables with the JVM Signals, and never remove the the variables.

I still haven't found all the causes though, 'cause the ClassLoader still doesn't get collected even if I place the dom4j jar in a parent ClassLoader.

Guillaume

[gpoirier]

It turns out I had found pretty much all the causes, but my detection algo wasn't working properly in all cases, I haven't found why yet. Anyway, with a few changes, I am able to make PetClinic work without memory leak when reloading the webapp. However, it has to be using the JDBC implementation, or else there's some jars that needs to be in a parent ClassLoader. For hibernate, it's the dom4j.jar that causes problems, and when using ojb, it seems to be the ojb jar itself. Both appears to be leaving ThreadLocal stuff in the signal dispatcher thread.

When using JDBC, the following ServletContextListner prevents the leak:
public class CleanupListener implements ServletContextListener {
public void contextInitialized(ServletContextEvent event) {
}
public void contextDestroyed(ServletContextEvent event) {
try {
Introspector.flushCaches();
for (Enumeration e = DriverManager.getDrivers(); e.hasMoreElements();) {
Driver driver = (Driver) e.nextElement();
if (driver.getClass().getClassLoader() == getClass().getClassLoader()) {
DriverManager.deregisterDriver(driver);
}
}
} catch (Throwable e) {
System.err.println("Failled to cleanup ClassLoader for webapp");
e.printStackTrace();
}
}
}
I don't know if it would be worth updating the sample applications with such cleanup - and may be add a DriverManager cleanup ServletContextListener in spring with IntrospectorCleanupListner?

Guillaume

[MHarhen]

if (driver.getClass().getClassLoader() == getClass().getClassLoader()) {
DriverManager.deregisterDriver(driver);
}


Can you explain why you are checking for identical classloaders?

Thanks

[sakkew]

I started getting OutOfMemoryError when i implemented OpenSessionInView into my junittests. I have more than 100 testcases, and each of them does the setup and teardown shown below. The problem disappears if i remove the shown lines. The problem occurs after 70 of the testcases have finished.
I have tried this with spring 1.1 and cglib 2.02.

public void setUp() throws Exception
{
super.setUp();
sessionFactory = (SessionFactory) ApplicationContextFactory.getApplicationContext(). getBean(SESSION_FACTORY);
Session session = SessionFactoryUtils.getSession(sessionFactory, true);
TransactionSynchronizationManager.bindResource(ses sionFactory, new SessionHolder(session));
...
}

public void tearDown() throws Exception
{
...
SessionHolder sessionHolder = (SessionHolder) TransactionSynchronizationManager.unbindResource(s essionFactory);
SessionFactoryUtils.closeSessionIfNecessary(sessio nHolder.getSession(), sessionFactory);
Introspector.flushCaches();
super.tearDown();
}

[gpoirier]

Can you explain why you are checking for identical classloaders?

Thanks
The reason is because you don't want to remove drivers registered by other webapps, or registered in a parent ClassLoader, because those ClassLoader are not being thrown away when a single webapp is unloaded.

Actually, I suppose the check could be more complete, if the driver's ClassLoader is a child of the CleanupListener class, the driver should be unregistered too, but I guess a webapp shouldn't have drivers registered in child ClassLoader anyway.


sakkew,

It seems that in your tests you have instance variables that you left non-null after the tear down. The way JUnit works, it creates all the test cases and keep them in a vector until all the test cases have been executed. So that means if you create expensive resources in each test case, you'll eventually run into an OOM Error.

However, looking at the code you added and removed, it seems like it doesn't create any more instance variables, so there must be something else. How are you running the test cases, are you pointing a source directory to eclipse which runs all the tests found there? If you try alternative ways of running your cases, does it make a difference? For example, using ant, or creating a test suite that include all you tests and running it, throught eclipse JUnit runner or throught JUnit's own runner.

I'll try to reproduce the problem later when I have time to see if I can figure out what's the issue there.

[daliakamal]

Hello

I add the DummyListener to my application but it make Tomcat hangs after 2 redeploy.
Can u help me?

[Chris Messina]

I was having trouble with locked jars (jars don't delete) when undeploying my web applications. The jars in question are my oracle driver jar, and struts-tiles jar. I have tried using this listener posted in this thread but the jars still will not delete on undeploy.

Is this not cause by the same issue, or is there something else I need to do other than create the listener class in my project and then configure it in the web.xml file?

[Chris Messina]

I was finally able to get the tiles jar to undeploy. I had to remove the doctypes from my tiles definition files, and I had to set the validateDefinitions property of the TilesConfigurer to false (otherwise it throws exceptions when there are no doctypes). This is unfortunate, I would have liked to keep validation of the tiles defintions (prefer fail fast when possible), but being able to undeploy and redeploy an application without bringing down the whole server is more important.

I still have not figured out how to get the ojdbc14 jar to undeploy though. I have that listener which loops through the drivers returned from DriverManager, and then deregisteres any driver who's classloader is the same as the classloader for the listener. I have also added the following attributes to the Context tag of my context.xml file antiJARLocking="false" antiResourceLocking="false" reloadable="true"

I will have to continue looking, but I wanted to post here in case anyone else was having trouble with undeploying a struts-tiles jar (I'm specifically using version 1.3.5 of tiles).