JRun running out of memory

I have had ColdFusion / JRun memory problems on projects in the past but are always very tricky to pin down. However, recently when a couple of my sites have started ranking at top positions on Google I have been experiencing a lot more crashes from our ColdFusion 7 server. Enough was enough and I decided I must sort this out once and for all.

I am running four ColdFusion instances with all of the sites I host spread across them. This has the benefit of me being able to allocate larger chunks of memory to each instance, monitor their load easier and reduce the risk of downtime for the other instances if one does happen to crash.

One of the sites in question normally receives on average about 1200 visitor sessions per day but on this one day the visits shot to over 11,000 - Great you might think. I suppose yes it would be, however with the amount of sites running on the server, the memory very quickly started shooting up and I had to restart the ColdFusion instance several times during the day.

Each time the server crashed, when I logged onto it to terminated the hung ColdFusion instance I would get the following error on the desktop:-

The instruction at "0x0000000" referenced memory at "0x0000000". The memory could not be "read".
Click on OK to terminate the program

This is in no way ideal as the eBay API I use on one of the sites takes about 15 minutes to download, build the stubs and then compile the Java files into classes. This meant that during that time I could no longer process requests properly and they started queuing up and causing further problems. I managed to alleviate this by only allowing the initial request to invoke the web service and then denying everyone else for another 20 minutes until Jikes had finished compiling everything. Since then, I have migrated my code to use the eBay XML API rather than a web service - Starts instantly and is just as easy to implement. Anyway, that's all by the by.

Jrun Server Error

Even after doing this, I still had the problem with JRun crashing and giving the dreaded error:-

Last resorts

I tried several things in desperation such as reducing session timeouts to 5 minutes. This didn't help, and only annoyed webmasters using our CMS who would take ages typing something in only to find when they went to save their session had ended. And yes, before someone mentions it, I could put an AJAX function to auto refresh their session every 1 minute. I'll get round to that later.

I tried altering some of the code that maintains data in session variables and using client variables instead, no joy there either.

Eventually, I took time out and actually thought about what was causing the problem rather than wild stabs in the dark - That's when it hit me! A while back I was doing some SQL server profiling on one of my databases and noticed there were duplicate threads active for the same datasource which just weren't being closed.

Nirvana

I hastily logged onto our web server and amended the datasource of the busy site, unticked the "Maintain connections across client request" check box. The result was INSTANT! As you can see from the FusionReactor graphs below, once I unticked that box and hit submit, the memory dropped like a stone.

At the point in time just before I hit submit the memory usage had steadily increased to 380mb used. After saving the datasource it dropped down to 167mb and stayed steady at that level for the next few days while I was monitoring it.

Notice the ColdFusion instance was restarted at 10:46 and by 12:38 it had reached 309mb free

After the changes were made to the 2 data sources, the memory dropped like a stone and has been sitting very happily at circa 250mb for days now.

Conclusion

It would appear that on high-traffic sites, or as in our case servers with 100 or so ColdFusion driven web sites that if the busier ones maintain connections to the database then it is possible that you may experience these JRun memory problems.

I haven't yet played around with Limiting connections as I was just happy not to have to restart the ColdFusion service every few hours.

It is also possible that specifying a shorter timeout in the datasource may also help - So a "proper" visitor who is visiting more that just one page on your site will benefit from the fact that the connection has been maintained. While a "passer by" who hits the home page, decides they're not really interested and leaves will only tie up the resource for 1 minute maximum.

I would be interested to know of other peoples experiences - so please, post your comments here.

Key phrases:-

• JRun out of memory
• ColdFusion service crashes or hangs
• Server Error
• The server encountered an internal error and was unable to complete your request.
• Could not connect to JRun Server.

Applies to:-

• ColdFusion MX7 Enterprise
• MSSQL Server
• MYSQL

Hardware:-

• Dell Poweredge 2850
• 4gb RAM
• Dual 3.2ghz HT Xeon processors
• Dell PowerVault P220S storage device

Links:-

• http://www.fusion-reactor.com/home.html (Most recommended)