Domino Resource Issues under XPages

A couple of weeks ago, we launched a new XPages app. Hopefully the first of many. It was very impressive and we got a lot of hits....

...until the server crashed.

Not a big deal. Our server is set to restart automatically, and it was up and running in no time. Then about 1.5 days later we had another crash.

We've decided to tackle this on a few fronts, first of all we're rewriting some parts of the app to be a bit less intense and to take better advantage of the recycler. That's all cutting edge development stuff, so it's really not "me".

On the admin side, I wanted to see if we could release resources a bit. I checked the server close to a crash but there's no indication on the Windows 2003 side of things. Of course, the Notes Logs tell a different story.

HTTP JVM: CLFAD0211E: Exception thrown. For more detailed information, please consult error-log-0.xml located in e:/Lotus/Domino/data/domino/workspace/logs

HTTP JVM: >>>>

HTTP JVM: The XPages runtime engine faced an OutOfMemoryError

HTTP JVM: You can fix this by increasing the value of the HTTPJVMMaxHeapSize variable in notes.ini

HTTP JVM: >>>>

HTTP JVM: Out of memory exception occurred servicing request for: /publicsite/OurNewXpagesDB.xsp - HTTP Code: 500. For more detailed information, please consult error-log-0.xml located in e:/Lotus/Domino/data/domino/workspace/logs

HTTP Web Server: Command Not Handled Exception [/publicsite/OurNewXpagesDB.xsp] Anonymous

HTTP JVM: CLFAD0211E: Exception thrown. For more detailed information, please consult error-log-0.xml located in e:/Lotus/Domino/data/domino/workspace/logs

So, we decided to try a http restart next time these error messages started building up.

It worked!

Doing a TELL HTTP RESTART bought us two more days of uptime.

To Restart or not to Restart?

We looked around and found that our problems weren't as unique as we'd imagined. There are a few people on 8.5 and above (we're currently 8.5.2) who have this problem.

Apparently also, a Tell HTTP Restart flushes memory but doesn't do much for the JVM.

To flush the JVM, we need to think about;

TELL HTTP QUIT

then after a few minutes

LOAD HTTP

or as the guy in the IT Crowd says "have you tried turning it off and on again?"

We'll be doing this until we get our application sorted but the question is; should this be part of our normal nightly routine? Maybe it's good practice to restart your web server's service nightly - especially if you have a cluster which could take the load while the restart occurs.

Does anyone think that this is "best practice"?

How to Create a Bootable DVD Using Nero Burning ROM 9

I often need to create bootable CDs and DVDs but it's weird because I frequently end up buring myself a new coaster instead. It's not that the process is difficult, just that nero has a few too many options and I forget which ones to choose and end up picking the wrong one. I figured that the best way to avoid this mistake in future would be to write the steps down. Procedure Insert CD or DVD into your DVD Burner. Start Nero Burning ROM 9 Choose DVD-ROM (Boot) or CD-ROM (Boot) depending on what you're creating You'll be prompted for a disk image source. Choose a Nero Source - you'll usually find them somewhere like this... C:\Program Files\Nero\Nero9\Nero Burning Rom\DOSBootImage.ima Leave the Boot Locale as English - unless you really need a different keyboard layout Tick the box marked [X] Enable Expert Settings Choose Hard Drive Emulation and leave any other settings as they are. Click the button marked New Add any files you want but don't try to add operati

How to Change Your Notification Options for New Lotus Notes Mail in version 8.x

Don't worry, I'm not patronizing you (my readers), I just decided to re-document this for one of our internal users and thought you might want to be able to use it in your own user documentation. WHAT IS THIS DOCUMENT ABOUT? Some people who don't get a lot of mail, like to be notified when such an event occurs. Notification can be; via a sound via a pop-up box via the system tray (where the computer clock is) The pop up box looks like this; Other people, who like myself, get too much mail would rather not be notified. The aim of this document is to tell you how (and where) to turn these options on and off. CHANGING YOUR SETTINGS To change your settings from the Notes 8.x client; On the Menu, click File , then Preferences... On the left hand side , click on the little plus sign to the left of Mail to expand the options. Click on the option marked Sending and Receiving . In the middle section, under receiving, you can control your notifications. If you untick the box mark

How to Create an Auto-Response Mail Message in Lotus Notes 8.5.3+

Why would you do this? Suppose that you have an externally accessible generic email address for your company; support@mycompany.com or info@mycompany.com. You might expose this to the web and allow people to send messages to you. Setting up an auto-response email will tell the senders that their message reached its destination and that it will be dealt with accordingly. It's also good practice to include links to FAQs or other useful information. Why 8.5.3 The techniques we'll be using here work in older versions of Notes but some of the options seem to have moved around in 8.5.3. I figured it was a good time to show you where they've moved to. The Procedure Start Domino Designer and open the Mail file to be modified. A really quick way to do this is to right-click on the application tab and choose "Open in Designer". In the Left hand panel of designer, expand Code and then double-click Agents. A new window should appear. Click the action

Tim Tripcony said…

You mention taking advantage of the recycler... I assume you're referring to calling the recycle() method on variables pointing to Domino objects (database, view, document, etc.) before assigning them to something else (such as when iterating through documents in a view). You should always, always do this. Forgetting to do this causes memory leaks, and is a Very Bad Thing.

If the JVM is running out of memory, a failure to do the above is the most likely culprit. You can delay this occurrence by increasing the amount of memory allotted to the JVM, or by (as you suggest) periodically restarting the HTTP task. But neither option cures the disease, it just treats the symptoms, so to speak. Unless you fix the memory leak to begin with, your users are likely to experience gradually increasing performance degradation as the server approaches the point at which it would run out of memory entirely.

Assuming you're running 8.5.2 (if not, upgrade), one additional option available to you that certainly is best practice is to update the application properties to tell the application to serialize all XPages to disk. This setting was specifically added by IBM to allow XPage applications to scale to more users; it causes all operations to consume the bare minimum of memory by saving all information about the page structure to the hard drive as soon as each request has returned a response to the browser, instead of storing that structure in memory. If the user triggers any events against the page, the page structure is then loaded back into memory before the event is handled. The tradeoff is that, by not holding all of this information in memory for the duration of the user session, there is a slight performance hit for events... but the response delta is typically in the sub-second range, so this is generally acceptable when scaling to thousands of concurrent users.

1:09 pm

P.S. Instead of issuing separate "TELL HTTP QUIT" and "LOAD HTTP" commands, the single "RESTART TASK HTTP" command does both. This is not the same as "TELL HTTP RESTART", which, as you mentioned, flushes some memory (and reloads some configuration settings) but does not actually restart the JVM. "RESTART TASK HTTP" actually shuts down the entire HTTP task, then loads it again. The only time I've seen this command fail to do that is when there are hung threads (like web agents that did not complete successfully).

1:14 pm

Simon O'Doherty said…

Try doing.

tell http xsp heapdump
tell http xsp javadump

Then have a look at the output. May give some hints.

8:14 pm

Paul Hannan said…

Hi,
Do you also have the 852 FixPacks installed upon the servers?
p.

5:58 am

Graham Dodge said…

Gavin,

Is the app an upgraded R7.x app with extensive Lotusscript code possibly calling new Java agents? It may pay to revise old code and make sure that all agents are properly disposing of their artifacts when they end.

8:25 am

Chris Whisonant said…

Out of curiousity, what is the value that you have if you do a SHOW CONFIG HTTPJVMMaxHeapSize ? It could be that you need to increase this value. IBM has tinkered with what they think the default settings should be. See, for instance, my blog post here. I would recommend that you increase this value to 128M if it's not already. And if it is, then add another 128M to it.

11:23 pm

Real World Computing

Search This Blog