Tuesday, July 26, 2011

Scheduling Maintenance Tasks on the Domino Server

I'm sure that most Domino admins have been doing these tasks regularly but if you don't have occasion to touch them often, you might this post useful.

As I mentioned last week, we've been having issues with the JVE running out of memory on our server. One of the "solutions" (band-aids) was to restart the domino HTTP task. I've been experimenting to see how long I can leave it and the shortest it's been is about 30 hours.

It seemed to be a good idea to stop and then restart the domino HTTP service on a timer. In admin-talk, that means a program document.

Procedure
The steps I'm using here are for Domino 8.5.2 but I don't think they've changed in years.

1. Start Domino Administrator
2. Click on the tab marked Configuration
3. In the left hand navigator,click on Server (and expand it)
4. In the left hand navigator,click on Programs

This will take you to a screen where you can expand your servers and see what other regular tasks are scheduled. If you've inherited a domino admin job, this is one of those important things to check out.

Typical things to find in here include;
  • UPDALL -R
  • COMPACT -B
  • FIXUP
Note that the case on the parameters is very important.

There's a good list of the parameters here.

Interestingly, today I stumbled across an old technote from IBM suggesting that these tasks shouldn't be programmed. I'd be interested to hear comments from other people as to whether or not they do them on their servers.


Anyway, enough with the side-notes, we're adding our own tasks.

5. Click on the big button marked Add Program

This takes you to a new screen (a new notes document).

6. Fill in the program name, command line and server to run on.

a. The program name will usually be NSERVER (for Windows Servers) or - apparently - SERVER for iSeries. If you were doing the other tasks mentioned earlier (UPDALL, FIXUP or COMPACT), then the program name would be one of them. I looked for a document which listed the Possible Program names but couldn't find one. (IBM... this should be a picklist).

b. The Command line will change but if you want to run a domino command line, you usually start with -c and put the rest in inverted commas.

I created one document for each of these two and scheduled them 15 minutes apart - with the quit document running first obviously.

-c "tell http quit"
-c "load http"

I could possibly have used Tim's excellent suggestion of "TELL HTTP RESTART" but I was too chicken to try it out on a schedule. Maybe later.

c. Pick your server... Luckily this IS selectable.

7. On the Schedule side of things, pick the days and times for the commands to run. If you don't want them repeating again throughout the day, leave the repeat interval as 0.

8. Finally, Click the Save and Close button.

Now I just have to sit here and wait until 4:15am... just kidding (I hope).

More Fun Ideas to Try
You'll find that you can run several commands at once by redirecting the input from a text file but I'll leave that for IBM to explain.

Monday, July 18, 2011

Domino Resource Issues under XPages

A couple of weeks ago, we launched a new XPages app. Hopefully the first of many. It was very impressive and we got a lot of hits....

...until the server crashed.

Not a big deal. Our server is set to restart automatically, and it was up and running in no time. Then about 1.5 days later we had another crash.

We've decided to tackle this on a few fronts, first of all we're rewriting some parts of the app to be a bit less intense and to take better advantage of the recycler. That's all cutting edge development stuff, so it's really not "me".

On the admin side, I wanted to see if we could release resources a bit. I checked the server close to a crash but there's no indication on the Windows 2003 side of things. Of course, the Notes Logs tell a different story.

HTTP JVM: CLFAD0211E: Exception thrown. For more detailed information, please consult error-log-0.xml located in e:/Lotus/Domino/data/domino/workspace/logs
HTTP JVM: >>>>
HTTP JVM: The XPages runtime engine faced an OutOfMemoryError
HTTP JVM: You can fix this by increasing the value of the HTTPJVMMaxHeapSize variable in notes.ini
HTTP JVM: >>>>
HTTP JVM: Out of memory exception occurred servicing request for: /publicsite/OurNewXpagesDB.xsp - HTTP Code: 500. For more detailed information, please consult error-log-0.xml located in e:/Lotus/Domino/data/domino/workspace/logs
HTTP Web Server: Command Not Handled Exception [/publicsite/OurNewXpagesDB.xsp] Anonymous
HTTP JVM: CLFAD0211E: Exception thrown. For more detailed information, please consult error-log-0.xml located in e:/Lotus/Domino/data/domino/workspace/logs

So, we decided to try a http restart next time these error messages started building up.

It worked!
Doing a TELL HTTP RESTART bought us two more days of uptime.


To Restart or not to Restart?
We looked around and found that our problems weren't as unique as we'd imagined. There are a few people on 8.5 and above (we're currently 8.5.2) who have this problem.

Apparently also, a Tell HTTP Restart flushes memory but doesn't do much for the JVM.

To flush the JVM, we need to think about;

TELL HTTP QUIT
then after a few minutes
LOAD HTTP

or as the guy in the IT Crowd says "have you tried turning it off and on again?"

We'll be doing this until we get our application sorted but the question is; should this be part of our normal nightly routine? Maybe it's good practice to restart your web server's service nightly - especially if you have a cluster which could take the load while the restart occurs.

Does anyone think that this is "best practice"?