Thursday, September 18, 2008

Domino Recovery Tricks - and the Problem with 8.0.2

Installing Domino 8.0.2
Yesterday, we did some tests on Domino 8.0.2. I wasn't expecting any real problems given the ease of the last few Domino upgrades - up to 7.0.2. I guess I was wrong.

First of all, the initial attempt at installation failed with one of those non-specific error messages. I rebooted the server and tried again - no problems. I've seen this problem a few times though - it's something to do with the JVM being held open on Windows Server 2003 even though Domino itself has been closed.

Upon starting Domino, I chose not to upgrade the designs of the databases. Ideally you should not upgrade the designs until your last production server is on at least the same "major" version - in this case, 8.x.

I then did a bit of testing. As usual, I found no problems. Domino is one of the few systems I know which can be upgraded and tested (albeit roughly) in 15 minutes.

The Problem
Then I found the problem. Our most critical database refused to render web pages. This was shown in Microsoft Internet Explorer as a general "dummy spit" message which told me nothing.

Luckily, I use Mozilla Firefox most of the time. I was only using IE for testing because;

a. Most of our clients use it
b. It had no passwords cached

Mozilla gave me a very different story - a proper error message.

Error 500
HTTP Web Server: Application Exception - Duplicate Subform found. A given
subform cannot be used on the same form more than one time.

I'd read about this problem - and I thought that we'd checked that specific database for the problem - apparently not.

Anyway - I wasn't going to compound the problem by attempting a last-minute fix. Time to rollback.

DRP Tricks
I decided to try a couple of tricks first.

First I tried re-installing 7.0.2 over 8. In the first instance the install
failed completely but a retry after a reboot worked wonders.

I didn't really expect a simple re-install to fix the problem though I
would have been pretty impressed if it had.

The next step was to move (copy and delete) the entire


Directory - except for the data directory of course.

I then used ArcServe (r12) to restore this folder from tape - but not
directly to the original location - just in case I stuffed up and overwrote data or something. I restored to a different drive.

After restore, I copied the files to the right location and rebooted.

Everything came up normally - problem solved.

I guess we won't be moving to Domino 8 until we resolve the subform issue on that database.


Nate said...

"Ideally you should not upgrade the designs until your last production server is on at least the same "major" version - in this case, 8.x."

HUH!??!?! Where did you get that idea? The system templates are designed to be BACKWARDS compatible, not FORWARDS. You should upgrade things like the directory template from the very first server you install.

Gavin Bollard said...

I got that idea from years ago when I was working in a company with lots of R4.5 servers replicating with each other - and using a giant shared address book.

One of our IT people decided to upgrade their test server and a laptop to R5 and then accidentally replicated the upgraded address books around the network.

The servers kept upgrading their designs then had trouble reading the address books.

It was a very messy problem to fix.

Nate said...

That particular upgrade was a troublesome one, primarily due to the "Allow more fields in this database" flag. The use of the extended UNK table unfortunately broke backwards compatibility between 4.x and 5. (Well, there was a version, I think 4.6.2, that was ok with it.)

Since then, IBM has maintained an almost religious adherence to backwards compatibility in templates. Whether you're talking about mail templates or directories, you should be able to apply any FORWARD version to an older server release, and have it perform flawlessly. This is particularly so with the directory template, since unless you upgrade it, you frequently WILL encounter problems on later releases.

In fact, I'd kinda like to get some other server folks in on this discussion to corroborate or correct me on this matter. So watch for some additional comments. :-)

Miguel Calvo said...

did you find a solution for the subform issue ?
We've found the same problem in a 8.5.1 deployment.

Gavin Bollard said...


As it turns out, the only solution to the subform problem is to go through each of the forms in your database looking for embedded and computed subforms.

You need to make sure that there's no condition in which the same subform could be used twice.

If it is, for example ours was in the header and the footer, then you need to copy the subform and rename it Footer...

It probably makes sense to rename (or copy) your top one to header too.

So if your subform is called DocumentMenu, you need to have two;


Then adjust the code on the form to call the appropriate subform.

It's a simple fix, but still a DESIGN fix - and IBM isn't going to change it back.

Also, if you're considering the 8.5.1 client, you should look at all databases which have attachments rendered on the web. There's a much worse design change required there.