Monday, December 29, 2014

A little more on the Poodle (IBM Domino 9.0.1 and the Poodle Vulnerability)

In my last post, I discussed the steps required to get your Domino 9.0.1 server patched against the poodle bug which exists in both SSL 3.0 and TLS.  At the end of the post, I mentioned that I still had one server which was refusing to apply the patch.

This is how I got around the problem. 


The Server that Wouldn't
After taking the patches all the way back to Fix Pack 1 and slowly patching forward one-by-one with reboots in between, I realised that whatever was affecting my server had been installed for a long time.

There was no easy way to resolve the problem and obviously I couldn't roll back forever.  I just had to accept the fact that the server would not take the patch and take some more drastic measures.



Backup First
Of course, before taking any drastic measures, you should always backup first. Personally, I like to have a proper backup as well as a local copy of the main domino files.  That way I don't have to worry about streaming and tapes.

I manually took a copy of the D:\Lotus\Domino folder on the server (but not sub-folders) and saved it elsewhere.  Then, being careful to overwrite only obvious program and library files, EXE, DLL plus any other non-data files, I copied the files from a server which was otherwise at the same level (64 bit Domino 9.0.1) but had been successfully patched.

After the copy, I restarted the server and retested for the poodle problem. Luckily, the file-copy procedure had resolved it.

Yet again, I'm thankful for the low tech nature of domino and the fact that it doesn't go crazy with registry entries. 


What if I don't have a working Server?
So, if you don't have a working poodle-proof server and you're faced with a similar issue to mine, then one of the simplest options is to install a brand new Domino 9.0.1 server and patch it, then try copying the files across.  There's a pretty good chance that it will do the job.

Wednesday, December 24, 2014

Taming the Poodle in IBM Domino 9.0.1


There's been a lot of talk lately about the Poodle Vulnerability and IBM have provided a rather slow and confusing response full of similarly named files across multiple web pages. We've mostly gotten our systems sorted now but as it was a difficult process, I thought I'd share some of the things I've learned.  If nothing else, I'm sure that other people could benefit from the fix lists being in the one place.  Huge thanks to the guys in the IBM Notes groups on LinkedIn who provided most of the best insights here.

The poodle vulnerability isn't a new thing (it's 15 years old) but recently browser vendors, particularly Mozilla (and soon Google) have have issued upgrades which block access to vulnerable sites by default. I guess that means it's time to deal with it. 

This is what the Firefox error message looks like.

One more thing.... In case you've already dealt with poodle and you think that your system is safe because Mozilla Firefox is no longer complaining, here's a good test; 

https://www.ssllabs.com/ssltest/


Just enter your server domain in the box (and make sure that you tick the box about NOT showing up on the boards. If you're safe, you'll get an A or a B.  If you're still vulnerable to poodle, you'll get an F.


Being on the Latest IBM Domino server
So, the first port of call is to be on the latest IBM server if possible.  That's Release 9.0.1.

IBM has actually provided fixes for a few other versions of domino, so it's not the end of the world if you're not on the latest, provided that you're on one of these.

  • 9.0.1 Fix Pack 2
  • 9.0
  • 8.5.3 Fix Pack 6
  • 8.5.2 Fix Pack 4
  • 8.5.1 Fix Pack 5

However, I've heard that the 8.5 releases don't include the full fix (I could be wrong). So, a 9.x release is a better bet.

Working out what version/patch your Server is
There are a bunch of different ways to do this but my favourite is to simply fire up the Notes admin client and connect to the server.  You'll see the patch level  in the top left.


Release 9.0.1FP2HF590 is the first version of Domino (for 64 bit Windows) which is fully patched against Poodle.   If your server has a higher number (eg: 9.0.2 or 9.0.1FP3 or 9.0.1FP2HF700) then you should be fine. If it's a lower number, then you probably have work to do.


Patching
The patches are actually quite simple to install provided that you download all of them and install them in the right order.  It's best to make folders (and copy them all to an install folder on your server) before starting anything.

The patches you'll need for a basic unpatched Windows 64 bit Domino 9.0.1 installation are as follows (in this order);

Domino 9.0.1 Fix Pack 2
http://www-01.ibm.com/support/docview.wss?uid=swg24037141

You don't have to install Fix Pack 1 because fix packs are cumulative (they contain the earlier fixes too).

Interim Fixes 1, 2 and 3

Unlike the fix packs, interim fixes are not cumulative, you need to install them all and in the exact order.

  • Domino 9.0.1 Fix Pack 2 Interim Fix 1 (This was released twice, so make sure that you have the version from Nov 5, not Nov 3)
  • Domino 9.0.1 Fix Pack 2 Interim Fix 2 (released December 12, 2014)
  • Domino 9.0.1 Fix Pack 2 Interim Fix 3 (released December 19, 2014)

You should be able to find most of the patches here;
http://www-01.ibm.com/support/docview.wss?uid=swg21657963

Once you've got all your patches in the right place, you'll need to find an outage window to shut down your server. It's strongly recommended that you do a backup before proceeding.

To install the fixes, simply run the EXE files and follow the prompts, they should take only a few minutes each.  It's not always necessary for a server reboot between fixes but I've had variable results trying to cheat, so it's worth the extra time on important production servers.

Be careful because some of the fixes look like they're done (with graphs sitting at 100% and a thank you message and a close button).  It's not finished until the bar graphs disappear.

Before you do your final reboot, there's an INI File setting that you need to change.


The INI File Setting
Use Notepad to edit your system's Notes.ini file.
Add a line (pretty much anywhere in the file), which says;

DISABLE_SSLV3=1

I'm fairly certain that this overrides the unsupported command;

DEBUG_UNSUPPORTED_DISABLE_SSLV3=17

So if you see that setting you can probably remove it.
After this you can reboot your server, do your testing and redo the Qualys test.
https://www.ssllabs.com/ssltest/


Other Fears and Concerns.
I've tested this on servers running Traveler, Domino and iNotes.  It seems to work. I haven't tested on the current version of Sametime.

There's some discussion around suggesting that this affects mail,  I didn't see any impact on mail.

You don't need to get a new SSL certificate. The old ones should still work with TLS.


Other things to Consider
There seems to be a patch for Traveler too, so this is probably worth applying.
I haven't gotten around to testing that one yet.

https://www-304.ibm.com/support/docview.wss?uid=swg1LO82423


Problems.
The main problem I found was this message;

From what I can tell, it's indicating that you're installing fixes either in the wrong order (or that Interim  Fix 1 is the wrong dated version).

One time when I received this, I just had to go back to the beginning and install Fixpack 2 and then the hotfixes carefully.  On another server, I can't get past this message (and suspect that a certain amount of uninstalling or rollback will be required).


Like I said, backup before you do this and best of luck Taming your poodle.