Tuesday, March 21, 2006

More on Symantec's Premium Anti-Spam Service for Domino

A quick recap
If you remember, last time I blogged about this software we ended up turning it off because it was causing a lot of problems with a particular regular correspondant. Well...

We turned the service back on after about a week because we were getting a lot of spam. I changed the settings to notify us if an email was blocked, but not to delete/quarantine it.

We were notified of the spam, but were never notified of mail from the "offending" mail system (and mail from that system was not being delivered).

Clearly the problem was much more than a simple mis-detection issue.

The Real Culprit
After doing a lot of searching, I eventually found the answer. It was that Microsoft, being unsatisfied with using the industry standard 7 bit MIME format, decided to implement an 8 bit version.

Out of the box, Notes/Domino has compatibility with this format turned off (after all, it isn't the industry standard).

Normally what would happen is that;

  1. An Exchange server would connect to Domino

  2. It would offer a non-standard 8 bit MIME Email

  3. Domino would slap it for breaking standards and refuse to accept the email

  4. Exchange would then resubmit as a standard email

  5. Notes would accept the correctly formatted email.



Unfortunately, since Symantec Premium Anti-Spam uses Microsoft SMTP to collect Emails on port 25 and then passes these to the Domino server on Port 26 (it changes the Domino ports) our server was working as follows;


  1. An Exchange server would connect to Domino

  2. It would offer a non-standard 8 bit MIME Email

  3. The Microsoft SMTP Service would happily accept the non-standard formatted email on behalf of the Symantec premium Anti-Spam service

  4. The sending server would receive a delivery confirmation

  5. Symantec would judge the email as Non-Spam and would forward it to Notes/Domino on Port 26

  6. Domino would slap it for breaking standards and refuse to accept the email

  7. The Symantec Service (or the Microsoft SMTP Service) would discard the email since it was too hard to resubmit in 7 bit MIME.



The Solution
The issue could have been resolved at our end either by changing our SMTP Server to accept only 7 bit, or by allowing our Domino server to accept 8 bit.  I checked both solutions and found that the Microsoft solution involved several patches, changes to files etc, and a lot more work which would probably be wiped out by their next Windows update.

I've opted to change Notes instead and followed the instructions in this document.

To cut a long story short, you do the following;


  1. Make sure you already have a Configuration Settings document for the server(s) to be configured.

  2. From the Domino Administrator, click the Configuration tab and expand the Messaging section.

  3. Click Configurations.

  4. Select the Configuration Settings document for the mail server or servers you want to administer, and click Edit Configuration.

  5. Click the Router/SMTP - Advanced - Commands and Extensions tab.

  6. Locate the 8 Bit MIME Extension field in the Inbound SMTP Commands and Extensions section and set it to Enabled.

  7. Click Save & Close:



I don't think that I needed to restart the server for it to begin working, but I have certainly done so since then (it's always a good idea after an update to restart and see if your changes are still in effect).

Thursday, March 09, 2006

Some Lotus Figures

IBM Put a few figures on slides at LotusSphere.
I'm just repeating them here for easy reference.

Number of Notes/Domino 7 Seats: 12 Million
Number of ND7 Downloads: 11,575
Number of Notes Users: 125 Million
Number of Competitive Migrations: 1300

Recent Acquisitions:
Bowstreet (December 2005)
Bowstreet provide portal development tools.

PureEdge Solutions Inc (July 2005)
PureEdge are open standards forms developers.

LotusSphere comes to you...

Yesterday I attended LotusSphere comes to you in at the Sydney Hilton. Overall I think the half-day was a great success.

If I had to complain about anything it would be that when I got my gift for attending, I left it with my papers while I went to get my lunch (only a few crowded meters away). When I got back, some knave had nicked it. (sigh)...

Well, back to the day...

The Keynote Speech
This was very good and very reassuring. IBM has done a wonderful job over the last year and a bit to push Notes/Domino as well as Workplace. I think we all walked away with a very clear vision of the future.

The EForms Stuff...
I think that a lot of this stuff had people chomping at the bit... E-Forms, though not new, are terribly exciting and I can see that IBM has positioned themselves very well here. Now if only we had some of the E-Forms tools to play with (or a Macromedia Captivate tutorial on creating a form).

Exchange Versus Domino
Ok, it wasn't called this... it was called Selling Notes/Domino in your organization. This wasn't a terribly popular session, but I choose this stream because I wasn't interested in being bombarded with Workplace propaganda. I'll move to Workplace in my own time thank you.

This session was quite useful for me because it gave me some IBM approved phrases to use to describe Notes/Domino. Next time someone says what is Notes/Domino? I can say...

"It's a complete application development environment with email, web and directory services built-in"

There was a little bit of MS-Bashing, but for the most part it was about how to bust the myths around the office.

Some useful things I picked up;

When they say: I use outlook at home...
You say: Yes, but do you do corporate email at home? Do you schedule meetings at home? Do you book rooms from home?
(Not to mention that these are the only areas where outlook and Notes can compete).

When they say: But I prefer outlook/exchange -or- I like
outlook/exchange.
You say: We should really focus on Business issues rather than emotional ones.

The other things that they pointed out were the bleeding obvious, but it's nice to be reminded of them, especially when I do so few, or so poorly...

  • Provide Training for Users - They'll spend most of their time in Notes...

  • Provide Quick Reference Cards

  • Provide Tips of the Week

  • Take the time to find out what the areas of pain are (eg: PDA Synch)



Activity-Based Computing
This is a great idea and probably will be a hit BUT...

  • It wasn't really well presented.

  • It is too far away to be properly relevant.

  • Most people aren't ready to go there yet.

  • There's a strong NON-IT component over which most IT Managers have no control.



I think IBM needs to rethink the way they're pitching the last.

Overall it was a good half-day, though I'll admit that the venue was a bit cramped which made it difficult to meet with the vendors.

Still, Well Done IBM!
I'll be there next year.

Wednesday, March 01, 2006

Fun with Mail and Symantec Premium Anti-Spam

Today we finally resolved our problem getting emails from one of our major contacts. The problem had been going on for most of February, but had generally been unnoticed.

The Symptoms
There generally were no symptoms, people from Company X would send us emails and they wouldn't arrive. There was no mention in the Notes Logs about the emails and there was nothing in the Symantec Anti-Virus software to indicate that there was an issue.

The company had recently changed their domain name from x1.com.au to x2.com.au (all domain names changed to protect the innocent) and we had, relatively recently upgraded to Domino 7.0, so there was much finger-pointing in both directions.

Steps Taken
The first thing I did was look through our firewall logs, but I quickly gave up - they were too difficult because of the sheer number of entries. I tried several methods of dumping the files, but they weren't readable and I couldn't see any filtering options.


My first mistake was to not spend enough time checking these logs, it could have saved a lot of time. The answer to filtering the logs turned out to be a tiny icon that I thought was merely screen decoration. The icon did not popup any 'hover text' which really is a must if you're going to have tiny icons.


So, what did I do?
Well... ...these were over a couple of weeks, not all at once...

  1. Added the new domain to our Symantec MailSecurity Whitelists

  2. Added the new domain to our Domino DNS Whitelists

  3. Reinstalled Symantec Mail Security 4.x - In case something was overwritten in the 6.5.4 to 7.0.0 upgrade.

  4. Upgraded to Domino 7.0.1 - To fix the security issues blogged about earlier

  5. Installed Symantec Mail Security 5.0 - Because it was supposed to be more ND7 Compatible

  6. Re-Signed the Databases using an Admin ID instead of the Server.id

  7. Replaced the Design of the Symantec Databases and ran fixup for good measure



Nothing worked.

What did work was to completely disable the Symantec MailSecurity Application but for obvious reasons we didn't want to do that.


Tracking the Problem
The Firewall logs reported the mail as incoming.
From the firewall, the mail goes straight to Symantec MailSecurity on Port 25.

Once MailSecurity has finished it's work, the messages should be forwarded to the Domino server on Port 26. Unfortunately, MailSecurity was simply killing/losing the message, and thus there was nothing in the Notes log to indicate that a message had been received.

Eventually, we disabled just the Premium Anti-spam.
The result was a server crash (ouch).

I had to kill the server task using

NSD.EXE -KILL

When the server came back up, it was able to receive mail from the affected domain.

Hypothesis
The problem seems to be that when they changed domain name, they also changed their email disclaimers. Something in their disclaimers was upsetting the Premium Anti-Spam component of Symantec MailSecurity.


Moving Forward...
Once Premium Anti-Spam was disabled all went well.

I've lost a lot of respect for this utility since it is the second time that it has been responsible for major Domino issues and only the first time in 5.5 years that our server has had a serious crash.

Amusingly, one of the reasons for my intense dislike of this component is that it requires Microsoft SMTP. Funny that having a small Microsoft Mail facility on the server should cause major issues twice in about 6 months, yet Domino itself has not.

The software is disabled and will stay that way unless we get overwhelmed with Spam.
We will not be renewing that license.

(We will renew the Symantec MailSecurity licence, just not the Premium Anti-Spam component).