There generally were no symptoms, people from Company X would send us emails and they wouldn't arrive. There was no mention in the Notes Logs about the emails and there was nothing in the Symantec Anti-Virus software to indicate that there was an issue.
The company had recently changed their domain name from x1.com.au to x2.com.au (all domain names changed to protect the innocent) and we had, relatively recently upgraded to Domino 7.0, so there was much finger-pointing in both directions.
The first thing I did was look through our firewall logs, but I quickly gave up - they were too difficult because of the sheer number of entries. I tried several methods of dumping the files, but they weren't readable and I couldn't see any filtering options.
My first mistake was to not spend enough time checking these logs, it could have saved a lot of time. The answer to filtering the logs turned out to be a tiny icon that I thought was merely screen decoration. The icon did not popup any 'hover text' which really is a must if you're going to have tiny icons.
So, what did I do?
Well... ...these were over a couple of weeks, not all at once...
- Added the new domain to our Symantec MailSecurity Whitelists
- Added the new domain to our Domino DNS Whitelists
- Reinstalled Symantec Mail Security 4.x - In case something was overwritten in the 6.5.4 to 7.0.0 upgrade.
- Upgraded to Domino 7.0.1 - To fix the security issues blogged about earlier
- Installed Symantec Mail Security 5.0 - Because it was supposed to be more ND7 Compatible
- Re-Signed the Databases using an Admin ID instead of the Server.id
- Replaced the Design of the Symantec Databases and ran fixup for good measure
What did work was to completely disable the Symantec MailSecurity Application but for obvious reasons we didn't want to do that.
Tracking the Problem
The Firewall logs reported the mail as incoming.
From the firewall, the mail goes straight to Symantec MailSecurity on Port 25.
Once MailSecurity has finished it's work, the messages should be forwarded to the Domino server on Port 26. Unfortunately, MailSecurity was simply killing/losing the message, and thus there was nothing in the Notes log to indicate that a message had been received.
Eventually, we disabled just the Premium Anti-spam.
The result was a server crash (ouch).
I had to kill the server task using
When the server came back up, it was able to receive mail from the affected domain.
The problem seems to be that when they changed domain name, they also changed their email disclaimers. Something in their disclaimers was upsetting the Premium Anti-Spam component of Symantec MailSecurity.
Once Premium Anti-Spam was disabled all went well.
I've lost a lot of respect for this utility since it is the second time that it has been responsible for major Domino issues and only the first time in 5.5 years that our server has had a serious crash.
Amusingly, one of the reasons for my intense dislike of this component is that it requires Microsoft SMTP. Funny that having a small Microsoft Mail facility on the server should cause major issues twice in about 6 months, yet Domino itself has not.
The software is disabled and will stay that way unless we get overwhelmed with Spam.
We will not be renewing that license.
(We will renew the Symantec MailSecurity licence, just not the Premium Anti-Spam component).