Skip to main content

Firewalls and Other Single Points of Failure

Last Friday night, we installed a new firewall to replace our "unsupported" Symantec model.

A Bizarre Sense of Timing
Curiously, we'd been complaining to Symantec for years about several of their systems and had been slowly ripping and replacing. Our complaints over their lack of interest in the firewall they sold us had been fairly strong over the last six months but had met with limited responses.

I got a call out of the blue on Friday afternoon from "upper" management telling us that they were now committed to sorting out the firewall for us.

I had to tell them that their firewall only had three hours of life left. I didn't feel guilty though. I don't think that anyone should be willing to tolerate a lack of service.

A Simple Swap-over
We cleverly timed the firewall swapover project to coincide with a weekend where there was a scheduled power outage. This meant that instead of having one firewall (point of failure) on site, we had two.

We spent a while reviewing the rules and it should have been a simple swap-over of cables. It should have been....

but it wasn't.

At first the firewall wouldn't connect - to the internet. Fine, it's a reasonable assumption that you'd need to cycle power on the router. The problem... well, since we had backup for the firewall, it follows that the router becomes the single point of failure.

The router didn't come up properly - all lights remained solid. We tried a number of power cycles but it steadfastly refused to come up.

Luckily it was a managed router and we were still well within our time limit for 24x7 serivce.

The Helpless Helpdesk
We rang our ISP's helpdesk but went through to another state - Victoria. Apparently they couldn't raise anyone in their Sydney office and the technician suggested that they may have left their 24x7 helpdesk to go for "Friday drinks". (sigh). They promised to get in touch as soon as possible.

We got a call back over 90 minutes later and received the incredibly technical description that "your router has fried".

Great! It's a managed router, so kindly deliver a new one... Right???

Wrong! Sorry, we're fresh out of routers. (It felt like we were in Monty Python's Cheese shop sketch). We were told that it was unlikely that they'd be able to come up with a router before Monday.

"Ok" we said, "not what we were hoping for... what time do your techs start on Monday?". I think we were all stunned by the response of 9am. (I'm a 6.30am starter myself).

We then started trying to negotiate, suggesting that if we could buy a replacement router over the weekend, we could install it ourselves if they'd give us the relevant passwords and settings. "Sure", they replied, "but it will cost $350 to change you from a managed to unmanaged router". We figured that this was an acceptable cost and told the technicians. They quickly killed the idea by telling us that they wouldn't be able to give us any connection info.

We were stuck.

Luckily, we had one of those "super-tech's" onsite at the time, you know the type who like to open all their equipment just to see where the motherboard was made... He decided to open our "fried" router and prod around for bloated capacitors. There were none.

Of course, one of the big rules about electronic equipment is that sometimes, when it's hot, it just wants a bit of nudity. With the cover off and the circuit board exposed, we plugged the router back in - it immeidately hummed into life.

The Power Outage
So now we had a working firewall and connection but an impending power shutdown lasting up to 6 hours... would the equipment come back online? Would the motherboards cool down and pop all their solder? I had to wait until Sunday to find out.

On Sunday morning, I arrived at work and started switching all the equipment back on. Desktops, Printers, etc... it all came back... all except the computer room that is.

It turned out that the building's power had flipped a fuse, so I had to call the electricians back. Once fixed, everything, even our very sick router, came back.

I was going to suggest that the moral of the story is that you need to have more than one piece of hardware that matches the spec of devices which are single points of failure but I think the moral is really... whenever there is an opportunity for the single point of failure to make its presence known... it will.

Comments

Popular posts from this blog

How to Change Your Notification Options for New Lotus Notes Mail in version 8.x

Don't worry, I'm not patronizing you (my readers), I just decided to re-document this for one of our internal users and thought you might want to be able to use it in your own user documentation. WHAT IS THIS DOCUMENT ABOUT? Some people who don't get a lot of mail, like to be notified when such an event occurs. Notification can be; via a sound via a pop-up box via the system tray (where the computer clock is) The pop up box looks like this; Other people, who like myself, get too much mail would rather not be notified. The aim of this document is to tell you how (and where) to turn these options on and off. CHANGING YOUR SETTINGS To change your settings from the Notes 8.x client; On the Menu, click File , then Preferences... On the left hand side , click on the little plus sign to the left of Mail to expand the options. Click on the option marked Sending and Receiving . In the middle section, under receiving, you can control your notifications. If you untick the box mark

How to Create an Auto-Response Mail Message in Lotus Notes 8.5.3+

Why would you do this? Suppose that you have an externally accessible generic email address for your company; support@mycompany.com or info@mycompany.com. You might expose this to the web and allow people to send messages to you. Setting up an auto-response email will tell the senders that their message reached its destination and that it will be dealt with accordingly.  It's also good practice to include links to FAQs or other useful information. Why 8.5.3 The techniques we'll be using here work in older versions of Notes but some of the options seem to have moved around in 8.5.3.  I figured it was a good time to show you where they've moved to. The Procedure Start Domino Designer and open the Mail file to be modified.  A really quick way to do this is to right-click on the application tab and choose "Open in Designer". In the Left hand panel of designer, expand Code and then double-click Agents.  A new window should appear. Click the action

How to Do a Mail Merge to Email using Lotus Notes

Why do one? In today's "green" world, it makes much better sense to send out emails than letters but you still want to personalize them. Sadly, by itself Lotus Notes doesn't support mail merge to email. Of course, we know that outlook does (but then it lets anyone and anything send emails for you - even when you don't want them to). So, how to do it in Notes? OpenNTF The first port of call is OpenNTF ( http://www.openntf.org/ ). This place is full of great things but most of them are really badly documented. Still, these guys give things away for free and they develop in their spare time, so we should be grateful for what we get. There's a great little project there called MailMerge Excel to Notes . Go there, click on releases and download the ZIP file. Getting to the Code The installation is tricky though I've noted that since I asked the author about the install, it's been updated (so maybe these steps are less necessary). Unzip the files to somewher