Skip to main content

The Perfect Storm: How a Combination of Random Events can lead to Disaster

No, this post isn't about that Wolfgang Peterson & George Clooney movie but it does borrow from the concept in a business sense.

This week we skirted close to disaster but the rest of the business was completely oblivious to the danger.

I wasn't worried, it was all under control but my boss said to me later, "we nearly had the perfect storm today". I thought about his words and realised that he was right.

I was wondering just how many times we, and other businesses nearly have the perfect storm and don't even realise.


What is a Perfect Storm?
In the book and movie of "The Perfect Storm", the storm wasn't a cyclone or any kind of normal severe storm event. It was just a normal storm in which several other conditions were perfect. It's simply the combination of several unlikely events which results in disaster.


Our Experience
Factor 1: Backup
This week we've struggled a bit with backup. We had a faulty tape which meant that we missed out on our backup job for one night. I replaced the tape for the next night and tried to run the backup again. It failed again - this time because the previous night's tape had left a bit of gunk in the drive. I cleaned the drive but although the system was ready for backup, we now had two days worth of "unsaved work".

Factor 2: Backup ISP and Server
Like many critial businesses, we have offsite redundancy. In our case, we have an offsite domino server which is part of our cluster. Our offsite provider had told us a little while ago that they needed to switch ISPs and re-run our infrastructure. We were told a month ago that there was no pressing urgency. Of course last week, on the night of our last successful backup, we were told that it was starting to become urgent.

You can imagine our surprise when we came into work on Thursday to find ourselves disconnected. Our main systems worked, the internet worked, everything was Ok. It's just that our redundant server was no longer accessible.

Factor 3: Board Day
I work at one of those lucky companies who don't have board members in attendance every day. In fact they're usually only around on monthly basis. Guess what.... the day of the storm was a board day.


A Lucky Escape
We were careful, we took precautions. Earlier in the week we'd had some work done on our main production databases but on that day, they were out of bounds.

Like I mentioned earlier, nothing happened. The production servers all stayed up, we eventually got our offsite server back and we got a backup that night. All was well but you have to think...

If anything had gone wrong on the day, we had no backup and no offsite server. We were hosting one of the most critical meetings of the year and the most important people in our company were all onsite watching.

The question is; would you recognize the perfect storm if it started forming near your company?

Comments

Post a Comment

Popular posts from this blog

How to Change Your Notification Options for New Lotus Notes Mail in version 8.x

Don't worry, I'm not patronizing you (my readers), I just decided to re-document this for one of our internal users and thought you might want to be able to use it in your own user documentation. WHAT IS THIS DOCUMENT ABOUT? Some people who don't get a lot of mail, like to be notified when such an event occurs. Notification can be; via a sound via a pop-up box via the system tray (where the computer clock is) The pop up box looks like this; Other people, who like myself, get too much mail would rather not be notified. The aim of this document is to tell you how (and where) to turn these options on and off. CHANGING YOUR SETTINGS To change your settings from the Notes 8.x client; On the Menu, click File , then Preferences... On the left hand side , click on the little plus sign to the left of Mail to expand the options. Click on the option marked Sending and Receiving . In the middle section, under receiving, you can control your notifications. If you untick the box mark...
12 comments
Read more

How to Create a Bootable DVD Using Nero Burning ROM 9

I often need to create bootable CDs and DVDs but it's weird because I frequently end up buring myself a new coaster instead. It's not that the process is difficult, just that nero has a few too many options and I forget which ones to choose and end up picking the wrong one. I figured that the best way to avoid this mistake in future would be to write the steps down. Procedure Insert CD or DVD into your DVD Burner. Start Nero Burning ROM 9 Choose DVD-ROM (Boot) or CD-ROM (Boot) depending on what you're creating You'll be prompted for a disk image source. Choose a Nero Source - you'll usually find them somewhere like this... C:\Program Files\Nero\Nero9\Nero Burning Rom\DOSBootImage.ima Leave the Boot Locale as English - unless you really need a different keyboard layout Tick the box marked [X] Enable Expert Settings Choose Hard Drive Emulation and leave any other settings as they are. Click the button marked New Add any files you want but don't try to add operati...
11 comments
Read more

How to Create an Auto-Response Mail Message in Lotus Notes 8.5.3+

Why would you do this? Suppose that you have an externally accessible generic email address for your company; support@mycompany.com or info@mycompany.com. You might expose this to the web and allow people to send messages to you. Setting up an auto-response email will tell the senders that their message reached its destination and that it will be dealt with accordingly.  It's also good practice to include links to FAQs or other useful information. Why 8.5.3 The techniques we'll be using here work in older versions of Notes but some of the options seem to have moved around in 8.5.3.  I figured it was a good time to show you where they've moved to. The Procedure Start Domino Designer and open the Mail file to be modified.  A really quick way to do this is to right-click on the application tab and choose "Open in Designer". In the Left hand panel of designer, expand Code and then double-click Agents.  A new window should appear. Click the action ...
22 comments
Read more