Monday, June 25, 2007

The Move - Complete

My blogging has slowed down considerably lately due to an office move which used up most of my time. There were two aspects of the move;

1. The move itself
Whereby all equipment had to be disconnected and packed up then unpacked/reconnected at the new site. There was a lot of planning and testing involved with this.

2. A Temporary Domino Server Switch
In some ways, this was more work than the move. Management decided that they wanted our systems to remain running throughout the move. We used the server setup for the "temporarily abandoned" clustering project to set up a new offsite domino server. We did a lot of testing of applications and needed to make a few changes to some of our applications.

I'm happy to say that the off site server worked well and kept our systems running throughout the move. I'll provide more detail on changes made and difficulties encountered in later posts.

Tuesday, June 05, 2007

When your ISP tries to kill you....

I've just edited and anonymised my last post in case I can't control myself here...
If you haven't read it, you should read that one before this one.

We've had no SMTP inbound mail since lunchtime yesterday.

We contacted our ISP (PT) but they weren't interested in the problem. They told me to reboot the Domino server....

No thanks.

Internal mail is working fine.

I tried using a VB Application to send SMTP mail to the server's internal IP Address. It worked fine - proving that if mail were reaching the server it would process it.

All indications were that there was a duplicate IP on the internet - but that couldn't be right - we got that problem fixed yesterday...

After hours of wasted IT time, we finally discovered the truth. Our ISP (PT) who duplicated our IP Address yesterday had removed the address as requested. They then put the address back into their system and allocated it to a new customer.

I can't think of a closing statement that wont get me in trouble with the law so I'll leave it up to your imaginings.

Monday, June 04, 2007

When your ISP is your own Worst Enemy...

[This post was edited to provide anonymity to the ISP - Not that they deserve it]

Our current internet service provider, (I'll call them PT to anonymise them in case this blog takes too much of an annoyed tone) is a medium sized provider in Australia. When I say current, I don't mean that we plan to change... heck, we only change ISP's once, but have had five ISPs. We made a switch from Ozemail shortly after they were taken over by a less service-conscious entity (can't remember who). From there, we moved to Davnet, who were taken over by NTT Australia, who were taken over by PT. All of the service we've received since the initial switch to Davnet has been top-quality. Until, of course, we get to PT.

In our early days of PT service, we were affected by an unplanned outage that lasted approximately 2 days. It was (and still is) the biggest total outage in my computing history. Even worse, they didn't seem to be in control of the situation but were at the mercy of other (larger) ISPs. That outage led to a three-year period of not signing a proper agreement with them in case we decided to ditch them.

Last year, when most people affected had either forgiven/forgotten the outage, or left the company, we signed a two year agreement. As is typical of these things, management decided to move offices shortly after signing the agreement. I'm sure that there will be plenty more blog posts to come detailing the effects of this move. One important aspect of the move is that we needed to have an internet connection available in our new office. We renegotiated with PT and the connection was installed last Friday.

Coincidently, from around midday last Friday, our current internet connection became very flaky indeed. It would not find a lot of sites at all and would render others at a snail's pace. One of our first thoughts on the matter was that the problem could be associated with the new connection. Sure, we'd filled out the documents and written very clearly that it was not to become active until after our moving date - but who knows, accidents happen right?

We contacted PT technical support who assured us that this had not happened. At any rate, several of our staff members had spoken to the technicians on-site before the event and reminded them not to make the connection active. If there was any doubt, the fact that they were installing in an empty half-renovated building would have provided some useful visual clues.

We spent the remainder of Friday troubleshooting our own network, hampered by the fact that we didn't realize the effects of updates to our local PC software firewalls on our ability to ping and tracert. (We figured that out after quite a bit of conflicting data).

There were other connectivity problems in Australia last weekend with my own local ISP (OPNet) dropping the ball locally for almost an entire day. This seriously impaired my ability to work on the problem.

This morning (Monday) the problem was there waiting for us when we started work and amid cries of anguish from staff who couldn't check their stock reports, we set to work. Of course, we're not stupid... we did call our ISP four times and ask specifically about the impact of the connection in our new office on our current connection. They assured us that it wasn't a problem and that there were no traffic issues. It also followed that since they could access the internet, the problem must be at our end.

Several experiments and many hastily redrawn contingency network diagrams later, we decided to take the plunge and disconnect our systems from the internet - replacing our firewall with a single laptop which plugged directly into the ADSL modem. We were expecting some serious surfing speed improvements but saw none... the problem was still there.

A last call back to PT pointing the finger firmly in their direction made them sit up and take notice....

guess what...

they had connected that other site after all... and guess which public IP address they allocated...

Sigh.