Monday, June 04, 2007

When your ISP is your own Worst Enemy...

[This post was edited to provide anonymity to the ISP - Not that they deserve it]

Our current internet service provider, (I'll call them PT to anonymise them in case this blog takes too much of an annoyed tone) is a medium sized provider in Australia. When I say current, I don't mean that we plan to change... heck, we only change ISP's once, but have had five ISPs. We made a switch from Ozemail shortly after they were taken over by a less service-conscious entity (can't remember who). From there, we moved to Davnet, who were taken over by NTT Australia, who were taken over by PT. All of the service we've received since the initial switch to Davnet has been top-quality. Until, of course, we get to PT.

In our early days of PT service, we were affected by an unplanned outage that lasted approximately 2 days. It was (and still is) the biggest total outage in my computing history. Even worse, they didn't seem to be in control of the situation but were at the mercy of other (larger) ISPs. That outage led to a three-year period of not signing a proper agreement with them in case we decided to ditch them.

Last year, when most people affected had either forgiven/forgotten the outage, or left the company, we signed a two year agreement. As is typical of these things, management decided to move offices shortly after signing the agreement. I'm sure that there will be plenty more blog posts to come detailing the effects of this move. One important aspect of the move is that we needed to have an internet connection available in our new office. We renegotiated with PT and the connection was installed last Friday.

Coincidently, from around midday last Friday, our current internet connection became very flaky indeed. It would not find a lot of sites at all and would render others at a snail's pace. One of our first thoughts on the matter was that the problem could be associated with the new connection. Sure, we'd filled out the documents and written very clearly that it was not to become active until after our moving date - but who knows, accidents happen right?

We contacted PT technical support who assured us that this had not happened. At any rate, several of our staff members had spoken to the technicians on-site before the event and reminded them not to make the connection active. If there was any doubt, the fact that they were installing in an empty half-renovated building would have provided some useful visual clues.

We spent the remainder of Friday troubleshooting our own network, hampered by the fact that we didn't realize the effects of updates to our local PC software firewalls on our ability to ping and tracert. (We figured that out after quite a bit of conflicting data).

There were other connectivity problems in Australia last weekend with my own local ISP (OPNet) dropping the ball locally for almost an entire day. This seriously impaired my ability to work on the problem.

This morning (Monday) the problem was there waiting for us when we started work and amid cries of anguish from staff who couldn't check their stock reports, we set to work. Of course, we're not stupid... we did call our ISP four times and ask specifically about the impact of the connection in our new office on our current connection. They assured us that it wasn't a problem and that there were no traffic issues. It also followed that since they could access the internet, the problem must be at our end.

Several experiments and many hastily redrawn contingency network diagrams later, we decided to take the plunge and disconnect our systems from the internet - replacing our firewall with a single laptop which plugged directly into the ADSL modem. We were expecting some serious surfing speed improvements but saw none... the problem was still there.

A last call back to PT pointing the finger firmly in their direction made them sit up and take notice....

guess what...

they had connected that other site after all... and guess which public IP address they allocated...

Sigh.

No comments: