About 165,000 Web Sites Knocked Offline By Navisite Outage

[Thaising]

Approximately 165,000 Web sites have been offline since Saturday, thanks to a failed data center migration involving Andover, Mass.-based Web hosting company NaviSite Inc.

The problems started Saturday when NaviSite attempted to migrate and replace hundreds of servers operated by Baltimore-based Alabanza Corp., a Web hosting company acquired by NaviSite in August.

According to NaviSite spokesman Rathin Sinha, NaviSite decided to physically move 200 of the 850 servers operated by Alabanza to NaviSite's data center in Andover and then virtually migrate the data from the rest of the older servers to new boxes, also in Andover.

NaviSite let its customers know that their sites would be down for a while on Saturday, with the migration expected to be finished that day, Sinha said. But when NaviSite attempted to transfer the data from the 650 servers still in Baltimore it ran into a number of synchronization failures that kept multiplying.

As Saturday progressed, NaviSite realized it would probably miss its completion deadline; as a result, company officials decided to physically transfer another 200 servers from Baltimore to Andover to help reduce the scope of the virtual migration and speed up the data transfer.

But then NaviSite ran into more problems. As the hosts came up, their URLs did not, so although customers could access their Web sites from their IP address, they could not do so using their URLs, Sinha said.

"That was unanticipated," he said.

As NaviSite tried to solve that problem, the network became overloaded because of all the customers trying to get online, Sinha said. "What happened was first the URL could not match with the IP address and then IP did not match with the machine, so it took some time, and all this time we have a highly trafficked overloaded network," he said. "If there is one little problem, they multiply because there is a lot of dependencies."

Although Sinha said a "big chunk" of sites are back online, he could not say when everything might be back to normal. He also couldn't say how much this failed migration would cost -- NaviSite is a publicly-traded company.

To put it mildly, one of NaviSite's customers, Cynthia Brumfield, president of Emerging Media Dynamics Inc., an analyst firm in Washington, seems to be furious.

In an interview, Brumfield said she's going into her fourth day without access to her Web sites. And she said she doesn't believe the way NaviSite is spinning the story. While NaviSite said it has brought a large number of Web sites back online, she claims it hasn't.

"According to people who have talked to NaviSite's tech personnel, they were ill equipped for the relocation and ignorant of how to accomplish even basic tasks," she said in a blog post. "At this point, NaviSite's poorly planned data center consolidation has slipped from mere incompetence to outrageous indifference to its customers' needs and should be grounds for legal action, if not government sanctions of some kind."

Brumfield said that, in effect, NaviSite yanked the servers for 200,000 Web sites, put them on trucks and then didn't know what to do once the servers arrived in Andover. "But what's worse, NaviSite had informed its clients of a completely different timetable and process for the server relocation than the one implemented," she said in the blog posting.

In the interview, Brumfield said that because all of her backup files are also stored on Alabanza's servers, she has no choice but to hop on a plane to Boston on Wednesday and drive to Andover to retrieve her data. And she said she's bringing a video camera with her to document NaviSite's response to her request.

Could these be the problems where some posters can't enter wiki and other websites?

Edited November 9, 2007 by Thaising

[Guest Reimar]

I don't think this is the problem for wikipedia!

Ok, I can't connect to Wikipedia at TRUE from Samut Prakarn but I can from Bangkok on True and from Minburi at TOT without any problems. But I found out that my computer at Samut Prakarn having some problems with Firefox and even re-install of Firefox isn't solve this problems.

The report above shows alot mistakes which was done by NaviSite! How much NaviSite could be held responsible is one point. But how much NaviSite will compensate and apologize to the customers is am other pont.

A few years ago one of my bigger customer was running the Website and the Mail-Server by Enom USA. One day no any E-Mail was arrivinge, the normal amount of one working day for this company was app. 6,000 e-mails! We was able to open the website not even NOT the Webmail. That was showing that the Servers was working except the Mail-Server! Trying to contact Enom via Web impossible! The Ticket Systems wasn't work! Even calling to the company via all known phone numbers: Not anyone was picking up the phone! And all of this for 7 day's!

Final after 7 days the system was back online but: NOT any apologize or even a simple excuse!

My customer was starting a legal action against Enom but until today not any movemant! And thats now 4.5 years! US American Justice!