TOT power cut hits 750,000 internet users

[george]

TOT power cut hits 750,000 internet users

BANGKOK: -- A power cut at a TOT Plc data centre on Saturday evening left about 750,000 internet users with no connections, according to TOT president Yongyuth Wattanasin.

He said the main and supplementary power systems were cut shortly before 7pm, so customers nationwide were unable to use the internet. Mr Yongyuth said the cut would not affect the estimated 5,000 ATMs that the state telecom agency serves according to a report in Thai Rath.

He said maintenance would take only 15 minutes if technicians could access the damaged site but as of 9pm, the system had not yet been reconnected. It was not clear whether the power cut was related to actions by political protesters. Anti-government marchers earlier in the day seized the headquarters of TOT and its sister agency CAT Telecom.

Earlier on Saturday, anti-government protesters drew heat from internet users after power was cut to a CAT Telecom facility in Bang Rak.

Several servers went down and international gateways were knocked out of service. However, backup generators were deployed and service began to return to normal after about two hours.

-- Bangkok Post 2013-12-01

link:http://www.bangkokpost.com/breakingnews/382466/power-cut-knocks-out-tot-internet.

[noitom]

We want to hear from the CEO and the government PM in charge of communications that it has been determined not to be a terrorist act.

[Lite Beer]

Power cut at CAT disrupts Internet
The Sunday Nation

BANGKOK: -- A Blackout allegedly caused by anti-government protesters at the CAT Telecom office in Bang Rak yesterday afternoon affected its Internet server, a Cabinet minister claimed.

The power cut at CAT in Bang Rak affected True’s mobile users, as the firm’s mobile node is there. It affected 62 per cent of True’s coverage in Bangkok. The system had not been restored at 8pm.

Information and Communications Technology Minister Anudith Nakornthap said the widespread disruption of CAT Internet was because a group of anti-government protesters had cut power to the international server in the CAT Telecom building and the back-up power supply system was also cut off. Officials were repairing the system at the time of going to press and power had been restored to some parts of the building.

However, Satit Wongnongtaey, a key member of the People’s Democratic Reform Committee, denied any involvement with the power cut.

Witnesses said a group of men came to the building but they could not be identified.

Another account from a witness at the building said that a group of Silpakorn University students were presenting works at the building's eighth floor when suddenly the lights went out and rumours started doing the rounds that anti-government protesters had cut the power supply.

A witness looked out of the balcony and saw some 30 trucks of protesters giving a speech in front of the building for 30 minutes. The blackout continued for over an hour even after they had left

-- The Nation 2013-12-01

[Tokay]

I have True fiber optic and we lost out internet for a little while (a first) at the same time. I wasn't aware True used CAT or TOT bandwidth. Oh well. *sigh*

[bkkjames]

I have True fiber optic and we lost out internet for a little while (a first) at the same time. I wasn't aware True used CAT or TOT bandwidth. Oh well. *sigh*

I think you will find on True's website, documentation etc that they call themselves a CAT reseller....

[timoclark]

So how is it that only The Nation, Manager.co.th, Bluesky Channel, and Pantip.com websites, used by the opposition to share information, were on the server that was targeted? ..... and this building requires a fingerprint scan to enter. If you try to access pantip.com from within Thailand now, you will get an ICT message saying that basically saying that they are watching you and what you post on Pantip, and will watch as to whether you are "crossing the line." Fishy.

[EyesWideOpen]

The Arab spring was fueled by the internet , and Facebook in particular which was used to exchange messages on

the uprising. If this government is about to topple, the first thing they will do is cut the internet.

[Somtamnication]

I am not worried about these thugs cutting off the internet, because it can be restored. What I am worried about is the govt or military turning the switch off.

Now THAT will worry me.

[Iamemjay]

i am on a TOT connection but out in the rural didnt notice anything at those times so it must have just been bkk that was affected

We are in the boonies and we lost internet access for 6 hours - not the two hours that has been reported

[jacko45k]

However, backup generators were deployed and service began to return to normal after about two hours.

Normal, in my case with TOT, is still pretty poor!

And still is

Edited December 1, 2013 by jacko45k

[canuckamuck]

I was about to uninstall my antivirus (free trial) and install a version I had purchased instead. So I decided to make sure the internet signal was good, so I could do the updates (internet is flaky here sometimes). So I checked the internet and it was good. I did the uninstall and then the internet disappeared.

I thought I had messed up my system. I am relieved to see that it was TOT's problem instead.

[Yunla]

I'm on True internet too, and the connection here was laggy and sporadic R-0s from evening in through the night. I wasn't sure if it was the peaceful democracy ppl smashing up the ISP box, or if it was the ruling dictatorship that finally got round to shutting down the web for our own goods.

[taiping]

My CAT fibre-optic Internet connection in the Ekamai area was cut-off about 3:30 pm on 30/11/13 and was only resumed at 8:30 am on 1/12/13.

All my contact phone numbers to to CAT were down. I got through to a CAT technician's mobile phone, and he said the CAT power had been shut down as a precautionary measure.

This is all causing untold damage to the country.

[haroldc]

The statement by the ICT Minister is absurd. If the external power supply to CAT's servers was cut off (by the way, exactly how was it cut off?), battery-powered uninterpretable power supplies should have begun supplying power instantaneously. Simultaneously, back-up diesel generators should have started automatically and, once running and stabilized, should have automatically begun supplying power to the servers and to the uninterpretable power supplies for re-charging the batteries.

It is inconceivable that the external power supply, the uninterpretable power supplies, and the diesel generators could all fail simultaneously unless 1) they were intentionally disabled by an agent provocateur with knowledge of and access to all three systems or 2) CAT's infrastructure is so poorly designed and/or maintained that it's entire management should be fired for incompetence and gross negligence.

FYI, a have more than 20 years' experience designing high-availability data centers.

Edited December 1, 2013 by haroldc

[huahinjoe]

I think screwing with the net was really unfair. Feeling they deliberately spitting into our coffee, so Sir Suthep send your thugs back to SuthepThani to cut rubber trees there instead internet cables in major datacenters. Causing distruption for the majority innocent ain't gonna help dude to shine you and your sheep's future...

[GeorgeO]

I have True fiber optic and we lost out internet for a little while (a first) at the same time. I wasn't aware True used CAT or TOT bandwidth. Oh well. *sigh*

I also figured it was protest related when I lost my connection last night. However, I am happy to accept some little inconvenience if their overall aim is achieved.

Has anyone seen YL and her cronies heading for the airport yet?

[SjaakNL2013]

We want to hear from the CEO and the government PM in charge of communications that it has been determined not to be a terrorist act.

Noitom, who is WE ?

[Tiffer]

Many people have reported a significant drop in connection speed / availability over the last 10 days or so. As this comes on top of a generally poor standard of service, a few hours without the internet could easily go unnoticed ^^

[TallGuyJohninBKK]

I don't know why CAT and TOT's power went down yesterday, and I wouldn't be too quick to believe everything the government says about the reason.

Maybe it was the protesters. Maybe it was the government. Maybe it was CAT themselves screwing up. In these kinds of situations, it's wise to avoid rash judgments based on early media reports, that often turn out to be wrong or misinformed.

The government here has publicly stated they want to fight the protesters with negative publicity. Presumably, that's why the media suddenly yesterday had reports about YL's kid supposedly being targeted... and now the Internet outage.

Those are the kinds of issues that common ordinary people would not think well of. And if the government can succeed in blaming the protesters for those kinds of things, they're scoring points in the PR war.

After all, when it comes to actually running the country, they really don't have much of a (good) record to stand on.

Edited December 1, 2013 by TallGuyJohninBKK

[user82374298374]

I'm a network engineer, and I know first-hand what happened. Not rumors or speculation.

A group of anti-government protestors went to the CAT Internet Data Center (IDC)/Point of Presence (PoP) in Bangrak; they got someone who could access the building to let them in, and then they proceeded to cut the power mains and to remove the control panels for the generators. UPSes for various servers could run for a while, but since the other routers, switches, etc. they connected to were down, it didn't matter.

The Bangrak facility is what's known as a carrier hotel, where lots of various ISPs peer together, and CAT operate the International Internet Gateway, or IIG, there. IIG went down, which completely took down international peering for a lot of smaller Thai ISPs, and it took about 50% of True's international capacity (True has its own international peering points in Singapore, the USA, etc.).

Lots of companies like Thai Airways have their Web and DNS servers colocated within the Bangrak IDC, so they were down, too.

Backup generators were dispatched and installed and fired up, and the commercial power mains connectivity was restored, as well. After some routing reconvergence and dealing with some routers/switches which needed configuration tweaking, things were pretty much back to normal. True Move had some switches which weren't behaving properly at first, so they temporarily downgraded mobile data users to EDGE speeds until they could get the problems resolved

You see, the protestors didn't realize that if they caused Internet disruption in Thailand, they couldn't use Facebook, Twitter, WhatsApp, Line, et. al. to coordinate their activities, and their television channels would go off the air, as well, because they use the Internet to deliver video, and most of their infrastructure is colocated within the Bangrak facility.

The protestors' interruption of power at TOT affected wireline home and commercial broadband users of TOT's branded Internet access service, not IDCs or Internet peering points, nor mobile wireless users. Power has been restored there, as well.

Both (all?) sides in this conflict need the Internet in order to use social media to coordinate their activities, and the government needs the Internet to get their message across, as well as to perform normal governmental functions, for online commerce involving orders for Thai exports, et. al., to proceed apace. Just as the protestors in 2008 who took over Suvarnabhumi ended up causing huge losses and disruption for the entire country, including their patrons, the protestors who were causing Internet disruptions last night were causing the same kind of problems for their own supporters. I think they've learned their lesson now, and both sides will leave key Internet infrastructure alone, just as they're leaving Suvarnabhumi alone.

[aonangkrabi]

In Krabi CAT internet is still unavailable, so I send this message using my back-up Mifi router that connects to DTAC 3G.

Are there more parts of the country that are still without internet?

[besth]

My CAT fibre-optic Internet connection in the Ekamai area was cut-off about 3:30 pm on 30/11/13 and was only resumed at 8:30 am on 1/12/13.

All my contact phone numbers to to CAT were down. I got through to a CAT technician's mobile phone, and he said the CAT power had been shut down as a precautionary measure.

This is all causing untold damage to the country.

In Jomtien, my CAT service cut off about the same time. Back about 1/2 hour ago.

CAT number only gave a recorded message "sorry for the inconvenience, we are working blah, blah, blah"

[PaullyW]

The statement by the ICT Minister is absurd. If the external power supply to CAT's servers was cut off (by the way, exactly how was it cut off?), battery-powered uninterpretable power supplies should have begun supplying power instantaneously. Simultaneously, back-up diesel generators should have started automatically and, once running and stabilized, should have automatically begun supplying power to the servers and to the uninterpretable power supplies for re-charging the batteries.

It is inconceivable that the external power supply, the uninterpretable power supplies, and the diesel generators could all fail simultaneously unless 1) they were intentionally disabled by an agent provocateur with knowledge of and access to all three systems or 2) CAT's infrastructure is so poorly designed and/or maintained that it's entire management should be fired for incompetence and gross negligence.

FYI, a have more than 20 years' experience designing high-availability data centers.

I have some similar background, and I've NEVER come across a Thai firm (or Singaporean, Malaysian, Indonesian, Chinese) firm that would invest the money in these backup and backup to the backup systems. Being what they are, they are simply too cheap or don't understand.

In my days of doing this type of consulting around Asia, I'd resigned myself to just saying what I had to say, expecting no agreement and moving on -- so long as my recommendations were made formally (to cover my butt).

Thais are probably the worst of the bunch, though, with the Indonesians being a close second. They just cannot/will not take any precautions. Same on the roads. Same throughout much of Thai life. It's how they like to live.

[jacko45k]

FYI, a have more than 20 years' experience designing high-availability data centers.

But did you ever work for TOT?????

[aaacorp]

In samed Tot is still down returning unavailable dns .

[user82374298374]

There are still lingering problems for some broadband users, mainly outside of Bangkok, because of routers/switches which need some remediative configuration because they didn't automagically re-establish connectivity when upstream devices came up after power restoration.

[pomchop]

Great job Thugs? What's next shut down the airport again? March to the US Embassy? ....and YOU want to oust the current government and hand pick your flunkies with no elections?

I don't think so.