British Airways vows "never again" after costly IT collapse

[paulbj2]

First observation, this HAS happened to British Airways before - some years back, if I recall correctly, a JCB (digger) driver put his back-hoe through a pair of power cables just outside the building that housed the mainframe computer system that ran BABS (British Airways Business System). One cable was the incoming mains supply so flash and BIG bang and emergency lighting came on in the building. Everything worked exactly according to plan: the two massive static diesel engines running the back up power supplies fired up right on cue and then the short-term battery backup power supply in the building ran out after 5 or 10 minutes and everything in the building died! Unfortunately for BA, the second cable that the JCB had cut was the supply cable from the static diesel power supply system to the building housing the mainframe. Oops! The entirety of BA's systems were down until a new power cable could be connected into the building and in the mean time BA and all the other airlines (which were quite numerous back then) that used the BABS system were almost completely paralysed! If I remember correctly, the final bill for this charade was around £40 million but that was back in the last century so it'll cost a whole lot more today.

 

The only way you can guard against a failure of this sort is the way that banks do it. They have a complete duplicate system at another location that is kept perfectly in sync with the main system at all times. If your primary system fails, it should be possible to switch over to the backup system within minutes and everything keeps running pretty much like nothing happened. If after the power cable catastrophe that I mentioned, at Heathrow, a few years back, BA still don't have a solid, bank-style parallel stand-by system in place, then very, very senior heads should roll including the CEO!

 

How do I know all this - I used to work for BA as a contractor and I have also worked for banks as an IT consultant!   

[Stargeezr]

BA saying Never Again, is just inviting something similar to happen again to put egg on all their faces.

  What a foolish lot. THey should have said that they are trying their best to rectify this problem and are

hoping that this event will never happen again.  Technology can be your enemy, when you apply Murphys law.

[Prbkk]

7 hours ago, dick dasterdly said:

Too expensive to employ good staff.

 

Far cheaper to employ as few staff as possible, many of whom don't have the faintest idea what they're doing....

Add to that: almost guarentee that at some point there will have been a strategic review of some description looking to outsource ( aka give up control over) aspects of the business, almost certainly presided over by the razor-sharp minds of an accounting company . We've seen it so many times you lose count ( but when the shit hits the fan like this, a very strong chance that a moronic consultant has written a report recommending the change that led to the disaster).

[Trouble]

15 hours ago, ezzra said:

It begs the question, whey BA didn't take ' steps ' in insure that a calamity

such as this would have happened? why is that a catastrophe has to happened

before everyone saying' never again? what? you don't have enough qualified

people working at BA to prevent such thing from happening?....

In this world, sh*t happens.  Have we come to such a state of affairs now that when something goes wrong it is immediately necessary for us to go looking to place blame. There was a power surge that was evidently powerful enough to prevent the backup systems form working. Not something BA anticipated but it happened.Sure it was unfortunate for everyone concerned but the attitude that I won't fly BA again is ridiculous.  It could just as easily happen to any airline.  We are dependent on computers and electronics and things are going to happen now and then that cause problems. We better be getting used to it.  God forbid Facebook or Instagram go down for a couple of days.  People might actually have to stop and talk to each other personally.

[Basil B]

1 hour ago, paulbj2 said:

The only way you can guard against a failure of this sort is the way that banks do it. They have a complete duplicate system at another location that is kept perfectly in sync with the main system at all times. If your primary system fails, it should be possible to switch over to the backup system within minutes and everything keeps running pretty much like nothing happened

It is called a "Mirror" and I would have thought that they would need at least 2, each located in different parts of the world.

 

But they still need pen and paper as the ultimate back up...

[paulbj2]

1 minute ago, Trouble said:

In this world, sh*t happens.  Have we come to such a state of affairs now that when something goes wrong it is immediately necessary for us to go looking to place blame. There was a power surge that was evidently powerful enough to prevent the backup systems form working. Not something BA anticipated but it happened.Sure it was unfortunate for everyone concerned but the attitude that I won't fly BA again is ridiculous.  It could just as easily happen to any airline.  We are dependent on computers and electronics and things are going to happen now and then that cause problems. We better be getting used to it.  God forbid Facebook or Instagram go down for a couple of days.  People might actually have to stop and talk to each other personally.

I think you might look at it rather differently if your bank's systems went belly up and you couldn't get at any money for days or when they came back up, all your money had disappeared. Ordinarily, banks have completely parallel systems in place so that if the main system fails the backup system can be up and running in minutes. A few recent system failures in the UK make me wonder if the banks there are trying to save some pennies by doing away with such expensive parallel systems. The whole point of such systems is that they are a long way from the master system so an outage in one, like a power surge, will not impact the backup system. In the bank where I worked, it was in a different town! 

 

If BA don't have such a parallel backup then I would say that was a very very risky way to run their business!

[Ulic]

This is what happens when you outsource your IT needs to India. End of story. I don't believe a word

British Airways says. Just self serving damage control. " In no way our fault, force majeure" 

Edited May 30, 2017 by Ulic

[paulbj2]

2 minutes ago, Basil B said:

It is called a "Mirror" and I would have thought that they would need at least 2, each located in different parts of the world.

 

But they still need pen and paper as the ultimate back up...

Indeed. I didn't use the technical term as not all the people in this conversation would necessarily know what a "mirror" was.

 

The last time this happened to BA was in the late 80s or early 90s (before I worked there). A JCB cut the power supply cables to the building at Heathrow  that housed the mainframe that ran BABS (British Airways Business System) unfortunately, the digger cut through the mains supply cable and the cable from the backup generators that were in buildings about 5 metres away so when the emergency battery power ran out after a few minutes, the lights went out along with the mainframe!  There was no mirror - it was chaos for a couple of days - I believe the cost to BA was of the order of £40 million. You might have thought they would have learned their lesson from that but... 

 

You cannot run a modern airline the size of BA on pen and paper - full stop!

 

Just one of the required pre-flight procedures is a weight and balance calculation that must be done to ensure that the load is distributed within the envelope specified by the airframe manufacturer which ensures that the aircraft is balanced so as to allow safe handling and trimming of the aircraft with a take-off fuel load and at all fuel loads that can be anticipated during the scheduled flight. That calculation can be done manually and indeed mostly was done manually until the 1990s when computerized systems started to take over this task. Doing this fast and accurately, manually, was considered to be something of a black art. Moreover the larger the aircraft, the more challenging it is to do. There are very few people today who have the necessary skills, expertize and practice to do the weight and balance by hand quickly and accurately and of course many modern aircraft are much larger than when it was habitually calculated by hand - best of luck getting that done without the computerized systems (BA actually said that one of the systems that had gone down as a result of the outage was their computerized weight and balance system). Without a weight and balance calculated and the form completed, an aircraft is not permitted to take off. That is just one of the legal responsibilities that an airline has to comply with. A full passenger and baggage manifest is another, along with a full cargo manifest if the aircraft is carrying cargo (BA have no dedicated cargo fleet but, surprisingly, carry quite a lot of air cargo on passenger flights).

[paulbj2]

3 minutes ago, Ulic said:

This is what happens when you outsource your IT needs to India. End of story. I don't believe a word

British Airways says. Just self serving damage control. " In no way our fault, force majeure" 

Funnily enough, I tried to contact my old boss at BA to get the inside story but his email address no longer exists; I guess, as he was in the IT department, his job is one on the ones that went to India.

[Basil B]

These things happen but I suspect what has angered so many people is the way it has been handled.

 

Though tonight I suspect there will be no stranded passengers still sleeping on the floor at LHR, thousands of people have probably cancelled their holidays or business trips and others have been delayed days, issues still remain.

• Passengers worldwide are still without their bags and have been bombarding the airline's lost luggage phone lines, website and social media feeds.
• British Airways has also been criticised on social media for directing some passengers towards a premium-rate phone line that costs up to 55p a minute from a mobile device.
• The airline also admitted that it was aware some passengers were unable to file a delayed bag report on its website

I feel the impact will see a little more than Alex Cruz and Willy Walsh forgoing next year bonuses.

 

 

 

[paulbj2]

3 minutes ago, Basil B said:

These things happen but I suspect what has angered so many people is the way it has been handled.

 

Though tonight I suspect there will be no stranded passengers still sleeping on the floor at LHR, thousands of people have probably cancelled their holidays or business trips and others have been delayed days, issues still remain.

• Passengers worldwide are still without their bags and have been bombarding the airline's lost luggage phone lines, website and social media feeds.
• British Airways has also been criticised on social media for directing some passengers towards a premium-rate phone line that costs up to 55p a minute from a mobile device.
• The airline also admitted that it was aware some passengers were unable to file a delayed bag report on its website

I feel the impact will see a little more than Alex Cruz and Willy Walsh forgoing next year bonuses.

 

 

 

I thought Willy Walsh had gone but I am a bit out of touch with happenings in the UK. Certainly I think Cruz should fall on his sword for this one!

 

First things first - why was there no mirror? We are not talking just about BA but Iberia, and a couple of others; I'm sure they all use the same system - it would be insane if they didn't!

 

Was there no planning for this sort of eventuality? I don't know where their systems are located now; I know BABS has been retired and replaced with something a bit more modern but what I don't know. Back in the days when I worked there, the mainframe that ran BABS was housed in a building not that far from one of the runways, so if an aircraft got it a very wrong for any reason, it was foreseeable that it could have taken out that building. At an airport, that sort of potential hazard is kind of obvious, isn't it! There was a BA 747 whose pilot "got it a bit wrong", rather big time, and mistook the A4 Bath Road for one of the runways. He pulled up and went round only just in time. Sadly, the Captain of the aircraft was so traumatized by what he had nearly done, that he committed suicide ( http://www.independent.co.uk/news/uk/pilot-in-near-miss-found-dead-in-car-1561147.html )

 

[Basil B]

3 minutes ago, paulbj2 said:

I thought Willy Walsh had gone but I am a bit out of touch with happenings in the UK.

William Matthew "Willie" Walsh is an Irish airline executive. He is CEO of International Airlines Group, and has been CEO of Aer Lingus and British Airways.

 

Moved up the Greasy Pole... IAG is BA's  parent company.

[paulbj2]

31 minutes ago, Basil B said:

William Matthew "Willie" Walsh is an Irish airline executive. He is CEO of International Airlines Group, and has been CEO of Aer Lingus and British Airways.

 

Moved up the Greasy Pole... IAG is BA's  parent company.

Yeah I think he was around when I was there. I am trying to think of the name of the guy who ran the show when I was working there. A nice, very down to earth, approachable bloke; I had a few pleasant chats with him when I doing some work with his secretariat. Bob Ayling - that's the guy's name!

[jacko45k]

Did they not promise that last time there was similar chaos?

Hollow promises!

 

They might as well sell some aircraft now!

[Basil B]

5 hours ago, jacko45k said:

Did they not promise that last time there was similar chaos?

Hollow promises!

 

They might as well sell some aircraft now!

LHR my just be considering alternative uses for T3 & T5... 

[thaibeachlovers]

Never again- what?

here's some suggestions

Never again

 sack your IT staff and outsource to a third world country

 have an incompetent in charge

 employ useless counter staff

 employ arrogant flight staff ( I won't ever fly BA again because of them )

 offer no refreshments to stressed and angry customers. 

 listen to accountants before public relations staff.

[paulbj2]

6 minutes ago, thaibeachlovers said:

Never again- what?

here's some suggestions

Never again

 sack your IT staff and outsource to a third world country

 have an incompetent in charge

 employ useless counter staff

 employ arrogant flight staff ( I won't ever fly BA again because of them )

 offer no refreshments to stressed and angry customers. 

 listen to accountants before public relations staff.

Yes but he saving SOOOooo much money!!!

 

Bringing in the cost accounts brought Leak (Hi-Fi manufacturer in the 50s, 60s and 70s), Jaguar and a host of other British companies known for quality, to their knees and finally led them to Carey Street. This is one of the few things the British are good at, don't knock it!

[thaibeachlovers]

2 hours ago, Basil B said:

LHR my just be considering alternative uses for T3 & T5... 

I can't see BA failing. Obviously a lot of people use them out of misguided loyalty, and that probably won't change.

[GuiseppeD]

1 hour ago, thaibeachlovers said:

I can't see BA failing. Obviously a lot of people use them out of misguided loyalty, and that probably won't change.

 

As I stated earlier in the thread, although Saturday's T5 fiasco was obviously an unmitigated disaster, when my wife and I finally re-booked our first long-haul BA flight for Sunday to BKK our Club World flight was extremely enjoyable, the service and food was top notch, the lie-flat bed was, IMHO, more comfortable and spacious than that of Thai and EVA business class.

 

We've also been informed today that our missing luggage will now be delivered tomorrow to northern Thailand.  Notwithstanding, we received 200 pounds compensation for the luggage debacle at the BA lost luggage counter at Swampy on Sunday which enabled my wife to purchase a few changes of clothes on Monday.

 

I also have a pot of 150k Avios to spend, which have only taken me six months to accumulate, so all in all, I would most definitely fly BA long-haul again in the future and certainly not through misguided loyalty.

 

Oh, and we're in for a nice little compensation package as well on our return to the UK. 

 

 

Edited May 31, 2017 by GuiseppeD

[Andaman Al]

Never say never again.

[Basil B]

3 hours ago, Andaman Al said:

Never say never again.

It was a long time ago that I said never again to BA & LHR and last weekends fiasco can only resolve my determination.

[thaibeachlovers]

On ‎31‎.‎05‎.‎2017 at 4:10 PM, GuiseppeD said:

 

As I stated earlier in the thread, although Saturday's T5 fiasco was obviously an unmitigated disaster, when my wife and I finally re-booked our first long-haul BA flight for Sunday to BKK our Club World flight was extremely enjoyable, the service and food was top notch, the lie-flat bed was, IMHO, more comfortable and spacious than that of Thai and EVA business class.

 

We've also been informed today that our missing luggage will now be delivered tomorrow to northern Thailand.  Notwithstanding, we received 200 pounds compensation for the luggage debacle at the BA lost luggage counter at Swampy on Sunday which enabled my wife to purchase a few changes of clothes on Monday.

 

I also have a pot of 150k Avios to spend, which have only taken me six months to accumulate, so all in all, I would most definitely fly BA long-haul again in the future and certainly not through misguided loyalty.

 

Oh, and we're in for a nice little compensation package as well on our return to the UK. 

 

 

Sorry but only cattle class customers know what it is like to be treated like **** by British companies.

[SheungWan]

On 5/31/2017 at 10:26 PM, Basil B said:

It was a long time ago that I said never again to BA & LHR and last weekends fiasco can only resolve my determination.

Dropping BA is understandable, but chucking out :HR as well? I don't think so and for a lot of us the only alternative is Gatwick and anyway the biggest complaint these days with LHR is that it remains somewhat dingy. Not LHR's fault that BA screwed up.

[SheungWan]

3 hours ago, thaibeachlovers said:

Sorry but only cattle class customers know what it is like to be treated like **** by British companies.

My guess is that with all BA flights messed up, using the Business Class lounges would have in their turn have become somewhat of a scrum.

[Basil B]

6 hours ago, SheungWan said:

Dropping BA is understandable, but chucking out :HR as well? I don't think so and for a lot of us the only alternative is Gatwick and anyway the biggest complaint these days with LHR is that it remains somewhat dingy. Not LHR's fault that BA screwed up.

Recall Landing BHX @ 12:36 (Wheels on Tarmac), home by 13:30 (54 mins), if I had landed LHR I doubt I would have got through immigration within an hour of touch down.

Edited June 1, 2017 by Basil B

[SheungWan]

8 minutes ago, Basil B said:

Recall Landing BHX @ 12:36 (Wheels on Tarmac), home by 13:30 (54 mins), if I had landed LHR I doubt I would have got through immigration within an hour of touch down.

I will be landing at LHR next month. If its the same as last time immigration/customs will be pretty quick. How fast one gets home depends on how far your home is from the airport and available transport links. Oh, there is one complaint from me and that is that last time the small-duty free shop just before entering the arrivals concourse in Terminal 3 had stopped stocking the Giant Toblerones Buy 3 Offer. Now if this happens again I am going to be really unhappy!

Edited June 1, 2017 by SheungWan

[JAG]

I will be landing at LHR next month. If its the same as last time immigration/customs will be pretty quick. How fast one gets home depends on how far your home is from the airport and available transport links. Oh, there is one complaint from me and that is that last time the small-duty free shop just before entering the arrivals concourse in Terminal 3 had stopped stocking the Giant Toblerones Buy 3 Offer. Now if this happens again I am going to be really unhappy!

That's outrageous! How can one retain ones "sang froid" as a seasoned globetrotter if one does not emerge from arrivals brandishing at least three Giant Toblerone?