SYDNEY, Nov 17 (Reuters) – Australia’s second-largest telco, Optus, had no crisis plan when a network-wide outage left nearly half the country without phone or internet for 12 hours, an executive told parliament on Friday, acknowledging the company’s defences had failed.
The Singapore Telecommunications-owned (STEL.SI) company had recently war-gamed scenarios in which the routers that direct voice and internet data failed in entire states, but it never expected a nationwide shutdown because it had alternate connections built into its network.
“We didn’t have a plan in place for that specific scale of outage,” Optus managing director of networks Lambo Kanagaratnam told a Senate hearing on the Nov. 8 failure that left much of the country unable to make payments, receive healthcare or contact emergency services for most of a day.
“It was unexpected. We have high levels of redundancy and it’s not something that we expect to happen,” he added, using the telecommunications term for alternate routes to send data when an initial pathway fails.
The comments underscore concerns about the resilience of Australia’s telecommunications networks, which have been in the spotlight since a massive data breach at Optus last year exposed the personal data of 10 million Australians. Now the company faces a fresh reputational crisis after the service blackout, which it has said was triggered by a standard software upgrade at Singtel.
Singtel has said that while Optus experienced an outage after its software upgrade, the upgrade itself was not the cause.
“Singtel will support Optus as it learns from what has occurred and continues to improve,” the Singapore company said a day earlier.
Kanagaratnam told the hearing Optus never expected a total shutdown because it had filters designed to stop all 90 of the company’s routers from being overloaded with data. But the filters failed, cutting the company’s ability to send data on alternate routes.
“The outage was a result of our defence not working as it should have,” he said. “Our network should have been able to deal with the change.”
The length of the outage – from about 4am to 4pm local time – was because Optus had to physically reboot all 90 routers plus another 50 core network devices, he added.
Optus CEO Kelly Bayer Rosmarin, asked why the company took six hours to dispel public concerns it was under a cyber attack, told the hearing “there were some strange coincidences that made us quite worried about that” because the Singtel board was in the country that day.
Bayer Rosmarin said 228 calls to Australian emergency hotline Triple-0 failed to connect because of the outage, but the telco had followed up all incidents and “thankfully everybody is OK”.
Asked whether Optus was overly reliant on third-party contractors, Bayer Rosmarin said “it is something I do think we should look at, in terms of the right level of outsourcing and insourcing”.
($1 = 1.5473 Australian dollars)
Reporting by Byron Kaye. Editing by Gerry Doyle
Our Standards: The Thomson Reuters Trust Principles.