Optus said those updates were sent “following a routine software upgrade”. The Optus source added that the software update occurred at the Singtel Internet Exchange, not on the Optus network.
Once received, the routing information changes then propagated through multiple layers in the Optus network and exceeded “preset safety levels” on key routers that could not handle them, resulting in the routers disconnecting from Optus’ core network “to protect themselves”.
The changes meant technicians had to be deployed to key locations across the country to access routers that had gone offline so that they could restore services, which delayed restoration.
The Singtel Internet Exchange is one of the many internet peering exchanges that help move vast amounts of data across the world. Peering allows two networks to connect and exchange traffic directly, sometimes without having to pay a third party to carry data across the internet. Not only does it lower the cost of moving traffic around, but it also improves performance and redundancy.
Poor filters
Despite the Singtel Internet Exchange being one of the triggers for the outage, an Optus source said the Australian telco was primarily responsible for the outage.
Loading
“It’s still Optus’ fault when it comes to its own network (unless Singtel required Optus to accept routes blindly),” an Optus engineer, who was unclear what commercial agreements may have been in place, said.
The source, as well as other external network engineers this masthead spoke to, said Optus should have had filtering rules in place that dismissed or ignored the update from the Singtel exchange.
On the day of the outage, a senior Optus executive phoned a counterpart at content distribution network Akamai, another of the telco’s peers, believing it had triggered the outage.
By Friday, Optus engineers were still adamant that Akamai was partly to blame. However, Akamai definitively ruled this out in a statement on Monday night. “Akamai did not trigger the outage,” a spokesperson told this masthead.
As more details of the cause of the Optus outage emerge, Telstra chief executive Vicki Brady said the nation’s biggest telco had won new customers as a result of Optus’ network meltdown. But she pushed back on calls for extra regulation, arguing that would force it to share its infrastructure in the event of another mass disruption.
Optus customers seek an explanation for the outage on Wednesday.Credit: Chris Hopkins
Last week’s outage affected about 10 million Optus customers, leading some to desert the company barely a year after it suffered the worst cyber breach in Australian history.
The exact number of customers ditching Optus is yet to be determined, but speaking at an annual Telstra investor day in Sydney on Tuesday, Brady said Telstra had seen an uptick in new customers.
“We take zero pleasure in any competitor or operator experiencing major outages,” the Telstra boss said.
While it was early days since the incident and Telstra was continuing to see elevated levels of new customers signing up, Brady said she “wouldn’t anticipate major shifts in [market] share”.
Loading
Telstra’s executives were quick to spruik the resiliency of its network in front of investors, highlighting increased investment and reliability.
Network sharing in the picture
Channa Seneviratne, Telstra’s executive in charge of technology development and innovation, said the telco’s mobile network was superior to competitors due to its “best in class resilience and security”.
Seneviratne said Telstra was in favour of network sharing in the event of a natural disaster like a bushfire, which would enable Optus or Vodafone customers to send texts or make and receive calls.
Communications Minister Michelle Rowland and Emergency Management Minister Murray Watt and their departments have begun developing such emergency roaming capabilities – work that was already underway before last week’s outage.
Minister for Communications Michelle Rowland.Credit: Alex Ellinghausen
But that was a very different proposition to network-sharing in the event of an outage, said Seneviratne.
“Last week we lost one of the national networks, roughly one-third of our carriers. We don’t design our networks to have overhead to manage another carrier’s whole customer base, it just doesn’t work that way.
“I’m sure it’s going to be the topic of conversations with some of the parliamentary inquiries, and we’ll be happy to participate and co-operate with that,” Seneviratne said.
Loading
“It’s a very different proposition, emergency roaming in a selected area, versus opening up the possibility of having 11 million Optus customers suddenly come into our network. You probably end up having two national outages.”
Telstra’s executives carry SIM cards of other carriers with them in the event of a Telstra outage, he added. Optus chief Kelly Bayer Rosmarin last week raised eyebrows when she was forced to dial into ABC radio via WhatsApp, given Optus’ networks were down.
The Business Briefing newsletter delivers major stories, exclusive coverage and expert opinion. Sign up to get it every weekday morning.