What happened?
As Australia awoke this Wednesday morning, real-time data provided by Netblock’s, a UK company that tracks internet connectivity, showed that traffic was down to just 17% of its of its ordinary levels.
⚠️ Update: Real-time network data show connectivity on #Australia‘s Optus network is now flatlining at 17% of ordinary levels amid an ongoing nation-scale outage; incident is heavily impacting payment systems with transport sector and carriers relying on Optus also affected 📵 pic.twitter.com/KFvjbyAKae
— NetBlocks (@netblocks) November 7, 2023
This has caused significant problems for Australians as Optus is responsible for 40% of internet connections and owns the second largest mobile network operator.
The Optus network is resold by MVNO’s such as Aussie Broadband, Amaysim, CatchConnect, Coles Mobile, Dodo, Moose Mobile who were also affected.
Local and international news has reported the chaos that ensued, with hospital, emergency responders and transport networks unable to operate at full capacity. Optus customers were also unable to contact the emergency 000 service.
Mobile customers could roam onto Telstra of TPG’s networks, if it was available, but Optus landlines were completely cut off.
The outage also effected people’s daily lives, with wifi powered cat feeders leaving cherished pets hungry, and a pensioner unable to collect his digital prescription from a chemist.
Reports of outages began to come through at around 4am AEDT, and increased to 8,180 by 5am, according to Downdetector.
But what caused the outage?
By midmorning, Optus CEO, Kelly Bayer Rosmarin had told ABC Radio her team was pursuing every possible avenue, but the number of hypotheses they had tested so far had failed to resolve the fundamental issue.
At 1pm, Rosmarin said there was a path to restoration and that some services had been restored, but did not expand on the route cause.
The minister for communications, Michelle Rowland said that the length of the outage indicated a “deep networking problem” and later said that there was no evidence that Optus had been the victim of a cyber-attack.
The crisis is the second in a year for Rosmarin, who had to deal with a security breach that led to 10 million Australian’s personal data being exposed.
After network services were largely restored, Rosmarin said in an interview on ABC news television that a thorough route cause analysis had to be undertaken before she could expand on the details of the outage, but said that the Optus outage was caused by a technical network issue.
Cloudflare, another IT that tracks internet activity, noticed a sudden rise in Border Gateway Protocol (BGP) announcements from Optus, at around the same time as its network outage.
BGP refers to a gateway protocol that enables the internet to exchange routing information between autonomous systems, according to cybersecurity firm Fortinet.
The managing director of a network analysis company Enex TestLab, Matt Tett, told Guardian Australia that Optus appeared to have a failure in routing at 4am that caused the exponential increase in BGP announcements.
Tett said that if this was the reason, the fault might not have been Optus itself, but could have originated from a partner the company works with on service provision.
Facebook, Instagram and Whatsapp suffered a five hour outage in October 2021, that Cloudflare also attributed to a BGP issue.