Optus has pinned the cause of a national outage on a routine software upgrade.
In a statement the telco said routers weren’t able to handle changes to the network and disconnected to protect themselves, leading to the nine-hour outage on November 8.
“The restoration required a large-scale effort of the team and in some cases required Optus to reconnect or reboot routers physically, requiring the dispatch of people across a number of sites in Australia.
“This is why restoration was progressive over the afternoon.”
Optus said the widespread impact of the outage meant investigations into the issue “took longer than we would have liked”.
“We are committed to learning from what has occurred and continuing to work with our international vendors and partners to increase the resilience of our network,” the statement said.
“We will also support and fully cooperate with the reviews being undertaken by the Government and the Senate.”
Questions remain for telco
RMIT University’s associate professor Mark Gregory said Optus had identified the cause as “human error”.
While this ruled out hardware failure or a cyber attack, he said questions still remained unanswered by the telco.
“Optus has not explained what went wrong with the test process that should have occurred before the routing software upgrade occurred,” he said.
“Also, there is no explanation as to why there appears to have been a lack of redundancy of the key routers, so that if there was a problem the key routers would swap to the redundant routers, which you would expect to be running the previous iteration of software.”
Research fellow at the Centre for Defence Communications and Information Networking at The University of Adelaide Mark Stewart said Optus had faced a “well-known and predictable problem… commonly associated with software updates”.
“A major telco should have a disaster recovery plan which is more sophisticated than your average corporate network,” he said.
“At a minimum, they should have had a plan to revert the changes or remotely reboot their systems.”
Following the disruption the telco offered customers at least 200GB of extra data.
Optus users were also warned of phishing scam messages claiming to offer compensation for the outage.
The telco said it would not send any communications with links about the outage, to provide “peace of mind” for customers.