Today, millions of Optus customers woke up to find that their Optus services weren’t working.
Businesses couldn’t process payments, trains in Melbourne had service disruptions, ambulances in Victoria couldn’t call people back if they needed more information.
All major hospitals in Victoria were affected by the outage, as were hospitals in the Metro North Hospital and Health Service in Brisbane.
The outage was first noticed by Optus at 4:05am, and Optus claimed that services were starting to gradually come online at 12:55pm, some 8 hours and 50 minutes later, though it still took a few more hours for all services to come back online.
The question is, though, how did this happen? And can this happen again?
According to Dr Bill Corcoran, an ARC Future Fellow at Monash University, this whole disaster was likely caused by human error.
“One of the things that happened today is that something went ‘sproink’ in network control for Optus.”
A telecommunications network is made up of a lot of parts.
The most important parts are the mobile towers, fibre optic cables, and other infrastructure that connects the network, and all of that equipment appears to be unharmed.
But for the information to flow properly through that network (which you could think of as a network of pipes, including some wireless pipes), there are servers and computers that act as a kind of “brain” to direct everything to where it needs to go.
It’s similar to how “the cloud” isn’t really a cloud, but a data centre located in the middle of nowhere filled with racks of servers and hard drives.
An example of a “sproink”, to use Dr Corcoran’s term, would be spilling a glass of water on one of the servers and having it short circuit, though Dr Corcoran thinks something much bigger must have gone wrong to cause the kind of outage we’ve seen today.
“What this seems to indicate, without knowing exactly what’s happened, is a cascade of failure.
“Something’s failed, and that’s made something else fail, which made something else fail. Getting to the bottom of what’s causing all of this sounds like a nightmarish job.”
There have been other outages in Australia caused by bad software or firmware updates pushed to network control centres in the past, but usually this is restored within an hour or so by just winding the update back.
One memorable outage was back in 2016 when an engineer improperly turned off a piece of key equipment, taking down the Telstra mobile network for several hours.
The year 2016 was actually pretty rough for outages all round, with Telstra suffering several and Virgin mobile suffering so many that this is probably the first time you’ve thought of Virgin Mobile in years.
For those worried that this could be another Optus hack, Dr Corcoran thinks it’s more likely that the call is coming from inside the house, so to speak.
“I would be very, very surprised if it was a hack. I would suggest that this is more something that has gone wrong internally.
“From what we understand with these hacks, they’re generally short and sharp, and then they run out of steam because they need someone attacking all the time to get this to stick.
“But this seems like it has been quite sustained. And that’s a big issue.”
Optus customers have had a particularly bad 14 months, first with the hack leaking identity documents, then with Optus raising plan prices and taking people off older plans in August, and now with this outage.
But while the Optus data breach last year can be put down to incompetence, Dr Corcoran doesn’t want to rule out this Optus outage being just a case of bad luck, which is bad news for Telstra and Vodafone customers.
“I wouldn’t necessarily say that this is just an Optus problem. I can imagine that this could happen at any of the telcos. I would probably just say that Optus have gotten somewhat unlucky and something’s gone wrong and they’re wearing it this time. But you can imagine that this could reoccur in a bunch of different places.
“It would be nice to know the specifics so we can know if this is going to occur again.”
As for whether Australia is particularly vulnerable to these kinds of breaches and problems; while this Optus outage is unusually long and affected an unusually large number of people, there is nothing particularly special or vulnerable about the Australian telecommunications network.
These outages happen overseas, too, we just notice it more because we live here.
News.com.au has approached Optus for comment.