The Senate has published a report from telco Telstra about What Went Wrong.
We knew it was about a machine involved in ‘timing’, that reset itself to 2006. When you read below, note that we think of GPS about being about location, but it’s actually about keeping timing syncronised, which is vital in big networks and systems.
There’s more.
“We now understand the likely technical cause of the outage from our initial investigation.
“The team carrying out the maintenance work followed a defined maintenance procedure, however, when the device was re-powered, a GPS card within it did not operate as expected.
“We now believe this occurred because of an intentional design change that had previously been made to the equipment to fix an earlier fault had not been properly documented. This meant the maintenance team was not aware of the way the device would behave when restarted.
“A software update had also not been applied to the device.
“Had that software update been completed or had the design change been properly reviewed and documented post the earlier incident, and reflected in the maintenance procedure, the outage may not have occurred.
“That is clearly unacceptable. If maintenance work can trigger this kind of outage, it suggests our controls were not good enough.
“We are accountable for that, and our investigation will address why that design change was not documented, why the software update was not completed, and what needs to change in our controls so known risks are captured, prioritised and closed before they can affect customers.”