Four international flights not yet airborne were held on the ground in New Zealand.<\/p>\n
Airways said a latent software defect during the initiation of a sectorisation process caused a display anomaly.<\/p>\n
Sectorisation involved splitting the airspace Airways managed from one overall sector into two sectors.<\/p>\n
\u201cSectorisation occurs several times a day,\u201d Airways said in a statement accompanying the report into air traffic system disruption.<\/p>\n
But an incoming message from a pilot during this \u201ccomputationally complex\u201d process prevented sectorisation from happening, the report said.<\/p>\n
\u201cThe particular timing led to an invalid memory access and prevented the sectorisation process completing correctly,\u201d the state-owned enterprise added.<\/p>\n
\u201cThis issue then affected the state that aircraft appeared in the reserve software and required a restart to restore the correct data state.\u201d<\/p>\n
New Zealand\u2019s oceanic airspace covers an area almost four times the size of Australia.<\/p>\n
Much of that is over the South Pacific and Southern Ocean with minimal traffic, or just a few long-haul flights flying at constant altitude across the ocean.<\/p>\n
The Tasman Sea can be busier, and before increasing daily transtasman traffic, controllers initiated a routine sectorisation process.<\/p>\n
\u201cIn simple terms, when a sectorisation process occurs, some elements are removed and then recreated to align with the new sectors,\u201d Airways said.<\/p>\n
The agency said that during testing and operational use in the days before the outage, routine sectorisation had been performed many times without incident.<\/p>\n
Screens went blank<\/p>\n
One controller asked a colleague to return from their break early to open the Tasman sector.<\/p>\n
\u201cThe process should take five to 10 seconds,\u201d the Airways report said.<\/p>\n
But on starting sectorisation, a controller\u2019s data display went blank, as did their colleague\u2019s.<\/p>\n
That was regarded as \u201ca complete workstation failure\u201d, the report added.<\/p>\n
\u201cWithout the data display, the controller has limited ability to interact with the system.\u201d<\/p>\n
The controllers had not experienced this before, so one sought expert advice from two technicians who were on site.<\/p>\n
At roughly the same moment, the principal desk in Christchurch started flagging multiple unusual indicators.<\/p>\n
Attempts were made to consolidate work on a standby workstation.<\/p>\n
But that third workstation was not accepting sectorisation.<\/p>\n
\u201cThe atmosphere was stressful, time-critical and noisy,\u201d the report added.<\/p>\n
\u201cExternal attention from outside parties arose quickly.\u201d<\/p>\n
Each of the controllers on shift at the start of the event was towards the end of their shift, but all completed the shift in full, the report added.<\/p>\n
Aircraft held<\/p>\n
About 11 minutes after the initial display anomaly and following the move to the reserve platform, a decision was made to hold international departures in domestic airspace.<\/p>\n
And 26 minutes after the initial display anomaly, operations resumed on a reserve controller workstation for aircraft operating within the Pacific sector.<\/p>\n
Airways said 50 minutes into the disruption, the platform was restored.<\/p>\n
And at 57 minutes, the Oceanic Control System (OCS) resumed full normal operations, including for the Tasman.<\/p>\n
The agency said aircraft separation and safety were maintained throughout the incident. <\/p>\n
\u201cBy the next morning, the cause hadn\u2019t yet been identified,\u201d the report added.<\/p>\n
\u201cAs a mitigation, a rollback to the previous software version was undertaken while priority work to identify and rectify the cause continued.\u201d<\/p>\n
For the investigation, the agency said, it interviewed four controllers, all of whom had more than 20 years\u2019 experience.<\/p>\n
A duty manager, operations manager and five technicians were also interviewed.<\/p>\n
The report said one controller at the time was unaware that several aircraft would shortly be required to divert.<\/p>\n
Air NZ call<\/p>\n
The report said Air New Zealand contacted the duty manager, asking if the OCS system had an issue.<\/p>\n
The duty manager confirmed it did, but said the OCS was moving to a reserve platform and would soon be operational.<\/p>\n
Air NZ said it had aircraft airborne that could hold for only about 30 minutes.<\/p>\n
The duty manager thought that would be sufficient because moves to the reserve platform were normally complete in about 10 minutes.<\/p>\n
Control system <\/p>\n
Airways said it bought the OCS from Canadian manufacturer CAE Inc in 2000.<\/p>\n
After that, Airways assumed responsibility for the software maintenance and support.<\/p>\n
The agency said the system had demonstrated 99.99% service availability percentage in the 12 months before the incident.<\/p>\n
The software development cycle is tied to a global schedule of data changes known as Aeronautical Information Regulation and Control, with which Airways must comply.<\/p>\n
It said the OCS had a dual-platform architecture with two physically independent systems, available at all times and capable of providing most system functions.<\/p>\n
One of those is a reserve platform, intended as a temporary backup system and without as many functions as the main platform.<\/p>\n
Tests<\/p>\n
The Herald has been contacted by an industry insider who queried whether the software was tested properly.<\/p>\n
Airways, in today\u2019s report, said testing happened in layers.<\/p>\n
\u201cFirst, developers test in their own branches of code. Next, test builds are provided to the Auckland-based Oceanic System Specialists team for further checking.\u201d<\/p>\n
It said once a release baseline was built, it was deployed into a laboratory environment for integration testing. <\/p>\n
\u201cThis environment mirrors operational platforms but cannot reproduce every dimension of the operational Oceanic system,\u201d the report added.<\/p>\n
\u2018Dangling pointer\u2019<\/p>\n
Airways said a technician found a long-latent timing defect known as a \u201cdangling pointer\u201d in the display process.<\/p>\n
\u201cThe defect had been in the code base for around 20 years.\u201d<\/p>\n
Findings<\/p>\n
Airways said it developed and applied a software patch by August 22 to address the data display application issue when sectorisation failed to complete.<\/p>\n
\u201cThe issue occurred due to an incoming CPDLC [controller pilot data link communications] message from an aircraft, during a system sectorisation process.\u201d<\/p>\n
CPDLC is a means of communication between controllers and pilots.<\/p>\n
Airways said it also applied a patch in August after the sectorisation process failed to complete because of a data display issue on the main platform.<\/p>\n
It also pledged to identify areas for improvement in terms of information flow between controllers and technical staff.<\/p>\n
\u201cImportant contextual information was not made available to the Christchurch-based duty manager from Auckland in a timely manner to enable them to effectively co-ordinate a response,\u201d the report added.<\/p>\n
It said that issue was reviewed.<\/p>\n
Airways also said customers were not informed of the service disruption in a timely manner.<\/p>\n
The agency said it would set up another external phone line to use for operational matters.<\/p>\n
\u201cWe acknowledge our communication with airlines during the incident did not meet our expected standards,\u201d Airways chief executive James Young said.<\/p>\n
\u201cWe apologise for this and are addressing this to ensure we have a more robust process for communicating changes to our air traffic management with our customers directly,\u201d he added.<\/p>\n
\u201cWe also apologise to the passengers whose flights were delayed or cancelled as a result of the unforeseen technical issue.\u201d <\/p>\n