During PG&E’s earnings call (opens in new tab) Thursday morning, executives touted the utility’s 10% annual earnings per share growth and 19% improvement in reliability.
What they didn’t acknowledge was the mass power outage days before Christmas that left a third of San Francisco in the dark and caused gridlock on its streets. A review of the company’s 376-page 10-K annual report (opens in new tab) found that the only mention of San Francisco was not about the blackout but about the utility’s business risk due to the renewed push for the city to take over its own grid.
A much-awaited hearing called by city officials to grill PG&E executives was meant to provide some answers about what caused the multiday blackout that began Dec. 20. Instead, the hearing revealed a cascade of missteps — from delayed response times to flawed restoration estimates — that raised fresh doubts about PG&E’s ability to avoid future outages.
A series of screw-ups
Part of the problem was the utility’s delayed response. The outage began Dec. 20 at 1:04 p.m., caused by a fire at a substation at Mission and 8th streets. It took PG&E’s technician 45 minutes to arrive at the substation, according to CEO Sumeet Singh.
Shortly after arriving, the tech — who came from outside of San Francisco — saw the substation was on fire and contacted the San Francisco Fire Department.
Patrick Rabbit, operations chief of the SFFD, said the department got the call at 2:17 p.m, more than an hour after PG&E learned of the power outage.
Putting out the fire proved difficult because the substation’s building plans were on paper, which firefighters had to read outdoors in the rain, Rabbit said. The amount of smoke and the size of the building made the fire difficult to locate.
Singh said at Thursday’s hearing that a failed circuit breaker caused the fire, and PG&E is still investigating why it failed. He said PG&E has not digitized the building plans for all its San Francisco substations but plans to do so in the coming weeks. He didn’t know which substations still have paper building plans.
The PG&E substation at Eighth and Mission streets ignited on Dec. 20, triggering a massive blackout that left one-third of the utility’s San Francisco customers without power. | Source: Jessica Christian/SF Chronicle/AP
PG&E took two hours to send a representative to the city’s Emergency Operations Center at 1011 Turk St. and did not proactively reach out to the city to inform officials about the power outage, according to Mary Ellen Carroll, executive director of the Department of Emergency Management. She was notified at 1:18 p.m. by her own team about a power outage affecting multiple neighborhoods. City officials had to specifically request a PG&E liaison, who arrived around 3 p.m, she said.
Singh said extended and repeated west-side outages after Dec. 20 occurred because power cables serving the Richmond and Sunset districts were damaged and required the substation to operate in an “abnormal configuration.” He failed to provide details of the configuration.
“It had to do with the complexity of the damage, the complexity of the interconnected nature of the system,” he said.
In other words, the fire screwed up a bunch of complicated, interlinked stuff. Singh said the substation repairs and reconfigurations were complete by Jan. 20 and claimed that the changes have improved power reliability.
Singh later said that in 2010, PG&E invested $200 million in rebuilding the substation at Mission and 8th streets, which has caught fire three times: in 1996, in 2003 (opens in new tab), and Dec. 20. The 2003 fire also occurred on Dec. 20, and also during rain.
After the 2003 fire, state regulators found that the utility had not fixed known hazards (opens in new tab), including missing smoke detectors and aging power cables — one of which dated to 1963 and was the source of the blaze. The California Public Utilities Commission fined PG&E for $6.5 million (opens in new tab) in 2006.
Mahmood was puzzled by the substation’s failure after the expensive upgrades.
“So you put $200 million in this specific substation?” he asked at the hearing.
“Yes, that’s correct,” Singh said.
“And it still broke?” Mahmood said.
False estimates, false hope
Singh said PG&E’s inaccurate power restoration times were “unacceptable.” He blamed the complexity of the infrastructure and the fire functioning as a barrier for inspectors.
“This resulted in multiple inaccurate estimates, and we own that gap in our process,” Singh said.
Another cause of the incorrect estimates was AI.
Singh said the company has been using a machine learning algorithm for the last year or two to estimate when power will be restored, based on previous outages. But the algorithm didn’t include data from the Mission and 8th Street substation’s 2003 outage. Singh said the complex substation circuitry provided an additional challenge for the algorithm, which was trained on data from smaller outages.
PG&E CEO Sumeet Singh said the company’s AI restoration model wasn’t built to handle an outage of Dec. 20’s scale, resulting in inaccurate estimates for San Francisco customers. | Source: George Kelly/The Standard
After providing the initial, false, AI-generated restoration time, PG&E had workers inspect equipment to provide an update. But Singh said crews couldn’t safely assess the damage for hours, which “resulted in multiple inaccurate estimates.”
When asked by Mahmood how PG&E will change its method of restoration estimates, Singh said PG&E has created a “rapid escalation team” of workers to double-check the automated restoration time before customers are notified.
Another point of frustration was PG&E’s delay in providing resources to those affected by the outage. City officials had to specifically request a customer resource center in the Richmond and near Civic Center.
Singh’s explanation? PG&E opens resource centers only for areas with planned power outages due to wildfire risks, so it had no protocol to do so in the case of an unplanned outage.
Adding insult to injury, Mayor Daniel Lurie requested that PG&E provide power to the War Memorial Opera House before his daughter’s performance in “The Nutcracker” even as thousands of west-side residents — and City Hall — remained in the dark.