Weekly Feature

2016-04-13 / Front Page

Series of human, technological failures result in 911 outage


A perfect storm of failures, both human and technological, were at the heart of a nearly 3½-hour shutdown of Erie County’s 911 system in the early morning hours of March 30.

Such was the conclusion of county administrators from Information Support Services, Public Works, Emergency

Services and Central Police Services as legislators grilled the officials on April 7 as to how the county’s 4-year-old, costly emergency system could suffer such an extensive and severe outage.

Emergency Services Commissioner Daniel Neaverth Jr. laid out a timeline of events, as compiled by his department, in regard to how and why the outage took place. When an air conditioner broke down in a power supply room in the county’s public safety building, a cooling system circuit board was supposed to activate a second cooling system and issue an emergency alert to county employees. It failed to do both tasks.

By approximately 5 a.m., it became apparent that only one workstation in the Town of Tonawanda and one in Hamburg were receiving rollover calls for all of Erie County.

When the power supply room heated to more than 100 degrees, a late-shift building engineer responded to the audible alarms. According to Neaverth, that’s when human error entered the equation and ultimately played a major role in the shutdown.

“What he believed to be a reset button was actually an emergency kill switch for power, resulting in all of the power at all of the public safety work campuses losing power and falling offline,” he said.

Neaverth said the power kill button is designed to terminate electrical hazards in situations in which firefighters may have to enter a building with active electricity. The activation of the button in this instance, however, bypassed street power, the Uninterrupted Power Systems, and even the generator unit until the system was reset by the contract electrician from Emerson Electric.

Legislature Majority Leader Joseph Lorigo directed pointed questions to Department of Public Works Commissioner John Loffredo

“What happens if the exact same thing happens tomorrow? Do we have people trained in what to do?” he asked.

In response, Loffredo said officials are reviewing the timeline of events and are unsure whether the buildings and grounds employee thought to have pushed the kill switch actually did so.

County officials are sure though that from the time of the power outage at 3:30 a.m., the public safety campus did not regain the ability to process 911 calls until 6:54 a.m. — nearly 3½ hours in which those dialing 911 were put into a lengthy queue.

Compounding the problem was an additional software failure that cut off 911 calls to police departments.

Representatives from Verizon and Siemens, the company responsible for the building’s climate-control system, declined to attend the Legislature’s April 7 hearing. Verizon contends that the company performed a “root cause analysis” to determine the key issues and maintains that its network did not fail. Neaverth and his department, conversely, say blame should ultimately be placed on Verizon for the failure.

Public Safety Committee Chairman Edward Rath III scheduled an additional hearing for Thursday, May 5, to receive updates as well as to allow Verizon and Siemens the opportunity to answer direct questions from legislators. Rath noted that subpoenas could be issued if the two companies don’t respond.

“If we have to compel them to come in, then that’s what we’re going to do,” he said.

Steve Matisz, a supervisory fire dispatcher for the Town of Amherst, told legislators that he is aware of four other outages that have taken place in the county’s new system, though none have been as extensive as the most recent one.

“Our challenge is, how do we get people to know where to call in this situation? For 30 years, we’ve been putting on the side of our trucks, ‘dial 911,’” Matisz said.

He added that his concern going forward is that a similar systemic breakdown could put the lives of county residents in danger.

“I worry that somebody may go in to look for somebody and they’re dead on the floor with a phone in their hand because the call didn’t go through.”

Return to top

Clarence Special Events 2018
Click for schedule