T-Mobile explains cause of major Monday outages

The chair of the Federal Communications Commmission said T-Mobile’s nationwide, hours-long outage Monday was “unacceptable” and that the regulators will investigate.

“We’re demanding answers — and so are American customers,” FCC Chair Ajit Pai tweeted.

T-Mobile, one of the country’s three largest cellphone service providers, said it had a “voice and text wireless issue” that began around noon EDT Monday. The company said at 1 a.m. Tuesday that all problems had been resolved.

The company blamed an internet traffic issue that caused problems with its network for the outage.

AT&T and Verizon both said their networks were operating normally. But calls between their customers and T-Mobile customers could have run into trouble because of T-Mobile’s issues, creating the impression of a widespread communications failure.

T-Mobile elaborated on the cause of the massive outage, saying that the issue stems from a fiber circuit failure in the Southeast.

“Our engineers worked through the night to understand the root cause of (Monday’s) issues, address it and prevent it from happening again,” Neville Ray, T-Mobile’s president of technology, wrote in a blog post Tuesday evening. “The trigger event is known to be a leased fiber circuit failure from a third-party provider in the Southeast.”

Ray said T-Mobile’s existing safeguards that are meant to ensure such circuit failures won’t impact customers had failed and resulted in an “overload situation.” That overload caused what Ray described as an “IP traffic storm” that spread from the Southeast to create capacity issues across the network that supports T-Mobile’s voice-over-LTE calls.

Outages affected the network across the United States for much of the afternoon and evening Monday. The website Downdetector, which tracks outages across popular services and websites, indicated that the issue seemed to peak at 3 p.m. on Monday, with more than 93,000 reports of issues on the network.

Customers on other carriers had reported problems with service as well, according to Downdetector, but both AT&T and Verizon had said their networks were operating normally on Monday. Ray also said that Sprint customers were unaffected by the service disruptions.

The FCC has fined telecom companies in the past for network outages. T-Mobile paid a $17.5 million fine for two nationwide service outages on the same day in August 2014, which together lasted three hours and prevented customers from being able to call 911.

Public safety officials were worried about lack of access to 911 on Monday as well. For example, the Redmond, Washington, police department tweeted that T-Mobile customers should have “an alternate plan in place in the event you need to call 911.”

T-Mobile, which is based in Bellevue, Washington, became one of the country’s largest carriers, along with AT&T and Verizon, after buying rival Sprint. The company has started integrating the two networks.