Skip to main content

T-Mobile explains why its network went down, hard, on Monday


“We did not meet our own bar for excellence,” writes T-Mobile’s CTO
If you’ve been wondering what could knock out one of the United States’ three big cellular carriers’ ability to deliver calls and text messages — and keep it that way for most of an entire day — T-Mobile now has a partial answer that pertains to its extensive nationwide outage Monday.

The short version, if we’re reading this correctly: a fiber-optic circuit failed, and its backup circuit also failed, which caused a chain reaction that strained the network to the point that many calls and texts couldn’t make it through.

Every day we see the vital role technology plays in keeping us connected, and we know T-Mobile customers rely on our network to ensure they have connections with family, loved ones and service providers. This is a responsibility my team takes very seriously and is our highest priority. Yesterday, we didn’t meet our own bar for excellence.

Many of our customers experienced a voice and text issue yesterday, specifically with VoLTE (Voice over LTE) calling. My team took immediate action — hundreds of our engineers worked tirelessly alongside vendors and partners throughout the day to resolve the issue starting the minute we were aware of it. Data connections continued to work, as did our non-VoLTE calling for many customers and services like FaceTime, iMessage, Google Meet, Google Duo, Zoom, Skype and others allowed our customers to stay in touch. Additionally, many customers were able to use circuit-switched voice connections and customers on the Sprint network were unaffected. VoLTE and text in all regions were fully recovered by 10 p.m. PDT last night. I’m happy to say the network is fully operational… and we’re working day in and day out to keep it that way.

Our engineers worked through the night to understand the root cause of yesterday’s issues, address it and prevent it from happening again. The trigger event is known to be a leased fiber circuit failure from a third party provider in the Southeast. This is something that happens on every mobile network, so we’ve worked with our vendors to build redundancy and resiliency to make sure that these types of circuit failures don’t affect customers. This redundancy failed us and resulted in an overload situation that was then compounded by other factors. This overload resulted in an IP traffic storm that spread from the Southeast to create significant capacity issues across the IMS (IP multimedia Subsystem) core network that supports VoLTE calls.

We have worked with our IMS (IP Multimedia Subsystem) and IP vendors to add permanent additional safeguards to prevent this from happening again and we’re continuing to work on determining the cause of the initial overload failure.

So, I want to personally apologize for any inconvenience that we created yesterday and thank you for your patience as we worked through the situation toward resolution.

Neville Ray

T-Mobile President of Technology

It’s not clear which third-party provider’s fiber circuit failed. There was a report on Monday that Level 3, one of the world’s major internet backbone providers, was experiencing an outage, but a spokesperson told TechCrunch differently.

Comments

Post a Comment

Popular posts from this blog

Black Adam Can Be The DCEU’s Wolverine

The upcoming Black Adam movie could position its title character as DC's equivalent to Wolverine. Though Black Adam has taken quite a long time to get rolling, the first day of DC FanDome gave a peak at what audiences can expect, from name-dropping several Justice Society of America members who make appearances to Dwayne Johnson stressing the harsh origins of the title character. Johnson's description of Black Adam himself also may offer a sense of just what role he'll occupy in the world of the DCEU. Johnson would place particular emphasis on Black Adam having been a slave who achieves his freedom after being granted power from the gods and who now seeks to unleash his own brand of justice on the world. Aside from the moral ambiguity of Black Adam, Johnson has also repeatedly spoken of him as a game-changing figure in the DCEU with the power level that he operates at. Meanwhile, with the way Black Adam is being presented as a character, he could carry some notable parallel...
Post a Comment
Read more

Studio Ghibli Meets The Legend Of Zelda in Stunning Breath of the Wild Poster

The Legend of Zelda  is  one of Nintendo's longest running, and most popular, game franchises , with the recent title of  Breath of the Wild  giving us a fresh new take on characters such as Link, Zelda, and Gannon, as one fan has decided to show what this game might look like as a product of  Studio Ghibli . Ghibli, which is often considered to be one of the most legendary animation studios, having created anime feature length films such as  Princess Mononoke, Howl's Moving Castle, Spirited Away , and many others that have propelled the studio's brand across the world. Studio Ghibli is set to release a new movie later this year in  Earwig and the Witch , taking the opportunity to venture into the world of computer graphics for this upcoming anime. Following this entry, Ghibli is looking to adapt the popular story known as " How Do You Live ", meaning that there is still plenty of work for the legendary animation house to do.  Hayao Miyazaki came out of retirement...
Post a Comment
Read more

The Earth is making the moon rust

(CNN)The moon is getting rusty. Scientists had the same reaction you probably did when they reached this conclusion. It shouldn't be possible -- after all, there's no oxygen on the moon, one of the two essential elements to create rust, the other being water. But the evidence was there. India's lunar probe, Chandrayaan-1, orbited the moon in 2008, gathering data that has led to numerous discoveries over the years -- including the revelation that there are water molecules on its surface. The probe also carried an instrument built by NASA that could analyze the moon's mineral composition. When researchers at NASA and the Hawai'i Institute of Geophysics and Planetology analyzed the data recently, they were stunned to find hints of hematite, a form of iron oxide known as rust. There are plenty of iron-rich rocks on the moon -- but rust is only produced when iron is exposed to oxygenand water. At first, I totally didn't believe it. It shouldn't exist based on the...
Post a Comment
Read more