After a year of running containers in production, T-Mobile felt comfortable enough Tuesday night before a crowd of Seattle techies to give them a peek behind the curtain at its live production systems.
Around 100 people packed into the offices of SURF Incubator for a classic beer-and-pizza tech talk led by Thom McCann, senior manager at T-Mobile and overseer of the wireless company’s container strategy. T-Mobile is still edging onto the cloud: it’s running about 15 percent of its app workloads on Amazon Web Services, McCann said, leaving an awful lot still running on on-premises servers, but that is changing.
McCann and two other T-Mobile engineers — Amreth Chandrasehar and Yogesh Sharma — showed attendees the monitoring dashboards of T-Mobile’s containerized applications. They showed how T-Mobile can replicate its entire cloud infrastructure running in Oregon on AWS servers over to the US-East region in about seven minutes, a process that could take up to six hours for applications that aren’t running in containers.
The company is running 180 containers on its production servers, and 400 on non-production servers that are more or less held to the same standard as the production ones, McCann said. And on any given day, depending on its needs, it spins anywhere between 200 to 500 containers up and down as developers test changes to its applications. (For scale, Google runs 2 billion containers a week, and that’s likely an outdated stat by this point.)
McCann tempted fate by telling developers that T-Mobile will have experienced zero downtime in the year it first started using containers in production if it makes it through Sunday without any issues. Most of that container adoption has come in the last five months as more and more people within the 1,300-strong IT department at T-Mobile realized how their apps could benefit from this approach.
And the accountants like it too. Containers have allowed T-Mobile to make better use of their AWS Reserved Instances, which are use-em-or-lose-em, and can better track cloud spending across development teams, McCann said.