After early problems scaling its infrastructure nearly derailed the company before it got off the ground, Twitter wound up developing some world-class systems for keeping the service alive. It announced Thursday that it is moving a key part of that system off its own infrastructure and onto Google Cloud.
Twitter’s cold storage and flexible compute Hadoop clusters will now run on Google’s infrastructure, CTO Parag Agrawal said in a blog post Thursday. Hadoop is an open-source project for organizing “big data” that is nearly ubiquitous at web-scale companies like Twitter, and companies who were early to Hadoop tended to run the compute portion and the storage portion of the software on the same machine in their own data centers.
More recently, the rise of cloud computing and deployment options like containers have made it possible to separate Hadoop compute from storage, which gives companies like Twitter more flexibility and paves the way to let it scale its services into the future, assuming Twitter user growth ever really accelerates again. It looks like Twitter still plans to maintain its own infrastructure for most of its needs, but this deal with Google could be an opening for future services to move over down the road.
The deal also shows that the pioneering webscale companies of Twitter’s generation are kicking the tires on hybrid cloud deployments after making massive investments in their own infrastructure a decade ago. Hybrid cloud is often thought of as a way station for big non-tech enterprise companies who see value in cloud computing but are primarily concerned with not breaking anything. When tech companies like Twitter that know a thing or two about infrastructure start seeing opportunities to run workloads in the cloud, that could open up some new areas of competition for cloud vendors.
