Nobody really knows what lies ahead for the future of open-source software as cloud computing becomes the dominant force in enterprise tech, but the times are definitely changing.
Just about anything that Amazon Web Services does has massive ripple effects throughout this world, and last week’s decision to release its own open-source version of Elasticsearch, a popular engine for searching and analyzing internal company data maintained by newly public company Elastic, was no exception. AWS open-source czar Adrian Cockcroft was careful to describe the Open Distro for Elasticsearch as a distribution, rather than a fork, but the move underscores a fundamental conflict between companies based around open-source projects and the growing popularity of cloud service providers.
Elasticsearch was first released as an open-source project in 2009, when startups on a budget were the primary customers for cloud computing. Elastic launched in 2012 to maintain that project and build commercial services around it, and the company recorded $70 million in quarterly revenue during its last earnings release after going public last year. Its market capitalization was $6.38 billion during Thursday morning trading.
Open-source commercial ventures like Elastic are generally banking on the fact that installing, running, and patching an open-source project as part of a production-caliber application is pretty hard without specialized engineering talent that lots of companies can’t afford to hire. Those companies are willing to pay someone to manage a lot of the heavy lifting around those projects, and in many cases companies like Elastic also build premium and proprietary bells and whistles that sit alongside the free project.
Likewise, cloud companies like AWS offer a lot of popular open-source projects as cloud services, which those licenses generally allow and which their customers demand. AWS first launched an Elasticsearch service in 2015 based around the core open-source project, but according to a blog post written by Cockcroft it noticed last year that Elastic was co-mingling open-source code with proprietary code, making it hard to understand which was which.
“For example, neither release notes nor documentation make it clear what is open source and what is proprietary. Enterprise developers may inadvertently apply a fix or enhancement to the proprietary source code,” Cockcroft wrote.
These distinctions are important because once a developer chooses a service — delivered through the cloud or hosted on self-managed servers — as part of their application, that developer needs to know that the code behind the service will remain available and predictable for years to come. Developers who inadvertently use proprietary code in a commercial application without a paid license can find themselves spending more time with lawyers than code, and that’s not a productive use of their time.
One of the unique issues between AWS and Elastic is that a lot of proprietary code covered some fairly basic aspects of using Elasticsearch, such as authentication: technically, users have to either come up with their own way to log into Elasticsearch or pay Elastic for the privilege.
So AWS teamed up with long-time partners Expedia and Netflix to create Open Distro for Elasticsearch, which is an Apache-licensed open-source release that AWS intends to contribute to the main project. “With the first release, our goal is to address many critical features missing from open source Elasticsearch, such as security, event monitoring and alerting, and SQL support,” Cockcroft wrote.
Elastic CEO Shay Banon did not take kindly to AWS’s move, suggesting in his own blog post last week that AWS first approached Elastic wanting “preferential treatment” compared to other customers before Elastic said no and AWS released its version. “We have a commitment that we will treat a single developer contributing to our products the same as others,” he wrote.
In response to a GeekWire inquiry, Elastic’s Steve Kearns, vice president of product management, said in a statement that Elastic thinks it has clearly delineated the type of code in the Elasticsearch repository. (An AWS representative did not respond to a similar inquiry.)
“All of the code for our proprietary features are kept in a separate top-level folder called “x-pack,” to avoid any mixing or confusion. We also include a header on every source file, indicating whether it is licensed under Apache 2.0 or the Elastic License, to prevent any ambiguity,” Kearns said in a statement.
As we’ve documented over the past year, AWS has been the favored punching bag of a certain class of companies built around open-source projects over the last decade. The ease at which those open-source projects can be offered as revenue-generating cloud services without any obligation to contribute code back to the project has not sat well with several companies, who last year changed the licensing terms of their open-source projects to make it harder or impossible for AWS to offer those projects as cloud services.
A lot of the rancor stems from fear, fear of the growing power of AWS and its control of the still-expanding cloud infrastructure computing market. Companies that might have used open-source projects as part of applications running on their own servers and paid companies like Elasticsearch for premium features are now running on AWS, and it’s very easy to use the AWS version of an enterprise tech service once you’re already on the platform.
“That commercial open source providers took these actions having been advised of these and other risks in advance says everything about how these businesses view their prospects in a world increasingly dominated by massive providers of cloud infrastructure and an expanding array of services that sit on top of that,” wrote Stephen O’Grady, principal analyst and co-founder of industry analyst firm RedMonk, in a blog post on the matter last week.
It’s not hard to understand why AWS would want Elastic to offer a basic set of services as part of the main open-source project, and also not hard to understand why Elastic wasn’t necessarily interested in adjusting its product-development strategy to suit AWS. AWS did not approach Elastic about a commercial reseller deal like the ones Elastic has cut with other cloud providers like Microsoft and Google prior to releasing its distribution, Kearns said.
This latest dust-up illustrates once again that open-source software is going through some growing pains as the assumptions behind enterprise computing change thanks to cloud computing.
The concept of open-source enterprise software paved the way for the tech boom of the last decade, as fledging companies were able to draw on the contributions of others to build their own tech infrastructure without having to reinvent the wheel over and over again. When bigger companies with deep pockets solved tricky tech problems that commercial software couldn’t or wouldn’t address, lots of other companies benefitted from that work.
But if open-source developers are unable to monetize their code by forming independent companies designed to support that work, we’re going to see a lot more top-down open-source projects run by the Big Three cloud vendors and the foundations over which they exert great influence. In some ways this seems inevitable as new power brokers emerge in enterprise tech; both open-source enterprise technology and cloud companies themselves are rebels against an older era of IT procurement dominated by rapacious software vendors and stagnant hardware.
Yet this risks grinding the incredible pace of enterprise tech innovation over the last decade to a halt, as important projects get tied up in corporate politics or released in service of a corporate agenda. Large companies are notorious for missing key shifts in technology trends while focused on their existing businesses, and startups have been filling that gap on a reliable basis for more than a decade.
“I am skeptical of companies who don’t really have open source in their founding DNA to do the best thing for the community in the long-run, and I worry that the cloud is just moving us back to a world of proprietary software,” wrote Camille Fournier, managing director of Two Sigma and author of a popular book on engineering management, in a post on Github last week.