Shippable: Evolve Your Continuous Integration Strategy

Continuous Integration (CI) is a hot topic right now. Companies have embraced the idea that automatically integrating software changes early and often is an effective approach to reduce cycle time, increase quality, and reduce end-to-end costs.

But what constitutes early and often? (Hint: the earlier the better, and you should do it at multiple stages of your software delivery process.)

To identify where to introduce CI, it helps to understand how the need for CI changes as the complexity of your software delivery process grows. Consider the following variations in complexity:

In the simple environment, the software delivery process is managed by a single person who “does it all” for the application. In this situation, only a few opportunities to introduce issues since a single person develops the changes serially and is solely responsible for having introduced any change that breaks the system. Yet even in this simple case, the developer can benefit from CI by automatically testing the code every time it is changed, immediately discovering any bugs and ensuring all functionality that worked previously still works.

Where things really get interesting is as the complexity grows. Let’s look at how the software delivery process changes for a typical multi-tier enterprise application:

• Multiple developers work simultaneously on different parts of the code
• The application contains a large number of components and dependencies among components
• Each of the tiers of the application likely run on different machines with different software
• Multiple pre-preproduction environments likely exist, including development environments, system integration testing environments, performance testing environments, and user acceptance testing environments
• Development, testing and operations are likely split between different people or teams

In this complex environment, opportunities to introduce errors grow exponentially as the number of developers, components, environments, and people making changes to the environments grows. And these errors typically compound the later in the software delivery process they are discovered, making them increasingly costly to resolve. Even a relatively small enterprise application team quickly finds itself struggling to get software changes through the lifecycle. Some of the most common reasons errors are introduced include:

• Inconsistent dependencies/packages running locally on a development machine in comparison to pre-production and production environments
• With distributed source code repositories (e.g. Git), local copies get out of sync with changes made by other developers
• Changes are integrated in bulk making it difficult to identify what change caused an issue
• Downstream dependencies break due to changes introduced upstream
• Environments do not match across Pre-Prod and Prod resulting in code that works in one environment, but not in others

With the right CI strategy and tools, you can drastically reduce the impact of these common issues and, in particular, the compounding nature of them. To do so, first, you will need a strategy to eliminate or catch these errors as early as possible, so less complexity must be dealt with in later stages. Second, you’ll need to repeatedly check that changes continue to integrate successfully as the system changes and as changes progress through the various software delivery process stages and environments. An effective CI system will enable both of these strategies.

In the above example, in fact, there isn’t just one Continuous Integration process to run, but actually four distinct CI processes. CI should occur for each of the following scenarios within the software delivery process:

1. By each individual developer before they push local code commits to their individual fork
2. By the owner of the master branch, prior to merging code commits into the master for a pull request
3. By the owner of the master branch, after merging a pull request into the master, for all outstanding pull requests
4. By the owner of each pre-production environment, prior to deploying code changes to their environment.

Most companies have established processes for performing integration testing for #2 and #4, either with manual or automated testing. Some have successfully moved to continuous integration for these, integrating and testing changes automatically on a daily basis. Few, however, have implemented continuous integration for individual developers or for outstanding pull requests in the queue after successfully merging a prior pull request. If you have not yet implemented continuous integration for each of these scenarios, you’re falling behind and need to develop your strategy in order to evolve faster. If you need any assistance with forming your Continuous Integration strategy, please feel free to reach out to me to discuss, at [email protected].

One final thought. Above, I discussed the Multi-Tier Environment as a typical enterprise application architecture. That architecture will be joined by microservices and containerized applications soon, with some with significant benefits, but also added complexity. So, don’t delay. It’s the right time to take your continuous integration strategy to the next level.

Tom Trahan is the VP, Business Development at Shippable, a startup providing continuous integration services in the cloud. Formerly, Tom was Co-Founder and CEO of EnterPAS, a Director at Microsoft, and a consultant at SAP. Reach him at [email protected] and follow Shippable @beshippable.