Typical Continuous Delivery, where builds are promoted along a pipeline or lifecycle is broken for complex architectures – especially web service based applications. I love these build promotion approaches and have cheered them on for six years, but they use a very imperfect model.
In the diagram below, we look at a typical build deployment pipeline including a build, a variety of testing environments, approvals, and eventual release to the live production environments. The three components (Services 1,2,3) are all set to use this model independently.
There are several problems when using this model:
- The web services are interdependent. A change in an API for #1 may break #2 unless an update for it is also released.
- Because things are released together, the “sign-off” rarely applies to the release of a single version of a single component. Instead, a collection of items is approved.
- The pipeline always starts with a build. While appropriate for most web services, releases often include components that are not built: incremental static content moves, database updates or report migrations.
- The build pipeline is too rigidly ordered (a complaint I’ve made before).
The book also suggests an “Integration Pipeline” in Chapter 13. In this approach each major component has a short testing pipeline and is marked as good. A new build containing all the latest good components is created and moves through its own pipeline towards production release. The authors bend over backwards encouraging you to avoid this practice if you can.
I understand why all too well. The “build” that contains the latest good versions of each of the components is hard to work with. Sometimes we only want to release half the components. Sometimes, the “latest good” of one component is incompatible with the “latest good” of another and we would rather release a slightly older version of that component. Only with exceptional discipline can we eliminate the numerous failure modes of assembling all the “latest good” and people are not good at discipline.
Using a build pipeline to model release orchestration and planning activities is simply a bad fit. It feels like wearing shoes a size too small. You can squeeze into them and they protect the soles of your feet, but expect pain.
What we want to do instead is to decouple our deployment pipeline or delivery chain from the concept of “a build”. Instead, we should focus on the idea of a Release Set or Snapshot which is a lightweight deployable unit for these larger systems.
Snapshots are similar to Integration Pipeline builds in that they reference versions of related components. But rather than retrieve and assemble those related components into a final application in as part of a super build, we instead, use a release set to logically collect the component versions. The Release Set or Snapshot is not “built” so much as “declared” and is analogous to the classic Release Manager’s spreadsheet that details the versions of each component that should be tested and released together. The Release Set also provides a natural granularity for managing approvals.
We also need to reject the notion that a Release Set will contain exactly one version of each component. For instance, in the Report migration example above, there might be a number of different new reports that have been independently developed and tested. If the component is naturally promoted in an incremental fashion, we include multiple, ordered versions in the Release Set.
Last week I worked with a team that was trying to figure out how to better update a complex CRM application. As an example, the team shared a recent release plan detailing updates to each of several dozen components. Modeling that in a build pipeline would be a nightmare. With release snapshots, we have a promising way forward.
I’ll be talking a lot more about the limitations of traditional CD, approaches for accomadating those limitations, and alternate models like Release Sets in our upcoming webcast, “Next Gen Continuous Delivery.” Please join us for that event.
This is not the end of the story. What if you are in situation when say Service 1 is external consumer you have no control of? You end up supporting multiple versions in Prod with some roadmap for retiring old versions and rolling out new ones. I’d be happy to implement something like this with uDeploy. Extend uDeploy to SOA scenario.
Evgeny,
Just wanted to leave you a note. We were white-boarding external dependencies again last week. Keep your eyes peeled. I think we’ve got a nice solution coming down the pipe.
Hey Eric. Great post.
I think it’s important to distinguish between the deployment pipeline pattern, and the implementation of it in tools. The deployment pipeline pattern allows you to control and the flow of inventory (in the form of changes) from version control to production, and get visibility into what is deployed into each environment, where it came from, what processes it has been through, who approved them, and so forth.
I think release sets are a great way to implement a deployment pipeline in a complex environment where you have multiple collaborating services, as you describe.
Of course many tools’ _implementation_ of the deployment pipeline won’t yet let you do that (yet) – I look forward to seeing how UrbanCode has addressed this.
I think we agree on the point of the pipeline for the most part. I like a view of the path that’s been taken, and what (normally) should come next.
The “trouble” is more with tools implementing and the “build centric” view of the world which is probably a result of the industry’s collective experience coming at this problem after implementing CI . With AnthillPro we’ve been doing the build promotion model a long time, and the challenge of missing logical release sets has been a growing pain as more and more teams move to SOA type models that lack the API backwards compatibility that really supports independent promotion. I know you’ve written extensively, and well, about why this sort of compatibility is a good thing.
uDeploy provides the release set approach and has helped us overcome some of these challenges in the field. There’s still work to be done, but there’s definate promise in wiring together the traditional CD type tools with a tool like uDeploy that adds in release snapshots. We provided a rough overview in a webcast showing AnthillPro and uDeploy together. That webcast was soft on functional testing (unfortunately realistic), and directed promotion paths, but starts laying out the road map.
Pingback: Always Agile · Pipeline antipattern: Uber-Binary
Hi Eric
I agree with much of what you say, but I think Jez’s point re. patterns vs tooling is an important one. One of the strengths of the Continuous Delivery book is its focus upon concepts and patterns rather than implementation, but the downside is that a lack of prescription can lead to poor tooling decisions.
Continuously Delivering multiple, complex applications with source and/or binary dependencies is a tough problem to solve, and while the Integration Pipeline approach can work it lives and dies by the packaging implementation. I’ve written on the same topic recently at http://www.stephen-smith.co.uk/pipeline-antipattern-uber-binary/, particularly on why the Uber-Binary (Build Of Builds) packaging implementation is such a bad idea. I also have a follow-up in the works describing a modified approach that I’ve successfully scaled up on an estate of ~95 greenfield and legacy applications.
Cheers
Steve
As a tools guy, I can tend to get too far down into the weeds. Arguably, the “build of builds” approach is just one implementation of “a package (or packaes) suitable for deployment by composing the appropriate collections of binaries”.
But, with the tools most of us have on hand, we will tend to use the build of builds approach. Our tools are build centric, and so we perform more builds.
I think the “Release Set” I’ve proposed is largely inline with your manifest file as the artifact. I just tried to coin a new term to aggressively sever from the “build” concept. When all you have is a hammer, it’s helpful when things yell, “I’m not a nail!”
Pingback: Maslow's Hammer: The curse of tools | UrbanCode Blogs