2.8 Delivery processes & tools
Your delivery toolchain impacts the entire team, every single day. Getting it right means massive productivity gain.
Before thinking about your toolchain, it’s important to have a clear understanding of what you’ll be delivering. Accordingly, checking your definition of ready will ensure you’ve completed the following activities and have necessary information at hand.
Before starting
Your delivery processes and accompanying tools address how you build and deliver software. The purpose of the pipeline is reliability and repeatability — on-time delivery, expected features, no surprises, sprint after sprint. This activity identifies the capabilities your pipeline needs to support and the products that will fulfill them. Companion blog posts on the Customer Obsessed Engineering blog go deep on individual processes (security, observability, canary releases and more).
You’ll need 2.6 Target state architecture and 2.7 Architecture foundation finished first — together they establish the product components your toolchain has to support.
Delivery pipeline basics
The obvious purpose of your delivery pipeline is to build and deliver software. There is a more important purpose: ensuring reliability and repeatability of software delivery. That can only be done through automation. Automation removes tedious and error-prone manual operations, which drives reliability and repeatability — and accelerates the entire pipeline.
A basic delivery pipeline needs to support those core principles. That means instilling the following features as a starting point:
Ephemeral environments
An ephemeral environment is just a build environment that is transitory. It exists while it is needed, and goes away when it’s no longer needed. The valuable thing about ephemeral environments is that they are automated — there’s no “standing up” a new environment manually. Every environment is built behind the scenes, when needed, by the delivery pipeline. The best way is to tie code repository branches to environments. Every time a new branch is created, a new environment spins up. That’s true for production, for staging, for transient test environments, even for feature branches.
Security
Static Application Security Testing (SAST) needs to be there from day one. One of the most costly software features to address after delivery (or even late in the delivery cycle) is security. I’ve seen software projects delayed by over a year because security was not considered from day one.
Including the basics of security from day one means starting with code security scanning, dependency scanning, secret detection and secret management as a bare minimum. It doesn’t have to be complicated. There are quick paths to have all of this from day one.
Test batteries
If everything is automated, we have to verify the software behaves correctly in an automated fashion. Integrate quality gates into the build pipeline. If tests fail, the build fails, and the code can’t be merged (no exceptions).
Zero trust architecture
A zero trust architecture means no unnecessary access to infrastructure and environment configuration. It goes hand-in-hand with ephemeral environments and release orchestration — by automating delivery, we cut out manual steps.
Staging, testing, and especially production environments should be strictly zero trust, providing no access to any team members. That means using Infrastructure as Code (“IaC”) to stand up and tear down servers, pushing configuration change through code, never by taking manual action.
Release orchestration
Software needs to be delivered in an automated, reliable fashion. Release orchestration automates the process, cutting out potentially error-prone manual steps. It also shifts release management to the business and away from the development or operations team. The product owner (or any authorized role) decides when to push the application to production.
Observability
Telemetry and observability need to be a part of your toolchain from the start. It doesn’t have to be all-encompassing and can evolve over time. A zero trust architecture and fully IaC driven infrastructure means zero access to many environments — but access won’t be needed if the development team can easily view metrics, logs, and telemetry from a dashboard, and push configuration changes through code.
Canary releases
Canary releases make it possible to try out a change or new feature on a limited audience. It’s the best way to make sure nothing is going off the rails during release, and also a great way to try out new features in beta or on limited recipients to gather feedback.
There are a couple of ways to do it. The easiest is to deploy a build to a fraction of your servers. By adding feature flags to the development process, you can also turn features on and off — then segment by customer number or preference (“I’d like to try out new beta features”).
Guide to delivery processes & tools
Our goal in this activity is to establish your overall delivery toolchain. The activity is divided into three steps:
- Reviewing and selecting capabilities
- Selecting technical solutions
- Establishing your toolchain roadmap
In the first step we’ll review potential pipeline features that would benefit the team if integrated into the toolchain.
Once you have a catalog of features identified, the next step is to identify the technical solution — the product or approach that fits the purpose. Then comes establishing priorities and deciding when each feature will be built.
Throughout all of this, the toolchain and your delivery pipeline should be regarded as an integral part of your software product. The features it supports are of equal importance. Establishing the ability to reliably and repeatedly deliver, sprint after sprint, is essential to the success of your project and, hence, the product.
Inputs
- Participation from team leads, architectural leadership, and delivery team.
- A component map of all the different project capabilities you need to plan for (such as testing, release management, data management, security, etc).
- A service catalog listing technologies used in product delivery.
- Your high-level architecture model, a simplified model describing how components fit into the ecosystem and what patterns you’ll use to glue it all together.
- A specific architecture realization, demonstrating how the abstract model will be specifically implemented in your target state architecture.
Process
The first step is to review your overall delivery processes and agree on what your future state needs to include. Separately, you can identify the technologies, tools and products that meet those needs. Depending on the scale of your project, you may want to plan this over several sessions.
You can use this template as a starting point for documenting your delivery processes. It includes placeholders for all of the processes named above, as well as space to enter potential solutions. As with the service catalog template, identify which solutions are approved versus under consideration.
As is often the case, you may need to run a spike or breakout session to do additional analysis. As candidate designs are fleshed out, those designs can be brought back to the team for wider discussion.
Prepare
- Organize your working sessions appropriately, based on the scale of your system. Use your delivery processes map (above) and associated template as an aid in coordinating topics and outcomes.
- Share materials ahead of time with your working team, and set clear scope for your working sessions.
- Schedule accordingly, knowing that some of the larger technical decisions need their own time for evaluation. Plan between 2 and 4 hours, giving the team plenty of time to work independently on breakout tasks.
- Set up your delivery processes template and make sure everyone has access, so you can collaboratively work on it.
Execute
- Process selection.
- Your first workshop should focus on identifying candidate processes that you will build into your delivery chain.
- Identify everything you want to do, even those items that you’ll put off until later. Knowing that you plan to build them will influence early design and implementation.
- Those items that you view as essential, go ahead and mark as high priority. You’ll come back later to reprioritize.
- Technology selection.
- Next, in separate workshops if needed, identify candidate solutions.
- Continue filling out your delivery processes, adding candidate solutions. If you have consensus on what you’ll use, go ahead and mark it as approved. Otherwise, note the solution is pending evaluation.
- In many cases, a single solution will check off several requirements. For instance, adopting GitLab as your delivery platform will check off all of the basics and then some.
- As needed, schedule breakout sessions for more detailed analysis: Either to investigate and select candidate technologies, or to evaluate competing options.
- Prioritization.
- As a group, review your delivery processes and tool selection. Prioritize accordingly, keeping in mind that you’ll need to have the basics in place to simply deliver your first build.
- Roadmapping.
- With priorities clear, create feature descriptions and appropriate implementation details for each one. Schedule your features alongside software development tasks.
- Keep in mind that delivering a single build to a test or staging environment is going to require your foundation: Ephemeral environments, some amount of SAST (such as code quality and secret detection). Don’t take shortcuts. Build your foundation from the start.
Outputs
- Delivery processes catalog. A catalog of all processes and candidate technologies you’ll use in your delivery pipeline. The list should provide a mechanism to monitor and review proposed technology choices, and prioritize tools (to aid in scheduling feature development).
- Prioritized delivery chain features. A list of features describing the functionality, and prioritizing the development, of your toolchain.
A note on GitLab
GitLab is an excellent starting platform. From a rapid productivity perspective, there’s nothing like it. You can go from “zero to 60” and have a fully functional delivery pipeline on day one with GitLab. It gives you environment management (and it’s easy to build ephemeral environments, too), release orchestration and auto deploy, secret detection, static application security testing, infrastructure as code scanning, dependency scanning, container scanning, dynamic threat prevention, API security and fuzz testing.
You don’t need to use GitLab. You can build all of this with GitHub Actions or a Jenkins build pipeline. But then you have to build it. GitLab has a huge advantage: you can have your pipeline up-and-running, with all of these security features, in minutes by turning on “Auto DevOps.” It’s a preconfigured pipeline that gives you a solid foundation from day one, with virtually zero work.