Description

Remember when one team would build software, then hand it off to another team to deploy it and get it working in production? I seem to recall we > came up with better ways to deliver software. We even made up cool buzzwords like “DevOps” and “Continuous Delivery.”

Many years later, I see people using LLMs to iterate on building an application and treating production readiness as an afterthought. That might > be fine for demos and personal projects. But if we’re going to use agents to build real, business-critical software, we need to use agents to > make sure the software is fit for purpose. We need to know that the software and its infrastructure performs, scales, recovers when things go > wrong, and stays secure and compliant.

I maintain that continuously ensuring software is production-ready as it’s developed is at least as important when using agents as when we > hand-code it. I’ll talk about how to pull the path to production inside the agentic development flow. And I’ll share why doing this kind of blew up on me the first time, and how I had to adjust my thinking to make it work.

Some of the influences for this talk

Harness engineering for coding agent users, Birgitta Böckeler

Harness engineering beyond skills: Using sensors to keep your coding agent in check, (Video) Birgitta Böckeler, Chris Ford

How I Use AI to Code, Chris Parsons

As we build, so we believe, Adam Jacob

Skills Are Context, and Context Needs Tests, Paul Stack

Credits for photos used in the talk:

• Homa Appliances on Unsplash
• Hyundai Motor Group on Unsplash
• Peter Herrmann on Unsplash
• Kathrine Coonjohn on Unsplash
• Abhinav Bhardwaj on Unsplash (also the image used for this post)

Some of the influences for this talk

Harness engineering for coding agent users, Birgitta Böckeler

Harness engineering beyond skills: Using sensors to keep your coding agent in check, (Video) Birgitta Böckeler, Chris Ford

How I Use AI to Code, Chris Parsons

As we build, so we believe, Adam Jacob

Skills Are Context, and Context Needs Tests, Paul Stack

I’m experimenting with how to use agents to build AWS infrastructure to run a sandboxed instance of OpenClaw for myself. It’s a nice dogfooding project that I’m actively relying on and needs constant work to keep up with frequent software releases and changes as I add new capabilities and data.

I could just fire up Claude Code with AWS credentials, give it a prompt or spec for the infrastructure I need to run OpenClaw, and let it loose with the AWS CLI - “vibe infrastructure engineering”. Claude is happy to crack on with it. The agent will make mistakes along the way, but it knows the outcome I want and will iterate on the implementation until it gets there.

The challenge is that my coding agent does a messy job of it. It takes multiple tries to find a command that works and casually blows away data and application configuration. When I ask for a minor change it replaces a solid, efficient implementation with a slow, less-reliable bodge. At one point it started implementing a Slack gateway, something already built into OpenClaw. Working with an AI agent reminds me of working with an old-school system administrator who likes to build everything by hand.

Code for Consistency

Infrastructure code gives me consistency. I define the set of resources I need and how to assemble them. Then I can build, rebuild, and replicate the system, and know it’s essentially the same. Infrastructure code is an anchor for my coding agent. Rather than starting from scratch for every change, the agent starts with the existing code, and I guide it to incrementally improve the quality of the infrastructure. The agent can also use existing code as a reference for new infrastructure with similar requirements, so it will thrash less deciding how to build it.

Having code helped me move from my original OpenClaw build to a new AWS account with additional capabilities. I built it using Swamp as my infrastructure coding tool. I decided to try Swamp because it’s written from the ground up to be used by AI coding agents. I had Claude build the new instance infrastructure by copying the relevant Swamp models and workflows from the original repository. Starting with a specification or export of the existing AWS resource structure would have needed much more time to work out a new implementation.

Code for Confidence

One of the reasons I’ve always liked using code to build infrastructure is that it gives me confidence when making a change, which helps me to make changes faster and more often.

I can inspect and run checks on the code before deploying it. I can deploy a disposable replica and run active tests to make sure it works the way I want. I can test the operational tolerances of my system by running stress tests, failure scenarios, and attacks on a replica. And if I still end up hosing my production instance, I can quickly rebuild it, presuming I’ve also automated data and configuration persistence.

The confidence I get from using code to manage my infrastructure makes it less scary to change it. So I can knock out fixes, improvements, updates, and new capabilities whenever I need to, rather than leaving them on a backlog to get around to. For years this has served me well working with fast-paced human software delivery teams. Agentic fire-hoses of software and infrastructure code make confidence in the reliability of the workflow even more important.

Infrastructure code by itself isn’t enough to make agentic infrastructure as reliable as I need it to be. I’ll go into more details of creating an agentic harness for infrastructure in a separate post, building on what my colleague Birgitta Böckeler has been writing about harness engineering.

Code for Composability

The benefit of using infrastructure code with agents that I’m particularly excited about is being able to reuse infrastructure implementations.

The Infrastructure as Code ecosystem includes many attempts at shareable libraries, but we’ve never achieved anything close to what operating system and programming language packaging systems have. Maybe infrastructure code is too brittle, or organizations’ requirements are too idiosyncratic. But unlike static libraries, agents can adapt code to specific needs and context. As Mitchell Hashimoto and Adam Jacob have pointed out, agents are particularly good at assembling pre-existing components.

If a coding agent can adapt working, pre-built code to the needs of a specific workload in a specific organization’s context, then we may have the basis for a healthy ecosystem of infrastructure components. Even as models get better at writing infrastructure code that works on the first try, having pre-built components will be faster and more consistent. Off-the-shelf infrastructure components, combined with a robust delivery harness that enforces compliance and operational standards, can empower developers to build and deploy production-grade systems, something I previously described as the return of NoOps.

Photo by Georg Eiermann

Reactions To My Previous Article

I wrote recently that the next generation of infrastructure tools is for developers, and that agentic software engineering may help make this transition real. I deliberately used the term “NoOps” to echo an early DevOps-era argument: operations work doesn’t disappear, but the role changes.

Back in 2012 “NoOps” triggered backlash it sounded like there would be no need for ops folks when infrastructure was automated. But the point was different. We needed to shift from manually handling every release to building and managing the systems that handle releases.

At the time, that shift showed up through Continuous Delivery and related operational practices that later evolved into what we now call platform engineering.

I’m using “NoOps” in that same sense now. Not “ops disappears,” but “ops changes shape.” I expected a reaction. Ops folks are proud of our craft, and rightly so. Reliable operations is hard, and not something developers can do casually with a magical stochastic parrot.

Some response has dismissed this as “AI = vibe coding slop,” but that misses the point. I’ve been making this argument since long before LLMs: many ops teams still hand-craft infrastructure, even when they’re using automation tools. They are still a manual step for routine changes.

Agentic workflows create a chance to push self-service further. Not by encouraging YOLO changes, but by having experts build the platforms, guardrails, and workflows developers can safely use.

I think software delivery workflows will look very different within a few years, especially in higher-performing teams. Infrastructure professionals will still be essential. But our value will be in designing the loop, not being the loop.

New Article: Humans and Agents in Software Engineering Loops

That leads to the question I wanted to explore next: where do humans fit in an agentic delivery workflow?

I wrote Humans and Agents in Software Engineering Loops to work through that.

The key issue is how we govern how agents build a system. Do we give no guidance (“vibe coding”)? Do humans manually inspect every line (“humans in the loop”)? Or do we build systems that shape both outcomes and behavior (“humans on the loop”)?

For anyone who lived through the rise of DevOps, IaC, and Continuous Delivery, this should feel familiar. We stopped patching release artifacts by hand and improved the delivery system itself — tests, checks, and automation — so the next change would be safer by default.

Agentic software engineering looks familiar to me. We’ll build and run systems that produce, deliver, and operate software, including infrastructure code.

Some people argue this won’t work because LLMs are unreliable. But human engineers aren’t deterministic either. We’ve always managed that with a mix of deterministic and non-deterministic elements. That doesn’t go away.

Tools Worth Watching

Infrastructure as Code vendors are starting to adapt for agentic workflows. They’re making their tools easier for agents to use safely and repeatedly in engineering loops.

A few examples worth paying attention to:

HashiCorp Agent Skills:

Skills for generating Terraform code, refactoring modules, working with stacks, and writing tests (plus Packer support). A natural fit for teams happy in HashiCorp’s ecosystem. The open question is how well this fits teams using alternative packaging and deployment tools with Terraform.

Pulumi skills: Skills focused on migration (Terraform/CDK/CloudFormation/ARM) and Pulumi authoring practices. Useful if you’re considering a move to Pulumi and want agent support during both migration and steady-state development.

Swamp: An agent-oriented CLI and workflow model built for AI-native infrastructure automation from the start. Strong candidate for teams designing end-to-end agentic workflows rather than bolting agents onto existing processes.

Formae: A newer IaC platform with strong emphasis on drift/convergence and explicit agent integration via MCP + skills. Particularly interesting for teams dealing with messy, mixed estates and out-of-band changes.

For most teams, this won’t be a single-tool decision. Terraform and Pulumi skills are pragmatic options for teams with an existing infrastructure codebase. Swamp and Formae are more opinionated, agent-first operating models.

DevOps has taken us a long way. Fifteen years ago a 6-month cycle for releasing software was considered pretty good. These days it’s considered pretty poor. But infrastructure is still a bottleneck for most teams releasing software. In most engineering organizations beyond a fairly small size, developers rely on another team to manage infrastructure for them.

Tools like Pulumi and CDK aimed to solve this by allowing developers to write infrastructure code in a familiar programming language, so they wouldn’t need to rely on specialists to do it for them. But it turns out that YAML and HCL aren’t the real obstacle. Knowing how to wire low-level infrastructure resources like network routes, security groups, gateways, and SSL certificates together to do something useful takes time and expertise.

Developers who gain this expertise and write the infrastructure code for non-trivial systems turn into infrastructure specialists. Some are happy to take that journey, many would prefer to focus on the software.

Platform engineering, Developer Experience, and a handful of other fairly recent movements aim to empower developers by making the software delivery fully self-service. But often, these names are applied to teams which still work by hand-coding infrastructure. Developer workflows for getting code into production still involve waiting for someone else to make changes to networking, databases, or compute clusters. I’ve previously described this antipattern as infrastructure management teams.

Better tools for infrastructure experts only reinforce the wall.

Then there’s an emerging genre of tools that aim to solve the problems with Infrastructure as Code. Their creators believe that the biggest obstacle that needs to be removed is the clunkiness of managing code files in git and coping with the drift between code and deployed infrastructure. I have written about potential successors to Infrastructure as Code, including Infrastructure as Model tools like System Initiative.

System Initiative spent the last 6 years building their contender for the future of infrastructure automation. Rather than code, it represented infrastructure resources as a graph that could be dynamically manipulated by whatever extensions or integrations you chose to build.

Nobody wanted it.

One theory of why System Initiative struggled for adoption is that their approach was “too weird.” But I believe the real issue was there wasn’t a strong value proposition. System Initiative was another tool for infrastructure experts to manage infrastructure for developers. If you need to wait a week for the platform team to set up the new environment, it doesn’t matter that they don’t need to edit YAML files. The wall remains.

The next generation of infrastructure tools is for developers.

What we need is to empower developers to build and manage their own infrastructure. A developer should be able to specify what they need in terms that are relevant to the problem they’re solving. “I need this service to accept inbound HTTPS connections from the public Internet.” “I need an SQL database instance to store private user data, configured for security, availability, backups, and regulatory compliance.”

There was a brief moment back when DevOps was new where “NoOps” was a hot topic. The term has faded, partly because people read it as declaring ops and infra specialists to be obsolete. If developers are going to be whipping up databases and network connectivity, someone needs to provide the frameworks to ensure that what gets deployed complies with policies and governance.

The real point of NoOps was that ops folks should provide tools and systems that remove themselves completely from the developer’s workflow. And that point is more relevant than ever.

The System Initiative team has shut down their original product and is pivoting to something that seems aligned with that point. Their founder, Adam Jacob, recently gave a talk where he shared his epiphany about what comes next. His message is that the increase in software development velocity from AI is going to force ops to follow, like it or not.

Adam and his team gave me an early build of their new product, swamp. Swamp is a command-line tool designed to be understood and run by AI coding agents like Claude Code, so it fits naturally into the developer’s workflow.

I don’t know whether swamp itself will become the next big thing. Tools like Pulumi Neo are converging on a similar model. Especially for organizations using agentic development, bringing infrastructure management into the agent, embedded into the way developers build their software, has the potential to finally remove the dev/infra wall.

Photo by Navy Medicine on Unsplash

I’ve written a post, Interesting Tools, where I list tools that I think are interesting for implementing the ideas in the talk, and another, Infrastructure Platform Teams, where I discuss team topologies for self-service infrastructure.

In my previous article, I observed that many engineering leaders find infrastructure management to be a bottleneck for software delivery, even after adopting cloud and infrastructure automation. Development teams are blocked waiting for environments to be built, updated, modified, extended, and fixed. There never seem to be enough environments, and yet cloud costs spiral. Messy and fragile environments are a bottleneck for developing and releasing changes. Infrastructure and platform teams are overstretched while technical debt piles up.

That article described some of the approaches tool vendors are taking to make it easier to work with infrastructure, moving past the limitations of the traditional Infrastructure as Code model. Using general purpose programming languages, embedding infrastructure code within application code, using dynamic models of infrastructure rather than static code, and using Generative AI all have the potential to make infrastructure more accessible to its users.

But as powerful as some of these approaches are, I believe that, in themselves, they don’t address the root issues that stop organizations from getting as much value as they should from cloud infrastructure. Why not? Because they focus on improving the efficiency of specific infrastructure management tasks rather than improving the end-to-end value streams that rely on infrastructure.

The value that infrastructure provides to an organization is not in the specific tasks of provisioning or updating infrastructure. It’s in the outcomes that are achieved by the direct and indirect users of the infrastructure.

In this article I describe an approach for identifying what value organizations get from digital infrastructure and how to trace that to the specific infrastructure delivery capabilities. I discuss how analyzing value streams that depend on infrastructure capabilities, particularly software delivery value streams, can identify opportunities to remove bottlenecks. Then I touch on how different approaches for delivering infrastructure capabilities can help with those bottlenecks, such as self-service delivery.

This article lays the foundation for some follow-up content I’m planning that discusses how to implement infrastructure delivery capabilities that improve the end-to-end effectiveness of value streams that rely on infrastructure. The good news is, many of the tool vendors who are working on ways to improve the experience of working directly with infrastructure are also working on ways to improve delivery and outcomes. My goal with this series is to help with thinking about how to design, select, and implement solutions using a value-driven approach.

Does infrastructure have business value?

It’s worth repeating, the value of infrastructure management isn’t measured by how quickly infrastructure resources are deployed and changed; it’s measured by the outcomes delivered by those who use those resources. Digital infrastructure usually has multiple layers of stakeholders represented in a value chain with the organization’s customers at the end.

At that end of the value chain, where revenue is earned, the contribution of infrastructure is indirect and usually unclear. At the opposite end it can be hard to see exactly what value we get from each network route we configure and storage bucket we provision.

Unfortunately, infrastructure is most clearly visible to the business as cost and failure.

This gap makes it tempting to dismiss decisions about infrastructure work as irrelevant to business value. It’s common for business leaders to assume that infrastructure is an undifferentiated utility akin to plumbing. Even many infrastructure engineers don’t see the need to understand the workloads that run on their systems, much less their organization’s strategy.

But if infrastructure didn’t need to be tailored to the organization then we wouldn’t have a constant stream of new buzzwords and movements popping up focusing on doing just that. DevOps, SRE, platform engineering, developer experience - these all continue to get attention because there is a layer of capabilities that needs to be built on top of the low-level IaaS cloud in order for software teams to deliver value specific to their organization.

The hyperscale cloud vendors have done a pretty good job of finding the line where infrastructure really is a utility and putting it behind IaaS APIs. Assembling the undifferentiated resources the APIs provide into useful services for your team to use turns out to need custom work, and plenty of it. Hordes of vendors of all sizes promise turnkey solutions to make that work disappear, and yet it still hasn’t disappeared.

So if cloud and platform vendors can’t give us a generic, one-size-fits-all solution for making the cloud directly useful to our organization, then we need to roll up our sleeves. Our first step is to work out what our organization, specifically, needs from our infrastructure.

What value does infrastructure deliver for your organization?

A good place to start understanding how to get the most value from infrastructure automation is to look at the organization’s strategy, goals, and plans, and think about where infrastructure plays a role. Here are a few examples I’ve come across multiple times with clients:

The business grows and sustains revenue by delivering new digital products and services, and delivering new features, improvements, and fixes to existing products. Infrastructure plays a key role in making sure that software delivery teams have what they need to develop and deliver code into production quickly, reliably, and safely.

The company expands into markets by deploying existing products to new geographical regions and new customers. These different types of expansion often involve building and maintaining new production infrastructure.

The organization manages costs and improves operational effectiveness by consolidating digital services created by different teams, including acquired businesses. This is often a pure infrastructure play, such as working out how to reduce eight different Kubernetes implementations to a more reasonable number.

These activities can usually be mapped to revenues and costs, which makes it possible to attribute financial value to high-level infrastructure delivery capabilities. We can work out those high-level capabilities by looking at the value chain for a given business activity, analyzing the value streams that support that value chain, and then understanding which infrastructure delivery capabilities are most important for improving those value streams.

For example, an online travel business has a goal to increase revenues, which they can achieve by increasing the number of bookings. Product and UX research suggests bookings could be increased by adding a new AI feature for users searching for flights. The development team will implement the new feature in code, and by integrating with an external AI service.

Delivering the new AI flight search feature depends on several infrastructure delivery capabilities, including providing test environments, adding a vector database to all of the environments including production, and changing network configurations to connect with the AI service.

This example shows how to trace the connection between providing an infrastructure delivery capability, like providing an environment to deploy and test a software build, to business goals. The impact of a given infrastructure capability can be assessed by understanding how it contributes to the effectiveness of the value streams it supports. So our next step is to examine one of those value streams.

Measuring software delivery effectiveness

A software delivery value stream traces the process for getting a change into production. The process of developing a new feature, like the travel company’s AI flight search, usually involves delivering a stream of changes into production over the course of weeks or even months. Early changes may be turned off in production, available only to testers and early adopters as development progresses. Once the feature is made available to end users, further changes iterate and evolve the feature based on how it performs commercially and user feedback.

Different types of software changes may be delivered using different value streams (or perhaps different branches of a complex stream). For the purposes of understanding infrastructure effectiveness, there are a number of types of streams that involve infrastructure in different ways. Some examples:

• A software team develops and delivers a code change to production without requiring any changes to existing infrastructure
• A software team develops and delivers a code change to production that requires a change to existing infrastructure
• A product team creates a new software product or service
• A business team introduces existing services to a new customer, requiring adding at least some dedicated-tenancy production infrastructure

The effectiveness of a software delivery value stream can be measured using frameworks like the DORA metrics and the DX platform. The key metrics, per DORA, are time to deliver a change, rate of delivery, production failure rates, and recovery time from production failures.

Once you have an idea of how effective your software delivery value streams are, value stream mapping exercises help to discover ways to improve it. These exercises involve gathering data from various systems and stakeholders to understand the steps involved in delivering a software change. Some key data points to look at within a value stream include:

Cycle times across different stages and subsets of the overall value stream. Where is the most time spent?

Waiting times between steps in the value stream. This leads to investigating the causes of waiting time, which can be a large proportion of cycle times and the overall lead time, and are often related to infrastructure.

Handovers between teams across stages, another common bottleneck. A common handover is when a separate person or team handles a task such as preparing a test environment.

Failure demand, where the value stream loops and extra steps are needed to investigate and correct problems.

Capacity, where there are limits to how many changes can be in progress at a time. These constraints may be because of expertise needed, or system limitations like sharing a test environment across teams.

Effort and expertise needed to execute a step in the value stream. Limits on these can affect capacity and waiting time for handovers. Constraints on expert effort are made worse by failure demand, which soaks up expert time to investigate and fix.

It’s worth noting that, in addition to software delivery, infrastructure can play a role in other value streams including production support and troubleshooting. Additional value streams can be mapped for activities for ensuring operational quality and managing risk, such as patching and upgrading services. Those infrastructure value streams can in turn impact software delivery value streams, for example by taking environments offline for maintenance.

The results of value stream mapping are useful for identifying where to focus efforts to improve the results of the end-to-end delivery process. It’s too common to automate an obvious part of the process that doesn’t make much difference to the big picture. For example, I’ve seen teams proud of having replaced a four-hour manual provisioning process with automation that takes ten minutes, only to discover that the two-week delivery cycle (including design, review, and approval stages) meant their users didn’t notice.

On the other hand, it’s important to understand that the impact of automating a step in the value stream isn’t only about completing that task more quickly. It could have added benefits like removing handovers and cutting waiting times.

Analyzing infrastructure capabilities

Platform and infrastructure improvement initiatives can improve value streams by optimizing underpinning infrastructure capabilities. Some of the software delivery value streams listed earlier involve making changes to infrastructure, like adding a new vector database or opening network connections to an external service’s API. But even with a basic value stream for delivering a software code change that doesn’t need any changes to infrastructure, infrastructure capabilities are required to get the change from the developer’s workspace to production.

For example, releasing a software change normally involves deploying a build with the change to one or more testing environments. The capability of “providing an environment for testing” can impact the effectiveness of the software delivery value stream in various ways, including:

Wait time for an environment to be available. This happens when there are a limited number of environments for a particular type of testing.

Wait time for an environment to be prepared. Configuration and data may need to be reset to a clean starting state after an environment was used for testing another build, especially if the previous build failed.

A failure when deploying or running tests due to an issue with the environment’s configuration or data. This leads to time spent resolving the issue and re-running steps in the value stream. Often resolving failures means asking people from other teams for help.

Issues with wait time and failure can impact the effectiveness of the end-to-end value stream, leading to longer lead times and fewer releases. If problems with providing or preparing environments occur when deploying the software change to production, then deployment failure rate and recovery time metrics will suffer.

There are a number of metrics that can be useful for measuring the capability of providing an environment for testing:

• How long does it take before the environment is ready to use? What is the turnaround time between deployments? What is the wait time for deployment?
• What are the capacity limits - how many of a given type of environment can be made available at a time? What is the demand?
• How much effort does it take to prepare an environment for deployment, and clean up afterwards? What skills are required? For example, it could need an infrastructure engineer, a DBA, and a QA to prepare an environment and its data.
• What is the frequency of failures in an environment, whether it’s a deployment failure or a test failure caused by an issue with the infrastructure or data?
• What is the utilization of infrastructure used for testing? Environments that are often idle between occasional deployments rack up wasted cloud spend. Even in a data center it’s a waste to have too much unused hardware capacity.

Improving an infrastructure capability’s effectiveness

An infrastructure capability can be implemented and delivered in different ways, each with varying impacts on the end to end value stream. Once you’ve pinpointed how a capability is creating bottlenecks, you should consider changing the way the capability is provided.

How is the capability implemented?

Static environments with manual preparation represent the least effective option, although the most common. A centralized infrastructure team manually configures and resets environments when needed. This approach introduces handovers between teams, which create waiting times before the deployment step can begin, and limits capacity to the number of pre-built environments. As we saw in the value stream analysis, these bottlenecks can dramatically impact end-to-end delivery effectiveness.

Static environments with automated preparation improves on this by removing some manual effort and reducing preparation time. Define the starting state, including data, dependencies, integration points, configuration settings, etc., and have a way to automatically reset the environment to that state. This reduces waiting times for environment preparation and can eliminate some handovers. However, teams may still wait for an environment to be available. The mitigation to make sure there is enough capacity to prevent teams from waiting is to keep enough environments provisioned to meet peak demand, which can lead to low utilization (i.e. wasted hardware or cloud spend). Also, automated environment preparation needs to be highly reliable. Failures not only lead to delays, but also the need to ask someone else to help, adding to the effort absorbed. Destroying and rebuilding the environment each time tends to be a more reliable approach than cleaning up a running environment, but also slower.

Ephemeral environments - build and destroy on demand - is the most effective approach. This builds on automated preparation but goes further by creating environments only when needed and destroying them when done. You get better utilization and cost management if you destroy or strip down environments when they aren’t in use. If you have the ability to add capacity on demand, whether by using cloud or a shared pool of data center resources, you can eliminate waiting times for environment availability. This approach can dramatically reduce cycle times and remove capacity constraints from the value stream.

Service delivery models

Beyond the technical implementation, the effectiveness of an infrastructure capability also depends on who provides it and how. Typical approaches are:

Centralized specialist teams handle all requests for the capability. This creates handovers and waiting times in the value stream, as software teams must request environments and wait for them to be provided.

Embedded specialists within software delivery teams (such as DevOps engineers) can reduce handovers and waiting times, though they may become bottlenecks themselves when multiple developers in a team need help. DevOps skills are also in high demand, so it’s often not feasible to hire people in every team for this.

Self-service capabilities that allow anyone authorized in the software team to provision what they need (for example, using a developer portal) can eliminate handovers entirely and remove waiting times from the value stream.

Fully automated provision, for example, integrated into deployment pipelines, represents the ultimate evolution, where environments are created and destroyed as part of the automated software delivery process without human intervention.

Each progression toward greater automation and self-service reduces the bottlenecks, handovers, and waiting times that impact software delivery value stream effectiveness.

Implementing better infrastructure delivery

The value of infrastructure management is in the outcomes of the work done by the people and teams who use the infrastructure. I recommend that engineering leaders who want to make infrastructure delivery more useful to the organization start by tracing the connections from business goals and value chains down to infrastructure management capabilities. This article has outlined an approach to do this to identify ways to use infrastructure automation to improve business outcomes.

I’ve used software delivery to illustrate this approach because it’s one of the easier parts of the value chain to trace the connection from business goals to infrastructure delivery.. Software is developed with specific business outcomes in mind, so it should be easy to understand the value in automating the process of providing test environments to make the end-to-end software delivery process faster and more reliable.

With other infrastructure capabilities, like ensuring systems are continuously patched and upgraded, and adding automated backups and failovers, the connections can be less obvious. But it’s entirely possible to make those connections using this approach, and it’s essential for avoiding accumulating invisible but potentially existential risk and inefficiency.

Understanding the value of improving infrastructure delivery is only the first step in making it happen. As I’ve suggested, business outcomes can often be improved by changing the way infrastructure capabilities are implemented. Automating tasks to provision and change infrastructure will only make the end-to-end processes the infrastructure supports better if they’re designed and implemented with an understanding of that process in mind.

I plan to expand on how to do that in later articles in this series. As a preview, there are three key elements of implementing infrastructure delivery to support effective value streams for infrastructure users. One is team structures and workflows. I’ll draw Team Topologies as well as continuing with value stream mapping to describe some different ways of providing infrastructure services. The second element is architecture, which is essentially about implementing infrastructure as composable components at two different levels, one for deployment (stacks) and one for consumption (compositions). My thinking here is heavily influenced by microservices, ports and adapters, and product thinking. The third element is delivery, which is the nuts and bolts of packaging, delivering, deploying, configuring, and integrating infrastructure components.

Although it sounds like I could describe each of these elements in a separate article, I think they’re too tightly connected to explain separately. My next article will probably be a fairly high-level description of these three elements and how they work together. Stay tuned!

Image by Deleece Cook

When my colleagues and I carry out software delivery effectiveness assessments, we often find time and effort wasted providing teams with the infrastructure they need. Value stream mapping shows teams losing time waiting for infrastructure work to be implemented, going back and forth to clarify needs and iterate as needs change during development, and troubleshooting and improving systems once they’ve gone live.

This is the first of several posts I’m working on to explore the question of how to improve software delivery effectiveness by empowering software delivery teams to provide infrastructure for themselves.

Infrastructure as Code is the dominant paradigm for automating infrastructure management today. But it creates plenty of pain points, especially for software teams that would like to manage their own infrastructure. A number of potential solutions that have emerged over the past few years, and new ones emerging now, focus on removing these pain points by improving the user experience of working with dynamic infrastructure.

Can software delivery be made more effective by giving software developers a friendlier, more effective interface for defining and provisioning infrastructure for themselves? Three potential approaches are offering more developer-friendly languages for writing infrastructure code, giving them GenAI chatbots or agents to provision code for them, or replacing infrastructure code with dynamic and extensible models or graphs.

Developer friendly languages

Traditional Infrastructure as Code tools provide special-purpose declarative DSLs (Domain Specific Languages), like Terraform, Ansible, and CloudFormation as an interface for defining the infrastructure you want. The tool generates a model of desired state, compares it with the reality, and then executes the changes to make the reality match the intention.

Many software developers I know find building infrastructure with these tools painful. They believe that alternative tools like Pulumi and AWS CDK that let them use general-purpose, imperative languages will remove the barriers for developers to manage infrastructure without relying on a separate team of infrastructure specialists.

Dynamic programming languages are more powerful than declarative languages for some areas of infrastructure automation, especially building component libraries. But they don’t really make low-level infrastructure coding more accessible to software developers. If you don’t have a solid understanding of networking, it isn’t any easier to wire together subnets, routing tables, firewall rules, gateways, and load balancers into safe and useful application connectivity in Python than it is with Terraform’s HCL. And Typescript doesn’t save you from needing to understand the tradeoffs of the many configurations of S3 bucket configuration for whichever one of the dozens of different purposes you might be using one for.

Some infrastructure experts are more comfortable using a general purpose language. And imperative languages are more appropriate than declarative languages for building component libraries. But they don’t reduce the effort or expertise needed to manage infrastructure for software delivery teams, so they don’t resolve the problem of making an organization’s software delivery process more effective.

Infrastructure from (application) Code

Some tools and platforms take the idea of developers writing infrastructure code a step further by making it possible to embed infrastructure code with application code. Platforms like Winglang and Darklang introduce new languages for coding applications and infrastructure together. Other solutions, like Ampt, Nitric, and Shuttle, introduce SDKs or annotation support for coding or defining infrastructure required in existing software languages like Golang and Rust.

The idea behind this approach is that a developer can declare the infrastructure required at the point where it is used. The code that reads and writes entries in a database specifies how to provision and configure the database. Code that writes messages to a queue creates the queue. An application declares the details for handling inbound network requests along with the business logic for handling those requests.

Infrastructure from Code does more than empower developers, it also ensures smooth alignment between building infrastructure and using it. Deployments often fail because of changes to one side or the other of infrastructure and application. Many of these failures can be avoided if the compiler or IDE immediately catches mismatches, like writes to an S3 bucket that hasn’t been created.

A major limitation of Infrastructure from Code is that it doesn’t address the concerns of infrastructure that isn’t directly used by a single application. Shared compute and networking resources; and platform services like monitoring and identity management need to be defined and managed separately. Environment management is also either hand-waved or tightly bound into application code. Enabling developers to manage application infrastructure is powerful, but most non-trivial systems have a broader scope.

Infrastructure as Model

There is an emerging crop of post-code infrastructure automation systems being developed by various startups like System Initiative and ConfigHub. These aim to remove the fiddliness of managing code in repositories. They also close the gap between live infrastructure and the representation that engineers use to define the changes they would like to make.

Infrastructure as Code uses code, whether declarative or imperative, to generate a model of the desired state, compares it with a model of the current state, and then makes changes to the existing infrastructure to converge it with the desired state. There are four states - code, desired state, model of current state, and actual state - can get out of sync in various ways that are at best confusing, and at worst lead to a broken state that is painful to correct.

Infrastructure as Model tools center on the model of current and desired state. They provide interfaces for defining the desired state. Demos focus on drag-and-drop GUI interfaces that demonstrate how much easier it is to compare desired and current state than is possible with traditional infrastructure code. But the real power with these tools is the programmable extensibility of the graph of desired and live state, and the events involved in building and converging them.

I confess that I don’t know exactly how Infrastructure as Model will end up being used in practice, but the possibilities are exciting. The implementations I’ve seen so far, as with using general purpose languages for Infrastructure as Code, tend to focus on improving the experience for infrastructure experts. I believe this is because the tools are still in early stages of development and are building the foundational functionality of assembling low-level cloud infrastructure into useful solutions. As Infrastructure as Model implementations evolve, they will hopefully address the higher-level concerns needed to make software delivery processes more effective.

GenAI as infrastructure assistant

Most of the approaches I’ve discussed so far give users a more convenient interface for working with low-level infrastructure resources, but they still require expertise and effort to select, configure, and deploy those resources in the most useful way. LLMs have the potential to support developers by providing this expertise.

Developers can use coding assistants like Github Copilot, Amazon CodeWhisperer, Cursor, or Pulumi AI to accelerate their use of old-school Infrastructure as Code. Infrastructure as Model platforms are already incorporating LLM support, and various tools like Firefly Copilot can use GenAI as a natural-language interface for building and managing infrastructure.

Imagine a developer describing their application’s networking requirements so an AI agent can provision and configure the necessary infrastructure resources. They can tell GenAI what they need to use an S3 bucket for, and it selects configuration options that seem appropriate.

If done poorly, tools like this will be AI-assisted ClickOps, leaving organizations with an unmaintainable mess of snowflake infrastructure. There are several challenges to getting reliable, useful results from AI-assisted infrastructure management.

First, the AI needs to understand what the user needs, which means the user needs to be able to explain what they need accurately, clearly, and comprehensively. GenAI will guess to fill in any gaps or ambiguity. As anyone who has worked with GenAI very much knows, it’s important to understand the domain very well, and to know how to craft and iterate on prompts, to avoid making a mess.

A GenAI-based infrastructure management solution could provide software teams with well-designed prompts and guardrails could help them to manage their own infrastructure. The work to provide those tools then looks very similar to providing infrastructure automation with a code-based toolchain, including being done by people with deep expertise in infrastructure. Call it “infrastructure as prompts”.

There are interesting possibilities for using LLMs for managing infrastructure. However, using them to improve the effectiveness of software delivery, as with the other approaches I’ve discussed in this article, requires going beyond optimizing the low-level work of defining infrastructure, whether using code or another interface.

Valuable progress, but not addressing the bigger issues

The solutions here, including developer-friendly infrastructure coding languages, interfaces that work directly with infrastructure state models, or adding LLM-based AI assistance, all target the low-level tasks of defining and deploying infrastructure resources. They can make those tasks easier for infrastructure experts or for developers.

But none of them are convincing as a solution that will lead to engineering leaders declaring that managing environments and platforms on the cloud is no longer a bottleneck for software delivery.

Yes, Infrastructure as Code is an awkward mechanism for assembling low-level resources into meaningfully useful services and environments for delivering and running software. But replacing code with a more convenient interface for infrastructure experts to use doesn’t address the largest friction points. The biggest gap is between low-level infrastructure details and the solutions needed by software teams.

I’m working on followup articles to explore that gap and how to address it. The next article describes an approach for identifying what value organizations get from digital infrastructure and how to trace that to the specific infrastructure delivery capabilities.

Image by Brett Jordan

This survey was a starting point for me. I want to explore ways to measure the effectiveness of the practices and patterns I describe in the Infrastructure as Code book. I’ve added some material on understanding the business value of infrastructure management capabilities to the third edition. These feel like a reasonable foundation to build on, to be able to connect approaches to managing infrastructure with effective business outcomes.

This survey was never going to do that, since I don’t have the skills and time to invest in a comprehensive, rigorous survey. This is no DORA report. It’s more of a finger in the air, almost a way of collaboratively brainstorming with a few willing participants.

About thirty people completed the survey, which wouldn’t be much if I was hoping to get conclusive answers about what practices work best, but was plenty to get me thinking.

The first insight I came away with is about framing “infrastructure effectiveness” around capabilities infrastructure provides for the business. There are four high-level capabilities that I tried to explore in the survey:

• Adding production infrastructure to onboard customers
• Adding production hosting in new regions
• Delivering infrastructure changes to existing infrastructure
• Supporting the delivery of software changes

I asked several specific questions for some of these capabilities. One asked about the current effectiveness of the respondent’s organization at that capability. How long does it take to do the thing? Another asked about their need for that capability. How often do you need to do it? A third asked how much impact it would make to improve the effectiveness.

I didn’t ask those questions for all of the capabilities, so one of my takeaways is that I need to think a bit harder about how to express those questions, especially in the software change delivery area.

In this post I’ll focus on the first capability. I’ll write more later on other insights and inspirations I’ve taken away from the survey.

Adding infrastructure to onboard customers is fairly common

This capability mainly applies to services that are at least partly single-tenancy. I would expect that most consumer-facing services will be designed for multi-tenancy, because otherwise the time and expense of adding new infrastructure will be a massive drag on growth. I know a few software companies that moved to SaaS models have struggled with this.

But 86% of my respondents who answered this question do need to add new infrastructure to onboard new customers. 28 respondents is not many, so this is as likely to reflect the kinds of people who follow me on my mailing list or social media, and who were interested enough to complete my survey for whatever reason. Many (maybe most) of my clients are in this situation.

There’s a roughly even split between those who need to do this at least once a month and those for whom it’s less frequent. I’d guess there’s a correlation to deal size: some companies have a few, large clients, while others have a larger number of smaller clients.

Differences between frequent and infrequent deployers of new production infrastructure

I would expect the group that needs to deploy new infrastructure more often to be faster at doing it. But the results on that question had a weak positive correlation (0.27), which suggests that while it’s generally true, there’s a gap. This is backed up by their answers to the question about how much of a difference it would make to the organization to reduce the time and effort for deploying new infrastructure. Most of those that need to do it more often feel a need to improve. That might seem obvious, but if they were good enough at it already, they probably wouldn’t feel as much need to work on it.

There’s a stronger correlation with those organizations that do deploy customer infrastructure more often and effectiveness at the other capabilities, like adding new regions, updating existing infrastructure, and delivering software changes. That makes sense - they have more incentive to get their infrastructure automation working well.

Another finding is that those teams that deploy customer infrastructure more often will be stricter about not forking or copying infrastructure code for each customer. The results (again, for my small sample) show a .8 correlation between those who deploy at least once a month and using configurable infrastructure code. There is a .38 correlation between those who deploy less often and maintaining separate copies of infrastructure code for each environment.

One final observation is that there doesn’t seem to ant noticeable correlation between tech and tool choices and deployment frequency.

Next

I’ll have a poke into what I learned about software delivery support effectiveness. As I mentioned, this is an area that I need to think harder about for framing and assessing in the future.

The reality is that speed and quality are mutually reinforcing. The ability to make changes quickly is essential to creating and maintaining a high level of quality. And a high level of quality is the only way to sustain the ability to make changes quickly.

Rather than a two-dimensional line, the best way to show the relationship between quality and speed in a system is with a matrix of quadrants.

The upper-left quadrant is where teams try to prioritize quality by sacrificing speed. They use heavyweight change management processes, believing that involving more people in making a change, with more rigorous activities to inspect, justify, and sign off on each change, will “raise the bar”. What happens in practice is that having so much friction for making a change discourages people from making smaller fixes and improvements. The list of “known issues” grows, technical debt builds up, and people become accustomed to the system’s bugginess and awkward usability.

So sacrificing speed to achieve quality achieves neither.

In the lower-right corner, teams try to “move fast” (and maybe “break things”). The most important thing is getting features into users’ hands, testing is seen as a luxury. Perfection is the enemy of building a business before the money runs out. The problem with this approach is that shoddy code is hard to change. The team reaches a point where it’s hard to make a simple change because the codebase is a tangled mess far more quickly than they had assumed.

So sacrificing quality to achieve speed achieves neither. This conclusion is backed by research (see Accelerate by Dr. Forsgren et al, plus multiple DORA reports).

In practice, sacrificing either quality or speed leads to a fragile, messy codebase that is difficult to change, which is the lower-left quadrant. Nobody aims for the lower-left quadrant, but the vast majority of teams end up there.

The upper-right quadrant is where teams prioritize both speed and quality. This was the idea behind agile software development, at least before agile became gentrified by those who believe in process over people.

The upper-right quadrant is not about finding a balance between speed and quality. It’s about using speed to achieve quality, and quality to achieve speed.

It’s about “Zero Defects” [Kent Beck] (which is not about never having any bugs, but making sure you fix them immediately when you find them). It’s about building the Best Simple System For Now [Daniel Terhorst-North].

A key to breaking the “tradeoff” mindset is in how you define quality. Quality is not features. It’s not extensibility. Quality is doing what needs to be done now, and doing it correctly. Quality code is easy to change. A good process drives quality by minimizing the friction of making a change safely.

Simple design and simple implementation make quality easy, because it means the code is easy to understand, change, test, and fix. Making small, frequent, incremental changes in a continuous flow reduces the friction that comes with heavier process (branching and merging). Pairing keeps people focused on clean, simple, and correct implementation. Really, we could do worse than to refresh ourselves on old school Extreme Programming. Daniel Terhorst-North’s CUPID approach for “joyful coding” is all about making sure your code is easy to work with.

Yeah, this is a site about Infrastructure as Code. But as with so much, the tenets of good software apply across the full system stack.