In January I set up an “Infrastructure Effectiveness Survey” and asked people to fill it in, promising that I’d share what I learned from it later on. Later on has arrived!
This survey was a starting point for me. I want to explore ways to measure the effectiveness of the practices and patterns I describe in the Infrastructure as Code book. I’ve added some material on understanding the business value of infrastructure management capabilities to the third edition. These feel like a reasonable foundation to build on, to be able to connect approaches to managing infrastructure with effective business outcomes.
This survey was never going to do that, since I don’t have the skills and time to invest in a comprehensive, rigorous survey. This is no DORA report. It’s more of a finger in the air, almost a way of collaboratively brainstorming with a few willing participants.
About thirty people completed the survey, which wouldn’t be much if I was hoping to get conclusive answers about what practices work best, but was plenty to get me thinking.
The first insight I came away with is about framing “infrastructure effectiveness” around capabilities infrastructure provides for the business. There are four high-level capabilities that I tried to explore in the survey:
- Adding production infrastructure to onboard customers
- Adding production hosting in new regions
- Delivering infrastructure changes to existing infrastructure
- Supporting the delivery of software changes
I asked several specific questions for some of these capabilities. One asked about the current effectiveness of the respondent’s organization at that capability. How long does it take to do the thing? Another asked about their need for that capability. How often do you need to do it? A third asked how much impact it would make to improve the effectiveness.
I didn’t ask those questions for all of the capabilities, so one of my takeaways is that I need to think a bit harder about how to express those questions, especially in the software change delivery area.
In this post I’ll focus on the first capability. I’ll write more later on other insights and inspirations I’ve taken away from the survey.
Adding infrastructure to onboard customers is fairly common
This capability mainly applies to services that are at least partly single-tenancy. I would expect that most consumer-facing services will be designed for multi-tenancy, because otherwise the time and expense of adding new infrastructure will be a massive drag on growth. I know a few software companies that moved to SaaS models have struggled with this.
But 86% of my respondents who answered this question do need to add new infrastructure to onboard new customers. 28 respondents is not many, so this is as likely to reflect the kinds of people who follow me on my mailing list or social media, and who were interested enough to complete my survey for whatever reason. Many (maybe most) of my clients are in this situation.
| How Often | How Many | Percentage |
|---|---|---|
| Multiple times a week | 5 | 18% |
| Multiple times a month | 6 | 21% |
| Multiple times a year | 10 | 36% |
| Less than once a year | 3 | 11% |
| None | 4 | 14% |
There’s a roughly even split between those who need to do this at least once a month and those for whom it’s less frequent. I’d guess there’s a correlation to deal size: some companies have a few, large clients, while others have a larger number of smaller clients.
Differences between frequent and infrequent deployers of new production infrastructure
I would expect the group that needs to deploy new infrastructure more often to be faster at doing it. But the results on that question had a weak positive correlation (0.27), which suggests that while it’s generally true, there’s a gap. This is backed up by their answers to the question about how much of a difference it would make to the organization to reduce the time and effort for deploying new infrastructure. Most of those that need to do it more often feel a need to improve. That might seem obvious, but if they were good enough at it already, they probably wouldn’t feel as much need to work on it.
There’s a stronger correlation with those organizations that do deploy customer infrastructure more often and effectiveness at the other capabilities, like adding new regions, updating existing infrastructure, and delivering software changes. That makes sense - they have more incentive to get their infrastructure automation working well.
Another finding is that those teams that deploy customer infrastructure more often will be stricter about not forking or copying infrastructure code for each customer. The results (again, for my small sample) show a .8 correlation between those who deploy at least once a month and using configurable infrastructure code. There is a .38 correlation between those who deploy less often and maintaining separate copies of infrastructure code for each environment.
One final observation is that there doesn’t seem to ant noticeable correlation between tech and tool choices and deployment frequency.
Next
I’ll have a poke into what I learned about software delivery support effectiveness. As I mentioned, this is an area that I need to think harder about for framing and assessing in the future.
