HashiCorp and the Rise of Platform Engineering

HashiCorp was explicit in its approach to the enterprise at HashiConf Global this year: DevOps is not really a thing.

What people call DevOps does have a certain appeal: two different specialist groups should communicate with each other better. This is a noble goal.

It is not, however, unique to tech. Many other areas of life have specialists that need to communicate with each other, and experience some of the friction and complexity of doing so. For example:

• Architects and general contractors.
• General contractors, electricians, and plumbers.
• Surgeons and anesthetists.

These are all fairly closely related jobs yet they are specialties in their own right. They each have their own jargon, shorthand, and life experience. Plumbers are more intimately aware of how water works than electricians, though each can come to appreciate the others' skills and experience.

DevOps has long seemed, to me, to be like trying to create an integrated ArchiBuilder, Contractician, or AnestheoSurgeon role when what we need is better coordination between different roles.

And it seems that a growing number of vendors, including HashiCorp, agree with me.

Divide and conquer

It makes sense to me to have specialist developers for different kinds of software development. There are frontend developers and backend developers because these two kids of systems are different in important ways. It is possible to do both, but the increasing complexity of modern systems means keeping across all of it at a high competency level is increasingly difficult.

We don't expect anesthetists to be equally capable knee surgeons, so why do we expect operations specialists to be equally good at writing application code? Focusing on one thing at a time is what allows you to get really good at it.

Which isn't to say there isn't a role for generalists as well!

Having experience of more than one specialty, and being able to bridge between them, translating between the different specialist jargon and metaphor, is also valuable. It's more of a case of AND than OR. Have teams of specialists, yes, but also have a few people who float across the silos to glue them together into a more cohesive whole.

Rise of the platform developer

Platform development is about writing infrastructure software, as distinct from application software. Platform software is things like Kubernetes, and CI/CD software, and monitoring software. It's the stuff that keeps the system running and healthy and available for applications developers to use to build their apps.

Databases are infrastructure software. So are operating systems. You don't need to write your own, but you do need to glue them together somehow, and that's not really the role of application developers who want to use infrastructure software to get things done, but don't want to develop or maintain it.

Platform developer is a useful role description for the kind of people who write infrastructure software. They use standard developer tools to do so, but the software they build is a platform that others build upon. Their customers are other, different kinds of developers. They need a platform they can simply use, rather than having to build it themselves.

As enterprise IT becomes more cloudlike it is unsurprising that it should start to deal more in products and platforms than projects. Platform developers create and maintain long-lived products and services that other parts of the enterprise consume.

Ops is important

It also helps to separate out development of platforms from operations of those platforms. It's operators who need to write scripts to automate their tasks and make their lives easier. Learning some software development methods, like version control, can help them to do a better job, but they probably don't need to know the full formal definition of a lambda calculus to get things done. They don't need to be able to reverse a binary tree.

Platform developers need specialist tools to help them operate the infrastructure that supports applications. Modern IT platforms are built up from many different components. While this has always been true, previously there were more proprietary components that would be installed and configured by the vendor. In some cases, such as Fibre Channel zoning and array configuration, operational tasks would also be performed by the vendor.

This remains true for outsourced platforms, such as public cloud, or Red Hat's OpenShift, or various as-a-Service infrastructure offerings. But the varied and flexible nature of modern IT systems means that platform teams frequently need additional glue tools.

Systems administrators have long glued systems together with scripts, and built their own custom tools for one-off or repetitive tasks. They are increasingly adopting the processes and systems of application developers because it makes sense to do so. Version control is useful. So, too, are test suites, which leads to continuous integration. From there, CI/CD pipelines are a short step, and now platform teams look a lot like application teams of some years ago.

But the work of platform development remains different from application development, and so the goals and techniques will also continue to be somewhat different. Both groups can learn much from each other, but there is no one-size-fit-all approach, so trying to lump everything together as a single DevOps concept tends to break down.

The future of platforms

Better, then, to acknowledge the similarities and the differences, and allow platform developers to adopt their own tools and techniques where it makes sense. By all means, steal and copy the techniques of application developers, but expecting platform developers to work in exactly the same way would be folly.

This is why I value HashiCorp's distinction between application teams and platform teams, between application developers and platform developers. While both are involved in software development, their goals are different. Allowing platform developers to claim the label of developer helps them to seek out software development tools and techniques while also allowing them to stand apart from application developers.

Ultimately these roles are complementary, rather than in conflict. We need both, and we need to seek out ways to help them work together, but also separately.

I suspect that we will see more of this acknowledgement as the DevOps label reduces in importance and marketers seek some other catchall term they can claim as a new category. It is an unfortunate but very real feature of the way the tech industry works, so all we can do is acknowledge this and adapt accordingly.

Hopefully we will progress towards more robust platforms with a clearer understanding of what works well, and what to avoid, rather than flailing about trying to jam square pegs into round holes.

We need tools that fit our needs more than we need larger hammers.

I travelled to HashiConf Global 2022 as a guest of HashiCorp. See my full disclosure post here.