Andrew Holway on DevOps: Held together with spit and hope

 
Later this month, Andrew Holway of Otter Networks will be speaking at Leaders in Tech | Berlin. His talk “Devops is Fscked” (Meetup), will look at how fully automated platforms such as Kubernetes are making it possible for software developers to utilize cloud services directly and eliminating the need for siloed infrastructure management.
I met up with Andrew to learn about his journey from sailor to supercomputer designer to freelance DevOps engineer to CTO — and now to teaching companies to eliminate their dependency on DevOps.
 
You started off in a totally different kind of engineering, on boats, doing rigging and some electronics, but ended up working with supercomputers. How did you start working with DevOps?
I was basically like a fixer, a problem solver. I worked in supercomputing for 5 years where I developed large scale supercomputer platforms for processing data from high-profile academic science projects such as the Large Hadron Collider. I came to Berlin and I started working in commercial sector stuff, and then in Amazon Web Services. I noticed that a lot of what I would build for my clients in AWS was basically only supportable by me. I didn’t realize it at the time, but when I look back on it… the nature of what I do is very complex and there’s generally only one of me. When I leave a situation, there tends to be something that needs to be taken care of. And often that thing cannot be taken care of because only I can take care of it.
This has been a common thread throughout every DevOps job that I’ve done. I’ve put together some solution, but when you’re building it, you’re often learning. That means you’re also generating a bunch of technical debt, just by the very nature of learning.
Most of IT is built out of technical debt. Everybody’s constantly learning about how to do the thing that they’re doing. That’s especially bad in DevOps because everything’s changing so fast. It’s especially bad when you’re consuming services from external partners, such as Amazon Web Services. Everything’s glued together with spit and hope.
Tell me about a previous company or team you worked with where DevOps wasn’t working.
I worked with a company in Berlin… I got hired by the then-CTO. It was a quite complicated system; it had many moving parts. Everything was written in Ruby and every single system was deployed in a different way. Everything! The main website was deployed on some servers in Rackspace. There were some other bits that were deployed onto ECS. Then there were some bits that were deployed in Docker on another platform. And some other bits were using another deployment method and another technology.
All these DevOps engineers that were there during the heyday of the company, they treated it as a massive learning exercise. Nothing was finished, it was all half-done experiments.
The DevOps engineers didn’t ever go back and fix whatever they had done before?
They had four engineers and all of them did whatever they wanted. Then they all went off and worked for other companies.
I’m pretty sure it was the unmanageable infrastructure which killed that company. It was insane, everything was unstable, everything was half-done, it was mind-blowingly bad. It was unmaintainable. This happens a lot. You have these DevOps engineers running amok. I’ve been that person: there’s no oversight, nobody else knows what they’re doing.
When did you realize that DevOps consulting wasn’t solving the actual problem?
The moment I walked out the door, nobody knew what I’d done, nobody understood it. You can write documentation until you’re blue in the face, but the knowledge of how to deal with all of this stuff, I know now is tacit knowledge. Tacit knowledge is knowledge which is very difficult to share or transfer verbally or written down. This is why documentation in software development is kind of a joke.
A carpenter can’t read a book on carpentry and sit and talk to a million carpenters, and then just make a table. Technical knowledge is tacit, you have to feel the tools in your hands. You can’t just read the documentation and do it.
How does Kubernetes change the relationship between developers and DevOps?
All of the complexity, which DevOps engineers previously had to manage, ceased to exist: in Kubernetes, all of the “Ops” have been “Dev”ed. Everything that I ever tried to achieve as a DevOps engineer is all delivered by Kubernetes out of the box: scalability, reliability, cost-effectiveness.
What is the ideal end of state for software developers using Kubernetes and productivity engineering?
They’ll have ownership of their own domain. They’re autonomous, they can write, they deploy, and manage their own applications. They  don’t need any third parties to do that. I consider that we’ve won when we never hear from people ever again. That’s our goal: to set people up so they don’t need to talk to us again.
 
To learn more about Andrew Holway’s predictions about DevOps and Kubernetes, strategies for transmitting of tacit knowledge, and the growing need for productivity engineering, join Leaders in Tech | Berlin on February 13th, 2019: http://eventbrite.com/e/leaders-in-tech-berlin-devops-is-fscked-tickets-54523716905