If there is one topic that is on every engineering leader’s mind, regardless of the size of the company, it is how to ship more code more often and how to run that code reliably and efficiently. So, it was a no-brainer that we chose to kick off the first of our Madrona Engineering Community (MEC) meetups with an event on DevOps and Agile practices.

MEC meetups foster ongoing knowledge sharing and networking among engineering leaders across the Madrona portfolio. Our members are VPs of Engineering and CTOs across our portfolio companies.

As far as first events go, we started off with a bang! We had a packed house with over fifty of our engineering leaders attending as our panel of experienced speakers dove into the current and future for DevOps. Our panel was:

• Tom Casey, SVP Engineering at DocuSign
• John Ludeman, SVP Engineering at Skytap
• Adam Johnson, Founder/CEO of IOPipe
• Moderator, Joe Duffy (Founder/CEO of Pulumi

The make up of the panel was by design — to cover companies at different scale. DocuSign, for example, serves hundreds of millions of users and more than 1.5 million documents per day. Skytap is a mid-stage company that has seen its usage quadruple over the last two years, deploying up to 45,000 VMs and serving up to 2 petabytes of storage a day. IOPipe, in contrast, is in the early days of building its product with a distributed team churning out multiple releases every day! The DevOps goals and plans of each of these companies are different, which led to an interesting discussion under the expert probing of our masterful moderator Joe Duffy.

DevOps = ownership + accountability

While Joe did his best to instigate the speakers in taking opposing positions, he had no luck when it came to how they thought of DevOps. To each, DevOps was about aligning engineering teams with creating customer value and achieving business goals. The key, as each speaker emphasized, was to give developers end-to-end ownership (and the accountability that comes with it). Developers must be able to see the value they are delivering to the customers, or not. And, they must be provided with all the tools and processes, and more importantly, the organization and cultural support needed to succeed.

DevOps ≠ shipping more features

In the fog of war, there is often a tendency to add every feature request to the scrum. Shipping, however, is not a good measure of success. Adoption, utilization, learning, and direct revenue are much better measures of value creation. When it comes to DevOps, reemphasizing value creation and not just features shipped is important. DocuSign, for instance, made driving adoption (of their product) a company wide priority because, direct revenue is typically a trailing indicator of feature value. To keep everyone focused, they evaluated every shipped feature in light of adoption. It mostly worked for them.

DevOps is always a work in progress

No matter how small or big the company is, DevOps is never done. John from Skytap, for example, shared how, after operating for a few years, they realized that their organizational structure was getting in the way of shipping code quickly and efficiently. So, they reorganized and refocused the engineering teams to make sure that they stayed agile as the business scaled rapidly. Tom from DocuSign shared a similar experience, done at a greater scale in a more gradual manner. And, they both fully expect to redo their DevOps process and organization multiple times as their businesses continue to grow quickly.

No “no-ops” unless you also believe in unicorns

In this day and age of serverless, the topic of “no-ops” inevitably came up. To much of Joe’s delight, it elicited some strong reaction from the speakers. Long story short, the verdict was that no-ops does not work at scale. Every SaaS company, once it reaches a certain scale, needs specialized ops to ensure it can deliver the best user experience without blowing up its cost budget. DocuSign has “devs who op,” and every company could do with more of them. Similarly, the topic of SRE did not find a lot of love from the panel. The view was that, for most SaaS applications, it is impossible — or, at least not advisable — to insulate the developers from running the application.

Telemetry trumps testing

The other interesting insight that emerged from the discussion was that, no matter how much one invests in testing, telemetry is indispensable and if anything, more important. Skytap, for example, believes investing more in application telemetry than in running exhaustive regression tests. Similarly, DocuSign puts some of its best devs on telemetry. The emphasis on telemetry helps to speed up release cadence and detect unforeseen bugs.

Looking forward …

While most modern companies have adopted DevOps to a certain degree, most also struggle with several issues. First, balancing agile development with predictability (a must to run a business) is always hard. Hence, every engineering leader clamors for effective planning tools. The other issue is how to measure engineering productivity. Lines of code written or number of bugs fixed could be great vanity metrics, but they do not serve the real purpose. Finally, as we get ready for applications running on multiple clouds, we do not yet fully know the new challenges that will bring to DevOps.

Maybe, that’s where we will pick up at the next MEC meetup!