Deepgram’s
Scott Stephenson
on revolutionizing voice AI and building a developer tool

Deepgram’s
Scott Stephenson
on revolutionizing voice AI and building a developer tool

Madrona Partner
Karan Mehandru
Initial Investment
Acceleration
Deepgram
Madrona Partner
Karan Mehandru
Initial Investment
Acceleration

What was the problem that inspired the creation of Deepgram, and why were you the right person to tackle that?

In a previous life, I was a particle physicist working deep underground building dark matter particle detectors. It was awesome — I was two miles underground in a government-controlled region of China, basically in a James Bond lair. But, when my team got our experiment working, we didn’t have much to do — we were collecting data for six months. Being in this exotic location underground, I thought a documentary crew should follow us because it was pretty unbelievable. Who gets to do something like that?

We decided to try to make our own, and the easiest thing to do was to record audio all day, every day. We built some devices to record 24/7, which left us with over 1,000 hours of audio — and nobody wants to go back and listen to your life again. It’s too much data, takes too long, and is pretty boring, despite how exciting you might think your life is. We assumed a product must exist that could understand open-ended audio to help us process all of ours because we were doing something very similar with waveforms in our particle physics work. But we didn’t find anything suitable — we just saw antiquated technology.

We didn’t set out to build a company, but once we realized that literally every company, even the most cutting-edge companies, was using antiquated technology, it became obvious that we were the ones to build it. We wanted to build a developer tool company that would allow anybody like us, developers, to understand what’s inside their audio. And that’s Deepgram — we do speech recognition and text-to-speech. Now we’re a voice AI API that allows anybody to transcribe audio, generate audio, search through audio, and figure out how many speakers there are. And we’re the largest speech-to-text API in the world now.

I had been thinking I would enjoy starting a company for a while, but I never had anything I thought was worth dedicating all of my time to until we realized this big hole was in the world. I like to say that what we were working on in physics was discovering the fundamental laws of nature. But working on Deepgram would be discovering the fundamental laws of intelligence. We could take the same mentality from physics and put it into AI. It totally works. There was a problem, though, which would be figuring out how to sell it to the world. You have to figure out how to distribute, which is very different. But I thought if anybody’s going to be able to do it, it would be us.

Visiting the folks at Madrona, going to the events, seeing the cast of characters and people they bring to events, and the portfolio companies — I was pretty blown away.

What is it like to work with Madrona?

Because we are a developer tool, we wanted to work with somebody who understood that type of product and the mentality a technical CEO/founder would have, which I saw Madrona had as soon as I met the folks there. I thought, “Oh, you guys get this kind of company.” That was the thing that stood out to me most — Madrona understands this kind of company. With that and the fact that we were finishing each other’s sentences, we thought, “Let’s go.” When we did the funding round, everything went quickly because we were amazing fits.

We have a handful of investors at Deepgram, and the two I’m talking with all the time are Wing and Madrona — Wing led our Series A, and Madrona led the Series B. We’ve got a bunch of others, but Madrona and Wing are the ones that are in the trenches with us.

Tell us about a Madrona Moment.

Deepgram is a 9-year-old YC company with many investors, most of whom are in Silicon Valley. That means I’ve been to tons of events in the valley and have seen how different VCs interact with their portfolio companies and the community.

That’s what stands out the most to me about Madrona. Visiting the folks at Madrona, going to the events, seeing the cast of characters and people they bring to events, and the portfolio companies — I was pretty blown away. Of all the events I’m invited to, those are my favorite. And it’s not only because of the event, it’s because of the people who come as well. They have amazing LPs, amazing portfolio companies, and an amazing community. Every time I go to an event, I meet somebody, probably several people, that I’m very happy to meet. Unfortunately, that is not normal for me when I attend a VC event. I would love every VC event to be that way, but the Madrona ones have a particularly high hit rate. Whenever I’m tapped to speak at or attend a Madrona event, I usually like, “Sign me up,” because the community is so great.

What have you learned about yourself during this process?

I am a decisive person with lots of energy who is always thinking about things. But, the business world has a way of teaching you to be patient. Many times, you shouldn’t be patient, but there are certain times when you have to wait for something to develop, whatever it is, because you’re working with human-based systems. The most logical thing to you may not be the most logical for the world at that moment. It needs to develop over time. When we started the company nine years ago, we were predicting all of what is happening in AI today, so we thought, “Let’s rush there as soon as possible. Right?” For instance, one of our first products was a vector-based embedding with a search. Embedding databases are popular today, but this was nine years too early. So, I realized, okay, we need to look at where the market is, supply the market, and look out ahead a year or two — not a decade. Look out a decade, but don’t build for the decade quite yet. Build for a year or two into the future. So, that’s one thing I’ve learned — sometimes you have to be patient because the market is just not developed yet.

What is the most important lesson you have learned during your startup journey?

For us, the most important lesson was about distribution. We’re deeply technical founders — we never had to deal with standard distribution mechanisms. Coming from academia, the norm is that when there is a superior method to do something, everybody adopts it within six months, maybe a year. You publish a paper about it. You do a few talks. And then everybody’s using it. That is not how the business world works. Many folks say distribution matters even more than a product. That’s probably true in the long run. Even if you’re the best product builder — the most technical — if you don’t distribute and other people do it better, you’ll lose. Then you learn that lesson the hard way.

As I said, Deepgram is almost nine years old now, and we’re a foundational AI company. We build our own models. That’s what we do. But we didn’t train a single model for almost a year and a half in 2018. We were an eight-person company, and our model was already the best in the world. It didn’t matter if it got any better. That wasn’t what was stopping us from growing. It was getting the distribution right and making sure the product was reliable. As a technical founder, there will be long stretches of time when you’re not doing what you would normally be inclined to do. My team is full of deeply technical people who always want to be training models. But all we did for that year and a half was smile and dial. If your day isn’t 90 or 95 percent stuff you don’t want to do as a technical founder in the early years, you may be driving yourself right off that cliff. It is a pendulum. When you first start your company as a technical founder, 90 percent of the time is spent on technical stuff. But then, once it starts to work and you need to scale, it swings back the other way. Then, finally, as you build your executive team, you can scale your time back up to focus on more frontier things.

More leader conversations

A-Alpha Bio
A-Alpha Bio’s
David Younger
on advancing drug discovery and the role of company culture
Terray
Terray’s
Jacob Berlin
on why data is the difference for AI-driven drug development
TerraClear
TerraClear’s
Brent Frei
on the moment of insight that leads to building a product
Stacklok
Stacklok’s
Craig McLuckie
on solving open-source security challenges
Rec Room
Rec Room’s
Nick Fajt
on democratizing game development
MotherDuck
MotherDuck’s
Joran Tigani
on shaping serverless data with DuckDB
Echodyne
Echodyne’s
Eben Frankenburg
on commercializing high-performance radars
Impinj
Impinj’s
Chris Diorio
on pioneering an industry and perseverance
Fauna
Fauna’s
Eric Berg
on building an infrastructure business to scale
Runway
Runway’s
Cristóbal Valenzuela
on AI-driven art and lifelong learning
Unstructured
Unstructured’s
Brian Raymond
on tackling engineering bottlenecks
Rover
Rover’s
Aaron Easterly
on the love of pets and finding the right partners
Highspot
Highspot’s
Robert Wahbe
on customers, AI, and building relationships that last
Common Room
Common Room’s
Linda Lian
on integrating community into your company
Pulumi
Pulumi’s
Joe Duffy
on bringing more power to developers
Crowd Cow
Crowd Cow’s
Elizabeth Liu
on the power of e-commerce to transform an industry
Spotnana
Spotnana’s
Sarosh Waghmar
on challenging a decades-old industry
Statsig
Statsig’s
Vijaye Raji
on using data to build better products
OctoAI
OctoAI’s
Luis Ceze
on bringing ML to all platforms
RelationalAI
RelationalAI’s
Molham Aref
on finding investors who believe in your passion
Flexe
Flexe’s
Karl Siebrecht
on transforming the logistics industry
Pendulum
Pendulum’s
Mark Listes
on finding hidden narratives in social media
SeekOut
SeekOut’s
Anoop Gupta
on hiring and untapped talent pools
Nautilus Bio
Nautilus Bio’s
Sujal Patel and Parag Mallick
on the proteome