These are the lessons I learned working inside the CTO’s office of a database technology company.

In the fall of 2017, I transitioned into the CTO’s office under Jonathan Ellis after bringing DataStax Graph to market and iterating on it for a year. The goal was to help Jonathan with technology strategy development for DataStax and product technology decisions across the product portfolio.

I was the CTO at Aurelius before our acquisition. However, at an early stage startup “CTO” usually means “head of engineering” because your priority is leading the engineering team and developing software. At DataStax, a database software company with 450 employees at the time, CTO was a role outside the core engineering organization responsible for long term technology strategy and medium term product technology decisions.

This post outlines some of the lessons I learned in this new role inside the CTO’s office.

What does a CTO do?

Figuring out what a CTO does on a day-to-day basis was the first lesson I needed to learn. “Technology strategy” sounds cool and important but isn’t a great job description.

There are two types of CTOs: external and internal [1]. External CTOs spend most of their time explaining a company’s technology to their customers, the press, and the outside world. Those CTOs have technical expertise and credibility and also build a public persona. They write blog posts, give talks, do press interviews, and generally have a high degree of visibility outside their company. Internal CTOs identify which technology areas to invest in, consult with the CEO and executives on technology topics, and provide technical guidance to architects and engineers within the organization. They spend a significant amount of time keeping up to date with the latest technology trends and try to ensure that the company is well-positioned within the rapidly evolving technology landscape.

Most CTOs cover both internal and external responsibilities but emphasize one over the other depending on their organization’s needs [2].

At DataStax [3], the CTO role was primarily internal. We spend most of our time analyzing emerging technology trends to identify areas of investment and work with engineering teams to ensure that we were making the right technology choices for future product success.

Provide Guidance

The CTO has the power to dictate technology choices but should never use that power unless in desperate circumstances. That’s the hardest lesson I had to learn and a mistake I made repeatedly in the beginning.

I thought that being in the CTO’s office meant that I should think hard about a particular technology decision, do all my research, and then emerge with a solution that gets handed to the engineering team. The engineering team either accepts that solution as gospel or needs to be convinced that it is the path forward.

That approach leads to poor outcomes because the engineering team will feel disenfranchised and steamrolled. I thought that if I invested enough energy into coming up with the “right” solution, the engineering team would eventually see the truth and come around. However, handing down a solution isn’t empowering, it is patronizing. The funny thing is: when I am on the receiving end, that dynamic is obvious to me. Being in the CTO’s office blinded me to that reality.

Any broad technology decision is fundamentally a multi-objective optimization problem and the optimal solution depends on how you define the objective function. For example, take the problem of “which indexing engine should we use for columnar predicates?”. The answer depends on what exactly you are optimizing for. Time to market? Query latency? Query throughput? Memory consumption? Maintenance effort? Often, you are trying to optimize for all of these and then it becomes a matter of weighing them.

The hard part is usually agreeing on the priorities and what you are optimizing for. Once everybody agrees on that, finding the “right” solution is not difficult. However, as an engineer, I am obsessed with finding the solution. Hence, I tended to speed through the “defining the problem” part to get to the good stuff: finding the solution.

Being a good CTO requires unlearning that engineering reflex. Rather than blurting out the solution, I need to spend my effort on defining the problem and providing guidance to get everybody on the same page regarding the problem definition.

Once the problem is well-defined, engineers can come up with their solutions. Making it “their” solution ensures that they feel a sense of ownership and are invested in the outcome. To write good software, engineers need to own the solution.

And if you are convinced that there is a better solution than the one that your engineering team came up with, asking a question based on the mutually agreed upon problem statement is an effective tool to guide the team in the right direction. “We agreed that 99th percentile read latencies are super important, hence I am wondering what you estimate the read latencies of your solution to look like?” is 100 times more effective than “your solution sucks because it has bad read latencies”.

There may be times when you have to use the CTO veto power because you are running out of time or aren’t making progress. If you find yourself using it more than once every 6-12 months, ask yourself whether your problem definitions are too vague or whether your engineering team is incompetent.

Effective Listening

Aside from “defining the problem”, the most important skill you need in the CTO’s office is listening. And listening very carefully. Listen to your customers, listen to the leaders in the company, and listen to the engineers. Most of the problems that you are dealing with in the CTO’s office are highly complex. They are highly complex because they are not “just” engineering problems but require an understanding of customer needs, business constraints, and running hypotheses for what the future is going to look like.

To be able to define such complex problems requires that you can assemble as much relevant context as possible. And then you need to prioritize that context to avoid decision paralysis. You need to carefully listen to design a mental model that allows you to do those two things effectively.

I’m bad at listening. When somebody starts talking, my brain runs a Markov Chain on what they are going to say so that I can prepare my answer. While it is emotionally satisfying to predict what people say, I lose the ability to pick up subtle nuances in communication. Those nuances are critical to parse the intent behind somebody’s communication. Are they stating an opinion or a belief? Are they describing reality or a desired state?

To counter my brain’s natural tendency, I am trying to use active listening. Active listening means that I approach a conversation to be able to reflect back what the other person has said as accurately and concisely as possible. I am forcing my brain to play a different game: lossless compression of what I heard. That is a surprisingly good way to absorb what the other person is communicating.

Stay Attuned to Customer Needs

One type of listening that is particularly important is listening to your customers. It is important to keep in mind that all the technical work you are doing serves one purpose: solving customer problems.

It is easy to lose sight of that in the CTO’s office and get obsessed with technology. You probably don’t end up in the CTO’s office if you aren’t passionate about technology. But that passion may lead you to embrace technology due to some perceived intrinsic merit. I would catch myself falling in love with technology and then invent a reason for why it will benefit our customers.

The only way to counteract this natural inclination is to keep customer problems top of mind. Instead of reasoning your way from a particular technology to a customer’s problem - which you can always do with enough creativity - always start your reasoning with the customer’s problem.

For instance, Blockchain was all the rage during my time in the CTO’s office fueled by the Bitcoin boom. There were a lot of people inside and outside of DataStax who were excited about the technology and pointed out how it could help address certain customer problems. That put us under pressure to come up with a Blockchain strategy for DataStax.

While Blockchain is a very exciting technology, Jonathan argued that none of our customers' primary problems are best solved by a cryptographically secured ledger. While it turned out to be the right call in hindsight, it was very controversial at the time. The difference is that Blockchain enthusiasts reasoned from the vantage point of the technology until they found a connection to customer problems whereas Jonathan reasoned from our customer problems and didn’t find an obvious path to Blockchain.

The latter type of reasoning is only possible if you stay attuned to your customers' problems and understand them deeply.

Have an Explicit Innovation Strategy

During the 2nd half of my tenure at the CTO’s office, I focused on technology innovation. The goal was to develop novel technologies to solve our customers' thorniest problems. My failure to bring one promising innovation to market taught me the importance of having an explicit innovation strategy.

In the beginning, we spend a lot of time digging through customer data and interviewing customers to identify their highest-value problems. After some experimentation and a few feedback cycles, we eventually settled on the problems of data modeling and data access.

Users of Apache Cassandra, the database at the core of DataStax’s products, often struggle with designing the optimal data model for their application. And they write a lot of boilerplate code to make their Cassandra data accessible to applications through standard APIs like REST or GraphQL. These two problems consume a lot of development time and are a big reason for developer frustration.

We speculated that by solving the two problems together, we could give database users a simplified development path. We could develop an API layer that automatically generates all the data mapping and data modeling based on a user’s API definition. That way, the user can focus on what they care about - the API they need for their application - and we handle all the gory database details for them.

We validated this approach with customers before starting to experiment with different technologies to generate the intelligent mapping into the database. After a handful of iterations, we build a working proof of concept that garnered promising feedback. We called it AppStax.

Unfortunately, I did not successfully navigate the tricky path from technology prototype to released product. I did not have a clear strategy for integrating this technology innovation into our existing product portfolio. In hindsight, I realize that’s where a lot of innovation projects go wrong. At the time, I made a series of decisions to bring AppStax to market quickly, all of which did not succeed because they were not aligned with the overall product roadmap.

As I was deeply embedded into the innovation project, I did not see that my localized decision making was not aligned with the overall product roadmap and the broader engineering effort. That’s why it is critical to have an explicit innovation strategy that maps out how an innovation project rejoins the core product development stream. Disruptive innovation needs to happen outside the core development process for some time to maximize agility and experimentation, but it is equally important to plan for convergence back into the stream. Otherwise, that project will float off into no-man’s land and die.

That’s what happened to AppStax. After 9 months of flailing around, the project got canned. AppStax was eventually reborn as Stargate within the core product development process. That was a painful transition that could have been avoided with a clear strategy.

Summary

Working in the CTO’s office taught me a lot about the role of the CTO and how to effectively provide technology guidance across an organization. I am deeply thankful to Jonathan Ellis for giving me this opportunity and his continued mentorship.

I also want to thank the AppStax team for joining me on this crazy journey: Luke Tillman, Daniel Henneberger, Lawrence Lazare, Sterling Koch, Matt Scandalis, Steve Donnelly, Kevin Gallardo, and Dmitri Bourlatchkov.

Footnotes

[1]

Credit to Billy Bosworth for this characterization of CTOs and a big thank you for his feedback and guidance on the CTO role.

[2]

I learned a lot about what CTOs do in general from the book “CTOs at Work” and the podcast of the same name.

[3]

The opinions expressed in this post are my own and do not reflect those of my current or former employers.

[4]

Photos by ThisisEngineering RAEng, Hayden Walker, and Franco Antonio Giovanella on Unsplash