240: What KPIs to Track as a Startup

Steli Efti:

Hey everybody, this is Steli Efti.

Hiten Shah:

This is Hiten Shah.

Steli Efti:

In today’s episode we’re going to talk about what KPIs to track as a startup,  how to do it right, and what mistakes to avoid when it comes to the  key performance indicators that you’re tracking, the key metrics, the reports you’re generating as a  startup, the data you’re looking into. You’re one of the most prolific farmers when it  comes to tracking data, right? You’ve had multiple companies that are in the analytics space  itself, and you’re a fairly analytical guy, Hiten, so I wanted to talk to you  about this. Because I see almost every startup if not every startup that’s starting very  early on has to ask themselves the question, what are the numbers that I need  to keep track of? How do I measure success? What I see is that startups  in all phases are struggling with this for one reason or another. In the early  stages they might struggle because they’re either tracking too many things that don’t matter, or  they haven’t found, or they track the wrong one metric. They pick the vanity metric,  they’ll pick the “bad” metric versus a good one to have as their key performance  indicator, the one number that they’re keeping the most track of. Also, as startups grow  and in the later phases, there’s always this struggle of tracking too much data, not  tracking enough, not having the right data internally, not having the right reports, having doubt about the accuracy of some numbers. It seems like there’s always a struggle. It’s very  rare that I meet a startup or a company that has perfect control over their  numbers and is very confident, and calm, and collected, and focused on it. Most startups,  they track a lot but they’re kind of flustered, and they don’t know what to  track and what not to track. I thought that talking about that would make a  lot of sense. We’re probably going to focus most of the attention on the earlier  side of things. You start a new project, and you’ve started a few new companies just recently. How do you decide what are the numbers, what are the KPIs we  are going to be tracking? What’s the framework that you use to arrive at the  right KPIs?

Hiten Shah:

KPIs. This terminology came from back in the day in some MBA management books and  stuff like that, right? It’s not exactly startup friendly, because honestly startups don’t really have  many performance indicators in the earliest stages that are trackable and you could set goals  on. The first thing I would say is to me, there’s a couple lenses I  put on a business, especially a startup, and that would determine how you think about  KPIs. The first lens is what’s the stage of the company? Is it super early  stage, pre-product or early product, or is it kind of later stage? I’m just going  to bucket everything later stage overall, where it’s post product with some amount of customers  and/or users depending on the type of business. That leads me to my second lens,  which is type of business. Every single type of business has a different set of  KPIs, and by type of business I mean is it a subscription business? Is it  a SaaS subscription business? Is it an ecommerce subscription business? Is it a B2B business?  Are there people who are selling? Is there a sales team? Is it a consumer  product? How does that consumer product make money? On advertising or through one time payments  like the ecommerce shop or even like somebody who just sells ebooks and courses versus  software? All these things, those two lenses are what’s really important to me when determining  the KPIs in the business. For example, if you’re early stage, and you don’t know  if you have product market fit, initial product market fit, and the product is working,  then your metrics and your KPIs are all around, is the product working? For me,  those metrics include, the key performance indicator there is mostly around customer satisfaction and/or a  specific engagement metric such as retention or number of certain actions people are taking every  day, every month, on average, per individual. That’s early stage where I like to focus.  As a business gets later stage and the business model gets defined, that points to another point, which is, is the business model defined? Is it working or not? Because  if it’s defined, then all your metrics go towards the business model, right? For example,  right now I have a business called Crazy Egg, over the last until literally yesterday  we’ve had one feature. It’s a heat map. Today, because yesterday this changed, we actually  have more than one feature, we have three now.

Steli Efti:

Holy shit.

Hiten Shah:

Yeah, we 3X’ed the product, and as a result, and this is a business model  that’s been going since 2005. This business has been one feature for 12 years.

Steli Efti:

Holy shit.

Hiten Shah:

Yeah, all of a sudden we’re like, “Holy crap!” Our metrics changed now. We were  focused on different metrics before, like churn and things like that. Now we just have  one metric, and that metric that we care about, our key performance indicator today, because  we went from one feature to three and it’s still a business model thing we’re  looking to improve. Because we already have a business model, we charge a subscription fee,  is the average new monthly recurring revenue, MRR, for every customer that comes in that’s  a new first time payment customer. The reason for that is it’s get older users,  older customers used to new features. It’s much easier to introduce new customers that are  just learning about your product and using it for the first time or the first  few times, for the first trial or whatever, to start using all these features. We  decided deliberately not to spend so much effort on a metric that’s based on existing  users, because we know how that’s going to be. We’ve run tests for this, we  know that’s going to be a little bit of a slog. Instead, all of our  metrics are focused on these new people coming in, and is our MRR for these  new people going up on average compared to what it’s been?

Steli Efti:

That’s all.

Hiten Shah:

Contact specific KPIs is probably what I’m advocating for more than anything else.

Steli Efti:

I love it. I remember, and I’ve told this story before in other places, but  I remember one of the most impactful things in our very first week going through  Y Combinator and talking to Paul Graham, PG, was we went on our first office  hour walk with him, and we described to him what our business, the startup that  we were running back then, was doing. The entire conversation was revolving around, “What’s the  measure of success here? How do we know if this is working? What’s the core  thing we want people to be doing that shows us that they’re getting value and  this is working?” Once we arrived at what that core number what, what that core  thing was that people needed to do to show us that it’s really working, he  turned around and he said, “Can you promise me 10% growth off that number every  single week from now until the end of YC?” I think at the point where  he asked, we had like 10. The number was literally 10. I was thinking, “Well,  if I get one more person to do this next quick, that’s 10%, that seems  pretty easy.” I’m like, “Sure, no problem!” He’s like, “All right.” Shake hands, and as I was shaking his hand I’m like, back then it was like the core number  was fully enrolled members. Enrolled members back then meant people that signed up with a  credit card to make donations that was based on a totally different business that we  were running today. As I was shaking his hand, I’m like, “Do you have your credit card with you?” He started laughing and was like, “Yeah, I do.” I cried,  “Let’s enroll you right now to our site.” He was like, “See? That’s why I  wanted you guys to focus on one thing.” Besides just me that day walking through  YC and getting everybody who had a credit card to sign up, what it did  is it allowed us, because we had this laser focus on one number we wanted  to improve on, it allowed us to make every decision based on answering the question,  “Will this help us growing at least 10% next week?” That’s how we prioritized every single decision in the first three-four-five months of the company. It was a beautiful way.  Now you can’t do that forever, and as the company grows, as the business grows  it becomes a little bit more complex than that, but for the early days I  thought that it was a beautiful way to prioritize, because you have so many things  you could do, so many ideas. How do you prioritize? How do you make sure  that you progress in a certain way? Also, and this has gotten old by now,  but back then in 2011, I think we ended up having 25% week over week  growth for the first three months. If you have like 10 or 20% week over  week growth in a metric that truly matters, that’s not total BS, it makes a  beautiful nice up and to the right curve. It creates the right kind of picture,  the right kind of momentum. That was incredibly helpful to us, to really zero in  on one thing and what matters. Also really challenging, and we’ve talked about this before,  challenging yourself to not pick the number that looks the best or that’s the easiest  to improve on, but to really take the number that is the most meaningful number,  the closest to reality on what it means if what you are trying to accomplish,  you’re truly accomplishing. I love this example with Crazy Egg, where it’s like right now  the most important metric is new net MRR by new customers. We’ve tripled the product,  we did it because we’re XYZ goals, and the best way for us to measure  this is working is to focus on this one group of customers, which is new  customers, and see how much MRR they are generating now when we offer them so  much more product than the past customers. I love that. What are some common mistakes  that startups make beyond the maybe not tracking anything? I don’t even know that that  still happens today. I can’t remember a single case where I met a startup where  they just don’t track numbers or don’t track any KPIs. Tracking the wrong ones is  definitely something I see frequently. Just giving me bullshit numbers, and then when I challenge  them on that, they just all fall apart. “Oh, we have over a million users  on Facebook,” or, “Over a million users for our chat bot related mobile app.” Then  I go, “Can you define what a user is? Is it somebody who downloaded your  mobile app, and is logging in every day, and is doing XYZ access? How did you define a user?” Then they tell me, “No, user is somebody that has replied  to a Facebook message or chat bot message that was initiated by our mobile app.”  I’m like, “Okay, so if I get a chat bot thing on Facebook and I reply, and I’ve never downloaded your app, I know nothing about you, I don’t know  what your app is called, I’m counted as a user because it’s some kind of  a growth hack you’re trying to use?” They’re like, “Yeah.” I’m like, “Okay, how many  people have downloaded your mobile app?” “4,000.” I’m like, “All right, mother fuckers, you can’t  be walking around telling me you have a million users with your mobile app company  when you really have just generated a million people to just reply to some chat  bot.” There’s a lot of these BS metrics that people focus on, and I think  that that’s a huge mistake people make. It’s kind of tricking themselves and others into  them having more success than they truly have, but what other mistakes? You’ve seen so  much on the KPI and tracking numbers side. What are the common mistakes you see  again and again on like companies struggle with this or making mistakes with how they  track KPIs?

Hiten Shah:

Yeah, yeah. Number one, as much as I love Y Combinator, and Paul Graham’s advice  to you, there’s one big word of caution I would have, which you already mentioned.  If you pick the wrong metric to improve, you’re going to improve the wrong metric.  The context in your business was like, look, these are people actively signing up with  a credit card.

Steli Efti:

Yeah.

Hiten Shah:

That’s generally a great metric to use regardless of what business you’re in, especially if  your business is at a stage where it’s ready to accept a credit card. Right?

Steli Efti:

Right.

Hiten Shah:

I’ll say something kind of interesting which is that methodology, that thinking is exactly why  Facebook came out and said, “We have millions of advertisers using our advertising system.” They  kept touting that metric, and the reason is because they had this attitude that, “Wait,  our business doesn’t make sense unless our ad system is actually getting people to pay  money and use it.” That’s not how Google thought about it, to be honest with  you. They thought about it like, “We want the largest advertisers.” It’s not like Facebook  doesn’t want to largest advertisers. They want to go out there and say to the  market for their optics, even for their business, that “We have millions of people advertising,  it doesn’t matter what size business they are.” I think that you have to pick the right metric. It’s really important, like that’s the number one thing. When you pick  a goal like we’re going to increase 10% every week, make sure that it’s the  right metric or you’re going to have a million chat bot conversations and 4,000 app  downloads, and no one’s going to find your business, or you’re not going to be able to build a business. Number two, I’ll give a few. Number two is like  they don’t measure it properly. When Paul Graham says 10% every week, let me just  do math. If you have 10 right now, and you add 10%, you’re adding one,  right?

Steli Efti:

Yeah.

Hiten Shah:

Now you have 11, right?

Steli Efti:

Yep, yep.

Hiten Shah:

Every week that number goes up, you have to add 1.1 in order for that  to increase. Your 10% week over week goal isn’t week over week as much as  it is compounded based on where you started.

Steli Efti:

Yep.

Hiten Shah:

Which is something people get so wrong, it really infuriates me, because what they think  is, okay, let’s start with a little bit higher numbers. I have 100 sign ups,  I want to grow 10% week over week, I got 10 next week, so now I’m at 110. I made it. It isn’t your goal to get like one more  on top of the 10, even though in the beginning that sounds about right. It’s  actually to get 10% of the total.

Steli Efti:

Yep.

Hiten Shah:

Right? It’s the 10% of the total. Not 10% of what you did last week.  I can’t tell you how often this is overlooked and mistaken for the wrong thing,  because compounding growth is what you’re going for. Honestly, I haven’t heard YC really explain  this very well, and yet people tout it, and then they get confused. That’s my  number two, and it’s not just YC, it’s a growth rate you’re going for. An  increasing growth rate, just like your interest compounds every year, every month when you have  money in the bank, and there’s an interest rate. That’s what you’re going for. That’s  such an important thing that I’ve seen so many people miss, and the reason they  miss it is if you have a subscription SaaS business, monthly recurring revenue is naturally  a compounding metric. It’s hacked, like you just focus on that. That makes it really  easy to calculate, and everybody knows what that means if they’re in a subscription business.  If they don’t know, within a few minutes of launching their subscription business, they realize  what that means, because the money compounds. I got 10 bucks, as long as I  don’t lose those 10 bucks from that user, I get to keep having that every  month. That’s compounding, that’s what I’m talking about. That’s number two. I think number three  is like, I’m going to come back to this, and come back to something from  earlier. I just see people having too many metrics. They have tons of KPIs, and  I’m always saying I’m coming back to it because you already mentioned this, but I  think it’s underscored in your story. Even at Crazy Egg, there’s a million metrics we  could focus on. There’s so many metrics even that are sort of indicators of our  average new MRR, such as people using these features during their trial, which will lead  to an increase to average new MRR for us, obviously. We don’t count those as  the main metric we’re trying to improve, and there’s a framework you can use called  leading indicators and lagging indicators. A leading indicator is one that tells you you’re going  in the right direction. I’m sorry, a lagging indicator is one that tells you on  a high level that you’re actually, that’s where your goal should be. Usually it’s like  revenue or churn or something like that. A leading indicator is an indicator, a metric,  a KPI that helps you understand whether you’re moving in the right direction or not.  For us, if people are using more than one feature in our product, that’s a  leading indicator to them paying more money than our existing users, as an example. There’s  this idea of leading and lagging indicators that gets really important as you get more  advanced around understanding KPIs, because you don’t want to focus your main KPI on a  leading indicator. You actually want to focus it, in an ideal world, on a lagging indicator initially. Then all these leading indicators are what you discover, and then you might  flip the switch and say, “Oh, if we can get everyone using three features, average  new MRR goes up, let’s just change our KPI that we focus on to getting  feature adoption, and getting every single user.” Right now, let’s say on average, out of our whole user base, people are using 1.5 features. We want to increase that to  three in the first seven days of using our product. That’s the kind of thinking  you can use once you start learning more. I think, and this probably goes to  my last point about this, which is these things are fluid, but don’t change them  unless you’re actually learning why you need to change them and learning what the better  KPI should be. It’s not like you should change them often. If I see someone  changing their KPI every two-three weeks, that’s bad. It’s bad for two reasons. One, they  keep changing it, they’re changing their goal, which is bad in a lot of cases,  or two, they didn’t start in the right way by thinking of context when they  determined it early, or number three actually, they’re listening to too many advisors. That’s common,  unfortunately. Come talk to Steli or I, because we’re definitely better at cutting through crap  like that, but in general you don’t want to change them, but if you learn  something new about your business, you’d be stupid to be so rigid that you’re not  going to change it. It’s likely even in our business, as we move and learn  about this one metric, and what the leading indicators are that are going to increase  our average new MRR, we might change it. We are not likely to because we’ve  done a bunch of work to determine this one. That being said, I could imagine  three or four months from now, once we get to a good place there or  have a new learning, we might change it because that leading indicator is actually the  key thing for us that we learn about. That’s kind of four or five tips  that I have around metrics. These are really core. I did run an analytics company  for a while, and still probably run one depending on how you look at it.  This is not much more complicated than that, that’s why the KPI topic’s really interesting  to me, because if you don’t know where you want to go or you don’t  have a metric, you don’t have some measurement system and are focused on one metric  ideally, then you’re not going to get anywhere. This is by team if you have  a larger company, but usually a company has one metric they focus on, and the  teams are focused on metrics that relate to that one.

Steli Efti:

Sweet. I think this was both powerful and beautifully put. All right, there’s not much  more than that. If you need more help or feedback, you can always reach out  to us, HNShah@gmail.com, Steli@Close.io. This is it for this set, so we’ll hear you very  soon.

Hiten Shah:

Yeah, bring us the math. See you.

Steli Efti:

Bye bye.