Yeah, I can imagine that we're talking about maybe like hundreds of instructors in one variant and hundreds in the other. And that's not going to really hit significance in two weeks. So you have to be able to find other ways to judge the success of something. So what do you lean on? How do you look at like, we did this big change in the platform. Was it successful or not? It's

really hard. I'll be honest. Like it's usually you're just guessing. And so you're just trying to educate your guests as much as possible because, you know, like let's say that without having any data or any information at all, your accuracy on something is like 10%. That sounds about right. Maybe it's a little bit higher, but, you know, let's say it's 10%. Your goal is to try to get to 30% or 40%. So how do you do that? First, you have have deep customer insight. So you need to understand the psychology of your consumer so that your decisions are in line with what they want. And so you intuitively know what people want, right? Think about something like charisma. It's really hard when you're trying to make friends and influence people to have data to tell you whether or not you're actually influencing the person in front of you, right? Because there's only five people in front of you at a given time or even one or two. And so you rely on your intuition. So you need to do the same thing. You need to have that same level of intuition that a Dale Carnegie follower has, but you need to have that for your customer. And so that intuition is super important. The second thing is that you need to try to get as much data, both quantitative and qualitative as possible. So that's doing interviews before and after a product is launched that is closely following the behavior and taking signal from the behavior, even if it isn't statistically significant. And what that means is that you're not just looking at, hey, we shipped this and 50% of people converted and our conversion rate used to be 40%, you're looking at 50% of the people converted and they mostly converted, they were mostly these types of users. And these types of users are our core market or actually 50% converted, but none of them paid, that kind of thing that converted to give the email, but maybe they didn't pay. And so there's a lot of different nuances to looking at the data you have less data, but you still need to look at the data. And then the final thing is, I do believe it's valuable to understand industry trends and just learn from experts. I think I'm hard pressed to find anyone who's successful in startups and tech and business. I'm sure that they exist, but who don't follow the industry trends of what's going on in their industry, at least, and oftentimes across industries. And that's through Twitter and LinkedIn. That's through podcasts like yours. That's through going to conferences. And it's also through literally your competitors and your other your favorite products and analyzing them and that gives you a uh an instinct and it hones your instinct uh without having to make the mistakes yourself right because you get to see thousands of experiments that are happening in real time all across the internet um you can sort of witness some of those and learn from them. So those are some ways to help make these decisions. But again, you also just have to accept you're not going to get above 30 or 40% accuracy. So that's fine. You can still have massive wins with 30 to 40% accuracy. You just have to have lots of at-bats.

I'm having flashbacks to my enterprise B2B SaaS days where really it just mattered like what your top 10 customers thought about any feature launch. Is that kind of like what you think about for like your big platform rollouts is I want to reach out to my top 10 instructors and understand what they think about it?

Yeah, it changes over time. So, yeah, when we started, it was really just the top 10 instructors, and we pretty much built the product for them. But as I told you earlier, our top instructors ended up actually not being our long-term instructors. We ended up changing. and enterprise software companies also solve have this problem your top customers are actually like not the people you want to solve for you might actually want to solve for your mid-tier of customers or your lowest tier of customers it's really depends at maven we are focused on the sort of you know i'd say the like fifth percentile to the 50th percentile or maybe the 30th percentile. I'd say that's like five to 30 is kind of our range. So the top instructors need bespoke things that we try to solve in other ways. We just give them really good support and we're very responsive to their needs. But the fifth percentile to the 30th percentile instructors, those are the people that we really feel like are our core that we can scale and that we should be getting feedback from and so yeah we have a you know i'm on text relationship with like 15 to 20 of our instructors um i'm on email with 50 um you know and i'm on slack with another 10 or 20 of them so i i'm constantly in touch with them and i'm not even the person who's the most in touch with our instructors in the company. So if I as CEO, I'm not involved in what they think, you can imagine what our team is doing. Okay.

So in this world where it's hard to look back at your feature launches and say what's successful, how do you do planning? How do you make your strategic bets on the product side? You already walked through kind of your ICP being that fifth to 30th percentile instructor for the product, but how do you go further? How do you think about, okay, these are the needs I'm going to target this quarter or this half, and how do you structure that? I

mean, I would say that I'm highly gut and intuitive driven. So it's funny because people on my team would probably tell you I'm very process oriented and I am. But at the end of the day, the final decision I understand is very gut driven. So I follow a pretty strict process. You know, I think about, I'm basically always in the first like, you know, five to 10 years of a company trying to grow the business, right? Like that's all that matters is, uh, and you have a growth goal. Like our growth goal is usually like two X or three X. Um, and I'm trying to hit it and trying to work backwards from what would it take to get there? And so you make your product decisions based on what you think is going to be the most likely to, uh, help you do that. And you have some principles, you know, that help you. So one is you have to have an excellent product. So you can't just do a bunch of growth hacky things. If the instructors, in our case, our supply side, don't like using Maven, they're going to sour on us as a business and they're going to be likely to turn. And so there's a level of investment that we make almost every quarter on product excellence that does not have a direct ROI return on growth, but has an indirect ROI return on retention as well as we think on growth. The second principle is that we need to make sure that a significant amount of our resources are going towards growth bets that can two or three X the business on their own. And so we figure out what we think the best idea is. And one of the benefits we have is that we have a marketplace. So we have like literally hundreds of instructors that are experimenting with ways to grow their courses. And we take the ideas that we think are the most scalable and we try to build for them. And so our ideas often come from the experimentation that's happening by the supply side of our marketplace. And then finally, you have to accept at least at Maven, there's a natural cadence to Maven's growth. We do not see results in two weeks or a month. At Sprig, we saw results of something we did in one to two weeks, right? We might take a month to make sure that it was statistically significant, but we basically knew what was happening because buying food is a daily habit. And so you have this user base of your user base, which is a lot smaller, right? But they're buying every day. And as a result of the fact that they're buying every day, within two weeks, they're going to tell you whether or not the thing that they're buying is something they want to continue to buy or not. On Maven, the typical buyer is buying once a quarter or once every six months. And so how do you, how do you like make decisions when, and so what we do actually is we rotate, we make a set of decisions. We let them cook for six to, you know, three to six months and we rotate and work on a different part of the product area. And then we come back to the other one. And we found this sort of cadence where we allow ourselves to spend, you know, one month shipping a product, a new feature or product or whatever. And then we spend another month on a different part of the product and another month on a different part of the product. Then we go back and look at what happened over the last quarter and try to make decisions on what we're going to do going forward. Oh,

very cool. Can you give an example of like a big product change and kind of why you had to wait and then what it looked like three to six months later? Yeah,

it's a great question. We made a huge bet on hybrid courses about a year ago. So a year ago, almost before a year ago, almost all Maven courses were fully live. And it turns out that while we believe in cohort-based, we realized that there are a lot of students who want to be able to watch recordings and get more efficient information when they need it, and who want to be able to review information and sort of have that hybrid approach while still having some live content. So we saw a couple of instructors. We weren't even sure it was a good idea, but we had a couple of instructors who started to build hybrid courses anyways on Maven. And they'd be like clamoring for extra features for it. And we were like, you're a minority right now. You don't matter. But eventually they ended up showing us that we had, they had like better NPS often because they had both video based and live you know components their course which students preferred so but it took us about you know six to twelve months to come to conviction that we should build this probably another like two months to build it and we shipped a basic version of our hybrid course product. And we tried to ship it as fast as possible. So, you know, maybe it was more like five weeks or six weeks we shipped it. And which is a big change, right? This is a huge change to the underlying syllabus and things of the product offering because it used to be a calendar. And now it has to have, you know, recorded content that you can watch whenever you want. And so we made this change. And within a month or two, the adoption of this was not that great. I don't remember the exact numbers, but it was a significant minority. But it was still a minority of people of courses had started to adopt this feature. but we had conviction that those courses were uh what we were looking at were were those courses having success with the feature were students responding to this feature and they were so even though we had low adoption and there were a bunch of instructors who did not adopt it because they're teaching their course once a quarter so it's going to take three to six months for them to adopt it they might not adopt it in the first cohort they might adopt in the second cohort which is you know four to six months later over time we stuck with it and over the course of last year now all maven courses offer more value than they ever have before because not only do you have the live time with the instructor but the instructors are now finding it worthwhile to record content that adds and provides more richness to the material that is being taught live it allows you to do more discussions live and less you know rote lecturing because it's because it's in video format or it's in written format and it also allows the instructor to scale, which means that they are likely to build a better course. They can spend more time on the course and less time on teaching. So this has been a huge win for us, but it took about six to 12 months to actually get the results of it. And now we're super confident it was a great decision, but it took a while okay

i like that and how did you how do you structure like your various product areas because you mentioned you'd rotate away from hybrid course sounds like it's basically touching all the key instructor surface areas so what other part of the product can you move to yeah

i mean um so keep in mind that we don't need to structure things to, we don't need to be that structured because we only have, you know, 18 people. So as a result of the small team size, it's very easy for us to, you know, simply, you know, just intuitively rotate from one subject area to another. So areas that we might rotate to. One month we might work on email, and we might do a bunch of stuff on our email system, both in terms of instructor tooling, but also in terms of our own homegrown email system. We might work another month, we might work on discovery, and we'll try to improve the way that you can find and view courses on our marketplace. Another month, we might work on recommendations, which is an input to discovery and to email, but is not directly in those two camps, right? It's sort of a developer tool. That's not the right word. It's an infrastructure thing that allows you then, that enables other product offerings. So we would work on our course recommendation algorithm um another month we might work on lightning lessons and improving our free to paid funnel which is what lightning lessons is for uh so and then and then within the admin side of the product uh there is probably i think there's like eight to ten tabs. There's your admin, just your basic controls. There's the landing page editor and what it looks like to the end user. There's the syllabus, there's events, there's community, there's the people tab where you get to see the other people in your cohort. And there's a bunch more that I, there's your earnings, there's your analytics. So we're always rotating between all these things um and we're just picking whatever we feel like is going to make the biggest impact and going after it okay

got it and you mentioned like months do you guys think in months what is kind of the unit that your engineering team operates on are they operating on sprints are they doing that in a tool like linear how are you guys taking kind of these are the feature ideas all the way through to how we develop them yeah

this is a great example where i didn't fully answer the question and you helped me so thank you uh because i think this is the second part to that to that uh that statement that i made which is that every month we do a, you know, we do a planning cycle, basically. So we review what the results were last month, and then we plan for what the next month is. And we use a pod structure that changes. So sometimes we'll have three engineers on a pod and two on another. Sometimes we'll have three pods where we have, you know, two engineers, two engineers, and one engineer, or one and a half, because my co-founder is our sixth engineer, and he often, you know, he codes a fair bit as well. So I would say that every month, we do a retro on last month, and then we start to plan for the following month, and what the pod structure is going to look like, and what bets we're going to make and um i'm fairly uh grateful to have rish in our head of product who's extremely process oriented in in this way and sort of runs a a clear and transparent process in which everyone gets to share their input um but that the the final decisions get made by you know essentially mostly him. And occasionally I'll step in and sort of override him, but it's very rare. So he's pretty much making all of these decisions. Got

it. And what comes to like a monthly planning process? How developed does an idea need to be for you guys to prioritize it? Another

great question, because we actually do a funny thing. Maybe other people do this too, but we have scoping time and we have shipping time. So essentially, we are not always allocating resources to building things. We are often allocating resources to scoping things. designers spend two weeks scoping a project that we are not 100% sure we're going to actually build. And we'll have, you know, we'll allocate them to a pod that is essentially a scoping group of people that are trying to figure out, like, what are they going to do? What should we do about the discovery? Like, what could we do if we push this forward? And I think that's been really, really successful for us. Okay.

And then when do you come a feature do you typically do like a product review on the designs before they start coding or how do you kind of get involved as the founder who is always partly head of product yeah

um i mean every week we do a product review meeting where uh people sort of bring up the topic that they're working on that week. It's usually our designers or our product lead who are bringing things in. And we're looking at whatever they're building. It's not just me. It's also my co-founder, Shrayons. It's our head of supply. It's our head of demand. So it's the leadership team plus the two designers who, in our company at Maven, our designers are very much a sort of extension of the leadership team. So we view them as quite significant contributors to the product roadmap and a significant part of sort of creating what the product is going to look like. not just as narrow as designing the flows, but sometimes often thinking about the PRD level items, like is the product going to do X or is it going to do Y? And so the designers are also in that meeting. And so this meeting happens every week and we're reviewing everything all the time, basically. Also, we have a very good async culture so on top of that weekly meeting in between there are channels for each pod and those channels are relatively active they're not super active because i don't want them to be like 50 messages a day right i mean 50 posts a day of course you're going to have a lot of messages in the threads, but 50 unique posts, we don't want that. We want three to five or 10 maybe in a really heavy project period. But often those posts are looms sharing or they're links to Figma. And so you're able to look at projects along the way in between the Thursday to Thursday is when our product reviews are.