431: Should You Run a Ton of Experiments Early on as a Founder?

[0:00:01]

Steli Efti: Hey everybody, this is Steli Efti.

[0:00:03]

Hiten Shah: And this is Hiten Shah.

[0:00:05]

Steli Efti: And today on The Startup Chat, we’re going to talk about how to run marketing experiments, especially when you are at the very beginning or super small. So here’s the reason why I want to talk about this with you, Hiten Shah. I was emailing back and forth with a single founder that is just building NBP, and he’s trying to start his start-up, and one challenge that he described to me was the challenge that he was trying to be super data-driven, and super kind of experimentation-driven, and was saying that he was trying all these different experiments, but it would take like a week or two to collect enough data. And then a lot of these experiments that he’s tried haven’t yet worked, and he was asking me if… He was basically getting to a point where he was doubting that this will work well, and saying to me, “Is it the case that maybe just really well-funded start-ups with lots of people can afford to run all kinds of crazy AB tests and all kinds of different growth experiments, because most experiments won’t work, and when you are a one-person team, with very little money, you don’t have infinite time to just keep experimenting with great, crazy ideas that most of them don’t work. You might just have to do bread and butter things that are easier to succeed with, even if they are not necessarily as data-driven or as experimental in nature.” So, basically asking, are experiments only for big companies, or for start-ups with more resources, or can a single person be super experiment-driven and try to grow their start-up from day one, based on running different tests, and driving different experiments? And I could not think of anybody better than to state this problem to and get probably a funny reaction that’s like, “This is the totally wrong way of thinking about life”. That’s the problem that he brought up to me, and I was super curious to hear your kind of first reaction on that.

[0:02:22]

Hiten Shah: Yeah, I think the killer here, in terms of mentality, is comparison. I think that’s part of the problem. To me, marketing is just about driving traffic with a goal in mind.

[0:02:41]

Steli Efti: Right.

[0:02:43]

Hiten Shah: Well, what’s your goal? Well, your goal is to get traffic. Figure out how you’re going to find the right visitors that are going to sign up for your product, right? I don’t think it’s more complicated than that. But yet I think people tend to make it more complicated than that. And the comparison isn’t helpful. It’s not helpful to be like, “Yo, these big companies can spend all this money I can’t.” That’s, I think, where the problem starts and ends, is when people start thinking like that. And that tends to be where I see a lot of people get in a lot of trouble. Because they start really comparing themselves to larger companies instead of just worrying about, “Well, how do I get traffic, what do I need to do?”

[0:03:28]

Steli Efti: Yeah.

[0:03:28]

Hiten Shah: Because that’s what matters. Nothing else matters.

[0:03:31]

Steli Efti: The other thing though is, I think in this particular case, like I asked him what experiments he’s actually done, and it’s like… I think it might be more of an issue not of the experimentation, but of the trying way too complicated and sophisticated things way too early.

[0:03:51]

Hiten Shah: Mm-hmm (affirmative).

[0:03:51]

Steli Efti: Right so, he was describing to me, “Oh, I thought I’ll try an email campaign,” an outbound email campaign, so I came up with this super, to me, over-engineered, over-complex way of scraping some email addresses, and then he wrote four totally different emails, with completely different subject lines, and he set up this email sequence and he spent like two weeks setting all this stuff up, and then he started sending these emails, and realized that most people didn’t open any of his subject lines. Right? So, now you have a case of you spend, I don’t know, four or five weeks of this outbound email experiment and he has like, not many results to show for, so he’s at a loss of what to do now. He’s, “Well, I ran this experiment, I thought would generate a clear winner and a clear loser and based on that data I would know what to do next to scale this campaign and improve and optimize it. But I’ve done all this work and none of these emails really have worked, so now I sit here. I have five weeks of work and have nothing to show for it. And nothing I have learned.” That was one of his complaints, “I did all this.” And to me when I briefly looked over that outbound email campaign is one experiment that he did. He did some Facebook ad stuff that I didn’t have a chance to really look into, but it felt to me that he over engineered and made something way to complex and already had four different versions of things but didn’t get that basics right, so it was sending emails to people who didn’t want to hear about it or sending emails to people that don’t open emails, or just had really bad subject lines to begin with. So if you have five different subject lines that are all bad, it doesn’t really matter. You’re not gonna learn or gain any real insights by sending to five different ones. I had the feeling that, he bought into this idea of being analytical as a founder and using data and running experiments until you uncover, like, every time you run an experiment you’ll learn something and you learn what’s better than before and I think that constantly pushes you forward with great momentum. He’s been doing this for two or three months and working really, really hard, and feels like he’s not really getting anywhere. And to me it seemed like that was kind of a big part of his challenge and problem.

[0:06:25]

Hiten Shah: Yeah, I mean, look, if you’re gonna be really involved, complicated things early on, and maybe you don’t have experience doing those things, maybe you shouldn’t do those things. Try doing things that are much simpler, easier to pick up. Not because you’re a small company but because it just makes your life easier and it’s like [inaudible]. So early on you don’t necessarily know what’s gonna get you traffic, but you can look at what other people have done in your space to get traffic, and figure out what things you can do. And then start doing them. My process is always about… Okay this is my space, this is the customer I’m targeting. How do other people get traffic in this space? How have they done their marketing? And then I build from there. The reason I build from there and the reason I start there is because if other folks are getting traffic in this space then it’s likely that those are the kinds of channels and experiments that I can also run myself. That’s it.

[0:07:25]

Steli Efti: Yeah.

[0:07:26]

Hiten Shah: I mean, it sounds pretty bland and boring but at the end of the day, if you are looking at getting traffic, your best bet is to start with what exist and what’s out there and what you can kind of dissect and go from there. And what you can dissect is how other people are doing things. And so I don’t have an answer like, “Oh go and try these channels or these other channels,” but my biggest answer is just literally see what’s working for other people, make a comprehensive list of it. Do some homework, and then go figure out what you should do. Otherwise, you might be stuck like the person you were talking to is stuck in doing things that he thinks are right, but doesn’t really know if they’re gonna work.

[0:08:05]

Steli Efti: Yeah and we’ve talked about this, many, many times. You need to really take it one step at a time. This is actually impressive, I was surprised myself. In episode 30, so this was a while back, but still as relevant as ever, in episode 30 Hiten Shah and I talked about how to get your first 10 customers, right? I would suggest listening to that episode. I think it goes back to the basics of like not over complicating, not over engineering. It’s a small way of how we’ve taught ourselves as a start up community not to build the perfect and final product before we ship the very first product we’ve ever shipped. But to build minimal and viable products, right? And to learn and iterate. So same thing when it comes to customer acquisition. You don’t run 400 different experiments, if you are this massive acquisition team, with this incredible experience, and all this data and all this traffic and all that. If you approach it like that when you’re a single founder you’re just overwhelming yourself. You’re setting yourself up for failure. You can run experiments, but an experiment doesn’t have to 400 different variations. It doesn’t have to be that complex. And so trying to learn from the companies and start ups and founders and people that acquire and market to your customer base, or the type of customer you wanna acquire and trying to go slow and one step at a time and start with things that might not scale immediately, and then experiment with a few of these things but don’t over do it with just insane variations and variables, and setting up these super complicated systems, when you’re not even sure yet who your real customers are, when you’re not even sure you’re what your real product is. I think that really is that thing that a lot of founders stumble upon and stumble on. And so it’s back to the keep it simple stupid, mind frame hey? It’s not about experiments or not experiments?

[0:10:10]

Hiten Shah: Absolutely.

[0:10:10]

Steli Efti: Experiments are not just good or bad, it’s just about in the phase that you are in, the very beginning of your company, you need to keep it simple. You need to do very few things and you need to move very fast and learn based on these things. And we’ve talked so many times about like, customer intimacy, instead of just sending… Setting up some landing pages and paid ad campaigns and outbound emails, you know, maybe you just need to try to figure out a way to visit somebody that could be a customer, or meet up with a bunch of people and invite them for coffee, that could be customers. Or that are founders that have experience with acquiring the type of customers you want or, or building the type of product you want to build. And ask them some questions, show them your NBP. Get feedback and gain insights and in person even if it’s slow versus just setting up these scalable beautiful ad campaigns and outbound email campaigns and try to run all these crazy experiments and then find yourself weeks and weeks and weeks into all of this work with nothing to show for it. I think that that’s really at the core of what we been teaching for a long time now.

[0:11:18]

Hiten Shah: Yeah, you’ve got to be deliberate. Make sure that you’re doing things that actually matter.

[0:11:22]

Steli Efti: Boom, that’s it from us this episode. We will hear you very soon.

[0:11:25]

Hiten Shah: Happy-

[0:11:26]