The Unofficial Shopify Podcast A/B Testing: Is it a 'sack of money' button?
What's a good conversion rate? One that's better than last month.
But how did you get there?
Nick Disabato has built a career on research-focused A/B testing. Over the past year, he's helped Shopify Plus store KeySmart achieve extraordinary success. He joins us today to discuss that journey and how you can improve your store 5% monthly with his approach to split testing.
"Never forget: focusing on your customers brings you more customers. Are you focused on helping your customers, or are you focusing on what your coworkers want?"
—
Subscribe to The Unofficial Shopify Podcast on iTunes
Subscribe to The Unofficial Shopify Podcast on Stitcher
Subscribe to The Unofficial Shopify Podcast via RSS
Join The Unofficial Shopify Podcast Facebook Group
—
Learn:
- How testing makes you money
- What makes a successful test
- Testing's impact on design
- Why testing defangs your internal debates
Links:
Free Guide
I want to send you a sample chapter of Ecommerce Bootcamp, free.
Tell me where to send your sample at ecommerce-bootcamp.com
Transcript
Kurt: 00:06 Before we continue, I wanted to share a quick tip from our sponsor, Referral Candy. We'll find out what's working, then do more of that, so look at your top sales channels and then double down. It's the 80 20 rule and action. For many stores, word of mouth is a top channel, but how do you double down on the word of mouth? Check out Referral Candy - increase word of mouth sales by giving your store a refer a friend program. They're giving you guys 50 bucks to get started with it. Just go to Kurtelster.com/referralcandy to get started.
Kurt: 00:37 Hello and welcome to this episode of the Unofficial Shopify Podcast. I'm your host, Kurt Elster, agency owner, Ethercycle, author of the Ecommerce Bootcamp, and a lot of other things. Find out more on Kurtelster.com. That's my podcast radio voice. At this point, I can't say Unofficial Shopify Podcast any other way. I apologize for that. Joining me today is a wonderful gentleman who, through a mastermind group that he started, has changed my life and a lot of inspiration to me and is also an quite the interesting character who's been on the show before. Please welcome Nick Disabato.
Nick: 01:11 Hi there. How's it going? Really happy to be here.
Kurt: 01:15 So Nick, the last time you were here, it was a good episode. I enjoyed it. I like talking to you, but for you, you had a great outcome from it. You landed your favorite client.
Nick: 01:27 No offense to my other clients. All of my clients are really my favorite client. I landed a fantastic client. They are a lifestyle, everyday carry brand called Keysmart. If you go to getkeysmart.com - they are wonderful. It's essentially like a multifunction tool for your keys and it makes your keys a little bit more organized, has a few extra tools and all these other things. Just to talk about what I do for a living - I run A/b tests for e-commerce and SaaS businesses. Those in the audience who don't know what A/B tests are - you have a change, you want to vet the economic impact of it, you test it against the control and you determine what the actual lift is. You've come up with ideas through research, you end up making that a core part of your design process so that you're not making bad decisions that could potentially hurt your business and you're more carefully and scientifically vetting what could actually convert better. Keysmart's revenue has gone up. I forget the last calculation I had done. I think it's something like 75 percent as a direct result of my A/B testing over the past nine months. Nevermind the fact that they have also been growing significantly as a company. So that helps as a force multiplier, right? Like they're getting more traffic, they're getting more sales, and then people who come in are more likely to convert that last bit is because of my work.
Kurt: 03:02 On Shopify Plus, they have a really cool custom theme. I've done a few few modifications to it for them. There's like a lot of brands you hear talked about on Shopify that are very popular and you don't hear Keysmart talked about that often, which I always find strange because it's a cool product, but it also is quantifiably one of the most successful stores on Shopify.
Nick: 03:25 I even mentioned to Andy, who's one of the people there, that we should just say we have like thousands of happy customers. Then he said we have millions of happy customers. I'm like, great.
Kurt: 03:37 I was talking to Andy, who has been on the show before to say, wow, your facebook campaigns and your marketing... Andy's the one who did this stuff and it's phenomenal. I've never seen anything this successful. And he said, yeah, we could scale it, but we can't ship fast enough. That's your problem?! Your bottleneck is because you literally can't get the product out the door quick enough. That's nuts. I've never heard anyone say that
Nick: 04:05 That's a really good problem to have. I like when I can cause problems for my clients.
Kurt: 04:13 That's what you did. How did that happen? What is the test you ran? How did you go about it? How did you know what tests to run? Cause I know with a/b testing, everyone thinks it'll help them pick the right button color. And it's not that at all. It's quite a bit more complicated than that.
Nick: 04:32 Yeah, everybody wants to know where they should start with testing. When they ask me that, it's as if they want the one weird tip that causes the revenue to go up by 75 percent. That's not at all true. I'm so sorry. What I do is research what your customers are doing and then come up with informed guesses as to what these tests may be. Button colors generally don't work. Headlines work. If you have a clear idea of what kind of headline you should be writing, there is no such thing as just writing a headline for the sake of it being "more persuasive." So what I'm doing is going into Google Analytics and figuring out if mobile is converting dismally. OK, why? Well, the page time is taking really long time to load.
Nick: 05:20 Well, that kind of sucks. Why is the page to taking a long time to load? Oh, you have a one megabyte product image on your page and you never bothered compressing it and it loads great on my comcast for business connection, but then I go into chrome and simulate a 3DG connection that's dropping occasionally and the page takes 38 seconds to load. You're not closing a sale. That was one example of research. What I did was go into one of the product pages and then extensively compress the product image and it ended up being like a 38K product image. It looked a little granular. Whatever. It's on your cell phone, you're on a train. You don't know what it actually looks like, and it's probably smaller than the actual product in person, but I ended up converting something like 11 percent higher because way more people were able to load the page effectively and make a purchasing decision.
Nick: 06:17 They don't care how compressed your product image is or what CDN is serving it. That was the most basic mobile optimization thing. But I went in, said this isn't working, let's fix it. It's leaking money. That wasn't a headline, it wasn't a button color, it was something that should have probably happened at the beginning of this site being built out, but nobody caught it. So that's one thing. Another thing that I run is heat maps on your site. I determine where people are clicking, how long down the page people are scrolling, that sort of stuff. One thing that I find very frequently in Shopify is that they keep the same navigation on the same template from page to page. So you end up having like the full blown navigation and all these things all over every page of the site. And that includes your shopping cart and your checkout pages.
Nick: 07:13 That sucks for a variety of reasons. This is a rare moment where I'm going to recommend something pretty fervently and say it will probably convert better. When I say probably, I mean it's likely to. Don't blame me if it doesn't. You have to test it, but try removing those links in your header navigation. When people get to the shopping cart page - Amazon does it, Ebay does it, and it works extremely well for keeping people focused on conversion. They're not just like, oh, shiny. And then go somewhere else when they're just about to pay you. The last thing people want to do at any point in the transaction is fill out a form, but you have to make them do it and you'll have to make them do it at the last step. So I strongly recommend doing that.
Nick: 07:59 We have most importantly over the past nine months crafted a process and an internal culture around constantly checking our own beliefs around things. I think that's been the biggest outcome. We have a part time developer on staff right now who is constantly making changes to try and optimize stuff from a programming and technical debt perspective, which allow us to run tests considerably faster. Ideally, you always want a test to be running as much as humanly possible. You want there to be kind of consistent tempo around it so you want to be building the next test while a given test is running. So, we have a Trello board for vetting test ideas and researching them and we move things along on this Trello board and when we get to the point where we need to be building it, then I coordinate with the developer to build it. If you don't have a developer, one thing I would recommend installing as few plugins as humanly possible in your Shopify store, and I know that sounds so cringe worthy because plugins are a huge value add to Shopify, but they add a lot of code dependencies and craft that might actually bite you later on. I'm not saying this about Keysmart necessarily, but I have seen it enough.
Kurt: 09:17 It's true of any store that installs into several plugins; even one could start adding these bizarre dependencies. And then to your earlier point about performance optimization, there are two things that generally cause those performance slowdowns. One is the giant image like you described. It's very common because people want their image to look the best. So they save it out in the highest possible size that causes these bloated load times. Plugins and Apps - each time you add one that starts adding code dependencies. You'll see sites that load jquery like four and five times because of these apps aren't paying attention to each other and you've installed something and installed it. But yeah, it's a little bit of a rabbit hole there.
Nick: 09:56 Yeah, absolutely. So, you know, that's definitely something that can weigh down your site and keep you from being able to make changes and deploy. And it's so funny, it's this combination of like the full facebook move fast and break things mentality, but you're doing it in a way that isn't so fast. We need to research it. So you're making a lot of changes, but a lot of it is like almost infrastructural where you're figuring out, OK, well how am I creating a separate product so that I can make changes to that and shunt people there as a variant page. How can I create variations on Shopify's end using if/then logic. There's no in built framework for this. So you're running a test maybe on maybe one product page that has if/then logic, if you're particularly sophisticated, or you're just running two skews and hoping to God that the inventory works out. I've definitely encountered both of those situations. If you're in a position like Keysmart, you cannot afford to have two different skews and hope that the inventory works out for reasons you had just mentioned, Kurt, right? It can be really difficult there.
Kurt: 11:10 So going back, you had mentioned heat maps. Two questions. What is good heat mapping software and what am I supposed to get out of heat mapping?
Kurt: 11:24 I'm going to start with the second one first. You're supposed to understand where people are interacting with the page and where people aren't interacting with the page. Then you figure out how that squares up with your business goals. I've written a huge, huge, deep dive on heat maps recently that actually goes through a couple of example ones and then says, OK, well people aren't clicking on our primary call to action and why is that happening? Well this area is lighting up like a Christmas tree. Why are they going? And you just have to ask yourself, why are they going, how can I make them do something else? Because they're not going to be cajoled into actually doing the thing that you want. You have to investigate their motivations, right?
Nick: 12:07 Why is the page persuading people to go to the things that I don't want them to go to? Then you start to come up with some speculations about it and you'd say, well, OK, well maybe they're not ready to buy. Maybe it's a high involvement product, maybe it's not our flagship product, but it's the cheapest product. Maybe a masthead - we've never bothered swapping out the masthead image. Maybe it's just the first place that people go after viewing a facebook ad. And so we haven't like actually segmented this sensibly enough. The thing you're trying to figure out about heat maps is like real world customer behavior, right? Good tools for it - for most of my Shopify clients, I already have heat maps that I get out of my a/b testing framework, which is visual website optimizer.
Nick: 12:54 You could go to vwo.com. It's a little bit overkill if you're not actually running a/b tests yet. In that case, I would recommend hotjar.com. They're like $29 a month. It's just comically cheap and you get heat maps, scroll maps, you get to see people's cursor and finger as they go around the screen. It's amazing and always really compelling evidence for a client. Usually when you show heat maps to clients, they feel like they're staring directly into the matrix because they've never actually seen the real world behavior in that visual way before. They go a little feral over it, which is great. Right. But heat maps and Google analytics are only two of the things that I do. I also actually get paying customers on the phone and talk to them for an hour about why, what competitors they vetted in, why they chose to buy it at this point in time, what motivated them, whether they're using it now, and what their problems were beforehand.
Nick: 14:03 I actually ran an annual survey for Keysmart in particular recently and it was asking about like other every day carry things. That's going to shape a lot of the other products that we're going to be putting out later. So there's a lot of like other strategic things that you can be doing; it's not just about vetting the impact of the design decisions, but you end up like drilling down to the business needs in a lot of ways and saying, OK, well what does this business stands for? Is it selling a bunch of this widget to people or is it providing a broader ecosystem around the thing that we care about? Neither bad answers, right? You just need to know what the answer is so you don't go down the wrong rabbit hole.
Kurt: 14:39 This sounds a lot like a lot less like traditional a/b testing in more business and user research which are incredibly valuable things. But is it a/b testing?
Nick: 14:59 Well I say it's research-driven a/b testing. A/b testing is a tool that you use in the service of optimization and it is usually the last step if you're getting a certain number of sales. If you're on Shopify Plus and you're listening to this, you probably get enough sales for a/b testing. If you get 500,000, that's probably the minimum, especially if you have one flagship product and all your traffic is going in there. You should not be stabbing in the dark on your design decisions. You should not be arguing internally about your design decisions and wheel spinning and then saying, well, a/b tests our way out of the hole. That is not a good strategy for making changes to your site. That is how you end up getting a 12.5 percent success rate on A/b tests industry wide. And that is true, right? But if you research stuff and just say people aren't clicking here; even something that basic ups the success rate to around 58-59% in Draft's case. That's tests that are generating revenue, not mailing list sign ups, not people are engaging with the page more. No, no, no. Screw those things. What matters is that you are increasing revenue, decreasing costs, or decreasing risks to the business. And in A/b testing, you're getting at least two of those things every single time.
Nick: 16:32 You have to end up backing it up with research. It's absolutely essential. If people say, I want to cut the research and I just want you to run a/b tests for me, I'd probably nope out of the project.
Kurt: 16:44 Knowing you, I can assure you that's what you would do at that point. You're just shooting in the dark.
Nick: 16:55 Yeah, honestly I charge you. I have probably a moral obligation to not take the project at that point because I would take early five figures of your money, do a bunch of research via Marionette for you, not get good business results, and you would waste money on me. And then we would part ways and everyone would feel frustrated. And you would think that a/b testing writ large as a failure. My goal in my career is to make sure that people understand that design decisions have an economic impact. I'm doing a tremendously bad disservice to the cause of design if I would take a project that did not actually have research as a component for it.
Kurt: 17:32 I don't have a good follow-up question.
Kurt: 17:42 Give me a good next question. I've got nothing.
Nick: 17:54 This is a big mindset thing. Like it's, it's something people are used to design decisions by debate, right? There used to be around ideas.
Kurt: 18:06 You know, even getting hired as a designer, you go through this constant back and forth with clients and that's why you have to back up a lot of design decisions. Saying, I didn't just pick that because it looked pretty, here's the reasoning behind it. I've cited my sources. Even then you're going to get push-back. The person's going to say, well, my dog doesn't like blue. So you have to change those kinds of things.
Nick: 18:28 Yeah. And I actually run into the study testing clients, like I've had one recently, a keysmart were lovely clients. They are insanely brilliant. They're a wonderful team and there is a forgivable foible at play here where we have the Trello board where we're suggesting different design decisions and all these things and we'll start batting it around. And in the critique process, most good designers, they have what's called the yes. No, yes. Other places called the Shit Sandwich. And uh, where you say, I love this idea, I think that we might need to change it in a little bit this way or what's your thinking on that? Or something like that. And then you end up with another yes. Like, uh, again, I think that this is really cool. I just wanted to know what's going on here. So that is classic critique, active listening technique, non-violent communication that allows people to not feel threatened or imposed upon when you're proposing something.
Kurt: 19:20 Right. Um, which is great, right? Um, it's super useful and what they see is, that's a great idea. And they're like, great, I just shipped it and I'm like, well no, this is a board for testing ideas that have to be tested and researched and there's gotta be a process. And so I ended up having to like spell out, here's what happens when a t how, here's how test ideas get on fire, hose them on, here's what happens when they get on, they need to go through this process, not only to make me feel good about having this actually go the way we wanted to, but also to kind of expose it to the harsh light of day, right? Like we need to make sure that not only is it a good idea to us, but it's actually a revenue generating idea for the business.
Kurt: 20:07 And that involves research that involves spending a little more time actually thinking about the ramifications of the decision that involves squaring it up against all the other decisions that we've put together in the past. Right? Like the more tests we run, the more likely it is we're going to continue coming up with decisions that don't work for us or that we've already tried. And some other form and we want to make sure we're not spinning our wheels on this. Um, so I had to go and sit down and say, you know, you want to do ab testing like you hired me. That seems obvious enough, but your still thinking in a way that is like the socratic inquiry design decisions that, that everybody does and I get why you do it. It's because you have like 10 years background in this industry and that's all you've known.
Kurt: 20:57 And then I asked to come in and be the fun ruiner right? I'm really good at ruining people's fun. I'm really good at it. Um, and I don't like having to come in and be the fun ruiner I like it when people agree broadly with the concept of Ab testing and then figure out the execution behind it. Right? And there's a lot of like, psychological impact that too, you know, like when a test fails and you have to say, well we should keep testing. You don't look good politically by doing that as a consultant or as a worker or anything. If you're the champion of the project, you look terrible to your boss and the best clients are ones where we'll spend like three months planning a task and putting together this giant, ambitious reworking. It fails miserably and they're like, it's OK.
Kurt: 21:47 I saved you from wasting more time on what would be a boondoggle of a project. But if you let, you know, if you get emotionally invested in it and you have people you know, fighting for, um, you know, they're, they're designed candidate for just to save face. It doesn't work. And that's where it split testing gives everyone this easy out. You know, when I argue with my children clear, like they just want an out, but they don't want to have to say, well, I was wrong. They'll always take the out if you give it to them. And I think that's um, as a tool for ending Itar internal debates, split testing is wonderful and you know, in your own language you say, well, it defines them.
Kurt: 22:26 Yeah, absolutely. I mean, it, it, if you have a process for considering design decisions, defangs them. And it also removes what in Ux parlance, it's called the hippo or the highest paid person's opinion. I love it. I relish it. Nothing more than when the like high school marketing intern comes up with a testable design decision that bumps revenue by 15 percent. And I just know about the CEO is design decisions. It's like one of my favorite things. It's so satisfying. I'm like, you know John over here, I'm actually about 15 percent more revenue for the business and we're paying him barely minimum wage credit.
Kurt: 23:07 No one wants. No one's gonna argue with it because who doesn't? If you're, if you're the business owner, if you're the employee, you're goal is roy in Split testing gives you this beautiful framework to do that. Right?
Kurt: 23:21 Roi Is also measurable in decreasing costs. Right? And I can come in. The most classic example for Ab testing is like this was a disastrous thing and it lost 11 percent revenue and now we're not rolling it out to everyone. So we avoided a bullet, right? That's the most classic one. But I tell you, I've run shopify Ab tests that pair to back the number of skews that we were offering and ended up decreasing overall like cost of goods and cost of manufacturing by like 25, 30 percent because it turned out nobody gave a crap about all the ancillary products that we were offering and offering it in one color, one size actually worked better for us.
Kurt: 24:04 You're eliminating, um, in many cases you're eliminating choice paralysis. People don't have to consider the thing they probably don't want. And even if it was like one in five, you still have a Pareto's principle, the 80 20 rule. And you're exploiting that by offering fewer products. People like storage often act like I'm crazy when I suggest that. I'm like, have you considered offering fewer products? Here are fewer options. So they're like, what? No more products means more money and it isn't the case. Not always.
Kurt: 24:33 Yeah. And then you can cite a bunch of consumer research around it, but like shopify store owners have printers in general, they have a habit of like they made a successful product and now let's make it green. Or let's make it slightly larger or let's put Swarovski crystals on it or something, you know, and, and they, they get antsy because there's this constant process of reinvention and it might juice the numbers temporarily because you're getting a little bit more engagement from like collectors or something like that. And that could work in the long-term if you're a brand like field notes and you've released something new every three months and you can run out of it really quickly, but most storefronts probably don't have that luxury or they're probably not creating goods that are amenable to doing that. And so I would, you know, removing products is one of those things and that's one where it's like maybe we are settling intubate, but like what's the monetary upside? And I asked them to like, you know, we got rid of these products. That's great. Like what's the upside for you? It was like, oh, probably we just produced manufacturing expenses by 25 percent. I'm like, peel jaw off the floor. Like, are you kidding me? Like oh, OK, fine.
Kurt: 25:52 Yeah. If I want to hire nick D, if I want to hire you to do my split testing, I know you actually run a business that's small by design. Um, so you can take out of the limited number of clients at any one time so I can hire you or,
Kurt: 26:12 or you can. Um, so there are, there are a few ways to, uh, enlist my services. So the easiest thing that you can do right now, if you go to ab testing manual [inaudible], I, uh, I'm writing a book and creating a video course around everything that you need to know about Ab testing for your store. Right? There are three different packages. One of them is just the book. If you just want to know tactics about how to run an ab test and research it. Another is the video that talks about all the strategy, like the things that we were just talking about around, like dealing with disappointment around Ab tests, dealing with the psychological impact of it, the mindset shift needed in an organization. And then the third thing, which is obviously my favorite, is I come in and do a giant tear down of your site and you get the video course and everything else and it's like a hour long video tear down, like I actually go through on screencasts and pick everything apart and offer a ton of testable ideas.
Kurt: 27:11 I also run heat maps and fine tune your google analytics install for you. So that's the deluxe wash if you want the really big package where I come in and run a b tests for you and dictate your strategy, um, that is probably going to be accepting new clients shortly before the holidays. Knock on wood. Um, I don't know when this episode runs, but um, I'm hoping to open up like one slot for a store owner probably end of November, ish. That might give us enough time to start ab testing and Ernest for the holidays. Um, it might get us enough time to get a plan going, but timeline depends heavily on like where your sites at and what you've gotten stalled, what your team looks like, that sort of thing. Um, but if you go to draft a dot and you, I spelled it all out, I'm probably your best option is grab a copy of the Ab testing manual, read through it, see if it makes sense as something that you should be doing for your business. You should really only be working on ab testing. You could work on optimization. Anyone can do that. Anyone can fix browser bugs or compress the images on their mobile pages. Um, but if you're running ab testing, you should probably have around 500 to 1000 transactions a month minimum. Ah, and that's not everyone who's listening to this, but it could be you someday. And maybe you'll think of me then.
Kurt: 28:35 So what's, we're coming to the riverside together. Want to see if you have any closing thoughts. What's one thing you wish every shopify store owner would do?
Kurt: 28:43 God, your biggest enemy is yourself. Most of the time when you think about the way that people are engaging with your product, you may be wrong and that is scary. You're the one who is the most informed about your product. Um, you think about it every day. It's your job, it's your life's work, um, but that's exactly why you shouldn't trust yourself on it, and the most important thing that you can do is listen to your customers and do what you can do, research it, whether or not you ab test anything after is that's up to you, but take the time to like run a survey. You can put together something on type form in 15 minutes and blasts it out to your mailing list and put it as a call out on your homepage for a week and then analyze what the impact is and it might teach you a lot.
Kurt: 29:29 What do you think about including a link? If you made a survey like that, it's easy to use type form [inaudible]. There's no reason you shouldn't have the data you get out of those things is unbelievably valuable. What do you think about including that in the, uh, order confirmation and the receipt?
Kurt: 29:42 The thing that I actually love doing kind of like life cycle emails too. So you get an order confirmation but, and actually deliver the product yet the order confirmation might be like jobs to be done type stuff like Clayton Christensen type questions. Like what led you to do this? What was the last thing you had an objection about before you went and purchased? Um, who else did you consider that sort of stuff. That's really great to get right at the height of purchase because it's also the height of enthusiasm. I love also sending a survey or sending a survey separately. Um, maybe like two or three months after they received the product. Like are you still using it? How did you enjoy it? Do you have any issues with it? Um, were there any problems with like assembly or something like that? Those are amazing. Amazing for figuring out.
Kurt: 30:29 Not just like how to actually talk about it on your website and get revenue generating changes, but maybe even for like how would you help with onboarding on the product, right? Like how you help with maintenance of the product or something like that. Like is it a leather wallet? Is it prone to cracking? Great. Sell a bottle of needs, foot oil on your site and get people to condition their wallets, pushed that a lot in a little card that you ship with the product, that sort of stuff. Um, it's, you know, optimization effects every part of the business.
Kurt: 31:00 Absolutely. Those are all great tips. Um, so what's one piece of information you'd like to correct about Ab testing?
Kurt: 31:07 Um, it is not a sack of money button. It is a tool, it is not a panacea for your job and it is a tool and it is one part of optimization and you have to be considering things more holistically than just this headline converts better.
Kurt: 31:24 Very good. And lastly, where can people go to learn more about you?
Kurt: 31:28 Draft Dot n u a n as in Nick, U as in the letter u university that um, and uh, yeah, if you want to learn more about the AB testing manual, ab testing manual, [inaudible] is your best option there.
Kurt: 31:45 Nick. Thank you. It's been my honor and pleasure.
Kurt: 31:48 A total honor. Thank you so much for having me back on.
Kurt: 31:50 So however this audio made it until find out more about an unofficial shopify podcast.com, and if you'd like to be notified whenever a new episode goes live, subscribe in Itunes, join our facebook group, unofficial shopify podcast insiders, or set up for my newsletter.
Speaker 5: 32:05 I'll shoot you an email whenever we post a new episode. Thanks everybody and we'll be back in. Our program was produced today by Paul Reeder. The unofficial shopify podcast is distributed by either cycle, LLC will be back next week with more value bombs for shopify store owners. If you're looking for more high quality and actionable advice on learning the business of e commerce, join thousands of other shopify store owners on our totally free newsletter at ecommerce bootcamp. That's e-commerce hyphen bootcamp.