Emphasizing the Importance of Holdout Testing in Marketing Strategies

What’s up everyone, today I have the pleasure of sitting down with Pranav Piyush, Co-Founder and CEO at Paramark.

Summary: Pranav guides us out of the labyrinth of multi-touch attribution under the clear sky of incrementality and causality, urging marketers to focus on whether their efforts genuinely drive sales that wouldn’t happen otherwise. Early-stage startups can benefit by prioritizing simple methods like geo-based testing over complex attribution models, allowing intuition to guide resourceful experimentation. By understanding the underlying motivations and true causality behind customer actions, marketers can craft campaigns that resonate deeply and drive real results. As businesses grow, balancing intuition with structured analytics becomes crucial. Holdout tests and marketing mix modeling provide actionable insights, ensuring strategies remain effective in a competitive landscape. This approach transforms marketing from a cost into an investment in sustainable growth, making each dollar count.

About Pranav

• Pranav started his career at well known brands like PayPal and Dropbox
• He co-founded Padlet, the popular collaboration app to make school less boring
• He’s former Head of Growth at Magento and Pilot.com before becoming VP of Marketing at BILL
• He’s also a Reforge Instructor for a new marketing measurement course
• And in March of last year, Pranav co-founded Paramark to help marketers measure and forecast the impact of their investments

What’s More Valuable? Analytical Skills or Creative Taste?

Marketing's creative nature often gets overshadowed by the obsession with data. Recently, HubSpot’s co-founder Brian Halligan suggested that marketers with good taste are undervalued compared to those with analytical skills. Pranav agrees, arguing that creativity now drives the most significant impact in marketing. We often question the overuse of the term "data-driven" in marketing, suggesting a shift towards being more "creatively driven." Pranav responds, arguing that data-driven and data-informed are all kind of bullshit. Relying solely on being "data-informed" is not sufficient. He emphasizes that without the ability to discern the success of a creative idea through data, creativity alone falls short.

Marketers face the challenge of making memorable impressions on people they've never met, and this requires innovation and creativity. While data is essential, Pranav notes that many marketers don't truly understand the depth of analytical skills. True data literacy involves grasping complex concepts like correlation and causation, which are often missing in marketers' education.

Pranav points out that the dichotomy between creativity and analytics is overly simplistic. Marketers need to integrate both skills. This blend is crucial not only in marketing but in other business functions like product development. He uses the example of launching a feature and gauging its success. If only 10% of the customer base uses it, understanding the broader impact on adoption, revenue, and retention is essential.

Despite recognizing the importance of analytical skills, Pranav emphasizes that good taste in marketing offers a unique advantage. Creativity leads to building compelling campaigns that resonate more profoundly with audiences. This insight suggests that while data provides valuable insights, it is creativity that ultimately distinguishes successful marketing efforts. Pranav further highlights the importance of rigorous testing and measurement. A successful feature or campaign isn't just about positive feedback; it needs to contribute to tangible business outcomes, such as increased revenue or cost savings. Without proper measurement, the value of creative initiatives remains unclear.

Key takeaway: To truly excel in marketing, you need to embrace a harmonious balance between analytical skills and creative taste. This means honing your ability to interpret data while also nurturing your creative instincts to craft memorable campaigns. Instead of relying solely on data or creativity, focus on integrating these skills. Use data to measure the success of your creative ideas, ensuring they lead to meaningful business outcomes like increased revenue or customer retention. By blending data literacy with creative insight, you'll develop campaigns that resonate deeply and drive tangible results.

Understanding Incrementality in Marketing

We often hear marketers claiming they understand ROI and reporting, yet the concept of incrementality often eludes them. Pranav sheds light on this by differentiating between attribution and incrementality. Attribution, as he explains, is rooted in the idea of cause and effect. However, its usage has been diluted over time, losing its original meaning.

Pranav appreciates our provided definition of incrementality: business results from marketing campaigns or channels that wouldn’t have occurred otherwise. He elaborates that if a prospect would have purchased a product without the influence of marketing, then that marketing effort isn't incremental. Conversely, if a prospect's decision to buy is directly influenced by marketing, then that effort is incremental.

He emphasizes the importance of understanding incrementality beyond traditional marketing channels, especially in B2B contexts. This involves considering scaled sales channels, partner channels, and affiliate channels. The essence of incrementality lies in recognizing the true impact of marketing efforts on sales and other business outcomes.

Pranav's insights underscore the need for marketers to move beyond surface-level metrics and understand the deeper implications of their strategies. By focusing on incrementality, they can more accurately measure the effectiveness of their campaigns and make informed decisions that drive real business growth.

Key takeaway: Focus on incrementality to truly gauge your marketing impact. Instead of just relying on attribution metrics, assess whether your efforts genuinely drive sales that wouldn't have happened otherwise. By understanding and applying incrementality across all channels, you can refine your strategies and foster real business growth.

Unpacking Multi-Touch Attribution

Multi-touch attribution (MTA) often gets hailed as the holy grail of marketing measurement. Many believe it's essential to solve attribution by capturing all touchpoints. However, Pranav argues that the obsession with MTA overlooks fundamental issues, particularly around causality.

When discussing attribution, we need to understand cause and effect. Pranav illustrates this with a simple example: if someone clicks on a Google link and converts, did that click cause the conversion? Sometimes it does, but other times it doesn't. He emphasizes the need to ask, "What prompted the search in the first place?" Without knowing this, we aren't truly understanding causality. We're merely observing sequences of actions without grasping their underlying motivations.

Pranav criticizes the current approach to MTA, which often amounts to behavioral analytics. This method logs sequences like A led to B led to C, but it doesn't clarify if A caused B. This lack of clarity is compounded by pressures on marketing and analytics teams to produce quick results, pushing them towards convenient but superficial solutions.

The martech industry, according to Pranav, has profited from building easy, superficial tools rather than delving into the complex but necessary task of understanding true causality. He believes this approach must change for the industry to advance meaningfully. By focusing on more robust methodologies, marketers can gain genuine insights into the effectiveness of their campaigns.