Why AI Analysts Often Seem Confident but Wrong

Would you like access to our advanced agency training for FREE? https://www.agencymastery360.com/training

Are you feeding your data into AI and assuming the insights it gives you are accurate? What if those confident-sounding answers are quietly steering you in the wrong direction?

More agency owners are turning to AI to analyze and interpret performance data, and for good reason. Used correctly, it can save massive amounts of time and move teams beyond using AI to crank out blog posts, ads, or emails faster. But when it comes to attribution, performance analysis, and real decision-making, AI has a dangerous flaw: it's often wrong with absolute confidence.

Today's featured guest understands where most agencies go wrong with AI-driven data analysis. He'll break down why large language models frequently misinterpret marketing data, how flawed inputs and assumptions lead to misleading insights, and what it actually takes to get reliable answers from AI without burning budget or making bad strategic calls.

Scott Desgrosseilliers is the founder and CEO of Wicked Reports, a marketing attribution platform built specifically for e-commerce brands doing between $5M and $50M in annual revenue. Scott has spent years deep in attribution, analytics, and now AI, figuring out how to separate real signal from noise in an ecosystem where every platform claims the win.

He'll talk about how most platforms may be misleading you and the framework he uses to bring sanity back to attribution for serious e-commerce brands.

In this episode, we'll discuss:

• Why AI is sounds smart but gets marketing attribution wrong.

• Injecting intention into AI.

• The Five Forces framework to improve your AI data.

Subscribe

Apple | Spotify | iHeart Radio

Sponsors and Resources

E2M Solutions: Today's episode of the Smart Agency Masterclass is sponsored by E2M Solutions, a web design, and development agency that has provided white-label services for the past 10 years to agencies all over the world. Check out e2msolutions.com/smartagency and get 10% off for the first three months of service.

Why AI Sounds Smart But Gets Marketing Attribution Wrong

One of the biggest myths around AI is that it's inherently "smart." Scott shared that it took eight months for Wicked Reports to release their AI analyst, not because the tech wasn't powerful, but because it was too confident while being wrong.

AI models are designed to sound affirmative. Ask them a bad question, and they'll still give you a polished answer. If you ask ChatGPT if you should jump off a bridge, it'll say, "Yes, that's a great idea," unless you explicitly train it to be critical. That's a massive problem when you're dealing with revenue attribution and ad spend decisions.

Another major issue is that AI lacks native understanding of time, which is foundational to attribution. Clicks, impressions, tags, and conversions happen in sequence over days or weeks. Without heavy rules, coaching, and sanity checks layered in, AI can't naturally interpret cause and effect. Left alone, it simply fills in gaps, and those hallucinations can cost you real money.

Why Intention and Metrics Matter More Than the AI Tool

The first thing Scott's team had to "inject" into the AI was intention. Not all campaigns exist to do the same job. Prospecting, retargeting, direct response, and existing customer campaigns each have different goals and therefore require different scoreboards.

If you don't tell the AI what the intention is for each row of data, it will make assumptions. And those assumptions are usually wrong. The "North Star" metrics and leading indicators change depending on what you're trying to accomplish. A prospecting campaign shouldn't be judged the same way as an abandoned cart flow.

The second big issue is AI's obsession with ROAS. ROAS is easy to latch onto because it gets rewarded with "thumbs up" feedback, but it's often misleading. If two-thirds of your reported revenue comes from repeat customers via email or SMS, AI might tell you your ads are crushing it when they're not. Simply separating new customers from repeat customers already puts you ahead of 95% of advertisers.

The Five Forces Framework for Making Better Attribution Decisions

To solve these problems, Scott introduced his Five Forces Framework, (intention, expectation, action, outcome, and optimization) a methodology most agencies simply aren't using.

The first force is Intention, which defines both the scoreboard and the timeframe. New customer acquisition might need a 30–90 day window to show results, while an abandoned cart campaign can be evaluated in seven days. Without this context, teams panic too early and kill campaigns that haven't had time to work.

The second force is Expectation, which is all about alignment. Brand owners often look at Shopify, GA4, Meta, Google, Klaviyo, and SMS dashboards—all showing different numbers. Without agreeing on a single version of truth, clients freak out and shut down top-of-funnel campaigns after five days because the data "doesn't look good yet." Setting expectations isn't a one-time conversation; it has to be reinforced constantly.

Reducing Drama: Use "Scale, Chill, and Kill" to Guide Ad Spend

The third force is Action, which includes launching the campaign but only after defining clear boundaries. Scott recommends setting "Scale, Chill, and Kill" zones before you spend a dollar.

For example, if your acceptable new customer acquisition cost is $50–$70, that's your Chill zone. Below $50? Scale it. Above $70? Kill it. These predefined rules remove emotion, reduce second-guessing, and dramatically lower what Scott calls "psychic stress" inside agencies and brands.

Once campaigns run, the fourth force—Outcome—is simply measuring performance against those zones. Did it scale, chill, or die?

Optimization Is More Than Creative Tweaks

Most agencies obsess over creative, constantly swapping headlines, images, and copy. For Scott, optimization should be more structured. At his agency, they use a decision log to rank potential actions by impact, focusing on whether the problem is the offer, the creative, the traffic, or the budget.

But Scott added a fourth optimization factor most teams miss: signaling. If you don't send the right signals back to ad platforms, your optimization efforts don't matter. Meta, in particular, is very good at claiming credit for conversions it didn't truly drive and if it sees quick conversions, it will chase more of those, even if they're just repeat customers.

Training Ad Platforms to Optimize for What Actually Matters

To fix this, Scott recommends creating separate events in Meta's Events Manager for new customer purchases versus repeat purchases. That way, ad sets can optimize specifically for the outcome you want.

If you're closing existing customers through email or SMS, you don't want Meta learning from those conversions. But when a new customer buys, Meta gets a clean signal and starts finding more people like them. Scott noted that when creative and offer are solid, sharpening signals alone can dramatically reduce acquisition costs within a month.

You can even go deeper by signaling based on SKU types, allowing platforms to optimize toward higher-quality or more strategic purchases—not just any conversion they can grab credit for.

Do You Want to Transform Your Agency from a Liability to an Asset?

Looking to dig deeper into your agency's potential? Check out our Agency Blueprint. Designed for agency owners like you, our Agency Blueprint helps you uncover growth opportunities, tackle obstacles, and craft a customized blueprint for your agency's success.