Episode 227: The Correlation masquerade: Crawling through the dungeon of martech architecture part 3/4
The boomerang effect, why attribution can't prove causation, holdout groups, the agent context graph, and building a causal memory layer.
This is the 404 Martech newsletter. 4 quotes to ponder, 0 chill for this thing, and 4 ideas to try at work, pulled from this week’s Humans of Martech conversation + fresh remote job postings you should check out.
Happy Tuesday folks!
The third floor of the dungeon of martech architecture is open!
The first 2 floors were about assembling raw data into a single source of truth and organizing it under a semantic layer. This floor is about something harder to see: telling an agent whether the thing it’s optimizing is actually the right thing to optimize.
Your warehouse full of historical data, mixed together with the context brain you’ve built, still can’t tell you why a campaign worked, or whether it worked for the reason the model assumes it did. And an agent that mistakes correlation for causation will scale that mistake faster than any human ever could.
Here’s the dungeon layout:
Part 1: 👑 CRM Gravity. We defeated the False Truth King and the Export Hydra, revealing the data warehouse as the real source of truth for modern GTM data.
Part 2: 🧙♀️ The Eye of Context. We learned why AI fails without shared meaning and why context engineering is the layer between data and agent authority.
Part 3: 🪃 The Correlation Masquerade (you are here). You’ll escape the correlation trap and build the causal memory layer that separates agents that optimize correctly from agents that confidently scale the wrong behavior.
Part 4: 🗻 The Dispatch Tower. You’ll tackle the governance chaos of 30 vendors all claiming authority, and confront the interface decision that most organizations already made without realizing it.
In part 3 today we’ll cover:
Why agentic AI optimizes for the wrong thing at scale
The boomerang effect that erodes revenue
Why attribution data can’t tell agents what caused a result
How bad signals masquerade as evidence
How to reduce exposure with holdout groups
How to build a causal memory layer with a context graph
Recommended martech tools/agencies
We only partner with products and agencies that are chosen and vetted by us. If you’re interested in partnering, hit me up.
🎨 Knak: Go from idea to on-brand email and landing pages in minutes, using AI where it actually matters.
📧 MoEngage: Customer engagement platform that executes cross-channel campaigns and automates personalized experiences based on behavior.
🦣 Mammoth Growth: Customer data agency that turns fragmented data into a unified foundation, unlocking sharper marketing insights and action.
🔁 GrowthLoop: The agentic, composable CDP that drives compound growth by uniting your cloud data + AI into one marketing engine.
4 quotes to ponder
“You can absolutely scale the wrong behavior if you let a generic agent optimize directly on historical correlations with unbounded authority.” - Jason Dobbs, Kumo.ai
Jason’s point is that the warehouse is a great context layer and a terrible decision-maker. It records what happened, and an agent that reads those correlations as instructions will efficiently scale a mistake. Prediction is an input to a decision, and it needs guardrails, approvals, and evidence before it ever becomes an action.
“You dropped the price 10% and sales went up 25% on Black Friday. Is that the impact of the campaign? The item that wasn’t on sale also went up 5%.” - Simon Lejeune, Wealthsimple
Simon makes incrementality the default response to any result shared on his team. The question agents never ask is what would have happened without the intervention. Most candidates he interviews think they understand it until he asks them to explain the first principle, and the explanation falls apart.
“I accidentally sent 19 million emails that were basically blank, and we got a massive open rate. If the AI optimizes for that, it’ll just send more blank emails.” - Jeff Lee, Calm
This is the correlation masquerade at its simplest. The proxy metric looked record-breaking and the actual experience was broken. An agent does not need bad context to make a bad decision here. It only needs permission to optimize the proxy.
“You snapshot everything you know about the customer, the intervention you tried, and the actual uplift, and you do that with every experiment. Then you can answer the what-if questions.” - Anthony Rotio, GrowthLoop
Anthony calls it the agent context graph, a record of causality data built one experiment at a time. It is the difference between an agent reading patterns and an agent reading evidence, and it doubles as a drift detector. Teams that started 2 years ago already have causal history, and you cannot create that retroactively.
0 chill for this thing
Martech experts everywhere agree, we have 0 chill for treating correlation as causation.
Obviously, correlation ≠ causation. We’ve all beaten that drum before. But it’s a drum worth hitting even harder with AI in the mix.
An agent reading the warehouse could find that high-LTV customers all viewed a certain product, so it makes that product the mandatory step in the welcome flow. The logic is reproducible and the data supports it and it’s pushed in a bunch of other spots.
But by default, we’re not asking the most important question: did the product cause the high LTV, or just come along for the ride?
The cost of not asking that question is the boomerang effect. The agent scales that pattern across every matching cohort efficiently and invisibly, and the metric you actually care about degrades even though your metrics look green. By the time it surfaces, you have done it to millions of people and wasted lots of cash.
We need to implement mechanisms so that we’re teaching the machine through experiments and a causal record.
4 things to try at work this week
Add a holdout group to every agent-driven campaign
Stop measuring agent campaigns against nothing. Hold back a percentage of the target audience that receives no intervention, then compare it to the group that did. That holdout is the baseline that tells you what would have happened anyway, and it becomes the first entry in your causal record. Make it a habit.
Cap agent scale at 10% until a human reviews the causal logic
Stop letting an agent roll a behavior out to the whole audience on its own from he get go. Set a hard rule that no agent scales any pattern beyond 10% of the audience until a person has reviewed the causal logic it assumed. It is the cheapest way to catch a boomerang before it reaches 5 million people, like an agent scaling a campaign full of blank emails because they had a record breaking open rate and a disastrous customer experience.
Make incrementality the default reply to any result
Stop accepting raw campaign revenue numbers as proof. Borrow Simon Lejeune’s meme from Wealthsimple and reply to every shared result with one question: what’s the actual impact? What would over happened if you didn’t run that campaign? Push for the incrementality breakdown until the team can separate what the campaign caused from what was going to happen regardless.
Stop assuming last year’s signal still holds
Measure new results against a continuously grounded baseline. Things are constantly shifting, so should your baseline. The market, your customers, or your product all move. Last year’s incrementality reports might not be relevant today. You catch model drift before performance collapses instead of after.
Listen, watch or read the full episode here.
Fresh remote job postings you should check out
Growth Marketing Lead at Glacis
Revenue Operations Analyst (US) at Transfr
Revenue Operations Manager (US) at IntusCare
RevOps Manager at Overflow