Same data, different answers: how AI models choose marketing metrics

TL;DR

• We gave 11 AI models the same GA4, Google Ads and Search Console tables and asked nine questions that need a business definition. For each answer we recorded the figure it led with, the others it reported, and the reason it gave.
• When one metric fits, they agree: all 11 reported sales as GA4 revenue, and all 11 worked out cost per lead from Google Ads conversions.
• When the question needs a decision, they split. "How much revenue did our marketing drive?" got five different headline figures, from $3,200 to $9,882.
• They don't stay consistent. Seven of the eight models that gave a lead count used GA4, then all of them priced those leads with Google Ads conversions. Put both in one report and the numbers don't reconcile.
• The data set decides the answer. The eight models that judged campaigns on Google Ads ROAS said cut Glazed Donuts. The three that used GA4 revenue said cut Brand.
• GA4 is the default brain. Outside the three questions that name Google Ads, only 3 of 66 headline figures came from Google Ads. No model led with Search Console on any question.

‍

Why look at the questions with no right answer

In The AI verification problem we scored 11 AI models on 16 questions that have one right answer. Some of the everyday questions you'd actually ask an AI aren't like that. "How many leads did we get?" depends on what you call a lead, and no table in GA4 or Google Ads says.

So for these questions we didn't ask whether the models were right. We asked what they chose: which data set they reached for, which metric they led with, what they reported alongside it, and how they explained the choice. Across 98 answers, that gives a fair picture of how an AI analyst thinks about marketing data when nobody has told it your definitions.

‍

How we tested it

The setup is the same as the first post. The business is Homer's Donuts, a donut shop in Springfield that sells online, runs five Google Ads campaigns and tracks its site with GA4 and Search Console. The name is ours; every number is real data from a real business's account for September 2026.

Each of the 11 models worked as an agent with the schema in its prompt, a run_sql tool and the tidy tables a connector produces: GA4 by day and by month, Google Ads campaigns by day, and Search Console by day. Each question was asked once per model, with no instructions about definitions.

Each chart below shows one question. The left column is the figure the model led with, the middle shows what else it reported, and the platform icon tells you which data set each figure came from. Under each row is the model's own reason, quoted from its answer. Use the Answer button to read the full reply and the SQL behind it. The bars at the bottom show how many of the 11 models led with each metric, and how many reported it alongside.

‍

When one metric fits, they agree

Asked for September's sales, all 11 models led with GA4 revenue, $13,676. Most added the 71 key events as a count of orders, and two noted Google Ads' own conversion value of $5,686 as a different number.

Conversion rate was nearly as uniform: 10 of 11 divided GA4 key events by sessions and reported 3.4%. Five also gave a rate per user, 4.5%, and DeepSeek worked one out per summed daily user, 3.5%. Gemini 3.8 Flash gave both without choosing.

What these have in common is that the data offers one obvious candidate. GA4 has a revenue column and nothing else that looks like sales. When that's true, the models behave the same way, and they mostly tell you what they used.

‍

Leads: a word no table defines

Leads is where they start to diverge. Six models counted every GA4 key event, 71. GLM used Google Ads conversions, 80. Gemini 3.8 Flash counted only key events with no revenue attached, 27, on the reasoning that a purchase isn't a lead. Claude Sonnet 5.5 and Qwen said the data has no lead metric, and Gemini 3.1 Pro listed six candidate figures without choosing one.

The interesting answers are the ones that split the counts. Gemini 3.1 Pro, Gemini 3.8 Flash, Grok, Qwen and DeepSeek all separated conversions or key events by whether they carried revenue, treating the ones without revenue as leads and the rest as sales. Qwen went furthest, counting visits to the contact and quote pages. None of these is wrong. They're different businesses' definitions, and the model had to guess which one is yours.

‍

Then they contradict themselves

The next question asked for cost per lead from Google Ads. All 11 models gave the same answer: $2,481 of spend over 80 Google Ads conversions, $31.01.

That agreement hides a problem. Seven of the eight models that gave a lead count in the previous question used GA4: six said 71 leads and Gemini 3.8 Flash said 27. Every one of them then priced leads using Google Ads' 80 conversions. Only GLM used the same definition both times.

Put those two answers in one monthly report and they don't reconcile: 71 leads at $31.01 each comes to $2,202, not the $2,481 you spent. Three models, Claude Opus 5.5, Kimi and Qwen, did report the GA4 view alongside: Google Ads spend over the 18 key events from paid search is $138 a lead, more than four times the headline.

‍

Marketing revenue: where does marketing stop?

"How much revenue did our marketing drive?" produced five different headline figures, one breakdown by channel, and three models that wouldn't pick one.

The split is about which traffic counts as marketing. Three models took everything except direct traffic, $9,882. Two counted paid and organic search, $7,547. Qwen added visits from AI assistants, $7,756. Claude Haiku 5.5 counted paid search only, $3,200. No model led with all of GA4's revenue, $13,676, though most reported it alongside. Every one of these boundaries is defensible, and the largest answer is three times the smallest.

‍

Did Google Ads pay for itself? It depends whose numbers

Google Ads says its campaigns returned $2.29 for every dollar. GA4 credits paid search with $3,200 of revenue on $2,481 of spend, a return of 1.29. Both say yes, but by very different margins, and the models split on which to believe.

Three led with GA4's 1.29, two with Google Ads' 2.29, Gemini 3.8 Flash with the profit on Google Ads' figures, and four reported both without choosing. Kimi didn't return an answer. Claude Opus 5.5 summed up the case for GA4: "Google Ads is probably counting more than purchases, and it also gives itself credit by its own rules." GLM made the case the other way: since the question is about a paid channel, lead with the platform's figure.

‍

Which campaign to cut: the metric picks the campaign

This is the question where the choice of data set changes the decision.

Eight models judged the campaigns on Google Ads ROAS and named Glazed Donuts, which returned the least conversion value per dollar. Three judged them on GA4 revenue per campaign and named Brand, because GA4 credits Brand with far less revenue than Google Ads does. Qwen put it plainly: Google Ads' claim for Brand is "~8x larger than any other campaign's, so its reported return is not credible." Same data, opposite budget actions, and the deciding factor was which platform the model trusted.

‍

Best channel and the month in review

Asked which channel performed best, 10 of 11 judged on GA4 revenue, and 9 named organic search. Claude Sonnet 5.5 named Direct, and Claude Haiku 5.5 judged on return on ad spend and named paid search, pointing out that Direct "has no media cost, so it isn't comparable on efficiency."

Asked how September went compared with August, the models agreed on what belongs in a summary and disagreed on what to lead with. Five opened with sessions, two with users, two with revenue, and Claude Haiku 5.5 with Google Ads spend. This was the only question where every model used Search Console. Ten took users from GA4's monthly table, 1,576. DeepSeek added up daily users instead and reported 2,016, the same mistake that sank the users-by-channel question in the first post.

‍

The marketing brain

Put the nine questions side by side and a few habits show up.

• GA4 is the default. 61 of the 98 headline figures came from GA4 and 25 from Google Ads. Outside the three questions that name Google Ads, only 3 of 66 headlines came from it.
• Search Console is an afterthought. No model led with it on any question. It appeared in every month-in-review answer and in two best-channel answers, and nowhere else.
• Some models commit, some hedge. DeepSeek, GLM and GPT-5.5 led with one figure on every question. Claude Opus 5.5, Claude Sonnet 5.5 and both Gemini models declined to pick on two questions each.
• Most explain themselves, not all. Claude Haiku 5.5, Claude Sonnet 5.5 and GLM said which definition they used in all nine answers. GPT-5.5 and Grok did in five.

Hedging isn't a flaw here. A model that says "this depends on what you count as a lead" is giving you a more honest answer than one that picks 71 without saying why. But it still leaves the decision with you.

‍

What this means for you

None of these answers is wrong, and that's the point. Each is a reasonable reading of a question that needs a business decision, and each model made that decision for you, differently, often without saying so. Ask the same model two related questions and it may decide differently each time.

The fix isn't a smarter model. It's deciding these things once:

• What counts as a lead. Every key event, specific events, or Google Ads conversions.
• Which revenue counts as marketing. All of it, everything but direct, or only paid and organic search.
• Whose numbers you trust for paid media. The platform's conversion value or your analytics.

Then make every answer use them. That's what a data model is for: in The AI verification problem we showed how it keeps answers right when the rules are known. It does the same for definitions, once you've made them.

‍

Method and limits

One business, one month and one run per model, so treat the splits as illustrations of how models choose, not fixed traits of each model. The models, versions and settings are the same as in the first post, all at their API's default reasoning effort, writing SQL against connector-style tables in DuckDB.

To compare the answers, Claude Sonnet 5.5 read each model's SQL and reply and matched its figures to a fixed list of metrics per question, choosing the one it led with and the others it reported. Every quote is copied word for word from the model's answer, and you can check each one with the Answer button. Campaign names, page paths and search terms have been relabelled to fit the name; the figures and the logic of every query are unchanged.