The AI verification problem. When AI agents disagree, which one do you trust?

Every query ran. That's the problem.

A thread on r/analytics recently asked what AI is and isn't replacing in analytics work. The most upvoted reply wasn't about jobs. It asked who catches the incorrect join, the flawed aggregation or the result that's silently wrong when nobody reads the code. The hard part isn't writing the query anymore. It's knowing whether you can trust the number that comes back.

We wanted to measure that on real marketing data, not a toy example. So we took one month of a real business's Google Ads, GA4 and Search Console data, gave it to 11 AI models as a warehouse they could query, and asked the questions you'd find in any monthly report, from ROAS to "how many leads did we get?". Every model could explore the tables, run as many queries as it liked and check its own work before answering. Then we asked the same questions through Jepto's data model instead of raw tables.

The short version: today's models are very good at the questions with one right answer. Where a question needs a decision, like how to split ad spend across landing pages, they make different decisions, and nothing in the output tells you which one you got.

‍

How we tested it

We'll call the business Homer's Donuts, a donut shop in Springfield that sells online, runs five Google Ads campaigns and tracks its site with GA4 and Search Console. The name is ours; every number below is real data from a real business's account. We used September 2026: $2,481 of Google Ads spend, 2,112 website sessions and $13,676 of revenue.

Each model worked as an agent, the way most people use AI with data today. We used the same lightweight agent loop for every model (built with Strands Agents) and changed only the model, so differences come from the model rather than the setup. Each run got one question and up to 15 tool calls, and had to finish with an answer table plus every query it used.

There were two setups.

SQL on a warehouse. The data sat in a warehouse in the shape most connectors produce: daily Google Ads campaign rows, GA4 rows by day, source, medium, campaign, landing page and device, a monthly GA4 property table, and Search Console daily totals plus a query and page table. The model saw the schema and had one tool, run_sql.

Jepto's data model through MCP. The model had Jepto's MCP tools for the same account: the data source list, the metric and dimension catalogue, and the analytics query tool. No SQL, no tables, just named metrics such as googleAds.roas and googleAnalytics4.totalUsers.

Here's the prompt every model got, word for word apart from the name and currency.

You are a marketing analyst. Today's date is 2026-10-01. Homer's Donuts is an American donut
shop in Springfield. Amounts are in USD. Use your tools to answer the question from the data. You have read access
to the company's DuckDB data warehouse through the run_sql tool.

Schema: (the eight tables, with the column notes a connector's documentation gives you)

When you're done, reply with the answer as a JSON table in a ```json block shaped like
{"columns": [...], "rows": [[...], ...]}, followed by every SQL query you used for that
answer, each in its own ```sql block.

The Jepto version swaps the warehouse sentence for "You have the Jepto MCP tools for the company's marketing data" and asks for the analytics_query calls instead of SQL. The question then follows, with the shape of the answer so results are comparable:

Question: Which landing pages got the best return on our Google Ads spend in September?

Return one row per landing page with columns named landing_page, spend, revenue and roas.

We ran every question three times per model and checked each answer against the platforms' own figures: Google Ads for spend, clicks, conversions and value, GA4 for sessions, users and revenue, and Search Console for clicks, impressions and position.

The models: Claude Opus, Sonnet and Haiku, GPT-5.5, Gemini 3.1 Pro and 3.8 Flash, Grok 4.7, Kimi K3, GLM 5.3, DeepSeek V4.1 Flash and Qwen 3.8 Max, all at their default settings.

‍

The results at a glance

‍

The good news: one-answer questions are solved

When a question has one right answer and the data supports it, the models found it. Google Ads ROAS for September was 2.29, and all 33 SQL answers said so. GA4 counted 1,576 users, and all 33 answers found the monthly table and used it rather than adding up days. Spend, sessions and revenue by campaign were right in 33 of 33, and so was the month-over-month comparison, the most involved question we asked. Conversions and cost per conversion were right in 32 of 33; the other counted Search campaigns only, a defensible reading we score as a different definition.

The account's click-through rate was 2.55%, and 32 of 33 answers returned it, even though we phrased the question as "the average click-through rate of our campaigns". The one exception is worth a look: GLM 5.3 averaged the five campaigns' rates, got 8.32%, said that's what it had done, and added that the account-level figure was 2.55%. It was the only answer that noticed the question could be read two ways.

The agent setup helped. Models routinely checked the date range, looked at the currency and time zone columns, and inspected the data before writing the final query. The median answer took two or three queries; the landing page question took ten.

If you've avoided AI for marketing SQL because you expect it to fumble basic aggregation, that's no longer where the risk is.

‍

Explore every answer

Before the mistakes, here's every answer. Pick a question to see each distinct result the models returned and which models returned it. Click any model to see the exact SQL it wrote, compared line by line with the closest correct answer, so you can see precisely where two answers part ways.

‍

Mistake 1: an average that looks careful

Search Console reports an average position for every day. For the month, the position you want weights each day by its impressions, which is how Search Console itself reports a period. 31 of 33 answers returned 12.43. Both runs that didn't came from one model, which took a plain average of the daily positions and reported 11.96.

SELECT
  SUM(clicks)::DOUBLE / SUM(impressions) AS ctr,
  AVG(position) AS avg_position
FROM gsc_site_daily
WHERE date >= DATE '2026-09-01' AND date <= DATE '2026-09-30';

That's the exact query GLM 5.3 wrote on its second run. AVG(position) counts every day the same, busy or quiet.

Half a position doesn't sound like much. But notice that the click-through rate in the same query is done properly, from totals. The model knew the rule for one ratio and not the other, and nothing in the result looks wrong.

‍

Mistake 2: a table that contradicts itself

We asked how many users each channel brought in September, and how many users there were in total. The total is easy: 1,576, in the monthly table. Users by channel aren't in the warehouse for the month, only for each day, and daily users can't be added up into a month. There's no correct way to answer the channel half of the question from these tables.

Every answer gave channel numbers anyway. 31 of 33 put the correct 1,576 in the total row and filled the channel rows by adding up daily users. Those channel rows add up to 2,016. So the table says 1,576 people visited, and also that the channels brought 2,016 of them. The other two used 2,016 as the total.

select session_medium, sum(total_users) u from ga4_daily where date>='2026-09-01' and date<'2026-10-01' group by 1 order by 2 desc;
select * from ga4_property_monthly where month='2026-09';

That's Claude Sonnet's first run, in full: one query for the channels, one for the total, and no note that they disagree. 11 of the 33 answers did warn that the channel rows were overstated. The other 22 handed over a table whose rows add up to more than its own total, without a word.

‍

Mistake 3: one question, many answers

Google Ads doesn't report spend by landing page, and GA4 doesn't report spend at all. To get ROAS by landing page you have to decide how to split each campaign's spend across the pages its visitors landed on, then match that to GA4 revenue. Every model had to make that decision, and it took them a median of ten queries, more than any other question. You can see every version, and the SQL behind it, in the explorer above.

Most answers split each campaign's spend by its share of sessions on each page, and still didn't agree. Homepage spend ranged from $1,026 to $1,461 depending on which sessions were counted, which put the homepage's ROAS anywhere from 0.32 to 0.46: six different figures from 32 answers. One page's ROAS ranged from 6.97 to 15.04. Some answers listed 49 pages and others 6, and one allocated 13% more spend than the account actually spent.

None of these is a bug. Each is a defensible modelling choice, and each produces a clean, plausible table. The problem is that the choice is invisible unless you read the SQL, and the same model made different choices on different runs.

‍

Mistake 4: two platforms, two versions of revenue

The real data contained a trap we didn't plan. Google Ads credited the campaigns with $5,686 of conversion value. GA4 recorded about $3,200 of revenue from those campaigns' sessions. The brand campaign is the extreme case: $1,486 of value in Google Ads, $0 of revenue in GA4.

Both numbers are "right". They count different things: Google Ads attributes conversions to ad clicks using its own model, often across days and devices, while GA4 credits the session. When we asked for revenue by campaign, all 33 answers used GA4, and in every one the brand campaign showed no revenue at all. None mentioned that Google Ads credits the same campaign with $1,486. Ask the question slightly differently and you get the Google Ads view, with brand as one of the best performers. Same month, same business, opposite conclusions about one campaign.

Google Ads also reports fractional conversions (14.18 for the brand campaign) because of data-driven attribution, which surprises anyone who expects whole orders.

‍

The questions a business actually asks

Everything so far had a right answer we could check. Real questions usually don't. Nobody asks for "the impression-weighted average position"; they ask how many leads they got, or which campaign to cut. So we asked all 11 models nine questions like that, once each, against the same warehouse.

On some, the models converged. All 11 reported sales as GA4 revenue, worked out cost per lead from Google Ads conversions, and measured conversion rate per session. Nine of 11 named organic search as the best channel. Where they converged, they mostly did so by picking the same default, not by checking it with anyone.

On the others, they split. "How many leads did we get?" got 71 from five models (every GA4 key event), 80 from one (Google Ads conversions), several figures with no headline from three, and from two an honest "we can't tell you": the warehouse has no form-fill or enquiry event, so there's no true lead count. "How much revenue did our marketing drive?" got headlines from $3,200 (paid search only) to $13,676 (every sale on the site), with $7,500, $7,800 and $9,882 in between, depending on whether organic search, AI assistants and direct visits count as marketing.

The answer that matters most for a budget meeting split 8 to 3. Asked which Google Ads campaign to cut, eight models said Glazed Donuts, whose September return collapsed on Google Ads' own figures. Three said Brand, which shows no revenue at all in GA4. Both are reasonable readings of the same data, and they recommend opposite actions.

The models weren't hiding anything: 82 of the 98 answers said which definition they used. But saying it isn't the same as being consistent. All five models that counted 71 leads then worked out cost per lead from Google Ads' 80 conversions, so their two answers use two different definitions of a lead. Ask both questions, put the answers in one report, and the numbers don't reconcile. Nobody made a mistake you could point to. The business never decided what a lead is, so each answer decided for it.

‍

Search Console's missing clicks

One more trap, this time in the data itself. Search Console hides searches that too few people make, for privacy. Its daily site totals include them; its query and page table doesn't. In September the site got 371 clicks from Google Search, but the query table adds up to just 112. A model that answers "how many organic clicks" from the query table is off by 70%, with no error. All 33 answers used the site table, as they should, so this one didn't catch anybody. It will catch the next question that needs a query-level breakdown.

‍

The same questions through a data model

‍

Why agreement isn't verification

On the questions with one right answer, the models agreed, and agreement meant something. On the questions that need a decision, agreement was never the point. Landing page ROAS has several defensible answers, and the models spread across them. Revenue by campaign has two true answers, and most models picked one without saying there was another. A second model, a rerun, or a majority vote can't tell you which definition you wanted.

What each case needed was a decision made before the question was asked. How is spend split across pages? Is revenue what Google Ads says or what GA4 says? How is position averaged over a month? An analyst who's been burned before carries those answers around. A model starting from a schema doesn't, so it guesses, and a guess that runs cleanly looks the same as a fact.

‍

How to check an AI answer yourself

‍

Method and limits

One business and one month, so treat the percentages as illustrations, not benchmarks. The data is real, which means some true answers depend on a definition. Where a question had more than one defensible answer, we marked answers as "different definition" rather than wrong and show every version in the explorer. Users were checked against GA4's own monthly figures, because users can't be added up from daily rows.

Campaign names, page paths and search terms have been relabelled to fit the name; the figures and the logic of every query are unchanged. Every model ran in the same Strands agent loop with default settings, a 15-tool-call budget and the same prompt; the SQL ran in DuckDB. Claude models ran through the Anthropic API, GPT through OpenAI, Gemini through Google, Grok through xAI, and Kimi, GLM, DeepSeek and Qwen through Fireworks.

‍

Where a data model fits

The last item on that checklist is the one that scales. If every definition lives in someone's head, or in whichever SQL a model writes today, every new question is a fresh chance to get it wrong. That's the idea behind a semantic layer, or data model: metrics are defined once, centrally, and every query, chart and AI answer uses the same definitions instead of rebuilding them from raw tables.

‍

Auditing an answer: SQL vs a data model query

Here's the difference in practice, on a question with a few more moving parts: which Google Ads campaigns improved their ROAS from August to September, with spend, conversions, cost per conversion and ROAS for each month. All 33 SQL answers got this right. The point isn't whether the answer is right. It's what you have to read to know that it is.

Here is Gemini 3.8 Flash's answer on its first run, exactly as written, next to the Jepto query for the same question.

SELECT
    campaign_name AS campaign,
    SUM(CASE WHEN date >= '2026-08-01' AND date <= '2026-08-31' THEN cost END) AS aug_spend,
    SUM(CASE WHEN date >= '2026-08-01' AND date <= '2026-08-31' THEN conversions END) AS aug_conversions,
    SUM(CASE WHEN date >= '2026-08-01' AND date <= '2026-08-31' THEN cost END) /
        SUM(CASE WHEN date >= '2026-08-01' AND date <= '2026-08-31' THEN conversions END) AS aug_cost_per_conversion,
    SUM(CASE WHEN date >= '2026-08-01' AND date <= '2026-08-31' THEN conversions_value END) /
        SUM(CASE WHEN date >= '2026-08-01' AND date <= '2026-08-31' THEN cost END) AS aug_roas,
    SUM(CASE WHEN date >= '2026-09-01' AND date <= '2026-09-30' THEN cost END) AS sep_spend,
    SUM(CASE WHEN date >= '2026-09-01' AND date <= '2026-09-30' THEN conversions END) AS sep_conversions,
    SUM(CASE WHEN date >= '2026-09-01' AND date <= '2026-09-30' THEN cost END) /
        SUM(CASE WHEN date >= '2026-09-01' AND date <= '2026-09-30' THEN conversions END) AS sep_cost_per_conversion,
    SUM(CASE WHEN date >= '2026-09-01' AND date <= '2026-09-30' THEN conversions_value END) /
        SUM(CASE WHEN date >= '2026-09-01' AND date <= '2026-09-30' THEN cost END) AS sep_roas,
    (SUM(CASE WHEN date >= '2026-09-01' AND date <= '2026-09-30' THEN conversions_value END) /
        SUM(CASE WHEN date >= '2026-09-01' AND date <= '2026-09-30' THEN cost END)) >
    (SUM(CASE WHEN date >= '2026-08-01' AND date <= '2026-08-31' THEN conversions_value END) /
        SUM(CASE WHEN date >= '2026-08-01' AND date <= '2026-08-31' THEN cost END)) AS improved
FROM google_ads_campaign_daily
GROUP BY campaign_name
ORDER BY campaign;

{
  "measures": [
    "googleAds.spend",
    "googleAds.conversions",
    "googleAds.costPerConversion",
    "googleAds.roas"
  ],
  "dimensions": [
    "googleAds.campaignId",
    "googleAds.campaignName"
  ],
  "startDate": "2026-09-01",
  "endDate": "2026-09-30",
  "compareStartDate": "2026-08-01",
  "compareEndDate": "2026-08-31",
  "periodTotals": true
}