Jepto

Pricing

Sign In

Try For Free

Analytics

The AI verification problem. When AI agents disagree, which one do you trust?

Dale McGeorge

Head of Product

circle-4circle-3circle-1circle-2

We gave 11 AI models a real business's Google Ads, GA4 and Search Console data and asked 16 questions with one right answer. On easy questions every model was right. As the questions got harder, accuracy fell to 45%, and the wrong answers looked exactly like the right ones. Then we ran the same questions through a data model.

TL;DR

  • We asked 11 AI models 16 marketing questions with one right answer on one real business's data, once each, and sorted the questions by how often the models got them right: Easy, Medium and Hard.
  • Easy questions, where the answer sits in one table and means one thing, were 100% right. If that's all you ask, AI on raw data works.
  • Medium questions, which need a rule the schema doesn't state, were 92% right. Most misses came back as a clean number with no warning.
  • Hard questions, where the tables can't be combined the obvious way, were 45% right. Three models agreed on the same wrong answer, and each hard answer cost three times an easy one.
  • A data model writes the rules down once, so accuracy holds as the questions get harder. With Jepto's data model, the same models scored 97%, 94% and 73% on Easy, Medium and Hard, and Hard rises to 97% if you leave out ROAS by landing page, which no data source can answer exactly. It costs a few cents more per answer.

The data

What every model could query

Homer's Donuts' GA4, Google Ads and Search Console data for August and September 2026.

GA47 metrics · 8 dimensions

Metrics

UsersNew usersSessionsEngaged sessionsEngagement rateKey eventsPurchase revenue

Dimensions

DateMonthSession sourceSession mediumSession campaignLanding pageDevice categoryEvent name

Tables

  • Daily, by source, medium, campaign, landing page and device
  • Monthly totals for the property
  • Google's export: one row per event

Google Ads5 metrics · 9 dimensions

Metrics

ImpressionsClicksCostConversionsConversion value

Dimensions

DateCampaignCampaign IDCampaign statusAccountCurrencyTime zoneDeviceChannel type

Tables

  • Daily, by campaign
  • Google's export: campaign stats by day and device, and daily campaign snapshots

Search Console4 metrics · 6 dimensions

Metrics

ClicksImpressionsCTRAverage position

Dimensions

DateQueryPageCountryDeviceSearch type

Tables

  • Daily totals for the site
  • Daily, by query, page, country and device
  • Google's export: site and URL impressions, positions counted from zero

Each question used either the tidy tables a connector produces or Google's own BigQuery export, the same for all 11 models.

‍

The benchmark

The benchmark

As questions get harder, raw SQL gets less accurate. A data model holds accuracy up.

16 questions with one right answer, asked once to each of 11 models on one real account's data.

AI + raw SQLAI + data model
EasyCorrect

One table, one obvious metric

The answer sits in a single table at the right level of detail, and the question means one thing.

Raw SQL
100%
Data model
97%

6 questions · e.g. Google Ads ROAS, monthly users, sessions

MediumCorrect

A rule the schema doesn't state

A ratio has to be rebuilt from totals, a snapshot table joined once, or positions counted from one, not zero.

Raw SQL
92%
Data model
94%

6 questions · e.g. average position, ROAS by campaign on Google's export · a miss on four of the six

HardCorrect

Data the tables can't combine correctly

Metrics that can't be added up, dates stored differently in two tables, spend split across pages, sessions matched to campaigns.

Raw SQL
45%
Data model
73%

4 questions · e.g. users by channel (4 of 11), Search spend by device (8 of 11)

In Hard, ROAS by landing page has no single right answer on either side, because no source records spend by page, so it isn't counted as right.

Read it top to bottom. The purple bars are what you get today by pointing an AI at your tables. The green bars are the same models, with the same questions, asking Jepto's data model instead.

‍

Every query ran. That's the problem.

A thread on r/analytics recently asked what AI is and isn't replacing in analytics work. The most upvoted reply wasn't about jobs. It asked who catches the incorrect join, the flawed aggregation or the result that's silently wrong when nobody reads the code. The hard part isn't writing the query anymore. It's knowing whether you can trust the number that comes back.

So we measured it. Not "can AI write SQL", which it can, but where it stops being safe to trust the answer without checking, and what it takes to move that line.

‍

How we tested it

We'll call the business Homer's Donuts, a donut shop in Springfield that sells online, runs five Google Ads campaigns and tracks its site with GA4 and Search Console. The name is ours; every number is real data from a real business's account for September 2026.

Each model worked as an agent, the way most people use AI with data today: the same lightweight agent loop for every model (built with Strands Agents), the schema in the prompt, a run_sql tool, up to 15 queries, and an answer table at the end. The data sat in two layouts: the tidy tables a connector produces, and the tables Google's own BigQuery exports create. Every question was asked once per model, the way you'd ask it for real.

The models: Claude Opus 5.5, Sonnet 5.5 and Haiku 5.5, GPT-5.5, Gemini 3.1 Pro and 3.8 Flash, Grok 4.7, Kimi K3, GLM 5.3, DeepSeek V4.1 Flash and Qwen 3.8 Max, all at default settings.

Then we sorted the 16 questions by how many models got them right. Three levels fell out, and each one has something in common. Questions with no single right answer, like "how many leads did we get?", can't be scored this way, so they have their own post.

‍

Easy: one table, one obvious metric

Google Ads ROAS for September was 2.29. All 11 models said so. GA4 counted 1,576 users for the month, and all 11 found the monthly table rather than adding up days. Spend, sessions and revenue by campaign, August against September, website sessions and organic clicks: 66 answers, 66 right.

Easy questions

How many models got each easy question right

11 models, one run each. Questions without a note used the tidy connector tables.

AI + raw SQLAI + data model
EasyCorrect

What was our Google Ads ROAS in September?

Raw SQL
100%
Data model
100%

How many users visited the website in September?

Raw SQL
100%
Data model
82%

For each Google Ads campaign, show September spend, website sessions and website revenue.

Raw SQL
100%
Data model
100%

Which Google Ads campaigns improved their ROAS from August to September? Show each campaign's spend, conversions, cost per conversion and ROAS in both months.

Raw SQL
100%
Data model
100%

How many sessions did the website have in September? (Google's export)

Raw SQL
100%
Data model
100%

How many clicks did we get from Google organic search in September? (Google's export)

Raw SQL
100%
Data model
100%

What these have in common is that the answer lives in one place, at the right level of detail, and the question can only mean one thing. If your AI questions look like this, raw SQL is fine, and it's the cheapest way to ask.

‍

Medium: a rule the schema doesn't state

Medium questions need one piece of knowledge the tables don't carry. Average position over a month has to be weighted by impressions. Google's Search Console export counts positions from zero. A snapshot table repeats each campaign once a day, so it has to be joined once, not 30 times.

Medium questions

How many models got each medium question right

11 models, one run each. Questions without a note used the tidy connector tables.

AI + raw SQLAI + data model
MediumCorrect

What was our organic search click-through rate and average position in September?

Raw SQL
91%
Data model
100%

How many conversions did paid search drive in September, and what was the cost per conversion?

Raw SQL
100%
Data model
100%

What was the average click-through rate of our Google Ads campaigns in September?

Raw SQL
91%
Data model
100%

How many users visited the website in September? (Google's export)

Raw SQL
100%
Data model
64%

What was the ROAS of each Google Ads campaign in September? (Google's export)

Raw SQL
91%
Data model
100%

What was our average position in Google organic search in September? (Google's export)

Raw SQL
82%
Data model
100%

Most models knew the rule: 61 of 66 answers were right. The misses are the problem. GLM 5.3 calculated the impression-weighted position, 12.43, then reported a plain average of the days, 11.96. DeepSeek and Qwen reported 11.43 from Google's export because they didn't add one. Each of those came back as a clean, single number, and nothing in it tells you it's the wrong one. One miss did say what it was doing: asked for the average click-through rate of the campaigns, GLM averaged the five campaigns' rates, 8.32%, and said so. That's a fair reading of "average", but it isn't the account's click-through rate of 2.55%, so we scored it as a different definition.

‍

Hard: data the tables can't combine correctly

Hard questions need two tables combined in a way the obvious query gets wrong, or can't get right at all. Only 20 of 44 answers were right.

Hard questions

How many models got each hard question right

11 models, one run each. Questions without a note used the tidy connector tables.

AI + raw SQLAI + data model
HardCorrect

How much did we spend on Search campaigns (not Shopping) on each device in September? (Google's export)

Raw SQL
73%
Data model
91%

For each Google Ads campaign, show September spend, the website sessions it drove and the website revenue from those sessions. (Google's export)

Raw SQL
73%
Data model
100%

How many users did each channel bring to the website in September, and how many users did we have in total?

Raw SQL
36%
Data model
100%

Which landing pages got the best return on our Google Ads spend in September?

Raw SQL
0%
Data model
0%

Each of the four goes wrong in a different way.

Users by channel, with total. Asked how many users each channel brought in September, plus the total, 4 of 11 got it right. The total is in a monthly table: 1,576. Users by channel only exist per day, and daily users can't be added into a month, because the same person visits on several days. Ten models put 1,576 in the total row and filled the channels by adding up days, so their channel rows add up to 2,016. Claude Opus, Sonnet and Haiku and Qwen said the channel rows were overstated because the same person is counted again for each day they visit. Three more noticed the rows were too high but gave the wrong reason or the wrong total, and GPT-5.5, both Geminis and Grok didn't mention it. Rows above the total aren't proof on their own, because one person can arrive through two channels: GA4's own monthly figures by channel add up to 1,612 against a total of 1,575. The mistake is adding up days, which counts a person again for every day they came back. These tables have no monthly users by channel, so on raw SQL the right answer was the total plus a note that the channels can't be split exactly.

Search spend by device. Google's export keeps campaign settings in one table and daily stats in another, and both have a column that looks like a date filter. Three models applied the snapshot filter that belongs on the campaign table to the stats table too. That kept only 30 September and threw away the other 29 days, so their desktop figure was $18.81 instead of $1,462.55. The next section shows how that got past three models agreeing.

Sessions and revenue by campaign. This one joins Google Ads spend to GA4 sessions and revenue, which sit in two different exports. Three models missed in three different ways. Gemini 3.8 Flash's joins matched nothing, so every campaign came back with $0 spend, 0 sessions and $0 revenue. Grok 4.7 got spend and sessions right, then reported Google Ads' conversion value as website revenue: $1,486.50 for Brand, where GA4 records none. Qwen 3.8 Max ran out of query budget with its final query written but never run, so it returned no answer.

ROAS by landing page. This can't be answered exactly, on either side. In the tables the models had, Google Ads records spend by campaign and GA4 records revenue by landing page, but nothing records spend by page. To get a ROAS for each page, a model has to decide how to share each campaign's spend across the pages it sent visitors to, and every way of sharing gives a different answer. Six models ended up on a homepage ROAS of 0.45 and two on 0.42. Gemini 3.8 Flash reported 0.46, but its figures aren't what its own query returns. One allocated $2,816 of spend against the account's real $2,481, and Claude Haiku 5.5's run stopped on an API billing error on our side, so it has no answer. None counts as right, because there's no true figure to score against.

Hard questions also cost more. At each provider's list prices, the median hard answer cost 6 cents and took six or seven queries, against 2 cents for an easy one. ROAS by landing page took a median of twelve queries. Paying more didn't buy accuracy either: on the seven export questions, Claude Haiku 5.5 got all seven right for 2 cents in total, and Qwen 3.8 Max spent 57 cents to get five.

‍

Why checking is hard: search spend by device

Hard questions aren't only a problem of accuracy. The wrong answers look the same as the right ones, so the usual ways of checking don't work. Search spend by device shows why. Here's what happened.

The verification problem

Search spend by device in September: how a wrong answer gets checked

Real answers from our test on Google's raw BigQuery export tables. Eight models got $1,462.55; three models independently converged on $18.81.

Step 1Read the SQLLooks fine
Step 2Ask two more modelsLooks confirmed
Step 3Compare with Google AdsAll 3 wrong
Step 4Check one totalCaught
1

Transparency

“The agent shows me its SQL, so I can audit it.”

Gemini 3.1 Pro$18.81desktop · $9.86 mobile · $0.00 tablet
SELECT s.segments_device AS device,
       SUM(s.metrics_cost_micros) / 1e6 AS spend
FROM google_ads.ads_CampaignBasicStats_5550001111 s
JOIN google_ads.ads_Campaign_5550001111 c
  ON s.campaign_id = c.campaign_id
WHERE c._DATA_DATE = c._LATEST_DATE
  AND s._DATA_DATE = s._LATEST_DATE
  AND c.campaign_advertising_channel_type = 'SEARCH'
  AND s.segments_date BETWEEN '2026-09-01' AND '2026-09-30'
GROUP BY 1

Ask yourself: Ten clean lines that ran without an error. Can you tell from reading them whether the answer is right?

2

Agreement

“I'll have two more models run it independently.”

Gemini 3.1 Pro (Google)$18.81desktop · $9.86 mobile
Grok 4.7 (xAI)$18.81desktop · $9.86 mobile (own SQL)
Kimi K3 (Moonshot)$18.81desktop · $9.86 mobile (own SQL)

The catch: Three models, three providers, three separately written queries, identical to the cent. It feels like independent confirmation. What if they all made the same mistake?

3

The reveal

Three models agree. All three are off by 98.5%.

What all 3 models agreed on$18.81desktop ($28.67 total Search spend)
What Google Ads actually spent$1,462.55desktop ($1,953.12 total Search spend)
SELECT s.segments_device AS device,
       SUM(s.metrics_cost_micros) / 1e6 AS spend
FROM google_ads.ads_CampaignBasicStats_5550001111 s
JOIN google_ads.ads_Campaign_5550001111 c
  ON s.campaign_id = c.campaign_id
WHERE c._DATA_DATE = c._LATEST_DATE
  AND s._DATA_DATE = s._LATEST_DATE   -- keeps 30 September only
  AND c.campaign_advertising_channel_type = 'SEARCH'
  AND s.segments_date BETWEEN '2026-09-01' AND '2026-09-30'
GROUP BY 1

What went wrong: All three applied the campaign snapshot filter (_DATA_DATE = _LATEST_DATE) to the daily stats table too. On the stats table, that column is the partition date, so the filter kept 30 September and threw away the other 29 days.

4

Real verification

Check one total against a source the models didn't produce.

Sum of the 3 models' answer$28.67all devices, Search campaigns
Google Ads interface total$1,953.12Search campaigns, 1 to 30 September

The lesson: One total from the platform, for the dates you meant, exposes the mistake in seconds. A second model only checks whether another LLM writes the same SQL.

Seeing the SQL told you what the models did, not whether it was right. Three models agreeing told you they made the same choice, not that it was correct. Only the platform's own total exposed the mistake. And correct answers don't help you check either: all 11 models got website sessions right with 11 different queries, so there's no reference query to compare against.

‍

What a data model changes

A data model, or semantic layer, is the knowledge from each level written down once: which table holds monthly users, how position is weighted, how the Google Ads export stores its dates, what you count as a lead. The AI asks for googleAds.roas or googleAnalytics4.totalUsers by name, and the rule comes with it.

That's why the green bars stay up. Easy questions don't change much, because there was little to get wrong. Hard questions go from 45% to 73%: users by channel went from 4 of 11 to 11 of 11, because the data model reads users by channel from GA4 for the whole period instead of adding up days. Search spend by device is one query instead of a join across two snapshot tables, and 10 of 11 got it; the one miss, DeepSeek, left Shopping in. ROAS by landing page is the only Hard question no model gets right either way, because it needs spend split across pages that no source records.

Medium goes from 92% to 94%. The misses were on users from Google's export: GPT-5.5, Gemini 3.8 Flash, Kimi and GLM asked the data model for active users, 1,563, instead of total users, 1,575. Active users is a real GA4 metric, but it isn't the total the question asked for. Medium also has one judgement call. Asked how many conversions paid search drove, four of the eleven models read "paid search" as Search campaigns only and left out Shopping, for 60 conversions instead of 80. One model did the same on raw SQL. Google calls Search campaigns paid search too, so we counted both readings as right. That's a definition, not a calculation error: a data model applies the definition you've written down, but it can't pick one for you. We look at how the models pick definitions in how AI models choose marketing metrics.

Here it is model by model: how many of the 16 questions each AI got right, with and without the data model.

Every model

Each AI with and without the data model

16 questions with one right answer, asked once to each model on each side. Most accurate on the data model first.

AI + raw SQLAI + data model

Opus 5.5 · MediumCorrect

Raw SQL
94%
Data model
94%

Sonnet 5.5 · High

Raw SQL
94%
Data model
94%

Haiku 5.5 · Medium

Raw SQL
94%
Data model
94%

Gemini 3.1 Pro · High

Raw SQL
81%
Data model
94%

Qwen 3.8 Max · Xhigh

Raw SQL
81%
Data model
94%

Grok 4.7 · High

Raw SQL
75%
Data model
94%

Kimi K3 · Max

Raw SQL
81%
Data model
88%

DeepSeek V4.1 Flash · Default

Raw SQL
81%
Data model
88%

GLM 5.3 · Default

Raw SQL
69%
Data model
88%

GPT-5.5 · Medium

Raw SQL
88%
Data model
81%

Gemini 3.8 Flash · Medium

Raw SQL
81%
Data model
81%

Every model ran at its API's default reasoning effort, shown next to its name, with no temperature or reasoning settings changed.

It also changes what you have to check. To verify the SQL answer to "which campaigns improved their ROAS from August to September?", you read 16 expressions. To verify the data model query, you check three things: the metrics, the months and the grouping.

Same question, two answers

What you have to know to trust each one

SQL on a warehouse

Correct, if all 16 expressions are

SELECT
    campaign_name AS campaign,
    SUM(CASE WHEN date >= '2026-08-01' AND date <= '2026-08-31' THEN cost END) AS aug_spend,
    SUM(CASE WHEN date >= '2026-08-01' AND date <= '2026-08-31' THEN conversions END) AS aug_conversions,
    SUM(CASE WHEN date >= '2026-08-01' AND date <= '2026-08-31' THEN cost END) /
        SUM(CASE WHEN date >= '2026-08-01' AND date <= '2026-08-31' THEN conversions END) AS aug_cost_per_conversion,
    SUM(CASE WHEN date >= '2026-08-01' AND date <= '2026-08-31' THEN conversions_value END) /
        SUM(CASE WHEN date >= '2026-08-01' AND date <= '2026-08-31' THEN cost END) AS aug_roas,
    SUM(CASE WHEN date >= '2026-09-01' AND date <= '2026-09-30' THEN cost END) AS sep_spend,
    SUM(CASE WHEN date >= '2026-09-01' AND date <= '2026-09-30' THEN conversions END) AS sep_conversions,
    SUM(CASE WHEN date >= '2026-09-01' AND date <= '2026-09-30' THEN cost END) /
        SUM(CASE WHEN date >= '2026-09-01' AND date <= '2026-09-30' THEN conversions END) AS sep_cost_per_conversion,
    SUM(CASE WHEN date >= '2026-09-01' AND date <= '2026-09-30' THEN conversions_value END) /
        SUM(CASE WHEN date >= '2026-09-01' AND date <= '2026-09-30' THEN cost END) AS sep_roas,
    (SUM(CASE WHEN date >= '2026-09-01' AND date <= '2026-09-30' THEN conversions_value END) /
        SUM(CASE WHEN date >= '2026-09-01' AND date <= '2026-09-30' THEN cost END)) >
    (SUM(CASE WHEN date >= '2026-08-01' AND date <= '2026-08-31' THEN conversions_value END) /
        SUM(CASE WHEN date >= '2026-08-01' AND date <= '2026-08-31' THEN cost END)) AS improved
FROM google_ads_campaign_daily
GROUP BY campaign_name
ORDER BY campaign;

6 things to check

  • Does every CASE use the right month and the right column?
  • Do both ROAS figures divide value by cost, not cost by value?
  • Is cost per conversion total cost over total conversions, not an average of days?
  • Does improved compare September with August, not the reverse?
  • Is grouping by campaign name safe, or was a campaign renamed?
  • Are the dates in the account's time zone?
Jepto data model

Correct, if the definitions are

{
  "measures": [
    "googleAds.spend",
    "googleAds.conversions",
    "googleAds.costPerConversion",
    "googleAds.roas"
  ],
  "dimensions": [
    "googleAds.campaignName"
  ],
  "startDate": "2026-08-01",
  "endDate": "2026-09-30",
  "granularity": "month"
}

3 things to check

  • Are these the metrics you mean?
  • Do the dates cover August and September, one row per month?
  • Is each campaign one row?

Both are the queries Gemini 3.8 Flash wrote in our test, one with raw SQL and one with the data model. Both answers were right.

It isn't cheaper per answer yet. Across the eleven models, a data model answer cost a median of 5 cents on easy and medium questions and 11 cents on hard ones, against 2 cents and 6 cents for raw SQL, because the agent looks up the schema and reads fuller responses before it answers. A few cents per question is small. A budget cut made on the wrong answer isn't, and that's the cost that matters.

Cube found a similar effect in their paired benchmark across three frontier models: a semantic layer added 17 to 23 points of accuracy, and with it the three models scored about the same.

We made the longer argument in You don't need a more expensive LLM. You need a data model. This is the evidence for it.

‍

How to check an AI answer yourself

Checklist

Six checks before you trust an AI-written number

  1. Place the question on the ladder.One table and one meaning is Easy; trust it. Anything that rolls up a ratio, joins two tables, counts people or uses a word like "lead" is Medium or harder; check it.
  2. Reconcile one total with the platform.Total spend, users or clicks for the same dates. This is the check that caught all three wrong models.
  3. Check the answer against itself.Do the rows add up to the total? For spend, clicks and sessions they must. Users can run slightly over, because one person can arrive through two channels, but not by a quarter.
  4. Ask which platform each number comes from.Google Ads revenue and GA4 revenue are different numbers.
  5. Don't count agreement as verification.A second model can find a mistake. It can't prove there isn't one.
  6. Write the definitions down once.Then every answer uses them.

‍

Method and limits

One business and one month, so treat the percentages as illustrations, not a universal benchmark. Questions were sorted into levels by how the 11 models did on them, then grouped by what they have in common. Where a question had more than one defensible answer, we counted a different definition as not right and show it in the appendix. The one exception is paid search conversions, where we counted both "all Google Ads campaigns" and "Search campaigns only" as right. We made that call after seeing the results. Scored the other way, the data model gets 64% on that question and 88% on Medium, and raw SQL gets 91% on both. Average CTR went the other way: GLM's average of the campaigns' rates is defensible, and we counted it as not right.

The connector-style tables and the Google Ads and Search Console exports hold the account's real figures. The GA4 export was rebuilt from the property's real daily totals, so sessions and revenue match GA4 exactly; we scored those questions against what the export itself supports. Campaign names, page paths and search terms have been relabelled to fit the name; the figures and the logic of every query are unchanged. Claude ran through the Anthropic API, GPT through OpenAI, Gemini through Google, Grok through xAI, and Kimi, GLM, DeepSeek and Qwen through Fireworks. The SQL ran in DuckDB.

The data model runs used Jepto's production MCP server, the same one available to ChatGPT and Claude users, on 28 and 30 October 2026, with all eleven models. Data model costs leave out the tokens of skill documents, which production loads directly rather than through tool calls.

‍

Appendix: every answer

Pick a question to see every distinct result the models returned and the exact SQL behind each one.

Answer explorer

Same question, different answers, and the exact SQL behind each one

Show results for

“Which landing pages got the best return on our Google Ads spend in September?”

True answer: no single true answer: spend has to be split across pages · 11 answers, 5 different results

homepage ROAS 0.456 of 11

Different definitionOne way of splitting spend across pages

homepage ROAS 0.422 of 11

Different definitionOne way of splitting spend across pages

homepage ROAS 0.461 of 11

WrongReported figures its own final query doesn't produce

Spend $2,8161 of 11

WrongAllocated 1.1x the real spend

No answer (API billing error)1 of 11

Query failedThe run stopped on an API billing error on our side

Was this post useful?
Nice one!

Thanks a lot for your feedback! If you’d like a member of our support team to respond to you, please send a message here

Please try again

Oops! Something went wrong while submitting the form.