We gave 11 AI models a real business's Google Ads, GA4 and Search Console data and asked 16 questions with one right answer. On easy questions every model was right. As the questions got harder, accuracy fell to 45%, and the wrong answers looked exactly like the right ones. Then we ran the same questions through a data model.
The data
What every model could query
Homer's Donuts' GA4, Google Ads and Search Console data for August and September 2026.
GA47 metrics · 8 dimensions
Metrics
Dimensions
Tables
Google Ads5 metrics · 9 dimensions
Metrics
Dimensions
Tables
Search Console4 metrics · 6 dimensions
Metrics
Dimensions
Tables
Each question used either the tidy tables a connector produces or Google's own BigQuery export, the same for all 11 models.
The benchmark
As questions get harder, raw SQL gets less accurate. A data model holds accuracy up.
16 questions with one right answer, asked once to each of 11 models on one real account's data.
One table, one obvious metric
The answer sits in a single table at the right level of detail, and the question means one thing.
6 questions · e.g. Google Ads ROAS, monthly users, sessions
A rule the schema doesn't state
A ratio has to be rebuilt from totals, a snapshot table joined once, or positions counted from one, not zero.
6 questions · e.g. average position, ROAS by campaign on Google's export · a miss on four of the six
Data the tables can't combine correctly
Metrics that can't be added up, dates stored differently in two tables, spend split across pages, sessions matched to campaigns.
4 questions · e.g. users by channel (4 of 11), Search spend by device (8 of 11)
In Hard, ROAS by landing page has no single right answer on either side, because no source records spend by page, so it isn't counted as right.
Read it top to bottom. The purple bars are what you get today by pointing an AI at your tables. The green bars are the same models, with the same questions, asking Jepto's data model instead.
A thread on r/analytics recently asked what AI is and isn't replacing in analytics work. The most upvoted reply wasn't about jobs. It asked who catches the incorrect join, the flawed aggregation or the result that's silently wrong when nobody reads the code. The hard part isn't writing the query anymore. It's knowing whether you can trust the number that comes back.
So we measured it. Not "can AI write SQL", which it can, but where it stops being safe to trust the answer without checking, and what it takes to move that line.
We'll call the business Homer's Donuts, a donut shop in Springfield that sells online, runs five Google Ads campaigns and tracks its site with GA4 and Search Console. The name is ours; every number is real data from a real business's account for September 2026.
Each model worked as an agent, the way most people use AI with data today: the same lightweight agent loop for every model (built with Strands Agents), the schema in the prompt, a run_sql tool, up to 15 queries, and an answer table at the end. The data sat in two layouts: the tidy tables a connector produces, and the tables Google's own BigQuery exports create. Every question was asked once per model, the way you'd ask it for real.
The models: Claude Opus 5.5, Sonnet 5.5 and Haiku 5.5, GPT-5.5, Gemini 3.1 Pro and 3.8 Flash, Grok 4.7, Kimi K3, GLM 5.3, DeepSeek V4.1 Flash and Qwen 3.8 Max, all at default settings.
Then we sorted the 16 questions by how many models got them right. Three levels fell out, and each one has something in common. Questions with no single right answer, like "how many leads did we get?", can't be scored this way, so they have their own post.
Google Ads ROAS for September was 2.29. All 11 models said so. GA4 counted 1,576 users for the month, and all 11 found the monthly table rather than adding up days. Spend, sessions and revenue by campaign, August against September, website sessions and organic clicks: 66 answers, 66 right.
Easy questions
How many models got each easy question right
11 models, one run each. Questions without a note used the tidy connector tables.
What was our Google Ads ROAS in September?
How many users visited the website in September?
For each Google Ads campaign, show September spend, website sessions and website revenue.
Which Google Ads campaigns improved their ROAS from August to September? Show each campaign's spend, conversions, cost per conversion and ROAS in both months.
How many sessions did the website have in September? (Google's export)
How many clicks did we get from Google organic search in September? (Google's export)
What these have in common is that the answer lives in one place, at the right level of detail, and the question can only mean one thing. If your AI questions look like this, raw SQL is fine, and it's the cheapest way to ask.
Medium questions need one piece of knowledge the tables don't carry. Average position over a month has to be weighted by impressions. Google's Search Console export counts positions from zero. A snapshot table repeats each campaign once a day, so it has to be joined once, not 30 times.
Medium questions
How many models got each medium question right
11 models, one run each. Questions without a note used the tidy connector tables.
What was our organic search click-through rate and average position in September?
How many conversions did paid search drive in September, and what was the cost per conversion?
What was the average click-through rate of our Google Ads campaigns in September?
How many users visited the website in September? (Google's export)
What was the ROAS of each Google Ads campaign in September? (Google's export)
What was our average position in Google organic search in September? (Google's export)
Most models knew the rule: 61 of 66 answers were right. The misses are the problem. GLM 5.3 calculated the impression-weighted position, 12.43, then reported a plain average of the days, 11.96. DeepSeek and Qwen reported 11.43 from Google's export because they didn't add one. Each of those came back as a clean, single number, and nothing in it tells you it's the wrong one. One miss did say what it was doing: asked for the average click-through rate of the campaigns, GLM averaged the five campaigns' rates, 8.32%, and said so. That's a fair reading of "average", but it isn't the account's click-through rate of 2.55%, so we scored it as a different definition.
Hard questions need two tables combined in a way the obvious query gets wrong, or can't get right at all. Only 20 of 44 answers were right.
Hard questions
How many models got each hard question right
11 models, one run each. Questions without a note used the tidy connector tables.
How much did we spend on Search campaigns (not Shopping) on each device in September? (Google's export)
For each Google Ads campaign, show September spend, the website sessions it drove and the website revenue from those sessions. (Google's export)
How many users did each channel bring to the website in September, and how many users did we have in total?
Which landing pages got the best return on our Google Ads spend in September?
Each of the four goes wrong in a different way.
Users by channel, with total. Asked how many users each channel brought in September, plus the total, 4 of 11 got it right. The total is in a monthly table: 1,576. Users by channel only exist per day, and daily users can't be added into a month, because the same person visits on several days. Ten models put 1,576 in the total row and filled the channels by adding up days, so their channel rows add up to 2,016. Claude Opus, Sonnet and Haiku and Qwen said the channel rows were overstated because the same person is counted again for each day they visit. Three more noticed the rows were too high but gave the wrong reason or the wrong total, and GPT-5.5, both Geminis and Grok didn't mention it. Rows above the total aren't proof on their own, because one person can arrive through two channels: GA4's own monthly figures by channel add up to 1,612 against a total of 1,575. The mistake is adding up days, which counts a person again for every day they came back. These tables have no monthly users by channel, so on raw SQL the right answer was the total plus a note that the channels can't be split exactly.
Search spend by device. Google's export keeps campaign settings in one table and daily stats in another, and both have a column that looks like a date filter. Three models applied the snapshot filter that belongs on the campaign table to the stats table too. That kept only 30 September and threw away the other 29 days, so their desktop figure was $18.81 instead of $1,462.55. The next section shows how that got past three models agreeing.
Sessions and revenue by campaign. This one joins Google Ads spend to GA4 sessions and revenue, which sit in two different exports. Three models missed in three different ways. Gemini 3.8 Flash's joins matched nothing, so every campaign came back with $0 spend, 0 sessions and $0 revenue. Grok 4.7 got spend and sessions right, then reported Google Ads' conversion value as website revenue: $1,486.50 for Brand, where GA4 records none. Qwen 3.8 Max ran out of query budget with its final query written but never run, so it returned no answer.
ROAS by landing page. This can't be answered exactly, on either side. In the tables the models had, Google Ads records spend by campaign and GA4 records revenue by landing page, but nothing records spend by page. To get a ROAS for each page, a model has to decide how to share each campaign's spend across the pages it sent visitors to, and every way of sharing gives a different answer. Six models ended up on a homepage ROAS of 0.45 and two on 0.42. Gemini 3.8 Flash reported 0.46, but its figures aren't what its own query returns. One allocated $2,816 of spend against the account's real $2,481, and Claude Haiku 5.5's run stopped on an API billing error on our side, so it has no answer. None counts as right, because there's no true figure to score against.
Hard questions also cost more. At each provider's list prices, the median hard answer cost 6 cents and took six or seven queries, against 2 cents for an easy one. ROAS by landing page took a median of twelve queries. Paying more didn't buy accuracy either: on the seven export questions, Claude Haiku 5.5 got all seven right for 2 cents in total, and Qwen 3.8 Max spent 57 cents to get five.
Hard questions aren't only a problem of accuracy. The wrong answers look the same as the right ones, so the usual ways of checking don't work. Search spend by device shows why. Here's what happened.
The verification problem
Search spend by device in September: how a wrong answer gets checked
Real answers from our test on Google's raw BigQuery export tables. Eight models got $1,462.55; three models independently converged on $18.81.
Transparency
“The agent shows me its SQL, so I can audit it.”
SELECT s.segments_device AS device,
SUM(s.metrics_cost_micros) / 1e6 AS spend
FROM google_ads.ads_CampaignBasicStats_5550001111 s
JOIN google_ads.ads_Campaign_5550001111 c
ON s.campaign_id = c.campaign_id
WHERE c._DATA_DATE = c._LATEST_DATE
AND s._DATA_DATE = s._LATEST_DATE
AND c.campaign_advertising_channel_type = 'SEARCH'
AND s.segments_date BETWEEN '2026-09-01' AND '2026-09-30'
GROUP BY 1Ask yourself: Ten clean lines that ran without an error. Can you tell from reading them whether the answer is right?
Agreement
“I'll have two more models run it independently.”
The catch: Three models, three providers, three separately written queries, identical to the cent. It feels like independent confirmation. What if they all made the same mistake?
The reveal
Three models agree. All three are off by 98.5%.
SELECT s.segments_device AS device,
SUM(s.metrics_cost_micros) / 1e6 AS spend
FROM google_ads.ads_CampaignBasicStats_5550001111 s
JOIN google_ads.ads_Campaign_5550001111 c
ON s.campaign_id = c.campaign_id
WHERE c._DATA_DATE = c._LATEST_DATE
AND s._DATA_DATE = s._LATEST_DATE -- keeps 30 September only
AND c.campaign_advertising_channel_type = 'SEARCH'
AND s.segments_date BETWEEN '2026-09-01' AND '2026-09-30'
GROUP BY 1What went wrong: All three applied the campaign snapshot filter (_DATA_DATE = _LATEST_DATE) to the daily stats table too. On the stats table, that column is the partition date, so the filter kept 30 September and threw away the other 29 days.
Real verification
Check one total against a source the models didn't produce.
The lesson: One total from the platform, for the dates you meant, exposes the mistake in seconds. A second model only checks whether another LLM writes the same SQL.
Seeing the SQL told you what the models did, not whether it was right. Three models agreeing told you they made the same choice, not that it was correct. Only the platform's own total exposed the mistake. And correct answers don't help you check either: all 11 models got website sessions right with 11 different queries, so there's no reference query to compare against.
A data model, or semantic layer, is the knowledge from each level written down once: which table holds monthly users, how position is weighted, how the Google Ads export stores its dates, what you count as a lead. The AI asks for googleAds.roas or googleAnalytics4.totalUsers by name, and the rule comes with it.
That's why the green bars stay up. Easy questions don't change much, because there was little to get wrong. Hard questions go from 45% to 73%: users by channel went from 4 of 11 to 11 of 11, because the data model reads users by channel from GA4 for the whole period instead of adding up days. Search spend by device is one query instead of a join across two snapshot tables, and 10 of 11 got it; the one miss, DeepSeek, left Shopping in. ROAS by landing page is the only Hard question no model gets right either way, because it needs spend split across pages that no source records.
Medium goes from 92% to 94%. The misses were on users from Google's export: GPT-5.5, Gemini 3.8 Flash, Kimi and GLM asked the data model for active users, 1,563, instead of total users, 1,575. Active users is a real GA4 metric, but it isn't the total the question asked for. Medium also has one judgement call. Asked how many conversions paid search drove, four of the eleven models read "paid search" as Search campaigns only and left out Shopping, for 60 conversions instead of 80. One model did the same on raw SQL. Google calls Search campaigns paid search too, so we counted both readings as right. That's a definition, not a calculation error: a data model applies the definition you've written down, but it can't pick one for you. We look at how the models pick definitions in how AI models choose marketing metrics.
Here it is model by model: how many of the 16 questions each AI got right, with and without the data model.
Every model
Each AI with and without the data model
16 questions with one right answer, asked once to each model on each side. Most accurate on the data model first.
Opus 5.5 · MediumCorrect
Sonnet 5.5 · High
Haiku 5.5 · Medium
Gemini 3.1 Pro · High
Qwen 3.8 Max · Xhigh
Grok 4.7 · High
Kimi K3 · Max
DeepSeek V4.1 Flash · Default
GLM 5.3 · Default
GPT-5.5 · Medium
Gemini 3.8 Flash · Medium
Every model ran at its API's default reasoning effort, shown next to its name, with no temperature or reasoning settings changed.
It also changes what you have to check. To verify the SQL answer to "which campaigns improved their ROAS from August to September?", you read 16 expressions. To verify the data model query, you check three things: the metrics, the months and the grouping.
Same question, two answers
What you have to know to trust each one
SELECT
campaign_name AS campaign,
SUM(CASE WHEN date >= '2026-08-01' AND date <= '2026-08-31' THEN cost END) AS aug_spend,
SUM(CASE WHEN date >= '2026-08-01' AND date <= '2026-08-31' THEN conversions END) AS aug_conversions,
SUM(CASE WHEN date >= '2026-08-01' AND date <= '2026-08-31' THEN cost END) /
SUM(CASE WHEN date >= '2026-08-01' AND date <= '2026-08-31' THEN conversions END) AS aug_cost_per_conversion,
SUM(CASE WHEN date >= '2026-08-01' AND date <= '2026-08-31' THEN conversions_value END) /
SUM(CASE WHEN date >= '2026-08-01' AND date <= '2026-08-31' THEN cost END) AS aug_roas,
SUM(CASE WHEN date >= '2026-09-01' AND date <= '2026-09-30' THEN cost END) AS sep_spend,
SUM(CASE WHEN date >= '2026-09-01' AND date <= '2026-09-30' THEN conversions END) AS sep_conversions,
SUM(CASE WHEN date >= '2026-09-01' AND date <= '2026-09-30' THEN cost END) /
SUM(CASE WHEN date >= '2026-09-01' AND date <= '2026-09-30' THEN conversions END) AS sep_cost_per_conversion,
SUM(CASE WHEN date >= '2026-09-01' AND date <= '2026-09-30' THEN conversions_value END) /
SUM(CASE WHEN date >= '2026-09-01' AND date <= '2026-09-30' THEN cost END) AS sep_roas,
(SUM(CASE WHEN date >= '2026-09-01' AND date <= '2026-09-30' THEN conversions_value END) /
SUM(CASE WHEN date >= '2026-09-01' AND date <= '2026-09-30' THEN cost END)) >
(SUM(CASE WHEN date >= '2026-08-01' AND date <= '2026-08-31' THEN conversions_value END) /
SUM(CASE WHEN date >= '2026-08-01' AND date <= '2026-08-31' THEN cost END)) AS improved
FROM google_ads_campaign_daily
GROUP BY campaign_name
ORDER BY campaign;6 things to check
improved compare September with August, not the reverse?{
"measures": [
"googleAds.spend",
"googleAds.conversions",
"googleAds.costPerConversion",
"googleAds.roas"
],
"dimensions": [
"googleAds.campaignName"
],
"startDate": "2026-08-01",
"endDate": "2026-09-30",
"granularity": "month"
}3 things to check
Both are the queries Gemini 3.8 Flash wrote in our test, one with raw SQL and one with the data model. Both answers were right.
It isn't cheaper per answer yet. Across the eleven models, a data model answer cost a median of 5 cents on easy and medium questions and 11 cents on hard ones, against 2 cents and 6 cents for raw SQL, because the agent looks up the schema and reads fuller responses before it answers. A few cents per question is small. A budget cut made on the wrong answer isn't, and that's the cost that matters.
Cube found a similar effect in their paired benchmark across three frontier models: a semantic layer added 17 to 23 points of accuracy, and with it the three models scored about the same.
We made the longer argument in You don't need a more expensive LLM. You need a data model. This is the evidence for it.
Checklist
Six checks before you trust an AI-written number
One business and one month, so treat the percentages as illustrations, not a universal benchmark. Questions were sorted into levels by how the 11 models did on them, then grouped by what they have in common. Where a question had more than one defensible answer, we counted a different definition as not right and show it in the appendix. The one exception is paid search conversions, where we counted both "all Google Ads campaigns" and "Search campaigns only" as right. We made that call after seeing the results. Scored the other way, the data model gets 64% on that question and 88% on Medium, and raw SQL gets 91% on both. Average CTR went the other way: GLM's average of the campaigns' rates is defensible, and we counted it as not right.
The connector-style tables and the Google Ads and Search Console exports hold the account's real figures. The GA4 export was rebuilt from the property's real daily totals, so sessions and revenue match GA4 exactly; we scored those questions against what the export itself supports. Campaign names, page paths and search terms have been relabelled to fit the name; the figures and the logic of every query are unchanged. Claude ran through the Anthropic API, GPT through OpenAI, Gemini through Google, Grok through xAI, and Kimi, GLM, DeepSeek and Qwen through Fireworks. The SQL ran in DuckDB.
The data model runs used Jepto's production MCP server, the same one available to ChatGPT and Claude users, on 28 and 30 October 2026, with all eleven models. Data model costs leave out the tokens of skill documents, which production loads directly rather than through tool calls.
Pick a question to see every distinct result the models returned and the exact SQL behind each one.
Answer explorer
Same question, different answers, and the exact SQL behind each one
“Which landing pages got the best return on our Google Ads spend in September?”
True answer: no single true answer: spend has to be split across pages · 11 answers, 5 different results
Different definitionOne way of splitting spend across pages
Different definitionOne way of splitting spend across pages
WrongReported figures its own final query doesn't produce
WrongAllocated 1.1x the real spend
Query failedThe run stopped on an API billing error on our side
Thanks a lot for your feedback! If you’d like a member of our support team to respond to you, please send a message here
Oops! Something went wrong while submitting the form.