TL;DR
We benchmarked seven AI providers on 25 analytical questions over a live 381,523-row manufacturing database. The Oriona agent was the only one to answer all 25 correctly.
A correct answer from Oriona cost $0.0046. The same answer cost 9.7x more from GPT-5.5, 10.7x more from Claude Opus 4.8, and 17.5x more from Claude Sonnet 5.
Oriona runs a cost-efficient open-weight model. The margin came from agent engineering, not model spend: schema discovery, deterministic SQL, a calculator tool, and answer verification.
Among raw models, price per correct answer spanned roughly 90x while accuracy varied by only 8 points. Paying more does not buy correctness.
Most enterprise AI budgets are measured in tokens. That is the wrong unit.
What a business actually buys when it deploys an AI assistant over its data is a correct answer to a real question. Tokens are just the input. And once you start measuring cost per correct answer instead of cost per million tokens, the ranking of AI providers changes completely.
On July 7, 2026, our team ran an experiment to find out exactly how much that unit costs. We put seven providers through the same exam: 25 analytical questions over a live 381,523-row manufacturing database, graded blind by an LLM judge. The Oriona agent was the only provider to answer 25 of 25 correctly, and it did so at roughly one tenth the cost per correct answer of the best frontier models in the field.
In this article, we’ll break down how the benchmark worked, what the numbers showed, and why the winning margin came from engineering rather than model scale.
The Benchmark: Same Database, Same Questions, Seven Providers
The setup was deliberately fair to the competition.
Every provider was pointed at the same PostgreSQL database: a demo deployment modeled on a packaging manufacturer operating 19 sites, holding 381,523 rows across 14 tables. Production runs, inventory snapshots, sales order items, downtime events, energy readings, complaints, machines, products, customers, and multi-currency FX rates.
Answering the questions well required genuine analytical SQL: joins across five or more tables, aggregations over calendar windows, and currency conversion through the FX table.
The 25 questions spanned three tiers. Simple lookups (“How many manufacturing sites does the company operate?”). Precise aggregations with strict output contracts (“total downtime hours caused by breakdowns in calendar year 2025, rounded to 2 decimal places”). And one open diagnostic that asked why the night shift at two Indian plants was running roughly 2.8x the day-shift scrap rate while a third plant showed no gap, and what it would cost to close it.
The field included three raw frontier models (GPT-5.5, Claude Opus 4.8, Claude Sonnet 5) and three raw open-weight models (Kimi K2.7, DeepSeek V4 Pro, DeepSeek V4 Flash), each given direct database access through a generic harness. This was not an agent competing against a blindfolded model. Every competitor could explore the schema and run its own queries. It was Oriona’s scaffolding against a competent generic harness, on identical data.
The Results: 100% Correct, at One Tenth the Cost
Oriona answered 25 of 25 correctly. No other provider did. Here is the field, ranked by cost per correct answer against Oriona:
Provider | Correct | Cost / correct | vs Oriona |
|---|---|---|---|
Oriona Agent | 25 / 25 | $0.0046 | 1.0x |
GPT-5.5 (raw) | 24 / 25 | $0.0441 | 9.7x |
Claude Opus 4.8 (raw) | 24 / 25 | $0.0489 | 10.7x |
Claude Sonnet 5 (raw) | 23 / 25 | $0.0797 | 17.5x |
Kimi K2.7 (raw) | 23 / 25 | $0.0089 | 1.9x |
DeepSeek V4 Pro (raw) | 22 / 25 | $0.0020 | 0.4x |
DeepSeek V4 Flash (raw) | 22 / 25 | $0.0009 | 0.2x |
Three things stand out.
Accuracy did not follow price. Among the raw models, cost per correct answer spans roughly 90x between the cheapest and the most expensive, yet their accuracy varies by only 8 points. Buying a more expensive model moved spend by an order of magnitude. It did not move correctness.
Oriona posted the highest judge score of the field, 0.934 against 0.851 for Claude Opus 4.8, the best raw model, and a perfect 1.000 on the factuality check alone.
The whole Oriona run cost $0.114. The Claude Sonnet 5 run cost $1.833 and got fewer answers right.
Why the Agent Won
Across the field, 12 answers failed. They cluster into three patterns, and each one maps to a piece of scaffolding that prevents it.
Precise aggregation failures. Questions like “average breakdown duration in hours, rounded to 2 decimal places” tripped Kimi K2.7 and both DeepSeek models. A raw harness computes and rounds inside the model, where language models routinely fumble arithmetic. Oriona routes it to a calculator tool, which makes the arithmetic deterministic.
Distinct-count failures. “How many customers have filed at least one complaint” caught both Claude models, typically an SQL subtlety like counting rows instead of distinct entities. Oriona runs an answer verification pass that re-checks the query against the question’s exact contract before responding.
The open diagnostic. The night-shift scrap question defeated three of the seven providers, because it requires chaining six or more queries into a single causal narrative. Oriona’s planning loop is built for exactly that decomposition.
The pattern is consistent. A raw model with database access can find the data. What it lacks is the discipline around the data: deterministic arithmetic, contract checking against the question’s exact wording, and a verification pass before answering. Those three account for the gap between Oriona’s 100% and the 88% to 96% every raw model managed.
And here is the part worth sitting with: Oriona runs on a cost-efficient open-weight model, not a frontier one. The margin came from engineering, not model spend.
An Honest Reading
Two raw models did undercut Oriona on price. DeepSeek V4 Flash posted $0.0009 per correct answer.
If your workload tolerates one wrong answer in every eight, and nobody acts on individual answers, that is a defensible choice. But its accuracy ceiling here was 88%, with the field’s lowest judge score and its slowest responses, and it failed the diagnostic question outright. A single wrong revenue figure in a board pack costs more than 10,000 Oriona answers.
The honest framing is a frontier, not a knockout. Below 1¢ per correct answer, Oriona is the only provider that also clears 95% accuracy. Nothing else in the field clears both bars.
Latency is the one dimension where frontier models led. Claude Opus 4.8 averaged 15.5 seconds per answer against Oriona’s 26.1. The agent spends that time on schema discovery, query execution, and verification, which is precisely where the extra accuracy points come from. For interactive analytics, 26 seconds sits well inside a usable answer loop.
We should also be clear about what the benchmark does not show. It is one dataset in one domain, a single head-to-head run, and grading is LLM-judged, which puts a shared noise floor under every number. The full paper states all three limits in detail, and the dataset and question set are available on request so the run can be reproduced.
What This Means for Your Team
If you are evaluating an enterprise AI assistant, the question to bring to every vendor is not “which model do you use?” It is “what does a correct answer cost, and how do you know it is correct?”
Model choice is the smallest lever in that equation. Schema discovery, deterministic SQL execution, tool-based arithmetic, and answer verification are the ones that decide whether your team can trust the number in front of them. And per-answer cost metering is what turns “trust” into something you can put on a budget line.
The full white paper includes the complete methodology, all seven provider results, both cost-accuracy charts, sample questions, and our full statement of the benchmark’s limits.