An AI agent can pull the right data, run SQL that works, and still answer a slightly different question from the one you asked.
It’s a hard mistake to catch, because nothing looks wrong. The numbers are precise, the query runs, and the explanation makes sense.
Preliminary results
We’ve been testing Refix on DataAgentBench, a benchmark from UC Berkeley’s EPIC Data Lab and Hasura PromptQL. It has 54 questions across 12 datasets.
So far, across our development runs, Refix has answered 48 of the 54 correctly, or 88.9%, based on our own review.
To be clear about what that number is: it combines first runs with follow-up attempts, including some that used the benchmark’s official hints. It also counts answers we checked and found correct even though the grader rejected them, either for formatting or because it never saw the attached file. It’s not a single-run accuracy score, and it’s not an official leaderboard result.
An average can hide a choice
In one evaluation, we asked Refix for the average rating of businesses in a city. It found the businesses, matched their reviews, and worked out an average.
But before it ran anything, it had quietly made a call: every business counts the same.
That gives you the average of each business’s rating. Pool all the reviews instead and you get a different number, because businesses with more reviews count for more.
Both are fair. “Average rating” just doesn’t say which one you meant.
Where the assumption came in
The specialist that queried the data did exactly what it was told. The assumption came in one step earlier, when Refix’s analyst turned the question into a task.
The analyst reads the question and hands the work to specialists. The specialists go through the data and do the math.
So the handoff is where the meaning can shift. “Average each business equally” makes a choice that “find the average rating” leaves open.
We changed how the analyst handles this. What you actually asked for now stays separate from what it’s assuming. And when an assumption could change the answer, Refix should lay out the options or ask you one focused question.
In a later run, Refix came back with both numbers:
Now you can see what each number means.
This comes up with everyday business metrics all the time. It’s why our ARPPU guide spells out the denominator and the reporting window. A number is only useful when everyone agrees on who’s in it.
A shared account doesn’t prove who made the sale
Another question asked which sales agent brought in the most sales under a contract eligibility policy. Orders were linked to accounts. Opportunities were linked to accounts and contracts.
An early attempt matched orders to qualifying opportunities through the account they shared.
That’s not enough. One account can have several opportunities, so a shared account doesn’t tell you which opportunity led to which order.
The specialist flagged that limit, but the analyst named a winner anyway, and the data couldn’t back it up.
Checking the attribution
We made Refix stricter about checking the links behind a conclusion. In a follow-up run, it spotted the missing link and asked for clarification instead of guessing.
Getting the whole answer through
Some questions tested whether Refix could carry a long analysis all the way to the end.
For a patent analysis, Refix produced results for 23 classification groups, with their titles, codes and peak years.
It sent the full table as a CSV and summarized the top groups in its reply. The grader only reads text, so it missed the attachment. We checked the file ourselves and counted it as correct in our review.
For the person who asked, a full table is something they can check, share and actually use.
What we check before trusting an answer
These runs have sharpened what we look at before we accept an analysis:
| Check | What it tells us |
|---|---|
| Population | The calculation includes the right records. |
| Weighting | Each record counts the way the question intends. |
| Relationships | The data actually supports the link or attribution. |
| Coverage | Parsing and missing values haven’t quietly dropped data. |
| Completeness | The user gets the full result they need to use the answer. |
What’s still open
Six questions are still unresolved in our preliminary review. They’re real problems, not formatting: classification, grouping, weighting and sales attribution.
We’re preparing a full, repeated evaluation to publish, with execution traces you can inspect.
So far, Refix is getting through detailed analyses across very different datasets. And the misses are what’s making those answers more dependable, from how it reads the question to the result a team actually uses.
