LiveSQLBench Base-Lite has two kinds of questions: 180 read questions that pull or analyze data, and 90 write questions that change the database. Refix does read-only database analysis today, so this article looks at the 180 read questions. Every read score below is out of 180.
The official grader gave Refix credit for 78 of the 180. After we fixed defects in the benchmark and reviewed the answers the grader rejected, our latest provisional count is 134 of 180, or 74.4%.
That adjusted count comes from going back after the run and checking what each answer actually means. It sits next to the official score, not in place of it. A query can answer the question and still fail the grader. It can also run fine and answer the wrong question.
TL;DR
- The official grader gave Refix 78/180, or 43.3%, on PostgreSQL read questions.
- Rescored against our corrected version of the benchmark, the same answers get 115/180, or 63.9%.
- We reviewed all 102 official read failures one by one. Holding back two disputed credits, the provisional adjusted count is 134/180, or 74.4%.
- Refix still made real mistakes, like mixing up related metrics and duplicating values through joins.
- The benchmark asks for a SQL query. Real use also means explaining, clarifying and answering follow-up questions.
The first result was 43.3%
LiveSQLBench Base-Lite has 270 tasks across 18 PostgreSQL databases. It comes with schemas, column descriptions, business definitions, reference SQL, and a grader that runs both queries and compares the results.
We ran the 180 read questions through Refix. The official grader accepted 78.
78 / 180 = 43.3%
The saved run also had the write tasks, which puts the full benchmark result at 140/270, or 51.85%. We leave writes out of this article because Refix only does read-only database analysis right now.
43.3% looked bad, so we went through every failure. Some were real Refix mistakes. But a lot were problems with the answer key, the grader, or questions you could fairly read more than one way.
The answer key couldn’t pass its own test
We ran the benchmark’s own reference answers through its official grader. They scored 264/270 against themselves.
One reference has invalid SQL. Five use the current date, so they now return no rows, and the grader rejects an empty result even when both sides are empty.
The grader also rewrites the SQL before running it. It strips DISTINCT and ROUND from both the reference and the submitted query. In 38 of the 180 read tasks, that changes the reference’s own result. It also turns PostgreSQL’s IS NOT DISTINCT FROM into invalid syntax.
A blind audit flagged 37 read references as defective. A separate, independent review then took a sample of eleven and reproduced ten clear defects. We found wrong joins, missing filters, wrong formulas, rounding done too early, and outputs that didn’t match what the question asked.
So the raw score mixes three things: problems with the benchmark, ambiguous questions, and Refix’s own mistakes.
| Outcome | What happened |
|---|---|
| The reference was wrong | Refix answered the question as written, and the answer key did something else. |
| The grader rejected a matching result | The values matched, but the order or format was different. |
| The question allowed more than one reading | The benchmark picked one without saying so. |
| The answer couldn’t be supported | Missing information made a supported answer impossible. |
| Refix was wrong | The query used the wrong metric, join, filter or time window. |
What each score measures
| Measure | Result | What it means |
|---|---|---|
| Official read score | 78/180, 43.3% | Accepted by the official grader, unchanged. |
| Retrospective corrected benchmark | 115/180, 63.9% | The same saved answers, rescored after our fixes to the benchmark and comparison rules. |
| Provisional review-adjusted count | 134/180, 74.4% | The original passes plus 56 rejected answers we accepted after review. Two disputed credits held back. |
All three use the same saved Refix answers. Nothing was rerun. We ran the evaluation ourselves and didn’t submit it to the official leaderboard.
The 74.4% comes from a review we did after seeing the failures. Some of our earlier comparison rules accepted answers that were actually wrong, so a mechanical match on its own doesn’t prove an answer is right. The adjusted count stays provisional until more independent checks are done.
We keep the official number next to it so you can see how much the method changes the result. It’s the same with metrics like ARPPU: the population and the definition have to stay consistent, even when changing them would give you a nicer number.
Where Refix gave the more useful answer
Several rejected answers followed the question more closely than the reference did. That difference shows up as soon as someone tries to use the result.
It returned the customer ID
Five credit tasks asked for customer IDs. The references returned an internal record key. Refix returned the field the schema actually describes as the customer ID.
We recalculated the rows and values for all five tasks on our own. Once you use the requested ID, they match.
If you’re looking those customers up in another system, you need the customer ID. An internal record key is for something else.
It worked out the daily average before filtering
One question asked for sessions with above-average daily engagement, and wanted the daily average shown next to each result.
Refix worked out the average across every session that day, then picked the sessions above it. The reference filtered first, then reported the average of only the sessions it had already picked.
Both queries returned 161 sessions. Refix’s daily average was 0.02097, across the whole day. The reference reported about 0.10, because it only averaged the high-engagement sessions.
If you’re comparing a session to a typical day, you need the whole day as the baseline. Averaging only the sessions you picked answers a different question.
It used the relationship the data supports
Several archaeology references joined child records by project alone. But the data also has an operator key. Joining on project alone matched one operator’s scan with another operator’s measurements, and repeated entities.
Refix joined on both project and operator, so each scan stayed with the right operator’s measurements. Independent calculations reproduced its results for the affected tasks.
It followed the written formula
In a vaccine maintenance task, the reference divided a whole-number incident count by 10. Integer division wiped out the incident effect on the score. Refix used decimal division and matched the formula as written.
Where Refix tripped on definitions and data caveats
Some failures came down to choices the question left open, like which population to average over, or how to weight several records for one person. Refix can ask the user to settle those before it commits to one reading.
Others were mistakes it should have caught. In one crypto task, Refix correctly found whale orders using liquidity at the best price. But it then returned that same number where the question asked for order-book depth. A follow-up could explain the difference and fix it, but Refix should have caught it before showing the result.
In an insider-trading task, a join repeated a transaction before an average was calculated. A later attempt caught the duplicate and fixed that number, though another average was still off. Refix needs to check what each row represents before it combines records.
A virtual-economy task raised a different issue: how a 90-day activity window applies to the customer group asked about. Our reviewers disagreed on the reading. In a conversation, that’s something Refix can point out and ask the user to decide. While it’s unresolved, we give it no credit in the adjusted count.
In real use, a conversation gives Refix a chance to clarify or correct an answer. We haven’t shown that it would fix every failure on its own.
The benchmark wants SQL, but teams want answers
The benchmark wants one SQL query and grades the result against a reference. What Refix usually gives you is an answer to a business question, with the explanation and context you need to use it.
That can take more than one query. It can also mean asking which definition your team uses, explaining missing data, or revising an answer after a follow-up. A SQL-only submission leaves little room for any of that. When Refix asks a question instead of returning SQL, the benchmark records no answer, even if asking was the useful next step.
Some questions in this run could be answered without clarifying anything. Even so, the score measures a narrower job than the full conversation Refix supports.
Why real work gives Refix more room to be useful
LiveSQLBench’s single-turn setup and hidden expected results make it reproducible. They also leave out a lot of how a team actually works with Refix over time.
In production, Refix can say which customer population it used and point out missing relationships. It can ask whether an average should weight businesses or reviews, and return customer IDs someone will recognize. When the data can’t support a requested split, it can explain what’s missing.
Refix also learns from those conversations. It can remember business definitions and preferences, keep corrections, and reuse analysis methods that already worked. The next question can start from context the team has already settled. This benchmark run didn’t measure that, and it doesn’t give us a real-world accuracy rate.
The benchmark counts a clarifying question as no answer. A team might well prefer that to a polished number built on an assumption nobody checked.
We test Refix as a read-only agent because it’s built to investigate data without changing the database. Actions in connected tools follow the team’s approval settings. Folding in database-write tasks Refix doesn’t support would make the headline answer a different product question.
