AI data analytics benchmark: 9 agents, one messy database

Disclosure: We commissioned this benchmark at Supaboard, and we're a participant. Claude designed and ran it for us; a claude configuration is also included in this comparison. These are vendor-reported results from our own repository. We have not published an independent grading audit. (Benchmark notes)

The same customers, different answers

The Data Analytics Index is an AI data analytics benchmark comparing agents on one messy SaaS database. Its most useful finding is the disagreement over customer identity: which records belong to the same business, which accounts count as paying, and how much revenue they represent. (Benchmark report)

Which paying customers are missing from the CRM?

An operations team could use the answer to check account coverage, assign owners or repair incomplete records. Before anyone acts, the agent has to match records across billing and CRM.

The answer key identifies 214 customers representing $59,658.33 in monthly recurring revenue. Supaboard received full credit. Hex reported 191 customers, Basedash 263, and ChatGPT 305. An account-repair task would begin with a different work list depending on which answer the team accepted. (CRM-gap results)

What stood out

• Customer matching and revenue segmentation exposed larger gaps than several headline metrics.
• Reported accuracy includes partial credit; a matching figure can appear in a sensitivity table rather than the headline.
• The nine configurations have 123 recorded attempts; we use one attempt-selection rule for the regrade.

Jump to results, test setup, question difficulty, or scoring and attempts.

AI agent benchmark leaderboard

This article covers nine configurations from the broader Data Analytics Index; all comparisons and aggregate counts below use those nine. The claude entry uses the report's Sonnet 5.5 results; ChatGPT uses its GPT-6 results. (claude source record, ChatGPT source record)

These charts use the vendor-reported scores reviewed on 21 October 2026. Scoring and attempt selection remain under review; the charts show the published results.

Supaboard received 35 of 39 available points; claude received 31. (Reported leaderboard)

Time is the reported mean in seconds. The Data Analytics Index is the percentage of available rubric points plus ten, on a scale with a maximum of 110. (Scores, timings)

Data Analytics Index for the nine configurations covered here. It includes partial credit and is an index, not a percentage of questions answered exactly. (Scoring method)

Metabase's zero-point result (an index of 10) reflects the rule requiring the agent to execute its queries: on many tasks, it returned SQL without running it. The transcript separately records results after Claude clicked Run; those evaluator-run results were excluded from Metabase's score. (Metabase results)

Both charts use the published scores and timing averages. (Reported scores, response times)

What's in the data?

Corvia Systems is a fictional company selling scheduling and dispatch software to clinics and field-service businesses. Its Postgres database contains roughly 20 tables covering billing, CRM, product usage, support, legacy billing and reference data. (Database description)

The records are deliberately messy. Systems lack clean shared keys, some records are duplicated or inconsistent, and definitions must be inferred from the data. Agents received database access without documentation, hints or a semantic layer. (Database description)

This tests how an agent investigates unfamiliar data. An evaluation with approved metric definitions and curated models would test a different workflow.

Each agent answered the same 13 questions, with each question in a fresh chat and the same short company introduction. (Question protocol) The questions covered recurring revenue, retention, receivables, sales, customer identity, usage and support. Some exchanges included a clarification instructing the agent to use its judgment and explain its assumptions. (recorded Supaboard answers)

Access was not identical. claude and ChatGPT used Neon connectors with owner access restricted to the benchmark project; the other agents shared a read-only database role. The report says no agent wrote to the database. (Access notes)

Customer identity was harder than several headline metrics

Across these nine configurations, annual recurring revenue (ARR) by industry had the lowest average score: 0.33 out of 3, with none receiving full credit. The CRM-gap question averaged 0.78, with full credit for Supaboard alone. These averages are recalculated from the published question scores. (Question difficulty)

By comparison, the cancellation-spike and annual-billings questions each earned full credit for 8 of these 9 configurations, while net revenue retention earned full credit for 7. (Question difficulty) Success on retention or billings did not guarantee success on customer reconciliation.

Selected tasks for the nine configurations in this article, recalculated from vendor-reported scores. Full credit follows the benchmark's rounding and alternative-answer rules. (Data Analytics Index)

For ARR by industry, the answer key puts Healthcare at 31.0%; the reported answers include 14.15% from Basedash, 29.16% from Supaboard, 36.5% from Lightdash and 47.2% from claude. (ARR-by-industry results) Agreement on the industry label concealed substantial disagreement about its share of revenue. The published comparison does not isolate the cause of each mismatch, so the figures alone cannot tell us which join or definition went wrong.

Even the simple-sounding customer-count question was difficult: Supaboard alone received full credit for the answer-key count of 863 paying customers. (Customer-count results) A buyer evaluating AI data analytics tools should include questions about who counts as a customer, alongside questions about how much revenue those customers generate.

Supaboard's strongest answers and its misses

Supaboard matched the CRM-gap answer and received full credit on customer count. Its misses matter just as much when those questions are part of your weekly review. (CRM-gap results, customer-count results)

Its weakest result was ARR by industry: 29.16% against the answer key's 31.0%, earning 1 point for a reasonable, correctly calculated interpretation under the rubric. (ARR-by-industry results)

For appointments, the original score table used Supaboard's rerun figure of 142,668 against the September all-appointments key of 139,943 and awarded 2 points. (Appointments results) It also received 2 points for enterprise support responsiveness, a question on which no agent received full credit. (Support results)

For seat utilization, Supaboard's headline was "0.57". Its full-credit figure of about 57.52% comes from the uncapped active-customer row dated 15 September in a sensitivity table. The rubric allows matching alternatives, which explains the credit for this sensitivity-table result. The month-end headline still does not exactly match the key. Before accepting that mid-month row in the regrade, we need to check the snapshot required by the full prompt. (Supaboard Q10, seat-utilization results)

Faster answers did not settle the comparison

Basedash had the shortest reported mean response time at 68 seconds, compared with 98 seconds for claude and 192 seconds for Supaboard. (Response-time results) Basedash also received full credit on retention, the cancellation spike, recognized revenue, annual billings and sales performance. (Question-level results) Its speed deserves consideration alongside its misses on customer reconciliation and industry ARR.

Those averages use the published mix of attempts. Supaboard Q6's timer is from its rerun, and Sigma Q7's 55 seconds is also explicitly a rerun timer. We will recalculate speed comparisons from the selected first conversations instead of carrying those averages into a regraded table. Platform timers and SQL-approval waits still need consistent treatment. (Supaboard Q6, Sigma Q7, timing method)

How the scoring works?

Each answer earns up to three points: three when the key figure appears, including in an alternative or sensitivity table; two for a figure within five percent; one for a reasonable stated method with a result checked against the database; and zero otherwise. Counts must match for full credit. Money and rates can be within 0.1% or match at the precision used in the answer. (Scoring and precision rules)

The maximum is 39 points. Divide the total by 39 and multiply by 100 for the normalized score, then add ten for the Data Analytics Index, whose maximum is 110. A score of 89.7% therefore means 89.7% of available rubric points, not that 89.7% of questions were answered exactly. (Index calculation)

We publish answer summaries and full agent responses. Full prompts, the database and the answer key are available to reviewers on request; repeated-run variability is not reported. This is a vendor-run evaluation on one synthetic database, with the conflicts of interest disclosed above. (Protocol and access notes)

One attempt-selection rule for every agent

For the regrade, we select the first completed recorded conversation for each question. Standardized clarification turns stay within that attempt. The final response and its supporting tables determine the score; an explicitly superseded earlier recommendation cannot rescue the final answer. A completed interaction that returns unexecuted SQL still counts as an attempt. Later attempts stay visible but cannot replace the selected one because they score better.

For these nine configurations, the published records contain 123 attempts across 117 agent-question pairs: six pairs have two attempts. These counts describe the published record, not a complete platform event log. (Full answer records)

Supaboard has second attempts on Q6, Q11 and Q13. Under this rule, its first conversations are selected in all three cases, including Q13's 863-customer answer. Dot Q1, Lightdash Q10 and Sigma Q7 also select their first conversations. (Supaboard records, Dot records, Lightdash records, Sigma records)

We cannot carry Supaboard's Q11 grade across automatically. Its first conversation states 80,467 human appointments and separately 10,459 Zapier appointments for September; the credited 142,668 total is explicit in the rerun. The first conversation references a chart whose full payload is absent from the README. We need that payload or a database check before assigning its replacement grade. We must also match timings to the selected attempts before publishing revised speed averages. (Supaboard Q11)

Questions to ask before using the benchmark

What does this AI data analysis benchmark measure?

It compares answer quality under the reported setup. Building a live dashboard, producing the weekly report and delivering it on schedule are separate jobs. The benchmark did not score those workflows. Evaluate them separately before treating similar answer scores as evidence that two products do the same work. (Question list)

Why compare general-purpose assistants with BI products?

Both can answer business questions from a connected database. The useful comparison is the evidence each returns for the same question. When choosing a product, also test the work around that answer: configuration, collaboration, recurring reporting and the effort required to keep it useful as your data changes.

Does full credit mean the headline answer matched?

Full credit can come from an alternative answer or a sensitivity table. For a team that reads only the summary, that distinction matters. Check where the matching result appears, which definition the agent recommends, and whether a reader can understand that recommendation without inspecting every table. (Scoring rules)

Can I reproduce the results?

We make the full prompts, database and answer key available to reviewers on request. Our public repository contains the recorded responses and scores. For a reproducible regrade, we also need an explicit attempt ledger, complete grading criteria and timings tied to the selected conversations. (Access notes)

What would make the Metabase result more informative?

A follow-up should first verify that the agent can execute queries in the intended configuration, then rerun the same questions and record the setup. Report that result separately from this run. The existing zero scores the agent's completed answers; the evaluator's later query executions are recorded in the transcript but excluded from the score. (Metabase responses)

Should I supply my business definitions?

Use the configuration you intend to rely on. A test with no documentation measures investigation of unfamiliar data. A test with approved definitions measures whether the agent applies your rules consistently. Record the definitions, permissions and accepted answers before running either test, so a later correction does not quietly change the comparison.

Test the whole weekly review

Supaboard is the one AI agent that does the whole analyst job on your own database. After an answer, the same agent can build a dashboard, prepare a report or deck, and run the recurring analysis on a schedule. Those capabilities are separate from the answer-quality results above. (Dashboards, Automations)

Give it seven days with one weekly review you already run. Bring twenty questions whose answers you know, teach the agent your definitions, and score its answers with evaluations. Then check whether it can produce and deliver the review you need. That tests both the answers and the work that follows them.

Start Free, or book a 30-minute demo to scope that evaluation. Seven days is the suggested evaluation window; access and AI allowances follow the current plan.