Skip to content
Start Free

Test your agent on the questions that matter.

Pair real questions with the SQL that answers them, then test how your agent answers. Inspect what fails, refine its context, and rerun the same questions to see what changed.

Start FreeBook a 30-minute demo

Free plan. No credit card required.

Finance agentInteractive example

6 passing

Test case

Passing

What was net revenue in September?

Example discrepancy

Reference
$112,000
Agent’s answer
$112,000

Matches the reference.

The change

Rule clarified: subtract refunds issued in the period.

Sample data. Results on these six test cases only.

How it works

  1. Write the eval

    A question, the SQL that answers it, and its source.

  2. Run it

    Each answer is graded 0–100; 70 or more passes.

  3. Refine and rerun

    Update a rule or an example, then run the same evals again.

Start with questions whose answers you know

Pick the questions your team asks every month and the tricky ones: refunds, test accounts, reporting periods. Add each with the SQL that answers it, and include variations your examples don’t cover.

  • A question, the expected SQL and its source
  • Save & Run checks the SQL straight away

Test cases

  • What was net revenue in September?Refunds
  • How many paying customers did we have in September?Test accounts
  • What did we invoice on the last day of September?Period
  • What was revenue in our first fiscal quarter?
  • What was our largest invoice in September?
  • How did net revenue change from August?

Sample data. Each case pairs a question with the SQL that answers it.

See what needs attention

Each run grades the agent’s answer to every eval from 0 to 100 against your SQL, and 70 or above passes. A failing eval comes with a note on how its answer differed, pointing to the rule, example or table note to look at.

  • Failing, Passing and Not run, at a glance
  • A note on how each answer differed
Finance agentInteractive example

September net revenue Failing: below 70

What differed: the answer’s query didn’t subtract refunds.

Look at: the net revenue rule, and whether an example covers refunds.

Sample data. Example eval and discrepancy.

Issue detail: refunds weren’t subtracted.

Change the agent. Compare the next run.

Refine the context, then rerun the same evals to see what moved. With five evals the agent shows its accuracy on the set you’ve written, so keep adding questions as your team asks them.

  • Accuracy shows once there are five evals
  • Run history keeps every run to compare

Run history

  1. Run A5 of 6 passing
  2. Clarified the refund rule
  3. Run B6 of 6 passing

Sample data. The same six cases in both runs.

Frequently asked questions

What is an eval?

A question your team asks, the SQL that answers it correctly, and the source it runs on.

How is an eval scored?

The agent answers the question, and its query is graded from 0 to 100 against yours. 70 or above is a pass.

What does the accuracy figure mean?

Once an agent has five evals, it shows accuracy: the average score across them. It describes the questions you’ve written, so add new ones as your team asks them.

What should I include?

Common questions, your important definitions, and cases that have caused mistakes. Add variations that weren’t used as examples when you set up the agent.

What should I do when an eval fails?

Read its note and check the expected SQL, then update the rule, example or table note it points to and run the evals again.

Put your agent’s answers to the test

Start FreeBook a 30-minute demo

Free plan. No credit card required.

Settings