Eval-Driven Development
The natural-language-to-SQL agent is backed by a first-class eval suite (evals/eval.ts, 20 cases drawn from real failure modes). Each case generates SQL with the same model and schema the app uses, runs it read-only against the 962k-row cadaster, and asserts on both the SQL text and the result shape.
From 85% to 100% in three runs
| Run | Score | What changed |
|---|---|---|
| 1 · Baseline | 17/20 (85%) | 3 real SQL-generation bugs surfaced |
| 2 · First fix | 18/20 (90%) | 2 bugs fixed; LLM non-determinism + a bad eval case exposed |
| 3 · Strengthened | 20/20 (100%) | general prompt rule + corrected eval case |
The three real bugs it caught
None were toy cases — each was a genuine failure of the agent's generated SQL that no user had reported.
UNION branch with ORDER BY / LIMIT
Question: “What is the oldest tree in Treptow-Köpenick?”
syntax error at or near "UNION"
A SELECT carrying its own ORDER BY/LIMIT must be parenthesised before UNION ALL. The agent emitted bare branches.
Heavy geom column on a large result set
Question: “List every tree in Köpenick, all of them”
canceling statement due to statement timeout (15s)
Serialising the PostGIS geom column for ~78k rows blows the timeout. Row-level output only needs lat/lng.
ORDER BY after UNION referencing a non-selected column
Question: “Trees within 200 m of a point”
column "geom" does not exist
After a UNION, ORDER BY can only reference columns in the combined SELECT list. Distance must be selected as an aliased column first.
What the loop taught
- The suite caught three real production bugs no user had reported.
- It also caught a poorly-specified eval case — refining the eval is part of the loop, not separate from it.
- LLM non-determinism means a single passing run is not proof: a class of bug (bare UNION + ORDER BY) hid in a case that happened to pass once. The fix was a forceful, general prompt rule, not a case-specific patch.