FAFelipe Alves
Data & AI Lab · conversational BI and evaluation

Ask the Ledger

When someone asks a finance question in plain language, does the system return the right figure, and does it say so when it cannot?

Typeexperiment
DataSynthetic · 2026-10-29
LicenceNone needed. Generated by this repository's own code (MIT).
Running costUS$0 to start on Cloudflare's free Workers AI allowance (10,000 neurons a day, read on Cloudflare's pricing page on 2026-10-29). Neurons per question have not been measured. Claude Haiku is not switched on.

The question

When someone asks a finance question in plain language, does the system return the right figure, and does it say so when it cannot? A finance team that wants staff to query the ledger without an analyst checking every answer would pay to know.

The live piece

Pick a question from the test set below. The page shows what the translator did with it, the figure it returned, and the right answer. Use the filter to see only the questions it got wrong. That part runs from saved output; nothing is sent anywhere.

This piece needs JavaScript to load the saved results. The full results are in data/ask-the-ledger/results.json in the repository.

Ask your own question (live box)

A small Worker, ledger-api, takes one question, asks a model for one read-only SQL query, checks the query itself, runs it on a copy of the same synthetic ledger and shows the query, the rows and a short explanation. Limits: 2 questions per visitor per day, 50 a day for everyone, 300 characters, 100 rows. When a limit is hit or the Worker cannot be reached, the recorded session below stays on the page.

This piece needs JavaScript. The recorded session is in data/ledger-api/recorded-session.json.

Recorded session

Loading the recorded session.

Model scorecard

The same 47 questions, but each is sent to the model, which must write SQL. The query goes through the Worker’s validator, runs on the ledger, and the result is compared with the hand-checked answer.

Loading the scorecard.

Method and results

The ledger is invented. 1,008 rows: six made-up companies, eight made-up accounts, 2025-01 to 2026-09, in thousands of AED. It comes from a fixed random seed, so the same numbers appear every run.

The semantic layer writes each business word down once. A question can only be answered if it maps onto one of these definitions. Words that could mean two things, such as “profit” or “costs”, are not mapped: the system must ask back.

MetricDefinitionWords it accepts
RevenueSum of all accounts in the Revenue group.revenue, sales, turnover
Cost of salesSum of all accounts in the Cost of sales group.cost of sales, cogs, cost of goods sold, direct costs
Gross profitRevenue minus cost of sales.gross profit
Gross marginGross profit divided by revenue, as a percentage. Zero when revenue is zero.gross margin
Operating expensesSum of all accounts in the Operating expense group.operating expenses, operating costs, opex, overheads
Operating profitRevenue minus cost of sales minus operating expenses.operating profit, operating income, operating result

The translator is keyword rules. It turns a question into a query: metric, period, entity, grouping, ranking. It is not a language model. That is deliberate. The test set and the scoring work on any translator, so a language model can be plugged in later and scored the same way. I have not done that.

The test set has 47 questions in four kinds, each with the right outcome decided by hand. The true figure for each answer is computed from the raw rows by separate code that does not use the semantic layer, so the two cannot share a bug.

Results. The translator was right on 40 of 47 (30 correct figures, 10 correct refusals).

Kind of questionRightTotal
Plain questions in the translator’s own vocabulary3030
Same meaning, different words or shape04
Ambiguous: should ask back24
Not answerable from the ledger89

What failed. 5 wrong answers given with no warning, and 2 questions wrongly declined.

  • “How much did we sell” and “what did Orchard Foods bring in” were declined, because “sell” and “bring in” are not in the word list.
  • “Revenue for the first half of 2026” was answered as the whole year to date: the translator does not know “first half” and fell back to the full year.
  • “Revenue in 2025 excluding Delta Services” returned Delta Services’ revenue alone. The translator saw the name and filtered to it. It does not read “excluding”.
  • “Revenue in Q2” and “Revenue in March” were answered for 2026. A missing year defaults to the latest year, where a careful analyst would ask which year.
  • “Operating profit margin” was answered as operating profit. The translator matched the metric and ignored “margin”.

The wrong answers matter more than the wrong refusals. A declined question costs a minute. A confident wrong figure can reach a report.

To rerun it: node pipelines/ask-the-ledger/evaluate.mjs. A test checks that the saved results match a fresh run.

The live box and its checks. Every question passes six checks in order: origin (only felipealveslab.com), per-visitor cap, daily cap, model call, SQL validation, then the answer. The validator accepts exactly one statement that starts with SELECT or WITH, rejects write and admin keywords, comments and quoted names, allows only the ledger table, and adds LIMIT 100 when none is present. 41 tests cover the validator, the caps, the origin check and the scorer; they pass. Requests that reach the model count against the caps even if they fail. The caps are approximate under bursts, because the counters are eventually consistent.

What has not been run. The Worker has not been deployed or called from this build, because it needs Felipe’s Cloudflare resources. No model has written SQL for the test set, so the scorecard above has no accuracy figure. To produce one, with Cloudflare credentials in your own shell: node pipelines/ledger-api/generate-queries.mjs queries.json, then node pipelines/ledger-api/score-sql.mjs queries.json. The recorded session uses two queries written by hand, not by a model; their rows were computed by running them on the ledger.

Known limits and running cost

  • The data is synthetic. Nothing here says how a real ledger behaves: real ledgers have adjustments, currencies, restatements and messy account names.
  • One author wrote both the questions and the translator. The perfect score on plain questions mostly reflects that I wrote them in words the translator knows. It is not a benchmark and it does not show the translator is good.
  • 47 questions is small. One question moves the score by about two points.
  • No language model has been scored. The harness and the Worker are ready for one. No result for one exists yet.
  • A free model may write weaker SQL than Claude Haiku. The scorecard is meant to show by how much. Haiku is not switched on.
  • The validator is a rule check, not a proof. The database holds only synthetic data, so a bypass would expose nothing real.
  • Cost: US$0 to start. Cloudflare’s pricing page, opened on 2026-10-29, gives the free Workers AI allowance as 10,000 neurons a day on the free plan and says calls fail once it is used up, so nothing is billed. Neurons used per question have not been measured, so whether 50 questions a day fits is not confirmed. If Haiku is switched on later, the hard stop is a prepaid US$5 credit with auto-reload off. The rest of the page reads saved files and costs nothing.

How it was built

My take