Projects

Finding the page that answers a question about a 10-K

An answer about a filing is only as good as the pages retrieval puts in front of the model. FinanceBench labels the answer page for every question, so I measured retrieval on its own, one step at a time, on questions I never tuned on.

Benchmark
FinanceBench: 150 public questions over 360 SEC filings, 53,399 pages
Held out
100 questions kept out of all tuning, which used the other 50
Result
Hit@10 from 0.14 to 0.71 on the held-out questions with a company and period filter and statement routing, and 0.75 with the company name left out of the query (within noise)
Serving
BM25 on one core: 126 queries a second at 12 ms p95

How it narrows the search

Every 10-K has a balance sheet, a risk section and the same accounting terms, so BM25 over all 53,399 pages finds the right filing for only 42% of held-out questions. The first step isn't ranking: the filter reads the company, fiscal year and quarter from the question and keeps 2.5 of 360 filings on average. Questions about a financial statement then get that statement's pages first.

  1. Question“What was the largest liability in American Express's Balance Sheet in 2022?”
  2. Filter1of 360 filingscompany, fiscal year and quarter, read from the question
  3. BM25260pagesover that filing's pages, the company's name left out of the query
  4. Routing1balance sheet pagethe question names the balance sheet, so balance sheet pages go first
  5. ResultPage 98ranked firstthe page FinanceBench labels as the answer
Figure 1. One held-out question through the pipeline. With routing switched off, the same search doesn't have page 98 in its top 10.

What each step is worth

I tuned on 50 questions and kept 100 aside; everything here is the 100. Hit@10 asks whether the labeled page made the top 10, which is what a model answering from ten pages needs.

Show the table, with MRR and latency
100 held-out questions. Hit@10: the labeled page is in the top 10.
Each row adds one step to the row aboveHit@1095% intervalMRR@10p50 / p95
BM25 over all 53,399 pages0.140.08 to 0.210.0912.7 / 5.3 ms
Only filings from the company the question names0.170.10 to 0.250.1002.5 / 5.2 ms
Only that company's filings for the year asked about0.410.32 to 0.510.1552.3 / 5.0 ms
Narrowed to the quarter's filing when a quarter is asked about0.430.34 to 0.530.1582.2 / 5.1 ms
Statement questions sent to statement pages first0.710.62 to 0.800.4992.5 / 5.1 ms
Company's name left out of the BM25 query0.750.66 to 0.830.5342.4 / 4.8 ms
Same pipeline, MiniLM embeddings instead of BM250.710.62 to 0.800.52711 / 25 ms
Same pipeline, BM25 and MiniLM fused (my pick on the dev split)0.730.64 to 0.820.54213 / 30 ms
Figure 2. Hit@10 on the 100 held-out questions. Each blue row adds one step to the row above; the gray rows run the finished pipeline with MiniLM embeddings. Lines are 95% bootstrap intervals.

The filter found the company in 97 of 100 questions and kept the right filing for 96, taking Hit@10 from 0.14 to 0.43 (+0.29, 95% interval +0.20 to +0.38). Routing added as much again. 78 of the 150 answer pages are financial statements, tables of numbers that share almost no words with a question like "what is the FY2019 fixed asset turnover ratio". Routing fires on 62 held-out questions and took Hit@10 to 0.71 (+0.28, +0.19 to +0.37).

Leaving the company's name out of the query hurts without routing (0.43 to 0.33) and adds +0.04 with it, inside noise (-0.02 to +0.10). It's the step I'm least sure of.

MiniLM didn't earn its cost. Fused with BM25 it scored 0.73 against 0.75 (-0.02, -0.08 to +0.03) at 13 ms a query instead of 2.4. It looked better on the 50 dev questions (0.78 against 0.66), which is why I committed to it before testing; that gap was noise. Without the filter MiniLM does win (0.24 against 0.14) by finding the right filing more often (61% against 42%), and the filter makes that redundant.

Where it still misses

Each held-out question the hybrid pipeline missed is filed under the first stage that lost it. The rules run on every evaluation, so the counts follow the code.

Figure 3. The 27 held-out questions whose answer page missed the top 10, by the first stage of the pipeline that lost it.

The biggest group, narrative pages that share almost no words with the question, is the case a cross-encoder reranker is meant for. I tried one, and it made retrieval worse (below). Routing's own failures are few: 2 questions needed a statement page without saying so, and 1 routed to the right kind of statement but ranked a different statement page first.

What I'd do next: go through the 12 misses where a narrative answer page shares at most two words with the question, one at a time, before testing another reranker or a query rewrite. They are the largest group, but the one cross-encoder I tried made retrieval worse, so I don't know yet whether the fix lies in the model or in the query. Tuning against these 12 would use them up as test questions, and all 150 FinanceBench questions are already spent (50 on tuning, 100 here), so I'd need new questions before trusting any claimed improvement.

As a service

With one FastAPI worker pinned to one core and the load generator on the other, BM25 serves 126 requests a second to one client (p95 12 ms) and up to 162 with more clients; the hybrid tops out at 55. No request failed.

  • BM25
  • Hybrid: BM25 and MiniLM fused

Requests a second

p95 latency, ms

Show the numbers
RetrieverClientsRequests/sp50p95
BM2511267.6 ms12 ms
BM25214014 ms21 ms
BM25414427 ms45 ms
BM25815449 ms85 ms
BM251616295 ms156 ms
Hybrid14918 ms35 ms
Hybrid25135 ms69 ms
Hybrid45468 ms129 ms
Hybrid855135 ms247 ms
Hybrid1652299 ms483 ms
Figure 4. The FastAPI service with one worker pinned to one CPU core and the load generator on the other, 30 seconds per level.

At half an AWS c7i.large's on-demand price, one vCPU, that's $0.10 per million BM25 queries and $0.25 for the hybrid, if the throughput carries over to that machine, which I haven't measured.

Two more checks

Adding filings without a rebuild. The index is kept in segments. With 36 filings held out and added back in batches, every score on 300 queries matched a one-pass build exactly. A full build takes 13.1 s; a batch took 0.42 to 0.64 s.

Chunk size. Smaller passages didn't help: 650-token chunks tie whole pages (0.75), and 256- and 128-token chunks score 0.71 and 0.72 with two to four times as many entries.

Does a reranker help?

A cross-encoder reads the question and a page together, which is the usual fix for the narrative-page misses above. I ran BAAI/bge-reranker-base over the top 30 pages of three retrieval systems and re-sorted them by its score. It made the two systems that work worse and left the all-pages baseline within one question of where it was.

Show the numbers
100 held-out questions, top 30 re-sorted.
SystemHit@10MRR@10Hit@10 against no reranker
Full pipeline0.730.542
+ reranker, one window0.630.336-0.10, 95% interval -0.20 to -0.01
+ reranker, three windows0.570.314-0.16, 95% interval -0.26 to -0.06
BM25 with filter and routing + reranker0.630.387-0.12, 95% interval -0.21 to -0.04
BM25 over all pages + reranker0.130.081-0.01, 95% interval -0.03 to +0.00
Figure 5. Hit@10 on the 100 held-out questions. Blue rows are retrieval systems; gray rows re-sort each one's top 30 with a cross-encoder. Lines are 95% bootstrap intervals.

On the held-out questions the full pipeline's Hit@10 fell from 0.73 to 0.63 (-0.10, 95% interval -0.20 to -0.01; 7 questions better, 17 worse) and its MRR@10 from 0.542 to 0.336. The 50 dev questions agree: 0.78 to 0.62.

The model reads 512 tokens, so a long page is scored on its first part. I reran the full pipeline with up to three windows per page, scored as the best window. Hit@10 was 0.57, lower than one window (0.63) and -0.16, 95% interval -0.26 to -0.06 against no reranker, so truncation doesn't explain it.

Scoring took 367 ms a page with one window and 909 ms with three on a laptop CPU, so re-sorting 30 pages would add about 11 s to a query. The reported system stays without a reranker. This is one off-the-shelf model, not fine-tuned and not tuned on this data; I haven't tried others.

What the model does with the pages

Retrieval is a means. To see what it buys, gpt-6-luna answered each held-out question from four page sets, and a stronger model, gpt-6.1-sol, graded every answer against FinanceBench's reference. An answer of "not in these pages" counts as incorrect.

Show the numbers
100 held-out questions. A declined answer counts as incorrect.
Pages givenCorrect95% intervalDeclinedIncorrect
Gold pages0.830.75 to 0.90413
Reported system's top 50.660.57 to 0.752113
No pages0.370.27 to 0.472340
BM25 over all pages, top 50.140.08 to 0.21797
Figure 6. Share of the 100 held-out questions gpt-6-luna answered correctly, as judged by gpt-6.1-sol, with 95% bootstrap intervals. Blue rows get the pipeline's pages; gray rows are the baselines.

With the gold pages it got 83 of 100 right. With the pipeline's top 5 it got 66, a gap of 0.17 (95% interval 0.09 to 0.25; 18 questions better with gold pages, 1 worse). The loss sits in the questions where retrieval missed: when a top-5 page was a gold page (66 questions) it scored 0.83, and when none was (34) 0.32.

The pages are worth +0.29 over no pages at all (95% interval +0.16 to +0.41), not the whole 0.66, because the model already knows some figures for well-known companies and got 37 of 100 with none. Searching all pages with BM25 was worse than no pages: the model declined 79 of 100 questions.

A model grading a model needs a check, so I graded 60 of the judge's verdicts myself, from 51 questions, against the same rubric and FinanceBench's references. I used AI tools to find the filings and check figures, so this grade is research-assisted, not blind to AI input. The judge agreed on 59 of 60 (kappa 0.95, 95% interval 0.83 to 1.00). The one difference was a Corning working-capital answer I marked correct and the judge marked incorrect: the reference counts only operating items ($831 million), the answer used total current assets minus total current liabilities ($2,278 million), and both say positive.

The labels follow the references, which are debatable in places: one says Pfizer was still spinning off Upjohn in 2023, when it had finished in 2020, and another gives a JPMorgan per-share figure that is tangible book value. So accuracy here means agreement with FinanceBench. It's one run with one generator and one grader, and it cost $0.46.

Run it yourself

FinanceBench is pinned to a commit and every PDF is checked against its SHA-256, so a rerun scores the same pages.

make install
make data      # 150 questions, 360 PDFs (about 670 MB), page text
make embed     # MiniLM over every page: hours on a laptop CPU, resumable
make eval      # every configuration on dev and test, and RESULTS.md
make studies   # the incremental-index check and the chunk-size study
make loadtest

CI runs the tests on Python 3.11 and 3.13, a regression gate that fails if Hit@10 on the dev questions drops by more than one question, a smoke test of the Docker image, a secret scan, and the browser port's parity test.

Retrieval numbers from the code at commit 95d3fe9, run on 2026-10-21. The reranker and generation runs are from 2026-10-28, on commit cf57ef1 with the study code applied but not yet committed; that code is in commit 34c4145 and the results in 3ea5084. Every results file and the command that wrote it: RESULTS.md.