Finding the page that answers a question about a 10-K
An answer about a filing is only as good as the pages retrieval puts in front of the model. FinanceBench labels the answer page for every question, so I measured retrieval on its own, one step at a time, on questions I never tuned on.
- Benchmark
- FinanceBench: 150 public questions over 360 SEC filings, 53,399 pages
- Held out
- 100 questions kept out of all tuning, which used the other 50
- Result
- Hit@10 from 0.14 to 0.71 on the held-out questions with a company and period filter and statement routing, and 0.75 with the company name left out of the query (within noise)
- Serving
- BM25 on one core: 126 queries a second at 12 ms p95
Ask a 10-K a question
The repo's BM25 pipeline, ported to JavaScript. The switches turn off the last two steps in Figure 2; turn off the first one on this question and the income statement drops out of the results.
- (it misses this one)
- 360 filings
- 1 filing: Coca-Cola 2021 10-K
- 183 pages
- income statement first
- top 3 by BM25
FinanceBench question 09724, one of the 100 held-out questions, labels page 62 as the answer. Here it's result 1. In the benchmark run, BM25 over all 53,399 pages put it below 100 and the hybrid with MiniLM at 1. Expected answer: 39.7%.
-
1Page 62Income statementLabeled answerBM25 1.01
…SUBSIDIARIES CONSOLIDATED STATEMENTS OF INCOME (In millions except per share data) Year Ended December 31, 2021 2020 2019 Net Operating Revenues $ 38,655 $ 33,014 $ 37,266 Cost of goods sold 15,357 13,433 14,619 Gross Profit 23,298 19,581 22,647 Selling, general and administrative expenses 12,144 9,…
Whole page
THE COCA-COLA COMPANY AND SUBSIDIARIES CONSOLIDATED STATEMENTS OF INCOME (In millions except per share data) Year Ended December 31, 2021 2020 2019 Net Operating Revenues $ 38,655 $ 33,014 $ 37,266 Cost of goods sold 15,357 13,433 14,619 Gross Profit 23,298 19,581 22,647 Selling, general and administrative expenses 12,144 9,731 12,103 Other operating charges 846 853 458 Operating Income 10,308 8,997 10,086 Interest income 276 370 563 Interest expense 1,597 1,437 946 Equity income (loss) — net 1,438 978 1,049 Other income (loss) — net 2,000 841 34 Income Before Income Taxes 12,425 9,749 10,786 Income taxes 2,621 1,981 1,801 Consolidated Net Income 9,804 7,768 8,985 Less: Net income (loss) attributable to noncontrolling interests 33 21 65 Net Income Attributable to Shareowners of The Coca-Cola Company $ 9,771 $ 7,747 $ 8,920 Basic Net Income Per Share $ 2.26 $ 1.80 $ 2.09 Diluted Net Income Per Share $ 2.25 $ 1.79 $ 2.07 Average Shares Outstanding — Basic 4,315 4,295 4,276 Effect of dilutive securities 25 28 38 Average Shares Outstanding — Diluted 4,340 4,323 4,314 Calculated based on net income attributable to shareowners of The Coca-Cola Company. Refer to Notes to Consolidated Financial Statements. 1 1 1 60 -
2Page 132BM25 5.27
…it Matters” in the Company’s 2022 Proxy Statement is incorporated herein by reference. Part IV ITEM 15. EXHIBITS AND FINANCIAL STATEMENT SCHEDULES (a) The following documents are filed as part of this report: 1. Financial Statements: Consolidated Statements of Income — Years Ended December 31, 2021,…
Whole page
ITEM 13. CERTAIN RELATIONSHIPS AND RELATED TRANSACTIONS, AND DIRECTOR INDEPENDENCE The information under the subheading “Director Independence and Related Person Transactions” under the principal heading “Governance” in the Company’s 2022 Proxy Statement is incorporated herein by reference. ITEM 14. PRINCIPAL ACCOUNTANT FEES AND SERVICES The information regarding Audit Fees, Audit-Related Fees, Tax Fees, All Other Fees and Audit Committee Pre-Approval of Audit and Permissible Non-Audit Services of Independent Auditors under the subheading “Item 3 Ratification of the Appointment of Ernst & Young LLP as Independent Auditors” under the principal heading “Audit Matters” in the Company’s 2022 Proxy Statement is incorporated herein by reference. Part IV ITEM 15. EXHIBITS AND FINANCIAL STATEMENT SCHEDULES (a) The following documents are filed as part of this report: 1. Financial Statements: Consolidated Statements of Income — Years Ended December 31, 2021, 2020 and 2019. Consolidated Statements of Comprehensive Income — Years Ended December 31, 2021, 2020 and 2019. Consolidated Balance Sheets — December 31, 2021 and 2020. Consolidated Statements of Cash Flows — Years Ended December 31, 2021, 2020 and 2019. Consolidated Statements of Shareowners’ Equity — Years Ended December 31, 2021, 2020 and 2019. Notes to Consolidated Financial Statements. Report of Independent Registered Public Accounting Firm. Report of Independent Registered Public Accounting Firm on Internal Control Over Financial Reporting. 2. Financial Statement Schedules: The schedules for which provision is made in the applicable accounting regulations of the Securities and Exchange Commission (“SEC”) are not required under the related instructions or are inapplicable and, therefore, have been omitted. 3. Exhibits: In reviewing the agreements included as exhibits to this report, please remember they are included to provide you with information regarding their terms and are not intended to provide any other factual or disclosure information about the Company or the other parties to the agreements. The agreements contain representations, warranties, covenants and conditions by or of each of the parties to the applicable agreement. These representations, warranties, covenants and conditions have been made solely for the benefit of the other parties to the applicable agreement and: • should not in all instances be treated as categorical statements of fact, but rather as a way of allocating the risk to one of the parties if those statements prove to be inaccurate; • may have been qualified by disclosures that were made to the other party in connection with the negotiation of the applicable agreement, which disclosures are not necessarily reflected in the agreement; • may apply standards of materiality in a way that is different from what may be viewed as material to you or other investors; and • were made only as of the date of the applicable agreement or such other date or dates as may be specified in the agreement and are subject to more recent developments. Accordingly, these representations and warranties may not describe the actual state of affairs as of the date they were made or at any other time. Additional information about the Company may be found elsewhere in this report and the Company’s other public filings, which are available without charge through the SEC’s website at http://www.sec.gov. 130 -
3Page 42BM25 5.13
…ur variable consideration, we generally calculate our estimates using a portfolio approach at the country and product line level rather than at the individual contract level. The result of making these estimates will impact the line items trade accounts receivable and accounts payable and accrued ex…
Whole page
Revenue Recognition Revenue is recognized when performance obligations under the terms of the contracts with our customers are satisfied. Our performance obligation generally consists of the promise to sell concentrates, syrups or finished products to our bottling partners, wholesalers, distributors or retailers. Control of the concentrates, syrups or finished products is transferred upon shipment to, or receipt at, our customers’ locations, as determined by the specific terms of the contract. Upon transfer of control to the customer, which completes our performance obligation, revenue is recognized. Our sales terms generally do not allow for a right of return except for matters related to any manufacturing defects on our part. After completion of our performance obligation, we have an unconditional right to consideration as outlined in the contract. Our receivables will generally be collected in less than six months, in accordance with the underlying payment terms. All of our performance obligations under the terms of contracts with our customers have an original duration of one year or less. Our customers and bottling partners may be entitled to cash discounts, funds for promotional and marketing activities, volume‑based incentive programs, support for infrastructure programs and other similar programs. In most markets, in an effort to allow our Company and our bottling partners to grow together through shared value, aligned financial objectives and the flexibility necessary to meet consumers’ always changing needs and tastes, we have implemented an incidence-based concentrate pricing model. Under this model, the concentrate price we charge is impacted by a number of factors, including, but not limited to, bottler pricing, the channels in which the finished products produced from the concentrates are sold, and package mix. The amounts associated with the arrangements described above represent variable consideration, an estimate of which is included in the transaction price as a component of net operating revenues in our consolidated statement of income upon completion of our performance obligations. The total revenue recorded, including any variable consideration, cannot exceed the amount for which it is probable that a significant reversal will not occur when uncertainties related to variability are resolved. As a result, we are recognizing revenue based on our faithful depiction of the consideration that we expect to receive. In making our estimates of variable consideration, we consider past results and make significant assumptions related to: (1) customer sales volumes; (2) customer ending inventories; (3) customer selling price per unit; (4) selling channels; and (5) discount rates, rebates and other pricing allowances, as applicable. In gathering data to estimate our variable consideration, we generally calculate our estimates using a portfolio approach at the country and product line level rather than at the individual contract level. The result of making these estimates will impact the line items trade accounts receivable and accounts payable and accrued expenses in our consolidated balance sheet. The actual amounts ultimately paid and/or received may be different from our estimates. Income Taxes Our annual effective tax rate is based on our income and the tax laws in the various jurisdictions in which we operate. Significant judgment is required in determining our annual income tax expense and in evaluating our tax positions. We establish reserves to remove some or all of the tax benefit of any of our tax positions at the time we determine that the position becomes uncertain based upon one of the following conditions: (1) the tax position is not “more likely than not” to be sustained; (2) the tax position is “more likely than not” to be sustained, but for a lesser amount; or (3) the tax position is “more likely than not” to be sustained, but not in the financial period in which the tax position was originally taken. For purposes of evaluating whether or not a tax position is uncertain, (1) we presume the tax position will be examined by the relevant taxing authority that has full knowledge of all relevant information; (2) the technical merits of a tax position are derived from authorities such as legislation and statutes, legislative intent, regulations, rulings and caselaw and their applicability to the facts and circumstances of the tax position; and (3) each tax position is evaluated without consideration of the possibility of offset or aggregation with other tax positions taken. We adjust these reserves, including any impact on the related interest and penalties, in light of changing facts and circumstances, such as the progress of a tax audit. Refer to the heading “Operations Review — Income Taxes” below and Note 14 of Notes to Consolidated Financial Statements. A number of years may elapse before a particular uncertain tax position is audited and finally resolved. The number of years subject to tax audits or tax assessments varies depending on the tax jurisdiction. The tax benefit that has been previously reserved because of a failure to meet the “more likely than not” recognition threshold would be recognized in income tax expense in the first interim period when the uncertainty disappears under any one of the following conditions: (1) the tax position is “more likely than not” to be sustained; (2) the tax position, amount, and/or timing is ultimately settled through negotiation or litigation; or (3) the statute of limitations for the tax position has expired. Settlement of any particular issue would usually require the use of cash. Refer to Note 11 of Notes to Consolidated Financial Statements. Tax laws require certain items to be included in the tax return at different times than when these items are reflected in the consolidated financial statements. As a result, the annual effective tax rate reflected in our consolidated financial statements is different from that reported in our tax return (our cash tax rate). Some of these differences are permanent, such as expenses that are not deductible in our tax return, and some differences reverse over time, such as depreciation expense. These timing differences create deferred tax assets and liabilities. Deferred tax assets and liabilities are determined based on temporary 40
It covers the 84 filings that FinanceBench questions cite. On all 438 question and setting pairs the tests run, it returns the same top 10 pages as the Python pipeline, with the same scores to the last bit.
How it narrows the search
Every 10-K has a balance sheet, a risk section and the same accounting terms, so BM25 over all 53,399 pages finds the right filing for only 42% of held-out questions. The first step isn't ranking: the filter reads the company, fiscal year and quarter from the question and keeps 2.5 of 360 filings on average. Questions about a financial statement then get that statement's pages first.
- Question“What was the largest liability in American Express's Balance Sheet in 2022?”
- Filter1of 360 filingscompany, fiscal year and quarter, read from the question
- BM25260pagesover that filing's pages, the company's name left out of the query
- Routing1balance sheet pagethe question names the balance sheet, so balance sheet pages go first
- ResultPage 98ranked firstthe page FinanceBench labels as the answer
What each step is worth
I tuned on 50 questions and kept 100 aside; everything here is the 100. Hit@10 asks whether the labeled page made the top 10, which is what a model answering from ten pages needs.
Show the table, with MRR and latency
| Each row adds one step to the row above | Hit@10 | 95% interval | MRR@10 | p50 / p95 |
|---|---|---|---|---|
| BM25 over all 53,399 pages | 0.14 | 0.08 to 0.21 | 0.091 | 2.7 / 5.3 ms |
| Only filings from the company the question names | 0.17 | 0.10 to 0.25 | 0.100 | 2.5 / 5.2 ms |
| Only that company's filings for the year asked about | 0.41 | 0.32 to 0.51 | 0.155 | 2.3 / 5.0 ms |
| Narrowed to the quarter's filing when a quarter is asked about | 0.43 | 0.34 to 0.53 | 0.158 | 2.2 / 5.1 ms |
| Statement questions sent to statement pages first | 0.71 | 0.62 to 0.80 | 0.499 | 2.5 / 5.1 ms |
| Company's name left out of the BM25 query | 0.75 | 0.66 to 0.83 | 0.534 | 2.4 / 4.8 ms |
| Same pipeline, MiniLM embeddings instead of BM25 | 0.71 | 0.62 to 0.80 | 0.527 | 11 / 25 ms |
| Same pipeline, BM25 and MiniLM fused (my pick on the dev split) | 0.73 | 0.64 to 0.82 | 0.542 | 13 / 30 ms |
The filter found the company in 97 of 100 questions and kept the right filing for 96, taking Hit@10 from 0.14 to 0.43 (+0.29, 95% interval +0.20 to +0.38). Routing added as much again. 78 of the 150 answer pages are financial statements, tables of numbers that share almost no words with a question like "what is the FY2019 fixed asset turnover ratio". Routing fires on 62 held-out questions and took Hit@10 to 0.71 (+0.28, +0.19 to +0.37).
Leaving the company's name out of the query hurts without routing (0.43 to 0.33) and adds +0.04 with it, inside noise (-0.02 to +0.10). It's the step I'm least sure of.
MiniLM didn't earn its cost. Fused with BM25 it scored 0.73 against 0.75 (-0.02, -0.08 to +0.03) at 13 ms a query instead of 2.4. It looked better on the 50 dev questions (0.78 against 0.66), which is why I committed to it before testing; that gap was noise. Without the filter MiniLM does win (0.24 against 0.14) by finding the right filing more often (61% against 42%), and the filter makes that redundant.
Where it still misses
Each held-out question the hybrid pipeline missed is filed under the first stage that lost it. The rules run on every evaluation, so the counts follow the code.
The biggest group, narrative pages that share almost no words with the question, is the case a cross-encoder reranker is meant for. I tried one, and it made retrieval worse (below). Routing's own failures are few: 2 questions needed a statement page without saying so, and 1 routed to the right kind of statement but ranked a different statement page first.
What I'd do next: go through the 12 misses where a narrative answer page shares at most two words with the question, one at a time, before testing another reranker or a query rewrite. They are the largest group, but the one cross-encoder I tried made retrieval worse, so I don't know yet whether the fix lies in the model or in the query. Tuning against these 12 would use them up as test questions, and all 150 FinanceBench questions are already spent (50 on tuning, 100 here), so I'd need new questions before trusting any claimed improvement.
As a service
With one FastAPI worker pinned to one core and the load generator on the other, BM25 serves 126 requests a second to one client (p95 12 ms) and up to 162 with more clients; the hybrid tops out at 55. No request failed.
- BM25
- Hybrid: BM25 and MiniLM fused
Requests a second
p95 latency, ms
Show the numbers
| Retriever | Clients | Requests/s | p50 | p95 |
|---|---|---|---|---|
| BM25 | 1 | 126 | 7.6 ms | 12 ms |
| BM25 | 2 | 140 | 14 ms | 21 ms |
| BM25 | 4 | 144 | 27 ms | 45 ms |
| BM25 | 8 | 154 | 49 ms | 85 ms |
| BM25 | 16 | 162 | 95 ms | 156 ms |
| Hybrid | 1 | 49 | 18 ms | 35 ms |
| Hybrid | 2 | 51 | 35 ms | 69 ms |
| Hybrid | 4 | 54 | 68 ms | 129 ms |
| Hybrid | 8 | 55 | 135 ms | 247 ms |
| Hybrid | 16 | 52 | 299 ms | 483 ms |
At half an AWS c7i.large's on-demand price, one vCPU, that's $0.10 per million BM25 queries and $0.25 for the hybrid, if the throughput carries over to that machine, which I haven't measured.
Two more checks
Adding filings without a rebuild. The index is kept in segments. With 36 filings held out and added back in batches, every score on 300 queries matched a one-pass build exactly. A full build takes 13.1 s; a batch took 0.42 to 0.64 s.
Chunk size. Smaller passages didn't help: 650-token chunks tie whole pages (0.75), and 256- and 128-token chunks score 0.71 and 0.72 with two to four times as many entries.
Does a reranker help?
A cross-encoder reads the question and a page together, which is the usual fix for the narrative-page misses above. I ran BAAI/bge-reranker-base over the top 30 pages of three retrieval systems and re-sorted them by its score. It made the two systems that work worse and left the all-pages baseline within one question of where it was.
Show the numbers
| System | Hit@10 | MRR@10 | Hit@10 against no reranker |
|---|---|---|---|
| Full pipeline | 0.73 | 0.542 | |
| + reranker, one window | 0.63 | 0.336 | -0.10, 95% interval -0.20 to -0.01 |
| + reranker, three windows | 0.57 | 0.314 | -0.16, 95% interval -0.26 to -0.06 |
| BM25 with filter and routing + reranker | 0.63 | 0.387 | -0.12, 95% interval -0.21 to -0.04 |
| BM25 over all pages + reranker | 0.13 | 0.081 | -0.01, 95% interval -0.03 to +0.00 |
On the held-out questions the full pipeline's Hit@10 fell from 0.73 to 0.63 (-0.10, 95% interval -0.20 to -0.01; 7 questions better, 17 worse) and its MRR@10 from 0.542 to 0.336. The 50 dev questions agree: 0.78 to 0.62.
The model reads 512 tokens, so a long page is scored on its first part. I reran the full pipeline with up to three windows per page, scored as the best window. Hit@10 was 0.57, lower than one window (0.63) and -0.16, 95% interval -0.26 to -0.06 against no reranker, so truncation doesn't explain it.
Scoring took 367 ms a page with one window and 909 ms with three on a laptop CPU, so re-sorting 30 pages would add about 11 s to a query. The reported system stays without a reranker. This is one off-the-shelf model, not fine-tuned and not tuned on this data; I haven't tried others.
What the model does with the pages
Retrieval is a means. To see what it buys, gpt-6-luna answered each held-out question from four page sets, and a stronger model, gpt-6.1-sol, graded every answer against FinanceBench's reference. An answer of "not in these pages" counts as incorrect.
Show the numbers
| Pages given | Correct | 95% interval | Declined | Incorrect |
|---|---|---|---|---|
| Gold pages | 0.83 | 0.75 to 0.90 | 4 | 13 |
| Reported system's top 5 | 0.66 | 0.57 to 0.75 | 21 | 13 |
| No pages | 0.37 | 0.27 to 0.47 | 23 | 40 |
| BM25 over all pages, top 5 | 0.14 | 0.08 to 0.21 | 79 | 7 |
With the gold pages it got 83 of 100 right. With the pipeline's top 5 it got 66, a gap of 0.17 (95% interval 0.09 to 0.25; 18 questions better with gold pages, 1 worse). The loss sits in the questions where retrieval missed: when a top-5 page was a gold page (66 questions) it scored 0.83, and when none was (34) 0.32.
The pages are worth +0.29 over no pages at all (95% interval +0.16 to +0.41), not the whole 0.66, because the model already knows some figures for well-known companies and got 37 of 100 with none. Searching all pages with BM25 was worse than no pages: the model declined 79 of 100 questions.
A model grading a model needs a check, so I graded 60 of the judge's verdicts myself, from 51 questions, against the same rubric and FinanceBench's references. I used AI tools to find the filings and check figures, so this grade is research-assisted, not blind to AI input. The judge agreed on 59 of 60 (kappa 0.95, 95% interval 0.83 to 1.00). The one difference was a Corning working-capital answer I marked correct and the judge marked incorrect: the reference counts only operating items ($831 million), the answer used total current assets minus total current liabilities ($2,278 million), and both say positive.
The labels follow the references, which are debatable in places: one says Pfizer was still spinning off Upjohn in 2023, when it had finished in 2020, and another gives a JPMorgan per-share figure that is tangible book value. So accuracy here means agreement with FinanceBench. It's one run with one generator and one grader, and it cost $0.46.
Run it yourself
FinanceBench is pinned to a commit and every PDF is checked against its SHA-256, so a rerun scores the same pages.
make install
make data # 150 questions, 360 PDFs (about 670 MB), page text
make embed # MiniLM over every page: hours on a laptop CPU, resumable
make eval # every configuration on dev and test, and RESULTS.md
make studies # the incremental-index check and the chunk-size study
make loadtest
CI runs the tests on Python 3.11 and 3.13, a regression gate that fails if Hit@10 on the dev questions drops by more than one question, a smoke test of the Docker image, a secret scan, and the browser port's parity test.
Retrieval numbers from the code at commit 95d3fe9, run on 2026-10-21. The reranker and generation runs are from 2026-10-28, on commit cf57ef1 with the study code applied but not yet committed; that code is in commit 34c4145 and the results in 3ea5084. Every results file and the command that wrote it: RESULTS.md.