TRACKER.md, including what is unfinished. The numbers on this page are measured, not projected.
The hard part of evaluating a retrieval system is that you usually cannot grade it without another model, and an LLM judge inherits the failure modes of the thing it is judging. SEC filings break that circularity. A 10-K states its revenue in prose and tags the same figure in XBRL, so an answer of “$60.9 billion” can be compared against a structured fact rather than against an opinion.
That makes exact-match grading possible for the numeric slice of the benchmark, which is the slice that matters most: a summary that is stylistically fine and numerically wrong is worse than useless in this domain. The tier-1 evaluation set is 72 frozen questions graded by exact match, with zero LLM grading.
1. Does long context kill RAG? The same question set, answered three ways, with the model held constant so that architecture is the only variable.
| Arm | Approach |
|---|---|
| A | Long context: whole filings stuffed into a 1M-token window |
| B | Naive RAG: fixed 512-token chunks, dense top-k |
| C | Hybrid agentic: structure-aware chunks, BM25 plus dense plus rerank, verification tools |
2. The quality and latency frontier. A self-hosted 8B model on vLLM walked down an optimisation ladder, FP8 then prefix caching then speculative decoding, with evaluation accuracy measured at every rung rather than throughput alone. The interesting question is not how fast it gets, it is where accuracy starts to pay for the speed.
Corpus: NVDA, AAPL, MSFT, JPM, XOM, PFE, WMT and COST, deliberately cross-sector, over FY2024 and FY2025. Cross-sector matters more than it sounds, because most of the interesting failures below only appear when a bank and an oil major are in the same corpus as a chip company.
Each of these produced no error. A naive pipeline would have shipped a corpus that looked fine and was wrong, which is the failure mode this project is mostly about.
SEC's own ticker map points XOM at a successor holding entity registered in 2026 with no filing
history. Every FY2024 and FY2025 Exxon filing sits under the predecessor CIK. The resolution was correct per
SEC's map and returned zero filings.
JPMorgan files roughly 25,000 documents a year, so its inline window reached back only twelve months, and all of FY2024 lived in 69 overflow files that nothing was reading.
Naive tag-stripping ingests the inline-XBRL header block. On NVIDIA's 10-K that was 125,940 of 381,767 characters of entity IDs and axis members that no human reader ever sees. They formed the two densest numeric-looking chunks in the document, which is exactly the kind of thing a retriever loves and a reader cannot use.
JPMorgan and Exxon incorporate MD&A by reference to page ranges, and NVIDIA files its statements under Item 15, leaving Item 8 with two lines. Measured across the corpus: 48 of 64 section slots are substantive.
Revenues for five of them, RevenueFromContractWithCustomerExcludingAssessedTax for
Apple and Microsoft, and RevenuesNetOfInterestExpense for JPMorgan, which is the correct concept
for a bank rather than a workaround. Separately, NVIDIA's 10-for-1 split means FY2023 EPS is both
11.93 and 1.19 depending on which filing you read.
| Stage | Result |
|---|---|
| EDGAR fetch | 64 filings, 8 tickers over FY2024 and FY2025, 230.8 MB, rate-limited and cache-first |
| Extraction | 231 MB of markup reduced to 17.0 MB of text, iXBRL machine metadata removed |
| Sectioning | 64 of 64 filings pass ordering and substance assertions |
| Chunking | 9,449 chunks, median around 477 tokens, table rows never split |
| XBRL facts | 2,010 resolved facts, restatements collapsed |
| Tier-1 evals | 72 frozen questions, graded by exact match, zero LLM grading |
| Tests | 127 passing, no network required |
Assertions over warnings. Every external boundary hard-fails on violated expectations. Warnings get read once and then ignored; raises stop the pipeline. Tests verify code, assertions verify data, and the two are not substitutes for each other.
Table rows are structurally unsplittable. Extraction emits one row per line and chunking is line-granular, so a figure can never be separated from its label. That is the single failure that would quietly undermine numeric verification, so it is prevented structurally rather than checked afterwards.
Negative results are documented. The decision log records a content-anchored sectioning approach that was built, measured, failed to converge, and was rejected with evidence rather than deleted.
Known exceptions are data, not folklore. By-reference filings and restated facts are allowlisted in config; anything unlisted hard-fails. The alternative is a pipeline whose real behaviour lives in the author's head.
The corpus itself is not committed, since it is 230 MB of regenerable HTML. What is committed is
data/raw/manifest.jsonl: every filing's accession number, period and SHA-256, so a re-fetch can
be verified byte for byte against the corpus that produced any published number.
This is a research and citation tool, not investment advice. It summarises and cites public SEC filings, its outputs may be incomplete or wrong, and it should not be used to make investment decisions. There is no affiliation with or endorsement by the SEC or any company in the corpus. Filing data is retrieved from EDGAR under SEC's fair-access policy, with a declared User-Agent and a self-imposed 8 requests per second ceiling against their limit of 10.