Measured on real filings,
not demo data.
Answer and retrieval quality of Lucy RAG on public financial QA benchmarks and recent financial statements, scored with the open RAGAS framework.
01 · Answer quality
Same model. Same questions. Only the context changed.
Claude Fable 5 answering 2024–2025 financial statement questions with general web search context, then with Lucy RAG context. RAGAS Answer Accuracy.
Same model, two context sources
Claude Fable 5 · Internal evaluation dataset · 2024–2025 financial statements · average of three runs.
RAGAS Answer Accuracy (%) · Higher is better
- Model + web search
- Model + Lucy RAG
Web search context
Lucy RAG context
From 79.2% to 95.2%: +16 percentage points attributable to retrieval alone, with the answer model held fixed.
Source: LucyRAG benchmark on SEC filings · Experiment 2
| Comparison | RAGAS Answer Accuracy (%) |
|---|---|
| Web search context | 79.2% |
| Lucy RAG context | 95.2% |
A 26B open-weight model with Lucy context
Internal evaluation dataset · 2024–2025 financial statements · average of three runs.
RAGAS Answer Accuracy (%) · Higher is better
- Model + web search
- Model + Lucy RAG
Gemma-4-26B + Lucy RAG
gemini-3.6-flash + web search
claude-fable-5 + web search
claude-opus-5 + web search
Gemma-4-26B with Lucy RAG context outscores these three frontier models with web search. gpt-5.6-sol with web search scored 98.8% on this set.
Source: LucyRAG benchmark on SEC filings · Experiment 2
| Comparison | RAGAS Answer Accuracy (%) |
|---|---|
| Gemma-4-26B + Lucy RAG | 86.3% |
| gemini-3.6-flash + web search | 84.5% |
| claude-fable-5 + web search | 79.2% |
| claude-opus-5 + web search | 78.6% |
02 · Public benchmark
FinQABench: one model, three conditions
Claude Fable 5 on FinQABench under three conditions. RAGAS Answer Accuracy averaged over three runs.
FinQABench answer accuracy
Claude Fable 5 · FinQABench · average of three runs.
RAGAS Answer Accuracy (%) · Higher is better
- Model alone
- Model + web search
- Model + Lucy RAG
claude-fable-5
Lucy RAG context gives Claude Fable 5 its highest FinQABench score. On FinanceBench, which leans on reasoning over retrieval, the same model scored 80.9% alone, 79.9% with web search and 79.4% with Lucy RAG.
Source: LucyRAG benchmark on SEC filings · Experiment 1
| Comparison | Model alone (%) | Model + web search (%) | Model + Lucy RAG (%) |
|---|---|---|---|
| claude-fable-5 | 73.3% | 87.5% | 89.9% |
These charts show Answer Accuracy. The source study also reports Answer Correctness, a separate metric on which Lucy RAG matched or beat web search in 7 of 8 model–dataset pairs. That result is not an Answer Accuracy win rate.
03 · Worked examples
Which answer was right?
Three benchmark questions answered by Claude Opus 5 under three conditions. Quotes are verbatim from the published runs. These examples use a different model from the Claude Fable 5 aggregate charts above.
Model alone
The model answers from memory, with no documents.
Model + web search
The model uses its built-in web search tool.
Model + Lucy RAG
The model answers from filing chunks Lucy retrieved.
Dataset: FinQABench · Model: Claude Opus 5
Source filing: Apple Inc. Form 10-K, fiscal 2022 (period ended September 24, 2022)
What is the total fair value of the Company's derivative assets and liabilities as of September 24, 2022?
Reference answer
$13,255 million
Model alone
Short answer: Apple's FY 2022 Form 10-K does not disclose a single "total fair value" of derivative assets and liabilities — that table was dropped in the FY 2022 filing.
Incorrect
Wrong. The table is in the filing; the model invented its absence.
Model + web search
This implies gross derivative assets of roughly $8.2 billion and gross derivative liabilities of roughly $7.8 billion, with the FX amounts above being the largest components.
Incorrect
Wrong. Pieced fragments together from the web and got both totals wrong.
Model + Lucy RAG
So gross derivative assets totaled $7,136 million and gross derivative liabilities totaled $4,752M + $1,367M = $6,119 million; adding the two sides together gives $7,136M + $6,119M = $13,255 million of gross derivative fair value.
Correct
Correct, with the table broken out by hedge designation and cited to the filing.
Full answer in the postDataset: FinQABench · Model: Claude Opus 5
Source filing: Apple Inc. Form 10-K, fiscal 2022, benefit plans note
What is the maximum number of shares authorized for issuance under the Director Plan at the time the plan was last amended on November 9, 2021?
Reference answer
approximately 45 million shares (split-adjusted)
Model alone
Short answer: that figure isn't in Apple's FY 2022 Form 10-K.
Incorrect
Wrong. The figure is in the benefit plans note.
Model + web search
The 10-K states that a maximum of approximately 11.2 million shares were authorized for issuance under the Director Plan at the time it was last amended on November 9, 2021 — consistent with the plan document itself…
Incorrect
Wrong. Confident and official-sounding, with the wrong number.
Model + Lucy RAG
Approximately 45 million shares (split-adjusted) were authorized for issuance pursuant to Apple's Non-Employee Director Stock Plan awards at the time the plan was last amended on November 9, 2021, as disclosed in the benefit plans note of Apple's fiscal 2022 Form 10-K.
Correct
Correct, and it names the note and the exhibit the plan is filed under.
Full answer in the postDataset: FinanceBench · Model: Claude Opus 5
Source filing: Corning Inc. Form 10-K, fiscal 2022, income taxes note and MD&A
How much has the effective tax rate of Corning changed between FY2021 and FY2022?
Reference answer
from 20% to 23%
Model alone
Corning's effective tax rate rose by roughly 3 percentage points — from about 21% in FY2021 to about 24% in FY2022.
Incorrect
Wrong. Right direction, wrong numbers, answered from memory.
Model + web search
Corning's effective tax rate declined modestly in FY2022 versus FY2021 — a decrease of roughly 2 percentage points (approximately 18% in 2022 versus approximately 20% in 2021).
Incorrect
Wrong. Mixed up sources and got even the direction wrong.
Model + Lucy RAG
Corning's effective tax rate rose by 2.7 percentage points, from 20.2% in FY2021 to 22.9% in FY2022. The MD&A shows the same movement on a rounded basis, 20% in 2021 to 23% in 2022.
Correct
Correct, with the exact rates, the rounded MD&A figures and the reconciling items.
Full answer in the post04 · Under the hood
The retrieval layer, before any LLM touches the result
Hybrid dense + BM25 retrieval with company, form type and date filters on self-hosted Qdrant. Retrieval quality, search latency, full response time and index memory are measured separately.
Gold passage in the top 10
FinanceBench and FinQABench · hybrid dense + BM25 retrieval with metadata filters.
Recall@10 (%) · Higher is better
- Recall@10
FinanceBench
FinQABench
FinanceBench / FinQABench: Recall@5 93.4% / 83.1% · nDCG@10 0.771 / 0.762 · MRR 0.774 / 0.745.
Source: Lucy retrieval layer with Qdrant · Retrieval quality
| Comparison | Recall@10 (%) |
|---|---|
| FinanceBench | 94.7% |
| FinQABench | 89.6% |
Time to a finished answer
gpt-5.6-sol · FinQABench · retrieval + answer generation.
End-to-end response time (s) · Lower is better
- Model + web search
- Model + Lucy RAG
Model + web search
Model + Lucy RAG
Lucy RAG was faster than web search in 7 of 8 model–dataset combinations. The separate retrieval study measured 32 ms per search on average; that is not time to a finished answer.
Source: LucyRAG benchmark on SEC filings · Experiment 1
| Comparison | End-to-end response time (s) |
|---|---|
| Model + web search | 8.8 s |
| Model + Lucy RAG | 5.3 s |
Retrieval latency by lane
Self-hosted Qdrant · per-search latency · excludes answer generation.
Retrieval latency (ms) · Lower is better
- p50
- Average
- p90
Narrow
XBRL
Broad
All lanes
Narrow lane applies section filters for precision. Broad lane is the recall safety net. XBRL lane runs only for financial statement questions.
Source: Lucy retrieval layer with Qdrant · 1,517 measured searches
| Lane | p50 (ms) | Average (ms) | p90 (ms) |
|---|---|---|---|
| Narrow | 9 ms | 23 ms | 62 ms |
| XBRL | 10 ms | 27 ms | 81 ms |
| Broad | 27 ms | 42 ms | 105 ms |
| All lanes | 16 ms | 32 ms | 82 ms |
4.9× smaller index
10-K corpus · 1,536 dimensions · unquantized vs. 4-bit TurboQuant.
Index memory (GB) · Lower is better
- Unquantized
- 4-bit TurboQuant
Unquantized
4-bit TurboQuant
Hybrid retrieval quality was nearly unchanged: +0.2% Recall@20, +0.1% MRR, −0.1% nDCG@20, measured without metadata filtering.
Source: Lucy retrieval layer with Qdrant · Quantization study
| Comparison | Index memory (GB) |
|---|---|
| Unquantized | 52 GB |
| 4-bit TurboQuant | 10.7 GB |
Try it on your own questions.
Explore Lucy RAG in the Playground, or see how filing evidence fits into your financial AI workflow.
Sources & methodology
- LucyRAG benchmark on SEC filings · Aug 31, 2026
- Lucy retrieval layer with Qdrant · Sep 21, 2026
- Public datasets: FinQABench · FinanceBench
Experiment 1 uses public financial QA datasets. Experiment 2 uses an internal dataset of recent 2024–2025 financial statements. Answer scores are averages of three runs. Retrieval and quantization measurements come from the separate Qdrant study.