The answer is in the filing.LUCY hands it to the model.
Evaluated on FinQABench, FinanceBench, and recent SEC filings, LucyRAG delivered answer quality comparable to or better than web search—particularly in answer correctness.
With the right filing context, even smaller open-weight models can compete with frontier systems.
Experiment Setup
- Experiment 1: Results are reported as the average of three runs usingFinQABench and FinanceBench.
- Experiment 2: Results are reported as the average of three runs using aninternal evaluation dataset focused on recent 2024–2025 financial statements.
- Evaluation: We used RAGAS, an open-source evaluation framework, and measured Answer Accuracy, one of the metrics provided by RAGAS.
1. Focused on Filings, Competitive with Web Search
Across FinQABench and FinanceBench, LucyRAG delivered answer quality comparable to general-purpose Web Search when evaluated with the same underlying LLMs.
For RAGAS Answer Accuracy, the performance gap was typically within0.02–0.06. For RAGAS Answer Correctness, LucyRAG matched or outperformed Web Search in 7 out of 8 model–dataset combinations.
Notably, on FinQABench, LucyRAG achieved higher Answer Correctness than Web Search across all evaluated models.
By focusing on SEC and DART filings, LucyRAG delivers competitive answer quality without relying on broad web search.
RAGAS answer accuracy
Summary
- Context drives performance
Both Web Search and LucyRAG significantly outperform the base model, with gains of up to +0.37 for Gemini on FinQABench. Providing relevant context is consistently better than providing none. - LucyRAG performs on par with Web Search
For the same model, the Answer Accuracy gap is only 0.005–0.035. On FinQABench, both Claude models achieved their highest scores with LucyRAG context (0.899 / 0.897). - LucyRAG is faster
LucyRAG had lower response latency than Web Search in 7 out of 8 model–dataset combinations (e.g., Sol on FinQABench: 5.3s vs. 8.8s).
Examples
See how the same model (Claude Opus 5) answers real financial questions across three setups: Base Model, Web Search, and LucyRAG.
Apple's total derivative fair value
FinQABenchQuestion
What is the total fair value of the Company's derivative assets and liabilities as of September 24, 2022?
Reference answer: $13,255 million
"Short answer: Apple's FY 2022 Form 10‑K does not disclose a single "total fair value" of derivative assets and liabilities — that table was dropped in the FY 2022 filing."
Wrong - the table is in the filing. The model made up its absence.
"This implies gross derivative assets of roughly $8.2 billion and gross derivative liabilities of roughly $7.8 billion, with the FX amounts above being the largest components."
Pieced fragments together from the web - and got the totals wrong.
Apple's FY2022 10-K (period ended September 24, 2022) reports derivative fair values on a gross basis:
| Item (as of Sept 24, 2022) | Designated as accounting hedges | Not designated | Total |
|---|---|---|---|
| Derivative assets – foreign exchange contracts | $4,317M [2] | $2,819M [2] | $7,136M [2] |
| Derivative liabilities – foreign exchange contracts | $2,205M [2] | $2,547M [2] | $4,752M [2] |
| Derivative liabilities – interest rate contracts | $1,367M [2] | — | $1,367M [2] |
So gross derivative assets totaled $7,136 million and gross derivative liabilities totaled $4,752M + $1,367M = $6,119 million [2]; adding the two sides together gives $7,136M + $6,119M = $13,255 million of gross derivative fair value. Assets are included in other current and other non-current assets, liabilities in other current and other non-current liabilities, and all are measured with Level 2 inputs[1].
On a net basis, the potential effects of rights of set-off, including collateral, would reduce both derivative assets and derivative liabilities by $7.8 billion, leaving a net derivative asset of $412 million [1].
Shares authorized under Apple's Director Plan
FinQABenchQuestion
What is the maximum number of shares authorized for issuance under the Director Plan at the time the plan was last amended on November 9, 2021?
Reference answer: approximately 45 million shares (split-adjusted)
"Short answer: that figure isn't in Apple's FY 2022 Form 10‑K."
Wrong - the figure is right there in the benefit-plans note.
"The 10-K states that a maximum of approximately 11.2 million shareswere authorized for issuance under the Director Plan at the time it was last amended on November 9, 2021 — consistent with the plan document itself…"
Confident, official-sounding - and wrong.
Approximately 45 million shares (split-adjusted) were authorized for issuance pursuant to Apple's Non-Employee Director Stock Plan awards at the time the plan was last amended on November 9, 2021, as disclosed in the benefit plans note of Apple's fiscal 2022 Form 10-K (period ended September 24, 2022) [1]. The amended plan itself is filed as an exhibit — "Apple Inc. Non-Employee Director Stock Plan, as amended November 9, 2021" — incorporated by reference from the Form 10-Q for the period ended December 25, 2021 [3]. Note that RSUs granted under the Director Plan reduce shares available for grant by a factor of two times the number of RSUs granted, and the plan expires on November 12, 2027[1].
Corning's effective tax rate, FY2021 → FY2022
FinanceBenchQuestion
How much has the effective tax rate of Corning changed between FY2021 and FY2022?
Reference answer: from 20% to 23%
"Corning's effective tax rate rose by roughly 3 percentage points — from about 21% in FY2021 to about 24% in FY2022."
Right direction, wrong numbers - answered from memory.
"Corning's effective tax rate declined modestly in FY2022 versus FY2021 — a decrease of roughly 2 percentage points (approximately 18% in 2022 versus approximately 20% in 2021)."
Mixed up sources - and got even the direction wrong.
Corning's effective tax rate rose by 2.7 percentage points, from 20.2% in FY2021 to 22.9% in FY2022 (22.9% − 20.2% = +2.7 pp, i.e. about a 13% increase relative to the FY2021 rate) [1][2]. The MD&A shows the same movement on a rounded basis, 20% in 2021 to 23% in 2022, with the provision for income taxes falling from $491 million on pre-tax income of $2,426 million to $411 million on pre-tax income of $1,797 million [11].
| Year ended Dec 31 | Effective tax rate | Statutory U.S. federal rate |
|---|---|---|
| 2022 | 22.9% [1] | 21.0% [1] |
| 2021 | 20.2% [1] | 21.0% [1] |
| Change | +2.7 pp | 0.0 pp |
The statutory U.S. federal rate itself was unchanged at 21.0% in both years[1][2]; the increase in the effective rate came from reconciling items, chiefly a net provision of $67 million for changes in tax reserves, $40 million from differences arising from foreign earnings and $38 million from valuation allowance assessments in 2022 [6], against 2021 benefits of $62 million of tax credits and $37 million related to share-based compensation, partly offset by a $52 million provision on foreign earnings[3].
2. The Right Context Can Outweigh Model Size
In an evaluation of numerical questions based on 10-K financial statements,LucyRAG powered by Gemma-4-26B delivered performance competitive with frontier models using Web Search.
Higher than Gemini (0.845), Claude Fable (0.792), and Opus (0.786)
LucyRAG (Gemma) vs frontier models with Web Search
RAGAS answer accuracy
LucyRAG (Claude Fable 5) vs frontier models with Web Search
RAGAS answer accuracy
Summary
- An open-weight model outperforms three frontier systems
With LucyRAG context, Gemma-4-26B achieved 0.863 Answer Accuracy, outperforming Gemini (0.845), Claude Fable (0.792), and Claude Opus (0.786) with Web Search. High-quality, filing-grounded context enabled a smaller open-weight model to outperform three frontier systems. - Same model, +0.16 from better context alone
Using Claude Fable 5 as the answer model, LucyRAG reached 0.952, compared with 0.792 for the same model using Web Search. Simply replacing web search results with filing-grounded context improved accuracy by 0.16, with virtually no regressions. - The gap to the strongest search system nearly disappears
Claude Fable 5 + LucyRAG (0.952) trails gpt-5.6-sol + Web Search (0.988) by just 0.036, matching its performance on nearly every question. The results also suggest that LucyRAG can benefit further as the underlying answer model becomes stronger.
Rather than searching broadly across the web, LucyRAG retrieves financial data directly from SEC filings and provides the relevant tables as structured Markdown Key-Value context.
This helps reduce the risk of mixing figures from different sources or reporting periods, while also lowering the burden on the model to interpret complex financial tables.
Accurate filing data and well-structured context can enable strong answer quality even with smaller models.