Skip to main content
Read as text
Lucy · SEC filing data · Benchmark

GPT-6 Luna on Lucy data matches 98% of GPT-6 Astra’s quality for 0.22% of the cost

Same filings, same questions, three models. The only change is what the model reads: Lucy data or the full filing.

98% Luna + Lucy data quality, relative to Astra + Full filing 1/459 Its cost, relative to Astra + Full filing ≈ 1/5 Cost on Lucy data for the same model, quality held November 9, 2026 · 60 filings · 2 topics · 6 conditions Two inputs Lucy data

Only the passages that matter, picked from a filing Lucy has split into Items

Full filing

The whole filing, converted to text with an open-source tool (markitdown)

Three models · input price per token Luna Low-cost gpt-6-luna 1× Sol Premium gpt-6.1-sol 20× Astra Frontier gpt-6-astra 100×
Luna + Lucy data sits top-left: near-frontier quality at the lowest cost Lucy data Full filing → same model, switched from full filing to Lucy data
Each dot = one model × one input · 60 filings × 2 topics
The task

Extraction with one right answer

  • Headcount: total, as-of date, and any breakdown by region or function
  • Major customers (10%+ of revenue): name, share, revenue or receivables basis, period

A good extraction finds every value, skips look-alikes (a subsidiary's headcount, say), and gets each value right.

A full filing averages about 190,000 tokens, yet the headcount usually sits in one or two paragraphs. With full filings, over 95% of the cost is the filing text going in.

The input

What goes into Lucy data

  1. Split by Item Each topic reads only the Items where its answer can be.
  2. Tables kept intact Every number stays attached to its row, column, and unit.
  3. Search inside Items Tens of thousands of tokens, not hundreds. Even the largest 20-F fits in one call.
  4. Rules per topic What counts as company-wide headcount, when a customer share is in scope.
Setup
Filings60 US annual reports, fiscal 2025, picked at random (40 10-K, 20 20-F)
Conditions2 inputs × 3 models = 6, same instructions for all. Luna + Lucy data also had a few check lines for commonly missed items.
Long filingsFull filings over the 1M-character input limit were split into 2–3 chunks and merged
ReasoningLuna high · Sol and Astra default (medium)
CostActual tokens × list price, no cache discount
GradingBlind AI grader (Opus); every item marked wrong rechecked by a second grader
Results

01 Cost: Lucy data cuts every model's cost to about 1/5

Cost per 10,000 extractions Full filing Lucy data ← cheaper (log scale)
One extraction = one filing × one topic · average input: Lucy data ≈ 37,000 tokens, full filing ≈ 190,000

02 Quality: Lucy data holds it, and lifts Luna

Recall: share of the filing's values found Full filing Lucy data better →
Averaged per filing over 81 filing-topics that have values · a ±1-point gap is within run-to-run swing
  • Sol, Astra: Lucy data and full filing within 1 point, inside run-to-run swing.
  • Luna: +6 points with Lucy data.
  • Core values (headcount, date, customer, share) were 99.7–100% correct in every condition; the few wrong values (5 in total) came mostly from full filings.

03 Big full filings didn't fit in one call

9 of 60 full filings went over the input limit, including 8 of the 20 20-Fs. They needed extra code to split, call several times, and merge. Every Lucy input fit in one call.

Takeaways
  • 98% · 1/459With Lucy data, low-cost Luna gets 98% of frontier Astra's quality at 1/459 the cost.
  • ≈ 1/5Sol and Astra keep their quality on Lucy data at about 1/5 the cost.
  • 9 of 60full filings were too long for one call and needed split-and-merge code. Every Lucy input fit.