Skip to main content
Read as text

Grounded post-training on Korean policy reports and DART filings

Grounded SFT and GRPO lift document QA by up to +1.0 judge points and carry over across document types. Closed-book answers do not move.

We built training data from Korean government policy research reports and DART corporate filings, post-trained Qwen3-1.7B with supervised fine-tuning (SFT) and GRPO, and evaluated it on documents and companies it had never seen. Grounded SFT followed by GRPO raised the judge score by +1.00 points on a 10-point scale for policy-report questions and +0.82 for DART questions, and one model trained on both sources reached +0.79 across 134 questions.

Retrieval remains essential. The same models scored around 2 out of 10 when the passage was withheld, so end-to-end performance comes from both together: a model whose reading skill has been raised by training, and a retriever that puts the right document in front of it.

Model: Qwen3-1.7B (4-bit, LoRA adapters). Judge: Claude Haiku rubric (1–10) and pairwise comparison against the base model. Single seed, synthetic evaluation sets built from held-out documents.

Summary

  1. Grounded SFT followed by GRPO improved answers from Korean policy reports by +1.00 judge points.
    • 6.37 → 7.37 on the 10-point Haiku rubric, 95% CI [+0.30, +1.70], on 54 grounded questions from 11 held-out reports. Better than the base model on 72% of questions (39 wins, 7 ties, 8 losses).
    • From the same SFT checkpoint, GRPO with a rule-based reward added +0.81 [+0.17, +1.46]. No reward model was needed, and each run took 52 to 99 minutes on a laptop.
  2. The skill transferred to DART filings, whether or not the model had ever seen a filing.
    • DART-trained SFT + GRPO: 6.25 → 7.08, +0.82 [+0.19, +1.45] on 80 questions from 8 companies excluded from training, 58% wins.
    • A model trained only on policy reports scored +0.80 [+0.11, +1.45] on the same filing questions. What the model learned was how to answer from a passage, not facts about any document type.
  3. One model trained on both sources matched the specialists, and nothing improved closed-book.
    • Mixed-data SFT + GRPO: +0.79 [+0.28, +1.30] across all 134 questions, 61% wins, statistically tied with the single-source models.
    • Asked the same questions without the passage, every variant scored at or below base (about 2 out of 10). Post-training taught the model to read what it is given; getting the right document in front of it is retrieval's job.
Change vs base model (Δ judge score, 1–10)
Evaluation setVariantnBaseAfterΔ [95% CI]Better than base
Policy reports, groundedGrounded SFT + GRPO546.377.37+1.00 [+0.30, +1.70]72%
DART filings, groundedDART-trained SFT + GRPO806.257.08+0.82 [+0.19, +1.45]58%
DART filings, groundedPolicy-trained SFT + GRPO806.257.05+0.80 [+0.11, +1.45]54%
Both sets, groundedMixed-data SFT + GRPO1346.196.98+0.79 [+0.28, +1.30]61%
Policy reports, closed-bookGrounded SFT + GRPO542.152.00−0.15 (CIs overlap)–

Δ is the per-question paired difference against the base model; "Better than base" is the share of questions the judge preferred over base in a two-order pairwise comparison. Closed-book scores use repetition penalty 1.1, so the base is 2.15 rather than the greedy 2.00.

Why Korean professional documents need their own test

Fluent Korean is not enough. The reader has to keep figures, units, conditions and exceptions straight.

Reading a Korean policy report or a corporate filing correctly takes more than fluent Korean. The reader has to know the terminology and the institutional context, keep figures, units, applicability conditions and exceptions straight, and compare information across documents. General-purpose Korean benchmarks barely touch this.

Two public sources have these properties in a form that can be checked. Policy research reports from government-funded institutes carry expert analysis across economics, industry, labor, education, science and law. DART filings (annual, half-year and quarterly reports filed with Korea's Financial Supervisory Service) are statutory documents dense with verifiable figures. Both let us write questions whose answers can be graded against the source, which is what reinforcement learning with verifiable rewards (RLVR) needs.

We asked a neighbouring question earlier for continued pretraining in Does continued pretraining on Korean financial text pay off?. This post asks it of post-training: SFT and GRPO on question–answer data built from the same kind of documents.

Training data

One cleaned document feeds every training stage: SFT, DPO, GRPO and the evaluation set.

We parsed the source documents into section-level chunks that keep headings, tables and metadata, then generated question–answer material from each chunk with an LLM. Every record points back to its source chunk.

Sources Policy research reports 63 reports, 6,527 chunks DART filings 62 filings, 41 companies, 3,388 chunks → Cleaned text Section chunks Structure, tables and metadata preserved → Training data SFT Grounded question, passage, reference answer DPO Chosen vs rejected answer, error type GRPO / RLVR Reference answer, key facts, numbers, scoring rules Evaluation Questions from documents and companies held out of training

A one-sentence example shows how the three formats differ. The source passage says:

Source passage “지원금은 적격 투자비의 30%이며, 사업장당 최대 2천만원이다.” The subsidy is 30% of eligible investment, capped at 20 million won per site.

SFT: read the passage, write the reference answer

{
  "messages": [
    {"role": "system", "content": "자료에 근거해 정확하게 답하세요."},   // Answer accurately from the passage.
    {"role": "user", "content": "[자료]\n지원금은 투자비의 30%, 최대 2천만원이다.\n[질문]\n투자비가 1억원이면 지원금은?"},
    {"role": "assistant", "content": "투자비의 30%는 3천만원이지만, 상한을 적용하면 지원금은 2천만원입니다."}
  ]
}

User turn: passage plus question ("If the investment is 100 million won, what is the subsidy?"). Assistant turn: the answer that applies the cap (30% would be 30 million, but the cap makes it 20 million).

DPO: the difference between applying the cap and forgetting it

{
  "prompt": "[자료]\n지원금은 투자비의 30%, 최대 2천만원이다.\n[질문]\n투자비가 1억원이면 지원금은?",
  "chosen": "상한을 적용해 2천만원입니다.",          // 20 million won, cap applied
  "rejected": "투자비의 30%이므로 3천만원입니다."   // 30 million won, cap ignored
}

Rejected answers are built around concrete mistakes (missed cap, wrong unit, misread condition), not just awkward prose.

GRPO: reference material for scoring sampled answers

{
  "id": "qa_001",
  "mode": "open",
  "prompt": "[자료]\n지원금은 투자비의 30%, 최대 2천만원이다.\n[질문]\n투자비가 1억원이면 지원금은?",
  "reference_answer": "상한을 적용해 2천만원입니다.",
  "key_facts": ["지원 비율은 투자비의 30%이다.", "지원 상한은 2천만원이다."],   // rate is 30%; cap is 20 million
  "numbers": ["30%", "2천만원"],
  "source_id": "passage_004"
}

The policy samples several answers per prompt; each is scored against the key facts and numbers without a separate reward model.

How we trained and evaluated

Same training budget for every variant; evaluation only on documents and companies the model never saw.

ItemSetting
ModelQwen3-1.7B (4-bit) with LoRA adapters, thinking mode off.
TrainingSFT on 800 examples, then GRPO for 60 steps with 6 sampled answers per prompt (720 samples). Every variant used the same budget.
RewardRule-based, no reward model: key-fact recall, numeric fidelity, Korean purity, semantic similarity, length, repetition and format.
DataQuestion–answer pairs generated by an LLM from section chunks of 51 policy reports and 47 DART filings (33 companies); items with numbers not in the source were filtered out.
EvaluationPolicy reports: 54 grounded questions from 11 held-out reports. DART: 80 grounded questions from 8 held-out companies. No overlap with training.
JudgeClaude Haiku: 1-to-10 rubric score and pairwise comparison against the base model in both orders; paired bootstrap confidence intervals.
Judge overallHaiku's overall rubric score (1–10) for one answer, covering accuracy, grounding, Korean naturalness and completeness. Δ vs basePer-question difference between a variant and the base model, averaged; the 95% interval is a paired bootstrap. Better than baseShare of questions where the judge preferred the variant over base in both comparison orders; disagreements count as ties. Grounded / closed-bookGrounded: the source passage is shown with the question. Closed-book: the same question without the passage.

Results

Bars show Δ judge score against the base model; whiskers are 95% confidence intervals.

SFT, mixed or other-source SFT, grounded or same-source SFT + GRPO, other variant SFT + GRPO, best in set

Policy reports: SFT alone did not move the judge; GRPO did

Δ judge score vs base

Policy reports, 54 grounded questions
Base model 6.37. Greedy decoding. Table below has the same numbers.

Judge score without the passage

Same 54 questions, closed-book
Repetition penalty 1.1. No variant beats the base model when the document is not provided.
VariantTraining dataJudge overallΔ vs base [95% CI]W / T / LBetter than baseKey-fact recall
Base–6.37–––0.645
Mixed SFTHalf grounded, half closed-book QA6.35−0.02 [−0.81, +0.72]26 / 9 / 1948%0.809
DPO765 preference pairs on top of mixed SFT6.22−0.15 [−0.80, +0.48]26 / 12 / 1648%0.806
Mixed SFT + GRPOMixed prompts7.17+0.80 [+0.13, +1.48]36 / 6 / 1267%0.854
Grounded SFTGrounded QA only6.87+0.50 [−0.11, +1.13]31 / 10 / 1357%0.772
Grounded SFT + GRPOGrounded prompts only7.37+1.00 [+0.30, +1.70]39 / 7 / 872%0.836
  • SFT raised key-fact recall from 0.645 to about 0.81 but not the judge score. The judge seems to have traded that against longer, less natural answers.
  • From the same SFT checkpoint, GRPO added +0.81 [+0.17, +1.46] (mixed) and +0.50 [−0.06, +1.04] (grounded). Numeric hallucination stayed at 0% for both GRPO models.
  • Grounded-only data led mixed data in every comparison (+0.52 after SFT, +0.20 after GRPO), but neither gap is statistically significant.
  • DPO did not help. Preference accuracy reached 0.97 during training, which suggests the deliberately degraded rejected answers were too easy to tell apart.

DART filings: the skill transfers from policy reports

Δ judge score vs base

DART filings, 80 grounded questions
Base model 6.25. 8 companies excluded from training. "Policy-trained" models never saw a filing.

Δ judge score vs base

Both sets combined, 134 questions
Base model 6.19 (weighted across both sets). Mixed training used the same budget as single-source runs.
VariantTrained onJudge overallΔ vs base [95% CI]W / T / LBetter than base
Base–6.25–––
Policy-trained SFTPolicy reports only6.90+0.65 [−0.04, +1.35]39 / 11 / 3049%
Policy-trained SFT + GRPOPolicy reports only7.05+0.80 [+0.11, +1.45]43 / 13 / 2454%
DART-trained SFTFilings only6.86+0.61 [−0.07, +1.29]44 / 18 / 1855%
DART-trained SFT + GRPOFilings only7.08+0.82 [+0.19, +1.45]46 / 14 / 2058%
  • All four post-trained models beat base on filing questions; the two GRPO models do so with confidence intervals clear of zero.
  • Policy-trained and DART-trained models are indistinguishable on DART questions (SFT −0.04 [−0.54, +0.46], GRPO +0.03 [−0.39, +0.41]). The model learned how to answer from a passage, and that habit did not depend on the document type it was learned from.
  • Transfer in the other direction was weaker: the DART GRPO model on policy questions scored +0.43 [−0.41, +1.28] over base, not significant, and below the policy-trained GRPO model.
  • Greedy decoding exposed repetition loops in the DART GRPO model (12 of 80 answers). A repetition penalty of 1.1 reduced this to 2 and raised its score from 6.83 to 7.08, so we used it for every later run.

Mixed training: one model covers both, no bonus from mixing

VariantPolicy set (54)DART set (80)All 134: Δ vs base [95% CI]W / T / LBetter than base
Base6.116.25–––
Policy-only SFT + GRPO7.097.05+0.87 [+0.37, +1.39]76 / 22 / 3657%
DART-only SFT + GRPO6.547.08+0.66 [+0.15, +1.19]80 / 22 / 3260%
Mixed-data SFT6.436.53+0.29 [−0.24, +0.84]61 / 29 / 4446%
Mixed-data SFT + GRPO6.707.17+0.79 [+0.28, +1.30]82 / 23 / 2961%

All answers in this comparison were generated with repetition penalty 1.1, so the policy-set base score is 6.11 here rather than the 6.37 above (greedy decoding).

  • The mixed GRPO model is clearly better than base and has the most pairwise wins, and the highest DART score (7.17), of any variant.
  • It is not better than the specialists: −0.08 [−0.42, +0.25] against the policy-only model and +0.13 [−0.19, +0.46] against the DART-only model. All five GRPO models sit between +0.66 and +0.87 over base, and none of their pairwise differences are significant.
  • Choosing hard prompts for GRPO (the 80 lowest-reward prompts per source under the SFT model) raised training reward more than random prompts (+0.050 vs +0.017) but flipped sign between evaluation sets (+0.47 on DART, −0.56 on policy). We read that as run-to-run variance of about 0.5 points, not as an effect of prompt selection.
  • Averaging the weights of the two GRPO adapters landed between them (+0.77), with no gain.

Closed-book: why retrieval still matters

VariantJudge overall, greedyJudge overall, rep. penalty 1.1 [95% CI]Repetition loops (greedy → 1.1)
Base2.002.15 [1.89, 2.43]1 → 2
Mixed SFT + GRPO1.671.96 [1.76, 2.17]22 → 1
Grounded SFT1.691.96 [1.76, 2.17]21 → 2
Grounded SFT + GRPO1.572.00 [1.78, 2.22]34 → 2

The same 54 policy questions asked without the passage score around 2 out of 10 for every model, against 6 to 7 with it. These questions ask for facts specific to reports the model never trained on, so the test shows only that no variant acquired knowledge it was not given. The 4-to-5-point gap is the clearest number in this study for retrieval: the answer is in the document, and the model's job starts once the document is in front of it. Fine-tuned models also looped under greedy decoding regardless of which data they saw, which points at LoRA fine-tuning plus greedy decoding rather than any particular dataset; serve such models with a repetition penalty.

Key findings and implications

  1. Grounded examples teach a transferable skill.
    • Showing the passage and requiring the answer to come from it was the ingredient every successful variant shared, and the skill moved across document types with almost no loss (+0.80 vs +0.82 on DART).
  2. Verifiable rewards are enough to run GRPO on this kind of data.
    • Key facts, numbers and language checks derived from the source replaced a reward model, and each GRPO run fit on a laptop in under two hours.
  3. Training and retrieval split the work.
    • Post-training did not inject facts at this scale. The 4-to-5-point gap between asking with and without the passage is the case for retrieval, the same conclusion our LucyRAG benchmark on SEC filings reached from the retrieval side: fetch the document, then let a model trained to read it answer.
  4. Differences under half a point need repeated seeds.
    • Four GRPO runs disagreed by that much on the same evaluation set. Single-run comparisons cannot resolve improvements smaller than that.

On a 1.7B model, grounded SFT plus GRPO built from Korean policy reports and DART filings produced a significant, transferable improvement in answering from documents (+0.79 to +1.00 judge points), and none without them. RAG remains essential, and the final performance comes from pairing a model improved by this kind of training with a retriever that finds the right document.

Next steps

These are single-seed results on synthetic evaluation sets, judged by the same LLM family that generated the data, so we treat them as a first signal rather than a final measurement. Next we will:

  • Repeat the main variants over several seeds and add a compute-matched SFT control, to separate the GRPO effect from extra training.
  • Build a human-reviewed evaluation set that includes unanswerable questions and misleading passages, judged by a model from a different family.
  • Test whether continued pretraining on the source text, followed by the same SFT and GRPO, changes the closed-book result.
  • Pair the trained reader with a retriever tuned for Korean filings, following our embedding fine-tuning work on DART filings, and measure the end-to-end answer quality.