Does continued pretraining on Korean financial text pay off?
Held-out perplexity fell 49.5% and 69.0% after continued pretraining on Korean filings. Gains in financial-term discrimination were real but uneven across models.
We continued pretraining two open base models, Gemma 4 E4B and Kanana 2 1.3B, on 200 DART annual reports, the K-IFRS standards and the Bank of Korea’s financial glossary, then measured each model against its own pre-CPT baseline.
- DART FY2025 annual reports200 companies, narrative text only
- K-IFRS63 standards, interpretations and framework documents
- Bank of Korea glossaryGlossary of 800 Economic and Financial Terms
- Language modelingIn-domain perplexity on annual reports from 30 held-out companies
- Financial-term discriminationContrastive accuracy and log-likelihood margin on a 300-item multiple-choice probe
- Gemma 4 E4BGeneral-purpose pretrained base model
- Kanana 2 1.3BKakao’s open-weight, Korean-specialized small language model
Key results
Each bar shows the change from the model’s own pre-CPT baseline.
In-domain perplexity, reduction after CPT
Contrastive accuracy on the term probe, gain after CPT
The same corpus cut perplexity sharply for both models, but improved financial-term discrimination far more for Gemma than for Kanana. Lower perplexity alone is not a reliable proxy for domain knowledge.
Not a model ranking: the goal is to measure the effect of CPT within each model, not to compare Gemma with Kanana.
Why Korean financial text needs its own evaluation
LLMs handle many languages, but their performance is uneven. Models pretrained mostly on English data often struggle to use knowledge expressed in lower-resource languages such as Korean.
KoSimpleQA makes the gap concrete: on 938 questions about Korea-specific knowledge, the best model answered only 31.6% correctly, and model rankings differed sharply from those on the English SimpleQA benchmark. Korean needs its own training data and its own benchmarks.
Finance raises the bar further. Filings are dense with accounting terms that are closely related but not interchangeable, such as consolidated versus separate financial statements, or amortized cost versus fair value.
We tested whether continued pretraining (CPT) on Korean financial text improves a general-purpose base model in this domain, and measured two capabilities before and after CPT:
- Language modeling on financial filings: does perplexity drop on held-out filings?
- Financial-term discrimination: does the model rank the correct term above close distractors more often, as measured by contrastive accuracy and log-likelihood margin?
1. Evaluation method
Perplexity for fluency on filing text, and log-likelihood scoring for term-level knowledge.
We trained with a standard causal language modeling objective and evaluated the two capabilities separately.
1.1 Language modeling on financial filings
We measured perplexity (PPL) on DART annual reports held out from training.
Lower perplexity means the model assigns higher probability to the actual text, so it predicts held-out filings better. Perplexity does not show whether the model can tell similar financial concepts apart, so we evaluated that separately.
1.2 Financial-term discrimination
Both models are base models without instruction tuning, so we did not ask them to generate answers. Instead we used log-likelihood scoring: for each prompt, we appended the correct answer and three hard negatives one at a time and computed the length-normalized log-likelihood of each candidate.
- Contrastive accuracy: the share of items where the correct answer outscores all three distractors
- Log-likelihood margin: the score of the correct answer minus the score of the strongest distractor
A positive margin means the correct answer ranks first. A larger margin means a stronger preference for it over close distractors.
2. Building the training corpus
Narrative text from 200 annual reports, plus accounting standards and a financial glossary, with a company-level split to prevent leakage.
2.1 Corpus composition
The corpus combines three sources. Annual reports were split into train, validation and test sets at the company level; K-IFRS and the glossary were used for training only.
| Split | Source | Documents |
|---|---|---|
| Train | DART annual reports | 200 companies |
| Train | K-IFRS | 63 documents |
| Train | BOK glossary | 1 document |
| Validation | DART annual reports | 20 companies |
| Test | DART annual reports | 30 companies |
2.2 Selecting and extracting annual reports
Selection criteria
- KOSPI- and KOSDAQ-listed companies with a December 2025 fiscal year-end, excluding SPACs
- One original (non-amended) annual report per company
- Eligible universe: 2,504 companies
Text extraction
- Kept narrative text only. Removed tables, XBRL tables, images, cover pages, detailed schedules and the expert confirmation section.
- Removed unit labels and lead-in lines before tables, and footnotes after them.
- Kept section headings only when followed by body text.
- Deduplicated repeated paragraphs of 40 or more characters within each report.
We excluded tables because they would have dominated the corpus: by token count, they were roughly eight times the size of the narrative text. Since the goal was domain adaptation to the narrative language of filings, we trained on narrative text only.
Many passages, like the accounting policy note below, use K-IFRS terminology verbatim.
금융자산은 금융상품의 계약당사자가 되는 때에만 연결재무상태표에 인식하고 있으며, 금융자산의 정형화된 매입이나 매도는 매매일 또는 결제일에 인식하고 있습니다. 금융상품의 최초 인식시점에 사업모형과 금융자산의 계약상 현금흐름의 특성에 따라 금융자산은 당기손익-공정가치측정금융자산, 기타포괄손익-공정가치측정금융자산, 상각후원가측정금융자산으로 분류하고 있습니다. 금융자산은 최초인식시 공정가치로 측정하고 있으며, 당기손익-공정가치측정금융자산이 아닌 경우 당해 금융자산의 취득과 직접 관련되는 거래원가는 최초인식하는 공정가치에 가산하여 측정하고 있습니다. (가) 당기손익-공정가치측정금융자산 금융자산을 단기매매목적으로 보유하고 있거나, 당기손익-공정가치측정금융자산으로 지정하는 경우와 기타포괄손익-공정가치측정금융자산 또는 상각후원가측정금융자산으로 분류되지 않는 금융자산은 당기손익-공정가치측정금융자산으로 분류합니다. (이하 생략)
2.3 Train, validation and test split
- We split at the company level: 30 test, 20 validation and 200 train companies. No company appears in more than one split.
- Financial companies (banks and financial holding companies, insurers, securities firms, asset managers and specialty lenders) make up only 2.4% of the universe, so we oversampled them to a 20% target. Actual shares: 17% train, 20% validation, 20% test. The remaining companies were sampled in proportion to industry and listing market.
- Holding companies and their listed subsidiaries can share up to 44% of their narrative text (Lotte Shopping and Lotte Corp., 44%; SK and SK Innovation, 39%). To prevent leakage, we excluded from all splits 207 companies linked to a test or validation company by name, affiliation or shared paragraphs.
2.4 K-IFRS and the BOK glossary
- K-IFRS (63 documents): 41 standards, 19 interpretations and 3 conceptual framework and practice documents. Tables, page numbers, figures and amendment histories were removed.
- BOK glossary: term entries only. Front matter, author credits, publication details and tables were removed.
50 확정기여제도의 회계처리는 보고기업이 각 기간에 부담하는 채무를 해당 기간의 기여금으로 결정하기 때문에 비교적 단순하다. 따라서 채무나 비용을 측정하기 위해 보험수리적 가정을 할 필요가 없고 그 결과 보험수리적손익이 생길 가능성도 없다. 그리고 기여금 전부나 일부의 납부기일이 종업원이 관련 근무용역을 제공하는 연차 보고기간 말 후 12개월이 되기 전에 모두 결제될 것으로 예상되지 않는 경우를 제외하고는 할인되지 않은 금액으로 채무를 측정한다. 인식과 측정 51 종업원이 일정 기간 근무용역을 제공하면, 기업은 그 대가로 확정기여제도에 납부할 기여금을 다음과 같이 인식한다. ⑴ 이미 납부한 기여금이 있다면 이를 차감한 후 부채(미지급비용)로 인식한다. (이하 생략)
카르텔(Cartel)이란 동종 혹은 유사 산업에서 경쟁하는 기업들이 상호간의 경쟁을 제한하고 완화함으로써 시장을 지배할 목적으로 결성한 기업연합체를 말한다. 카르텔에 속한 기업들은 협약을 맺고 가격책정, 생산량 할당 등에 대해 개별 기업 간의 행동을 조정할 기구를 만들고 이를 통해 가격과 생산량을 결정한다. (중략) 대표적인 국제규모의 카르텔로는 중동 산유국이 주축이 된 석유수출국기구(OPEC; Organization of Petroleum Exporting Countries)를 들 수 있다. 연관검색어: 독점/과점, 자연독점 캐리트레이드 상대적으로 금리가 낮은 국가의 통화로 자금을 빌려(carry) 금리가 높은 국가의 주식 채권 등의 자산에 투자(trade)하는 행위를 의미한다. (이하 생략)
3. Building the 300-item term probe
Three question types, each with one correct answer and three hard negatives from the same domain.
We built a separate probe of 300 multiple-choice items: 237 drawn from the BOK glossary and 63 from K-IFRS.
| Type | What it tests | Items |
|---|---|---|
| T1. Korean term recognition | Map a financial abbreviation to its Korean term | 100 |
| T2. Concept identification | Identify the concept that matches a definition or description | 100 |
| T3. Abbreviation discrimination | Identify the abbreviation that matches a description | 100 |
| Total | 300 |
Difficulty: 49 easy, 159 medium and 92 hard items. Items span 10 categories: investment, financial accounting, macroeconomics, risk management, disclosure and regulation, money and financial markets, payments and settlement, international finance, banking and lending, and insurance.
Each item includes a prompt, one correct answer (the positive), three hard negatives, a source citation (document and page) and a rationale for each distractor. Every answer and distractor was checked against the cited passage.
Each item also has a paraphrased prompt with the same meaning, so we can test whether results depend on wording. One example per type:
- Prompt
- 은행 자산건전성 분류에서 NPL의 한국어 명칭은
- Paraphrase
- 금융기관 여신 관리에서 쓰이는 NPL을 우리말로 옮긴 명칭은
- Answer
- 부실채권
- Distractors
- 대손충당금, 정상채권, 후순위채권
- Prompt
- 현재의무이고 이를 이행하기 위하여 경제적 효익이 있는 자원을 유출할 가능성이 높으며 해당 금액을 신뢰성 있게 추정할 수 있어 부채로 인식하되, 지출하는 시기 또는 금액이 불확실한 부채는
- Paraphrase
- 결산일에 기업이 부담해야 할 의무가 명백히 존재하고 금액을 합리적으로 추정할 수 있어 재무제표에 부채로 반영하지만, 결제에 필요한 미래 지출의 시기나 금액에는 불확실성이 있는 부채는
- Answer
- 충당부채
- Distractors
- 우발부채, 금융부채, 계약부채
- Prompt
- 은행이 유동성 부족에 대비하여 보유한 고유동성자산 규모를 30일간의 유동성스트레스 시나리오 하에서 예상되는 순현금유출액으로 나눈 단기 유동성 규제비율의 약어는
- Paraphrase
- 향후 30일 동안의 심각한 위기상황에서 예상되는 순현금유출액 대비 은행이 보유한 고유동성자산의 비율을 뜻하는 규제지표의 약어는
- Answer
- LCR
- Distractors
- NSFR, CET1, VaR
4. Did perplexity on financial filings improve?
Yes, sharply, for both models.
We compared in-domain perplexity before and after CPT on annual reports from the 30 held-out companies.
위험회피수단으로 지정되고 해외사업장순투자의 위험회피회계의 적용요건을 충족한 금융상품의 공정가치 변동 중 위험회피에 효과적인 부분은 기타포괄손익으로 인식하며, 위험회피에 비효과적인 부분은 당기손익으로 인식하고 있습니다. 위험회피에 효과적이어서 기타포괄손익으로 인식한 부분은 향후 그 해외사업장을 처분하거나 그 해외사업장의 일부를 처분하는 시점에 기업회계기준서 제1021호 ‘환율변동효과’에 따라 재분류조정으로 기타포괄손익에서 당기손익으로 재분류합니다. 2) 기타 파생상품 위험회피수단으로 지정되어 위험회피에 효과적인 파생상품을 제외한 모든 파생상품은 공정가치로 측정하며, 공정가치 변동으로 인한 평가손익은 당기손익으로 인식하고 있습니다. (이하 생략)
| Model | PPL before CPT | PPL after CPT | Change |
|---|---|---|---|
| Gemma 4 E4B | 4.5161 | 2.2805 | −49.5% |
| Kanana 2 1.3B | 11.0845 | 3.4361 | −69.0% |
In-domain perplexity fell substantially for both models after CPT. The result holds on companies the models never saw in training, with affiliates excluded, which shows that narrative filing text alone, without tables, was enough to improve language modeling in this domain.
The two models use different tokenizers, so their absolute perplexities and percentage reductions are not comparable across models. We compare each model only with its own baseline.
5. Did financial-term discrimination improve?
Yes for both models, but much more for Gemma.
We compared contrastive accuracy and log-likelihood margin before and after CPT on the original prompts of all 300 items.
5.1 Overall results
Gemma ranked the correct answer first on 177 items, up from 146, and its mean margin turned positive. More items were answered correctly, and on average the correct answer now outscores the strongest distractor.
Kanana’s correct items rose from 146 to 152. Its mean margin improved but stayed negative: CPT raised the relative score of correct answers, but on average the strongest distractor still scored higher.
Overall, both models improved after CPT, with a clearly larger gain for Gemma.
Contrastive accuracy, 300 items
Mean log-likelihood margin
Read accuracy and margin together. Accuracy is the share of items where the correct answer ranks first; margin is the score gap between the correct answer and the strongest distractor.
Margin can improve even when the correct answer does not move into first place, as long as the gap narrows. Conversely, mean margin can fall at constant accuracy if a few distractors pull far ahead. Together, the two metrics show both how often the model is right and by how much.
5.2 Which concepts improved?
Gemma 4 E4B
Gemma 4 E4B by question type
T1 barely moved; T2 and T3 improved clearly.
On T2, which asks the model to identify a concept from its definition, accuracy rose 11 points and the mean margin turned positive.
On T3, which asks for the abbreviation that matches a description, accuracy jumped from 33% to 52% (+19pp), and the mean margin improved from −1.516 to −0.387. The margin is still negative, so abbreviations remain the hardest category after CPT.
Kanana 2 1.3B
Kanana 2 1.3B by question type
Kanana improved clearly only on T2: accuracy rose 7 points there, fell 2 points on T1 and was roughly flat on T3. Trained on the same data, its gains were concentrated in concept identification.
On T1, the mean margin improved even though accuracy fell, a reminder that top-1 accuracy and score gaps do not always move together.
5.3 Do the gains hold when questions are rephrased?
To check whether the gains depend on specific wording, we also scored the paraphrased prompts, which keep the meaning but change the phrasing. The Paraphrase lines in the section 3 examples are the actual prompts used.
| Model | Accuracy, before → after | Mean margin, before → after |
|---|---|---|
| Gemma 4 E4B | 47.33% → 58.00% | −0.4634 → +0.0475 |
| Kanana 2 1.3B | 45.00% → 52.33% | −0.3662 → −0.2150 |
Both models improved on the paraphrased prompts as well, in line with the original prompts, so the gains are not tied to one specific wording.
A per-item check of whether each question is answered consistently under both prompt versions is still needed to measure robustness more precisely.
6. Key findings and implications
- CPT on Korean financial filings substantially improved in-domain language modeling.
- On annual reports from 30 held-out companies, in-domain perplexity fell 49.5% for Gemma 4 E4B and 69.0% for Kanana 2 1.3B.
- Continued pretraining on a filing corpus measurably improved both models’ ability to predict Korean financial text.
- Financial-term discrimination improved too, but by different amounts.
- Both models gained in contrastive accuracy and mean log-likelihood margin on the term probe.
- Accuracy rose 10.33pp for Gemma 4 E4B and 2.00pp for Kanana 2 1.3B. Gemma gained mainly on T2 and T3; Kanana mainly on T2.
- The effect of CPT depends on the capability measured and on the model.
- Language modeling improved sharply for both models; term discrimination did not improve to the same degree.
- Lower in-domain perplexity is not a reliable proxy for domain knowledge. Evaluating domain adaptation requires both language modeling metrics and knowledge probes.
Bottom line: continued pretraining on a Korean financial corpus improves in-domain language modeling and can also improve financial-term discrimination, which supports building domain-specific pretraining corpora for finance.
7. Limitations and next steps
Sources: DART FY2025 annual reports, K-IFRS, Bank of Korea Glossary of 800 Economic and Financial Terms. Background: KoSimpleQA (arXiv:2510.18368). Hover over a chart row or move keyboard focus to it to see exact values.