Skip to main content
Read as text

Fine-tuning embedding models for more accurate financial filing retrieval

Widening the similarity gap between positive chunks and hard negatives

In DART annual reports, the same line item often appears on more than one reporting basis: consolidated or separate, current period or prior period. We tested whether fine-tuning can teach an embedding model to tell these apart and rank the chunk that matches the question higher.

We built a training set from 30 DART annual reports and fine-tuned two embedding models, EmbeddingGemma 2 and *kanana-2-3b-embedding. We measured both before and after fine-tuning and benchmarked them against hosted embedding APIs from Upstage and OpenAI.

* kanana-2-3b-embedding is an embedding model released by Kakao, with about 3 billion parameters and support for Korean and English.

Summary

  1. Fine-tuning on filing data sharply improved the models’ ability to separate look-alike financial data.
    • On numerical questions, hard-negative accuracy rose from 0.272 to 0.862 for EmbeddingGemma 2 and from 0.606 to 0.891 for kanana-2-3b-embedding.
    • The gains were clearest where one line item is reported on several bases, such as consolidated versus separate or current versus prior period.
  2. A wider gap between positives and hard negatives translated into better ranking.
    • Across 10 Hyundai Motor examples, the average gap between the positive chunk and the closest hard negative widened from 0.018 to 0.238.
    • On numerical questions, nDCG@10 rose from 0.486 to 0.717 for EmbeddingGemma 2 and from 0.448 to 0.730 for kanana-2-3b-embedding.
  3. Fine-tuning helped numerical questions far more than narrative ones.
    • On narrative questions, hard-negative accuracy improved by +0.147 for EmbeddingGemma 2 and +0.028 for kanana-2-3b-embedding, compared with +0.590 and +0.285 on numerical questions.
    • Teaching a model to recognize reporting bases appears to pay off most for numerical lookups.

Change after fine-tuning (Δ)

ModelQuestion typeHard-negative accuracynDCG@10MRR@10Recall@10
EmbeddingGemma 2Numerical+0.59+0.23+0.21+0.05
EmbeddingGemma 2Narrative+0.15+0.10+0.090.00
kanana-2-3b-embeddingNumerical+0.29+0.28+0.28+0.16
kanana-2-3b-embeddingNarrative+0.03+0.03+0.02+0.01

Why filing retrieval has to tell similar figures apart

Finding a chunk with the right financial data is not enough. The retriever also has to pick the chunk that matches the reporting basis the question asks for.

The example below comes from our training data. The question asks for Hyundai Motor’s consolidated total assets. Before fine-tuning, the model scored a chunk from the separate financial statements higher than the correct consolidated chunk.

현대자동차의 연결 기준 제58기 자산총계는 얼마인가요?

Reference answer 368,844,849백만원

Positive 3. 재무상태 및 영업실적 › 가. 연결 재무상태표

Similarity 0.7982

### 자산총계
* 제58기 > 120,777,363: 368,844,849
* 제57기 > 115,764,287: 339,798,429
Hard negative 1. 요약재무정보 › (2) 요약재무정보(별도재무제표 기준)

Similarity 0.8209

### 자산총계
* 제58기: 98,580,388
* 제57기: 96,143,628
Same company, same line item, same fiscal year. Only the reporting basis differs: consolidated versus separate. The question and filing text appear in the original Korean.

We used hard negatives like this one as the training signal to test whether an embedding model can learn to distinguish reporting bases.

Training data

Each training example is a triplet: a question (the anchor), a chunk that contains the answer (the positive) and a chunk that does not but that the base model scored as highly similar (the hard negative). We built 7,515 triplets from 30 annual reports and split them by company: 15 for training, 5 for validation and 10 for testing. The appendix explains how we built the data and walks through real examples.

Appendix

Building embedding training data from DART annual reports

At the end of this post ↓

How we evaluated

We used four metrics.

Recall@10 Share of all relevant chunks that appear in the top 10 results. MRR@10 (mean reciprocal rank) How high the first relevant chunk ranks. nDCG@10 (normalized discounted cumulative gain) Ranking quality of the top 10 results, weighting each relevant chunk by its position. Hard-negative accuracy Share of triplets in which the question is closer to the positive than to the hard negative. It shows whether a model can tell look-alike data apart, such as consolidated versus separate figures or current versus prior periods.

Recall@10 alone misses ranking quality and the ability to reject hard negatives. If the positive ranks ninth and a hard negative ranks first, Recall@10 still counts a hit.

So we measured MRR@10, nDCG@10 and hard-negative accuracy alongside Recall@10. Together they show whether the right chunk is retrieved, where it ranks and whether the model can separate it from similar chunks that do not meet the question’s conditions.

ModelTypeNotes
EmbeddingGemma 2 baseOpen modelgoogle/embeddinggemma-2, base model
EmbeddingGemma 2 fine-tunedOpen model, fine-tunedFine-tuned on triplets from the 15 training companies
kanana-2-3b-embedding baseOpen model (Korean-focused, 3B)kakaocorp/kanana-2-3b-embedding, base model
kanana-2-3b-embedding LoRAOpen model, LoRA fine-tunedLoRA fine-tuned (r=16) on the same triplets
Solar Embedding 2 (Upstage)Commercial APIsolar-embedding-2-query / -passage
text-embedding-3-small (OpenAI)Commercial API—

Results

Numerical questions

Numerical questions separated the models most. Excluding OpenAI’s text-embedding-3-small, Recall@10 fell in a narrow band of 0.820 to 0.980, while hard-negative accuracy ranged from 0.272 to 0.891.

Getting the right chunk into the top 10 is a different skill from telling it apart from similar data that does not match the question.

EmbeddingGemma 2 base EmbeddingGemma 2 fine-tuned kanana-2-3b-embedding base kanana-2-3b-embedding LoRA Solar Embedding 2 (Upstage) text-embedding-3-small (OpenAI)

Hard-negative accuracy

Numerical questions, 1,815 triplets

nDCG@10

Numerical questions, 150 questions
ModelHard-negative accuracynDCG@10MRR@10Recall@10
EmbeddingGemma 2 base0.2720.4860.6100.907
EmbeddingGemma 2 fine-tuned0.8620.7170.8160.960
kanana-2-3b-embedding base0.6060.4480.5480.820
kanana-2-3b-embedding LoRA0.8910.7300.8320.980
Solar Embedding 2 (Upstage)0.5160.5810.7200.960
text-embedding-3-small (OpenAI)0.4810.2410.3000.540

Both fine-tuned models improved on all four metrics, with the biggest gains in hard-negative accuracy and the ranking metrics.

For EmbeddingGemma 2, hard-negative accuracy rose from 0.272 to 0.862, nDCG@10 from 0.486 to 0.717 and MRR@10 from 0.610 to 0.816. For kanana-2-3b-embedding, hard-negative accuracy rose from 0.606 to 0.891, nDCG@10 from 0.448 to 0.730 and MRR@10 from 0.548 to 0.832.

Recall@10 also rose, from 0.907 to 0.960 for EmbeddingGemma 2 and from 0.820 to 0.980 for kanana-2-3b-embedding.

Fine-tuning did more than surface the right chunk. It pushed the right chunk above look-alike chunks, with the clearest gains where the model had to tell reporting bases apart.

Narrative questions

Narrative questions separated the models less. On questions about the business, strategy and risk, every model except text-embedding-3-small reached Recall@10 of 0.980 or higher, even before fine-tuning.

Hard-negative accuracy

Narrative questions, 831 triplets

nDCG@10

Narrative questions, 150 questions
ModelHard-negative accuracynDCG@10MRR@10Recall@10
EmbeddingGemma 2 base0.7630.7590.7600.980
EmbeddingGemma 2 fine-tuned0.9100.8550.8530.980
kanana-2-3b-embedding base0.9050.8440.8540.987
kanana-2-3b-embedding LoRA0.9330.8740.8740.993
Solar Embedding 2 (Upstage)0.9120.8420.8360.993
text-embedding-3-small (OpenAI)0.8110.6130.6130.860

Fine-tuning still lifted hard-negative accuracy on narrative questions, but by less. EmbeddingGemma 2 rose from 0.763 to 0.910 (+0.147). kanana-2-3b-embedding rose from 0.905 to 0.933 (+0.028), a smaller gain from an already high baseline.

Ten examples, before and after fine-tuning

To see what the metrics mean for a single query, we scored 10 numerical questions from Hyundai Motor’s annual report before and after fine-tuning. Each question has one positive and three hard negatives, and both runs use the same questions and chunks.

Key results (before → after)

  • Questions where a hard negative outscored the positive: 2 → 0 (of 10)
  • Average gap between the positive and the closest hard negative: 0.018 → 0.238
  • Average similarity of positives: 0.781 → 0.714
  • Average similarity of hard negatives: 0.734 → 0.416 (30 chunks)

Similarity by question

Positive Hard negative Gap to the closest hard negative Similarity 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Gap

metric_00 현대자동차의 2025년 연결 자본변동표에서 자기주식 처분에 따른 자본 합계는 얼마인가요?

Before +0.006 After Improved +0.229

metric_01 현대자동차 차량 부문 한국 법인 HMC의 2025년 생산능력은 몇 대인가요?

Before +0.013 After Improved +0.144

metric_02 현대자동차의 제58기 별도재무제표 기준 매출액은 얼마인가요?

Before +0.027 After Improved +0.393

metric_03 현대자동차의 연결 기준 제58기 자산총계는 얼마인가요?

Before −0.023 After Improved +0.232

metric_04 현대자동차의 2025년 연결 당기순이익은 얼마인가요?

Before +0.046 After Improved +0.409

metric_05 2025년 현대자동차 차량부문 부품 매입액 합계는 얼마인가요?

Before +0.005 After Improved +0.194

metric_06 현대자동차의 제58기 연결 기준 매출액은 얼마인가요?

Before −0.013 After Improved +0.173

metric_07 현대자동차의 2025년 말 종속법인 등에 대한 보증 잔액 합계는 얼마인가요?

Before +0.002 After Improved +0.055

metric_08 현대자동차의 2025년 별도 기준 당기순이익은 얼마인가요?

Before +0.054 After Improved +0.377

metric_09 2025년 12월 31일 기준 현대자동차 및 연결대상 회사의 생산설비 기말 계는 얼마인가요?

Before +0.062 After Improved +0.180
Show chunk-level scores
ChunkIDBeforeAfterChange
005380_metric_00: 현대자동차의 2025년 연결 자본변동표에서 자기주식 처분에 따른 자본 합계는 얼마인가요?
Positive3::0120.75440.7802+0.026
Hard negative 13::0050.74820.4167−0.331
Hard negative 23::0090.70770.5516−0.156
Hard negative 33::0100.72880.5243−0.205
005380_metric_01: 현대자동차 차량 부문 한국 법인 HMC의 2025년 생산능력은 몇 대인가요?
Positive2::0070.80140.8080+0.007
Hard negative 12::0080.78870.6644−0.124
Hard negative 22::0810.77370.5383−0.235
Hard negative 32::0040.69280.4735−0.219
005380_metric_02: 현대자동차의 제58기 별도재무제표 기준 매출액은 얼마인가요?
Positive3::0010.78030.7224−0.058
Hard negative 13::0000.75370.3293−0.424
Hard negative 24::0110.70150.2089−0.493
Hard negative 33::8750.70000.2939−0.406
005380_metric_03: 현대자동차의 연결 기준 제58기 자산총계는 얼마인가요?
Positive4::0110.79820.6731−0.125
Hard negative 13::0010.82090.3053−0.516
Hard negative 21::0100.70350.3118−0.392
Hard negative 33::0050.71780.4415−0.276
005380_metric_04: 현대자동차의 2025년 연결 당기순이익은 얼마인가요?
Positive3::0040.75300.7271−0.026
Hard negative 13::0250.70740.3181−0.389
Hard negative 23::0260.69420.3075−0.387
Hard negative 32::0140.69370.2649−0.429
005380_metric_05: 2025년 현대자동차 차량부문 부품 매입액 합계는 얼마인가요?
Positive2::0050.75740.6640−0.093
Hard negative 12::0140.71690.2821−0.435
Hard negative 22::0110.75210.4705−0.282
Hard negative 32::0130.70540.3530−0.352
005380_metric_06: 현대자동차의 제58기 연결 기준 매출액은 얼마인가요?
Positive3::0000.76340.6526−0.111
Hard negative 13::0010.77660.3369−0.440
Hard negative 22::0130.73260.4799−0.253
Hard negative 34::0110.71630.4629−0.253
005380_metric_07: 현대자동차의 2025년 말 종속법인 등에 대한 보증 잔액 합계는 얼마인가요?
Positive4::0220.82410.6978−0.126
Hard negative 111::0000.79870.6284−0.170
Hard negative 211::0030.82230.6428−0.179
Hard negative 311::0020.78560.5919−0.194
005380_metric_08: 현대자동차의 2025년 별도 기준 당기순이익은 얼마인가요?
Positive3::4220.78780.7584−0.029
Hard negative 12::0990.73340.3665−0.367
Hard negative 23::7090.72750.3812−0.346
Hard negative 33::7020.67400.2786−0.395
005380_metric_09: 2025년 12월 31일 기준 현대자동차 및 연결대상 회사의 생산설비 기말 계는 얼마인가요?
Positive2::0100.79060.6599−0.131
Hard negative 12::0080.72840.4795−0.249
Hard negative 22::0140.71000.2983−0.412
Hard negative 32::0070.71510.4666−0.248

After fine-tuning, positives scored slightly lower on average, but hard negatives fell much further, so the gap widened. Before fine-tuning, a hard negative outscored the positive on 2 of the 10 questions. Afterward, the positive scored highest on all 10.

Fine-tuning did not simply raise scores for the right chunks. It taught the model to push away look-alike chunks that do not meet the question’s conditions.

Key findings and implications

  1. Domain-specific hard negatives improved the models’ ability to separate look-alike financial data.
    • Our hard negatives included chunks that read almost the same but differ in reporting basis, such as consolidated versus separate or current versus prior period.
    • After fine-tuning, hard-negative accuracy on numerical questions rose from 0.272 to 0.862 for EmbeddingGemma 2 and from 0.606 to 0.891 for kanana-2-3b-embedding.
  2. A wider similarity gap meant better ranking.
    • Across 10 Hyundai Motor examples, the average gap between the positive and the closest hard negative widened from 0.018 to 0.238.
    • On numerical questions, nDCG@10 rose from 0.486 to 0.717 for EmbeddingGemma 2 and from 0.448 to 0.730 for kanana-2-3b-embedding, so positives ranked higher.
  3. The effect was larger for numerical questions than for narrative ones.
    • Both models improved on narrative questions, but their hard-negative accuracy gains were larger on numerical questions.
    • For financial retrieval, training data should capture differences in reporting basis, not just general semantic similarity.

Overall, hard negatives built around financial-domain distinctions, combined with fine-tuning, improved retrieval quality. The results make a case for building domain-specific training data for financial retrieval.

Evaluation data: DART annual report test split (training_dataset/dart/triplets/). Coverage: FY2025 annual reports. Question generation: gpt-6-luna. Labeling: Jev 1.13 (TypeSafe). Evaluation outputs: outputs/dart/api_eval/evaluation.json (API models), outputs/dart/colab/colab_run/evaluation.json (EmbeddingGemma 2), outputs/dart/kanana_lora/evaluation.json (kanana-2-3b-embedding).

Appendix

Building embedding training data from DART annual reports

We built 7,515 training triplets from 30 annual reports. An LLM wrote the questions, and Jev scored every chunk in each report to decide which ones contain the answer.

Training example structure

Each example is a triplet: one question and two chunks. Training pulls the question toward the positive chunk and pushes it away from the negative one.

Anchor Question

Written by an LLM from a chunk of an annual report

Positive Answer chunk

Contains the answer to the question

Hard negative Look-alike wrong chunk

Does not contain the answer, but the base model scores it about as close to the question as the positive

30 annual reports (one per company) 33,169 chunks 900 questions 7,515 triplets
Question typeWhat it asks forQuestionsPositives per questionTriplets
NumericalA specific figure, with the basis (consolidated or separate) and the period stated4483.95,283
NarrativeAn explanation of the business, strategy or risks4491.72,232

We split the data by company: 15 training companies (3,687 triplets), 5 validation companies (1,182) and 10 test companies (2,646). Reports from test companies never appear in training.

How we built it

  1. Chunk preparation

    1,106 chunks per report on average

    We reused the existing parsed chunks as is. Tables, XBRL financial statements and narrative paragraphs each form their own chunks.

  2. Question generation

    LLM 30 per report

    The LLM sees one chunk and writes a question with a reference answer. Numerical questions must state the basis (consolidated or separate) and the period, and answers keep the units used in the table.

  3. Exhaustive labeling

    Jev

    Each question is checked against every chunk in its report. With no first-stage retrieval to shortlist candidates, no positive can slip through.

    LabelWhat Jev is askedUsed to
    answerDoes this chunk contain the full reference answer?Select positives
    relevantDoes this chunk provide information that helps answer the question?Filter out ambiguous chunks

    Jev returns a probability between 0 and 1 for each label.

  4. Labeling rules

    Rules

    We bucket chunks by answer score. If the chunk a question was generated from is not labeled positive, the question is dropped.

    Negative Discard Positive 0 0.2 0.7 1
  5. Hard-negative mining

    kanana-2-3b 3 per positive

    Training examples

    Below are examples of the three hard negatives that the base kanana-2-3b model scored as most similar to each question. Questions and filing excerpts appear in the original Korean.

    Numerical question Hyundai Motor, FY2025 annual report 005380_metric_03

    현대자동차의 연결 기준 제58기 자산총계는 얼마인가요?

    Reference answer 368,844,849백만원

    Positive 3. 재무상태 및 영업실적 › 가. 연결 재무상태표 4::011 Similarity 0.7982
    가. 연결 재무상태표
    (단위 : 백만원)
    ⋮
    ### 자산총계
    * 제58기 > 120,777,363: 368,844,849
    * 제57기 > 115,764,287: 339,798,429
    ⋮
    Hard negative 1 1. 요약재무정보 › (2) 요약재무정보(별도재무제표 기준) 3::001 Similarity 0.8209
    (2) 요약재무정보(별도재무제표 기준)
    (단위 : 백만원)
    ⋮
    ### 자산총계
    * 제58기: 98,580,388
    * 제57기: 96,143,628
    ⋮
    Hard negative 2 1-회사의 개요 › 3. 자본금 변동사항 1::010 Similarity 0.7035
    3. 자본금 변동사항
    **자본금 변동추이** (단위 : 원, 주)
    ⋮
    ### 종류: 보통주
    #### 구분: 발행주식총수
    * 58기 (2025년말): 204,757,766
    * 57기 (2024년말): 209,416,191
    ⋮
    Hard negative 3 2. 연결재무제표 › 2-4. 연결 자본변동표 3::005 Similarity 0.7178
    2-4. 연결 자본변동표
    ⋮
    ## 자본변동표 - 연결
    ⋮
    ### 기초자본
    (dart:EquityAtBeginningOfPeriod)
    * 자본 합계 · 2025-01-01~2025-12-31 : 120275933000000 KRW
    ⋮
    Numerical question Hyundai Motor, FY2025 annual report 005380_metric_04

    현대자동차의 2025년 연결 당기순이익은 얼마인가요?

    Reference answer 10,364,775백만원

    Positive 2. 연결재무제표 › 2-3. 연결 포괄손익계산서 3::004 Similarity 0.7530
    2-3. 연결 포괄손익계산서
    ⋮
    ## 포괄손익계산서, 세후로 표시된 OCI 구성요소 - 연결
    ⋮
    ### 연결당기순이익
    (ifrs-full:ProfitLoss)
    * 2025-01-01~2025-12-31 : 10364775000000 KRW
    * 2024-01-01~2024-12-31 : 13229908000000 KRW
    ⋮
    Hard negative 1 3. 연결재무제표 주석 › 1. 회사 및 종속기업의 개요 (연결) 3::025 Similarity 0.7074
    (2) 당기의 연결대상 주요 종속기업의 요약재무상태 및 경영성과는 다음과 같습니다.
    (단위 : 백만원)
    ⋮
    ## 종속기업명
    ⋮
    ### 현대캐피탈(주)(*)
    * 총자산: 42,534,581
    * 총부채: 35,246,438
    * 매출액: 5,389,767
    ⋮
    Hard negative 2 3. 연결재무제표 주석 › 1. 회사 및 종속기업의 개요 (연결) 3::026 Similarity 0.6942
    전기의 연결대상 주요 종속기업의 요약재무상태 및 경영성과는 다음과 같습니다.
    (단위 : 백만원)
    ⋮
    ## 종속기업명
    ⋮
    ### 현대캐피탈(주)(*)
    * 총자산: 40,413,943
    * 총부채: 33,865,870
    * 매출액: 5,885,785
    * 당기순손익: 432,704
    ⋮
    Hard negative 3 4. (제조서비스업)매출 및 수주상황 › 가. 매출실적 2::014 Similarity 0.6937
    가. 매출실적
    ③ 현대로템 매출실적 (연결기준)
    ⋮
    (단위: 억원)
    ⋮
    ### 사업구분: 디펜스솔루션 부문
    #### 품 목: 방산물자 > 내수
    * 2025년 (제27기): 8,861
    * 2024년 (제26기): 7,735
    ⋮
    Numerical question Hyundai Motor, FY2025 annual report 005380_metric_07

    현대자동차의 2025년 말 종속법인 등에 대한 보증 잔액 합계는 얼마인가요?

    Reference answer 2,974,593백만원

    Positive 4-이사의 경영진단 및 분석의견 › 5. 부외거래 4::022 Similarity 0.8241
    5. 부외거래
    당기말 현재 종속법인 등에 대한 보증내역은 아래와 같습니다.
    ⋮
    (단위 : 백만원)
    ⋮
    ### 현대로템
    * 관계: 국내 종속법인
    * 목적: 계약이행보증
    ⋮
    ### 합 계
    * 거래내역:
      * 기초: 3,148,523
      * 증가: -
      * 감소: -173,930
      * 기말: 2,974,593
    Hard negative 1 2. 우발부채 등에 관한 사항 › 가. 당기말 현재 연결실체가 종속기업을 제외한 특수관계자 등의 채무 등에 대해 제공한 11::000 Similarity 0.7987
    2. 우발부채 등에 관한 사항
    가. 당기말 현재 연결실체가 종속기업을 제외한 특수관계자 등의 채무 등에 대해 제공한
    지급보증 내역은 다음과 같습니다.
    ⋮
    (단위 : 백만원)
    ⋮
    ## 상대방
    ⋮
    ### 관계기업
    ⋮
    Hard negative 2 2. 우발부채 등에 관한 사항 › 가. 당기말 현재 연결실체가 종속기업을 제외한 특수관계자 등의 채무 등에 대해 제공한 11::003 Similarity 0.8223
    전기말 종속법인 등에 대한 보증내역은 아래와 같습니다.
    ⋮
    (단위 : 백만원)
    ⋮
    ### 현대로템
    * 관계: 국내 종속법인
    * 목적: 계약이행보증
    ⋮
    ### 합 계
    * 거래내역:
      * 기초: 1,040,245
      * 증가: 2,109,348
      * 감소: -1,070
      * 기말: 3,148,523
    Hard negative 3 2. 우발부채 등에 관한 사항 › 가. 당기말 현재 연결실체가 종속기업을 제외한 특수관계자 등의 채무 등에 대해 제공한 11::002 Similarity 0.7856
    (*) 2025년 2월 중 Hyundai Assan Otomotiv Sanayi Ve Ticaret A.S. (HAOSVT)의 사명을
    Hyundai Motor Turkiye Otomotiv A.S, (HMTR) 으로 변경하였음.
    ※ 외화 지급보증금액은 보고기간종료일 현재의 서울외국환중개주식회사가 고시하는 기준환율로
    환산한 금액임.
    ⋮
    전기말 연결실체가 제공한 지급보증의 내역은 다음과 같습니다.
  6. Triplet assembly

    We pair each positive with its three hard negatives, then split the triplets into training, validation and test sets by company.

Coverage: 30 FY2025 annual reports. Question generation: gpt-6-luna. Labeling: Jev 1.13 (TypeSafe). Hard-negative mining: kanana-2-3b (base model).