Fine-tuning embedding models for more accurate financial filing retrieval
Widening the similarity gap between positive chunks and hard negatives
In DART annual reports, the same line item often appears on more than one reporting basis: consolidated or separate, current period or prior period. We tested whether fine-tuning can teach an embedding model to tell these apart and rank the chunk that matches the question higher.
We built a training set from 30 DART annual reports and fine-tuned two embedding models, EmbeddingGemma 2 and *kanana-2-3b-embedding. We measured both before and after fine-tuning and benchmarked them against hosted embedding APIs from Upstage and OpenAI.
* kanana-2-3b-embedding is an embedding model released by Kakao, with about 3 billion parameters and support for Korean and English.
Summary
Fine-tuning on filing data sharply improved the models’ ability to separate look-alike financial data.
On numerical questions, hard-negative accuracy rose from 0.272 to 0.862 for EmbeddingGemma 2 and from 0.606 to 0.891 for kanana-2-3b-embedding.
The gains were clearest where one line item is reported on several bases, such as consolidated versus separate or current versus prior period.
A wider gap between positives and hard negatives translated into better ranking.
Across 10 Hyundai Motor examples, the average gap between the positive chunk and the closest hard negative widened from 0.018 to 0.238.
On numerical questions, nDCG@10 rose from 0.486 to 0.717 for EmbeddingGemma 2 and from 0.448 to 0.730 for kanana-2-3b-embedding.
Fine-tuning helped numerical questions far more than narrative ones.
On narrative questions, hard-negative accuracy improved by +0.147 for EmbeddingGemma 2 and +0.028 for kanana-2-3b-embedding, compared with +0.590 and +0.285 on numerical questions.
Teaching a model to recognize reporting bases appears to pay off most for numerical lookups.
Change after fine-tuning (Δ)
Model
Question type
Hard-negative accuracy
nDCG@10
MRR@10
Recall@10
EmbeddingGemma 2
Numerical
+0.59
+0.23
+0.21
+0.05
EmbeddingGemma 2
Narrative
+0.15
+0.10
+0.09
0.00
kanana-2-3b-embedding
Numerical
+0.29
+0.28
+0.28
+0.16
kanana-2-3b-embedding
Narrative
+0.03
+0.03
+0.02
+0.01
Why filing retrieval has to tell similar figures apart
Finding a chunk with the right financial data is not enough. The retriever also has to pick the chunk that matches the reporting basis the question asks for.
The example below comes from our training data. The question asks for Hyundai Motor’s consolidated total assets. Before fine-tuning, the model scored a chunk from the separate financial statements higher than the correct consolidated chunk.
Same company, same line item, same fiscal year. Only the reporting basis differs: consolidated versus separate. The question and filing text appear in the original Korean.
We used hard negatives like this one as the training signal to test whether an embedding model can learn to distinguish reporting bases.
Training data
Each training example is a triplet: a question (the anchor), a chunk that contains the answer (the positive) and a chunk that does not but that the base model scored as highly similar (the hard negative). We built 7,515 triplets from 30 annual reports and split them by company: 15 for training, 5 for validation and 10 for testing. The appendix explains how we built the data and walks through real examples.
Appendix
Building embedding training data from DART annual reports
Recall@10Share of all relevant chunks that appear in the top 10 results.
MRR@10 (mean reciprocal rank)How high the first relevant chunk ranks.
nDCG@10 (normalized discounted cumulative gain)Ranking quality of the top 10 results, weighting each relevant chunk by its position.
Hard-negative accuracyShare of triplets in which the question is closer to the positive than to the hard negative. It shows whether a model can tell look-alike data apart, such as consolidated versus separate figures or current versus prior periods.
Recall@10 alone misses ranking quality and the ability to reject hard negatives. If the positive ranks ninth and a hard negative ranks first, Recall@10 still counts a hit.
So we measured MRR@10, nDCG@10 and hard-negative accuracy alongside Recall@10. Together they show whether the right chunk is retrieved, where it ranks and whether the model can separate it from similar chunks that do not meet the question’s conditions.
Model
Type
Notes
EmbeddingGemma 2 base
Open model
google/embeddinggemma-2, base model
EmbeddingGemma 2 fine-tuned
Open model, fine-tuned
Fine-tuned on triplets from the 15 training companies
kanana-2-3b-embedding base
Open model (Korean-focused, 3B)
kakaocorp/kanana-2-3b-embedding, base model
kanana-2-3b-embedding LoRA
Open model, LoRA fine-tuned
LoRA fine-tuned (r=16) on the same triplets
Solar Embedding 2 (Upstage)
Commercial API
solar-embedding-2-query / -passage
text-embedding-3-small (OpenAI)
Commercial API
—
Results
Numerical questions
Numerical questions separated the models most. Excluding OpenAI’s text-embedding-3-small, Recall@10 fell in a narrow band of 0.820 to 0.980, while hard-negative accuracy ranged from 0.272 to 0.891.
Getting the right chunk into the top 10 is a different skill from telling it apart from similar data that does not match the question.
Both fine-tuned models improved on all four metrics, with the biggest gains in hard-negative accuracy and the ranking metrics.
For EmbeddingGemma 2, hard-negative accuracy rose from 0.272 to 0.862, nDCG@10 from 0.486 to 0.717 and MRR@10 from 0.610 to 0.816. For kanana-2-3b-embedding, hard-negative accuracy rose from 0.606 to 0.891, nDCG@10 from 0.448 to 0.730 and MRR@10 from 0.548 to 0.832.
Recall@10 also rose, from 0.907 to 0.960 for EmbeddingGemma 2 and from 0.820 to 0.980 for kanana-2-3b-embedding.
Fine-tuning did more than surface the right chunk. It pushed the right chunk above look-alike chunks, with the clearest gains where the model had to tell reporting bases apart.
Narrative questions
Narrative questions separated the models less. On questions about the business, strategy and risk, every model except text-embedding-3-small reached Recall@10 of 0.980 or higher, even before fine-tuning.
Hard-negative accuracy
Narrative questions, 831 triplets
nDCG@10
Narrative questions, 150 questions
Model
Hard-negative accuracy
nDCG@10
MRR@10
Recall@10
EmbeddingGemma 2 base
0.763
0.759
0.760
0.980
EmbeddingGemma 2 fine-tuned
0.910
0.855
0.853
0.980
kanana-2-3b-embedding base
0.905
0.844
0.854
0.987
kanana-2-3b-embedding LoRA
0.933
0.874
0.874
0.993
Solar Embedding 2 (Upstage)
0.912
0.842
0.836
0.993
text-embedding-3-small (OpenAI)
0.811
0.613
0.613
0.860
Fine-tuning still lifted hard-negative accuracy on narrative questions, but by less. EmbeddingGemma 2 rose from 0.763 to 0.910 (+0.147). kanana-2-3b-embedding rose from 0.905 to 0.933 (+0.028), a smaller gain from an already high baseline.
Ten examples, before and after fine-tuning
To see what the metrics mean for a single query, we scored 10 numerical questions from Hyundai Motor’s annual report before and after fine-tuning. Each question has one positive and three hard negatives, and both runs use the same questions and chunks.
Key results (before → after)
Questions where a hard negative outscored the positive: 2 → 0 (of 10)
Average gap between the positive and the closest hard negative: 0.018 → 0.238
Average similarity of positives: 0.781 → 0.714
Average similarity of hard negatives: 0.734 → 0.416 (30 chunks)
Similarity by question
PositiveHard negativeGap to the closest hard negative
Similarity
0.20.30.40.50.60.70.80.9
Gap
metric_00현대자동차의 2025년 연결 자본변동표에서 자기주식 처분에 따른 자본 합계는 얼마인가요?
Before
+0.006
After
Improved+0.229
metric_01현대자동차 차량 부문 한국 법인 HMC의 2025년 생산능력은 몇 대인가요?
Before
+0.013
After
Improved+0.144
metric_02현대자동차의 제58기 별도재무제표 기준 매출액은 얼마인가요?
Before
+0.027
After
Improved+0.393
metric_03현대자동차의 연결 기준 제58기 자산총계는 얼마인가요?
Before
−0.023
After
Improved+0.232
metric_04현대자동차의 2025년 연결 당기순이익은 얼마인가요?
Before
+0.046
After
Improved+0.409
metric_052025년 현대자동차 차량부문 부품 매입액 합계는 얼마인가요?
Before
+0.005
After
Improved+0.194
metric_06현대자동차의 제58기 연결 기준 매출액은 얼마인가요?
Before
−0.013
After
Improved+0.173
metric_07현대자동차의 2025년 말 종속법인 등에 대한 보증 잔액 합계는 얼마인가요?
Before
+0.002
After
Improved+0.055
metric_08현대자동차의 2025년 별도 기준 당기순이익은 얼마인가요?
Before
+0.054
After
Improved+0.377
metric_092025년 12월 31일 기준 현대자동차 및 연결대상 회사의 생산설비 기말 계는 얼마인가요?
Before
+0.062
After
Improved+0.180
Show chunk-level scores
Chunk
ID
Before
After
Change
005380_metric_00: 현대자동차의 2025년 연결 자본변동표에서 자기주식 처분에 따른 자본 합계는 얼마인가요?
Positive
3::012
0.7544
0.7802
+0.026
Hard negative 1
3::005
0.7482
0.4167
−0.331
Hard negative 2
3::009
0.7077
0.5516
−0.156
Hard negative 3
3::010
0.7288
0.5243
−0.205
005380_metric_01: 현대자동차 차량 부문 한국 법인 HMC의 2025년 생산능력은 몇 대인가요?
Positive
2::007
0.8014
0.8080
+0.007
Hard negative 1
2::008
0.7887
0.6644
−0.124
Hard negative 2
2::081
0.7737
0.5383
−0.235
Hard negative 3
2::004
0.6928
0.4735
−0.219
005380_metric_02: 현대자동차의 제58기 별도재무제표 기준 매출액은 얼마인가요?
Positive
3::001
0.7803
0.7224
−0.058
Hard negative 1
3::000
0.7537
0.3293
−0.424
Hard negative 2
4::011
0.7015
0.2089
−0.493
Hard negative 3
3::875
0.7000
0.2939
−0.406
005380_metric_03: 현대자동차의 연결 기준 제58기 자산총계는 얼마인가요?
Positive
4::011
0.7982
0.6731
−0.125
Hard negative 1
3::001
0.8209
0.3053
−0.516
Hard negative 2
1::010
0.7035
0.3118
−0.392
Hard negative 3
3::005
0.7178
0.4415
−0.276
005380_metric_04: 현대자동차의 2025년 연결 당기순이익은 얼마인가요?
Positive
3::004
0.7530
0.7271
−0.026
Hard negative 1
3::025
0.7074
0.3181
−0.389
Hard negative 2
3::026
0.6942
0.3075
−0.387
Hard negative 3
2::014
0.6937
0.2649
−0.429
005380_metric_05: 2025년 현대자동차 차량부문 부품 매입액 합계는 얼마인가요?
Positive
2::005
0.7574
0.6640
−0.093
Hard negative 1
2::014
0.7169
0.2821
−0.435
Hard negative 2
2::011
0.7521
0.4705
−0.282
Hard negative 3
2::013
0.7054
0.3530
−0.352
005380_metric_06: 현대자동차의 제58기 연결 기준 매출액은 얼마인가요?
Positive
3::000
0.7634
0.6526
−0.111
Hard negative 1
3::001
0.7766
0.3369
−0.440
Hard negative 2
2::013
0.7326
0.4799
−0.253
Hard negative 3
4::011
0.7163
0.4629
−0.253
005380_metric_07: 현대자동차의 2025년 말 종속법인 등에 대한 보증 잔액 합계는 얼마인가요?
Positive
4::022
0.8241
0.6978
−0.126
Hard negative 1
11::000
0.7987
0.6284
−0.170
Hard negative 2
11::003
0.8223
0.6428
−0.179
Hard negative 3
11::002
0.7856
0.5919
−0.194
005380_metric_08: 현대자동차의 2025년 별도 기준 당기순이익은 얼마인가요?
Positive
3::422
0.7878
0.7584
−0.029
Hard negative 1
2::099
0.7334
0.3665
−0.367
Hard negative 2
3::709
0.7275
0.3812
−0.346
Hard negative 3
3::702
0.6740
0.2786
−0.395
005380_metric_09: 2025년 12월 31일 기준 현대자동차 및 연결대상 회사의 생산설비 기말 계는 얼마인가요?
Positive
2::010
0.7906
0.6599
−0.131
Hard negative 1
2::008
0.7284
0.4795
−0.249
Hard negative 2
2::014
0.7100
0.2983
−0.412
Hard negative 3
2::007
0.7151
0.4666
−0.248
After fine-tuning, positives scored slightly lower on average, but hard negatives fell much further, so the gap widened. Before fine-tuning, a hard negative outscored the positive on 2 of the 10 questions. Afterward, the positive scored highest on all 10.
Fine-tuning did not simply raise scores for the right chunks. It taught the model to push away look-alike chunks that do not meet the question’s conditions.
Key findings and implications
Domain-specific hard negatives improved the models’ ability to separate look-alike financial data.
Our hard negatives included chunks that read almost the same but differ in reporting basis, such as consolidated versus separate or current versus prior period.
After fine-tuning, hard-negative accuracy on numerical questions rose from 0.272 to 0.862 for EmbeddingGemma 2 and from 0.606 to 0.891 for kanana-2-3b-embedding.
A wider similarity gap meant better ranking.
Across 10 Hyundai Motor examples, the average gap between the positive and the closest hard negative widened from 0.018 to 0.238.
On numerical questions, nDCG@10 rose from 0.486 to 0.717 for EmbeddingGemma 2 and from 0.448 to 0.730 for kanana-2-3b-embedding, so positives ranked higher.
The effect was larger for numerical questions than for narrative ones.
Both models improved on narrative questions, but their hard-negative accuracy gains were larger on numerical questions.
For financial retrieval, training data should capture differences in reporting basis, not just general semantic similarity.
Overall, hard negatives built around financial-domain distinctions, combined with fine-tuning, improved retrieval quality. The results make a case for building domain-specific training data for financial retrieval.
Building embedding training data from DART annual reports
We built 7,515 training triplets from 30 annual reports. An LLM wrote the questions, and Jev scored every chunk in each report to decide which ones contain the answer.
Training example structure
Each example is a triplet: one question and two chunks. Training pulls the question toward the positive chunk and pushes it away from the negative one.
Anchor
Question
Written by an LLM from a chunk of an annual report
Positive
Answer chunk
Contains the answer to the question
Hard negative
Look-alike wrong chunk
Does not contain the answer, but the base model scores it about as close to the question as the positive
30annual reports (one per company)
33,169chunks
900questions
7,515triplets
Question type
What it asks for
Questions
Positives per question
Triplets
Numerical
A specific figure, with the basis (consolidated or separate) and the period stated
448
3.9
5,283
Narrative
An explanation of the business, strategy or risks
449
1.7
2,232
We split the data by company: 15 training companies (3,687 triplets), 5 validation companies (1,182) and 10 test companies (2,646). Reports from test companies never appear in training.
How we built it
Chunk preparation
1,106 chunks per report on average
We reused the existing parsed chunks as is. Tables, XBRL financial statements and narrative paragraphs each form their own chunks.
Question generation
LLM30 per report
The LLM sees one chunk and writes a question with a reference answer. Numerical questions must state the basis (consolidated or separate) and the period, and answers keep the units used in the table.
Exhaustive labeling
Jev
Each question is checked against every chunk in its report. With no first-stage retrieval to shortlist candidates, no positive can slip through.
Label
What Jev is asked
Used to
answer
Does this chunk contain the full reference answer?
Select positives
relevant
Does this chunk provide information that helps answer the question?
Filter out ambiguous chunks
Jev returns a probability between 0 and 1 for each label.
Labeling rules
Rules
We bucket chunks by answer score. If the chunk a question was generated from is not labeled positive, the question is dropped.
Negative
Discard
Positive
00.20.71
Hard-negative mining
kanana-2-3b3 per positive
Training examples
Below are examples of the three hard negatives that the base kanana-2-3b model scored as most similar to each question. Questions and filing excerpts appear in the original Korean.
5. 부외거래
당기말 현재 종속법인 등에 대한 보증내역은 아래와 같습니다.
⋮
(단위 : 백만원)
⋮
### 현대로템
* 관계: 국내 종속법인
* 목적: 계약이행보증
⋮
### 합 계
* 거래내역:
* 기초: 3,148,523
* 증가: -
* 감소: -173,930
* 기말: 2,974,593
Hard negative 12. 우발부채 등에 관한 사항 › 가. 당기말 현재 연결실체가 종속기업을 제외한 특수관계자 등의 채무 등에 대해 제공한11::000
Similarity0.7987
2. 우발부채 등에 관한 사항
가. 당기말 현재 연결실체가 종속기업을 제외한 특수관계자 등의 채무 등에 대해 제공한
지급보증 내역은 다음과 같습니다.
⋮
(단위 : 백만원)
⋮
## 상대방
⋮
### 관계기업
⋮
Hard negative 22. 우발부채 등에 관한 사항 › 가. 당기말 현재 연결실체가 종속기업을 제외한 특수관계자 등의 채무 등에 대해 제공한11::003
Similarity0.8223
전기말 종속법인 등에 대한 보증내역은 아래와 같습니다.
⋮
(단위 : 백만원)
⋮
### 현대로템
* 관계: 국내 종속법인
* 목적: 계약이행보증
⋮
### 합 계
* 거래내역:
* 기초: 1,040,245
* 증가: 2,109,348
* 감소: -1,070
* 기말: 3,148,523
Hard negative 32. 우발부채 등에 관한 사항 › 가. 당기말 현재 연결실체가 종속기업을 제외한 특수관계자 등의 채무 등에 대해 제공한11::002
Similarity0.7856
(*) 2025년 2월 중 Hyundai Assan Otomotiv Sanayi Ve Ticaret A.S. (HAOSVT)의 사명을
Hyundai Motor Turkiye Otomotiv A.S, (HMTR) 으로 변경하였음.
※ 외화 지급보증금액은 보고기간종료일 현재의 서울외국환중개주식회사가 고시하는 기준환율로
환산한 금액임.
⋮
전기말 연결실체가 제공한 지급보증의 내역은 다음과 같습니다.
Triplet assembly
We pair each positive with its three hard negatives, then split the triplets into training, validation and test sets by company.
Fine-tuning embedding models for more accurate financial filing retrieval
Widening the similarity gap between positive chunks and hard negatives
In DART annual reports, the same line item often appears on more than one reporting basis: consolidated or separate, current period or prior period. We tested whether fine-tuning can teach an embedding model to tell these apart and rank the chunk that matches the question higher.
We built a training set from 30 DART annual reports and fine-tuned two embedding models, EmbeddingGemma 2 and *kanana-2-3b-embedding. We measured both before and after fine-tuning and benchmarked them against hosted embedding APIs from Upstage and OpenAI.
* kanana-2-3b-embedding is an embedding model released by Kakao, with about 3 billion parameters and support for Korean and English.
Summary
Fine-tuning on filing data sharply improved the models’ ability to separate look-alike financial data.
On numerical questions, hard-negative accuracy rose from 0.272 to 0.862 for EmbeddingGemma 2 and from 0.606 to 0.891 for kanana-2-3b-embedding.
The gains were clearest where one line item is reported on several bases, such as consolidated versus separate or current versus prior period.
A wider gap between positives and hard negatives translated into better ranking.
Across 10 Hyundai Motor examples, the average gap between the positive chunk and the closest hard negative widened from 0.018 to 0.238.
On numerical questions, nDCG@10 rose from 0.486 to 0.717 for EmbeddingGemma 2 and from 0.448 to 0.730 for kanana-2-3b-embedding.
Fine-tuning helped numerical questions far more than narrative ones.
On narrative questions, hard-negative accuracy improved by +0.147 for EmbeddingGemma 2 and +0.028 for kanana-2-3b-embedding, compared with +0.590 and +0.285 on numerical questions.
Teaching a model to recognize reporting bases appears to pay off most for numerical lookups.
Change after fine-tuning (Δ)
Model
Question type
Hard-negative accuracy
nDCG@10
MRR@10
Recall@10
EmbeddingGemma 2
Numerical
+0.59
+0.23
+0.21
+0.05
EmbeddingGemma 2
Narrative
+0.15
+0.10
+0.09
0.00
kanana-2-3b-embedding
Numerical
+0.29
+0.28
+0.28
+0.16
kanana-2-3b-embedding
Narrative
+0.03
+0.03
+0.02
+0.01
Why filing retrieval has to tell similar figures apart
Finding a chunk with the right financial data is not enough. The retriever also has to pick the chunk that matches the reporting basis the question asks for.
The example below comes from our training data. The question asks for Hyundai Motor’s consolidated total assets. Before fine-tuning, the model scored a chunk from the separate financial statements higher than the correct consolidated chunk.
Same company, same line item, same fiscal year. Only the reporting basis differs: consolidated versus separate. The question and filing text appear in the original Korean.
We used hard negatives like this one as the training signal to test whether an embedding model can learn to distinguish reporting bases.
Training data
Each training example is a triplet: a question (the anchor), a chunk that contains the answer (the positive) and a chunk that does not but that the base model scored as highly similar (the hard negative). We built 7,515 triplets from 30 annual reports and split them by company: 15 for training, 5 for validation and 10 for testing. The appendix explains how we built the data and walks through real examples.
Appendix
Building embedding training data from DART annual reports
At the end of this post ↓
How we evaluated
We used four metrics.
Recall@10 Share of all relevant chunks that appear in the top 10 results. MRR@10 (mean reciprocal rank) How high the first relevant chunk ranks. nDCG@10 (normalized discounted cumulative gain) Ranking quality of the top 10 results, weighting each relevant chunk by its position. Hard-negative accuracy Share of triplets in which the question is closer to the positive than to the hard negative. It shows whether a model can tell look-alike data apart, such as consolidated versus separate figures or current versus prior periods.
Recall@10 alone misses ranking quality and the ability to reject hard negatives. If the positive ranks ninth and a hard negative ranks first, Recall@10 still counts a hit.
So we measured MRR@10, nDCG@10 and hard-negative accuracy alongside Recall@10. Together they show whether the right chunk is retrieved, where it ranks and whether the model can separate it from similar chunks that do not meet the question’s conditions.
Model
Type
Notes
EmbeddingGemma 2 base
Open model
google/embeddinggemma-2, base model
EmbeddingGemma 2 fine-tuned
Open model, fine-tuned
Fine-tuned on triplets from the 15 training companies
kanana-2-3b-embedding base
Open model (Korean-focused, 3B)
kakaocorp/kanana-2-3b-embedding, base model
kanana-2-3b-embedding LoRA
Open model, LoRA fine-tuned
LoRA fine-tuned (r=16) on the same triplets
Solar Embedding 2 (Upstage)
Commercial API
solar-embedding-2-query / -passage
text-embedding-3-small (OpenAI)
Commercial API
—
Results
Numerical questions
Numerical questions separated the models most. Excluding OpenAI’s text-embedding-3-small, Recall@10 fell in a narrow band of 0.820 to 0.980, while hard-negative accuracy ranged from 0.272 to 0.891.
Getting the right chunk into the top 10 is a different skill from telling it apart from similar data that does not match the question.
EmbeddingGemma 2 base EmbeddingGemma 2 fine-tuned kanana-2-3b-embedding base kanana-2-3b-embedding LoRA Solar Embedding 2 (Upstage) text-embedding-3-small (OpenAI)
Hard-negative accuracy
Numerical questions, 1,815 triplets
nDCG@10
Numerical questions, 150 questions
Model
Hard-negative accuracy
nDCG@10
MRR@10
Recall@10
EmbeddingGemma 2 base
0.272
0.486
0.610
0.907
EmbeddingGemma 2 fine-tuned
0.862
0.717
0.816
0.960
kanana-2-3b-embedding base
0.606
0.448
0.548
0.820
kanana-2-3b-embedding LoRA
0.891
0.730
0.832
0.980
Solar Embedding 2 (Upstage)
0.516
0.581
0.720
0.960
text-embedding-3-small (OpenAI)
0.481
0.241
0.300
0.540
Both fine-tuned models improved on all four metrics, with the biggest gains in hard-negative accuracy and the ranking metrics.
For EmbeddingGemma 2, hard-negative accuracy rose from 0.272 to 0.862, nDCG@10 from 0.486 to 0.717 and MRR@10 from 0.610 to 0.816. For kanana-2-3b-embedding, hard-negative accuracy rose from 0.606 to 0.891, nDCG@10 from 0.448 to 0.730 and MRR@10 from 0.548 to 0.832.
Recall@10 also rose, from 0.907 to 0.960 for EmbeddingGemma 2 and from 0.820 to 0.980 for kanana-2-3b-embedding.
Fine-tuning did more than surface the right chunk. It pushed the right chunk above look-alike chunks, with the clearest gains where the model had to tell reporting bases apart.
Narrative questions
Narrative questions separated the models less. On questions about the business, strategy and risk, every model except text-embedding-3-small reached Recall@10 of 0.980 or higher, even before fine-tuning.
Hard-negative accuracy
Narrative questions, 831 triplets
nDCG@10
Narrative questions, 150 questions
Model
Hard-negative accuracy
nDCG@10
MRR@10
Recall@10
EmbeddingGemma 2 base
0.763
0.759
0.760
0.980
EmbeddingGemma 2 fine-tuned
0.910
0.855
0.853
0.980
kanana-2-3b-embedding base
0.905
0.844
0.854
0.987
kanana-2-3b-embedding LoRA
0.933
0.874
0.874
0.993
Solar Embedding 2 (Upstage)
0.912
0.842
0.836
0.993
text-embedding-3-small (OpenAI)
0.811
0.613
0.613
0.860
Fine-tuning still lifted hard-negative accuracy on narrative questions, but by less. EmbeddingGemma 2 rose from 0.763 to 0.910 (+0.147). kanana-2-3b-embedding rose from 0.905 to 0.933 (+0.028), a smaller gain from an already high baseline.
Ten examples, before and after fine-tuning
To see what the metrics mean for a single query, we scored 10 numerical questions from Hyundai Motor’s annual report before and after fine-tuning. Each question has one positive and three hard negatives, and both runs use the same questions and chunks.
Key results (before → after)
Questions where a hard negative outscored the positive: 2 → 0 (of 10)
Average gap between the positive and the closest hard negative: 0.018 → 0.238
Average similarity of positives: 0.781 → 0.714
Average similarity of hard negatives: 0.734 → 0.416 (30 chunks)
Similarity by question
Positive Hard negative Gap to the closest hard negative Similarity 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Gap
metric_00 현대자동차의 2025년 연결 자본변동표에서 자기주식 처분에 따른 자본 합계는 얼마인가요?
Before +0.006 After Improved +0.229
metric_01 현대자동차 차량 부문 한국 법인 HMC의 2025년 생산능력은 몇 대인가요?
Before +0.013 After Improved +0.144
metric_02 현대자동차의 제58기 별도재무제표 기준 매출액은 얼마인가요?
Before +0.027 After Improved +0.393
metric_03 현대자동차의 연결 기준 제58기 자산총계는 얼마인가요?
Before −0.023 After Improved +0.232
metric_04 현대자동차의 2025년 연결 당기순이익은 얼마인가요?
Before +0.046 After Improved +0.409
metric_05 2025년 현대자동차 차량부문 부품 매입액 합계는 얼마인가요?
Before +0.005 After Improved +0.194
metric_06 현대자동차의 제58기 연결 기준 매출액은 얼마인가요?
Before −0.013 After Improved +0.173
metric_07 현대자동차의 2025년 말 종속법인 등에 대한 보증 잔액 합계는 얼마인가요?
Before +0.002 After Improved +0.055
metric_08 현대자동차의 2025년 별도 기준 당기순이익은 얼마인가요?
Before +0.054 After Improved +0.377
metric_09 2025년 12월 31일 기준 현대자동차 및 연결대상 회사의 생산설비 기말 계는 얼마인가요?
Before +0.062 After Improved +0.180 Show chunk-level scores
Chunk
ID
Before
After
Change
005380_metric_00: 현대자동차의 2025년 연결 자본변동표에서 자기주식 처분에 따른 자본 합계는 얼마인가요?
Positive
3::012
0.7544
0.7802
+0.026
Hard negative 1
3::005
0.7482
0.4167
−0.331
Hard negative 2
3::009
0.7077
0.5516
−0.156
Hard negative 3
3::010
0.7288
0.5243
−0.205
005380_metric_01: 현대자동차 차량 부문 한국 법인 HMC의 2025년 생산능력은 몇 대인가요?
Positive
2::007
0.8014
0.8080
+0.007
Hard negative 1
2::008
0.7887
0.6644
−0.124
Hard negative 2
2::081
0.7737
0.5383
−0.235
Hard negative 3
2::004
0.6928
0.4735
−0.219
005380_metric_02: 현대자동차의 제58기 별도재무제표 기준 매출액은 얼마인가요?
Positive
3::001
0.7803
0.7224
−0.058
Hard negative 1
3::000
0.7537
0.3293
−0.424
Hard negative 2
4::011
0.7015
0.2089
−0.493
Hard negative 3
3::875
0.7000
0.2939
−0.406
005380_metric_03: 현대자동차의 연결 기준 제58기 자산총계는 얼마인가요?
Positive
4::011
0.7982
0.6731
−0.125
Hard negative 1
3::001
0.8209
0.3053
−0.516
Hard negative 2
1::010
0.7035
0.3118
−0.392
Hard negative 3
3::005
0.7178
0.4415
−0.276
005380_metric_04: 현대자동차의 2025년 연결 당기순이익은 얼마인가요?
Positive
3::004
0.7530
0.7271
−0.026
Hard negative 1
3::025
0.7074
0.3181
−0.389
Hard negative 2
3::026
0.6942
0.3075
−0.387
Hard negative 3
2::014
0.6937
0.2649
−0.429
005380_metric_05: 2025년 현대자동차 차량부문 부품 매입액 합계는 얼마인가요?
Positive
2::005
0.7574
0.6640
−0.093
Hard negative 1
2::014
0.7169
0.2821
−0.435
Hard negative 2
2::011
0.7521
0.4705
−0.282
Hard negative 3
2::013
0.7054
0.3530
−0.352
005380_metric_06: 현대자동차의 제58기 연결 기준 매출액은 얼마인가요?
Positive
3::000
0.7634
0.6526
−0.111
Hard negative 1
3::001
0.7766
0.3369
−0.440
Hard negative 2
2::013
0.7326
0.4799
−0.253
Hard negative 3
4::011
0.7163
0.4629
−0.253
005380_metric_07: 현대자동차의 2025년 말 종속법인 등에 대한 보증 잔액 합계는 얼마인가요?
Positive
4::022
0.8241
0.6978
−0.126
Hard negative 1
11::000
0.7987
0.6284
−0.170
Hard negative 2
11::003
0.8223
0.6428
−0.179
Hard negative 3
11::002
0.7856
0.5919
−0.194
005380_metric_08: 현대자동차의 2025년 별도 기준 당기순이익은 얼마인가요?
Positive
3::422
0.7878
0.7584
−0.029
Hard negative 1
2::099
0.7334
0.3665
−0.367
Hard negative 2
3::709
0.7275
0.3812
−0.346
Hard negative 3
3::702
0.6740
0.2786
−0.395
005380_metric_09: 2025년 12월 31일 기준 현대자동차 및 연결대상 회사의 생산설비 기말 계는 얼마인가요?
Positive
2::010
0.7906
0.6599
−0.131
Hard negative 1
2::008
0.7284
0.4795
−0.249
Hard negative 2
2::014
0.7100
0.2983
−0.412
Hard negative 3
2::007
0.7151
0.4666
−0.248
After fine-tuning, positives scored slightly lower on average, but hard negatives fell much further, so the gap widened. Before fine-tuning, a hard negative outscored the positive on 2 of the 10 questions. Afterward, the positive scored highest on all 10.
Fine-tuning did not simply raise scores for the right chunks. It taught the model to push away look-alike chunks that do not meet the question’s conditions.
Key findings and implications
Domain-specific hard negatives improved the models’ ability to separate look-alike financial data.
Our hard negatives included chunks that read almost the same but differ in reporting basis, such as consolidated versus separate or current versus prior period.
After fine-tuning, hard-negative accuracy on numerical questions rose from 0.272 to 0.862 for EmbeddingGemma 2 and from 0.606 to 0.891 for kanana-2-3b-embedding.
A wider similarity gap meant better ranking.
Across 10 Hyundai Motor examples, the average gap between the positive and the closest hard negative widened from 0.018 to 0.238.
On numerical questions, nDCG@10 rose from 0.486 to 0.717 for EmbeddingGemma 2 and from 0.448 to 0.730 for kanana-2-3b-embedding, so positives ranked higher.
The effect was larger for numerical questions than for narrative ones.
Both models improved on narrative questions, but their hard-negative accuracy gains were larger on numerical questions.
For financial retrieval, training data should capture differences in reporting basis, not just general semantic similarity.
Overall, hard negatives built around financial-domain distinctions, combined with fine-tuning, improved retrieval quality. The results make a case for building domain-specific training data for financial retrieval.
Building embedding training data from DART annual reports
We built 7,515 training triplets from 30 annual reports. An LLM wrote the questions, and Jev scored every chunk in each report to decide which ones contain the answer.
Training example structure
Each example is a triplet: one question and two chunks. Training pulls the question toward the positive chunk and pushes it away from the negative one.
Anchor Question
Written by an LLM from a chunk of an annual report
Positive Answer chunk
Contains the answer to the question
Hard negative Look-alike wrong chunk
Does not contain the answer, but the base model scores it about as close to the question as the positive
A specific figure, with the basis (consolidated or separate) and the period stated
448
3.9
5,283
Narrative
An explanation of the business, strategy or risks
449
1.7
2,232
We split the data by company: 15 training companies (3,687 triplets), 5 validation companies (1,182) and 10 test companies (2,646). Reports from test companies never appear in training.
How we built it
Chunk preparation
1,106 chunks per report on average
We reused the existing parsed chunks as is. Tables, XBRL financial statements and narrative paragraphs each form their own chunks.
Question generation
LLM 30 per report
The LLM sees one chunk and writes a question with a reference answer. Numerical questions must state the basis (consolidated or separate) and the period, and answers keep the units used in the table.
Exhaustive labeling
Jev
Each question is checked against every chunk in its report. With no first-stage retrieval to shortlist candidates, no positive can slip through.
Label
What Jev is asked
Used to
answer
Does this chunk contain the full reference answer?
Select positives
relevant
Does this chunk provide information that helps answer the question?
Filter out ambiguous chunks
Jev returns a probability between 0 and 1 for each label.
Labeling rules
Rules
We bucket chunks by answer score. If the chunk a question was generated from is not labeled positive, the question is dropped.
Negative Discard Positive 0 0.2 0.7 1
Hard-negative mining
kanana-2-3b 3 per positive
Training examples
Below are examples of the three hard negatives that the base kanana-2-3b model scored as most similar to each question. Questions and filing excerpts appear in the original Korean.
5. 부외거래
당기말 현재 종속법인 등에 대한 보증내역은 아래와 같습니다.
⋮
(단위 : 백만원)
⋮
### 현대로템
* 관계: 국내 종속법인
* 목적: 계약이행보증
⋮
### 합 계
* 거래내역:
* 기초: 3,148,523
* 증가: -
* 감소: -173,930
* 기말: 2,974,593
Hard negative 1 2. 우발부채 등에 관한 사항 › 가. 당기말 현재 연결실체가 종속기업을 제외한 특수관계자 등의 채무 등에 대해 제공한 11::000 Similarity 0.7987
2. 우발부채 등에 관한 사항
가. 당기말 현재 연결실체가 종속기업을 제외한 특수관계자 등의 채무 등에 대해 제공한
지급보증 내역은 다음과 같습니다.
⋮
(단위 : 백만원)
⋮
## 상대방
⋮
### 관계기업
⋮
Hard negative 2 2. 우발부채 등에 관한 사항 › 가. 당기말 현재 연결실체가 종속기업을 제외한 특수관계자 등의 채무 등에 대해 제공한 11::003 Similarity 0.8223
전기말 종속법인 등에 대한 보증내역은 아래와 같습니다.
⋮
(단위 : 백만원)
⋮
### 현대로템
* 관계: 국내 종속법인
* 목적: 계약이행보증
⋮
### 합 계
* 거래내역:
* 기초: 1,040,245
* 증가: 2,109,348
* 감소: -1,070
* 기말: 3,148,523
Hard negative 3 2. 우발부채 등에 관한 사항 › 가. 당기말 현재 연결실체가 종속기업을 제외한 특수관계자 등의 채무 등에 대해 제공한 11::002 Similarity 0.7856
(*) 2025년 2월 중 Hyundai Assan Otomotiv Sanayi Ve Ticaret A.S. (HAOSVT)의 사명을
Hyundai Motor Turkiye Otomotiv A.S, (HMTR) 으로 변경하였음.
※ 외화 지급보증금액은 보고기간종료일 현재의 서울외국환중개주식회사가 고시하는 기준환율로
환산한 금액임.
⋮
전기말 연결실체가 제공한 지급보증의 내역은 다음과 같습니다.
Triplet assembly
We pair each positive with its three hard negatives, then split the triplets into training, validation and test sets by company.