AAMTジャーナル「機械翻訳」No. 83 Quality Estimation Reranking for Document-Level Translation

Abstract

Quality estimation (QE) reranking is a form of quality-aware decoding which aims to improve machine translation (MT) by scoring and selecting the best candidate from a pool of generated translations. While known to be effective at the sentence level, its application to the increasingly prominent domain of document-level translation remains underexplored. In this work, we evaluate QE reranking performance on document-level (rather than the typical sentence-level) translation, using various learned and large language model (LLM)-based QE metrics. We find that with our best learned metric, SLIDE, BLEURT-20 scores improve by +2.00 with only two candidates, and by +5.09 with 32, across both decoder-only LLM models and encoder-decoder neural machine translation (NMT) models. Using the best LLM-based metric, GEMBA-DA, gains of +1.63 and +4.30 are achieved under the same conditions. Although gains shrink with longer inputs, reranking with 32 candidates yields improvements of +2.34 (SLIDE) and +1.40 (GEMBA-DA) on our longest documents (512-1024 source tokens). These findings demonstrate the practical value of document-level QE, with minimal runtime overhead given suitable translation models and hardware.

1. Introduction

Machine Translation (MT) evaluation metrics are widely used to assess system performance, having been shown to align strongly with human evaluation [1]. In contrast, the standard decoding strategy of maximising model likelihood (MAP) has been shown to diverge from human evaluation [2] [3]. This motivates quality-aware decoding [4], where MT evaluation metrics are directly integrated into the translation process.

Quality-aware decoding uses an MT evaluation metric to select an optimal translation from a candidate pool. Minimum Bayes-Risk (MBR) decoding [5] uses reference-based metrics such as BLEU [6] or COMET [7], comparing translation candidates against each other to select the highest utility candidate. Conversely, reference-free metrics, or Quality Estimation (QE) metrics, such as Comet-Kiwi [8], may be used for QE reranking, where the highest-scoring candidate is chosen as the final output. Although MBR mitigates weaknesses of MAP decoding [9], it requires 𝑂(𝑁2) pairwise comparisons compared to the linear 𝑂(𝑁) complexity of QE reranking, and translation quality gains over QE reranking are not definitive [4] [10]. Therefore, we focus our work on QE reranking.

While document-level MT is becoming increasingly prominent [11], to date, document-level QE reranking has received little attention, despite one implementation showing promising results [12]. The leading QE metrics such as Comet-Kiwi being sentence-level raises uncertainty about their suitability for document-level evaluation.

In this work, we investigate the applicability of QE metrics to document-level QE reranking. We evaluate translation quality improvements compared to standard decoding, examining differences across QE metrics, translation models, candidate pool sizes, and document lengths, as well as the associated computational trade-offs.

Our contributions are as follows:

• We demonstrate that QE reranking improves document-level MT quality across multiple QE metrics and translation models.
• We analyse how reranking effectiveness varies with candidate pool size and document length.
• We quantify the computational trade-offs of document-level QE reranking.

2. Method

2.1. Translators

For candidate generation, we evaluate both decoder-only large language models (LLMs) and encoder-decoder neural machine translation (NMT) models. While the use of NMT models is the traditional approach, the broad pretraining of LLMs typically enables them to generate more diverse outputs [10], making them well-suited for QE reranking.

We experiment with ALMA-7B [13], a LLaMA2-7B LLM finetuned for translation, and NLLB-200-3.3B [14], a widely used multilingual NMT model. Although both were trained on sentence-level data, we found them sufficient for document-level translation, given the scarcity of publicly available document-trained models. For ALMA, the LLaMA pretraining and additional monolingual fine-tuning stage may further preserve document-level capabilities. We also evaluate Yaraku Translate, our proprietary NMT model trained directly for English-Japanese document translation.

For decoding, we adopt nucleus sampling (𝑝=0.9) for ALMA [15]. For NLLB, while beam search is the typical choice for NMT models, we opt for epsilon sampling (𝜀=0.02), shown to excel for MBR decoding [16]. Temperatures of 0.6 (ALMA) and 0.5 (NLLB) balance candidate pool diversity and quality, and have been effective for QE reranking [10]. For Yaraku Translate, we opt for diverse beam search [17] as an alternative to sampling, to address difficulties in achieving diverse yet high-quality outputs with sampling. We set 𝐺=16 groups and 𝜆=0.5 diversity strength.

2.2. QE Metrics

2.2.1. Learned QE Metrics

We adopt the COMET model family as baseline QE metrics, namely COMET-QE [18] and Comet-Kiwi [9]. Although trained for sentence-level evaluation, we test two strategies for adapting them to the document-level.

The first averages sentence-level predictions across a document. Documents are segmented into sentences using Punkt [19] for English and a simple regular expression for Japanese, then aligned by order. When source and target sentence counts differ, the shorter text is padded by duplicating its final sentence, ensuring equal segment counts so that all sentences are scored. While this approach closely aligns with the intended use case, it is flawed in practice since document-level translators rarely preserve one-to-one sentence alignment, which compounds with longer documents. We refer to these metrics as COMET-QE (sentence) and Comet-Kiwi (sentence).

The second strategy passes full documents as single segments. Although not the intended use-case, prior work shows this may perform comparably to metrics directly trained for longer-context evaluation [20], likely due to the long-context pretraining of the underlying InfoXLM encoders [21]. This method is constrained by sequence length limits (512 tokens for source + target for Comet-Kiwi, 512 per text for COMET-QE). We refer to these metrics simply as COMET-QE and Comet-Kiwi.

Doc-COMET-QE [22] extends COMET-QE by concatenating two preceding source and target sentences (where available) to provide additional document-level context. Sentence-level score is calculated only for the current sentence via masking, and document-level score as the average of all sentence-level scores. This is compatible with COMET-QE, which pools token representations and allows for selective masking, but not with Comet-Kiwi, whose representation collapses into a single [CLS] token. Although shown to improve accuracy over COMET-QE in isolated evaluations, Doc-COMET-QE inherits the same alignment problems as COMET-QE (sentence), limiting its utility for QE reranking. We use the same sentence alignment and padding approaches as outlined in COMET-QE (sentence).

SLIDE [23] is a document-level QE approach requiring no architectural changes, so we implement it on top of Comet-Kiwi. It segments documents into fixed-sentence-width, strided windows, scoring each window independently and averaging to obtain a document-level score. SLIDE is identical to Comet-Kiwi for any documents shorter than the window length but mitigates sequence length limitations for longer documents. The original work reported optimal performance with 𝑤=6, 𝑠=6 (window size, stride) in idealised conditions where documents segmented evenly. We instead adopt their proposed weighted partial window approach, which accommodates arbitrary document lengths. Since the best configuration is unclear, we experiment with both 𝑤=7, 𝑠=7 and 𝑤=7, 𝑠=1 as they both show good performance while representing two extremes of the method. We use the same padding approach as outlined in COMET-QE (sentence).

2.2.2. LLM-based QE Metrics

Given their strong performance in translation tasks, LLMs are a natural choice for document-level QE. We evaluate two prompting-based methods using Gemma 3 27B [24] as the backbone. Although originally developed for sentence-level QE, we expect them to transfer effectively to documents due to the long-context capabilities of LLMs in MT [25].

GEMBA-DA [26] tasks the LLM with Direct Assessment, assigning a score from 0 to 100 in a zero-shot manner. This method is efficient, requiring minimal token generation. We make minimal modifications to the original prompt to account for Gemma being instruction tuned. To mitigate the unpredictable nature of LLM output, we adopt the failure-recovery strategy from the original work, retrying with gradually higher temperature for up to five attempts, after which the candidate is discarded. This introduces a small chance of no valid candidates kept, so a fallback QE metric may be important.

EAPrompt [27] emulates the MQM human evaluation framework [28]. To compute the score, the LLM identifies major and minor errors, from which a weighted sum is computed via a regular expression. We weight major errors eight times higher than minor errors, shown to be effective for segment-level evaluation. We adopt one-shot prompting with language-pair-specific in-context examples, with minor prompt adjustments to accommodate the output style of Gemma. Both EAPrompt and GEMBA-DA frequently produce tied scores, which we resolve via random selection.

Although EAPrompt has shown state-of-the-art (SOTA) performance, our experiments revealed some limitations. First, unlike GEMBA-DA, it lacks a failure-recovery mechanism: erroneous outputs with no listed errors are indistinguishable from valid assessments of perfect translations. Second, the scoring scheme is overly lenient on critical translation errors—e.g., a nonsensical translation may only receive one major error, while a flawed yet comprehensible translation has more scope for identifying several errors. This issue is likely amplified at the document level, where long contexts increase the risk of critical errors. To address this, we introduce EAPrompt-Critical, which adds a critical error category weighted at 100.

3. Experiments

For our main experiment, we generate a large pool of candidate translations for each source text in our dataset, score them with each QE metric, and then trim the candidate pool to various sizes. From each pool, we select the top-scoring candidate and evaluate it against the reference translation using several reference-based metrics.

3.1. Dataset

We use the WMT23 test set [29] as our source of ground-truth translations, evaluating bidirectionally between English and Japanese. As the dataset is segmented at the document, paragraph, and sentence levels, we merge segments to obtain document-level translations. We augment the data with a balanced mix of full documents and individual paragraphs to better examine the relationship between document length and performance. As the dataset was released after the COMET models we evaluate, there is no risk of overfitting, although some risk remains for Gemma. Dataset length distributions are shown in Figure 2.

3.2. Evaluation Metrics

We evaluate QE reranking performance using reference-based metrics, treating the reranked translation as the hypothesis and the dataset translation as the reference. We consider two families: neural metrics, using BLEURT-20 [30] and COMET-22 [31], the latter setting the SOTA in WMT 2022 shared task [32]; and LLM-based evaluation, using the reference-based prompting framework GEMBA- DA [26], which we had found to outperform EAPrompt. For the backbone LLM, we use GPT-4.1-mini [33] for its strong natural language understanding capabilities and to minimise the risk of overfitting with our QE metrics using Gemma. Notably, this is the only evaluation metric directly compatible with document-level translation, as both COMET and BLEURT impose a strict 512 token cap, limiting their reliability on long documents.

We acknowledge the risk of overfitting when using the same metric family for both QE and evaluation, which can lead to evaluation scores diverging from human judgement [4]. Nonetheless, we include COMET-22 as an evaluation metric to enable comparison across a broad range of evaluators and discuss the implications of overfitting in Section 4.

4. Results

4.1. Pool Size

We first examine the effect of candidate pool size on reranking performance. A pool size of one serves as the baseline, equivalent to no QE reranking. As shown in Figure 1, scores generally increase with larger pools, confirming the effectiveness of QE reranking at the document level. Gains are observed for both LLMs and NMT models, with the largest improvements in Yaraku Translate, likely reflecting its document-level training. Performance does not reach a full plateau at pool size 32, suggesting larger pools could yield further gains. While improvements are consistent across all evaluators, the leading QE metric varies, likely due to evaluator-specific biases such as overfitting (e.g., COMET-based QE metrics evaluated with COMET-22), and sequence length constraints in COMET-22 and BLEURT, which limit their ability to fully capture the advantages of LLM-based QE metrics which can handle longer sequences. Full results are given in Table 1.

Among COMET-based metrics, Comet-Kiwi consistently outperforms COMET-QE. Notably, scoring entire documents in one pass also outperforms per-sentence averaging in all settings. The gap is smallest, however, for Yaraku Translate, which enforces sentence alignment, suggesting the benefit of aligned outputs. In contrast, Doc-COMET-QE provides no benefit and ranks among the weakest QE metrics. Both configurations of SLIDE performed very similarly to each other and to standard Comet-Kiwi, only providing marginal gains in some cases. This is expected, as our SLIDE implementation is built upon Comet-Kiwi.

The LLM-based metrics show mixed results. When evaluated with BLEURT and COMET, GEMBA-DA performs well but slightly below the best-performing COMET-based metrics. This is unsurprising due to the potential overfitting risks of the COMET-based QE metrics. However, with GPT as the evaluator, GEMBA-DA achieves the best performance. Naturally, there is also an overfitting risk in this case, but the difference in backbone LLM attempts to minimise this. Surprisingly, while adding the critical category helps, EAPrompt generally showed poor performance.

4.2. Length

We expected QE performance to degrade with longer inputs: learned metrics were not trained for long sequences, and LLM attention tends to diverge over extended contexts [34]. The results shown in Figure 3 confirm this hypothesis. QE metrics perform best on short inputs, but performance remains stable up to ~256 source tokens, indicating reasonable capability in multi-sentence contexts. Beyond this point, performance rapidly declines, reflecting the 512 token limit for combined source and target texts in most QE metrics and evaluators. Nonetheless, most QE metrics continue to provide a performance gain over the baseline even at the longest tested sequence lengths.

The extent of degradation varies across metrics. SLIDE exhibits similar performance to Comet-Kiwi for short sequences but retains slightly higher performance for long sequences, thanks to the sliding window approach avoiding the token limit. Additionally, when GPT is used as an evaluator, the lead of GEMBA-DA as a QE metric becomes the biggest for long sequences, highlighting the long-context capabilities of LLMs.

4.3. Runtime

Runtime is difficult to assess in a hardware-agnostic manner, as it depends heavily on GPU memory, implementation details, batch size, and document length. In our experiments, we used a cluster of 4 NVIDIA A6000 GPUs, sharding models so that the maximum pool size could be processed in a single batch. To ensure fairness, we fixed the batch size for both translation and QE models at 32 (the largest pool size). Although many QE models could support larger batch sizes, this cap highlights the potential slowdown of metrics requiring multiple evaluations per candidate. We show the translator and QE metric runtime in Figure 4.

Among translators, Yaraku Translate shows little sensitivity to pool size, likely reflecting the efficiency of its dynamic beam search decoding. Counterintuitively, the smaller NLLB model is slower than the larger ALMA, exploding exponentially with both pool size and input length. This is primarily due to difficulty generating stop tokens; hallucinated outputs often reach the token limit, delaying the entire batch. This highlights the value of document-level translation models and effective stopping strategies. To partially mitigate this issue, we apply an adaptive maximum token limit defined as:

where 𝐿_in is the input token length, 𝛼_a=10, 𝛼_m=2 are additive and multiplicative margin factors, 𝑁_ceil=2048 is a hard ceiling, and 𝜇_tgt, 𝜇_src denote the average dataset token lengths for the current target and source languages, respectively. This caps hallucinated translations while retaining sufficient headroom for legitimate document-length variability.

For learned QE metrics, runtime grows modestly with pool size and sequence length. Methods requiring multiple evaluations per candidate, namely SLIDE, Doc-COMET-QE, and the sentence-based COMET variants, exhibit steeper growth on longer documents. Between SLIDE configurations, while 𝑠=1 and 𝑠=7exhibit similar performance, 𝑠=1 incurs significantly higher runtime, making 𝑠=7the more practical choice. Comparison between learned QE metrics and LLM-based methods is complicated by experiment setup limitations (Gemma was hosted on a GH200 NVIDIA GPU), yet GEMBA-DA runs substantially faster than both EAPrompt variants, as it requires minimal token generation. GEMBA-DA, however, exhibits higher runtime growth with larger pool sizes, likely reflecting the higher probability of triggering its failure-prevention strategy.

5. Conclusion

We investigated the applicability of QE reranking to the document translation domain and found consistent translation quality gains over standard decoding across various QE metrics and translation models. Gains increased with translation candidate pool size and were not saturated at 32, indicating further potential improvement. Methods that score full documents in one pass consistently outperform sentence-level averaging, even with QE metrics designed for sentence-level scoring. SLIDE was found to be the best performing QE metric, matching Comet-Kiwi on short inputs while being more performant on long documents. Among LLM-based methods, GEMBA-DA was found to be competitive when evaluated with COMET-22 and BLEURT, and leads under GPT evaluation. Although performance gains decline for long documents beyond 256 source tokens, QE reranking improves translation quality even for the longest documents in our dataset. Runtime analysis showed that all QE metrics represent a fraction of total translation runtime cost, allowing for near cost-free performance gains under certain conditions.

Limitations

This study has several limitations. First, although we tested multiple pool sizes, performance did not reach a clear saturation point (i.e., a peak followed by stagnation), which would have provided stronger evidence of the limits of QE reranking. Second, resource constraints prevented us from exploring more diverse LLM prompting methods for evaluation, limiting our ability to fully exploit the potential of LLMs. Third, most QE models remain constrained by a 512-token limit, restricting their applicability for longer documents. Lastly, this study lacks human evaluation, which reduces the reliability of our tested evaluation metrics, and would have allowed us to better explore the extent of overfitting.

References

[1] Q. Ma, J. Wei, O. Bojar, and Y. Graham, ‘Results of the WMT19 Metrics Shared Task: Segment-Level and Strong MT Systems Pose Big Challenges’, in Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), 2019, pp. 62–90.

[2] B. Eikema and W. Aziz, ‘Is MAP Decoding All You Need? The Inadequacy of the Mode in Neural Machine Translation’, in Proceedings of the 28th International Conference on Computational Linguistics, 2020, pp. 4506–4520.

[3] M. Freitag, D. Grangier, Q. Tan, and B. Liang, ‘High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metrics’, Transactions of the Association for Computational Linguistics, vol. 10, pp. 811–825, 2022.

[4] P. Fernandes et al., ‘Quality-Aware Decoding for Neural Machine Translation’, in Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2022, pp. 1396–1412.

[5] S. Kumar and W. Byrne, ‘Minimum Bayes-Risk Decoding for Statistical Machine Translation’, in Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004, pp. 169–176.

[6] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, ‘Bleu: a Method for Automatic Evaluation of Machine Translation’, in Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, 2002, pp. 311–318.

[7] R. Rei, C. Stewart, A. C. Farinha, and A. Lavie, ‘COMET: A Neural Framework for MT Evaluation’, in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020, pp. 2685–2702.

[8] R. Rei et al., ‘CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Task’, in Proceedings of the Seventh Conference on Machine Translation (WMT), 2022, pp. 634–645.

[9] M. Müller and R. Sennrich, ‘Understanding the Properties of Minimum Bayes Risk Decoding in Neural Machine Translation’, in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 2021, pp. 259–272.

[10] G. Vernikos and A. Popescu-Belis, ‘Don’t Rank, Combine! Combining Machine Translation Hypotheses Using Quality Estimation’, in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024, pp. 12087–12105.

[11] L. Wang et al., ‘Benchmarking and Improving Long-Text Translation with Large Language Models’, in Findings of the Association for Computational Linguistics: ACL 2024, 2024, pp. 7175–7187.

[12] K. Kudo et al., ‘Document-level Translation with LLM Reranking: Team-J at WMT 2024 General Translation Task’, in Proceedings of the Ninth Conference on Machine Translation, 2024, pp. 210–226.

[13] H. Xu, Y. J. Kim, A. Sharaf, and H. H. Awadalla, ‘A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Models’, in The Twelfth International Conference on Learning Representations, 2024.

[14] N. Team et al., ‘No Language Left Behind: Scaling Human-Centered Machine Translation’, arXiv [cs.CL]. 2022.

[15] H. Touvron et al., ‘Llama 2: Open Foundation and Fine-Tuned Chat Models’, arXiv [cs.CL]. 2023.

[16] M. Freitag, B. Ghorbani, and P. Fernandes, ‘Epsilon Sampling Rocks: Investigating Sampling Strategies for Minimum Bayes Risk Decoding for Machine Translation’, in Findings of the Association for Computational Linguistics: EMNLP 2023, 2023, pp. 9198–9209.

[17] A. K. Vijayakumar et al., ‘Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Models’. 2017.

[18] R. Rei, C. Stewart, A. C. Farinha, and A. Lavie, ‘Unbabel’s Participation in the WMT20 Metrics Shared Task’, in Proceedings of the Fifth Conference on Machine Translation, 2020, pp. 911–920.

[19] T. Kiss and J. Strunk, ‘Unsupervised Multilingual Sentence Boundary Detection’, Computational Linguistics, vol. 32, no. 4, pp. 485–525, 2006.

[20] D. Deutsch, J. Juraska, M. Finkelstein, and M. Freitag, ‘Training and Meta-Evaluating Machine Translation Evaluation Metrics at the Paragraph Level’, in Proceedings of the Eighth Conference on Machine Translation, 2023, pp. 996–1013.

[21] Z. Chi et al., ‘InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training’, in Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2021, pp. 3576–3588.

[22] G. Vernikos, B. Thompson, P. Mathur, and M. Federico, ‘Embarrassingly Easy Document-Level MT Metrics: How to Convert Any Pretrained Metric into a Document-Level Metric’, in Proceedings of the Seventh Conference on Machine Translation (WMT), 2022, pp. 118–128.

[23] V. Raunak, T. Kocmi, and M. Post, ‘SLIDE: Reference-free Evaluation for Machine Translation using a Sliding Document Window’, in Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), 2024, pp. 205–211.

[24] G. Team et al., ‘Gemma 3 Technical Report’, arXiv [cs.CL]. 2025.

[25] M. Karpinska and M. Iyyer, ‘Large Language Models Effectively Leverage Document-level Context for Literary Translation, but Critical Errors Persist’, in Proceedings of the Eighth Conference on Machine Translation, 2023, pp. 419–451.

[26] T. Kocmi and C. Federmann, ‘Large Language Models Are State-of-the-Art Evaluators of Translation Quality’, in Proceedings of the 24th Annual Conference of the European Association for Machine Translation, 2023, pp. 193–203.

[27] Q. Lu, B. Qiu, L. Ding, K. Zhang, T. Kocmi, and D. Tao, ‘Error Analysis Prompting Enables Human-Like Translation Evaluation in Large Language Models’, in Findings of the Association for Computational Linguistics: ACL 2024, 2024, pp. 8801–8816.

[28] M. Freitag, G. Foster, D. Grangier, V. Ratnakar, Q. Tan, and W. Macherey, ‘Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation’, Transactions of the Association for Computational Linguistics, vol. 9, pp. 1460–1474, 2021.

[29] M. Freitag et al., ‘Results of WMT23 Metrics Shared Task: Metrics Might Be Guilty but References Are Not Innocent’, in Proceedings of the Eighth Conference on Machine Translation, 2023, pp. 578–628.

[30] T. Sellam, D. Das, and A. Parikh, ‘BLEURT: Learning Robust Metrics for Text Generation’, in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020, pp. 7881–7892.

[31] R. Rei et al., ‘COMET-22: Unbabel-IST 2022 Submission for the Metrics Shared Task’, in Proceedings of the Seventh Conference on Machine Translation (WMT), 2022, pp. 578–585.

[32] M. Freitag et al., ‘Results of WMT22 Metrics Shared Task: Stop Using BLEU — Neural Metrics Are Better and More Robust’, in Proceedings of the Seventh Conference on Machine Translation (WMT), 2022, pp. 46–68.

[33] OpenAI et al., ‘GPT-4 Technical Report’, arXiv [cs.CL]. 2024.

[34] F. Barbero et al., ‘Why do LLMs attend to the first token?’, in Second Conference on Language Modeling, 2025.