Preferred Networks

PLaMo 3 Translate: World-Class Translation with Open Weights—Here’s How We Built It

Kaito Suzuki

Kaito Suzuki

日本語版はこちら。

Overview

  • We have released the weights for PLaMo 3 Translate, our translation-specialized LLM, on Hugging Face. The models come in four sizes—0.8B, 2B, 8B, and 31B. Even the smallest 0.8B model outperforms existing translation-specialized LLMs, including models with more than 30 times as many parameters, in Japanese–English translation and translation between Japanese and other languages.
  • The key was meticulous data refinement through techniques such as tournament-based data synthesis, back-translation, and iterative self-distillation, combined with distillation into smaller models. In this post, we describe how we developed these models and how they perform. For comparisons with frontier LLM APIs and translation APIs, see our earlier release post (ja) [1].

Released Model Weights

PLaMo 3 Translate comes in four model sizes: 0.8B, 2B, 8B, and 31B. All four are available on Hugging Face. For each size, we provide BF16 weights, several quantized variants, and models in GGUF and MLX formats. Choose the version that best suits your environment.

We quantized the models using tools such as AutoRound [2]. In our benchmarks, all of the quantization formats retained nearly the same accuracy as the BF16 versions. See the model cards for details.

PLaMo 3 Translate supports translation across 53 languages. It accepts surrounding context and glossaries, and lets users choose between plain and polite Japanese. For detailed usage instructions, see the individual model cards on Hugging Face.

PLaMo 3 Translate is released under the PLaMo Community License. For private purposes such as personal study and hobbies, you can download the models free of charge and use them freely. Noncommercial use, including academic use, has no income or revenue limit and requires no prior registration with PFN.

Commercial use is also available free of charge if the user or its affiliates had income or revenue of JPY 1 billion or less in the most recent fiscal year, provided that the user registers in advance through the dedicated form. Commercial use above this threshold requires a separate license agreement; please contact PFN.

Table 1. Links to model weights

The following table can help you choose a distribution format.

FormatTarget HardwareInference EngineTypical Use
BF16BF16-compatible GPUTransformers / vLLM [3]Prioritize unquantized accuracy when memory is sufficient
FP8NVIDIA Ada Lovelace/Hopper/BlackwellvLLM [3]Balance accuracy, memory use, and speed
W4A16NVIDIA Ampere or latervLLM [3]Run with reduced GPU memory using 4-bit weights
W4AFP8NVIDIA HoppervLLM [3]Combine 4-bit weights and FP8 computation to balance memory use and throughput
NVFP4NVIDIA BlackwellvLLM [3]Use FP4 computation to reduce memory use and improve speed
GGUFCPUs / GPUsllama.cpp [4]Run locally, optionally using both CPU and GPU
MLX-4bitApple siliconmlx-lm [5]Run on a Mac with reduced memory use

Example Use Cases

Even the 0.8B PLaMo 3 Translate model delivers strong translation performance. We believe this opens up new use cases that were previously out of reach for existing LLMs.

One example is on-device inference. The demo below shows PLaMo 3 Translate 0.8B translating directly on a smartphone. This is an internal demo; PFN has not released an app like this at this time.

As the video below shows, the model delivers fast, natural translations even in airplane mode, with no internet connection. By offering a range of model sizes, including small LLMs suited to different hardware, we hope to make new use cases like this possible.

Training Overview

The overall training process for PLaMo 3 Translate is shown below.

Figure 1. Overview of the PLaMo 3 Translate training process

We start with a third-generation PLaMo base model (*1)—a model that has completed pretraining but has not yet been fine-tuned—and apply two stages of supervised fine-tuning (SFT), followed by direct preference optimization (DPO) [6]. The stages are as follows:

Phase 1 SFT

  • We train on (source text, translation) pairs so that the model learns to produce a translation given a source text.
  • We use a large volume of data to adapt the model to the translation task and establish its basic translation capabilities.

Phase 2 SFT

  • As in Phase 1, we train the model on (source text, translation) pairs.
  • Compared with Phase 1, we collect more diverse source texts that better reflect real-world use cases and refine their synthetic translations even more thoroughly. Together, the two phases move from large-scale training to training on a smaller, higher-quality dataset, with the aim of further improving translation performance.

DPO

  • For each source text, we select a preferred translation (chosen) and a comparison translation (rejected) from multiple candidates. We train the model to assign higher probability to the chosen translation than to the rejected one, as a final refinement of translation quality.

For models of 8B and smaller, we also perform distillation from the 31B model between SFT and DPO. The aim is to further improve the smaller models by making full use of the completed larger model's capabilities.

Details for each stage are explained below.

Evaluation

Before discussing the training process in detail, we describe how we evaluated translation performance during development.

We used the following four types of benchmarks:

  • PFTB Benchmark
  • SaaS Benchmark
  • WMT24++ [7]
  • FLORES+ [8]

Details are provided below.

PFTB Benchmark

This is an internally developed benchmark designed to measure Japanese–English and English–Japanese translation performance across a wide range of domains. We collected texts ranging from academic papers and product reviews to poetry, edited them as needed, and used multiple powerful LLMs to create high-quality reference translations.

Our earlier study, "Investigation of Automatic Evaluation Methods for High-Precision Translation Models" [9], showed that n-gram-based metrics can provide effective evaluations when sufficiently high-quality reference translations are available. We therefore score translations by their n-gram similarity to the references, using charsim [10].

The examples below illustrate the benchmark. Each passage is reasonably long and poses a moderately complex translation challenge.

Source textReference translation
English → Japanese example
Tokugawa Ieyasu (born Matsudaira Takechiyo; January 31, 1543 – June 1, 1616) was the founder and first shōgun of the Tokugawa shogunate of Japan, which ruled from 1603 until the Meiji Restoration in 1868. He was the third of the three "Great Unifiers" of Japan, along with his former lord Oda Nobunaga and fellow Oda subordinate Toyotomi Hideyoshi. The son of a minor daimyo, Ieyasu once lived as a hostage under daimyo Imagawa Yoshimoto on behalf of his father. He later succeeded as daimyo after his father's death, serving as ally, vassal, and general of the Oda clan, and building up his strength under Oda Nobunaga.徳川家康(幼名:松平竹千代、1543年1月31日 – 1616年6月1日)は、1603年から1868年の明治維新まで日本を統治した徳川幕府の創設者であり、初代将軍である。日本の「三英傑」の三人目であり、かつての主君である織田信長、同じく織田氏の家臣であった豊臣秀吉と並ぶ。小大名の子であった家康は、かつて父の名代として大名・今川義元のもとで人質として暮らした。父の死後、家督を継いで大名となり、織田氏の同盟者、家臣、武将として仕え、織田信長のもとで勢力を築いた。
Japanese → English example
こんにちは、オンライン英会話スクールのプリファードです! この度、オンライン英会話スクールでは、初月無料キャンペーンを実施いたします! この機会に、自宅で気軽に英会話を始めてみませんか? キャンペーン期間 2029年11月1日 ~ 2029年11月30日 ★キャンペーンの魅力 ① 初月無料! 期間中に登録すると初月のレッスン料金が無料! ② 人気講師のレッスンが受け放題! 豊富な講師陣の中から、自分に合った講師を選べます。 ③ 無料教材も充実! 初心者でも安心して学べる教材をご用意しています。 ★おすすめレッスンピックアップ おすすめ1: 初心者向け英会話 おすすめ2: ビジネス英会話 その他、様々なニーズに合わせたレッスンをご用意! 【キャンペーンを賢く利用するコツ】 ・ 今すぐ登録して、初月無料の特典をゲット! ・ レッスンスケジュールを確認して、計画的にレッスンを受けよう! ▼今すぐチェック! https://www.preferred.jp/ このチャンスを逃さず、新しい自分への一歩を踏み出してみませんか? 皆さまのご参加を心よりお待ちしております!Hello, this is Preferred, your online English conversation school! We're now running a first-month-free campaign at our online English conversation school! Why not take this opportunity to start learning English conversation from the comfort of your home? Campaign Period November 1, 2029 – November 30, 2029 ★Campaign Highlights ① First month free! Register during the campaign period and your first month's lesson fees are free! ② Unlimited lessons with popular instructors! Choose from our extensive roster of instructors to find the one that's right for you. ③ Comprehensive free learning materials! We offer materials that even beginners can study with confidence. ★Recommended Lesson Picks Recommendation 1: English Conversation for Beginners Recommendation 2: Business English Conversation Plus a variety of lessons tailored to diverse needs! 【Tips for Making the Most of This Campaign】 ・ Register now and grab your first-month-free benefit! ・ Check the lesson schedule and plan your lessons ahead! ▼Check it out now! https://www.preferred.jp/ Don't miss this chance—why not take the first step toward a new you? We sincerely look forward to your participation!

SaaS Benchmark

We developed this benchmark in-house to evaluate translation in a way that reflects the PLaMo Translation SaaS experience, using inputs formatted like real requests. It currently covers English-to-Japanese translation only. As with PFTB, we evaluate translations using charsim against reference translations.

Examples are shown below. To reflect the needs of the translation SaaS and the browser extension, the benchmark asks models to translate real web pages while preserving HTML tags and other markup. Surrounding context is also supplied as part of the translation request, although it is not shown in these examples.

Source textReference translation
For example, for <code>Date</code> instances, you can use a range of methods to get different information from a single date value, such as the <a href="0">year</a>, <a href="1">month</a>, <a href="2">day of the week</a>, etc. You can also set those values through the <code>setX</code> counterparts like <a href="3"><code>setFullYear</code></a>.例えば、<code>Date</code>のインスタンスでは、1つの日付値から<a href="0">年</a>、<a href="1">月</a>、<a href="2">曜日</a>などのさまざまな情報を取得するために、各種メソッドを使用できます。それらの値は、<a href="3"><code>setFullYear</code></a>などの<code>setX</code>系の対応メソッドを通じて設定することもできます。

WMT24++

WMT24++ extends the evaluation data used at WMT24 (the Ninth Conference on Machine Translation) to cover translation from English into 55 languages and dialects. We use the Japanese track as an English-to-Japanese benchmark and evaluate performance with COMET-22 [11].

Because the benchmark consists mainly of short sentences and we had some concerns about the quality of its reference translations, we relied primarily on PFTB and SaaS benchmark scores during development.

FLORES+

We use this benchmark to measure multilingual translation performance, evaluating the following languages (*2):

Arabic, Italian, Indonesian, English, Japanese, Dutch, Korean, Spanish, Thai, Chinese (Simplified), Chinese (Traditional), German, French, Vietnamese, Russian

Evaluating every language pair would be costly, so we evaluate translation in both directions between Japanese and the other languages, and between English and the other languages.

We use chrF++ [12] as the metric. When evaluating translation into Japanese, we explicitly ask each model to use polite Japanese so that variation between plain and polite forms does not cause large swings in scores.

Data Construction

Training a translation model requires a large number of (source text, translation) pairs. For PLaMo 3 Translate, we construct most of this data by collecting source texts from the web and other sources, then synthesizing translations with an LLM.

Source Data Collection

We collect source data using a combination of methods. One of these is mining large-scale web corpora. During mining, we pay attention to the following points:

  • Avoid including too many spam pages
  • Collect as diverse a range of texts as possible
  • Ensure that languages are not mixed
  • Favor texts originally written by native speakers of each language over texts machine-translated from other languages

As discussed below, some training outputs are derived from these collected source texts through back-translation. We therefore follow these guidelines to keep the texts on the output side of our training pairs as clean as possible.

We combine rule-based methods, clustering, and other techniques to mine the data. We also apply rules tailored to each language community. A few examples from this broader set of rules are listed below:

  • Measure the density of words common in spam pages and exclude pages with a high probability of being spam
  • Cap the amount of data taken from any single host so that we mine from as many different hosts as possible
  • Use clustering results to increase the diversity of domains among the mined texts
  • Exclude Japanese passages containing Chinese-specific characters that are not normally used in Japanese, treating them as contaminated with Chinese
  • Although Simplified and Traditional Chinese can be converted into each other by mechanically substituting characters, the resulting text can differ from text originally written in each variety. To exclude converted text as far as possible, examine errors characteristic of conversion tools and the relative frequencies of vocabulary specific to Simplified- and Traditional-Chinese-speaking communities
  • Use a dedicated word-based filter when collecting Indonesian data to exclude the closely related Malay language
  • Exclude domains believed to contain a large number of machine-translated texts

Synthesizing Translations

Once we have collected the source texts, we synthesize translations with Gemma 4 31B IT [13]. We chose this open model for its strong performance relative to its size and its multilingual capabilities.

Rather than simply instructing the LLM to translate, we combine several techniques to synthesize more refined translations.

  • Tournament-based Data Synthesis
  • Back-translation
  • Iterative Self-Distillation

We describe each approach in turn below.

Tournament-based Data Synthesis

The diagram below illustrates tournament-based data synthesis.

Figure 2. Tournament-based Data Synthesis

Gemma 4 is a powerful open model, but its translations from a single inference pass often leave room for improvement. We therefore invest more computation in repeated inference to synthesize better translations.

For PLaMo 3 Translate, we developed our own tournament-based data synthesis method. We first generate several translation candidates, then repeatedly improve them by having Gemma 4 "fuse" pairs of candidates in a tournament. For each pair, it selects a winner and incorporates any strengths of the losing translation into the winner.

In preliminary experiments, we compared several approaches, such as generating a large number of candidates and selecting the best ones. Tournament-based data synthesis produced the highest-quality translations.

Back-translation

The diagram below illustrates back-translation.

Figure 3. Back-translation

We deliberately collect source texts written by native speakers, so the source side consists of natural writing in each language. Applying tournament-based data synthesis gives us forward-direction pairs: source text → synthetic translation. These teach the model to read fluent, naturally written text.

Training only in this direction would leave the model learning exclusively from synthetic text on the output side. To avoid compromising output fluency, we also construct reverse-direction pairs: synthetic translation → source text. These teach the model to write fluent, naturally written text.

Using naturally written human text as training targets may enable a model to outperform the LLM used to synthesize its training data. However, simply swapping the source and target does not always yield a suitable training pair for the reverse direction. Languages differ in which information they express explicitly and which they leave to context. Even a natural, accurate translation may therefore not provide enough information to fully reconstruct the original source text.

We therefore provide Gemma 4 with both the source text and its synthetic translation, and have it adjust the source to make the pair suitable for reverse-direction translation. This step preserves the source’s natural wording as much as possible while correcting passages that no longer align with the synthetic translation. We use the resulting “synthetic translation → revised source text” pairs for reverse-direction training.

Back-translation itself is a long-established technique in translation model development, but advances in LLMs now make it possible to perform it accurately and at scale. The final dataset includes both forward- and reverse-direction data.

Data Improvement via Iterative Self-Distillation

Tournament-based data synthesis and back-translation allow PLaMo translation models to produce better translations than the LLM that generated their training data, even though they are trained on synthetic translations. We can then use the latest PLaMo translation model to improve the data further. We call this process iterative self-distillation.

The diagram below shows how iterative self-distillation improves the training data.

Figure 4. Iterative self-distillation

First, we use the latest PLaMo translation model, trained with SFT on the current dataset, to generate multiple candidates. We then improve these candidates through tournament-based data synthesis. In a final round, we compare the resulting translation with the one already in the dataset and replace the existing translation if the new one is judged to be better.

We repeat the cycle, running SFT on the improved data and using the resulting model to refine the data further. Because of the computational cost, we apply iterative self-distillation only to the Phase 2 SFT data.

Final Dataset

The final dataset sizes are shown below.

Phase 1 SFT DataApprox. 33.9B tokens
Phase 2 SFT DataApprox. 3.4B tokens
TotalApprox. 37.3B tokens
Table 2. Dataset sizes

If we assume about 100,000 characters per paperback, this amounts to hundreds of thousands of books.

The Phase 1 SFT dataset is relatively simple and is collected mainly by mining large-scale corpora. Even so, its source–translation pairs are already of very high quality, thanks to tournament-based data synthesis and back-translation.

The Phase 2 SFT dataset combines mined texts with more diverse data reflecting real-world use cases, including inputs tailored to SaaS request formats. In addition to tournament-based data synthesis and back-translation, we refine these translations through iterative self-distillation. Although smaller than Phase 1, Phase 2 still contains the equivalent of tens of thousands of paperback books.

The Phase 2 SFT dataset includes the following categories, among others:

Academic papersLiterature
Technical documentsFinance
PatentsMedicine
NewsEncyclopedias
General web contentIdioms
Colloquial languageTranscripts
Browser translationFile translation
Table 3. Categories in the Phase 2 SFT dataset

Alongside diversity across domains, we emphasize data tailored to real SaaS use cases, such as browser and file translation.

Phase 1 SFT is designed to adapt the model to translation. Phase 2 then aims to improve translation quality and practical usefulness with more diverse, higher-quality data.

SFT

We use this dataset to apply two stages of supervised fine-tuning (SFT) to the pretrained base model.

The following figures show model performance after SFT.

Figure 5. Improvements in Japanese–English translation performance through SFT
Figure 6. Improvements in multilingual translation performance through SFT

Figure 5 shows Japanese–English translation performance, averaged across PFTB, SaaS, and WMT24++ COMET-22. Figure 6 shows multilingual performance measured with FLORES+ chrF++: Japanese ↔ other languages on the left, and English ↔ other languages on the right. The left and right panels use different score scales, so their values should not be compared directly.


For Japanese–English translation, Phase 1 SFT substantially improves on the base model, and Phase 2 SFT provides a further gain through more carefully refined data. Even the 0.8B model outperforms the 8B PLaMo 2 Translate model.

For multilingual translation, Phase 1 SFT produces a substantial improvement over the base model, especially for the smaller models. Phase 2 SFT adds little further improvement on these benchmarks.

Overall, SFT on carefully refined data delivered dramatic improvements in translation performance. This marks a major change from PLaMo 2 Translate, where the subsequent DPO stage produced substantial gains. For PLaMo 3 Translate, thoroughly refining the data has enabled significant performance improvements through SFT alone.

We continued refining the dataset throughout development. Dataset refinement alone allowed us to keep improving model performance while leaving the training methods and hyperparameters largely unchanged.

DPO

After SFT, we apply direct preference optimization (DPO) to further improve translation quality. For DPO, we prepare (source text, chosen, rejected) triples, where the chosen translation is a good translation of the source and the rejected translation is a relatively poor one.

Given the source text, DPO trains the model to increase the likelihood of the chosen translation relative to the rejected translation. Both translations are selected from the model’s outputs at that point in training. This final refinement step improves translation quality by penalizing undesirable behavior in the model’s own outputs.

The figure below illustrates how we select chosen and rejected translations.

Figure 7. Selecting chosen and rejected translations for DPO

We first generate many candidate translations with the current model, then rank them using an LLM as a judge. We select the best candidate as chosen and a poor candidate as rejected. As in data synthesis, the judge is Gemma 4 31B IT.

We perform two rounds of iterative DPO. After the first round, we use the updated model to generate new candidates, select new chosen/rejected pairs, and train again, aiming for further improvements.

The following graph shows the performance changes resulting from DPO.

Figure 8. Changes in Japanese–English translation performance through DPO

The figure shows the average score across PFTB, SaaS, and WMT24++ COMET-22. These benchmarks show only very small gains from DPO. Improvements are apparent in the translations themselves, but are difficult to capture in the scores.

We therefore examined DPO's effects in greater detail through pairwise comparisons with an LLM as a judge.

 PFTB English → JapanesePFTB Japanese → English
SFT vs DPO Round 1
(DPO Round 1 win rate)
57%59%
DPO Round 1 vs DPO Round 2
(DPO Round 2 win rate)
53%57%
Table 4. Changes in LLM-as-a-judge win rates for the 31B model through DPO

For each sample, Gemini 3.1 Pro compared the translations twice, with their presentation order reversed. When the two judgments disagreed, each translation received half a win. A win rate of 50% indicates that the models are evenly matched. The results show a small improvement with each round of DPO.

The following Japanese-to-English examples compare the 31B model after Phase 2 SFT with the released PLaMo 3 Translate 31B model after two rounds of DPO. The excerpts come from PFTB evaluations using the same inputs and generation settings; the model outputs have not been edited.

How to read the highlights: Yellow marks the SFT wording, and green marks the corresponding wording after DPO. We highlight selected improvements discussed below, rather than every difference between the outputs.

Example 1

Source text (Japanese)31B SFT (Phase 2)plamo-3-translate-31b (DPO Round 2)

Post title

`UnicodeDecodeError: 'utf-8' codec can't decode byte` が出てCSVが読み込めません

`UnicodeDecodeError: 'utf-8' codec can't decode byte` occurs and I cannot read the CSVI'm getting `UnicodeDecodeError: 'utf-8' codec can't decode byte` and can't load the CSV file.

Closing sentence

環境はWindowsです。

My environment is Windows.I'm using Windows.

What improves: “I'm getting …” frames the error as something the writer is experiencing, which is more natural in a developer's question. “I'm using Windows” likewise replaces the literal “My environment is Windows.” The error message and technical meaning are preserved.

Example 2

In the source document, the preceding paragraph discusses aging infrastructure, decarbonization, labor shortages, and digitalization. In this context, “環境変化” refers to the broader conditions affecting the business.

Source text (Japanese)31B SFT (Phase 2)plamo-3-translate-31b (DPO Round 2)
このような環境変化を踏まえ、当社グループは、2025年度を初年度とする「中期経営計画2025–2027」を策定し、「安全・安心な社会資本の整備」「環境負荷の低減」「デジタル施工による生産性向上」を基本方針として掲げました。本計画のもと、建設業界の構造変化に的確に対応し、持続的な成長と企業価値の向上を目指しております。In light of these environmental changes, our Group has formulated the "Medium-Term Management Plan 2025–2027," with fiscal 2025 as its first year, and has established "Development of Safe and Secure Social Infrastructure," "Reduction of Environmental Impact," and "Productivity Improvement through Digital Construction" as its basic policies. Under this plan, we aim to respond accurately to structural changes in the construction industry and strive for sustainable growth and the enhancement of corporate value.In response to these changing circumstances, our Group has formulated the "Medium-Term Management Plan 2025–2027," with fiscal 2025 as its first year, and has established "development of safe and secure social infrastructure," "reduction of environmental impact," and "productivity improvement through digital construction" as its basic policies. Under this plan, we aim to respond effectively to structural changes in the construction industry and achieve sustainable growth and enhanced corporate value.

What improves: “Changing circumstances” makes the broader business context clearer and avoids a possible reading of “environmental changes” as changes to the natural environment alone. “Respond effectively” and “achieve sustainable growth and enhanced corporate value” also read more naturally in this context. These examples illustrate specific improvements in wording; they do not imply that every change made by DPO is an improvement.

 

As an aside, early in development we also explored GRPO as an alternative to DPO. Although it improved benchmark scores, we did not adopt it because human evaluations indicated a decline in fluency.

For GRPO, we used an LLM as a judge to compute the reward. Even when we explicitly asked it to assess fluency, increases in reward did not translate into better human evaluations. We suspect this reflects the difficulty of reliably assessing fluency in each language.

Reinforcement learning is highly effective at maximizing the rewards it is given. We suspect that optimizing rewards for other criteria, such as translation fidelity, may have inadvertently come at the expense of fluency.

Since DPO also uses an LLM as a judge to select chosen and rejected translations, it might seem prone to the same problem. We therefore supplement the standard DPO loss with an SFT loss on the chosen translations and KL regularization against a reference model. We carefully tune the strength of these terms to keep the model from moving too far from its pre-DPO state. With these adjustments, we found that DPO could actually improve fluency.

Distillation

We complete training of the 31B model with SFT followed by DPO. For models of 8B and smaller, we add distillation using the 31B model as the teacher. Whereas SFT trains the model to produce the correct translation, distillation trains the student to match the teacher's next-token probability distribution at each position.

Our choice of distillation method builds on work by Kentaro Shinagawa during PFN's summer 2026 internship. A future internship blog post will describe the experiments that led to this method; here, we focus on the approach we ultimately adopted.

Our method builds on DistiLLM-2 [14] and adjusts the distributions used in training with AMiD [15], combining recent advances in LLM distillation. The following figure provides an overview.

Figure 9. Overview of the distillation method

Following DistiLLM-2, we combine off-policy and on-policy distillation. We train on the teacher's token distributions using two types of sequences:

  • Reference translation sequences: the (source text, translation) pairs in the dataset, used for off-policy distillation.
  • Student-generated sequences: source texts paired with translations generated by the current student at temperature 0, used for on-policy distillation.

DistiLLM-2 uses teacher-generated sequences for its off-policy component. Our implementation therefore differs slightly, but the underlying idea is similar.

Our training loss uses forward KL divergence between the teacher and student distributions on reference translation sequences, and reverse KL on student-generated sequences. Without going into the mathematical details, the intuition is:

  • Forward KL encourages the student to imitate the teacher’s behavior.
  • Reverse KL discourages behavior that differs from the teacher’s.

Broadly speaking, the reference sequences teach the student how the teacher behaves when producing a correct translation, while the student-generated sequences allow the teacher to correct the student’s own translation distribution.

Because the student changes during training, we regenerate its sequences every few dozen steps. This introduces a small delay, so the process is not strictly on-policy, but preliminary experiments confirmed that the delay does not affect the results.

The explanation above uses standard KL divergence. In practice, we adjust the loss with AMiD mixture distributions. The figure below illustrates the next-token distributions at one position in a student-generated sequence.

Figure 10. Illustration of an AMiD mixture distribution

As noted above, reverse KL penalizes behavior that differs from the teacher’s. If the student assigns probability to a token that the teacher considers almost impossible, the loss can become extremely large and make training unstable.

AMiD addresses this by combining the teacher and student distributions into a new distribution r, which serves as the student’s target. Both the mixing ratio and a parameter α control how the two distributions are combined.

With our settings, r largely preserves the probabilities of tokens that the teacher considers likely. For tokens that the teacher assigns near-zero probability, it also retains a small contribution from the student’s distribution. This softens the penalty when the student’s predictions differ from the teacher’s, allowing the student to move toward the teacher’s distribution more gradually. Preliminary experiments confirmed that training progressed more smoothly with this approach.

The figure shows the reverse KL case. For forward KL, we use the same form of mixture with the teacher and student roles reversed.

The figures below show how performance changes with distillation.

Figure 11. Improvements in Japanese–English translation performance through distillation
Figure 12. Improvements in multilingual translation performance through distillation

Distillation produces dramatic gains, especially for the smaller models. These results demonstrate the value of first building a large model, then using its capabilities to train smaller models.

After distillation, the models of 8B and smaller also undergo iterative DPO, just like the 31B model, to complete training.

Final Model Performance

The performance of the final models is shown below.

Figure 13. PLaMo 3 Translate performance

The horizontal axis shows total parameter count, and the vertical axis shows performance. As before, Japanese–English performance is averaged across PFTB, SaaS, and WMT24++ COMET-22. The multilingual results show average FLORES+ chrF++ scores for Japanese ↔ other languages and English ↔ other languages.

For comparison, we have also included existing translation-specialized LLMs: TranslateGemma [16] (Google DeepMind) and Hy-MT2 [17] (Tencent).

Overall, all PLaMo 3 Translate models substantially outperform existing translation-specialized LLMs on both Japanese–English and multilingual benchmarks. Even the smallest 0.8B model surpasses all of the existing translation-specialized LLMs in Japanese–English translation and translation between Japanese and other languages, including models with more than 30 times as many parameters. For translation between English and other languages, the 2B model already surpasses all of these existing models.

Performance declines somewhat as the models get smaller, but Japanese–English performance in particular holds up well. We attribute this to distillation, which helps us maximize the capabilities of the smaller models.

Taken together, these results lead us to believe that we have substantially advanced the Pareto frontier for model size and translation performance and built the world's best translation-specialized LLM.

Summary

  • The weights for PLaMo 3 Translate are now available on Hugging Face. We offer four sizes—0.8B, 2B, 8B, and 31B—with multiple quantization options and GGUF and MLX formats. We encourage you to try them.
  • We developed PLaMo 3 Translate through SFT, DPO, and, for the smaller models, distillation. Even the smallest 0.8B model outperforms existing translation-specialized LLMs with more than 30 times as many parameters. We believe these results justify calling PLaMo 3 Translate the world's best translation-specialized LLM.

References

[1] Kaito Suzuki (2026). "plamo-3-translate: A Frontier-Class Translation-Specialized LLM." Preferred Networks Blog. https://www.preferred.jp/ja/blog/tech/plamo-3-translate-product-release
[2] Cheng, W. et al. (2024). “Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs.” Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 11332–11350. https://doi.org/10.18653/v1/2024.findings-emnlp.662
[3] Kwon, W. et al. (2023). “Efficient Memory Management for Large Language Model Serving with PagedAttention.” Proceedings of the 29th Symposium on Operating Systems Principles (SOSP 2023). https://doi.org/10.1145/3600006.3613165 (vLLM: https://github.com/vllm-project/vllm)
[4] ggml-org. llama.cpp. GitHub. https://github.com/ggml-org/llama.cpp
[5] ml-explore. MLX LM. GitHub. https://github.com/ml-explore/mlx-lm
[6] Rafailov, R. et al. (2023). “Direct Preference Optimization: Your Language Model is Secretly a Reward Model.” Advances in Neural Information Processing Systems 36 (NeurIPS 2023). https://papers.neurips.cc/paper_files/paper/2023/hash/a85b405ed65c6477a4fe8302b5e06ce7-Abstract-Conference.html
[7] Deutsch, D. et al. (2025). “WMT24++: Expanding the Language Coverage of WMT24 to 55 Languages & Dialects.” Findings of the Association for Computational Linguistics: ACL 2025, pp. 12257–12284. https://doi.org/10.18653/v1/2025.findings-acl.634
[8] Open Language Data Initiative. FLORES+. Hugging Face. https://huggingface.co/datasets/openlanguagedata/flores_plus (Foundational dataset paper: NLLB Team (2024). “Scaling neural machine translation to 200 languages.” Nature 630, pp. 841–846. https://doi.org/10.1038/s41586-024-07335-x)
[9] Kentaro Imajo and Masanori Hirano (2026). “Investigation of Automatic Evaluation Methods for High-Precision Translation Models.” Proceedings of the 32nd Annual Meeting of the Association for Natural Language Processing, pp. 3623–3628. (In Japanese). https://www.anlp.jp/proceedings/annual_meeting/2026/pdf_dir/C8-1.pdf
[10] Kentaro Imajo (2026). “charsim: Character n-gram similarity less sensitive to parameters.” https://imoz.jp/scraps/202607_charsim.html
[11] Rei, R. et al. (2022). “COMET-22: Unbabel-IST 2022 Submission for the Metrics Shared Task.” Proceedings of the Seventh Conference on Machine Translation (WMT), pp. 578–585. https://doi.org/10.18653/v1/2022.wmt-1.52
[12] Popović, M. (2017). “chrF++: words helping character n-grams.” Proceedings of the Second Conference on Machine Translation, pp. 612–618. https://doi.org/10.18653/v1/W17-4770
[13] Gemma Team (2026). “Gemma 4 Technical Report.” arXiv:2607.02770. https://arxiv.org/abs/2607.02770
[14] Ko, J. et al. (2025). “DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMs.” Proceedings of the 42nd International Conference on Machine Learning (ICML 2025), PMLR 267, pp. 31044–31062. https://proceedings.mlr.press/v267/ko25a.html
[15] Shin, D. et al. (2026). “AMiD: Knowledge Distillation for LLMs with α-mixture Assistant Distribution.” The Fourteenth International Conference on Learning Representations (ICLR 2026). https://proceedings.iclr.cc/paper_files/paper/2026/file/54a468d19bc5727800096d9c7c51a423-Paper-Conference.pdf
[16] Finkelstein, M. et al. (2026). “TranslateGemma Technical Report.” arXiv:2601.09012. https://arxiv.org/abs/2601.09012
[17] Tencent Hunyuan Team (2026). “Hy-MT2: A Family of Fast, Efficient and Powerful Multilingual Translation Models in the Wild.” arXiv:2605.22064. https://arxiv.org/abs/2605.22064

*1 PLaMo Translation uses a translation-specialized LLM developed by PFN on top of PLaMo 3 NICT, a pretrained model resulting from joint research by PFN and the National Institute of Information and Communications Technology (NICT).

*2 Our evaluation covers a subset of the supported languages. PLaMo 3 Translate is also trained on the additional languages listed below, and its training includes translation among languages other than Japanese as well as translation to and from Japanese.

Hungarian, Polish, Hindi, Persian, Bengali, Tagalog, Hebrew, Tamil, Burmese, Urdu, Malay, Nepali, Catalan, Malayalam, Bulgarian, Serbian, Croatian, Slovak, Slovenian, Sinhala, Azerbaijani, Georgian, Armenian, Uzbek, Lithuanian, Khmer, Latvian, Estonian, Basque, Cebuano, Belarusian, Telugu, Javanese, Kazakh, Marathi, Swahili, Gujarati, Punjabi

Join the PFN team

Contact us for inquiries on our products, solutions, R&D and other case studies