nllb200-formosan-en-spm8k

Base model: facebook/nllb-200-distilled-600M
Direction: Formosan -> English
Companion model: FormosanBank/nllb200-en-formosan-spm8k
Release: private no-Bible SPM8k flight 20260712-232900, validation-selected step 270,000

This directional checkpoint replaces the earlier release with the strongest private_no_bible model from the fully rebuilt FormosanBank MT pipeline. It uses an 8,192-piece Formosan-aware SentencePiece extension and explicit direction, source-language, source-domain, and dialect control tags.

Supported Languages

Language	NLLB code
English	eng_Latn
Amis	ami_Latn
Bunun	bnn_Latn
Kavalan	ckv_Latn
Rukai	dru_Latn
Paiwan	pwn_Latn
Puyuma	pyu_Latn
Thao	ssf_Latn
Saaroa	sxr_Latn
Sakizaya	szy_Latn
Tao / Yami	tao_Latn
Atayal	tay_Latn
Seediq	trv_Latn
Tsou	tsu_Latn
Kanakanavu	xnb_Latn
Saisiyat	xsy_Latn

Input Format

Prefix every source with:

<to_eng> <src_LANG> <dom_BUCKET> <dialect_DIALECT>

Example:

<to_eng> <src_ami> <dom_unknown> <dialect_default> Pa'araw cingra to demak nira.

Use <dom_unknown> and <dialect_default> when metadata is unavailable.

Usage

Use the slow NllbTokenizer (use_fast=False with AutoTokenizer). These checkpoints were trained with transformers==4.56.1; fast-tokenizer added-token IDs can differ from the slow tokenizer IDs used in training. NLLB generation must start with the tokenizer EOS ID and force the target-language BOS ID.

import torch
from transformers import AutoModelForSeq2SeqLM, NllbTokenizer

model_id = "FormosanBank/nllb200-formosan-en-spm8k"
tokenizer = NllbTokenizer.from_pretrained(model_id)
model = AutoModelForSeq2SeqLM.from_pretrained(model_id)
model.to("cuda" if torch.cuda.is_available() else "cpu")

FORMOSAN_TO_LID = {
    "ami": "ami_Latn", "bnn": "bnn_Latn", "ckv": "ckv_Latn", "dru": "dru_Latn",
    "pwn": "pwn_Latn", "pyu": "pyu_Latn", "ssf": "ssf_Latn", "sxr": "sxr_Latn",
    "szy": "szy_Latn", "tao": "tao_Latn", "tay": "tay_Latn", "trv": "trv_Latn",
    "tsu": "tsu_Latn", "xnb": "xnb_Latn", "xsy": "xsy_Latn",
}

def translate_formosan_to_english(text: str, lang_code: str, source_bucket: str = "unknown", dialect: str = "default") -> str:
    tokenizer.src_lang = FORMOSAN_TO_LID[lang_code]
    prompt = f"<to_eng> <src_{lang_code}> <dom_{source_bucket}> <dialect_{dialect}> {text}"
    inputs = tokenizer(prompt, return_tensors="pt", truncation=True, max_length=384).to(model.device)
    outputs = model.generate(
        **inputs,
        forced_bos_token_id=tokenizer.convert_tokens_to_ids("eng_Latn"),
        decoder_start_token_id=tokenizer.eos_token_id,
        max_new_tokens=128,
        num_beams=4,
        no_repeat_ngram_size=3,
        repetition_penalty=1.15,
        early_stopping=True,
    )
    return tokenizer.batch_decode(outputs, skip_special_tokens=True)[0]

print(translate_formosan_to_english("Pa'araw cingra to demak nira.", "ami"))

Checkpoint Selection

The published checkpoint was selected only on validation chrF2, not on the hard test set.

Selection step	Validation samples	Validation loss	Perplexity	BLEU	chrF2	TER
270,000	1,920	2.2070	9.09	19.15	35.33	80.93

Validation generation sampled 128 rows per Formosan language every 10,000 updates. The full hard test was evaluated only after selection.

Training Setup

Setting	Value
Corpus	private_no_bible (English)
Base model	facebook/nllb-200-distilled-600M
Maximum updates	300,000
Published best step	270,000
Microbatch / accumulation	16 / 4
Effective batch	64
Maximum length	384
Learning rate	2e-05
Precision	bf16
Easy-source weight	0.05
Language sampling alpha	0.5
Metadata control tags	enabled and validated as single tokenizer IDs

Corpus and Split Integrity

Total	Train	Test	Validate	Minimum per-language test	Minimum per-language validate
759,493	681,034	58,129	20,330	7.5%	2.5%

The exact Formosan-Taiwan-Bible-Society-Bibles repository is excluded. Lexical entries are train-only. Independent validation found zero normalized source, target, or pair overlap; zero punctuation/spacing skeleton overlap; and zero one-edit source or target conflicts across train and evaluation. Connected similarity groups are assigned as units so variants cannot be split independently merely because they are not exact duplicates.

The English hard test contains 51,206 original-reference rows and 6,923 DeepL-pivoted reference rows. Synthetic rows were admitted only when the available human sentence groups could not satisfy a language's minimum evaluation floor; metrics should not be described as human-only.

Hard-Test Results

SacreBLEU was computed with 13a tokenization; chrF uses beta 2; TER is lower-is-better.

Direction	Samples	BLEU	chrF2	TER	Exact match	Empty output
Formosan -> English	58,129	13.08	31.17	86.50	2.42%	0.00%

Per-Language Results

Language	Code	Samples	BLEU	chrF2	TER
Amis	ami_Latn	10,741	10.50	28.90	93.74
Bunun	bnn_Latn	5,319	12.84	32.45	80.82
Kavalan	ckv_Latn	2,728	13.74	32.26	85.13
Rukai	dru_Latn	5,340	9.20	26.56	88.63
Paiwan	pwn_Latn	5,349	11.78	31.27	84.88
Puyuma	pyu_Latn	3,480	15.26	34.25	79.45
Thao	ssf_Latn	1,521	18.84	36.02	80.85
Saaroa	sxr_Latn	1,466	16.72	33.26	84.19
Sakizaya	szy_Latn	1,983	15.14	34.29	89.05
Tao / Yami	tao_Latn	1,729	16.97	33.51	83.90
Atayal	tay_Latn	5,985	16.31	33.12	80.59
Seediq	trv_Latn	6,386	14.66	32.55	86.39
Tsou	tsu_Latn	1,784	11.32	29.23	89.88
Kanakanavu	xnb_Latn	2,688	8.31	29.18	86.57
Saisiyat	xsy_Latn	1,630	10.52	29.28	90.05

Full source-bucket and length-bin breakdowns are in eval/metrics.json.

Intended Use

• Research, teaching, and prototyping for Formosan-language machine translation.
• Draft translation assistance where knowledgeable speakers can review the output.
• Comparative low-resource MT evaluation on the documented leakage-controlled split.

Limitations

• Output may be incorrect, ungrammatical, incomplete, or culturally inappropriate.
• Formosan generation is draft-only and requires speaker review.
• Aggregate scores across 15 languages conceal substantial per-language variation.
• This model is unsuitable for legal, medical, safety-critical, or authoritative community-facing use without expert review.
• Hard-split scores are not directly comparable with earlier evaluations that allowed stronger train-test similarity.

License

Released under cc-by-nc-4.0. Underlying corpus sources may impose additional restrictions. Confirm the rights needed for your use case.

Citation

@misc{formosanbank_nllb200_formosan_en_spm8k_2026,
  title  = {nllb200-formosan-en-spm8k: Directional NLLB-200 MT on the FormosanBank private no-Bible corpus},
  author = {FormosanBank contributors},
  year   = {2026},
  url    = {https://huggingface.co/FormosanBank/nllb200-formosan-en-spm8k}
}