OPUS-MT-train/models/en-ber/README.md

opus-2019-12-18.zip

• dataset: opus
• model: transformer-align
• pre-processing: normalization + SentencePiece
• download: opus-2019-12-18.zip
• test set translations: opus-2019-12-18.test.txt
• test set scores: opus-2019-12-18.eval.txt

Benchmarks

testset	BLEU	chr-F
Tatoeba.en.ber	29.7	0.544

opus+bt-2020-05-23.zip

• dataset: opus+bt
• model: transformer-align
• source language(s): en
• target language(s): ber
• model: transformer-align
• pre-processing: normalization + SentencePiece (spm1k,spm1k)
• download: opus+bt-2020-05-23.zip
• test set translations: opus+bt-2020-05-23.test.txt
• test set scores: opus+bt-2020-05-23.eval.txt

Training data: opus+bt

• en-ber: QED (23) Ubuntu (6)
• en-ber: total size = 29
• unused dev/test data is added to training data
• total size (opus+bt): 70855

Validation data

• ber-en: Tatoeba

• devset = top 5000 lines of Tatoeba.src.shuffled!

• testset = next 5000 lines of Tatoeba.src.shuffled!

• remaining lines are added to traindata

Benchmarks

testset	BLEU	chr-F
Tatoeba.en.ber	23.3	0.476