Helsinki-NLP/OPUS-MT-train

mirror of https://github.com/Helsinki-NLP/OPUS-MT-train.git synced 2024-10-26 21:19:02 +03:00

Joerg Tiedemann e2ed3d85d1 finetuning and backtranslation

2020-01-12 01:10:53 +02:00

382 B

Raw Blame History

Translate data as synthetic training data

Use Wiki data:

json processor: https://stedolan.github.io/jq/
wiki JSON dumps: https://dumps.wikimedia.org/other/cirrussearch/current/

NOTE: this only works for SentencePiece models

TODO

download base models from ObjectStorage
make it work with multilingual models (need to adjust preprocess-scripts for those models)