OPUS-MT-train/pivoting

Pivot-based data augmentation

The idea of this folder is to create synthetic training data by translating existing bitexts for a different language pair on one side. An example is to create training data for translation from Breton to English from bitexts in Breton and French. The French part of the auxiliary corpus is translated to English using a strong French-English translation model.

This assumes that

• auxiliary data sets are in ORIGINAL_DATADIR (defaults to ${PWD}/../work/data)
• packaged translation models can be found in ${PWD}/../models or ${PWD}/../models

Usage

Set variables SRC, TRG and PIVOT and run make all, for example to translate French-Breton data to English-Breton:

make SRC=en TRG=br PIVOT=fr all

You can print the data that will be translated and the model that will be used for that by running:

make SRC=en TRG=br PIVOT=fr print-all-data
make SRC=en TRG=br PIVOT=fr print-modelname

If this does not print anything then running make all does not make sense. For submitting a job via slurm you can add the suffic .submit to the call, e.g.

make SRC=en TRG=br PIVOT=fr all.submit

Specific models

Special targets for specific models are defined in lib/models.mk. Use them like this:

• Sami language model: make all-sami (can also do make print-all-data-sami and make print-modelname-sami)

TODO

• get models from ObjectStorage instead to fetch them from the local filesystem
• get auxiliary data from OPUS instead of pre-processed data in the OPUS-MT dir (with hard-coded path)