OPUS-MT-train/TODO.md

Things to do

data preparation

• do temperature-based balanced sampling (see https://arxiv.org/pdf/1907.05019.pdf)

slurm job pipelines

Create slurm jobs with dependencies to create pipelines of jobs. (add --dependencies to sbatch) see https://hpc.nih.gov/docs/job_dependencies.html https://hpc.nih.gov/docs/userguide.html#depend grep for job is (pattern: 'Submitted batch job 946074')

Issues

• racing situations in work/data with jobs that fetch data for the same language pairs!
• get rid of BPE to simplify the scripts

General settings

• better hyperparameters for low-resource setting (lower batch sizes, smaller vocabularies ...)
• better data selection (data cleaning / filtering); use opus-filter?
• better balance between general data sets and backtranslations

Backtranslation

• status: basically working, need better integration?!
• add backtranslations to training data
• can use monolingual data from tokenized wikipedia dumps: https://sites.google.com/site/rmyeid/projects/polyglot
• https://dumps.wikimedia.org/backup-index.html
• better in JSON: https://dumps.wikimedia.org/other/cirrussearch/current/

Fine-tuning and domain adaptation

• status: basically working
• do we want to publish fine-tuned data or rather the fina-tuning procedures? (using a docker container?)

Show-case some selected language pairs

• collaboration with wikimedia
• focus languages: Tagalog (tl, tgl), Central Bikol (bcl), Malayalam (ml, mal), Bengali (bn, ben), and Mongolian (mn, mon)

Other requests

• Hebrew-->English and Hebrew-->Russian (Shaul Dar)