moses-smt/mosesdecoder
1
1
Fork 0
mirror of https://github.com/moses-smt/mosesdecoder.git synced 2024-12-27 05:55:02 +03:00
Code Issues Packages Projects Releases Wiki Activity
1752993414
mosesdecoder/scripts/training/bilingual-lm
History
Barry Haddow 7125096c29 enable nplm training on separate host, fix ems for nplm
2016-01-06 11:55:12 +00:00
..
averageNullEmbedding.py Fixing lint. Only 600 or so lines of errors left! 2015-06-02 17:29:32 +07:00
create_blm_ini.py Improvements to handling of bilingual LM in EMS 2015-07-10 15:44:24 +01:00
extract_test.py Add license notices to scripts. 2015-05-29 18:30:26 +07:00
extract_training.py Add license notices to scripts. 2015-05-29 18:30:26 +07:00
extract.py Add license notices to scripts. 2015-05-29 18:30:26 +07:00
README move bilingual-lm training scripts 2014-11-26 10:32:37 +00:00
reduce_ngrams.py Add license notices to scripts. 2015-05-29 18:30:26 +07:00
tag.sh move bilingual-lm training scripts 2014-11-26 10:32:37 +00:00
test_nplm.py Add license notices to scripts. 2015-05-29 18:30:26 +07:00
train_nplm.py enable nplm training on separate host, fix ems for nplm 2016-01-06 11:55:12 +00:00

README 

Example usage:
#create training and test corpus
/home/abmayne/code/deepathon/nnjm/extract_training.py --working-dir  /home/abmayne/experiments/2014-iwslt/nplm/en-cs10k --corpus corpus/europarl.clean.10k --target-language cs --source-language en --align corpus/europarl.clean.10k.align
/home/abmayne/code/deepathon/nnjm/extract_test.py --working-dir  /home/abmayne/experiments/2014-iwslt/nplm/en-cs10k --corpus corpus/europarl.test.10k --target-language cs --source-language en --align corpus/europarl.test.10k.align

#Train and test different language models with basic nplm training
/home/abmayne/code/deepathon/nnjm/train_nplm.py --working-dir /home/abmayne/experiments/2014-iwslt/nplm/en-cs10k --corpus europarl.clean.10k --minibatch-size 128 --epochs 40 --output-model europarl.10k.bbn  --nplm-home  /home/abmayne/code/deepathon/nplm_one_layer --hidden 0 --threads 1 --output-model europarl.10k.1layer
/home/abmayne/code/deepathon/nnjm/test_nplm.py --working-dir /home/abmayne/experiments/2014-iwslt/nplm/en-cs10k --corpus europarl.test.10k --train-corpus europarl.10k.1layer --nplm-home  /home/abmayne/code/deepathon/nplm_one_layer --threads 1