moses-smt/mosesdecoder
1
1
Fork 0
mirror of https://github.com/moses-smt/mosesdecoder.git synced 2025-01-08 04:27:53 +03:00
Code Issues Packages Projects Releases Wiki Activity
1,770 Commits 136 Branches 26 Tags 147 MiB
21de1e121f
Commit Graph

300 Commits

Author SHA1 Message Date
bojar
21de1e121f Merge branch 'master' into moses-svn 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3192 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-21 14:48:32 +00:00
bojar
db8e3357f5 Merge branch 'zmert' into moses-svn 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3152 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-20 22:03:04 +00:00
bojar
aa6043a556 mert-moses must quote -n-best-list *if* passed through parallelizer (and must not otherwise) 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3151 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-20 17:02:58 +00:00
bojar
8ea058bfc7 fixed handling of the lmodel section; has only 3 ints 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3150 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-20 17:02:41 +00:00
pjwilliams
2edfc16912 Merge remaining script support for tree-based models from mt3_chart. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3137 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-16 09:45:51 +00:00
hieuhoang1972
a2233d0f8d xcode 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3136 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-14 16:53:39 +00:00
pjwilliams
264c9150d1 Use external consolidate' program in train-factored-phrase-model.perl.' 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3135 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-14 15:51:20 +00:00
pjwilliams
5faaedc0df Copy in consolidate,' consolidate-direct,' and the new version of 
`score' from mt3_chart.


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3134 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-14 15:50:17 +00:00
hieuhoang1972
06ee9a3be3 vs 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3132 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-13 16:50:44 +00:00
pjwilliams
53cb08efca Use a generic version of the SAFE_GETLINE macro in scripts/phrase-extract 
instead of defining one per source file.


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3131 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-13 16:29:55 +00:00
hieuhoang1972
0440dfe079 xcode 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3130 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-13 16:13:56 +00:00
pjwilliams
580acce9e2 Integrate rule extraction code from mt3_chart. There are now two extract 
programs: `extract' for the phrase-based model and `extract-rules' for
tree-based models.  They could be combined into a single program, but
they're probably sufficiently different that it isn't worthwhile.


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3129 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-13 15:34:39 +00:00
pjwilliams
51ae927ede Start merging in rule extraction code from mt3_chart branch. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3126 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-12 15:22:50 +00:00
pjwilliams
9c2536417f Remove file limit option for phrase extraction. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3122 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-12 11:56:54 +00:00
pjwilliams
99f1c92edb Remove redundant --ZipFiles option from extract. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3120 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-12 10:53:08 +00:00
pjwilliams
4c6c4b71cf Remove redundant --ProperConditioning option from extract. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3118 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-12 10:41:32 +00:00
bojar
7172d05a43 fixed a bug, too eager check for preprocessing type; not needed in some cases 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3111 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-09 23:21:13 +00:00
bojar
82d6cc714e use qruncmd to parallelize srunblocks 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3110 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-09 23:20:59 +00:00
bojar
c5f44a2abf better verbosity level for srunblocks: emit some (most importantly fatal) msgs 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3109 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-09 23:20:45 +00:00
bojar
390ee866d8 require TMT_ROOT only if TectoMT will be actually needed 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3108 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-09 23:20:30 +00:00
bojar
9fe574f6ed fixed weights for SemPOS_BLEU metric 
usage: MERTFLAGS="--semposbleu-weights <sempos_weight>:<bleu_weight>"
e.g. --semposbleu-weights 2:1 to increase the weight of SemPOS


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3107 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-09 22:47:22 +00:00
bojar
26d77d15b2 added option --semposbleu-weights to specify weight of SemPOS and BLEU in SemPOS_BLEU metric 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3106 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-09 22:47:04 +00:00
bojar
c954a507c6 fixed error message in zmert-moses.pl 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3105 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-09 22:46:49 +00:00
bojar
8e2e4eeecc bug fixed 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3104 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-09 22:46:34 +00:00
bojar
b8a6048e81 ZMERTSEMPOSSOURCE=factors working for SemPOS_BLEU 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3103 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-09 22:46:17 +00:00
bojar
866776810b typo fixed 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3102 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-09 22:45:59 +00:00
bojar
b87370aec2 few bugs fixed in ZMERTSEMPOSSOURCE=factors part 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3101 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-09 22:45:43 +00:00
bojar
19184783a9 fixed bug in --mert-verbose parameter 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3100 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-09 22:45:28 +00:00
bojar
9f3d2f427d fixed nbest-list conversion 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3098 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-09 22:44:41 +00:00
bojar
2fb22064a3 bug in passing nbest-lists to zmert 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3097 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-09 22:44:21 +00:00
bojar
cd64a02344 fixed bug - missing loop to copy one file to another 
Conflicts:

	scripts/training/zmert-moses.pl


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3096 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-09 22:43:55 +00:00
bojar
b276bacb72 bug in foreach 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3095 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-09 22:43:27 +00:00
bojar
94eb8a5c1b added default parameters for basic metrics 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3094 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-09 22:43:10 +00:00
bojar
2604a890d7 zmert training with new metric SemPOS_BLEU - linear combination of SemPOS and BLEU 
SemPOS_BLEU requires new TMT block ForSemPOSBLEUMetric.pm in TectoMT. Please,
update your TMT.


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3093 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-09 22:42:49 +00:00
bojar
5b1541b96d Fixed order of feature scores passed to zmert. Evaluation using BLEU works. 
Zmert uses different order of features than Moses. It is necessary to reorder
them when passing to Zmert in nbest-lists.
Previous versions used wrong copy of moses.ini, which linked to phrase tables
that were already filtered for tuning. Thus, phrase tables for evaluation were
missing a lot of phrases and the reported scores were too low.


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3092 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-09 22:42:20 +00:00
bojar
5a1c308673 Setting number of zmert iterations back to unlimited in zmert config. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3091 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-09 22:41:55 +00:00
bojar
671e7e2e4f Default size of nbest-list in zmert-moses.pl set back to 100. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3090 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-09 22:41:31 +00:00
bojar
1ce9a15483 Bug fixes im zmert-moses.pl and zmert.jar. Zmert works. 
Now it is possible to lauch Zmert with SemPOS metric.
It is possible to select a smaller model for McD parser by uncommenting line
with pdt20_train_autTag_golden_latin2_pruned_0.10.model in file zmert.tmt-scen.
Sentences are then analyzed faster (if you use TectoMT to get SemPOS tags).


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3089 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-09 22:41:11 +00:00
bojar
1b28cde2e8 small fixes of paths 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3088 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-09 22:40:29 +00:00
bojar
d47b82369f some small code modifications in zmert-moses.pl 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3087 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-09 22:39:47 +00:00
bojar
d34af9b769 updated zmert-moses.pl - zmert with sempos still not running 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3086 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-09 22:39:31 +00:00
bojar
49bb6b8882 Updated version of zmert training - still not finished SemPOS factor loading via TMT 
zmert-moses.pl - launches zmert training
zmert-decoder.pl - an is launched by zmert after each training iteration to compute scores
                   with updated lambdas
zmert.tmt-scen - TMT scenario to extract SemPOS factor for sentence without any factors


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3085 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-09 22:39:14 +00:00
bojar
90fecff3aa New mert training (Zmert) for Moses\n\nZmert jar includes SemPOS metric extension. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3084 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-09 22:38:53 +00:00
pjwilliams
b3f6e211fd Fix mistakes in previous commit (oh, and revert to my own svn username 
to prevent my shoddy check-ins from further sullying Hieu's good name).


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3083 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-09 14:02:58 +00:00
hieuhoang1972
c6d20e1f9f Update the training scripts to support the new format parameter for 
'ttable-file'


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3082 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-09 11:37:43 +00:00
sarst
7d3a22c8a7 Bugfixes for the new lexical reordering. Running without a reordering model, and with the reordering flag specyfying distance now works. Formatting of the extract.o file is now correct with 'f'-reordering models. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3054 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-06 12:08:04 +00:00
bhaddow
bd9f392875 Fix for training with non-lexicalised reordering 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3016 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-01 14:05:58 +00:00
sarst
943275e331 Set the default debug mode to 0 in train-factored-phrase-model.perl (which was the case before merging the hierarchical reordering branch to trunk) 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3009 1f5c12ca-751b-0410-a591-d2e778427230
 2010-03-30 12:09:19 +00:00
bhaddow
9573147e36 Fix --reordering-table option 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3004 1f5c12ca-751b-0410-a591-d2e778427230
 2010-03-26 09:32:15 +00:00
bhaddow
8060024cc1 Can now specify reordering table when executing step 9 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/branches/hierarchical-reo@2997 1f5c12ca-751b-0410-a591-d2e778427230
 2010-03-23 18:55:48 +00:00