moses-smt/mosesdecoder
1
1
Fork 0
mirror of https://github.com/moses-smt/mosesdecoder.git synced 2024-09-20 07:42:21 +03:00
Code Issues Packages Projects Releases Wiki Activity
1,899 Commits 136 Branches 26 Tags 147 MiB
e409b6827c
Commit Graph

1899 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
bgottesman
e409b6827c add --max-word-length option to cleaning script, with default value 1000; any segment with a word (or factor) exceeding this length in chars is discarded; motivated by symal.cpp, which has its own such parameter (hardcoded to 1000) and crashes if it encounters a word that exceeds it 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3410 1f5c12ca-751b-0410-a591-d2e778427230
 2010-08-23 16:35:14 +00:00
pjwilliams
2deb68af84 In PhraseDictionaryNodeSCFG, use separate maps for children with terminal and non-terminal keys. This removes the need to look up source terminals twice. The chart decoder spends a *lot* of time in PhraseDictionaryNodeSCFG::GetChild() (approx 38% of post-startup decoding time in my target syntax test, according to callgrind), so this makes a significant difference. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3409 1f5c12ca-751b-0410-a591-d2e778427230
 2010-08-17 13:41:46 +00:00
pjwilliams
98383c1393 Pare down PhraseDictionaryNodeSCFG. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3408 1f5c12ca-751b-0410-a591-d2e778427230
 2010-08-17 11:01:03 +00:00
redpony
570183461c plf checker 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3407 1f5c12ca-751b-0410-a591-d2e778427230
 2010-08-16 01:03:39 +00:00
hieuhoang1972
083a9af215 delete alignment info for terminals 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3405 1f5c12ca-751b-0410-a591-d2e778427230
 2010-08-13 10:03:13 +00:00
rafpayen
b431f951c5 git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3404 1f5c12ca-751b-0410-a591-d2e778427230 2010-08-13 09:58:17 +00:00
hieuhoang1972
ef09298824 move function calls with side effects out of asserts 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3403 1f5c12ca-751b-0410-a591-d2e778427230
 2010-08-12 12:54:55 +00:00
bhaddow
7b06173064 No longer required 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3402 1f5c12ca-751b-0410-a591-d2e778427230
 2010-08-10 15:33:06 +00:00
bhaddow
d7fe4353c7 no longer relevant 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3401 1f5c12ca-751b-0410-a591-d2e778427230
 2010-08-10 14:57:02 +00:00
hieuhoang1972
382799dd38 delete win32 make. out-of-date. find a better way of doing this 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3400 1f5c12ca-751b-0410-a591-d2e778427230
 2010-08-10 14:41:49 +00:00
hieuhoang1972
9600b6473c delete win32 make. out-of-date. find a better way of doing this 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3399 1f5c12ca-751b-0410-a591-d2e778427230
 2010-08-10 14:41:28 +00:00
bhaddow
08a8480136 rename mert-moses-new.pl to mert-moses.pl 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3398 1f5c12ca-751b-0410-a591-d2e778427230
 2010-08-10 14:29:07 +00:00
bhaddow
321f528ff5 remove zmert and cmert 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3397 1f5c12ca-751b-0410-a591-d2e778427230
 2010-08-10 14:28:03 +00:00
hieuhoang1972
8616a2bdee visual studio 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3396 1f5c12ca-751b-0410-a591-d2e778427230
 2010-08-10 13:51:20 +00:00
hieuhoang1972
8fc72ee74a xcode 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3395 1f5c12ca-751b-0410-a591-d2e778427230
 2010-08-10 13:35:08 +00:00
bhaddow
904133fcb7 Merge in the multiple models branch. These changes allow the moses server 
to support multiple translation, language and generation models within the
same process. The main design change is the introduction of a TranslationSystem
object to manage the models, which have been moved out of StaticData.
The changes should have no effect on existing systems.


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3394 1f5c12ca-751b-0410-a591-d2e778427230
 2010-08-10 13:12:00 +00:00
bhaddow
d31b030bc5 Write correct ttable type when binarising a phrase table 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3392 1f5c12ca-751b-0410-a591-d2e778427230
 2010-08-09 12:28:33 +00:00
hieuhoang1972
d7d297eaa2 get rid of debug messages 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3391 1f5c12ca-751b-0410-a591-d2e778427230
 2010-08-09 11:20:20 +00:00
hieuhoang1972
b219f63dde delete debug info 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3387 1f5c12ca-751b-0410-a591-d2e778427230
 2010-08-06 17:10:19 +00:00
bhaddow
f2660e8d41 Fix glue grammar generation for new ttable format 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3386 1f5c12ca-751b-0410-a591-d2e778427230
 2010-08-06 14:45:37 +00:00
hieuhoang1972
7e6b3766dd visual studio 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3385 1f5c12ca-751b-0410-a591-d2e778427230
 2010-08-03 15:56:41 +00:00
bhaddow
faf65dfcd2 Remove unused options. 
Merge in some changes from mert-moses.perl


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3384 1f5c12ca-751b-0410-a591-d2e778427230
 2010-08-03 15:54:31 +00:00
rafpayen
1896cc7fff better messages 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3383 1f5c12ca-751b-0410-a591-d2e778427230
 2010-08-03 15:34:34 +00:00
hieuhoang1972
277ba483e1 alignment info in the decoder 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3381 1f5c12ca-751b-0410-a591-d2e778427230
 2010-08-02 16:32:36 +00:00
hieuhoang1972
579253d3cd add lowercaser 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3380 1f5c12ca-751b-0410-a591-d2e778427230
 2010-08-02 14:05:23 +00:00
phkoehn
56447ed42c bug fix for nested zones 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3379 1f5c12ca-751b-0410-a591-d2e778427230
 2010-07-30 22:17:08 +00:00
rafpayen
2ef133e02b add empty fields in glue grammar to accomodate the new phrase table format 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3378 1f5c12ca-751b-0410-a591-d2e778427230
 2010-07-30 15:55:14 +00:00
nicolabertoldi
621428de44 improved description of configuration file for [ttable-file] parameter 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3376 1f5c12ca-751b-0410-a591-d2e778427230
 2010-07-30 09:10:17 +00:00
hieuhoang1972
8adef921ed new format for consolidate-direct 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3374 1f5c12ca-751b-0410-a591-d2e778427230
 2010-07-29 23:20:37 +00:00
hieuhoang1972
0ee6d75566 bug in Good turing 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3372 1f5c12ca-751b-0410-a591-d2e778427230
 2010-07-28 22:49:37 +00:00
hieuhoang1972
340ebbd333 bug in Good turing 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3370 1f5c12ca-751b-0410-a591-d2e778427230
 2010-07-28 21:52:32 +00:00
hieuhoang1972
ae9779dd7f separate PhraseAlignment class into separate file 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3369 1f5c12ca-751b-0410-a591-d2e778427230
 2010-07-28 21:28:14 +00:00
hieuhoang1972
7221bf2dd4 alignment info, for chart decoding, updated regression 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3368 1f5c12ca-751b-0410-a591-d2e778427230
 2010-07-28 09:53:21 +00:00
hieuhoang1972
8f11e17615 alignment info, new format 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3366 1f5c12ca-751b-0410-a591-d2e778427230
 2010-07-27 11:25:30 +00:00
hieuhoang1972
7c6007f018 alignment info, new format 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3365 1f5c12ca-751b-0410-a591-d2e778427230
 2010-07-27 11:21:12 +00:00
hieuhoang1972
fc56e031d4 alignment info, new format 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3364 1f5c12ca-751b-0410-a591-d2e778427230
 2010-07-27 11:10:13 +00:00
hieuhoang1972
3d9d756055 alignment info, new format 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3363 1f5c12ca-751b-0410-a591-d2e778427230
 2010-07-27 11:04:03 +00:00
rafpayen
b9e74aab90 change phrase-word-alignment to boolean flag instead of string 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3362 1f5c12ca-751b-0410-a591-d2e778427230
 2010-07-22 13:08:46 +00:00
hieuhoang1972
881117d9f5 alignment info in pt 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3361 1f5c12ca-751b-0410-a591-d2e778427230
 2010-07-18 19:49:08 +00:00
hieuhoang1972
b9339bdf0e svn properties 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3360 1f5c12ca-751b-0410-a591-d2e778427230
 2010-07-17 23:23:09 +00:00
hieuhoang1972
dd7d3d1b56 vs.net 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3359 1f5c12ca-751b-0410-a591-d2e778427230
 2010-07-17 23:11:55 +00:00
hieuhoang1972
31930eb6fc alignment info in pt 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3358 1f5c12ca-751b-0410-a591-d2e778427230
 2010-07-17 22:29:06 +00:00
pjwilliams
fab2e96d2f In extract-rules, if the source or target syntax contains an unsupported 
escape sequence (anything other than "<", ">", "&", "&apos",
and "&quot") then write a warning message and skip the sentence pair
(instead of asserting).


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3350 1f5c12ca-751b-0410-a591-d2e778427230
 2010-06-29 10:41:42 +00:00
mphi
1f6e9b488b the script now calculates the p-value and confidence intervals not only using BLEU, but also the NIST score; 
improved confidence interval representation (avg+-stddev);

fixed bugs



git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3345 1f5c12ca-751b-0410-a591-d2e778427230
 2010-06-22 20:17:42 +00:00
hieuhoang1972
a21c9bff68 debug output 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3334 1f5c12ca-751b-0410-a591-d2e778427230
 2010-06-13 21:06:34 +00:00
hieuhoang1972
f24fb6449e delete pragma once when using #define 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3324 1f5c12ca-751b-0410-a591-d2e778427230
 2010-06-10 15:36:08 +00:00
nicolabertoldi
6e67edd11f sorted lif of headers and sources 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3323 1f5c12ca-751b-0410-a591-d2e778427230
 2010-06-10 15:32:33 +00:00
nicolabertoldi
e6d39bf83a minor fix 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3322 1f5c12ca-751b-0410-a591-d2e778427230
 2010-06-10 15:19:07 +00:00
nicolabertoldi
79d91a572d memory-unmap moved into the destructor 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3316 1f5c12ca-751b-0410-a591-d2e778427230
 2010-06-09 14:34:15 +00:00
nicolabertoldi
f38d220b67 moving compilation of LanguageModelParallelBackoff in the SRILM-dependent region 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3315 1f5c12ca-751b-0410-a591-d2e778427230
 2010-06-09 14:08:55 +00:00