moses-smt/mosesdecoder
1
1
Fork 0
mirror of https://github.com/moses-smt/mosesdecoder.git synced 2025-01-06 19:49:41 +03:00
Code Issues Packages Projects Releases Wiki Activity
1,929 Commits 136 Branches 26 Tags 147 MiB
7929e4624e
Commit Graph

332 Commits

Author SHA1 Message Date
bojar
6a087d59c4 removed SCRIPTS_ROOTDIR from this 'my' declaration, it was obscuring previous 
declaration!
lines wrapped


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1865 1f5c12ca-751b-0410-a591-d2e778427230
 2008-07-14 16:24:18 +00:00
bojar
c20e682f18 Avoid NFS race condition: 
explicitly remove old cmert output files (hoping that they will be correctly
  replaced by a 'mv' in the shell script submitted to SGE by qsubwrapper
  occasionally reveals a race condition in NFS => weights seem unchanged =>
  mert finishes too early)


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1862 1f5c12ca-751b-0410-a591-d2e778427230
 2008-07-10 11:47:55 +00:00
bhaddow
83f234cf17 Implementation of Cer et al mert regularisation. Use with argument such 
as --scconfig regtype:min,regwin:3 in extractor and mert. Only tested
on toy example so far.


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1860 1f5c12ca-751b-0410-a591-d2e778427230
 2008-06-24 19:27:18 +00:00
hieuhoang1972
52c2843e6c perl regexpr bug, submitted by German Sanchis Trilles 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1855 1f5c12ca-751b-0410-a591-d2e778427230
 2008-06-19 21:57:29 +00:00
bhaddow
4195b70247 First cut of new mert outer loop 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1842 1f5c12ca-751b-0410-a591-d2e778427230
 2008-06-10 09:07:20 +00:00
hieuhoang1972
1b44c7c445 most popular alignment outputted, finally 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1818 1f5c12ca-751b-0410-a591-d2e778427230
 2008-06-04 14:49:56 +00:00
hieuhoang1972
8554a7c89d most popular alignment outputted, finally 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1817 1f5c12ca-751b-0410-a591-d2e778427230
 2008-06-04 14:42:51 +00:00
hieuhoang1972
3832f68fed most popular alignment outputted, finally 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1816 1f5c12ca-751b-0410-a591-d2e778427230
 2008-06-04 14:40:04 +00:00
hieuhoang1972
bf34eb891d don't output alignment if inverse 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1813 1f5c12ca-751b-0410-a591-d2e778427230
 2008-06-03 12:25:37 +00:00
hieuhoang1972
b48ce341e9 output most aligned instead of merged 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1798 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-28 13:49:04 +00:00
phkoehn
7498f469ab get scripts rootdir by FindBin 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1745 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-16 15:54:02 +00:00
hieuhoang1972
a2a3d33103 explicitly use bash 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1693 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-15 08:50:22 +00:00
hieuhoang1972
3fc0b8ddb4 git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1603 1f5c12ca-751b-0410-a591-d2e778427230 2008-05-04 12:52:52 +00:00
hieuhoang1972
a822d61d8f prevent -inf in lex re-ordering. Code contributed by Christian Hardmeier 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1596 1f5c12ca-751b-0410-a591-d2e778427230
 2008-04-18 09:04:38 +00:00
nicolabertoldi
1aff3d2382 correct handling of binary phrase tables 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1579 1f5c12ca-751b-0410-a591-d2e778427230
 2008-02-28 14:33:15 +00:00
nicolabertoldi
def0fff5cd changes to handle lattice input format 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1578 1f5c12ca-751b-0410-a591-d2e778427230
 2008-02-28 08:55:40 +00:00
hieuhoang1972
0bb92c2e79 merge properly 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1577 1f5c12ca-751b-0410-a591-d2e778427230
 2008-02-27 19:01:38 +00:00
hieuhoang1972
cb1f0e56dc optional output what lines are retained 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1576 1f5c12ca-751b-0410-a591-d2e778427230
 2008-02-27 18:38:31 +00:00
bojar
f056bdbfde fixed to correctly handle models in [distortion-file] section 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1572 1f5c12ca-751b-0410-a591-d2e778427230
 2008-02-26 10:24:53 +00:00
bojar
3957dc6b4c default to reordering factors of 0-0 even if decoding steps are set (users 
might have explicitly said e.g. t0-0!)


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1571 1f5c12ca-751b-0410-a591-d2e778427230
 2008-02-26 10:20:09 +00:00
bojar
f7a1fb5b9c corpus compression correctly used even for generation step 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1568 1f5c12ca-751b-0410-a591-d2e778427230
 2008-02-22 16:14:30 +00:00
bojar
8b3d44b2e2 SAFE_GETLINE made safer: will exit if the line does not fit into the buffer 
instead of just going on and getting the src/tgt/alignment files out of sync


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1565 1f5c12ca-751b-0410-a591-d2e778427230
 2008-02-22 14:42:01 +00:00
redpony
25750c6555 if giza returns sentences that have different lengths in different directions (due to truncation or other errors), don't silenty fail. print a blank line instead. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1562 1f5c12ca-751b-0410-a591-d2e778427230
 2008-02-19 20:48:14 +00:00
bojar
fa31d83421 even factors that are being added can be gzipped 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1561 1f5c12ca-751b-0410-a591-d2e778427230
 2008-02-19 17:32:51 +00:00
bojar
eec1bdb623 added support to open gzipped files 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1560 1f5c12ca-751b-0410-a591-d2e778427230
 2008-02-19 16:05:11 +00:00
nicolabertoldi
ae319da62b revert to /bin/sh for enhanced-mert; use of setenv (instead of export) in the csh scripts created by qsub-wrapper.pl 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1515 1f5c12ca-751b-0410-a591-d2e778427230
 2007-11-21 14:31:21 +00:00
nicolabertoldi
0176c5f8ec use fo csh instead of sh 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1514 1f5c12ca-751b-0410-a591-d2e778427230
 2007-11-21 07:59:17 +00:00
bojar
89ea9828ba added ttable iterator to this script, too 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1498 1f5c12ca-751b-0410-a591-d2e778427230
 2007-11-06 03:33:41 +00:00
nicolabertoldi
568f92b310 bug fixed 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1491 1f5c12ca-751b-0410-a591-d2e778427230
 2007-10-30 15:42:26 +00:00
jdschroeder
e52040bc12 added str length check to stop std::out_of_range error in a few more spots - similar bug to one corrected in v. 1319 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1488 1f5c12ca-751b-0410-a591-d2e778427230
 2007-10-26 12:42:55 +00:00
nicolabertoldi
fd3ecd4334 bug fixed 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1487 1f5c12ca-751b-0410-a591-d2e778427230
 2007-10-25 14:34:49 +00:00
nicolabertoldi
1b0576ba6c small bug fixed: temporary concatenated sorted file is now deleted only at the end 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1486 1f5c12ca-751b-0410-a591-d2e778427230
 2007-10-24 07:50:51 +00:00
nicolabertoldi
8710cc9bc9 features can be activated using a comma- or blank-separated list 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1485 1f5c12ca-751b-0410-a591-d2e778427230
 2007-10-23 16:55:02 +00:00
nicolabertoldi
9e70b5ffd8 Features are activated using their names 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1484 1f5c12ca-751b-0410-a591-d2e778427230
 2007-10-23 16:30:02 +00:00
nicolabertoldi
8fe62f2b95 some small bugs fixed and clean up 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1483 1f5c12ca-751b-0410-a591-d2e778427230
 2007-10-23 14:15:19 +00:00
nicolabertoldi
4720d1cb9f bug fixed 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1482 1f5c12ca-751b-0410-a591-d2e778427230
 2007-10-22 17:01:53 +00:00
nicolabertoldi
918dae011a bug fixed 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1481 1f5c12ca-751b-0410-a591-d2e778427230
 2007-10-22 14:00:10 +00:00
nicolabertoldi
e7ac20d4d6 bug fixed in the name of a temporary file 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1480 1f5c12ca-751b-0410-a591-d2e778427230
 2007-10-22 13:34:12 +00:00
nicolabertoldi
db9d0fc539 Added a more time-efficient (but more memory-consumptive) method to rescore nbest list 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1479 1f5c12ca-751b-0410-a591-d2e778427230
 2007-10-22 10:08:07 +00:00
nicolabertoldi
b827d51870 changes to cope with the new mert suite (enhanced-mert) 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1478 1f5c12ca-751b-0410-a591-d2e778427230
 2007-10-22 09:45:19 +00:00
jdschroeder
a969197e16 Fixed passing decoder parameters when tuning on single machine. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1477 1f5c12ca-751b-0410-a591-d2e778427230
 2007-10-22 09:27:09 +00:00
nicolabertoldi
5759005857 Suite of scripts to perform MERT on a subset of fetures. 
Look at the directory example to learn about its use.


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1476 1f5c12ca-751b-0410-a591-d2e778427230
 2007-10-22 09:04:03 +00:00
redpony
0cf583e249 add --hmm-align option. Allows using Giza++'s HMM word alignment model as the underlying word alignment. It is much faster than Model 4 alignment and not much worse. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1474 1f5c12ca-751b-0410-a591-d2e778427230
 2007-10-19 03:44:05 +00:00
nicolabertoldi
901823d83a explicit export of PYTHONPATH variable 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1473 1f5c12ca-751b-0410-a591-d2e778427230
 2007-10-18 15:33:04 +00:00
nicolabertoldi
81b439d728 minor changes in passing parameters to moses-parallel 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1472 1f5c12ca-751b-0410-a591-d2e778427230
 2007-10-18 12:36:49 +00:00
hieuhoang1972
4e1cad4bbe fixed sync/async bug 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1471 1f5c12ca-751b-0410-a591-d2e778427230
 2007-10-04 17:31:31 +00:00
redpony
57dcaa8e80 performance fixes for scorer 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1470 1f5c12ca-751b-0410-a591-d2e778427230
 2007-10-02 21:43:54 +00:00
hieuhoang1972
9cbc2922b4 separate word penalty for each decode step for async 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1469 1f5c12ca-751b-0410-a591-d2e778427230
 2007-10-02 12:52:00 +00:00
hieuhoang1972
d2d03c33e7 fixed bug which prevented mert working when phrase table NOT filtered or binarised 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1449 1f5c12ca-751b-0410-a591-d2e778427230
 2007-08-10 15:48:58 +00:00
hieuhoang1972
53fa2cb18a async - don't use binarising or filtering 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1443 1f5c12ca-751b-0410-a591-d2e778427230
 2007-08-05 20:29:46 +00:00
hieuhoang1972
9eba034662 turn off debugging 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1435 1f5c12ca-751b-0410-a591-d2e778427230
 2007-07-25 10:05:34 +00:00
hieuhoang1972
2beb0c44e9 mkdir before doing generation 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1425 1f5c12ca-751b-0410-a591-d2e778427230
 2007-07-14 09:54:05 +00:00
nicolabertoldi
75afdf04a5 I corrected direction of alignment 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1421 1f5c12ca-751b-0410-a591-d2e778427230
 2007-06-27 17:57:55 +00:00
nicolabertoldi
ac91cb78cc two additional (and simpler) ways of extracting alignments: source-to-target and target-to-source 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1420 1f5c12ca-751b-0410-a591-d2e778427230
 2007-06-27 16:34:14 +00:00
nicolabertoldi
7f9c2856c2 changes to reduce disk memory consumption during training 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1419 1f5c12ca-751b-0410-a591-d2e778427230
 2007-06-27 16:30:20 +00:00
phkoehn
960bebdd4a fixed clean script to handle '|'s 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1416 1f5c12ca-751b-0410-a591-d2e778427230
 2007-06-18 15:50:04 +00:00
redpony
c747cdd505 fix dumb error 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1414 1f5c12ca-751b-0410-a591-d2e778427230
 2007-06-01 16:27:56 +00:00
redpony
1f050e198a fix compile error, enable optimizations 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1413 1f5c12ca-751b-0410-a591-d2e778427230
 2007-06-01 16:26:26 +00:00
redpony
564bb5a64e make scorer use compiler optimization 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1412 1f5c12ca-751b-0410-a591-d2e778427230
 2007-06-01 16:22:30 +00:00
hieuhoang1972
aa25c7341d fixed bug with non-ascii data, recieved from Jaakko Väyrynen 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1392 1f5c12ca-751b-0410-a591-d2e778427230
 2007-05-21 13:06:40 +00:00
bojar
74954cb0ae prefer hardlinking, dropped dependency on a proprietary script 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1381 1f5c12ca-751b-0410-a591-d2e778427230
 2007-05-09 00:54:07 +00:00
bojar
31def05428 - added a comment where the binarizer is 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1380 1f5c12ca-751b-0410-a591-d2e778427230
 2007-05-07 07:19:02 +00:00
abarun
ba90a05233 Added script to perform Minimum Bayes Risk reranking 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1372 1f5c12ca-751b-0410-a591-d2e778427230
 2007-05-02 17:26:01 +00:00
hieuhoang1972
13e07cef5f multiple distance based distortion for async decoder 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1370 1f5c12ca-751b-0410-a591-d2e778427230
 2007-04-23 19:17:38 +00:00
redpony
485bda2db5 andreas zollman's changes to write span information 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1367 1f5c12ca-751b-0410-a591-d2e778427230
 2007-04-20 14:53:46 +00:00
jdschroeder
752d148c6e Changed initial setting of number of distortion weights from 0 to 1. For models with lexicalized reordering, this script was generating one too few weights in moses.ini 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1365 1f5c12ca-751b-0410-a591-d2e778427230
 2007-04-18 19:38:14 +00:00
hieuhoang1972
45dde20c54 comment out psyco library 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1354 1f5c12ca-751b-0410-a591-d2e778427230
 2007-04-12 17:48:48 +00:00
hieuhoang1972
75c20e7609 Add alignment info to phrase table 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1352 1f5c12ca-751b-0410-a591-d2e778427230
 2007-04-11 19:58:38 +00:00
hieuhoang1972
b84191c9d3 Add alignment info to phrase table 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1351 1f5c12ca-751b-0410-a591-d2e778427230
 2007-04-11 19:56:43 +00:00
hieuhoang1972
b9d2288c22 compileable with visual studio 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1349 1f5c12ca-751b-0410-a591-d2e778427230
 2007-04-11 11:35:36 +00:00
hieuhoang1972
e868986885 compileable with visual studio 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1348 1f5c12ca-751b-0410-a591-d2e778427230
 2007-04-11 11:18:37 +00:00
jdschroeder
84d12552c2 added additional numbered count to output phrase table names so multiple phrase tables can be filtered and used at the same time 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1337 1f5c12ca-751b-0410-a591-d2e778427230
 2007-04-04 17:54:12 +00:00
jdschroeder
667e85264a added LC_ALL=C call and temp directory specification to sort command, hoping to minimize failed sorts crashing processPhraseTable 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1336 1f5c12ca-751b-0410-a591-d2e778427230
 2007-04-04 17:07:16 +00:00
hieuhoang1972
c17316495c use pawd instead of pwd whenever we can 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1332 1f5c12ca-751b-0410-a591-d2e778427230
 2007-03-26 21:00:20 +00:00
bojar
e7821e84ef Added support for Ondrej Bojar's scoring of nbestlist, faster than python and 
does not rescore previous iterations. The scoring tool is however not included
in the scripts distribution.


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1329 1f5c12ca-751b-0410-a591-d2e778427230
 2007-03-26 09:26:05 +00:00
bojar
c7b78dfc2a Automagically fix moses.ini if it points to a non-gzipped version of a file but 
only the gzipped exists.


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1324 1f5c12ca-751b-0410-a591-d2e778427230
 2007-03-25 22:55:31 +00:00
bojar
9fe45b747a Not everyone has 'pawd'. Using pawd as the default but reverting to pwd to get 
the current directory.


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1323 1f5c12ca-751b-0410-a591-d2e778427230
 2007-03-25 05:46:05 +00:00
lexi_birch
d20ae825f8 Alternate graph code: 
Now can specify an alternate path in the config file for experiment.perl
All you need to do is change the line: decoding-steps
Every : you add will mean that the next steps are in an alternate graph
Ie. 
	decoding-steps = " t0 , g1 , t1 "
will produce a single graph but
	decoding-steps = " t0 , g1 : t1 "
will produce two alternate graphs

The only change it actually makes is to the create configuration file.

This is also a bugfix!
There was no ability to have multiple factors in the target
for the two filering scripts. They just ignored those lines
as if they were not valid table descriptions.




git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1322 1f5c12ca-751b-0410-a591-d2e778427230
 2007-03-22 12:42:23 +00:00
bojar
d4b1103bd7 Fixed the case when the intersection of two alignments is empty. Used to throw 
std::out_of_range at basic_string::replace, now emits an empty line.


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1320 1f5c12ca-751b-0410-a591-d2e778427230
 2007-03-21 00:52:28 +00:00
bojar
2f210366f6 Flushing stdout for intersect and union alignments. 
(No explicit flushing leads to truncated output! C++ is crap!)
Also report the number of sentences processed.


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1319 1f5c12ca-751b-0410-a591-d2e778427230
 2007-03-21 00:31:53 +00:00
maurocettolo
92788cadb5 added -V option to qsub call in qsub-wrapper.pl. This allows to make 
available to the submitted job the value of variable PYTHONPATH, set to
$pythonpath in mert-moses.pl through %ENV (no more need of using
setenv/export)



git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1317 1f5c12ca-751b-0410-a591-d2e778427230
 2007-03-16 13:20:27 +00:00
nicolabertoldi
9912815c53 csh requires setenv instead of export. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1316 1f5c12ca-751b-0410-a591-d2e778427230
 2007-03-15 18:00:57 +00:00
nicolabertoldi
ffa520a2a9 Robust (forced and not interactive) copy, move and remove of files. 
Add usage infos.


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1315 1f5c12ca-751b-0410-a591-d2e778427230
 2007-03-15 17:54:00 +00:00
hieuhoang1972
4b0ea463c8 add svn id comments to start of file 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1308 1f5c12ca-751b-0410-a591-d2e778427230
 2007-03-14 22:30:25 +00:00
hieuhoang1972
3c07c5df4d add svn id comments to start of file 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1307 1f5c12ca-751b-0410-a591-d2e778427230
 2007-03-14 22:22:36 +00:00
jorcisai
a83fe593c2 startup ini file is backed up in order to preserve the pathnames to the original models, so that the final moses.ini file doesn't point to the filtered models 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1304 1f5c12ca-751b-0410-a591-d2e778427230
 2007-03-14 18:31:53 +00:00
hieuhoang1972
71833f3bee merge from hieu-async branch 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1299 1f5c12ca-751b-0410-a591-d2e778427230
 2007-03-13 23:03:53 +00:00
bojar
87b168cd1b Handles gzipped input. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1296 1f5c12ca-751b-0410-a591-d2e778427230
 2007-03-12 02:24:34 +00:00
phkoehn
41ee7f69a2 adapted mert to work with multiple decoding paths 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1293 1f5c12ca-751b-0410-a591-d2e778427230
 2007-03-09 17:58:05 +00:00
maurocettolo
5439a7796d Fixed a minor bug in mert-moses.pl regarding sanity checks for specified lambda triples 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1254 1f5c12ca-751b-0410-a591-d2e778427230
 2007-03-01 13:02:22 +00:00
jorcisai
ad28cee802 distortion filename was incorrectly written into moses.ini file in step 9 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1243 1f5c12ca-751b-0410-a591-d2e778427230
 2007-02-26 14:36:53 +00:00
jorcisai
d5b4565f23 language model parser for --lm option is now again able to parse $type, but it is backward compatible 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1238 1f5c12ca-751b-0410-a591-d2e778427230
 2007-02-23 17:32:51 +00:00
jorcisai
c69bd4079b reordering model was left in the local directory instead of model directory 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1236 1f5c12ca-751b-0410-a591-d2e778427230
 2007-02-23 12:27:50 +00:00
jorcisai
872f2d3612 Trying to parse $type in --lm option, but not available. So we just need to parse three tokens. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1235 1f5c12ca-751b-0410-a591-d2e778427230
 2007-02-23 11:49:54 +00:00
phkoehn
6c5cb3a6ec changes to fit with edinburgh setup, added switch -generation-type: "single" only produces one probability, not both 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1231 1f5c12ca-751b-0410-a591-d2e778427230
 2007-02-22 19:37:11 +00:00
phkoehn
9f227aa26b minor bug fix with config file 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1224 1f5c12ca-751b-0410-a591-d2e778427230
 2007-02-19 18:35:51 +00:00
phkoehn
14839768c8 a large number of changes. besides little tweaks: 
* training script now has proper default behaviour for single-factor models, 
* mert script has better handling of default lambda parameters that now
  works with lexicalized reordering models, and also with multiple 
  models files (e.g. multiple language models)
* parallel mert script is more robust when single jobs fail: detects it
  and resubmits the crashed (or killed) jobs
* recaser added that builds on moses
* filtering script added that also binarizes filtered model files
  (this will be eventually replaced when the lexicalized reordering
  model also uses the binary format)


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1210 1f5c12ca-751b-0410-a591-d2e778427230
 2007-02-13 19:22:35 +00:00
hieuhoang1972
970af347e4 Bug fixed for distortion model proposed by Tim Murray 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1111 1f5c12ca-751b-0410-a591-d2e778427230
 2007-01-07 12:36:18 +00:00
hieuhoang1972
0aba61ca8b don't insist on using python 2.3 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1104 1f5c12ca-751b-0410-a591-d2e778427230
 2007-01-02 12:54:00 +00:00
lexi_birch
93937b529d Making remaining scripts os independent re pawd/pwd 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1103 1f5c12ca-751b-0410-a591-d2e778427230
 2006-12-29 13:45:21 +00:00
nicolabertoldi
26aff6ead9 managing of pwd/pawd 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1101 1f5c12ca-751b-0410-a591-d2e778427230
 2006-12-29 13:19:21 +00:00
lexi_birch
dee506806f Fix for mount bug using pwd on terabyte 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1096 1f5c12ca-751b-0410-a591-d2e778427230
 2006-12-28 22:17:06 +00:00
hieuhoang1972
e701f57f07 halcion days of the jhu workshop are over and grim reality has taken hold. 
default qsub not to use workshop specific queue

git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1079 1f5c12ca-751b-0410-a591-d2e778427230
 2006-12-16 22:33:55 +00:00
bojar
72ff1f8450 added yet another combiner for factored corpora 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1026 1f5c12ca-751b-0410-a591-d2e778427230
 2006-11-30 06:17:45 +00:00
bojar
412f04737c allows reducing factors from stdin 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1025 1f5c12ca-751b-0410-a591-d2e778427230
 2006-11-30 03:46:21 +00:00
phkoehn
0a088dbb38 fixed error in filtering for lexicalized reordering tables 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@998 1f5c12ca-751b-0410-a591-d2e778427230
 2006-11-22 15:50:20 +00:00
phkoehn
28ca9b57fd minor bug fixes for training and using lexicalized reordering 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@978 1f5c12ca-751b-0410-a591-d2e778427230
 2006-11-15 17:04:19 +00:00
bojar
e2518b7799 - support for sun grid engine prior to v.6.0 in qsubwrapper and mert-moses 
- changed temporary scripts to csh (because my sge runs them in csh regardless of my wishes)
- added a two tests + sample data for the full chain: train-mert-decode-eval
  (a parallel and a serial version)
- cleanup of other tests
- Makefile rules for running single tests in foreground or background


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@899 1f5c12ca-751b-0410-a591-d2e778427230
 2006-10-18 10:36:39 +00:00
nicolabertoldi
605e47d978 psyco library is maintained only for 386-compatible processors. 
I modified score-nbest.py to import psyco only if $MACHTYPE is equal to "i386"
If MACHTYPE does not matchpsyco library is not imported,
but script works properly.
I do not know if the control is efffective under Windows


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@879 1f5c12ca-751b-0410-a591-d2e778427230
 2006-10-12 09:49:09 +00:00
nicolabertoldi
24d3ed8a37 Changes to mert-moses.pl: 
- added a flag (--no-filter-phrase-table) to disallow filtering of phrase tables (useful if binary hrase tables are used)
- added a flag to compute bleu score without text normalization (--nonorm) (default is with normalization)
- added a flag to compute bleu score with the "closest reference length" (--closest), which is 
   alternative to "average reference length" (--average) or "shortest reference length" (default)
- added a parameter (--inputype=[0|1]) to manage different input types (0 for text, 1 for confusion network, default is 0)
Changes to moses-parallel.pl:
- corrected a typos


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@878 1f5c12ca-751b-0410-a591-d2e778427230
 2006-10-11 16:58:30 +00:00
bojar
d90b1d348e reuses lexical translation 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@876 1f5c12ca-751b-0410-a591-d2e778427230
 2006-10-10 13:53:45 +00:00
bojar
2eb05906aa skips giza if older output reusable 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@872 1f5c12ca-751b-0410-a591-d2e778427230
 2006-10-09 15:01:27 +00:00
bojar
998a8216ba skips mkcls and some other steps, if already finished 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@870 1f5c12ca-751b-0410-a591-d2e778427230
 2006-10-06 21:19:29 +00:00
nicolabertoldi
a73c412b88 added clean to some Makefiles 
use of "make clean" in scripts/Makefile



git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@867 1f5c12ca-751b-0410-a591-d2e778427230
 2006-10-06 15:07:30 +00:00
bojar
c8f5e2aeba fixed an error message 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@866 1f5c12ca-751b-0410-a591-d2e778427230
 2006-10-06 10:47:16 +00:00
phkoehn
a71f247596 bugfix: option rootdir misnamed roodir 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@835 1f5c12ca-751b-0410-a591-d2e778427230
 2006-09-28 19:25:08 +00:00
mfederico
ef42ad791e symal.cpp: just a minor change 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@833 1f5c12ca-751b-0410-a591-d2e778427230
 2006-09-28 16:14:21 +00:00
bojar
271b78d94c Just checking if I can commit. Added my name. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@820 1f5c12ca-751b-0410-a591-d2e778427230
 2006-09-20 14:20:17 +00:00
redpony
9582bcecff turn on O3 optimization for symal 
increase MAX_WORD in symal.cpp (I was hitting this limit in a chinese corpus that had some tokenization errors)



git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@816 1f5c12ca-751b-0410-a591-d2e778427230
 2006-09-18 15:36:04 +00:00
redpony
da7fed9e7e add --corpus-compression [gz|bz2] to allow corpora to be compressed 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@814 1f5c12ca-751b-0410-a591-d2e778427230
 2006-09-15 12:38:13 +00:00
redpony
7d50d155dc fix compilation error on gcc 4.1, fix warnings in mert 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@813 1f5c12ca-751b-0410-a591-d2e778427230
 2006-09-12 19:46:16 +00:00
redpony
c69cfaf33e Allow the factor delimiter, that is, the string that separates the factors in a 'word' to be specified to moses and to train-factored-phrase-model.perl. The default is still to use '|'. Multi-character delimiters are allowed (for example, '+++'). Added a regression test for multi-character delimiters. 
Remove JHU dependencies on make release.  It now looks for GIZA++ and sets the BINDIR inside train-factored-phrase-model.perl at installation time (note: because of this, this script MUST BE released before it can be run now).



git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@812 1f5c12ca-751b-0410-a591-d2e778427230
 2006-09-12 15:53:50 +00:00
phkoehn
572c577ef7 initial release 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@806 1f5c12ca-751b-0410-a591-d2e778427230
 2006-09-01 18:02:56 +00:00
nicolabertoldi
041e6ed3c5 Changes to compilation scripts: 
- irstlm/src/Makefile.am did not install some files
- irstlm/mkinstalldirs needed by OSX
- irstlm/regenerate-makefiles.sh substitutes 
  explicit calls of aclocal, autoconf and automake

Changes to scoring script used by MERT
- added the option ("-e") to compute BLEU wrt the
  "closest" reference length like in multi-bleu.perl
- now multi-bleu.perl manages 0 counts for ngram-statistics




git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@805 1f5c12ca-751b-0410-a591-d2e778427230
 2006-09-01 14:54:41 +00:00
bojar
53bbbbfa22 --continue now also attempts to step one extra step back if necessary moses output is not found 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@754 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-15 15:16:28 +00:00
bojar
568cff8e34 fixed serious stupid bug: value ranges were ignored and min. and max were set to the starting value 
this bug occurred only if lambdas were supplied on command line, not with the default lambdas and ranges


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@753 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-15 15:04:19 +00:00
bojar
5c2d19a156 reversed exit codes of symal and added safesystem to call symal 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@730 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-14 17:54:11 +00:00
bojar
7735bc6b6d the python compiled files should not be in the cvs 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@729 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-14 17:53:23 +00:00
mfederico
f211a2a738 New version with c++ module (symal) performing step (3). 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@728 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-14 17:43:17 +00:00
mfederico
6d6ac5c1e4 New version with faster computation of word alignments. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@718 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-14 15:18:12 +00:00
mfederico
c3ea1ef545 Filter to make GIZA++ alignment files more readable. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@717 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-14 15:06:58 +00:00
mfederico
e72010d6ce A tool to compute symmetric alignments from GIZA++ alignments. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@716 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-14 15:03:49 +00:00
bojar
840441dc1a die if phrase mismatch discovered 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@688 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-13 05:35:02 +00:00
bojar
6fc349f75f gives nice overview of model complexity (in terms of ambiguity in translation and generation tables) 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@670 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-11 22:33:48 +00:00
bojar
e1936af681 marking finished_step also after last iteration finished 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@655 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-11 17:05:04 +00:00
bojar
68ef1413cd allows arbitrary mixing of 'kept' and 'added' factors in output 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@627 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-10 22:00:02 +00:00
bojar
9b23b6d9c8 die in safesystem on child's death 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@612 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-10 01:42:49 +00:00
bojar
af1be61259 die when there are no phrases in input 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@611 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-10 01:37:29 +00:00
swadey
683435e058 - updated bleu and score-nbest to allow optional bypass of NIST-style normalization 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@608 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-10 00:24:19 +00:00
phkoehn
0595062d7d fixed error message on scripts root dir 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@607 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-09 23:52:50 +00:00
redpony
0ea85deef7 fix off-by-one error in tables-score (prevents null characters from being inserted) 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@595 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-09 19:37:14 +00:00
bojar
e97b542717 added --debug mode to training script to keep all intermediate files 
exit status of extract and score are 1 on error, not zero


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@585 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-08 22:28:26 +00:00
redpony
523527fa17 get rid of profiling 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@573 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-08 20:16:06 +00:00
redpony
db5a6bd11e fix bug that prevents | and _ from being tokenized properly. 
fix bug in --parallel


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@572 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-08 20:13:57 +00:00
bojar
81ddb0e4f9 added train-factored... to releases, added dependency on our copy of phrase-extract 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@569 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-08 19:22:53 +00:00
bojar
303f411387 simplified Makefile, removed duplicit implementation of tokenize() 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@568 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-08 19:04:59 +00:00
phkoehn
b83fc72dd2 initial version of phrase-extract and phrase-score used by training script 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@567 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-08 18:54:28 +00:00
bojar
264f045a6b fixing ensure_absolute 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@556 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-08 03:41:50 +00:00
bojar
ab5bb31797 allowing to override default paths 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@547 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-07 23:04:33 +00:00
bojar
26ce21f29b fixed unintended structure-sharing bug 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@541 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-07 22:17:02 +00:00
bojar
a41a4e95d6 now expects 3 numbers on [generation-file] lines before the pathname 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@538 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-07 21:55:06 +00:00
bojar
2d7cf749a6 Allowing scores in 'scientific' float format from moses. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@514 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-05 00:53:48 +00:00
bojar
10a0e23801 checking in reduce_combine 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@510 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-05 00:11:57 +00:00
bojar
a5c122dfc8 added mert to list of released files, make rules to release moses (personally or publicly) 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@505 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-04 22:25:38 +00:00
bojar
12f50a5f26 Added labelling of scores in nbestlist and fixed mert to understand that. 
Before release, these have to be checked:
- train-factored-phrase-model.perl (the whole process)
- mert on newly generated moses.ini with 2 weights for generation


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@492 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-04 04:45:48 +00:00
redpony
7d0e0f5698 fix 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@491 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-03 21:10:18 +00:00
redpony
7b11b66b6d enable --parallel in tfpm.perl 
add a script to build a generation table from a monolingual corpus.
add a script to post-process the german morpho tagger output


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@490 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-03 21:05:55 +00:00
bojar
18dac34fe2 checking in the current version of cmert we're using 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@489 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-03 20:00:00 +00:00
bojar
232727e0e4 removed the dependence on external lowercaser, lowercasing internally 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@488 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-03 18:07:10 +00:00
bojar
4d49e12bc4 checking the latest version from /export/bin to cvs 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@486 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-03 17:53:13 +00:00
bojar
32e73c3785 yet another clarification of messages 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@467 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-02 20:06:04 +00:00
bojar
60f9301ab7 Fixed matching of lambdas. (Back to the hardwired order.) 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@457 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-01 22:14:12 +00:00
phkoehn
63a86828ba Added setting "distortion-limit=6" to moses.ini 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@455 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-01 21:11:17 +00:00
bojar
9304d71469 improved passing (and checking) of command-line options 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@446 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-01 15:18:49 +00:00
callison-burch
c0968b9041 Updated the script so that it correctly passes the qflags argument along to the qsub_wrapper script. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@445 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-01 14:29:08 +00:00
bojar
e59035efca Default to use only our team's queue. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@437 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-01 02:43:36 +00:00
bojar
5f3965de12 various tiny bugfixes 
added basic testcases
moved qsub-wrapper to generic


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@434 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-01 01:18:13 +00:00
nicolabertoldi
7910b65cdf Check in generic/moses-parallel.pl 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@431 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-31 23:01:02 +00:00
phkoehn
6f80f8c12a Speed-up of lexical translation table training, old code was crap 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@429 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-31 22:10:34 +00:00
bojar
75a5f9e935 clearer error message 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@425 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-31 21:33:54 +00:00
bojar
540aadea2b Allowing to optimize unknown lambdas, release methodology 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@421 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-31 21:01:07 +00:00
bojar
1c2cd47881 checking in the current version of clone_moses_model 
working on a single scripts directory


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@416 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-31 20:13:25 +00:00
nicolabertoldi
ba76013a5c *** empty log message *** 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@411 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-31 18:26:44 +00:00
bojar
a325df6380 renamed pythonpath variable, correctly passing --jobs, checking for blank moses.ini 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@410 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-31 17:00:22 +00:00
bojar
51ad454a39 checking in this useful script 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@409 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-31 16:53:40 +00:00
bojar
57bcad0c5f the cleanup of mert-moses seems to be finished 
added first simple 'make release' goal


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@405 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-31 14:17:43 +00:00
bojar
54c6554d09 Removed the 'run-moses' functionality, so that the script is now usable by various variants of moses. (parallel and non parallel, mainly, but also by mert-moses.pl and others). 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@378 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-29 01:47:33 +00:00
bojar
9f4178e36e Added 
-rwxrwxr-x  1 pkoehn ws06osmt 5769 Jul 19 15:47 run-filtered-moses.perl
under a new name. Just for diffing purposes.


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@377 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-29 01:45:14 +00:00
bojar
46ea34ab20 Merged the parallel and non-parallel copies of this script. 
Changed the command line and added some options.
Added extensive checking of validity of input files and options.
Still not ready for deployment due to the following bugs:
- the generation of output moses.ini was not tested
- the --start-step option does not work (not critical)


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@376 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-29 01:41:52 +00:00
bojar
05b0a07892 Checking in the last version of 
-rwxr-xr-x  1 nbertoldi ws06osmt 12430 Jul 28 00:01 mert-moses-parallel.perl-2006-07-27
Just for diff purposes.


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@374 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-29 01:35:46 +00:00
bojar
6272fa6ecf Added the change in giza default options as done by ccb. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@373 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-29 01:18:58 +00:00
bojar
9061c682eb Checking in the version: 
-rwxrwxr-x  29 obojar ws06osmt 51861 Jul 24 18:15 train-factored-phrase-model.perl


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@372 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-29 01:13:30 +00:00