moses-smt/mosesdecoder
1
1
Fork 0
mirror of https://github.com/moses-smt/mosesdecoder.git synced 2024-09-20 15:48:05 +03:00
Code Issues Packages Projects Releases Wiki Activity
1,837 Commits 136 Branches 26 Tags 147 MiB
05eb33d5ac
Commit Graph

424 Commits

Author SHA1 Message Date
bojar
55e3ee4a30 just setting the executable bit 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2795 1f5c12ca-751b-0410-a591-d2e778427230
 2010-01-29 19:49:37 +00:00
bojar
2097e45edd a handy script for calculating out-of-vocabulary rate of n-grams 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2794 1f5c12ca-751b-0410-a591-d2e778427230
 2010-01-29 19:48:29 +00:00
naditomeh
ad3b0760b2 adding extract.cpp 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/branches/hierarchical-reo@2770 1f5c12ca-751b-0410-a591-d2e778427230
 2010-01-29 13:59:04 +00:00
naditomeh
03de8a99d8 adding extract.cpp 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/branches/hierarchical-reo@2769 1f5c12ca-751b-0410-a591-d2e778427230
 2010-01-29 13:41:34 +00:00
sarst
4eec020d5b bugfixes to train-factored-phrase-model.perl 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/branches/hierarchical-reo@2764 1f5c12ca-751b-0410-a591-d2e778427230
 2010-01-29 12:11:10 +00:00
sarst
a9ef19edf0 updated train-factored-phrase-model.perl to work with the new hierarchical reordering framework 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/branches/hierarchical-reo@2759 1f5c12ca-751b-0410-a591-d2e778427230
 2010-01-29 11:29:39 +00:00
bojar
9f784c6bf8 a handy script to get many translations from Google (can continue interrupted 
sessions)


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2744 1f5c12ca-751b-0410-a591-d2e778427230
 2010-01-29 01:48:13 +00:00
bojar
ed18df8dc7 allow env.var to override BINDIR and TARGETDIR 
exclude memscore from the released things if failed to compile


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2740 1f5c12ca-751b-0410-a591-d2e778427230
 2010-01-29 00:51:28 +00:00
phkoehn
4d814e53d2 del 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2728 1f5c12ca-751b-0410-a591-d2e778427230
 2010-01-28 17:38:26 +00:00
phkoehn
f1f395e05d added web interface, organized files in sub directories 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2724 1f5c12ca-751b-0410-a591-d2e778427230
 2010-01-28 17:24:39 +00:00
hieuhoang1972
53e54def0b indent 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2723 1f5c12ca-751b-0410-a591-d2e778427230
 2010-01-28 17:17:19 +00:00
hieuhoang1972
f5ebdbcec8 xcode 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2721 1f5c12ca-751b-0410-a591-d2e778427230
 2010-01-28 16:43:36 +00:00
phkoehn
244e334b3d bug fix 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2690 1f5c12ca-751b-0410-a591-d2e778427230
 2010-01-27 14:55:17 +00:00
bojar
0889b9efff renaming .pl -> .perl 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2674 1f5c12ca-751b-0410-a591-d2e778427230
 2010-01-26 17:23:41 +00:00
bojar
0e26f91865 don't organize to stacks by default, accept --organize-to-stacks 
read from stdin as well


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2673 1f5c12ca-751b-0410-a591-d2e778427230
 2010-01-26 17:20:28 +00:00
bojar
536c7bdbcc commiting a script by Loic Barrault to display moses search graph 
(-output-search-graph) using graphviz dot


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2672 1f5c12ca-751b-0410-a591-d2e778427230
 2010-01-26 17:01:12 +00:00
phkoehn
def35604af initial release of experiment.perl 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2669 1f5c12ca-751b-0410-a591-d2e778427230
 2010-01-25 17:38:53 +00:00
chardmeier
6317633148 Added memscore phrase scorer. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2653 1f5c12ca-751b-0410-a591-d2e778427230
 2010-01-18 14:52:34 +00:00
nicolabertoldi
3ad833d136 program to compute countings for phrase pairs 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2647 1f5c12ca-751b-0410-a591-d2e778427230
 2010-01-08 17:16:37 +00:00
nicolabertoldi
34d9feccc8 now it is possible to perform mert on a subset of features 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2646 1f5c12ca-751b-0410-a591-d2e778427230
 2010-01-08 15:56:45 +00:00
mphi
850e54f17d added a switch to the training script, which allows using different word alignment models: --final-alignment-model X, where X is either 1/2/3/4/5 (for the ibm models 1 to 5) or hmm; the latter is equivalent to using --hmm-align. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2644 1f5c12ca-751b-0410-a591-d2e778427230
 2010-01-08 12:00:28 +00:00
bhaddow
d8864fa6ad support for mgiza 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2640 1f5c12ca-751b-0410-a591-d2e778427230
 2009-12-15 17:36:06 +00:00
eisele
12dda84589 test, please ignore 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2598 1f5c12ca-751b-0410-a591-d2e778427230
 2009-10-23 15:45:35 +00:00
nicolabertoldi
5ad52827e3 minor change 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2576 1f5c12ca-751b-0410-a591-d2e778427230
 2009-10-09 07:47:14 +00:00
nicolabertoldi
427d421cf9 small change to be compliant with the previous change (2571->2572) 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2573 1f5c12ca-751b-0410-a591-d2e778427230
 2009-10-07 18:09:20 +00:00
nicolabertoldi
7b77734e3c the ordered list of features names are now stored in a file after each step and re-load in the case of re-starting of the training 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2572 1f5c12ca-751b-0410-a591-d2e778427230
 2009-10-07 18:03:41 +00:00
nicolabertoldi
6d45d03f48 removed local references 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2571 1f5c12ca-751b-0410-a591-d2e778427230
 2009-10-07 17:34:34 +00:00
nicolabertoldi
3731f83b8d minor changes 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2570 1f5c12ca-751b-0410-a591-d2e778427230
 2009-10-07 16:43:07 +00:00
nicolabertoldi
98387244c1 added a new regression test for --continue option of mert-moses-new.pl 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2569 1f5c12ca-751b-0410-a591-d2e778427230
 2009-10-07 16:37:25 +00:00
nicolabertoldi
124f88e55a enabled the --continue option to re-start an interrupted mert from the last finished step 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2568 1f5c12ca-751b-0410-a591-d2e778427230
 2009-10-07 16:35:57 +00:00
nicolabertoldi
e25b8c41b7 minor change 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2567 1f5c12ca-751b-0410-a591-d2e778427230
 2009-10-07 16:33:32 +00:00
nicolabertoldi
40f9b00bab use of compressed data 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2566 1f5c12ca-751b-0410-a591-d2e778427230
 2009-10-07 12:47:40 +00:00
nicolabertoldi
2d1e4697f2 adding a new regression test 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2564 1f5c12ca-751b-0410-a591-d2e778427230
 2009-10-05 18:01:57 +00:00
nicolabertoldi
a93041d3d8 adding a new regression test 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2563 1f5c12ca-751b-0410-a591-d2e778427230
 2009-10-05 18:01:15 +00:00
nicolabertoldi
572f54f474 removing useless files 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2562 1f5c12ca-751b-0410-a591-d2e778427230
 2009-10-05 17:59:41 +00:00
nicolabertoldi
fa6a5bfc35 with this change, the usage of initial points for mert works properly 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2558 1f5c12ca-751b-0410-a591-d2e778427230
 2009-10-01 16:07:13 +00:00
nicolabertoldi
d4083b1119 adding very basic regression regr-tests for mert-moses-new which use a virtual decoder simulating the generation of the nbest lists 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2557 1f5c12ca-751b-0410-a591-d2e778427230
 2009-10-01 15:42:55 +00:00
nicolabertoldi
3484a1fd93 fixed minor bugs 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2550 1f5c12ca-751b-0410-a591-d2e778427230
 2009-10-01 07:44:30 +00:00
hieuhoang1972
9b18ec4a29 add release files 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2527 1f5c12ca-751b-0410-a591-d2e778427230
 2009-08-25 10:13:47 +00:00
nicolabertoldi
f75e3993ac correction of parameter description 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2462 1f5c12ca-751b-0410-a591-d2e778427230
 2009-08-05 16:54:33 +00:00
nicolabertoldi
8384857f8c changes to mert-moses-new.pl to work with different reference length policies: shortest, average, closest (either "--shortest", "--average", or "--closest ) for BLEU and with case-sensitive/insensitive evaluation ( --nocase) 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2461 1f5c12ca-751b-0410-a591-d2e778427230
 2009-08-05 16:39:06 +00:00
hieuhoang1972
568e973b2e make gcc amd make calls consistent for eric to use in ubuntu package 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2347 1f5c12ca-751b-0410-a591-d2e778427230
 2009-05-27 12:54:04 +00:00
phkoehn
8833098925 generalized n-best list reporting for feature functions, added experimental version of global lexical model 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2343 1f5c12ca-751b-0410-a591-d2e778427230
 2009-05-26 19:30:35 +00:00
mphi
17c3cfffac added unpaired significance evaluation 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2328 1f5c12ca-751b-0410-a591-d2e778427230
 2009-05-12 18:56:01 +00:00
bhaddow
981e440cc2 Fix detection of binarised reordering table 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2240 1f5c12ca-751b-0410-a591-d2e778427230
 2009-03-13 12:28:34 +00:00
bhaddow
e1d7bb986c Add option for predictable seeding 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2220 1f5c12ca-751b-0410-a591-d2e778427230
 2009-02-25 19:31:17 +00:00
phkoehn
8d5aef137b bug fix 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2113 1f5c12ca-751b-0410-a591-d2e778427230
 2009-02-09 16:00:35 +00:00
phkoehn
a62f8ee316 added truecaser 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2112 1f5c12ca-751b-0410-a591-d2e778427230
 2009-02-09 15:32:34 +00:00
jdschroeder
cc95706045 mert-moses.pl now supports multiple input weights for lattices and confusion networks, using the --inputweights argument. 
I'll leave it to someone who knows mert-moses-new.pl better to make the changes there.

"zcat" is now abstracted as a $ZCAT variable in these files, and is set to "gzip -cd" which should work on more platforms (notably on the mac, where zcat fails unless an archive name ends in ".Z").

 


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2082 1f5c12ca-751b-0410-a591-d2e778427230
 2009-02-05 17:39:36 +00:00
phkoehn
98381c0193 fixed xml removal 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1995 1f5c12ca-751b-0410-a591-d2e778427230
 2009-01-24 05:21:36 +00:00
phkoehn
616842f278 fixed multi-bleu documentation 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1971 1f5c12ca-751b-0410-a591-d2e778427230
 2009-01-08 00:47:10 +00:00
nicolabertoldi
2075f9dda1 modification to mert script to allow the use of fewer nbest lists; features and scores are no more gzipped 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1965 1f5c12ca-751b-0410-a591-d2e778427230
 2008-12-30 17:33:16 +00:00
bojar
091c9ece28 raising line_max_length 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1953 1f5c12ca-751b-0410-a591-d2e778427230
 2008-12-05 10:01:05 +00:00
bojar
586d7e2f84 minor fix when handling gzipped corpora 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1952 1f5c12ca-751b-0410-a591-d2e778427230
 2008-12-04 17:49:59 +00:00
bojar
2c900c8bd7 uncompress input files for phrase extract, if needed 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1951 1f5c12ca-751b-0410-a591-d2e778427230
 2008-11-27 10:39:28 +00:00
bhaddow
1e13f6d2d6 Weights can sometimes be in exponential format. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1947 1f5c12ca-751b-0410-a591-d2e778427230
 2008-11-25 09:54:45 +00:00
hieuhoang1972
2807bc48ad absolute file name check, provided by Eric Kow 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1944 1f5c12ca-751b-0410-a591-d2e778427230
 2008-11-20 13:03:59 +00:00
hieuhoang1972
254284e57e patch to fix fiddly env variable and directory stuff, provided by Eric Kow@ 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1943 1f5c12ca-751b-0410-a591-d2e778427230
 2008-11-20 13:01:49 +00:00
phkoehn
abb2fc37b1 proper binarization of lexicalized reordering model 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1938 1f5c12ca-751b-0410-a591-d2e778427230
 2008-11-10 16:03:16 +00:00
phkoehn
bfbbefd710 bug fixes 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1917 1f5c12ca-751b-0410-a591-d2e778427230
 2008-10-26 03:32:29 +00:00
mphi
8a4c6a2c63 pus significance test into proper location 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1915 1f5c12ca-751b-0410-a591-d2e778427230
 2008-10-23 09:16:33 +00:00
mphi
88d3b775ce altered the bootstrap significance script algorithm according to (Riezler and Maxwell 2005 @ MTSE'05) 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1914 1f5c12ca-751b-0410-a591-d2e778427230
 2008-10-23 09:03:41 +00:00
phkoehn
a09242ad16 bug fix with phrase table name in moses.ini, when using hmm alignment 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1913 1f5c12ca-751b-0410-a591-d2e778427230
 2008-10-20 21:37:57 +00:00
mphi
f033e32979 Added implementation of Koehn's 2004 EMNLP paired bootsrap resampling 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1911 1f5c12ca-751b-0410-a591-d2e778427230
 2008-10-20 11:55:12 +00:00
phkoehn
1c7b305152 bug fix 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1910 1f5c12ca-751b-0410-a591-d2e778427230
 2008-10-19 07:32:48 +00:00
phkoehn
1b5d99ad26 added headers for standard compliance (gcc 4.3 on 64 bit linux) 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1905 1f5c12ca-751b-0410-a591-d2e778427230
 2008-10-16 21:14:38 +00:00
phkoehn
3a5981ce9d major improvements, see email to moses-support 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1904 1f5c12ca-751b-0410-a591-d2e778427230
 2008-10-15 23:25:14 +00:00
phkoehn
614876771d extended extract/score, to allow for one big file, not just parts 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1903 1f5c12ca-751b-0410-a591-d2e778427230
 2008-10-15 22:12:56 +00:00
jdschroeder
78534c1518 made all zcat calls through ZCAT variable. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1875 1f5c12ca-751b-0410-a591-d2e778427230
 2008-08-15 16:15:51 +00:00
jdschroeder
7a2ebedc20 minor bugfixes and error checking 
-added -rootdir option to enhanced-mert
	-fixed float regex in score-nbest.py and mert-moses.pl
	-allow for extra weights in constructing ini in mert-moses.pl
	-additional NFS bug checks in mert-moses.pl



git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1869 1f5c12ca-751b-0410-a591-d2e778427230
 2008-08-01 10:24:01 +00:00
bojar
6a087d59c4 removed SCRIPTS_ROOTDIR from this 'my' declaration, it was obscuring previous 
declaration!
lines wrapped


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1865 1f5c12ca-751b-0410-a591-d2e778427230
 2008-07-14 16:24:18 +00:00
bojar
2afe9e0357 avoid coredump files in parallel moses (usually just kills NFS for a while), 
debug on a smaller scale, if needed


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1864 1f5c12ca-751b-0410-a591-d2e778427230
 2008-07-14 13:58:42 +00:00
bojar
c20e682f18 Avoid NFS race condition: 
explicitly remove old cmert output files (hoping that they will be correctly
  replaced by a 'mv' in the shell script submitted to SGE by qsubwrapper
  occasionally reveals a race condition in NFS => weights seem unchanged =>
  mert finishes too early)


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1862 1f5c12ca-751b-0410-a591-d2e778427230
 2008-07-10 11:47:55 +00:00
bhaddow
83f234cf17 Implementation of Cer et al mert regularisation. Use with argument such 
as --scconfig regtype:min,regwin:3 in extractor and mert. Only tested
on toy example so far.


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1860 1f5c12ca-751b-0410-a591-d2e778427230
 2008-06-24 19:27:18 +00:00
hieuhoang1972
52c2843e6c perl regexpr bug, submitted by German Sanchis Trilles 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1855 1f5c12ca-751b-0410-a591-d2e778427230
 2008-06-19 21:57:29 +00:00
bhaddow
4195b70247 First cut of new mert outer loop 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1842 1f5c12ca-751b-0410-a591-d2e778427230
 2008-06-10 09:07:20 +00:00
hieuhoang1972
1b44c7c445 most popular alignment outputted, finally 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1818 1f5c12ca-751b-0410-a591-d2e778427230
 2008-06-04 14:49:56 +00:00
hieuhoang1972
8554a7c89d most popular alignment outputted, finally 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1817 1f5c12ca-751b-0410-a591-d2e778427230
 2008-06-04 14:42:51 +00:00
hieuhoang1972
3832f68fed most popular alignment outputted, finally 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1816 1f5c12ca-751b-0410-a591-d2e778427230
 2008-06-04 14:40:04 +00:00
hieuhoang1972
bf34eb891d don't output alignment if inverse 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1813 1f5c12ca-751b-0410-a591-d2e778427230
 2008-06-03 12:25:37 +00:00
hieuhoang1972
b48ce341e9 output most aligned instead of merged 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1798 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-28 13:49:04 +00:00
phkoehn
7498f469ab get scripts rootdir by FindBin 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1745 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-16 15:54:02 +00:00
hieuhoang1972
a2a3d33103 explicitly use bash 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1693 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-15 08:50:22 +00:00
hieuhoang1972
3fc0b8ddb4 git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1603 1f5c12ca-751b-0410-a591-d2e778427230 2008-05-04 12:52:52 +00:00
nicolabertoldi
def5f419c2 - handling of word graph generation in the parallel environment (-output-word-graph) 
- handling of '-' (i.e. /dev/stdout) for word graphs
- if either translations, nbests, searchgraphs or wordgraphs are output to stdout
  they are concatenated in this order
  BUT I STRONGLY RECOMMENT NO TO DO THAT


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1598 1f5c12ca-751b-0410-a591-d2e778427230
 2008-04-23 10:03:45 +00:00
nicolabertoldi
d514c277df - handling of search graph generation in the parallel environment (-output-search-graph) 
- modification of the parameter for nbest generation in the parallel environment:
  I make it similar to moses parameter (-n-best-list)
- handling of '-' (i.e. /dev/stdout) for nbest and search graphs


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1597 1f5c12ca-751b-0410-a591-d2e778427230
 2008-04-23 08:37:44 +00:00
hieuhoang1972
a822d61d8f prevent -inf in lex re-ordering. Code contributed by Christian Hardmeier 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1596 1f5c12ca-751b-0410-a591-d2e778427230
 2008-04-18 09:04:38 +00:00
nicolabertoldi
1aff3d2382 correct handling of binary phrase tables 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1579 1f5c12ca-751b-0410-a591-d2e778427230
 2008-02-28 14:33:15 +00:00
nicolabertoldi
def0fff5cd changes to handle lattice input format 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1578 1f5c12ca-751b-0410-a591-d2e778427230
 2008-02-28 08:55:40 +00:00
hieuhoang1972
0bb92c2e79 merge properly 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1577 1f5c12ca-751b-0410-a591-d2e778427230
 2008-02-27 19:01:38 +00:00
hieuhoang1972
cb1f0e56dc optional output what lines are retained 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1576 1f5c12ca-751b-0410-a591-d2e778427230
 2008-02-27 18:38:31 +00:00
bojar
f056bdbfde fixed to correctly handle models in [distortion-file] section 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1572 1f5c12ca-751b-0410-a591-d2e778427230
 2008-02-26 10:24:53 +00:00
bojar
3957dc6b4c default to reordering factors of 0-0 even if decoding steps are set (users 
might have explicitly said e.g. t0-0!)


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1571 1f5c12ca-751b-0410-a591-d2e778427230
 2008-02-26 10:20:09 +00:00
hieuhoang1972
cced54cf7d win32 fix provided by jc read 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1569 1f5c12ca-751b-0410-a591-d2e778427230
 2008-02-22 21:47:32 +00:00
bojar
f7a1fb5b9c corpus compression correctly used even for generation step 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1568 1f5c12ca-751b-0410-a591-d2e778427230
 2008-02-22 16:14:30 +00:00
bojar
7f3e34207a added some heuristics for Czech quotation marks 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1567 1f5c12ca-751b-0410-a591-d2e778427230
 2008-02-22 15:07:46 +00:00
bojar
6af3140978 added optional sentence uppercasing (use -u) 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1566 1f5c12ca-751b-0410-a591-d2e778427230
 2008-02-22 14:50:43 +00:00
bojar
8b3d44b2e2 SAFE_GETLINE made safer: will exit if the line does not fit into the buffer 
instead of just going on and getting the src/tgt/alignment files out of sync


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1565 1f5c12ca-751b-0410-a591-d2e778427230
 2008-02-22 14:42:01 +00:00
bojar
f89ab590ec added Nicola's enhancedmert to released files 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1564 1f5c12ca-751b-0410-a591-d2e778427230
 2008-02-22 13:30:05 +00:00
redpony
25750c6555 if giza returns sentences that have different lengths in different directions (due to truncation or other errors), don't silenty fail. print a blank line instead. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1562 1f5c12ca-751b-0410-a591-d2e778427230
 2008-02-19 20:48:14 +00:00
bojar
fa31d83421 even factors that are being added can be gzipped 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1561 1f5c12ca-751b-0410-a591-d2e778427230
 2008-02-19 17:32:51 +00:00
bojar
eec1bdb623 added support to open gzipped files 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1560 1f5c12ca-751b-0410-a591-d2e778427230
 2008-02-19 16:05:11 +00:00
nicolabertoldi
ae319da62b revert to /bin/sh for enhanced-mert; use of setenv (instead of export) in the csh scripts created by qsub-wrapper.pl 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1515 1f5c12ca-751b-0410-a591-d2e778427230
 2007-11-21 14:31:21 +00:00
nicolabertoldi
0176c5f8ec use fo csh instead of sh 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1514 1f5c12ca-751b-0410-a591-d2e778427230
 2007-11-21 07:59:17 +00:00
bojar
89ea9828ba added ttable iterator to this script, too 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1498 1f5c12ca-751b-0410-a591-d2e778427230
 2007-11-06 03:33:41 +00:00
bojar
09d8b5e657 improving documentation and allowing environment variables to override the 
default paths


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1497 1f5c12ca-751b-0410-a591-d2e778427230
 2007-11-06 03:16:17 +00:00
nicolabertoldi
568f92b310 bug fixed 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1491 1f5c12ca-751b-0410-a591-d2e778427230
 2007-10-30 15:42:26 +00:00
jdschroeder
e52040bc12 added str length check to stop std::out_of_range error in a few more spots - similar bug to one corrected in v. 1319 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1488 1f5c12ca-751b-0410-a591-d2e778427230
 2007-10-26 12:42:55 +00:00
nicolabertoldi
fd3ecd4334 bug fixed 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1487 1f5c12ca-751b-0410-a591-d2e778427230
 2007-10-25 14:34:49 +00:00
nicolabertoldi
1b0576ba6c small bug fixed: temporary concatenated sorted file is now deleted only at the end 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1486 1f5c12ca-751b-0410-a591-d2e778427230
 2007-10-24 07:50:51 +00:00
nicolabertoldi
8710cc9bc9 features can be activated using a comma- or blank-separated list 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1485 1f5c12ca-751b-0410-a591-d2e778427230
 2007-10-23 16:55:02 +00:00
nicolabertoldi
9e70b5ffd8 Features are activated using their names 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1484 1f5c12ca-751b-0410-a591-d2e778427230
 2007-10-23 16:30:02 +00:00
nicolabertoldi
8fe62f2b95 some small bugs fixed and clean up 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1483 1f5c12ca-751b-0410-a591-d2e778427230
 2007-10-23 14:15:19 +00:00
nicolabertoldi
4720d1cb9f bug fixed 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1482 1f5c12ca-751b-0410-a591-d2e778427230
 2007-10-22 17:01:53 +00:00
nicolabertoldi
918dae011a bug fixed 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1481 1f5c12ca-751b-0410-a591-d2e778427230
 2007-10-22 14:00:10 +00:00
nicolabertoldi
e7ac20d4d6 bug fixed in the name of a temporary file 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1480 1f5c12ca-751b-0410-a591-d2e778427230
 2007-10-22 13:34:12 +00:00
nicolabertoldi
db9d0fc539 Added a more time-efficient (but more memory-consumptive) method to rescore nbest list 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1479 1f5c12ca-751b-0410-a591-d2e778427230
 2007-10-22 10:08:07 +00:00
nicolabertoldi
b827d51870 changes to cope with the new mert suite (enhanced-mert) 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1478 1f5c12ca-751b-0410-a591-d2e778427230
 2007-10-22 09:45:19 +00:00
jdschroeder
a969197e16 Fixed passing decoder parameters when tuning on single machine. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1477 1f5c12ca-751b-0410-a591-d2e778427230
 2007-10-22 09:27:09 +00:00
nicolabertoldi
5759005857 Suite of scripts to perform MERT on a subset of fetures. 
Look at the directory example to learn about its use.


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1476 1f5c12ca-751b-0410-a591-d2e778427230
 2007-10-22 09:04:03 +00:00
redpony
0cf583e249 add --hmm-align option. Allows using Giza++'s HMM word alignment model as the underlying word alignment. It is much faster than Model 4 alignment and not much worse. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1474 1f5c12ca-751b-0410-a591-d2e778427230
 2007-10-19 03:44:05 +00:00
nicolabertoldi
901823d83a explicit export of PYTHONPATH variable 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1473 1f5c12ca-751b-0410-a591-d2e778427230
 2007-10-18 15:33:04 +00:00
nicolabertoldi
81b439d728 minor changes in passing parameters to moses-parallel 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1472 1f5c12ca-751b-0410-a591-d2e778427230
 2007-10-18 12:36:49 +00:00
hieuhoang1972
4e1cad4bbe fixed sync/async bug 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1471 1f5c12ca-751b-0410-a591-d2e778427230
 2007-10-04 17:31:31 +00:00
redpony
57dcaa8e80 performance fixes for scorer 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1470 1f5c12ca-751b-0410-a591-d2e778427230
 2007-10-02 21:43:54 +00:00
hieuhoang1972
9cbc2922b4 separate word penalty for each decode step for async 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1469 1f5c12ca-751b-0410-a591-d2e778427230
 2007-10-02 12:52:00 +00:00
hieuhoang1972
d2d03c33e7 fixed bug which prevented mert working when phrase table NOT filtered or binarised 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1449 1f5c12ca-751b-0410-a591-d2e778427230
 2007-08-10 15:48:58 +00:00
hieuhoang1972
53fa2cb18a async - don't use binarising or filtering 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1443 1f5c12ca-751b-0410-a591-d2e778427230
 2007-08-05 20:29:46 +00:00
hieuhoang1972
9eba034662 turn off debugging 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1435 1f5c12ca-751b-0410-a591-d2e778427230
 2007-07-25 10:05:34 +00:00
hieuhoang1972
2beb0c44e9 mkdir before doing generation 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1425 1f5c12ca-751b-0410-a591-d2e778427230
 2007-07-14 09:54:05 +00:00
nicolabertoldi
75afdf04a5 I corrected direction of alignment 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1421 1f5c12ca-751b-0410-a591-d2e778427230
 2007-06-27 17:57:55 +00:00
nicolabertoldi
ac91cb78cc two additional (and simpler) ways of extracting alignments: source-to-target and target-to-source 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1420 1f5c12ca-751b-0410-a591-d2e778427230
 2007-06-27 16:34:14 +00:00
nicolabertoldi
7f9c2856c2 changes to reduce disk memory consumption during training 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1419 1f5c12ca-751b-0410-a591-d2e778427230
 2007-06-27 16:30:20 +00:00
phkoehn
960bebdd4a fixed clean script to handle '|'s 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1416 1f5c12ca-751b-0410-a591-d2e778427230
 2007-06-18 15:50:04 +00:00
redpony
c747cdd505 fix dumb error 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1414 1f5c12ca-751b-0410-a591-d2e778427230
 2007-06-01 16:27:56 +00:00
redpony
1f050e198a fix compile error, enable optimizations 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1413 1f5c12ca-751b-0410-a591-d2e778427230
 2007-06-01 16:26:26 +00:00
redpony
564bb5a64e make scorer use compiler optimization 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1412 1f5c12ca-751b-0410-a591-d2e778427230
 2007-06-01 16:22:30 +00:00
hieuhoang1972
aa25c7341d fixed bug with non-ascii data, recieved from Jaakko Väyrynen 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1392 1f5c12ca-751b-0410-a591-d2e778427230
 2007-05-21 13:06:40 +00:00
bojar
74954cb0ae prefer hardlinking, dropped dependency on a proprietary script 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1381 1f5c12ca-751b-0410-a591-d2e778427230
 2007-05-09 00:54:07 +00:00
bojar
31def05428 - added a comment where the binarizer is 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1380 1f5c12ca-751b-0410-a591-d2e778427230
 2007-05-07 07:19:02 +00:00
abarun
ba90a05233 Added script to perform Minimum Bayes Risk reranking 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1372 1f5c12ca-751b-0410-a591-d2e778427230
 2007-05-02 17:26:01 +00:00
hieuhoang1972
13e07cef5f multiple distance based distortion for async decoder 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1370 1f5c12ca-751b-0410-a591-d2e778427230
 2007-04-23 19:17:38 +00:00
redpony
485bda2db5 andreas zollman's changes to write span information 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1367 1f5c12ca-751b-0410-a591-d2e778427230
 2007-04-20 14:53:46 +00:00
jdschroeder
3e1aabc487 Removed a few errant svn diff lines that found their way into the file. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1366 1f5c12ca-751b-0410-a591-d2e778427230
 2007-04-19 10:52:17 +00:00
jdschroeder
752d148c6e Changed initial setting of number of distortion weights from 0 to 1. For models with lexicalized reordering, this script was generating one too few weights in moses.ini 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1365 1f5c12ca-751b-0410-a591-d2e778427230
 2007-04-18 19:38:14 +00:00
redpony
c80d8b8d47 Support for the decoding of arbitrary word lattices. Must be given in the form of a "plf" file, which is a little tricky. I'll add documentation at some point; for now, refer to the example plf file in the "lattice-surface" regression test. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1359 1f5c12ca-751b-0410-a591-d2e778427230
 2007-04-18 14:08:46 +00:00
hieuhoang1972
45dde20c54 comment out psyco library 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1354 1f5c12ca-751b-0410-a591-d2e778427230
 2007-04-12 17:48:48 +00:00
hieuhoang1972
75c20e7609 Add alignment info to phrase table 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1352 1f5c12ca-751b-0410-a591-d2e778427230
 2007-04-11 19:58:38 +00:00
hieuhoang1972
b84191c9d3 Add alignment info to phrase table 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1351 1f5c12ca-751b-0410-a591-d2e778427230
 2007-04-11 19:56:43 +00:00
hieuhoang1972
b9d2288c22 compileable with visual studio 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1349 1f5c12ca-751b-0410-a591-d2e778427230
 2007-04-11 11:35:36 +00:00