bgottesman
|
e409b6827c
|
add --max-word-length option to cleaning script, with default value 1000; any segment with a word (or factor) exceeding this length in chars is discarded; motivated by symal.cpp, which has its own such parameter (hardcoded to 1000) and crashes if it encounters a word that exceeds it
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3410 1f5c12ca-751b-0410-a591-d2e778427230
|
2010-08-23 16:35:14 +00:00 |
|
pjwilliams
|
2edfc16912
|
Merge remaining script support for tree-based models from mt3_chart.
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3137 1f5c12ca-751b-0410-a591-d2e778427230
|
2010-04-16 09:45:51 +00:00 |
|
hieuhoang1972
|
0bb92c2e79
|
merge properly
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1577 1f5c12ca-751b-0410-a591-d2e778427230
|
2008-02-27 19:01:38 +00:00 |
|
hieuhoang1972
|
cb1f0e56dc
|
optional output what lines are retained
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1576 1f5c12ca-751b-0410-a591-d2e778427230
|
2008-02-27 18:38:31 +00:00 |
|
phkoehn
|
960bebdd4a
|
fixed clean script to handle '|'s
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1416 1f5c12ca-751b-0410-a591-d2e778427230
|
2007-06-18 15:50:04 +00:00 |
|
hieuhoang1972
|
aa25c7341d
|
fixed bug with non-ascii data, recieved from Jaakko Väyrynen
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1392 1f5c12ca-751b-0410-a591-d2e778427230
|
2007-05-21 13:06:40 +00:00 |
|
hieuhoang1972
|
3c07c5df4d
|
add svn id comments to start of file
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1307 1f5c12ca-751b-0410-a591-d2e778427230
|
2007-03-14 22:22:36 +00:00 |
|
bojar
|
87b168cd1b
|
Handles gzipped input.
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1296 1f5c12ca-751b-0410-a591-d2e778427230
|
2007-03-12 02:24:34 +00:00 |
|
bojar
|
c8f5e2aeba
|
fixed an error message
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@866 1f5c12ca-751b-0410-a591-d2e778427230
|
2006-10-06 10:47:16 +00:00 |
|
bojar
|
232727e0e4
|
removed the dependence on external lowercaser, lowercasing internally
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@488 1f5c12ca-751b-0410-a591-d2e778427230
|
2006-08-03 18:07:10 +00:00 |
|
bojar
|
4d49e12bc4
|
checking the latest version from /export/bin to cvs
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@486 1f5c12ca-751b-0410-a591-d2e778427230
|
2006-08-03 17:53:13 +00:00 |
|