Jeroen Vermeulen
ef028446f3
Add license notices to scripts.
...
This is not pleasant to read (and much, much less pleasant to write!) but
sort of necessary in an open project. Right now it's quite hard to figure
out what is licensed how, which doesn't matter much to most people but can
suddenly become very important when people want to know what they're being
allowed to do.
I kept the notices as short as I could. As far as I could see, everything
without a clear license notice is LGPL v2.1 or later.
2015-05-29 18:30:26 +07:00
Hieu Hoang
2f0ee5502e
delete debugging info
2015-05-18 16:26:26 +04:00
Jeroen Vermeulen
a25193cc5d
Fix a lot of lint, mostly trailing whitespace.
...
This is lint reported by the new lint-checking functionality in beautify.py.
(We can change to a different lint checker if we have a better one, but it
would probably still flag these same problems.)
Lint checking can help a lot, but only if we get the lint under control.
2015-05-17 20:04:04 +07:00
Nicola Bertoldi
5700fbaabf
Merge branch 'master' of https://github.com/moses-smt/mosesdecoder
2015-04-22 07:50:07 +02:00
Hieu Hoang
6162223690
add use warnings to all perl scripts
2015-04-13 20:42:33 +04:00
Hieu Hoang
54e55f2dcb
better detection of pigz, sort, split. In case they are not in the default directory
2015-04-06 11:31:44 +04:00
Hieu Hoang
2d1da3219d
consistently use 'env perl' command for environments where the 1st perl in PATH isn't the default perl. Which is kinda stupid
2015-04-02 17:38:56 +04:00
Nicola Bertoldi
a1539505c8
minor change to make extract-parallel.perl compliant with MacOSX split command
2015-02-04 09:02:51 +01:00
Hieu Hoang
90d4b2d713
use pigz rather than gzip if it exists
2015-01-13 15:16:22 +00:00
Matthias Huck
c27cbf55ea
source labels: integration into EMS
2014-08-07 21:02:51 +01:00
Matthias Huck
3a5dee12e8
implementation of phrase orientation in GHKM extraction
...
(...but a corresponding feature function for the chart-based decoder has not been written yet)
2014-07-28 18:27:12 +01:00
Hieu Hoang
f95a1bb75b
more shards
2014-06-23 09:36:13 -04:00
Rico Sennrich
7138056b8f
flexibility scores
2013-09-07 23:04:01 +02:00
Hieu Hoang
c104dee3b2
merge glue grammars, rather than writing them all to the same file. Required by Phil Williams & others when doing syntax extraction
2013-07-24 19:01:46 +01:00
Hieu Hoang
1b83b85f44
debug info from sort command
2013-03-18 16:48:40 +00:00
phikoehn
d5cf38cab2
Merge branch 'master' of git://github.com/moses-smt/mosesdecoder
2013-01-14 19:23:02 +00:00
phikoehn
344b150372
bug fixes with escaping / truecasing interactions
2013-01-14 19:22:29 +00:00
Barry Haddow
459acf87b1
Add support for instance weights file
2013-01-04 14:55:24 +00:00
phikoehn
b275c94dbf
allow for inclusion of extract from previous run
2012-12-12 07:02:59 +00:00
phikoehn
a84fbcb80a
bug fix for using domain feature in multi-process extract
2012-09-24 11:14:36 +01:00
Hieu Hoang
30e5b0575b
merge conflict
2012-09-03 19:12:00 +01:00
phikoehn
4a1a995878
a lot of changes
2012-08-18 23:48:26 +01:00
phikoehn
366ab93f8a
a lot of changes
2012-08-18 23:47:05 +01:00
Hieu Hoang
23bbf12752
redirect stderr from extract program to stderr of script that runs it, for easier debugging
2012-07-18 22:53:47 +01:00
Hieu Hoang
f0a3e67717
redirect stderr from extract program to stderr of script that runs it, for easier debugging. Recursive create of output directory, in case output directory doesn't exist
2012-07-18 22:33:49 +01:00
Hieu Hoang
b609473645
under changes to make scripts handle directories with spaces. Karel gave up, and it messing up cruise control
2012-07-18 11:02:24 +01:00
Karel Bílek
ad257b27d9
spaces in filepath does not wreak havoc anymore
2012-07-17 04:36:27 +02:00
Barry Haddow
25f5def387
Fix for case where no ttable is to be created
2012-06-03 22:03:41 +01:00
Hieu Hoang
f91b19fa41
Works with phrase-table scoring
2012-05-23 19:26:37 +01:00
Hieu Hoang
761ac4f0fc
parallel extract. Works with reordering
2012-05-23 19:02:36 +01:00
Hieu Hoang
e74f67abc5
use new gzipped extract file. Always save extract files already sorted
2012-05-13 10:23:59 +01:00
Hieu Hoang
066c209fcb
use new gzipped extract file. Always save extract files already sorted
2012-05-12 15:41:07 +01:00
Hieu Hoang
ef390a705f
1. use new gsort, rather than sort, if it is available
...
2. check return value of cat command in extract-parallel in case it runs out of disk space etc
2012-03-11 15:50:46 +07:00
Hieu Hoang
de9eeab7e9
mac osx compatible split & sort
2012-02-23 13:26:19 +00:00
Hieu Hoang
7073f7d891
bug fix by Guchun Zhang
2012-02-16 10:28:09 +07:00
Hieu Hoang
7f6f8a99f9
bug fix by Guchun Zhang
2012-02-16 10:26:48 +07:00
Hieu Hoang
53b41f7c45
parallel extract
2012-02-09 18:24:49 +07:00
hieuhoang1972
6a27dc4f17
example of how to run
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4084 1f5c12ca-751b-0410-a591-d2e778427230
2011-07-22 08:32:09 +00:00
hieuhoang1972
debca7632b
change order of arguments. Arguments for extract-parallel are simple extension of normal extract
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4027 1f5c12ca-751b-0410-a591-d2e778427230
2011-06-23 02:25:49 +00:00
hieuhoang1972
2cdc39f63f
parallelize extract using perl fork.
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4025 1f5c12ca-751b-0410-a591-d2e778427230
2011-06-21 21:48:56 +00:00
hieuhoang1972
4b5c8aaf10
parallelize extract using perl fork.
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4024 1f5c12ca-751b-0410-a591-d2e778427230
2011-06-21 21:34:18 +00:00
hieuhoang1972
4689d33d0f
parallelize extract using perl fork.
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4023 1f5c12ca-751b-0410-a591-d2e778427230
2011-06-21 21:31:47 +00:00
hieuhoang1972
9eb51e31fb
parallelize extract using perl fork.
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4022 1f5c12ca-751b-0410-a591-d2e778427230
2011-06-21 16:47:22 +00:00
hieuhoang1972
62ddd6eb53
parallelize extract using perl fork.
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4021 1f5c12ca-751b-0410-a591-d2e778427230
2011-06-21 16:45:42 +00:00
hieuhoang1972
bd64e748ff
parallelize extract using perl fork.
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4020 1f5c12ca-751b-0410-a591-d2e778427230
2011-06-21 16:38:15 +00:00
hieuhoang1972
56fc94c2a7
parallelize extract using perl fork. Not quite ready for prime time
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4019 1f5c12ca-751b-0410-a591-d2e778427230
2011-06-21 16:20:22 +00:00
