moses-smt/mosesdecoder
1
1
Fork 0
mirror of https://github.com/moses-smt/mosesdecoder.git synced 2024-09-11 11:25:40 +03:00
Code Issues Packages Projects Releases Wiki Activity
17,477 Commits 136 Branches 26 Tags 147 MiB
master
Commit Graph

2352 Commits

Author SHA1 Message Date
Pierre Lison
e26bbc215f sending the stderr output of the which command to /dev/null 2015-09-16 11:01:22 +02:00
Ulrich Germann
2e3a82a40c Merge pull request #125 from akivajp/master 
Fixed for removed option building cooccurrence table.
 2015-09-12 11:32:27 +02:00
Akiva Miura
7b15548144 Fixed for removed option building cooccurrence table. 2015-09-07 06:21:04 +09:00
Barry Haddow
90f15cc619 extra nplm settings 2015-09-04 10:07:50 +01:00
Barry Haddow
4746970bf8 Merge branch 'master' of github.com:moses-smt/mosesdecoder 2015-09-04 10:06:58 +01:00
Barry Haddow
e58ddf74e8 parameter changes 2015-09-04 10:06:54 +01:00
Tomáš Fulajtár
1a26cb8414 Added a simple support for the factored systems. 2015-08-27 15:15:32 +02:00
Hieu Hoang
d349bf8a94 dos2unix everything 2015-08-23 19:00:19 +04:00
Matthias Huck
d5c41634e8 EMS: fix filtering issue when output-splitter is defined 2015-08-21 18:58:36 +01:00
Matthias Huck
261cfdb024 perl shebang 2015-08-19 16:59:10 +01:00
Hieu Hoang
4ff776f564 dos2unix the whole lot 2015-08-19 16:30:31 +04:00
Hieu Hoang
97eed02301 dos2unix. Revert Matthias' change, use env 2015-08-19 13:25:29 +01:00
Matthias Huck
d583e0888f make-factor-de-pos.perl 2015-08-18 18:31:41 +01:00
Hieu Hoang
3a261c9fc9 don't hardcode amount of mem to be used by lmplz 2015-08-16 20:32:07 +04:00
Phil Williams
01a9dd2305 extract-target-trees.py: support for new-style trace files 2015-08-14 16:53:24 +01:00
Ulrich Germann
883c34aee9 Merge branch 'master' of http://github.com/moses-smt/mosesdecoder into mmt-dev 
Conflicts:
	moses/SearchNormalBatch.cpp
	moses/TranslationModel/UG/mm/ug_bitext.h
	moses/TranslationModel/UG/mm/ug_typedefs.h
	moses/TranslationModel/UG/mmsapt.cpp
	moses/TranslationModel/UG/mmsapt.h
 2015-08-07 14:14:19 +01:00
Barry Haddow
4c3a6a3f3f remove dash 2015-08-03 21:19:08 +01:00
Barry Haddow
57b0c351c0 Merge branch 'master' of github.com:moses-smt/mosesdecoder 2015-08-03 16:47:31 +01:00
Barry Haddow
f808b32030 support version of nplm that picks best on heldout 2015-08-03 16:47:25 +01:00
Barry Haddow
a39544bbcb fix inconsistency in the example 2015-08-03 16:46:57 +01:00
Hieu Hoang
bfd45fdfc3 don't use all threads 2015-08-01 11:35:47 +04:00
Hieu Hoang
7ac6f90a4d Merge branch 'master' of github.com:moses-smt/mosesdecoder 2015-07-31 22:29:07 +04:00
Hieu Hoang
f894dec0fd multi-threaded decoding by default /Vincent Nguyen 2015-07-31 22:28:45 +04:00
Rico Sennrich
89d16a491a fix ems regression (concatenate-split step) 2015-07-31 11:20:29 +01:00
Philipp Koehn
a5ee3c1b6d script to copy model files to local disk before running the decoder - useful for grid 2015-07-29 11:10:13 -04:00
Philipp Koehn
836ca8212a better support of grid engine cluster 2015-07-29 11:03:24 -04:00
Philipp Koehn
b29166e2fe fix of fix 2015-07-29 11:02:55 -04:00
Philipp Koehn
837bcde69d bug with legacy 2015-07-29 10:46:09 -04:00
Philipp Koehn
ae9cd14948 fixes 2015-07-29 10:44:57 -04:00
Hieu Hoang
5f4b497b63 Merge pull request #119 from cidermole/master 
Fix 'Use of uninitialized value' error through explicit setting of 0s…
 2015-07-29 16:25:39 +04:00
Barry Haddow
3a2116b2c9 add quotes so arguments don't get lost 2015-07-29 09:35:19 +01:00
Rico Sennrich
0173512ddc separate xml version of combine_factors.pl 
xml version causes slow-down for users who use factors, but not xml (which are most)
 2015-07-28 23:30:18 +01:00
Phil Williams
2cda286a06 experiment.meta: re-run fast_align symmetrization if symmetrization type changes 2015-07-28 16:55:55 +01:00
Rico Sennrich
a968536176 ems fix: pass-unless doesn't understand AND 2015-07-28 16:37:50 +01:00
Ulrich Germann
d67723fd29 Merge branch 'master' of http://github.com/moses-smt/mosesdecoder into ranked-sampling 
Conflicts:
	moses/TargetPhrase.cpp
	moses/TargetPhrase.h
 2015-07-28 14:29:49 +01:00
Barry Haddow
e53ad40859 Support for nplm in ems 2015-07-23 10:37:26 +01:00
Philipp Koehn
1a795f549e only extract reordering phrase pairs if use mmsapt phrase table 2015-07-20 11:47:24 -04:00
Philipp Koehn
777a88673d compress sort tmp files by default 2015-07-20 11:46:47 -04:00
Philipp Koehn
496f8c6d85 only extract reordering phrase pairs if use mmsapt phrase table 2015-07-20 11:44:22 -04:00
Philipp Koehn
fcf2934a2f customized phrase table pruning step 2015-07-20 11:43:02 -04:00
Rico Sennrich
7b19d83d43 xml support for combine-factors.pl 2015-07-20 10:45:18 +01:00
Rico Sennrich
1b1bafb1e8 ems: add option to factorize after truecase/split/etc. 2015-07-20 10:43:23 +01:00
Rico Sennrich
e85f353898 code simplification by removing language-specific, unused hack. 2015-07-20 10:39:01 +01:00
Ulrich Germann
df50e454d2 Proper handling of moses parameters with double dash in create_config(). 2015-07-17 14:32:30 +01:00
Phil Williams
c83628a92b Fix errors from multiline `` commands in transliteration Perl scripts 
Replace the backslash-newline sequence with backslash-backslash-newline in
multiline backquote command strings.  i.e. replace expressions like this:

  `some-command \
    -option1 \
    -option2`;

with ones like this

  `some-command \\
    -option1 \\
    -option2`;

If I understand this right, the shell converts a backslash-newline sequence
to an empty string (i.e. it discards it), but Perl does not.  Unless the
backslash itself is escaped, using a backslash-newline in a Perl command
string results in errors in most instances.  By escaping the backslash, it
gets passed through to the shell where it is interpreted as intended.
 2015-07-16 14:54:00 +01:00
Philipp Koehn
66ecf98cf7 minor bug fix 2015-07-14 11:01:22 -04:00
Rico Sennrich
ca72105fdf fix ems regression 2015-07-14 13:16:25 +01:00
David Madl
3c30210dad Fix 'Use of uninitialized value' error through explicit setting of 0s in hash. 
Fixes the following errors in bootstrap-hypothesis-difference-significance.pl on Perl v5.14.2:

Use of uninitialized value $coocUpd in numeric gt (>) at /fs/lofn0/dmadl/software/mosesdecoder/scripts/analysis/bootstrap-hypothesis-difference-significance.pl line 317.
Use of uninitialized value $b in numeric lt (<) at /fs/lofn0/dmadl/software/mosesdecoder/scripts/analysis/bootstrap-hypothesis-difference-significance.pl line 543.
Use of uninitialized value $coocUpd in addition (+) at /fs/lofn0/dmadl/software/mosesdecoder/scripts/analysis/bootstrap-hypothesis-difference-significance.pl line 314.
Use of uninitialized value $coocUpd in numeric gt (>) at /fs/lofn0/dmadl/software/mosesdecoder/scripts/analysis/bootstrap-hypothesis-difference-significance.pl line 317.
Use of uninitialized value $a in numeric gt (>) at /fs/lofn0/dmadl/software/mosesdecoder/scripts/analysis/bootstrap-hypothesis-difference-significance.pl line 552.
 2015-07-14 13:05:22 +01:00
Philipp Koehn
7e3050f7f2 allow saving of model from fast-align (for incremental use) 2015-07-14 05:27:03 -04:00
Barry Haddow
3fdbb00904 Improvements to handling of bilingual LM in EMS 2015-07-10 15:44:24 +01:00
Hieu Hoang
f66beabf4f Generation error in EMS due to pruning. Lets see if this works. 2015-06-28 14:03:54 +04:00
Hieu Hoang
82edbb98a7 comments in ini file about default weights 2015-06-28 10:40:43 +04:00
Hieu Hoang
57e213ed19 tighten up extract-parallel on osx. Can now use gsplit and bsd split 2015-06-26 12:18:21 +04:00
Hieu Hoang
ca54852641 tighten up extract-parallel on osx. Can now use gsplit and bsd split 2015-06-26 11:55:24 +04:00
Hieu Hoang
b83803203e prune generation table in ems 2015-06-25 18:10:31 +04:00
Hieu Hoang
dce0f33270 prune generation table in ems 2015-06-24 18:35:59 +04:00
Barry Haddow
425118aa5d bugfixes - working directory 2015-06-17 09:32:29 +01:00
Barry Haddow
ad8114ddb0 capitalisation 2015-06-15 16:23:12 +01:00
XapaJIaMnu
166bf7365f Forgot to update the weight config path 2015-06-12 16:56:36 +01:00
XapaJIaMnu
ffd3f2bb6e Added basic BilingualNPLM support to EMS and an example config. 2015-06-12 16:21:24 +01:00
Jeroen Vermeulen
dbcc264506 Remove unneeded script. 
Tom Hoar, the author of this script, asked me to remove it because it
doesn't actually do what the current name says, and can't work without
an additional script which isn't in the repository.
 2015-06-09 23:10:27 +07:00
Lexi Birch
b76194a16b Merge branch 'master' of https://github.com/moses-smt/mosesdecoder 2015-06-08 17:13:00 +01:00
Lexi Birch
501c51947b Allowing the truecaser to work on uncased ASR input, pass the -a flag 2015-06-08 16:58:50 +01:00
Jeroen Vermeulen
85c23ed7dc Fix some JS lint. 2015-06-02 18:05:12 +07:00
Jeroen Vermeulen
b3e577be76 Fixing lint. Only 600 or so lines of errors left! 2015-06-02 17:29:32 +07:00
Jeroen Vermeulen
0981d23705 Lint-fixing binge. 2015-06-02 16:02:39 +07:00
Rico Sennrich
5d8af9c289 support memory-mapped files for NPLM training 2015-05-29 16:07:26 +01:00
Jeroen Vermeulen
ef028446f3 Add license notices to scripts. 
This is not pleasant to read (and much, much less pleasant to write!) but
sort of necessary in an open project.  Right now it's quite hard to figure
out what is licensed how, which doesn't matter much to most people but can
suddenly become very important when people want to know what they're being
allowed to do.

I kept the notices as short as I could.  As far as I could see, everything
without a clear license notice is LGPL v2.1 or later.
 2015-05-29 18:30:26 +07:00
Jeroen Vermeulen
26170a4179 Friendlier error reporting in beautify.py. 2015-05-29 09:37:37 +07:00
Barry Haddow
c27aa193ea Revert "Min score parameter". Doesn't work without filter. 
This reverts commit ab2d396781.
 2015-05-28 17:44:26 +01:00
Barry Haddow
ab2d396781 Min score parameter 2015-05-28 17:10:21 +01:00
Phil Williams
842fc9780e senna2brackets.py: bug fixes + clean-up 2015-05-27 20:33:43 +01:00
Phil Williams
c086a8ee50 Add a wrapper script for parsing English text with SENNA 2015-05-26 16:44:13 +01:00
Rico Sennrich
f6f56d11af ems: parse-relax comes last in train; do same for dev/test 2015-05-25 15:52:07 +01:00
Hieu Hoang
582a845524 don't use zcat 2015-05-24 20:04:01 +04:00
Rico Sennrich
43527c82fc training script for monolingual Neural LM 
(+bugfixes and usability improvements for RDLM training)
 2015-05-22 15:31:08 +01:00
Rico Sennrich
a1678187fe wrapper for stanford dependency parser 2015-05-22 15:28:42 +01:00
Rico Sennrich
98ff2382d0 duplication of existing functionality 2015-05-20 17:35:38 +01:00
Rico Sennrich
6aac7ded9a EMS: more flexible way to concatenate LM training data. 
the implementation allows the user to specify which corpora to combine,
and to have multiple LMs on the same data.
 2015-05-20 17:20:02 +01:00
Hieu Hoang
36caf2eb9a escape ^# character otherwise morfessors skips line 2015-05-20 15:28:28 +04:00
Hieu Hoang
79ca96db0a should have tested this 2015-05-19 22:19:11 +04:00
Hieu Hoang
59071bf16c run on all cores if number of cores not given 2015-05-19 18:32:31 +04:00
Rico Sennrich
8ca6764c7d ems: allow LMs with user-specified training commands and moses.ini config entries 
intended for neural LMs, syntactic LMs, and the like. currently doesn't play nice with INTERPOLATED-LM.
 2015-05-18 19:07:37 +01:00
Rico Sennrich
fb06a2325e fix broken ems with interpolated lm disabled 2015-05-18 17:26:09 +01:00
Rico Sennrich
f85dd85f6b ignore-unless magic 2015-05-18 16:17:33 +01:00
Rico Sennrich
59376f500b still confused about pass-unless vs. ignore-unless 2015-05-18 14:40:56 +01:00
Rico Sennrich
45a97f9016 EMS: disable concatenated LM by default 2015-05-18 14:10:29 +01:00
Hieu Hoang
2c0aecb16b Merge branch 'master' of github.com:moses-smt/mosesdecoder 2015-05-18 16:27:03 +04:00
Hieu Hoang
2f0ee5502e delete debugging info 2015-05-18 16:26:26 +04:00
Rico Sennrich
27fd45d088 ems: training LM on concatenation of all LM training corpora 2015-05-18 12:18:49 +01:00
Hieu Hoang
14d2a67193 Merge branch 'master' of github.com:moses-smt/mosesdecoder 2015-05-18 12:28:35 +04:00
Hieu Hoang
5fdcf372ae rename a python file to have have .py, instead of .perl. In case beautify script depends on it 2015-05-18 12:27:35 +04:00
Jeroen Vermeulen
5aa70c6cdd Also reformat Perl, using Perltidy. 2015-05-18 00:45:15 +07:00
Jeroen Vermeulen
494c20f634 Add note about Perltidy. 2015-05-17 22:48:03 +07:00
Jeroen Vermeulen
e2a632a2b8 JavaScript lint. 2015-05-17 21:36:07 +07:00
Jeroen Vermeulen
5d0bbb6a45 Fix some JavaScript lint. Still a lot left. 2015-05-17 21:24:04 +07:00
Jeroen Vermeulen
a25193cc5d Fix a lot of lint, mostly trailing whitespace. 
This is lint reported by the new lint-checking functionality in beautify.py.
(We can change to a different lint checker if we have a better one, but it
would probably still flag these same problems.)

Lint checking can help a lot, but only if we get the lint under control.
 2015-05-17 20:04:04 +07:00
Jeroen Vermeulen
108da16374 Suppress CSS lint checking; accept longer lines. 
The CSS style that Pocketlint expects is just too different from what we
have.  Don't check those files for now.

Also, a maximum line length of 300 still gives too many warnings, so I'm
regretfully dumping the default to 400 characters.  The traditional 80
characters are already longer than the measured optimum for human reading,
so I hope some day we can address this!
 2015-05-17 20:03:27 +07:00
Jeroen Vermeulen
07a8fe06aa Also support checking for lint. 
Choose which action(s) you want for each run: --format and/or --lint.

Many different types of files are lint-checkable, but you need Pocketlint
installed (plus ideally, its plugins for the various languages).

Also, added option to control batching of the commands.
 2015-05-17 18:25:06 +07:00
Jeroen Vermeulen
9bdcb5f7c1 Fix more Python lint. 
This is about the last that isn't in contrib or generated files.  At this
point we can start doing regular lint checks, at least on the Python files,
without being completely inundated with warnings.
 2015-05-16 18:03:54 +07:00
Jeroen Vermeulen
61162dd242 Fix more Python lint. 
Most of the complaints fixed here were from Pocketlint, but many were also
from Syntastic the vim plugin.
 2015-05-16 17:26:56 +07:00
Jeroen Vermeulen
0ffe79579e Fix some python lint. 
I used mainly pocketlint, a very good Python linter, but also Syntastic,
a vim plugin.  Didn't get anywhere near fixing all of Syntastic's complaints
though.

Once I've cleaned up all (or at least most) of the Python lint, we can
start doing regular automated lint checks and keep the code clean.
 2015-05-16 14:58:03 +07:00
Jeroen Vermeulen
f1ed14eb33 Move ignored path prefixes into config file. 
The path prefixes listed in .beautify-ignore, in the project root, will not
be cleaned up.  C and C++ files everywhere else will be.

Also fixes bugs in the prefix-matching code, and makes the matching a little
bit more powerful: the prefix can now extend down into the directory tree.
 2015-05-15 16:40:06 +07:00
Jeroen Vermeulen
d7599134b8 Remove the --any-astyle option. 
The risk from flip-flopping on styles is too great: 2.04 formatting has
many changes from 2.01 formatting.

Also, fix a broken version check.
 2015-05-15 15:24:54 +07:00
Jeroen Vermeulen
3e821b56dd Rewrite beautify script in Python. 
The new version is much longer, but hopefully extensible, reusable, and
easy to read.  With a few more changes it will let us do three things:

1. Apply more checks and cleanups.
2. Clean up additional file types.
3. Use the same script for mgiza++, so we get a uniform code style.

The "more cleanups" could be more things like the removal of trailing
whitespace which we just added.  We may even want to run lint checkers of
some sort.

Additional file types could be e.g. Perl scripts, build configuration,
or documentation.

In order to make the script reusable we'll have to generalize the
skip_at_root list in list_c_like_files into something like a configuration
file.
 2015-05-15 13:46:00 +07:00
Hieu Hoang
5173b9f617 beautify. Add sed for trailing spaces 2015-05-13 11:29:16 +01:00
Hieu Hoang
87e1f1351f tighten up OSM build. More debugging output, to stderr not stdout. lmplz uses outdir as temp directory 2015-05-13 12:29:56 +04:00
Hieu Hoang
0cd62488bf morfessor wrapper 2015-05-12 20:40:19 +04:00
Hieu Hoang
abfc0671a3 osm tweaks and morfessor wrapper 2015-05-12 20:19:39 +04:00
Hieu Hoang
a922245864 default to using lmplz for convenience and because SRILM uses tonnes of memory 2015-05-12 11:44:05 +04:00
Hieu Hoang
03e507f354 default to using lmplz for convenience and because SRILM uses tonnes of memory 2015-05-12 10:40:52 +04:00
Hieu Hoang
8bb18b9ff0 add no-splitter-training argument. Splitter to be used by mada 2015-05-11 15:26:50 +04:00
Barry Haddow
85c1af4d72 Merge branch 'master' of github.com:moses-smt/mosesdecoder 2015-05-08 09:16:55 +01:00
Barry Haddow
f403f5e478 mmsapt doesn't require feature weights on first tuning iteration 2015-05-08 09:16:51 +01:00
Hieu Hoang
2acb590394 output bleu for multi-bleu hack 2015-05-05 17:54:35 +04:00
Hieu Hoang
d006c6ef8c don't output remaining args twice 2015-05-05 12:15:08 +04:00
Hieu Hoang
5fefb0da47 Merge branch 'master' of github.com:moses-smt/mosesdecoder 2015-05-05 12:02:13 +04:00
Hieu Hoang
8f272e04a9 output debugging messages to stderr, not stdout 2015-05-05 12:01:21 +04:00
Nicola Bertoldi
90a982e579 merge remote into local 2015-05-04 09:42:44 +02:00
Hieu Hoang
d456d9229e add multi-bleu-detok. Like multi-bleu scoring but will detokenize/post-process before scoring 2015-05-03 14:07:12 +04:00
Hieu Hoang
e5f76ee99e Merge branch 'master' of github.com:moses-smt/mosesdecoder 2015-05-03 11:50:31 +04:00
Hieu Hoang
73ae7d7e20 option not to use parallel 2015-05-03 11:50:10 +04:00
Philipp Koehn
a4a7c14593 allow breaking up training data for fast align (to avoid memory blowups for very large corpora) 2015-05-01 17:47:08 -04:00
Philipp Koehn
de6a9bd1b3 minor updates to factor scripts; brown-cluster may now run other scripts (e.g., truecaser) before assigning classes 2015-05-01 17:46:14 -04:00
Philipp Koehn
b369699661 various small changes, mostly related to better compliance with grid engine 2015-05-01 17:44:18 -04:00
Rico Sennrich
e98a2fc980 fix interpolation for LM with parser in pre-processing 2015-04-30 15:46:33 +01:00
Hieu Hoang
1278b8f5a7 Merge branch 'master' of github.com:moses-smt/mosesdecoder 2015-04-30 15:35:34 +04:00
Hieu Hoang
a6d34a660d another madamira wrapper. Just uses the tokenized file it outputs 2015-04-30 15:35:15 +04:00
Hieu Hoang
15e4b16f49 delete unused var 2015-04-30 14:01:03 +04:00
Hieu Hoang
ebc5a51d32 Merge pull request #111 from unhammer/extract-perl-safewait 
die if the forked extract exited with error
 2015-04-30 11:45:25 +04:00
Hieu Hoang
1c99b2b2b8 Merge pull request #110 from unhammer/extract-perl-abspath-when-ln 
avoid bad symlinks in extract-parallel
 2015-04-30 11:41:03 +04:00
Kevin Brubeck Unhammer
2af2f2ef36 avoid bad symlinks in extract-parallel 
train-model seems to pass a non-absolute path for the
model/aligned-argument, and then extract-parallel creates a bad symlink
 2015-04-30 09:36:59 +02:00
Kevin Brubeck Unhammer
c116fa0dbf die if the forked extract exited with error 
Should we pass on bad exit codes from RunFork to those waitpids as well?
Seems like the right thing, though I don't know the code.
 2015-04-30 09:33:37 +02:00
Nicola Bertoldi
3400b622c0 Merge branch 'master' of https://github.com/moses-smt/mosesdecoder 2015-04-30 08:35:41 +02:00
Hieu Hoang
8f9bf7ea38 add -config 2015-04-28 15:03:59 +04:00
Hieu Hoang
b7792b227a script to convert arabic to bw, and vice versa 2015-04-28 12:29:58 +04:00
Hieu Hoang
8adad4fc2e exec permission 2015-04-27 17:39:49 +04:00
Hieu Hoang
a47fc00635 option to output factors 2015-04-27 17:35:19 +04:00
Rico Sennrich
da648fd65b fix some RDLM training options 2015-04-27 10:52:16 +01:00
alvations
fa30ea6712 Merge branch 'master' of https://github.com/moses-smt/mosesdecoder into moses-smt-master 2015-04-26 20:38:27 +02:00
alvations
4a68c42b16 syncing to latest moses version 2015-04-26 20:37:10 +02:00
alvations
ec54ea3c4f put back some of the difference made after RELEASE3.0 and incorporated it with the -threads parameter 2015-04-26 20:30:15 +02:00
alvations
c01b0a6262 merging the filter-model-given-input.pl with alvations-master branch 2015-04-26 20:25:15 +02:00
alvations
dda3ddd80b Merge branch 'master' of https://github.com/moses-smt/mosesdecoder 2015-04-26 20:23:39 +02:00
alvations
e1fcc8082a use integer type when reading options instead of checking for undef. it's more elegant. 2015-04-24 19:30:40 +02:00
alvations
d453ccc9f5 removed wrongly added perl script... 2015-04-24 19:14:04 +02:00
alvations
0ccbcaece6 added $threads option in usage example 2015-04-24 19:11:57 +02:00
alvations
aa9207acfc fixed typo in $thread to $threads 2015-04-24 19:10:10 +02:00
alvations
6c63ca963c checks for undefined $threads 2015-04-24 19:06:37 +02:00
alvations
585784f62a added thread options for filter-model-given-input.pl 2015-04-24 18:57:28 +02:00
Hieu Hoang
4b47e1148c use ignore-unless /Philipp Koehn 2015-04-22 23:02:57 +04:00
Hieu Hoang
40933b4a78 hack to allow target side of tokenized parallel corpus to be used for LM 2015-04-22 19:01:12 +04:00
Nicola Bertoldi
5700fbaabf Merge branch 'master' of https://github.com/moses-smt/mosesdecoder 2015-04-22 07:50:07 +02:00
Hieu Hoang
c15f3ef068 duplicated functionality with ems/support/lmplz-wrapper.perl 2015-04-21 17:54:34 +04:00
Hieu Hoang
ab01d30687 make sure GetOptions doesn consume -T by confusing it with --text 2015-04-21 17:53:46 +04:00
Rico Sennrich
15d3c3f259 be more tolerant about xml input 2015-04-21 14:04:25 +01:00
Rico Sennrich
5a3d5b6bdd EMS: LM:mock-parse can be actual parser 2015-04-21 10:21:24 +01:00
Hieu Hoang
95435f2a2e better detection of pigz 2015-04-20 20:40:50 +04:00
Hieu Hoang
eb37437d09 don't output warnings. It wasn't originally there before the 'env perl' change. This script should be tightened up at some point, eg use strict, debug warning messages 2015-04-20 16:18:51 +04:00
Hieu Hoang
1b9dc6cfae more butinah tweaks 2015-04-19 11:50:50 +04:00
Hieu Hoang
637e8a17e8 add pre tokenization cleaning script. In case training has bad, overlying long lines which blows up some taggers/segmenters, eg. mada 2015-04-19 11:21:07 +04:00
Hieu Hoang
6162223690 add use warnings to all perl scripts 2015-04-13 20:42:33 +04:00
Hieu Hoang
8190a5e1d6 Merge pull request #107 from flammie/master 
Finnish detokenisation
 2015-04-11 14:20:37 +04:00
Dingyuan Wang
4aba64ed53 Merge pull request #106 from gumblex/master 
Fix some problems in EMS
 2015-04-11 09:26:25 +08:00
Michael Denkowski
2682cc0f9b typo fix 2015-04-07 17:06:18 -04:00
Flammie Pirinen
fc8ee03b8d examples 2015-04-07 16:19:07 +01:00
Flammie Pirinen
ef52bc66f6 full set of cases and caps 2015-04-07 16:16:43 +01:00
Flammie Pirinen
85230e8334 add fi to list to silence warnings 2015-04-07 16:00:48 +01:00
Flammie Pirinen
f9deb6de3b also lowercase if case fail 2015-04-07 15:55:02 +01:00
Flammie Pirinen
5817806ec7 fix detokenising : in abbrev. case suffix case 2015-04-07 15:51:29 +01:00
Hieu Hoang
54e55f2dcb better detection of pigz, sort, split. In case they are not in the default directory 2015-04-06 11:31:44 +04:00
Hieu Hoang
02185a85fb store temp run files in current directory, not /tmp 2015-04-05 17:02:48 +04:00
Hieu Hoang
93ad52d2f9 leave in runPath for debugging 2015-04-05 16:49:12 +04:00
Hieu Hoang
4cb8a1837e Merge branch 'master' of github.com:moses-smt/mosesdecoder 2015-04-05 16:45:17 +04:00
Hieu Hoang
7ffdddef13 script to submit ems job to grid engine as 1 job. Hardcoded for NYUAD at the mo 2015-04-05 16:44:24 +04:00
Dingyuan Wang
aea07b0a19 Fix some problems in EMS: 
* remove absolute links
* fix coverage bar highlighting
* change Base64 library to support UTF-8
 2015-04-03 23:47:25 +08:00
Rico Sennrich
8d8097632b re-apply commit 1fb51dc (use gunzip -c instead of zcat) 
plus be more tolerant about xml input
 2015-04-03 15:00:45 +01:00
Hieu Hoang
b2f9ba2b64 revert last commit to add MASTER_PATH. Not needed 2015-04-02 19:29:42 +04:00
Hieu Hoang
27b36e0c96 pass in PATH variable from master node. When you're running of a grid but really just qsubbing everything to 1 slave node 2015-04-02 19:15:21 +04:00
Hieu Hoang
2d1da3219d consistently use 'env perl' command for environments where the 1st perl in PATH isn't the default perl. Which is kinda stupid 2015-04-02 17:38:56 +04:00
Hieu Hoang
035c806059 Merge branch 'master' of github.com:moses-smt/mosesdecoder 2015-04-02 14:10:42 +04:00
Hieu Hoang
e76247e19b Conditional import of Thread package for perl installations that don't support threads 2015-04-02 14:07:57 +04:00
Hieu Hoang
e22d275c32 don't ignore lowercasing of factored LM. Must be consistent with pt 2015-04-01 23:25:57 +04:00
Phil Williams
6ce3060dd8 lmplz-wrapper.perl: use Getopt::Long's "pass_through" option 
This avoids the need to duplicate all of lmplz's options in the wrapper and
it prevents --prune 0 0 1 from being truncated to --prune 0 if the user forgets
to quote the arguments.
 2015-03-30 10:18:51 +01:00
alvations
496a2a716c Merge branch 'master' of https://github.com/moses-smt/mosesdecoder 2015-03-23 15:55:56 +01:00
Matthias Huck
506427368f filter-model-given-input.pl: drop "-encoding None" from phrase table binaization with processPhraseTableMin. Recommended by Marcin. 2015-03-23 14:38:24 +00:00
Hieu Hoang
c4af7d28b5 Merge pull request #100 from alvations/master 
Added the Gacha Filter used in WMT14 by the Manawi system
 2015-03-21 22:42:55 +00:00
alvations
e5feb1a73e Enforce python3 and also remove extra empty newline from STDOUT 2015-03-20 19:37:34 +01:00
alvations
8f2d687d27 added more description of usage in docstring 2015-03-20 19:00:36 +01:00
alvations
44cd32d058 fixed typo in error message 2015-03-20 18:53:03 +01:00
alvations
93ea5853e8 Added Gacha Filter used by Manawi system from WMT14 translation task 2015-03-20 18:48:47 +01:00
alvations
5536e13213 added Gacha Filter from WMT14 2015-03-20 18:44:59 +01:00
Rico Sennrich
3a673fc8dc EMS: support for syntactic metrics for MERT/MIRA 
- add "-n-best-trees" to TUNING:decoder-settings
 - add "mock-output-parser-references = $output-parser" to GENERAL (and define output-parser)
 - TUNING:tuning-settings should include the metric you want to optimize (e.g. "-batch-mira-args='--sctype BLEU,HWCM'")
 2015-03-20 17:15:33 +00:00
Rico Sennrich
ca08b1d205 reduce-factors: port xml support from train-model.perl 2015-03-20 14:44:48 +00:00
Rico Sennrich
b8ca33c34e RDLM training without editing bash scripts 2015-03-20 14:12:41 +00:00
Rico Sennrich
2271f295e6 nplm_train: more options 2015-03-20 14:12:41 +00:00
Rico Sennrich
eab513b635 relational dependency language model 2015-03-18 17:39:45 +00:00
Phil Williams
fc15e03ebe Replace truecase-egret.sh with more general tree-converter-wrapper.perl 2015-03-18 09:57:42 +00:00
Phil Williams
ac51e9f0a8 Always use "SyntaxInputWeight0" as name of SyntaxInputWeight feature 2015-03-18 09:56:46 +00:00
Phil Williams
0a8e5fb3bf EMS: fix TRAINING:use-syntax-input-weight-feature option 2015-03-13 17:18:56 +00:00
Hieu Hoang
ce8b0e0876 fix example for reusing tuned moses.ini file 2015-03-13 15:07:23 +00:00
Phil Williams
05872cf32f Add tree-converter-mosesxml.sh wrapper script 2015-03-12 22:27:43 +00:00
Phil Williams
4685474e9b parse-en-egret.perl: wrap tree in parentheses prior to conversion to XML 2015-03-12 09:49:28 +00:00
Philipp Koehn
530d0f5a11 some more better defaults for recaser 2015-03-11 17:56:02 +00:00
Philipp Koehn
2ce45229f8 better default configuration for recaser 2015-03-11 17:52:30 +00:00
Philipp Koehn
1632c5f39d proper handling of specified configuration file 2015-03-11 16:49:20 +00:00
Matthias Huck
01bed83cf9 GHKM extraction: option to strip non-terminal labels from BitPar syntactic parses right during extraction (i.e., remove any suffix starting with a hyphen from the label) 2015-03-10 21:25:32 +00:00
Phil Williams
c7cf33ee05 parse-en-egret.perl: use "ROOT" instead of "TOP" as label of root tree node 
This is to match the label Egret assigns to the root vertices of forests.
 2015-03-10 15:43:14 +00:00
Phil Williams
77faaaea6c Add truecase-egret.sh 
This is currently just a wrapper for Travatar's tree-converter tool.
 2015-03-10 14:36:28 +00:00
Phil Williams
f7b4d403e3 Add parse-en-egret.perl wrapper script. 2015-03-10 14:32:59 +00:00
Phil Williams
9e2eb702dc EMS: add TRAINING:use-syntax-input-weight-feature option 2015-03-10 11:40:49 +00:00
Phil Williams
91abb69cdf train-model.perl: add -use-syntax-input-weight-feature option 
Currently only used for forest input.
 2015-03-10 11:39:14 +00:00
Phil Williams
7eba58b942 EMS: add TRAINING:dont-tune-glue-grammar option 
Adds -dont-tune-glue-grammar to train-model.perl command during config file
generation step.  This is preferable to manually adding -dont-tune-glue-grammar
to TRAINING:training-options because changing its value won't trigger a re-run
of dependent steps that don't really need re-running (like word alignment).
 2015-03-10 10:20:19 +00:00
Phil Williams
e79644540c train-model.perl: add -dont-tune-glue-grammar option 2015-03-10 09:53:12 +00:00
Phil Williams
fd3dcb7bb0 filter-model-given-input.pl: add -[no]StripXml and -SyntaxFilterCmd options 
-noStripXml is required for tree and forest input in STSG-based models.

-SyntaxFilterCmd can be used to set the command for filtering rule tables in
syntax-based models.  The default is to use

    $SCRIPTS_ROOTDIR/../bin/filter-rule-table

The option -MinNonInitialRuleCount is deprecated.
 2015-03-10 08:57:56 +00:00
Phil Williams
70bef90b36 train-model.perl: add -score-command option 
This matches the existing -extract-command option.  Given the argument value
<name>, train-model.perl will use the score program in

  $SCRIPTS_ROOTDIR/../bin/<name>

The default value is "score".
 2015-03-10 08:48:54 +00:00
Matthias Huck
25f5470216 GHKM: write target parts-of-speech as a factor 2015-03-09 21:54:03 +00:00
Hieu Hoang
cb2e1b8a40 separate variables into lines. Easier to merge with other branches 2015-03-05 21:37:30 +00:00
Hieu Hoang
0f5556f6d9 separate variables into lines. Easier to merge with other branches 2015-03-05 21:28:51 +00:00
Rico Sennrich
2431f514dd fix EMS bug from dca8dd: cleaning step was skipped 2015-03-05 10:55:35 +00:00
Rico Sennrich
47c460fe1d remove unused variable 2015-03-05 08:31:50 +00:00
Matthias Huck
638e9c3f60 POS property: map tags to indices in consolidate 2015-03-04 22:48:34 +00:00
Matthias Huck
06e87d851e GHKM: extract POS phrase property (from preterminals in the syntactic parse tree) 2015-03-04 21:40:56 +00:00
Rico Sennrich
ff5502d323 off-by-one error in previous commit 2015-03-04 17:25:19 +00:00
Rico Sennrich
71ab598435 extract_test.py should also create numberized corpus 2015-03-04 17:10:06 +00:00
Rico Sennrich
dca8ddc746 EMS convenience: 
- merge clean-corpus-n-ratio.perl and clean-corpus-n.perl (use variable 'cleaner' in EMS to call cleaning script with extra arguments)
  - use low default weight for glue rules in syntax systems (especially useful with 'tuneable=false')
 2015-03-04 14:43:05 +00:00
Rico Sennrich
f9ec387a5b typo 2015-03-04 10:06:03 +00:00
Phil Williams
90e8d4940c EMS: add TRAINING:no-glue-grammar option 2015-03-03 12:36:09 +00:00
Nicola Bertoldi
37b7162018 Merge branch 'master' of https://github.com/moses-smt/mosesdecoder 2015-02-28 22:09:41 +01:00
Rico Sennrich
e2b1ac1e9d fix option --return-best-dev with hypergraph MIRA (which I broke in commit d39cbca0b9) 2015-02-27 14:47:37 +00:00
Philipp Koehn
39c0068e4f discount_fallback for lmplz 2015-02-26 22:21:50 +00:00
Marcin Junczys-Dowmunt
a3d2adca50 Update filter-model-given-input.pl 
Added -encoding None to force single pass for compact phrase table so it works with pipes.
 2015-02-26 14:04:06 +01:00
Ondrej Bojar
441a2bb190 safer binarizer execution, bash, sort tempdir 2015-02-24 00:36:29 +01:00
Matthias Huck
8025cbf350 Merge branch 'master' of https://github.com/moses-smt/mosesdecoder 2015-02-16 15:10:15 +00:00
Nicola Bertoldi
ff4a103826 Merge branch 'master' of https://github.com/moses-smt/mosesdecoder 2015-02-13 14:52:31 +01:00
Barry Haddow
34b139e2ae Remove debug 2015-02-13 12:14:18 +00:00
Phil Williams
92a21f9d3a train-model.perl: fix "argument isn't numeric" warning 2015-02-13 11:55:39 +00:00
Phil Williams
7e54e23fe2 Update transliteration scripts to use the on-disk phrase table 
The scripts now use CreateOnDiskPt instead of processPhraseTable (which
is no longer supported and was removed by commit f3a84fc01).
 2015-02-13 11:36:16 +00:00
Kenneth Heafield
ee39fdbaa5 Relative path 2015-02-10 10:43:10 -05:00
Charley C
e40606d08f default path update in train-recaser 2015-02-09 18:36:31 -05:00
Matthias Huck
53ce063214 tuneable-components config parameter for feature functions 2015-02-09 13:52:05 +00:00
Philipp Koehn
f69c1dab02 more efficient default recaser training 2015-02-04 09:18:09 +00:00
Nicola Bertoldi
a1539505c8 minor change to make extract-parallel.perl compliant with MacOSX split command 2015-02-04 09:02:51 +01:00
Hieu Hoang
78f79632b9 script to convert moses.ini v2 to v1 /Tom Hoar 2015-02-03 10:59:38 +00:00
Kenneth Heafield
925565a0b9 "just put it in. I'll verify it if i can be bovvered" --Hieu /usr/bin/env 2015-01-29 18:37:05 -05:00
Matthias Huck
449d9b294b Revert "env perl shebang" 
This reverts commit 34f2801f8a.

Caused problems because /bin/env doesn't exist on Ubuntu 12.04.
/usr/bin/env does, though.
 2015-01-29 21:15:20 +00:00
Kenneth Heafield
34f2801f8a env perl shebang 2015-01-27 18:35:54 -05:00
XapaJIaMnu
6ca1a4718c Expose learning rate as a parameter 2015-01-25 02:13:47 +00:00
Matthias Huck
9987beb453 SoftSourceSyntacticConstraintsFeature: Now for both non-terminals (as before) _and_ terminals. 
Also added score components based on relative frequency.
(TODO: logprobs right now; are plain probabilities better?)
 2015-01-23 18:41:18 +00:00
Hieu Hoang
59c4baec3f use utf8 german model 2015-01-22 16:10:12 +00:00
Kenneth Heafield
7c507bfa74 May is not an abbreviation 2015-01-19 16:37:57 -05:00
Hieu Hoang
30e31d4a95 don't normalise quotes if tokenizing like Penn /Phil Williams 2015-01-16 12:34:22 +00:00
Hieu Hoang
19d7c44aad move normalisation of quotes into normalize-punctuation.perl /Tom Hoar 2015-01-16 11:37:31 +00:00
Hieu Hoang
6d61db28fa use astyle 2.01. It's on Edinburgh server and doesn't screw up enum 2015-01-14 19:21:11 +00:00
Hieu Hoang
90d4b2d713 use pigz rather than gzip if it exists 2015-01-13 15:16:22 +00:00
Hieu Hoang
6186262a3b don't use processPhraseTable in EMS 2015-01-12 12:43:51 +00:00
Hieu Hoang
a8d4b81e71 Revert "Update train-model.perl" 
This reverts commit e1e14a91ee.
 2015-01-08 16:07:40 +00:00
Hieu Hoang
5336598734 beatify 2015-01-08 08:29:56 +00:00
Philipp Koehn
0441fd6ab9 added informative error message when trying to build a lexicalized reordering model with hierarchical model 2015-01-06 18:46:02 +00:00
Hieu Hoang
0a707597d8 Revert "Added error message on experiment.meta for the filter step 'No phrases in'" 
This reverts commit 2105423626.
 2015-01-03 21:58:15 +05:30
Eleftherios Avramidis
2105423626 Added error message on experiment.meta for the filter step 'No phrases in' 2014-12-28 18:09:33 +01:00
Philipp Koehn
59fdb3d99c same spec for dedicated script as for train-model.perl and filter-model-given-input.pl 2014-12-21 01:37:05 +00:00
Philipp Koehn
831f947874 long overdue feature: do not produce very low scoring translation table entries that are never used and just gum up the works 2014-12-21 01:14:42 +00:00
Rico Sennrich
67e101b07a Revert "Update train-model.perl" 
This reverts commit 41f06a01c0.
 2014-12-17 17:51:02 +00:00
Rico Sennrich
685f18ca1b documentation/readability 2014-12-16 17:42:17 +00:00
Nicola Bertoldi
d0cddf0f2d Merge branch 'master' of https://github.com/moses-smt/mosesdecoder 2014-12-16 17:35:47 +01:00
Nicola Bertoldi
4e77665d30 better handling of cache-based models with inconsistent parameters 2014-12-15 17:42:41 +01:00
Xiang Li
41f06a01c0 Update train-model.perl 
If the final alignment model is model 3-5, the hmm model will be trained.
 2014-12-16 00:37:15 +08:00
Nicola Bertoldi
e4eb201c52 merged master into dynamic-models and solved conflicts 2014-12-13 12:52:47 +01:00
Hieu Hoang
5ae5a630a6 Merge branch 'master' of github.com:moses-smt/mosesdecoder 2014-12-12 10:04:58 +00:00
Kenneth Heafield
8bbccd441a Fix #85 by changing the default LM. Hieu said it's ok in the issue. 2014-12-11 23:51:48 -05:00
Hieu Hoang
c48a3aadc1 chmod 2014-12-11 16:54:19 +00:00
Hieu Hoang
765d8d1350 Merge pull request #83 from lixiangnlp/patch-1 
Update train-model.perl
 2014-12-10 15:48:35 +00:00
Phil Williams
1353aa57dc experiment.meta: fixes for $input-parse-relaxer 2014-12-08 16:26:08 +00:00
Phil Williams
60e56efc6b phrase-extract: add syntax-common sub-library 
And remove some (near-)duplicate code from pcfg-common and score-stsg.
 2014-12-07 14:27:51 +00:00
Kenneth Heafield
f97ed79a70 Month abbreviations shouldn't be causing a sentence split. 
Yes this will break existing tokenized data :-(.
 2014-12-05 03:41:01 -05:00
Philipp Koehn
9d55ce13c0 change for thot integration 2014-12-02 14:05:56 -05:00
Xiang Li
e1e14a91ee Update train-model.perl 
The default hmm iterations of GIZA++ is 5. Even though the "hmm-align" option is not set. The hmm align is also activated when using the training script.
 2014-12-01 11:26:53 +08:00
Rico Sennrich
4ca730a67c improve bilingualLM alignment heuristics consistency 2014-11-26 10:32:41 +00:00
Rico Sennrich
ee759bfede move bilingual-lm training scripts 2014-11-26 10:32:37 +00:00
Tomáš Musil
4cb81e3093 lmtype now preferred as symbolic name 2014-11-24 12:20:36 +01:00
Hieu Hoang
c0be182bfa makemteval and small change to tokenizer. /Tom Hoar and Tomas Fulajtar 2014-11-21 13:55:13 +00:00
XapaJIaMnu
52c520c042 Resolve merge conflicts 2014-11-20 15:50:32 +00:00
Hieu Hoang
e27f6b0120 Merge branch 'master' of github.com:moses-smt/mosesdecoder 2014-11-15 14:32:49 +00:00
Hieu Hoang
67ad197d5a take out PYTHONIOENCODING=utf-8. Rely on Rico's python changes 2014-11-15 14:32:31 +00:00
XapaJIaMnu
a343837095 Add option to choose activation function during nplm training 2014-11-15 11:54:47 +00:00
Rico Sennrich
b0b5eef0c6 fix metric interpolation with mert 2014-11-14 14:35:32 +00:00
Hieu Hoang
acd3ac964a set PYTHONIOENCODING=utf-8 before running merge_alignment.py 2014-11-14 14:34:31 +00:00
Hieu Hoang
1c27e05a06 softlink for moses_chart 2014-11-14 13:56:56 +00:00
XapaJIaMnu
d5567b6cfb Training: Do the preparation step ourselves. No validation support yet. No decoder support yet. 2014-11-13 16:14:17 +00:00
Rico Sennrich
8fd3be9e4e add EOS token </s> to each sentence 2014-11-13 16:14:16 +00:00
Rico Sennrich
f26fc251d5 sort vocab by frequency 2014-11-13 16:14:16 +00:00
XapaJIaMnu
bb70f60f67 grrr 2014-11-13 16:14:16 +00:00
XapaJIaMnu
e330ab35d5 Short option must be only one letter 2014-11-13 16:14:16 +00:00
XapaJIaMnu
a74105ea7d Fix a wrong condition 2014-11-13 16:14:16 +00:00
XapaJIaMnu
e54c171850 Make it optional to prepare the validation set 2014-11-13 16:14:16 +00:00
XapaJIaMnu
a300824bd1 Add optional validation during training 2014-11-13 16:14:16 +00:00
XapaJIaMnu
0451142ece Add null token normalization for models to be used with the chart decoder. 2014-11-13 16:13:38 +00:00
XapaJIaMnu
aae894fe6b Add null token in vocabulary during construction 2014-11-13 16:13:38 +00:00
XapaJIaMnu
b4f51c05d1 Add option to reduce the ngrams from already prepared .ngrams file to train a model with smaller number of ngrams 2014-11-13 16:13:38 +00:00