moses-smt/mosesdecoder
1
1
Fork 0
mirror of https://github.com/moses-smt/mosesdecoder.git synced 2024-08-16 23:10:31 +03:00
Code Issues Packages Projects Releases Wiki Activity
17,477 Commits 136 Branches 26 Tags 147 MiB
master
Commit Graph

2352 Commits

Author SHA1 Message Date
Matthias Huck
c731851b92 integration of TargetPreferencesFeature in EMS create-config step 2016-01-27 17:54:57 +00:00
Hieu Hoang
38f999fa3f Merge ../mosesdecoder into perf_moses2 2016-01-13 14:57:20 +00:00
Matthias Huck
7764d86fcb tiny changes related to feature functions 2016-01-12 19:44:43 +00:00
Hieu Hoang
d0a48e71ad Merge ../mosesdecoder into perf_moses2 2016-01-12 09:23:07 +00:00
Matthias Huck
885b8b33a1 preparing extraction of Hiero soft syntactic preferences (target syntax) 2016-01-11 20:04:32 +00:00
Hieu Hoang
2c78a26b74 Merge ../mosesdecoder into perf_moses2 2016-01-10 13:21:53 +00:00
Matthias Huck
1d3feba8d0 preparing extraction of Hiero soft syntactic preferences (target syntax) 2016-01-09 23:02:31 +00:00
Hieu Hoang
bf19d71780 Merge ../mosesdecoder into perf_moses2 2016-01-06 21:03:33 +00:00
Barry Haddow
977e8eaf67 Merge branch 'master' of github.com:moses-smt/mosesdecoder 2016-01-06 11:55:16 +00:00
Barry Haddow
7125096c29 enable nplm training on separate host, fix ems for nplm 2016-01-06 11:55:12 +00:00
Hieu Hoang
f179072902 cpu affinity offset 2016-01-04 00:19:35 +00:00
Hieu Hoang
589f0d97f0 cpu affinity offset 2016-01-04 00:05:17 +00:00
Hieu Hoang
abf41d04e9 cpu affinity offset 2016-01-03 23:22:50 +00:00
Matthias Huck
0a39efb6c8 Hiero phrase orientation: modify some parameters 2015-12-18 17:24:42 +00:00
Matthias Huck
bd3f573452 Hiero phrase orientation 2015-12-10 12:56:37 +00:00
Philipp Koehn
33f4e93915 no binarizing/filtering with mmsapt 2015-12-01 23:10:37 +00:00
Ulrich Germann
4c78e7c0b2 scripts/generic/bsbleu.py can now handle gzipped files 2015-11-29 17:57:57 +00:00
Ulrich Germann
c8b859de67 Merge remote-tracking branch 'legacy/master' 
Conflicts:
	moses/server/TranslationRequest.cpp
 2015-11-24 19:22:37 +00:00
Jeroen Vermeulen
710915c088 Python implementation of parallel scoring. 
Re-implementation of score-parallel.perl.  Not a drop-in replacement;
the command line is similar but different and uses the standard Python
command-line parser.

Written without much knowledge of the original script, so documentation
in particular may seem nonsensical to experts.  If you see something
wrong, please help!
 2015-11-24 14:37:18 +01:00
Philipp Koehn
94cd1f7433 when building mmsapt phrase table, also use mmsapt reordering table 2015-11-23 18:12:56 -05:00
Barry Haddow
10df006eed Merge branch 'master' of github.com:moses-smt/mosesdecoder 2015-11-23 12:16:46 +00:00
Michael Denkowski
b002fade50 Minimal buffering for multi_moses.py 
Speeds things up when using multi-threaded instances
 2015-11-18 13:54:54 -05:00
Barry Haddow
21d8111287 Merge branch 'master' of github.com:moses-smt/mosesdecoder 2015-11-18 09:50:29 +00:00
David Madl
e36fb96557 LanguageModel, KenLM: avoid StaticData usage 
* drop global lmodel-oov-feature option, and add it to LM FF config line instead
	use oov-feature=1 (bool) option instead
* drop LanguageModel::GetWeight()
* KenLM: use m_verbosity of FF instead of IFVERBOSE macro which uses StaticData

* train-model.perl: move language model OOV feature onto LM feature spec line
 2015-11-17 16:15:13 +00:00
Barry Haddow
ccfe8ba018 remove unused method, and misleading comment 2015-11-10 21:35:08 +00:00
Ulrich Germann
ec71c2397b Allow multiple reference files to be specified on the command line; handle gzipped reference files. 2015-11-10 01:16:17 +00:00
Phil Williams
6a37dfd2ce Add a wrapper script for parsing English text with the BLLIP parser 2015-10-26 16:18:54 +00:00
Barry Haddow
d59bb883dc fix extra settings 2015-10-21 09:22:00 +01:00
Hieu Hoang
384e0b06d2 revert Greg's error handling commit. Cruise control breaks with message 
sh: 1: cannot create /dev/stderr: Permission denied
 2015-10-14 16:42:44 +01:00
Ubuntu
d6dda4f292 explicitly use bash /Uli Germannn 2015-10-13 13:40:48 +00:00
Ubuntu
81d106e74d Revert "Revert "Be more aware of errors in subprocesses"" 
This reverts commit 1fca3f8a75.
 2015-10-13 13:33:40 +00:00
Ubuntu
1fca3f8a75 Revert "Be more aware of errors in subprocesses" 
This reverts commit f6894f4623.
 2015-10-13 12:53:02 +00:00
Matthias Huck
62748f5296 Revert "EMS: fix filtering issue when output-splitter is defined" 
This reverts commit d5c41634e8.
 2015-10-12 18:05:46 +01:00
Tomáš Fulajtár
dd9eb54ec4 Named group added for the safer 'protected patterns' recognition regexp. 
In the original code there are the number references used , which might actualy colidate if any group is used inside the $protected_pattern string.  for example the protected_pattenr (loaded from file ) :  (http[s]?|ftp):\/\/[^:\/\s]+(\/\w+)*\/[\w\-\.]+.

If we use the number reference, the $2 will reffer to (http[s]?|ftp):, instead to  (.*) inside the :
 while ($t =~ /($protected_pattern)(.*)$/) {

Naming patterns resolves this issue.
 2015-10-12 18:47:45 +02:00
Tomáš Fulajtár
83e25a3f5e Merge pull request #1 from moses-smt/master 
update my local fork
 2015-10-12 18:37:26 +02:00
Greg Hanneman
f6894f4623 Be more aware of errors in subprocesses 2015-10-09 16:05:25 +00:00
Michael Denkowski
923f512be0 Extend multi_moses.py to support multi-threaded moses instances 2015-10-08 11:50:34 -04:00
Nadir
15b4aa91b0 Extra Space 2015-10-08 13:16:58 +01:00
Nadir
965aeb9012 Interpolated OSM - Bug Fix 2015-10-08 12:03:09 +01:00
Nadir
2ec6fed898 Interpolated OSM 2015-10-07 13:57:32 +01:00
Michael Denkowski
35538bf894 Actually override threads specified in moses.ini 2015-10-06 15:29:42 -04:00
Rico Sennrich
c0fedd275b change default nplm setting to 1 hidden layer 2015-10-06 11:49:45 +01:00
Michael Denkowski
160a7f254b Parallelization with multiple instances of moses 2015-10-02 18:19:43 -04:00
Barry Haddow
bb1b5d3abd config of dropout 2015-09-25 13:53:39 +01:00
Hieu Hoang
1f5ec65f3c Merge pull request #129 from jimregan/patch-1 
Basic tokenizer support for Irish (ga)
 2015-09-24 16:07:17 +01:00
Hieu Hoang
d81dfda511 option to cat model files 2015-09-24 09:00:54 -04:00
Jim Regan
36e4951134 ga (mostly) behaves more like fr/it 2015-09-23 14:36:57 +01:00
Jim Regan
dfe682d823 Create nonbreaking_prefix.ga 2015-09-23 14:35:18 +01:00
Jim Regan
09a9f1b061 ga (mostly) behaves more like fr/it 2015-09-23 14:33:18 +01:00
Pierre Lison
31cc22cf14 redirecting output of which and split --help to /dev/null 2015-09-16 12:56:25 +02:00
Pierre Lison
e26bbc215f sending the stderr output of the which command to /dev/null 2015-09-16 11:01:22 +02:00
Ulrich Germann
2e3a82a40c Merge pull request #125 from akivajp/master 
Fixed for removed option building cooccurrence table.
 2015-09-12 11:32:27 +02:00
Akiva Miura
7b15548144 Fixed for removed option building cooccurrence table. 2015-09-07 06:21:04 +09:00
Barry Haddow
90f15cc619 extra nplm settings 2015-09-04 10:07:50 +01:00
Barry Haddow
4746970bf8 Merge branch 'master' of github.com:moses-smt/mosesdecoder 2015-09-04 10:06:58 +01:00
Barry Haddow
e58ddf74e8 parameter changes 2015-09-04 10:06:54 +01:00
Tomáš Fulajtár
1a26cb8414 Added a simple support for the factored systems. 2015-08-27 15:15:32 +02:00
Hieu Hoang
d349bf8a94 dos2unix everything 2015-08-23 19:00:19 +04:00
Matthias Huck
d5c41634e8 EMS: fix filtering issue when output-splitter is defined 2015-08-21 18:58:36 +01:00
Matthias Huck
261cfdb024 perl shebang 2015-08-19 16:59:10 +01:00
Hieu Hoang
4ff776f564 dos2unix the whole lot 2015-08-19 16:30:31 +04:00
Hieu Hoang
97eed02301 dos2unix. Revert Matthias' change, use env 2015-08-19 13:25:29 +01:00
Matthias Huck
d583e0888f make-factor-de-pos.perl 2015-08-18 18:31:41 +01:00
Hieu Hoang
3a261c9fc9 don't hardcode amount of mem to be used by lmplz 2015-08-16 20:32:07 +04:00
Phil Williams
01a9dd2305 extract-target-trees.py: support for new-style trace files 2015-08-14 16:53:24 +01:00
Ulrich Germann
883c34aee9 Merge branch 'master' of http://github.com/moses-smt/mosesdecoder into mmt-dev 
Conflicts:
	moses/SearchNormalBatch.cpp
	moses/TranslationModel/UG/mm/ug_bitext.h
	moses/TranslationModel/UG/mm/ug_typedefs.h
	moses/TranslationModel/UG/mmsapt.cpp
	moses/TranslationModel/UG/mmsapt.h
 2015-08-07 14:14:19 +01:00
Barry Haddow
4c3a6a3f3f remove dash 2015-08-03 21:19:08 +01:00
Barry Haddow
57b0c351c0 Merge branch 'master' of github.com:moses-smt/mosesdecoder 2015-08-03 16:47:31 +01:00
Barry Haddow
f808b32030 support version of nplm that picks best on heldout 2015-08-03 16:47:25 +01:00
Barry Haddow
a39544bbcb fix inconsistency in the example 2015-08-03 16:46:57 +01:00
Hieu Hoang
bfd45fdfc3 don't use all threads 2015-08-01 11:35:47 +04:00
Hieu Hoang
7ac6f90a4d Merge branch 'master' of github.com:moses-smt/mosesdecoder 2015-07-31 22:29:07 +04:00
Hieu Hoang
f894dec0fd multi-threaded decoding by default /Vincent Nguyen 2015-07-31 22:28:45 +04:00
Rico Sennrich
89d16a491a fix ems regression (concatenate-split step) 2015-07-31 11:20:29 +01:00
Philipp Koehn
a5ee3c1b6d script to copy model files to local disk before running the decoder - useful for grid 2015-07-29 11:10:13 -04:00
Philipp Koehn
836ca8212a better support of grid engine cluster 2015-07-29 11:03:24 -04:00
Philipp Koehn
b29166e2fe fix of fix 2015-07-29 11:02:55 -04:00
Philipp Koehn
837bcde69d bug with legacy 2015-07-29 10:46:09 -04:00
Philipp Koehn
ae9cd14948 fixes 2015-07-29 10:44:57 -04:00
Hieu Hoang
5f4b497b63 Merge pull request #119 from cidermole/master 
Fix 'Use of uninitialized value' error through explicit setting of 0s…
 2015-07-29 16:25:39 +04:00
Barry Haddow
3a2116b2c9 add quotes so arguments don't get lost 2015-07-29 09:35:19 +01:00
Rico Sennrich
0173512ddc separate xml version of combine_factors.pl 
xml version causes slow-down for users who use factors, but not xml (which are most)
 2015-07-28 23:30:18 +01:00
Phil Williams
2cda286a06 experiment.meta: re-run fast_align symmetrization if symmetrization type changes 2015-07-28 16:55:55 +01:00
Rico Sennrich
a968536176 ems fix: pass-unless doesn't understand AND 2015-07-28 16:37:50 +01:00
Ulrich Germann
d67723fd29 Merge branch 'master' of http://github.com/moses-smt/mosesdecoder into ranked-sampling 
Conflicts:
	moses/TargetPhrase.cpp
	moses/TargetPhrase.h
 2015-07-28 14:29:49 +01:00
Barry Haddow
e53ad40859 Support for nplm in ems 2015-07-23 10:37:26 +01:00
Philipp Koehn
1a795f549e only extract reordering phrase pairs if use mmsapt phrase table 2015-07-20 11:47:24 -04:00
Philipp Koehn
777a88673d compress sort tmp files by default 2015-07-20 11:46:47 -04:00
Philipp Koehn
496f8c6d85 only extract reordering phrase pairs if use mmsapt phrase table 2015-07-20 11:44:22 -04:00
Philipp Koehn
fcf2934a2f customized phrase table pruning step 2015-07-20 11:43:02 -04:00
Rico Sennrich
7b19d83d43 xml support for combine-factors.pl 2015-07-20 10:45:18 +01:00
Rico Sennrich
1b1bafb1e8 ems: add option to factorize after truecase/split/etc. 2015-07-20 10:43:23 +01:00
Rico Sennrich
e85f353898 code simplification by removing language-specific, unused hack. 2015-07-20 10:39:01 +01:00
Ulrich Germann
df50e454d2 Proper handling of moses parameters with double dash in create_config(). 2015-07-17 14:32:30 +01:00
Phil Williams
c83628a92b Fix errors from multiline `` commands in transliteration Perl scripts 
Replace the backslash-newline sequence with backslash-backslash-newline in
multiline backquote command strings.  i.e. replace expressions like this:

  `some-command \
    -option1 \
    -option2`;

with ones like this

  `some-command \\
    -option1 \\
    -option2`;

If I understand this right, the shell converts a backslash-newline sequence
to an empty string (i.e. it discards it), but Perl does not.  Unless the
backslash itself is escaped, using a backslash-newline in a Perl command
string results in errors in most instances.  By escaping the backslash, it
gets passed through to the shell where it is interpreted as intended.
 2015-07-16 14:54:00 +01:00
Philipp Koehn
66ecf98cf7 minor bug fix 2015-07-14 11:01:22 -04:00
Rico Sennrich
ca72105fdf fix ems regression 2015-07-14 13:16:25 +01:00
David Madl
3c30210dad Fix 'Use of uninitialized value' error through explicit setting of 0s in hash. 
Fixes the following errors in bootstrap-hypothesis-difference-significance.pl on Perl v5.14.2:

Use of uninitialized value $coocUpd in numeric gt (>) at /fs/lofn0/dmadl/software/mosesdecoder/scripts/analysis/bootstrap-hypothesis-difference-significance.pl line 317.
Use of uninitialized value $b in numeric lt (<) at /fs/lofn0/dmadl/software/mosesdecoder/scripts/analysis/bootstrap-hypothesis-difference-significance.pl line 543.
Use of uninitialized value $coocUpd in addition (+) at /fs/lofn0/dmadl/software/mosesdecoder/scripts/analysis/bootstrap-hypothesis-difference-significance.pl line 314.
Use of uninitialized value $coocUpd in numeric gt (>) at /fs/lofn0/dmadl/software/mosesdecoder/scripts/analysis/bootstrap-hypothesis-difference-significance.pl line 317.
Use of uninitialized value $a in numeric gt (>) at /fs/lofn0/dmadl/software/mosesdecoder/scripts/analysis/bootstrap-hypothesis-difference-significance.pl line 552.
 2015-07-14 13:05:22 +01:00
Philipp Koehn
7e3050f7f2 allow saving of model from fast-align (for incremental use) 2015-07-14 05:27:03 -04:00
Barry Haddow
3fdbb00904 Improvements to handling of bilingual LM in EMS 2015-07-10 15:44:24 +01:00
Hieu Hoang
f66beabf4f Generation error in EMS due to pruning. Lets see if this works. 2015-06-28 14:03:54 +04:00
Hieu Hoang
82edbb98a7 comments in ini file about default weights 2015-06-28 10:40:43 +04:00
Hieu Hoang
57e213ed19 tighten up extract-parallel on osx. Can now use gsplit and bsd split 2015-06-26 12:18:21 +04:00
Hieu Hoang
ca54852641 tighten up extract-parallel on osx. Can now use gsplit and bsd split 2015-06-26 11:55:24 +04:00
Hieu Hoang
b83803203e prune generation table in ems 2015-06-25 18:10:31 +04:00
Hieu Hoang
dce0f33270 prune generation table in ems 2015-06-24 18:35:59 +04:00
Barry Haddow
425118aa5d bugfixes - working directory 2015-06-17 09:32:29 +01:00
Barry Haddow
ad8114ddb0 capitalisation 2015-06-15 16:23:12 +01:00
XapaJIaMnu
166bf7365f Forgot to update the weight config path 2015-06-12 16:56:36 +01:00
XapaJIaMnu
ffd3f2bb6e Added basic BilingualNPLM support to EMS and an example config. 2015-06-12 16:21:24 +01:00
Jeroen Vermeulen
dbcc264506 Remove unneeded script. 
Tom Hoar, the author of this script, asked me to remove it because it
doesn't actually do what the current name says, and can't work without
an additional script which isn't in the repository.
 2015-06-09 23:10:27 +07:00
Lexi Birch
b76194a16b Merge branch 'master' of https://github.com/moses-smt/mosesdecoder 2015-06-08 17:13:00 +01:00
Lexi Birch
501c51947b Allowing the truecaser to work on uncased ASR input, pass the -a flag 2015-06-08 16:58:50 +01:00
Jeroen Vermeulen
85c23ed7dc Fix some JS lint. 2015-06-02 18:05:12 +07:00
Jeroen Vermeulen
b3e577be76 Fixing lint. Only 600 or so lines of errors left! 2015-06-02 17:29:32 +07:00
Jeroen Vermeulen
0981d23705 Lint-fixing binge. 2015-06-02 16:02:39 +07:00
Rico Sennrich
5d8af9c289 support memory-mapped files for NPLM training 2015-05-29 16:07:26 +01:00
Jeroen Vermeulen
ef028446f3 Add license notices to scripts. 
This is not pleasant to read (and much, much less pleasant to write!) but
sort of necessary in an open project.  Right now it's quite hard to figure
out what is licensed how, which doesn't matter much to most people but can
suddenly become very important when people want to know what they're being
allowed to do.

I kept the notices as short as I could.  As far as I could see, everything
without a clear license notice is LGPL v2.1 or later.
 2015-05-29 18:30:26 +07:00
Jeroen Vermeulen
26170a4179 Friendlier error reporting in beautify.py. 2015-05-29 09:37:37 +07:00
Barry Haddow
c27aa193ea Revert "Min score parameter". Doesn't work without filter. 
This reverts commit ab2d396781.
 2015-05-28 17:44:26 +01:00
Barry Haddow
ab2d396781 Min score parameter 2015-05-28 17:10:21 +01:00
Phil Williams
842fc9780e senna2brackets.py: bug fixes + clean-up 2015-05-27 20:33:43 +01:00
Phil Williams
c086a8ee50 Add a wrapper script for parsing English text with SENNA 2015-05-26 16:44:13 +01:00
Rico Sennrich
f6f56d11af ems: parse-relax comes last in train; do same for dev/test 2015-05-25 15:52:07 +01:00
Hieu Hoang
582a845524 don't use zcat 2015-05-24 20:04:01 +04:00
Rico Sennrich
43527c82fc training script for monolingual Neural LM 
(+bugfixes and usability improvements for RDLM training)
 2015-05-22 15:31:08 +01:00
Rico Sennrich
a1678187fe wrapper for stanford dependency parser 2015-05-22 15:28:42 +01:00
Rico Sennrich
98ff2382d0 duplication of existing functionality 2015-05-20 17:35:38 +01:00
Rico Sennrich
6aac7ded9a EMS: more flexible way to concatenate LM training data. 
the implementation allows the user to specify which corpora to combine,
and to have multiple LMs on the same data.
 2015-05-20 17:20:02 +01:00
Hieu Hoang
36caf2eb9a escape ^# character otherwise morfessors skips line 2015-05-20 15:28:28 +04:00
Hieu Hoang
79ca96db0a should have tested this 2015-05-19 22:19:11 +04:00
Hieu Hoang
59071bf16c run on all cores if number of cores not given 2015-05-19 18:32:31 +04:00
Rico Sennrich
8ca6764c7d ems: allow LMs with user-specified training commands and moses.ini config entries 
intended for neural LMs, syntactic LMs, and the like. currently doesn't play nice with INTERPOLATED-LM.
 2015-05-18 19:07:37 +01:00
Rico Sennrich
fb06a2325e fix broken ems with interpolated lm disabled 2015-05-18 17:26:09 +01:00
Rico Sennrich
f85dd85f6b ignore-unless magic 2015-05-18 16:17:33 +01:00
Rico Sennrich
59376f500b still confused about pass-unless vs. ignore-unless 2015-05-18 14:40:56 +01:00
Rico Sennrich
45a97f9016 EMS: disable concatenated LM by default 2015-05-18 14:10:29 +01:00
Hieu Hoang
2c0aecb16b Merge branch 'master' of github.com:moses-smt/mosesdecoder 2015-05-18 16:27:03 +04:00
Hieu Hoang
2f0ee5502e delete debugging info 2015-05-18 16:26:26 +04:00
Rico Sennrich
27fd45d088 ems: training LM on concatenation of all LM training corpora 2015-05-18 12:18:49 +01:00
Hieu Hoang
14d2a67193 Merge branch 'master' of github.com:moses-smt/mosesdecoder 2015-05-18 12:28:35 +04:00
Hieu Hoang
5fdcf372ae rename a python file to have have .py, instead of .perl. In case beautify script depends on it 2015-05-18 12:27:35 +04:00
Jeroen Vermeulen
5aa70c6cdd Also reformat Perl, using Perltidy. 2015-05-18 00:45:15 +07:00
Jeroen Vermeulen
494c20f634 Add note about Perltidy. 2015-05-17 22:48:03 +07:00
Jeroen Vermeulen
e2a632a2b8 JavaScript lint. 2015-05-17 21:36:07 +07:00
Jeroen Vermeulen
5d0bbb6a45 Fix some JavaScript lint. Still a lot left. 2015-05-17 21:24:04 +07:00
Jeroen Vermeulen
a25193cc5d Fix a lot of lint, mostly trailing whitespace. 
This is lint reported by the new lint-checking functionality in beautify.py.
(We can change to a different lint checker if we have a better one, but it
would probably still flag these same problems.)

Lint checking can help a lot, but only if we get the lint under control.
 2015-05-17 20:04:04 +07:00
Jeroen Vermeulen
108da16374 Suppress CSS lint checking; accept longer lines. 
The CSS style that Pocketlint expects is just too different from what we
have.  Don't check those files for now.

Also, a maximum line length of 300 still gives too many warnings, so I'm
regretfully dumping the default to 400 characters.  The traditional 80
characters are already longer than the measured optimum for human reading,
so I hope some day we can address this!
 2015-05-17 20:03:27 +07:00
Jeroen Vermeulen
07a8fe06aa Also support checking for lint. 
Choose which action(s) you want for each run: --format and/or --lint.

Many different types of files are lint-checkable, but you need Pocketlint
installed (plus ideally, its plugins for the various languages).

Also, added option to control batching of the commands.
 2015-05-17 18:25:06 +07:00
Jeroen Vermeulen
9bdcb5f7c1 Fix more Python lint. 
This is about the last that isn't in contrib or generated files.  At this
point we can start doing regular lint checks, at least on the Python files,
without being completely inundated with warnings.
 2015-05-16 18:03:54 +07:00
Jeroen Vermeulen
61162dd242 Fix more Python lint. 
Most of the complaints fixed here were from Pocketlint, but many were also
from Syntastic the vim plugin.
 2015-05-16 17:26:56 +07:00
Jeroen Vermeulen
0ffe79579e Fix some python lint. 
I used mainly pocketlint, a very good Python linter, but also Syntastic,
a vim plugin.  Didn't get anywhere near fixing all of Syntastic's complaints
though.

Once I've cleaned up all (or at least most) of the Python lint, we can
start doing regular automated lint checks and keep the code clean.
 2015-05-16 14:58:03 +07:00
Jeroen Vermeulen
f1ed14eb33 Move ignored path prefixes into config file. 
The path prefixes listed in .beautify-ignore, in the project root, will not
be cleaned up.  C and C++ files everywhere else will be.

Also fixes bugs in the prefix-matching code, and makes the matching a little
bit more powerful: the prefix can now extend down into the directory tree.
 2015-05-15 16:40:06 +07:00
Jeroen Vermeulen
d7599134b8 Remove the --any-astyle option. 
The risk from flip-flopping on styles is too great: 2.04 formatting has
many changes from 2.01 formatting.

Also, fix a broken version check.
 2015-05-15 15:24:54 +07:00
Jeroen Vermeulen
3e821b56dd Rewrite beautify script in Python. 
The new version is much longer, but hopefully extensible, reusable, and
easy to read.  With a few more changes it will let us do three things:

1. Apply more checks and cleanups.
2. Clean up additional file types.
3. Use the same script for mgiza++, so we get a uniform code style.

The "more cleanups" could be more things like the removal of trailing
whitespace which we just added.  We may even want to run lint checkers of
some sort.

Additional file types could be e.g. Perl scripts, build configuration,
or documentation.

In order to make the script reusable we'll have to generalize the
skip_at_root list in list_c_like_files into something like a configuration
file.
 2015-05-15 13:46:00 +07:00
Hieu Hoang
5173b9f617 beautify. Add sed for trailing spaces 2015-05-13 11:29:16 +01:00
Hieu Hoang
87e1f1351f tighten up OSM build. More debugging output, to stderr not stdout. lmplz uses outdir as temp directory 2015-05-13 12:29:56 +04:00
Hieu Hoang
0cd62488bf morfessor wrapper 2015-05-12 20:40:19 +04:00
Hieu Hoang
abfc0671a3 osm tweaks and morfessor wrapper 2015-05-12 20:19:39 +04:00
Hieu Hoang
a922245864 default to using lmplz for convenience and because SRILM uses tonnes of memory 2015-05-12 11:44:05 +04:00
Hieu Hoang
03e507f354 default to using lmplz for convenience and because SRILM uses tonnes of memory 2015-05-12 10:40:52 +04:00
Hieu Hoang
8bb18b9ff0 add no-splitter-training argument. Splitter to be used by mada 2015-05-11 15:26:50 +04:00
Barry Haddow
85c1af4d72 Merge branch 'master' of github.com:moses-smt/mosesdecoder 2015-05-08 09:16:55 +01:00
Barry Haddow
f403f5e478 mmsapt doesn't require feature weights on first tuning iteration 2015-05-08 09:16:51 +01:00
Hieu Hoang
2acb590394 output bleu for multi-bleu hack 2015-05-05 17:54:35 +04:00
Hieu Hoang
d006c6ef8c don't output remaining args twice 2015-05-05 12:15:08 +04:00
Hieu Hoang
5fefb0da47 Merge branch 'master' of github.com:moses-smt/mosesdecoder 2015-05-05 12:02:13 +04:00
Hieu Hoang
8f272e04a9 output debugging messages to stderr, not stdout 2015-05-05 12:01:21 +04:00
Nicola Bertoldi
90a982e579 merge remote into local 2015-05-04 09:42:44 +02:00
Hieu Hoang
d456d9229e add multi-bleu-detok. Like multi-bleu scoring but will detokenize/post-process before scoring 2015-05-03 14:07:12 +04:00
Hieu Hoang
e5f76ee99e Merge branch 'master' of github.com:moses-smt/mosesdecoder 2015-05-03 11:50:31 +04:00
Hieu Hoang
73ae7d7e20 option not to use parallel 2015-05-03 11:50:10 +04:00
Philipp Koehn
a4a7c14593 allow breaking up training data for fast align (to avoid memory blowups for very large corpora) 2015-05-01 17:47:08 -04:00
Philipp Koehn
de6a9bd1b3 minor updates to factor scripts; brown-cluster may now run other scripts (e.g., truecaser) before assigning classes 2015-05-01 17:46:14 -04:00
Philipp Koehn
b369699661 various small changes, mostly related to better compliance with grid engine 2015-05-01 17:44:18 -04:00
Rico Sennrich
e98a2fc980 fix interpolation for LM with parser in pre-processing 2015-04-30 15:46:33 +01:00
Hieu Hoang
1278b8f5a7 Merge branch 'master' of github.com:moses-smt/mosesdecoder 2015-04-30 15:35:34 +04:00
Hieu Hoang
a6d34a660d another madamira wrapper. Just uses the tokenized file it outputs 2015-04-30 15:35:15 +04:00
Hieu Hoang
15e4b16f49 delete unused var 2015-04-30 14:01:03 +04:00
Hieu Hoang
ebc5a51d32 Merge pull request #111 from unhammer/extract-perl-safewait 
die if the forked extract exited with error
 2015-04-30 11:45:25 +04:00
Hieu Hoang
1c99b2b2b8 Merge pull request #110 from unhammer/extract-perl-abspath-when-ln 
avoid bad symlinks in extract-parallel
 2015-04-30 11:41:03 +04:00
Kevin Brubeck Unhammer
2af2f2ef36 avoid bad symlinks in extract-parallel 
train-model seems to pass a non-absolute path for the
model/aligned-argument, and then extract-parallel creates a bad symlink
 2015-04-30 09:36:59 +02:00
Kevin Brubeck Unhammer
c116fa0dbf die if the forked extract exited with error 
Should we pass on bad exit codes from RunFork to those waitpids as well?
Seems like the right thing, though I don't know the code.
 2015-04-30 09:33:37 +02:00
Nicola Bertoldi
3400b622c0 Merge branch 'master' of https://github.com/moses-smt/mosesdecoder 2015-04-30 08:35:41 +02:00
Hieu Hoang
8f9bf7ea38 add -config 2015-04-28 15:03:59 +04:00
Hieu Hoang
b7792b227a script to convert arabic to bw, and vice versa 2015-04-28 12:29:58 +04:00
Hieu Hoang
8adad4fc2e exec permission 2015-04-27 17:39:49 +04:00
Hieu Hoang
a47fc00635 option to output factors 2015-04-27 17:35:19 +04:00
Rico Sennrich
da648fd65b fix some RDLM training options 2015-04-27 10:52:16 +01:00
alvations
fa30ea6712 Merge branch 'master' of https://github.com/moses-smt/mosesdecoder into moses-smt-master 2015-04-26 20:38:27 +02:00
alvations
4a68c42b16 syncing to latest moses version 2015-04-26 20:37:10 +02:00
alvations
ec54ea3c4f put back some of the difference made after RELEASE3.0 and incorporated it with the -threads parameter 2015-04-26 20:30:15 +02:00
alvations
c01b0a6262 merging the filter-model-given-input.pl with alvations-master branch 2015-04-26 20:25:15 +02:00
alvations
dda3ddd80b Merge branch 'master' of https://github.com/moses-smt/mosesdecoder 2015-04-26 20:23:39 +02:00
alvations
e1fcc8082a use integer type when reading options instead of checking for undef. it's more elegant. 2015-04-24 19:30:40 +02:00
alvations
d453ccc9f5 removed wrongly added perl script... 2015-04-24 19:14:04 +02:00
alvations
0ccbcaece6 added $threads option in usage example 2015-04-24 19:11:57 +02:00
alvations
aa9207acfc fixed typo in $thread to $threads 2015-04-24 19:10:10 +02:00
alvations
6c63ca963c checks for undefined $threads 2015-04-24 19:06:37 +02:00
alvations
585784f62a added thread options for filter-model-given-input.pl 2015-04-24 18:57:28 +02:00
Hieu Hoang
4b47e1148c use ignore-unless /Philipp Koehn 2015-04-22 23:02:57 +04:00
Hieu Hoang
40933b4a78 hack to allow target side of tokenized parallel corpus to be used for LM 2015-04-22 19:01:12 +04:00
Nicola Bertoldi
5700fbaabf Merge branch 'master' of https://github.com/moses-smt/mosesdecoder 2015-04-22 07:50:07 +02:00
Hieu Hoang
c15f3ef068 duplicated functionality with ems/support/lmplz-wrapper.perl 2015-04-21 17:54:34 +04:00
Hieu Hoang
ab01d30687 make sure GetOptions doesn consume -T by confusing it with --text 2015-04-21 17:53:46 +04:00
Rico Sennrich
15d3c3f259 be more tolerant about xml input 2015-04-21 14:04:25 +01:00
Rico Sennrich
5a3d5b6bdd EMS: LM:mock-parse can be actual parser 2015-04-21 10:21:24 +01:00
Hieu Hoang
95435f2a2e better detection of pigz 2015-04-20 20:40:50 +04:00
Hieu Hoang
eb37437d09 don't output warnings. It wasn't originally there before the 'env perl' change. This script should be tightened up at some point, eg use strict, debug warning messages 2015-04-20 16:18:51 +04:00
Hieu Hoang
1b9dc6cfae more butinah tweaks 2015-04-19 11:50:50 +04:00
Hieu Hoang
637e8a17e8 add pre tokenization cleaning script. In case training has bad, overlying long lines which blows up some taggers/segmenters, eg. mada 2015-04-19 11:21:07 +04:00
Hieu Hoang
6162223690 add use warnings to all perl scripts 2015-04-13 20:42:33 +04:00
Hieu Hoang
8190a5e1d6 Merge pull request #107 from flammie/master 
Finnish detokenisation
 2015-04-11 14:20:37 +04:00
Dingyuan Wang
4aba64ed53 Merge pull request #106 from gumblex/master 
Fix some problems in EMS
 2015-04-11 09:26:25 +08:00
Michael Denkowski
2682cc0f9b typo fix 2015-04-07 17:06:18 -04:00
Flammie Pirinen
fc8ee03b8d examples 2015-04-07 16:19:07 +01:00
Flammie Pirinen
ef52bc66f6 full set of cases and caps 2015-04-07 16:16:43 +01:00
Flammie Pirinen
85230e8334 add fi to list to silence warnings 2015-04-07 16:00:48 +01:00
Flammie Pirinen
f9deb6de3b also lowercase if case fail 2015-04-07 15:55:02 +01:00
Flammie Pirinen
5817806ec7 fix detokenising : in abbrev. case suffix case 2015-04-07 15:51:29 +01:00
Hieu Hoang
54e55f2dcb better detection of pigz, sort, split. In case they are not in the default directory 2015-04-06 11:31:44 +04:00
Hieu Hoang
02185a85fb store temp run files in current directory, not /tmp 2015-04-05 17:02:48 +04:00
Hieu Hoang
93ad52d2f9 leave in runPath for debugging 2015-04-05 16:49:12 +04:00
Hieu Hoang
4cb8a1837e Merge branch 'master' of github.com:moses-smt/mosesdecoder 2015-04-05 16:45:17 +04:00
Hieu Hoang
7ffdddef13 script to submit ems job to grid engine as 1 job. Hardcoded for NYUAD at the mo 2015-04-05 16:44:24 +04:00
Dingyuan Wang
aea07b0a19 Fix some problems in EMS: 
* remove absolute links
* fix coverage bar highlighting
* change Base64 library to support UTF-8
 2015-04-03 23:47:25 +08:00
Rico Sennrich
8d8097632b re-apply commit 1fb51dc (use gunzip -c instead of zcat) 
plus be more tolerant about xml input
 2015-04-03 15:00:45 +01:00
Hieu Hoang
b2f9ba2b64 revert last commit to add MASTER_PATH. Not needed 2015-04-02 19:29:42 +04:00
Hieu Hoang
27b36e0c96 pass in PATH variable from master node. When you're running of a grid but really just qsubbing everything to 1 slave node 2015-04-02 19:15:21 +04:00
Hieu Hoang
2d1da3219d consistently use 'env perl' command for environments where the 1st perl in PATH isn't the default perl. Which is kinda stupid 2015-04-02 17:38:56 +04:00
Hieu Hoang
035c806059 Merge branch 'master' of github.com:moses-smt/mosesdecoder 2015-04-02 14:10:42 +04:00
Hieu Hoang
e76247e19b Conditional import of Thread package for perl installations that don't support threads 2015-04-02 14:07:57 +04:00
Hieu Hoang
e22d275c32 don't ignore lowercasing of factored LM. Must be consistent with pt 2015-04-01 23:25:57 +04:00
Phil Williams
6ce3060dd8 lmplz-wrapper.perl: use Getopt::Long's "pass_through" option 
This avoids the need to duplicate all of lmplz's options in the wrapper and
it prevents --prune 0 0 1 from being truncated to --prune 0 if the user forgets
to quote the arguments.
 2015-03-30 10:18:51 +01:00
alvations
496a2a716c Merge branch 'master' of https://github.com/moses-smt/mosesdecoder 2015-03-23 15:55:56 +01:00
Matthias Huck
506427368f filter-model-given-input.pl: drop "-encoding None" from phrase table binaization with processPhraseTableMin. Recommended by Marcin. 2015-03-23 14:38:24 +00:00
Hieu Hoang
c4af7d28b5 Merge pull request #100 from alvations/master 
Added the Gacha Filter used in WMT14 by the Manawi system
 2015-03-21 22:42:55 +00:00
alvations
e5feb1a73e Enforce python3 and also remove extra empty newline from STDOUT 2015-03-20 19:37:34 +01:00
alvations
8f2d687d27 added more description of usage in docstring 2015-03-20 19:00:36 +01:00
alvations
44cd32d058 fixed typo in error message 2015-03-20 18:53:03 +01:00
alvations
93ea5853e8 Added Gacha Filter used by Manawi system from WMT14 translation task 2015-03-20 18:48:47 +01:00
alvations
5536e13213 added Gacha Filter from WMT14 2015-03-20 18:44:59 +01:00
Rico Sennrich
3a673fc8dc EMS: support for syntactic metrics for MERT/MIRA 
- add "-n-best-trees" to TUNING:decoder-settings
 - add "mock-output-parser-references = $output-parser" to GENERAL (and define output-parser)
 - TUNING:tuning-settings should include the metric you want to optimize (e.g. "-batch-mira-args='--sctype BLEU,HWCM'")
 2015-03-20 17:15:33 +00:00
Rico Sennrich
ca08b1d205 reduce-factors: port xml support from train-model.perl 2015-03-20 14:44:48 +00:00
Rico Sennrich
b8ca33c34e RDLM training without editing bash scripts 2015-03-20 14:12:41 +00:00
Rico Sennrich
2271f295e6 nplm_train: more options 2015-03-20 14:12:41 +00:00
Rico Sennrich
eab513b635 relational dependency language model 2015-03-18 17:39:45 +00:00
Phil Williams
fc15e03ebe Replace truecase-egret.sh with more general tree-converter-wrapper.perl 2015-03-18 09:57:42 +00:00
Phil Williams
ac51e9f0a8 Always use "SyntaxInputWeight0" as name of SyntaxInputWeight feature 2015-03-18 09:56:46 +00:00
Phil Williams
0a8e5fb3bf EMS: fix TRAINING:use-syntax-input-weight-feature option 2015-03-13 17:18:56 +00:00