moses-smt/mosesdecoder
1
1
Fork 0
mirror of https://github.com/moses-smt/mosesdecoder.git synced 2024-08-17 07:20:48 +03:00
Code Issues Packages Projects Releases Wiki Activity
17,477 Commits 136 Branches 26 Tags 147 MiB
master
Commit Graph

5557 Commits

Author SHA1 Message Date
Hieu Hoang
ebae7ce520 move ConvertFromMoses() to Moses 2016-06-28 11:15:40 +01:00
Hieu Hoang
ad240a9f5b move Moses classes out of OnDiskPt - Word 2016-06-28 10:46:51 +01:00
Hieu Hoang
cef30724d5 move Moses classes out of OnDiskPt - TargetPhrase 2016-06-28 10:28:19 +01:00
Hieu Hoang
aca2db894c move Moses classes out of OnDiskPt 2016-06-28 10:08:30 +01:00
Hieu Hoang
2a6e5777fb Merge ../mosesdecoder into perf_moses2 2016-06-17 12:47:51 +01:00
Ulrich Germann
5282ad667c Global scope for VW training. 2016-06-16 14:22:58 +01:00
Prashant Mathur
e31bc247ea Merge branch 'master' of ssh://github.com/moses-smt/mosesdecoder 2016-06-15 05:33:42 -07:00
MosesAdmin
f895e09292 daily automatic beautifier 2016-06-07 18:20:39 +01:00
Hieu Hoang
b19e60eb13 Merge ../mosesdecoder into perf_moses2 2016-06-07 18:16:22 +01:00
Hieu Hoang
93dd29639a vector -> pair 2016-06-07 18:16:16 +01:00
Hieu Hoang
60b83e7963 debugging output for ReorderingConstraint 2016-06-07 17:24:33 +01:00
Hieu Hoang
81642a82c1 debugging zones. Work the same as moses 2016-06-07 17:23:31 +01:00
Hieu Hoang
747e416687 Merge ../mosesdecoder into perf_moses2 2016-06-07 16:26:34 +01:00
Ales Tamchyna
44c9e6db17 Merge branch 'vw_tgtcontext' of github.com:moses-smt/mosesdecoder 2016-06-07 17:14:16 +02:00
Hieu Hoang
b75ef6f619 Merge ../mosesdecoder into perf_moses2 2016-06-04 12:45:30 +01:00
MosesAdmin
ea306f62b7 daily automatic beautifier 2016-06-02 00:00:39 +01:00
Hieu Hoang
1c9220049b Merge ../mosesdecoder into perf_moses2 2016-06-01 21:03:37 +01:00
Marcin Junczys-Dowmunt
73ffe51bd8 Added Grammatical Error Correction specific scorer (M^2) and features 2016-06-01 18:36:43 +02:00
Hieu Hoang
36812013bf Merge ../mosesdecoder into perf_moses2 2016-05-31 14:36:15 +01:00
MosesAdmin
33932a0fc9 daily automatic beautifier 2016-05-30 00:00:33 +01:00
Philipp Koehn
942eb5a8b1 allow configuration of operation sequence model loading, allow specification of KENLM/OSM loading in experiment.perl / train-model.perl 2016-05-29 11:46:42 -04:00
Philipp Koehn
227e54162a corrected segment time out implementation 2016-05-29 11:44:31 -04:00
MosesAdmin
a18c6746a0 daily automatic beautifier 2016-05-28 00:00:33 +01:00
Philipp Koehn
c11a533f14 added segment level time out 2016-05-27 05:39:42 -04:00
Philipp Koehn
65810a3862 do not crash on bad XML 2016-05-27 04:07:37 -04:00
Hieu Hoang
0e4af190a3 Merge ../mosesdecoder into perf_moses2 2016-05-23 17:20:10 +01:00
Jonathan Chen
5b42fb9c05 removed duplicated option 2016-05-17 11:42:10 -05:00
Ales Tamchyna
7c5bb9328c less verbosity in VW 2016-05-17 13:16:09 +02:00
Hieu Hoang
dd6daf7719 Merge ../mosesdecoder into perf_moses2 2016-05-16 11:14:12 -04:00
Prashant Mathur
9e3bb2b926 bug fix 2016-05-14 14:02:29 +02:00
Prashant Mathur
95cebdad0c removing debug statements 2016-05-14 13:31:12 +02:00
Prashant Mathur
6f75c31be2 Adding a new cache based translation model (thread safe) 2016-05-14 13:21:55 +02:00
michaelhutt
9945502e50 fixes std::bad_alloc in processPhraseTableMin for large models 
+ only seems to affect the Intel compiler at O3
 2016-05-12 16:44:09 -04:00
Prashant Mathur
02027c14e8 Bug fixes for factored translation 2016-05-11 17:30:33 +02:00
MosesAdmin
3068ac254f daily automatic beautifier 2016-05-11 00:00:37 +01:00
Prashant Mathur
d379557cf2 Bug fix 2016-05-09 19:56:42 +02:00
Prashant Mathur
55144293a3 CBTM with additional feature scores: compiled version 2016-05-09 17:57:00 +02:00
Prashant Mathur
ef3199de3e Fix the evaluate method for GLM. GLM should evaluate transopts after 
initializing the input
 2016-05-03 12:01:43 +02:00
Hieu Hoang
c9a07dd25c Merge ../mosesdecoder into perf_moses2 2016-04-15 20:04:26 +04:00
Ulrich Germann
31c8946c8c Merge branch 'master' of https://github.com/moses-smt/mosesdecoder 2016-04-15 16:39:57 +01:00
Ulrich Germann
35c4abd874 Bug fix: make server interface recognize booleans encoded as proper xmlrpc_c::value_boolean when parsing requests. 2016-04-15 16:38:07 +01:00
Hieu Hoang
1ff1d04f76 Merge ../mosesdecoder into perf_moses2 2016-04-12 23:04:51 +04:00
MosesAdmin
7b205b0c8a daily automatic beautifier 2016-04-11 00:00:43 +01:00
Hieu Hoang
fa92641061 Merge ../mosesdecoder into perf_moses2 2016-04-10 12:41:57 +04:00
Hieu Hoang
657b08888c Merge https://github.com/msalameh83/mosesdecoder 2016-04-10 08:51:42 +01:00
Ales Tamchyna
2a58011d37 vw corner cases when looking inside factors 2016-04-04 16:32:39 +02:00
Ales Tamchyna
90f6f4d4a7 vw context bigram features; support using tag subsets as features in VW 2016-04-04 15:22:43 +02:00
msalameh83
9b0923d93c added dese-schemeoption; 2016-04-03 22:23:05 -06:00
msalameh83
fa36b602ec fix BeginSentence unsegLM score for optimistic 2016-04-03 16:55:21 -06:00
msalameh83
9b260d48c9 deseg initial commit 2016-04-01 11:53:54 -06:00
msalameh83
f23beaa27d deseg initial commit 2016-04-01 11:16:03 -06:00
Hieu Hoang
556a51b5d2 Merge ../mosesdecoder into perf_moses2 2016-04-01 11:11:39 +01:00
Hieu Hoang
3e2c51095c guard 2016-04-01 10:25:21 +01:00
Hieu Hoang
ff8caa1226 Merge ../mosesdecoder into perf_moses2 2016-03-28 23:58:20 +01:00
Hieu Hoang
b8bc4a9fb6 use max_partial_trans_opt, not m_max_phrase_length 2016-03-28 23:58:06 +01:00
Ales Tamchyna
4f85b605c9 bilingual features for VW 2016-03-24 18:09:07 +01:00
Lane Schwartz
923532982d Merge branch 'master' of github.com:moses-smt/mosesdecoder 2016-03-24 10:34:27 -05:00
Ales Tamchyna
2536352875 towards using aligned source words in context features 2016-03-24 15:07:48 +01:00
Hieu Hoang
9a560a3d0f recreate lex ro framework from Moses 2016-03-24 13:46:52 +00:00
Kenneth Heafield
31698d5eca Remove deprecated lazyken=0 settings. Should be load instead, but don't specify if it's the default. 2016-03-24 12:58:10 +00:00
Ales Tamchyna
4f807c47c8 implementing VWContextBilingual 2016-03-24 12:58:20 +01:00
Ales Tamchyna
90a3650b42 minor 2016-03-24 12:45:35 +01:00
MosesAdmin
ab2be02eef daily automatic beautifier 2016-03-24 00:00:47 +00:00
Kenneth Heafield
8a083bc921 Missing #include <set> (which BTW probably shouldn't be used) 2016-03-23 22:46:54 +00:00
Kenneth Heafield
49165a370b Pull out my config structure, still need it to go outside the ConstructOSMLM function 2016-03-23 22:15:14 +00:00
Lane Schwartz
d45b357e11 Merge branch 'master' of github.com:moses-smt/mosesdecoder 2016-03-23 17:03:31 -05:00
Kenneth Heafield
0665cb46e5 Chart decoding is supported by UTIL_THROW2, but not by OpSequenceModel 2016-03-23 21:31:47 +00:00
Kenneth Heafield
90b772734c Delete unnecessary virtual 2016-03-23 21:28:55 +00:00
Ales Tamchyna
073f9c5656 bugfix in context hash 2016-03-23 17:37:19 +01:00
Ales Tamchyna
c7a1d21abd refactoring VW state class 2016-03-23 16:45:53 +01:00
Lane Schwartz
167def1d52 Merge branch 'master' of github.com:moses-smt/mosesdecoder 2016-03-23 10:26:07 -05:00
Lane Schwartz
8893524339 Fixed bug 2016-03-23 10:25:57 -05:00
Hieu Hoang
356de67a6d Merge ../mosesdecoder into perf_moses2 2016-03-23 15:18:39 +00:00
Hieu Hoang
51300fd897 merge 2016-03-23 15:18:37 +00:00
Hieu Hoang
4c07496eb2 compile error with suffix array 2016-03-23 15:18:20 +00:00
Hieu Hoang
dccaa63b31 rename LexicalReorderingState.h -> LRState.h 2016-03-23 14:37:26 +00:00
Hieu Hoang
fa02a99f13 separate LRModel into its own file 2016-03-23 14:29:32 +00:00
Hieu Hoang
105faa5673 separate BidirectionalReorderingState into its own file 2016-03-23 13:57:25 +00:00
Ales Tamchyna
01301ac816 removing unused variables 2016-03-23 14:16:11 +01:00
Ales Tamchyna
f6f506e814 clarification 2016-03-23 14:10:25 +01:00
Ales Tamchyna
47e1ffc108 more comments 2016-03-23 14:05:29 +01:00
Ales Tamchyna
f85952c47c some comments 2016-03-23 13:36:54 +01:00
Ales Tamchyna
e6b00a0c22 re-factoring VW integration 2016-03-23 13:32:28 +01:00
Ales Tamchyna
acf7d7eedd update VW context feature interface 2016-03-23 12:38:57 +01:00
Hieu Hoang
ba9a9b3ef9 separate PhraseBasedReorderingState into its own file 2016-03-23 10:48:16 +00:00
Hieu Hoang
be20f2c149 separate HReorderingBackwardState into its own file 2016-03-23 10:43:55 +00:00
Hieu Hoang
46a94f2258 separate HReorderingForwardState into its own file 2016-03-23 10:38:42 +00:00
Hieu Hoang
3fe5574a95 compiles 2016-03-22 17:51:39 +00:00
Ales Tamchyna
73c0a86445 minor 2016-03-22 16:08:57 +01:00
Ales Tamchyna
c7222841d7 Merge branch 'master' of github.com:moses-smt/mosesdecoder into vw_tgtcontext 2016-03-22 15:42:43 +01:00
Ales Tamchyna
a7e33755f9 optimizations 2016-03-22 15:11:41 +01:00
Hieu Hoang
95f252aa37 Merge ../mosesdecoder into perf_moses2 2016-03-22 10:20:21 +00:00
David Madl
b760ad8a7e BUGFIX: fix ScoreComponentCollection::ZeroAll(), affecting MIRA 
I observed a bug that ScoreComponentCollection core entries would retain their score even after ZeroAll().  This may have affected the Moses implementation of MIRA.

* std::valarray::resize(0) means "resize to 0" [1]
* subsequent accesses using operator[] result in undefined behavior [2]

FeatureVector::clear() is used by ScoreComponentCollection::ZeroAll(), which in turn was used in these places:

./contrib/mira/Main.cpp:665:      cumulativeWeights.ZeroAll();
./contrib/mira/Main.cpp:666:      cumulativeWeightsBinary.ZeroAll();
./moses/Incremental.cpp:580:  features.ZeroAll();

It seems to me that the Moses implementation of MIRA may have been affected?

[1] http://www.cplusplus.com/reference/valarray/valarray/resize/
[2] http://www.cplusplus.com/reference/valarray/valarray/operator%5B%5D/
 2016-03-22 10:37:30 +01:00
Michael Denkowski
fa15d60afd Revert "Query member phrase tables in batch mode for forward compatibility" 
This reverts commit 2462c81f7a.

Looks like legacy is still the least-breaking way to query phrase tables
 2016-03-21 16:44:23 -04:00
Michael Denkowski
2a9034eaec Revert "daily automatic beautifier" 
This reverts commit 578d667ca2.
 2016-03-21 16:41:49 -04:00
Ales Tamchyna
75676a8435 Merge branch 'master' of github.com:moses-smt/mosesdecoder into vw_tgtcontext 2016-03-21 13:49:36 +01:00
MosesAdmin
578d667ca2 daily automatic beautifier 2016-03-19 00:00:38 +00:00
Michael Denkowski
2462c81f7a Query member phrase tables in batch mode for forward compatibility 
Use GetTargetPhraseCollectionBatch instead of GetTargetPhraseCollectionLEGACY
 2016-03-18 14:37:46 -04:00
MosesAdmin
e7627e04ed daily automatic beautifier 2016-03-18 00:00:58 +00:00
Matthias Huck
bbf8a615f2 Sparse distortion features 
in the manner of:

Spence Green, Michel Galley, Christopher D. Manning. 2010.  Improved
Models of Distortion Cost for Statistical Machine Translation. In NAACL
2010.
 2016-03-17 16:10:49 +00:00
Hieu Hoang
f96b48a041 Merge ../mosesdecoder into perf_moses2 2016-03-11 16:38:38 +00:00
Ales Tamchyna
3e5c0e8667 minor 2016-03-10 14:01:47 +01:00
Ales Tamchyna
d972d50b79 optimizing VW queries by caching 2016-03-10 13:41:56 +01:00
Ales Tamchyna
6b579c2156 VW feature functions now also output extracted features for caching 2016-03-09 19:27:24 +01:00
Ales Tamchyna
1007b376b2 return feature ids when adding features 2016-03-09 18:52:07 +01:00
Ales Tamchyna
4486c8ec43 important bug fix in target-side VW, adding optional debugging messages 2016-03-08 14:42:45 +01:00
Jeroen
ae5ad8a49e Detect write errors in LVoc. 
Write failures in LVoc::Write() were going unnoticed.  If disk space
runs out, the output file might get truncated without any indication
of failure.
 2016-03-07 18:55:18 +01:00
Ales Tamchyna
f13ba3bdfb bugfix 2016-03-07 18:04:17 +01:00
Ales Tamchyna
dab84877ae bugfix 2016-03-07 16:31:41 +01:00
Ales Tamchyna
3bc2124f78 prepend BOS words to target sentence when training VW 2016-03-07 16:18:33 +01:00
Ales Tamchyna
f169c59204 removing outdated method 2016-03-07 15:49:53 +01:00
Ales Tamchyna
7fbbc65902 update vw state 2016-03-07 15:45:51 +01:00
Ales Tamchyna
e4487c8d45 finished a naive implementation of vw target context, it compiles 2016-03-07 15:24:39 +01:00
Ales Tamchyna
0ca54e4d15 keep track of required context size for each VW features 2016-03-07 14:25:04 +01:00
Ales Tamchyna
0704e9c2e5 towards state tracking for VW 2016-03-07 13:52:40 +01:00
Ales Tamchyna
fa8df45656 evaluate VW with target context 2016-03-07 13:17:38 +01:00
Ales Tamchyna
48d9d8e0d4 towards evaluation in target context 2016-03-07 12:41:05 +01:00
Ales Tamchyna
0484739873 when target context features are used in VW, do not add any score before decoding, cache it instead 2016-03-07 11:57:44 +01:00
Ales Tamchyna
8003d28ffa note 2016-03-07 11:05:47 +01:00
Ales Tamchyna
97b1625ffa pseudocode for target-context evaluation in decoding with VW 2016-03-03 17:27:40 +01:00
Ales Tamchyna
82c3254d46 Merge branch 'master' of github.com:moses-smt/mosesdecoder into vw_tgtcontext 2016-03-03 17:03:28 +01:00
Ales Tamchyna
e74e147c84 minor 2016-03-03 17:02:22 +01:00
Ales Tamchyna
396982010b support training with target-context features in VW 2016-03-03 16:29:57 +01:00
Ales Tamchyna
b43e24f953 implemented a basic target context feature 2016-03-03 15:39:46 +01:00
Ales Tamchyna
91541f0b83 target context feature base class 2016-03-03 14:46:51 +01:00
Ales Tamchyna
4b5e49aff9 new vw feature type: target context 2016-03-03 13:50:52 +01:00
Ales Tamchyna
dce605029c make VW feature stateful 2016-03-02 17:28:21 +01:00
Hieu Hoang
b9217cec1d Merge ../mosesdecoder into perf_moses2 2016-03-01 11:08:39 +00:00
Matthias Huck
2d6f616480 Merge branch 'master' of https://github.com/moses-smt/mosesdecoder 
Conflicts:
	moses/FF/Model1Feature.cpp
 2016-02-22 18:30:11 +00:00
Matthias Huck
5c8c15d345 Model1Feature: skip-target-punctuation parameter 2016-02-22 18:28:23 +00:00
MosesAdmin
6da1343f88 daily automatic beautifier 2016-02-22 00:00:40 +00:00
Hieu Hoang
e6bc481ee2 Merge ../mosesdecoder into perf_moses2 2016-02-21 23:09:46 +00:00
Matthias Huck
71a0fcbc27 Model1Feature: skip over target-side punctuation 2016-02-21 20:44:09 +00:00
Matthias Huck
e284503018 Merge branch 'master' of https://github.com/moses-smt/mosesdecoder 2016-02-20 00:40:23 +00:00
Matthias Huck
ebb18b2dc2 WordTranslationFeature: fix loading of word lists 2016-02-20 00:39:02 +00:00
Kenneth Heafield
7a1baeecda load= option to KenLM exposing more load_method options 2016-02-20 00:07:48 +00:00
Ulrich Germann
63eefe03fd Merge branch 'master' of https://github.com/moses-smt/mosesdecoder 2016-02-19 10:23:21 +00:00
Ulrich Germann
98bcf53ec2 Namespace bug fix. 2016-02-18 16:39:34 +00:00
Hieu Hoang
6a363d893c Merge ../mosesdecoder into perf_moses2 2016-02-17 17:39:51 +00:00
Hieu Hoang
0a1856d510 Merge ../mosesdecoder into perf_moses2 2016-02-17 17:38:07 +00:00
Ales Tamchyna
ca6dba9108 update VW compatibility 2016-02-17 11:27:44 +01:00
Hieu Hoang
0d0f75b6e0 gets right score 2016-02-16 16:03:37 +00:00
Ulrich Germann
d7db5d2441 Merge branch 'master' of https://github.com/moses-smt/mosesdecoder 2016-02-15 16:49:03 +00:00
Ulrich Germann
bbcd10a62d Better option handling in check-coverage3. 2016-02-15 16:48:46 +00:00
MosesAdmin
ca5178b684 daily automatic beautifier 2016-02-13 00:02:04 +00:00
Hieu Hoang
938ef79404 Merge ../mosesdecoder into perf_moses2 2016-02-12 19:40:04 +00:00
Matthias Huck
1659d6b4c8 Option for target constituent constrained phrase extraction. TargetConstituentAdjacencyFeature. 2016-02-12 17:46:57 +00:00
Ulrich Germann
490edf7a82 Merge branch 'master' of https://github.com/moses-smt/mosesdecoder 2016-02-06 15:20:09 +00:00
Ulrich Germann
105225d398 Bug fix. 2016-02-06 15:19:44 +00:00
Hieu Hoang
06769e648d Merge ../mosesdecoder into perf_moses2 2016-02-04 13:05:36 +00:00