Hieu Hoang
ebae7ce520
move ConvertFromMoses() to Moses
2016-06-28 11:15:40 +01:00
Hieu Hoang
ad240a9f5b
move Moses classes out of OnDiskPt - Word
2016-06-28 10:46:51 +01:00
Hieu Hoang
cef30724d5
move Moses classes out of OnDiskPt - TargetPhrase
2016-06-28 10:28:19 +01:00
Hieu Hoang
aca2db894c
move Moses classes out of OnDiskPt
2016-06-28 10:08:30 +01:00
Hieu Hoang
2a6e5777fb
Merge ../mosesdecoder into perf_moses2
2016-06-17 12:47:51 +01:00
Ulrich Germann
5282ad667c
Global scope for VW training.
2016-06-16 14:22:58 +01:00
Prashant Mathur
e31bc247ea
Merge branch 'master' of ssh://github.com/moses-smt/mosesdecoder
2016-06-15 05:33:42 -07:00
MosesAdmin
f895e09292
daily automatic beautifier
2016-06-07 18:20:39 +01:00
Hieu Hoang
b19e60eb13
Merge ../mosesdecoder into perf_moses2
2016-06-07 18:16:22 +01:00
Hieu Hoang
93dd29639a
vector -> pair
2016-06-07 18:16:16 +01:00
Hieu Hoang
60b83e7963
debugging output for ReorderingConstraint
2016-06-07 17:24:33 +01:00
Hieu Hoang
81642a82c1
debugging zones. Work the same as moses
2016-06-07 17:23:31 +01:00
Hieu Hoang
747e416687
Merge ../mosesdecoder into perf_moses2
2016-06-07 16:26:34 +01:00
Ales Tamchyna
44c9e6db17
Merge branch 'vw_tgtcontext' of github.com:moses-smt/mosesdecoder
2016-06-07 17:14:16 +02:00
Hieu Hoang
b75ef6f619
Merge ../mosesdecoder into perf_moses2
2016-06-04 12:45:30 +01:00
MosesAdmin
ea306f62b7
daily automatic beautifier
2016-06-02 00:00:39 +01:00
Hieu Hoang
1c9220049b
Merge ../mosesdecoder into perf_moses2
2016-06-01 21:03:37 +01:00
Marcin Junczys-Dowmunt
73ffe51bd8
Added Grammatical Error Correction specific scorer (M^2) and features
2016-06-01 18:36:43 +02:00
Hieu Hoang
36812013bf
Merge ../mosesdecoder into perf_moses2
2016-05-31 14:36:15 +01:00
MosesAdmin
33932a0fc9
daily automatic beautifier
2016-05-30 00:00:33 +01:00
Philipp Koehn
942eb5a8b1
allow configuration of operation sequence model loading, allow specification of KENLM/OSM loading in experiment.perl / train-model.perl
2016-05-29 11:46:42 -04:00
Philipp Koehn
227e54162a
corrected segment time out implementation
2016-05-29 11:44:31 -04:00
MosesAdmin
a18c6746a0
daily automatic beautifier
2016-05-28 00:00:33 +01:00
Philipp Koehn
c11a533f14
added segment level time out
2016-05-27 05:39:42 -04:00
Philipp Koehn
65810a3862
do not crash on bad XML
2016-05-27 04:07:37 -04:00
Hieu Hoang
0e4af190a3
Merge ../mosesdecoder into perf_moses2
2016-05-23 17:20:10 +01:00
Jonathan Chen
5b42fb9c05
removed duplicated option
2016-05-17 11:42:10 -05:00
Ales Tamchyna
7c5bb9328c
less verbosity in VW
2016-05-17 13:16:09 +02:00
Hieu Hoang
dd6daf7719
Merge ../mosesdecoder into perf_moses2
2016-05-16 11:14:12 -04:00
Prashant Mathur
9e3bb2b926
bug fix
2016-05-14 14:02:29 +02:00
Prashant Mathur
95cebdad0c
removing debug statements
2016-05-14 13:31:12 +02:00
Prashant Mathur
6f75c31be2
Adding a new cache based translation model (thread safe)
2016-05-14 13:21:55 +02:00
michaelhutt
9945502e50
fixes std::bad_alloc in processPhraseTableMin for large models
...
+ only seems to affect the Intel compiler at O3
2016-05-12 16:44:09 -04:00
Prashant Mathur
02027c14e8
Bug fixes for factored translation
2016-05-11 17:30:33 +02:00
MosesAdmin
3068ac254f
daily automatic beautifier
2016-05-11 00:00:37 +01:00
Prashant Mathur
d379557cf2
Bug fix
2016-05-09 19:56:42 +02:00
Prashant Mathur
55144293a3
CBTM with additional feature scores: compiled version
2016-05-09 17:57:00 +02:00
Prashant Mathur
ef3199de3e
Fix the evaluate method for GLM. GLM should evaluate transopts after
...
initializing the input
2016-05-03 12:01:43 +02:00
Hieu Hoang
c9a07dd25c
Merge ../mosesdecoder into perf_moses2
2016-04-15 20:04:26 +04:00
Ulrich Germann
31c8946c8c
Merge branch 'master' of https://github.com/moses-smt/mosesdecoder
2016-04-15 16:39:57 +01:00
Ulrich Germann
35c4abd874
Bug fix: make server interface recognize booleans encoded as proper xmlrpc_c::value_boolean when parsing requests.
2016-04-15 16:38:07 +01:00
Hieu Hoang
1ff1d04f76
Merge ../mosesdecoder into perf_moses2
2016-04-12 23:04:51 +04:00
MosesAdmin
7b205b0c8a
daily automatic beautifier
2016-04-11 00:00:43 +01:00
Hieu Hoang
fa92641061
Merge ../mosesdecoder into perf_moses2
2016-04-10 12:41:57 +04:00
Hieu Hoang
657b08888c
Merge https://github.com/msalameh83/mosesdecoder
2016-04-10 08:51:42 +01:00
Ales Tamchyna
2a58011d37
vw corner cases when looking inside factors
2016-04-04 16:32:39 +02:00
Ales Tamchyna
90f6f4d4a7
vw context bigram features; support using tag subsets as features in VW
2016-04-04 15:22:43 +02:00
msalameh83
9b0923d93c
added dese-schemeoption;
2016-04-03 22:23:05 -06:00
msalameh83
fa36b602ec
fix BeginSentence unsegLM score for optimistic
2016-04-03 16:55:21 -06:00
msalameh83
9b260d48c9
deseg initial commit
2016-04-01 11:53:54 -06:00
msalameh83
f23beaa27d
deseg initial commit
2016-04-01 11:16:03 -06:00
Hieu Hoang
556a51b5d2
Merge ../mosesdecoder into perf_moses2
2016-04-01 11:11:39 +01:00
Hieu Hoang
3e2c51095c
guard
2016-04-01 10:25:21 +01:00
Hieu Hoang
ff8caa1226
Merge ../mosesdecoder into perf_moses2
2016-03-28 23:58:20 +01:00
Hieu Hoang
b8bc4a9fb6
use max_partial_trans_opt, not m_max_phrase_length
2016-03-28 23:58:06 +01:00
Ales Tamchyna
4f85b605c9
bilingual features for VW
2016-03-24 18:09:07 +01:00
Lane Schwartz
923532982d
Merge branch 'master' of github.com:moses-smt/mosesdecoder
2016-03-24 10:34:27 -05:00
Ales Tamchyna
2536352875
towards using aligned source words in context features
2016-03-24 15:07:48 +01:00
Hieu Hoang
9a560a3d0f
recreate lex ro framework from Moses
2016-03-24 13:46:52 +00:00
Kenneth Heafield
31698d5eca
Remove deprecated lazyken=0 settings. Should be load instead, but don't specify if it's the default.
2016-03-24 12:58:10 +00:00
Ales Tamchyna
4f807c47c8
implementing VWContextBilingual
2016-03-24 12:58:20 +01:00
Ales Tamchyna
90a3650b42
minor
2016-03-24 12:45:35 +01:00
MosesAdmin
ab2be02eef
daily automatic beautifier
2016-03-24 00:00:47 +00:00
Kenneth Heafield
8a083bc921
Missing #include <set> (which BTW probably shouldn't be used)
2016-03-23 22:46:54 +00:00
Kenneth Heafield
49165a370b
Pull out my config structure, still need it to go outside the ConstructOSMLM function
2016-03-23 22:15:14 +00:00
Lane Schwartz
d45b357e11
Merge branch 'master' of github.com:moses-smt/mosesdecoder
2016-03-23 17:03:31 -05:00
Kenneth Heafield
0665cb46e5
Chart decoding is supported by UTIL_THROW2, but not by OpSequenceModel
2016-03-23 21:31:47 +00:00
Kenneth Heafield
90b772734c
Delete unnecessary virtual
2016-03-23 21:28:55 +00:00
Ales Tamchyna
073f9c5656
bugfix in context hash
2016-03-23 17:37:19 +01:00
Ales Tamchyna
c7a1d21abd
refactoring VW state class
2016-03-23 16:45:53 +01:00
Lane Schwartz
167def1d52
Merge branch 'master' of github.com:moses-smt/mosesdecoder
2016-03-23 10:26:07 -05:00
Lane Schwartz
8893524339
Fixed bug
2016-03-23 10:25:57 -05:00
Hieu Hoang
356de67a6d
Merge ../mosesdecoder into perf_moses2
2016-03-23 15:18:39 +00:00
Hieu Hoang
51300fd897
merge
2016-03-23 15:18:37 +00:00
Hieu Hoang
4c07496eb2
compile error with suffix array
2016-03-23 15:18:20 +00:00
Hieu Hoang
dccaa63b31
rename LexicalReorderingState.h -> LRState.h
2016-03-23 14:37:26 +00:00
Hieu Hoang
fa02a99f13
separate LRModel into its own file
2016-03-23 14:29:32 +00:00
Hieu Hoang
105faa5673
separate BidirectionalReorderingState into its own file
2016-03-23 13:57:25 +00:00
Ales Tamchyna
01301ac816
removing unused variables
2016-03-23 14:16:11 +01:00
Ales Tamchyna
f6f506e814
clarification
2016-03-23 14:10:25 +01:00
Ales Tamchyna
47e1ffc108
more comments
2016-03-23 14:05:29 +01:00
Ales Tamchyna
f85952c47c
some comments
2016-03-23 13:36:54 +01:00
Ales Tamchyna
e6b00a0c22
re-factoring VW integration
2016-03-23 13:32:28 +01:00
Ales Tamchyna
acf7d7eedd
update VW context feature interface
2016-03-23 12:38:57 +01:00
Hieu Hoang
ba9a9b3ef9
separate PhraseBasedReorderingState into its own file
2016-03-23 10:48:16 +00:00
Hieu Hoang
be20f2c149
separate HReorderingBackwardState into its own file
2016-03-23 10:43:55 +00:00
Hieu Hoang
46a94f2258
separate HReorderingForwardState into its own file
2016-03-23 10:38:42 +00:00
Hieu Hoang
3fe5574a95
compiles
2016-03-22 17:51:39 +00:00
Ales Tamchyna
73c0a86445
minor
2016-03-22 16:08:57 +01:00
Ales Tamchyna
c7222841d7
Merge branch 'master' of github.com:moses-smt/mosesdecoder into vw_tgtcontext
2016-03-22 15:42:43 +01:00
Ales Tamchyna
a7e33755f9
optimizations
2016-03-22 15:11:41 +01:00
Hieu Hoang
95f252aa37
Merge ../mosesdecoder into perf_moses2
2016-03-22 10:20:21 +00:00
David Madl
b760ad8a7e
BUGFIX: fix ScoreComponentCollection::ZeroAll(), affecting MIRA
...
I observed a bug that ScoreComponentCollection core entries would retain their score even after ZeroAll(). This may have affected the Moses implementation of MIRA.
* std::valarray::resize(0) means "resize to 0" [1]
* subsequent accesses using operator[] result in undefined behavior [2]
FeatureVector::clear() is used by ScoreComponentCollection::ZeroAll(), which in turn was used in these places:
./contrib/mira/Main.cpp:665: cumulativeWeights.ZeroAll();
./contrib/mira/Main.cpp:666: cumulativeWeightsBinary.ZeroAll();
./moses/Incremental.cpp:580: features.ZeroAll();
It seems to me that the Moses implementation of MIRA may have been affected?
[1] http://www.cplusplus.com/reference/valarray/valarray/resize/
[2] http://www.cplusplus.com/reference/valarray/valarray/operator%5B%5D/
2016-03-22 10:37:30 +01:00
Michael Denkowski
fa15d60afd
Revert "Query member phrase tables in batch mode for forward compatibility"
...
This reverts commit 2462c81f7a
.
Looks like legacy is still the least-breaking way to query phrase tables
2016-03-21 16:44:23 -04:00
Michael Denkowski
2a9034eaec
Revert "daily automatic beautifier"
...
This reverts commit 578d667ca2
.
2016-03-21 16:41:49 -04:00
Ales Tamchyna
75676a8435
Merge branch 'master' of github.com:moses-smt/mosesdecoder into vw_tgtcontext
2016-03-21 13:49:36 +01:00
MosesAdmin
578d667ca2
daily automatic beautifier
2016-03-19 00:00:38 +00:00
Michael Denkowski
2462c81f7a
Query member phrase tables in batch mode for forward compatibility
...
Use GetTargetPhraseCollectionBatch instead of GetTargetPhraseCollectionLEGACY
2016-03-18 14:37:46 -04:00
MosesAdmin
e7627e04ed
daily automatic beautifier
2016-03-18 00:00:58 +00:00
Matthias Huck
bbf8a615f2
Sparse distortion features
...
in the manner of:
Spence Green, Michel Galley, Christopher D. Manning. 2010. Improved
Models of Distortion Cost for Statistical Machine Translation. In NAACL
2010.
2016-03-17 16:10:49 +00:00
Hieu Hoang
f96b48a041
Merge ../mosesdecoder into perf_moses2
2016-03-11 16:38:38 +00:00
Ales Tamchyna
3e5c0e8667
minor
2016-03-10 14:01:47 +01:00
Ales Tamchyna
d972d50b79
optimizing VW queries by caching
2016-03-10 13:41:56 +01:00
Ales Tamchyna
6b579c2156
VW feature functions now also output extracted features for caching
2016-03-09 19:27:24 +01:00
Ales Tamchyna
1007b376b2
return feature ids when adding features
2016-03-09 18:52:07 +01:00
Ales Tamchyna
4486c8ec43
important bug fix in target-side VW, adding optional debugging messages
2016-03-08 14:42:45 +01:00
Jeroen
ae5ad8a49e
Detect write errors in LVoc.
...
Write failures in LVoc::Write() were going unnoticed. If disk space
runs out, the output file might get truncated without any indication
of failure.
2016-03-07 18:55:18 +01:00
Ales Tamchyna
f13ba3bdfb
bugfix
2016-03-07 18:04:17 +01:00
Ales Tamchyna
dab84877ae
bugfix
2016-03-07 16:31:41 +01:00
Ales Tamchyna
3bc2124f78
prepend BOS words to target sentence when training VW
2016-03-07 16:18:33 +01:00
Ales Tamchyna
f169c59204
removing outdated method
2016-03-07 15:49:53 +01:00
Ales Tamchyna
7fbbc65902
update vw state
2016-03-07 15:45:51 +01:00
Ales Tamchyna
e4487c8d45
finished a naive implementation of vw target context, it compiles
2016-03-07 15:24:39 +01:00
Ales Tamchyna
0ca54e4d15
keep track of required context size for each VW features
2016-03-07 14:25:04 +01:00
Ales Tamchyna
0704e9c2e5
towards state tracking for VW
2016-03-07 13:52:40 +01:00
Ales Tamchyna
fa8df45656
evaluate VW with target context
2016-03-07 13:17:38 +01:00
Ales Tamchyna
48d9d8e0d4
towards evaluation in target context
2016-03-07 12:41:05 +01:00
Ales Tamchyna
0484739873
when target context features are used in VW, do not add any score before decoding, cache it instead
2016-03-07 11:57:44 +01:00
Ales Tamchyna
8003d28ffa
note
2016-03-07 11:05:47 +01:00
Ales Tamchyna
97b1625ffa
pseudocode for target-context evaluation in decoding with VW
2016-03-03 17:27:40 +01:00
Ales Tamchyna
82c3254d46
Merge branch 'master' of github.com:moses-smt/mosesdecoder into vw_tgtcontext
2016-03-03 17:03:28 +01:00
Ales Tamchyna
e74e147c84
minor
2016-03-03 17:02:22 +01:00
Ales Tamchyna
396982010b
support training with target-context features in VW
2016-03-03 16:29:57 +01:00
Ales Tamchyna
b43e24f953
implemented a basic target context feature
2016-03-03 15:39:46 +01:00
Ales Tamchyna
91541f0b83
target context feature base class
2016-03-03 14:46:51 +01:00
Ales Tamchyna
4b5e49aff9
new vw feature type: target context
2016-03-03 13:50:52 +01:00
Ales Tamchyna
dce605029c
make VW feature stateful
2016-03-02 17:28:21 +01:00
Hieu Hoang
b9217cec1d
Merge ../mosesdecoder into perf_moses2
2016-03-01 11:08:39 +00:00
Matthias Huck
2d6f616480
Merge branch 'master' of https://github.com/moses-smt/mosesdecoder
...
Conflicts:
moses/FF/Model1Feature.cpp
2016-02-22 18:30:11 +00:00
Matthias Huck
5c8c15d345
Model1Feature: skip-target-punctuation parameter
2016-02-22 18:28:23 +00:00
MosesAdmin
6da1343f88
daily automatic beautifier
2016-02-22 00:00:40 +00:00
Hieu Hoang
e6bc481ee2
Merge ../mosesdecoder into perf_moses2
2016-02-21 23:09:46 +00:00
Matthias Huck
71a0fcbc27
Model1Feature: skip over target-side punctuation
2016-02-21 20:44:09 +00:00
Matthias Huck
e284503018
Merge branch 'master' of https://github.com/moses-smt/mosesdecoder
2016-02-20 00:40:23 +00:00
Matthias Huck
ebb18b2dc2
WordTranslationFeature: fix loading of word lists
2016-02-20 00:39:02 +00:00
Kenneth Heafield
7a1baeecda
load= option to KenLM exposing more load_method options
2016-02-20 00:07:48 +00:00
Ulrich Germann
63eefe03fd
Merge branch 'master' of https://github.com/moses-smt/mosesdecoder
2016-02-19 10:23:21 +00:00
Ulrich Germann
98bcf53ec2
Namespace bug fix.
2016-02-18 16:39:34 +00:00
Hieu Hoang
6a363d893c
Merge ../mosesdecoder into perf_moses2
2016-02-17 17:39:51 +00:00
Hieu Hoang
0a1856d510
Merge ../mosesdecoder into perf_moses2
2016-02-17 17:38:07 +00:00
Ales Tamchyna
ca6dba9108
update VW compatibility
2016-02-17 11:27:44 +01:00
Hieu Hoang
0d0f75b6e0
gets right score
2016-02-16 16:03:37 +00:00
Ulrich Germann
d7db5d2441
Merge branch 'master' of https://github.com/moses-smt/mosesdecoder
2016-02-15 16:49:03 +00:00
Ulrich Germann
bbcd10a62d
Better option handling in check-coverage3.
2016-02-15 16:48:46 +00:00
MosesAdmin
ca5178b684
daily automatic beautifier
2016-02-13 00:02:04 +00:00
Hieu Hoang
938ef79404
Merge ../mosesdecoder into perf_moses2
2016-02-12 19:40:04 +00:00
Matthias Huck
1659d6b4c8
Option for target constituent constrained phrase extraction. TargetConstituentAdjacencyFeature.
2016-02-12 17:46:57 +00:00
Ulrich Germann
490edf7a82
Merge branch 'master' of https://github.com/moses-smt/mosesdecoder
2016-02-06 15:20:09 +00:00
Ulrich Germann
105225d398
Bug fix.
2016-02-06 15:19:44 +00:00
Hieu Hoang
06769e648d
Merge ../mosesdecoder into perf_moses2
2016-02-04 13:05:36 +00:00