moses-smt/mosesdecoder
1
1
Fork 0
mirror of https://github.com/moses-smt/mosesdecoder.git synced 2024-11-10 10:59:21 +03:00
Code Issues Packages Projects Releases Wiki Activity
12,583 Commits 136 Branches 26 Tags 147 MiB
e7afe5701e
Commit Graph

3797 Commits

Author SHA1 Message Date
Ulrich Germann
ead3fc565e Initial check-in. 2014-06-12 18:30:41 +01:00
Hieu Hoang
b6e497caa0 Merge ../mosesdecoder into nikolay_probingPT 2014-06-12 17:42:21 +01:00
Hieu Hoang
f3ec79d278 Merge branch 'master' of github.com:moses-smt/mosesdecoder 2014-06-12 13:15:51 +01:00
Hieu Hoang
eeedaaa09c incorrectly used boost shared_ptr 2014-06-12 13:15:02 +01:00
Hieu Hoang
31f662264e incorrectly used boost shared_ptr 2014-06-12 13:10:56 +01:00
Matthias Huck
9a7e568760 SourceLabelsPhraseProperty 2014-06-11 21:08:22 +01:00
Matthias Huck
7380a83f75 CountsPhraseProperty: add comment to header 2014-06-11 20:26:24 +01:00
Matthias Huck
e693a27e4e A simple phrase property class to access the three phrase count values. 
The counts are usually not needed during decoding and are not loaded
from the phrase table. This is just a workaround that can make them
available to features which have a use for them.

If you need access to the counts, copy the two marginal counts and the
joint count into an additional information property with key "Counts",
e.g. using awk:

$ zcat phrase-table.gz | awk -F' \|\|\| '  '{printf("%s {{Counts %s}}\n",$0,$5);}' | gzip -c > phrase-table.withCountsPP.gz

CountsPhraseProperty reads them from the phrase table and provides
methods GetSourceMarginal(), GetTargetMarginal(), GetJointCount().
 2014-06-11 20:02:31 +01:00
Hieu Hoang
ad4825ad44 integrated probing pt into decoder 2014-06-11 14:25:18 +01:00
Hieu Hoang
253f8e35fd integrated probing pt into decoder 2014-06-11 13:52:20 +01:00
XapaJIaMnu
095877742c Add ProbingPT files to Moses 2014-06-11 11:47:00 +01:00
Hieu Hoang
8d5328a957 ignore 0 span. Don't bomb out 2014-06-11 05:05:22 +01:00
Hieu Hoang
a96b0fcce3 Merge branch 'hieu' of github.com:hieuhoang/mosesdecoder into hieu 2014-06-10 18:05:00 +01:00
Hieu Hoang
341969e61d Merge ../mosesdecoder into hieu 2014-06-10 18:04:55 +01:00
Hieu Hoang
a695c9b50b Merge branch 'master' of github.com:moses-smt/mosesdecoder 2014-06-10 17:27:34 +01:00
Hieu Hoang
4b98495e78 don't load data when just showing weightd 2014-06-10 17:27:20 +01:00
Ulrich Germann
106a66eb17 Merge branch 'master' of https://github.com/moses-smt/mosesdecoder 2014-06-10 14:28:59 +01:00
Hieu Hoang
a27835727e make sure int are cast to float 2014-06-10 11:18:14 +01:00
Hieu Hoang
31a5b78b91 bug in SpanLength FF 2014-06-10 10:44:25 +01:00
Hieu Hoang
8c55f98c7a bug in SpanLength FF 2014-06-10 10:29:49 +01:00
Barry Haddow
91ccf8ef72 bidirectional case 2014-06-10 10:23:48 +01:00
Hieu Hoang
0ab1142238 Merge github.com:moses-smt/mosesdecoder into hieu 2014-06-10 10:17:05 +01:00
Hieu Hoang
8edb344492 minor const£ 2014-06-10 10:16:17 +01:00
Barry Haddow
a5e5a6590b basic implementation of non-hierarchical sparse features 2014-06-09 22:17:05 +01:00
Hieu Hoang
1fb71da5b5 delete debugging messages 2014-06-09 16:17:20 +01:00
Hieu Hoang
649b31c226 span length 2014-06-09 16:01:37 +01:00
Hieu Hoang
3c6a31128d Merge ../mosesdecoder into hieu 2014-06-08 17:07:41 +01:00
Hieu Hoang
d68257c34d use standard c++ getline instead of old Moses SAFE_GETLINE 2014-06-08 16:37:12 +01:00
Hieu Hoang
45ed0a5b1f Merge ../mosesdecoder into hieu 2014-06-08 13:22:34 +01:00
Barry Haddow
3dec0abf0a Call sparse reordering 2014-06-06 21:08:09 +01:00
Hieu Hoang
747661ba3e sparse features in on-disk-pt 2014-06-06 20:35:58 +01:00
Barry Haddow
4aa4fe0a04 Pass scc, not scores 2014-06-06 20:25:45 +01:00
Matthias Huck
a19aaada06 formatting 2014-06-06 20:16:40 +01:00
Matthias Huck
98a117ca37 rule-limit=0: unlimited 2014-06-06 20:16:19 +01:00
Matthias Huck
2ec50239e7 rule-limit=0: unlimited 2014-06-06 20:05:59 +01:00
Hieu Hoang
091ce3f016 Merge ../mosesdecoder into hieu 2014-06-06 17:25:26 +01:00
Barry Haddow
18d1bceea0 Move sparse reordering object to LR config 2014-06-06 14:20:23 +01:00
Hieu Hoang
e25e7b69be span length 2014-06-06 10:40:40 +01:00
phikoehn
ceadacd3af Merge branch 'master' of ssh://github.com/moses-smt/mosesdecoder 2014-06-05 21:33:35 +01:00
Philipp Koehn
15288213be allow < > in factors 2014-06-05 21:31:09 +01:00
Hieu Hoang
d95bd5bdae span length property 2014-06-05 20:57:00 +01:00
Hieu Hoang
a17a45fa7f span length 2014-06-05 17:20:38 +01:00
Hieu Hoang
b8e8d6a3d2 span length 2014-06-05 17:19:06 +01:00
Hieu Hoang
ce2a69ba25 Merge ../mosesdecoder into hieu 2014-06-05 17:18:26 +01:00
Hieu Hoang
33a0b97cc9 Merge branch 'master' of github.com:moses-smt/mosesdecoder 2014-06-05 17:15:24 +01:00
Hieu Hoang
d29bf716a4 minor change to debugging output. Memory pt used by both pb and chart-based models 2014-06-05 17:15:08 +01:00
Jun-ya NORIMATSU
cc71e0ee54 bugfix. 2014-06-05 19:26:27 +09:00
Jun-ya NORIMATSU
0659c31c05 Merge remote-tracking branch 'origin/master' 2014-06-05 14:24:44 +09:00
Jun-ya NORIMATSU
1180f5dd70 bug fix. 2014-06-05 14:24:10 +09:00
Ulrich Germann
5116f0072b Minor edits to ug_bitext.h. Added min_diverse to ug_bitext::job to ensure minimum number of translation alternatives before sampling stops. 2014-06-05 01:50:55 +01:00
Ulrich Germann
b92d599727 Bug fix in mmlex-lookup. 2014-06-05 01:48:11 +01:00
Ulrich Germann
2f109621bf Added configurable options and SetTableLimit to Mmsapt. 2014-06-05 01:47:29 +01:00
Ulrich Germann
a40fcbae02 Added utility lookup_mmsapt 2014-06-05 01:45:34 +01:00
Ulrich Germann
22ec93b85c Added operator [] to TargetPhraseCollection. 2014-06-05 01:44:11 +01:00
Ulrich Germann
c7a0520a18 Made moses shut up by changing unconditional 'cerr's to VERBOSE(1,...) 2014-06-05 01:43:26 +01:00
Ulrich Germann
004b8c9078 Changed Phrase.m_words from private to protected. 2014-06-05 01:41:50 +01:00
Ulrich Germann
3145bf3cc4 Commented out unused variable. 2014-06-05 01:40:51 +01:00
Ulrich Germann
0e98a08446 Commented out unused variable. 2014-06-05 01:40:27 +01:00
Ulrich Germann
5ae57f09d7 Commented out unused variable. 2014-06-05 01:39:44 +01:00
Ulrich Germann
ce853731ae Added mmsapt lookup utility. 2014-06-05 01:38:55 +01:00
Hieu Hoang
21da8299b8 eclipse 2014-06-04 23:02:11 +01:00
Hieu Hoang
c5aac0676e don't output SpanLength property if no non-terms 2014-06-04 17:15:54 +01:00
Hieu Hoang
4a3ac7411d span length 2014-06-04 16:52:57 +01:00
Barry Haddow
2a6a99c8bf Merge remote-tracking branch 'origin/master' into sparse-reordering 2014-06-04 16:25:21 +01:00
Hieu Hoang
8e2c4a89c0 Merge ../mosesdecoder into hieu 2014-06-04 12:53:06 +01:00
Barry Haddow
1a4b641421 Merge branch 'master' of github.com:moses-smt/mosesdecoder 2014-06-03 22:13:17 +01:00
Ulrich Germann
8dee1725fb Removed Phrase penalty as a built-in feature function. 2014-06-03 21:36:04 +01:00
Ulrich Germann
9c27dc08e9 Merge branch 'master' of https://github.com/moses-smt/mosesdecoder 
Conflicts:
	moses/FF/ConstrainedDecoding.cpp
 2014-06-03 18:55:53 +01:00
Hieu Hoang
fcf9e4b51c Merge ../mosesdecoder into hieu 2014-06-03 17:11:16 +01:00
Kenneth Heafield
346a8f41b0 Use boost::ref for Visual Studio / Mike Ladwig 2014-06-03 08:48:10 -07:00
Ulrich Germann
9cda39f802 Major bug fix in setting ug_bitext::jstats.my_cnt2. 2014-06-03 15:48:27 +01:00
Ulrich Germann
8c9a26a3be Added phrase feature function for log counts. 2014-06-03 15:45:29 +01:00
Ulrich Germann
0fc80660f3 TSA_tree_iterator::approxOccurrenceCount() now returns exact count for small values. 2014-06-03 15:44:20 +01:00
Ulrich Germann
a248349da3 Added more phrase table features and configuration options to Mmsapt. 2014-06-03 15:41:59 +01:00
Ulrich Germann
077f2ab51e Added some code to Word.cpp to allow factor-less decoding. 
Factor-less decoding currently works ONLY with suffix-array-based phrase tables.
 2014-06-03 15:38:28 +01:00
Hieu Hoang
038f213d10 Merge ../mosesdecoder into hieu 2014-06-03 13:24:12 +01:00
Jun-ya NORIMATSU
65bf623228 tuning. 2014-06-03 16:13:32 +09:00
Hieu Hoang
3f49dfaaef Merge branch 'hieu' of github.com:hieuhoang/mosesdecoder into hieu 2014-06-02 18:18:28 +01:00
Hieu Hoang
39ef678b32 Merge ../mosesdecoder into hieu 2014-06-02 18:18:17 +01:00
Hieu Hoang
31a583b0bc update comments for Evaluate() to say something about stackVec 2014-06-02 18:17:39 +01:00
Barry Haddow
5b0c696710 Range check in OSM 2014-06-02 16:23:55 +01:00
Hieu Hoang
934dd9b0ad Merge branch 'hieu' of github.com:hieuhoang/mosesdecoder into hieu 2014-06-01 20:37:44 +01:00
Hieu Hoang
e260241d1f Merge ../mosesdecoder into hieu 2014-06-01 20:30:05 +01:00
Hieu Hoang
94b43823e0 can use NieceTerminal FF as non-tuneable 2014-06-01 20:29:55 +01:00
Ulrich Germann
ce435d22bf Mmsapt now adds the phrase penalty only if num_features specified 
on the respective line in moses.ini is odd.
 2014-06-01 18:23:00 +01:00
Hieu Hoang
c6d80b5303 Merge ../mosesdecoder into hieu 2014-05-31 19:53:59 +01:00
Hieu Hoang
c4d0f7dc93 Merge branch 'master' of github.com:moses-smt/mosesdecoder 2014-05-31 19:52:41 +01:00
Hieu Hoang
7909dcf094 add decoder arg -weight-add 2014-05-31 19:52:30 +01:00
Ulrich Germann
d1e5a544f1 Word.CreateFromString() now bypasses factor splitting if the factor delimiter is th empty string (specify as: none). 2014-05-31 15:22:37 +01:00
Ulrich Germann
249c40ec74 Eliminated factor delimiter from the parameters of 
Phrase.CreateFromString() and Sentence.CreateFromString(),
as it was never used in those functions anyway ---
Word.CreateFromString() retrieves the factor delimiter
from StaticData directly.
 2014-05-31 14:33:31 +01:00
Ulrich Germann
c452875343 Mmsapt/ug_bitext: 
- Sampling now uses a static seed for consistency between runs.
- (In order to have consistency guarantees, specify workers=1 in the
- Mmsapt line in moses.ini; otherwise the randomness of thread scheduling may
  cause inconsistencies between runs.
- Denominator for fwd phrase probabilities can now be chosen via an
  external argument.
 2014-05-31 14:29:44 +01:00
Hieu Hoang
1ecad718b1 merge 2014-05-31 00:26:42 +01:00
Marcin Junczys-Dowmunt
71818ee994 missing intialization of member 2014-05-30 17:53:17 +02:00
Marcin Junczys-Dowmunt
6cdd2b6019 optional soft constraints for constrained decodings, multiple reference files 2014-05-30 17:36:19 +02:00
Barry Haddow
10e26ef00d config of sparse reordering 2014-05-30 11:27:59 +01:00
Hieu Hoang
e173e8ffab merge 2014-05-29 00:27:09 +01:00
Hieu Hoang
5e55ef089e merge 2014-05-28 22:52:22 +01:00
Hieu Hoang
e9d3052790 hard/soft constraints wrong way round 2014-05-28 22:51:26 +01:00
Barry Haddow
912c9c1f55 Configuration 2014-05-27 13:36:58 +01:00
Barry Haddow
244d9cd824 Stub out sparse reordering class 2014-05-27 11:05:56 +01:00
Barry Haddow
1bd851411f fix includes 2014-05-27 09:52:18 +01:00
Barry Haddow
1c00f8d9a7 Should be with other LR classes 2014-05-26 13:47:32 +01:00
Barry Haddow
c1be2d47e5 Merge remote-tracking branch 'origin/master' into sparse-reordering 
Conflicts:
	moses/FF/Factory.cpp
 2014-05-26 11:48:51 +01:00
Hieu Hoang
bbfc6b1c76 hard constraint 2014-05-25 00:48:10 +01:00
Hieu Hoang
6774595f21 call ReadParameters() 2014-05-24 17:26:24 +01:00
Hieu Hoang
6b7059e801 Merge ../mosesdecoder into hieu 2014-05-21 13:45:13 +01:00
Hieu Hoang
401c4940cf debugged FF NieceTerminal 2014-05-21 13:43:32 +01:00
Hieu Hoang
b60ac0321f rename DuplicateWord -> NieceTerminal 2014-05-20 20:02:49 +01:00
Hieu Hoang
3f93534624 Merge branch 'master' of github.com:moses-smt/mosesdecoder 2014-05-20 19:52:31 +01:00
Hieu Hoang
e31ce4d3be add FF DuplicateWord 2014-05-20 19:51:46 +01:00
Hieu Hoang
409c8d5349 Merge ../mosesdecoder into hieu 2014-05-20 00:16:58 +01:00
Matthias Huck
d921d23f7d comment 2014-05-19 22:09:27 +01:00
Matthias Huck
1740478238 Framework for additional phrase properties in decoding. 
Derive your property class from PhraseProperty. Do any expensive string
processing of the property value in there, not in the feature
implementation, and provide methods to access the information in
appropriate data formats. The property value string will thus have to
be processed only once (on loading) rather than each time the respective
phrase is applied and your feature needs to access the property value.
 2014-05-19 21:54:08 +01:00
Ulrich Germann
380b2da2d0 Bug fix in and in connection with Mmsapt::combine_pstats(). 2014-05-19 21:34:58 +01:00
Hieu Hoang
238864a53e Merge ../mosesdecoder into hieu 2014-05-16 16:17:01 +01:00
Hieu Hoang
65a1e176e5 bugs in FF MaxSpanFreeNonTermSource 2014-05-16 16:16:20 +01:00
Hieu Hoang
1e96ffcf3d rename FF RuleAmbiguity to RuleScope 2014-05-16 12:47:43 +01:00
Hieu Hoang
0989087a5b comment 2014-05-16 10:39:26 +01:00
Hieu Hoang
35c812f5df Merge ../mosesdecoder into hieu 2014-05-13 18:24:17 +01:00
Barry Haddow
294ad4f24f Only tuneable 2014-05-13 12:02:50 +01:00
Phil Williams
fc85c26eda moses_chart: remove ChartTrellis* k-best code 2014-05-13 10:24:30 +01:00
Hieu Hoang
6c9bc9f988 delete chart translation option if -inf score. Add FF MaxSpanFreeNonTermSource to use it 2014-05-12 19:44:31 +01:00
Hieu Hoang
5f7c424f7f minor typo 2014-05-12 16:16:35 +01:00
Hieu Hoang
bf7c033271 backoff for default GetTargetPhraseCollectionBatch(), used by binary pt and compact pt 2014-05-12 15:50:32 +01:00
Hieu Hoang
2830ed9968 use Nadir's backoff code 2014-05-12 15:40:18 +01:00
Hieu Hoang
1d401817a5 backoff fix for memory and on-disk phrase-tables 2014-05-12 15:16:11 +01:00
Hieu Hoang
44fec57c53 backoff fix 2014-05-12 14:06:47 +01:00
Hieu Hoang
f5851ecfbd Merge ../mosesdecoder into hieu 2014-05-08 21:44:26 +01:00
Hieu Hoang
6a80643ab6 minor compile error with clang 2014-05-08 21:44:06 +01:00
Hieu Hoang
7632fbd51c Merge ../mosesdecoder into hieu 2014-05-08 21:09:35 +01:00
Hieu Hoang
91afa4b4f1 delete chart translation option if -inf score 2014-05-08 18:48:10 +01:00
Hieu Hoang
b64d87f1ba Merge branch 'master' of github.com:moses-smt/mosesdecoder 2014-05-08 17:58:22 +01:00
Hieu Hoang
15566f7fc5 extend FF framework by add StackVec* argument to 2nd Evaluate(). NULL if phrase-based 2014-05-08 17:51:45 +01:00
Hieu Hoang
3ef5d76860 add MaxSpanFreeNonTermSource. Need to change FF framework to take in derivation 2014-05-08 16:34:04 +01:00
Barry Haddow
171e151705 Only surface factor in hypergraph 2014-05-08 16:31:29 +01:00
Hieu Hoang
594aee3322 Merge branch 'master' of ../mosesdecoder into hieu 2014-05-08 11:42:19 +01:00
Hieu Hoang
14d892a858 Merge branch 'master' of github.com:moses-smt/mosesdecoder 2014-05-08 11:26:32 +01:00
Hieu Hoang
d12f91841a Segfault in nbest list creation. UTIL_THROW_IF is the opposite of assert 2014-05-08 11:26:17 +01:00
Hieu Hoang
814fb20aff 0 scores for FF SetSourcePhrase 2014-05-07 19:29:09 +01:00
Hieu Hoang
4e977f0f1f Merge ../mosesdecoder into hieu 2014-05-07 12:21:49 +01:00
Hieu Hoang
b436570ab0 delete unused files 2014-05-07 12:21:19 +01:00
Hieu Hoang
71242a3578 Merge ../mosesdecoder into hieu 2014-05-07 12:11:24 +01:00
Barry Haddow
5dbf47afd3 Want number of source words covered in hg output 2014-05-02 12:01:03 +01:00
Barry Haddow
10c4077ae9 Add source-span to hg output, comments 2014-05-01 21:37:07 +01:00
Hieu Hoang
2beed771d2 minor error. Segfault if phrase length is bigger than max-phrase-length 2014-05-01 16:47:17 +01:00
Hieu Hoang
d4b270e796 redo creation of input paths for lattices. Should redo OOV handling as phrase-table 2014-05-01 15:10:16 +01:00
Hieu Hoang
17c10d45f3 merge 2014-05-01 14:52:38 +01:00
Hieu Hoang
1465cf94b7 debugging info 2014-04-30 20:47:58 +01:00
Hieu Hoang
26abab8e5c debugging info 2014-04-30 20:20:07 +01:00
Ulrich Germann
ae4ab9aae0 Merge branch 'master' of https://github.com/moses-smt/mosesdecoder 2014-04-29 15:50:39 +01:00
Hieu Hoang
4c74a5d638 bug in lattice decoding. Some arcs can span more than 1 node 2014-04-29 14:41:20 +01:00
Ulrich Germann
07d8b66366 Merge branch 'master' of https://github.com/moses-smt/mosesdecoder 2014-04-29 11:01:46 +01:00
Hieu Hoang
6182750b70 Merge branch 'master' of github.com:moses-smt/mosesdecoder 2014-04-28 14:29:49 +01:00
Hieu Hoang
9e33f37fb4 bug in lattice decoding. Some target phrase collection can have 0 size. Should execute OOV processing 2014-04-28 14:29:39 +01:00
Ulrich Germann
7c145d045b Merge branch 'master' into dynamic-phrase-tables 
Conflicts:
	contrib/server/Jamfile
	contrib/server/mosesserver.cpp
 2014-04-28 10:00:07 +01:00
Hieu Hoang
da6ade7d94 get-by-line-number.perl 2014-04-24 17:48:12 +01:00
Ulrich Germann
e5b739946f Merge branch 'master' of https://github.com/moses-smt/mosesdecoder 2014-04-23 18:29:39 +01:00
Hieu Hoang
d7380d6d9e don't add label to m_targetLabelSet if no hypotheses. Assert error in parsing 2014-04-23 17:11:25 +01:00
Hieu Hoang
2c14b506b4 merge problem 2014-04-23 16:11:09 +01:00
Hieu Hoang
6a9eb6c848 minor leak showing for lex reordering. Just refactor 2014-04-23 15:12:27 +01:00
Hieu Hoang
3f32e48f97 Merge branch 'master' of github.com:moses-smt/mosesdecoder 2014-04-23 12:11:50 +01:00
Hieu Hoang
00505ba048 minor leak 2014-04-23 11:36:25 +01:00
Hieu Hoang
b917ac552f strange assert failure when stack is empty 2014-04-22 19:05:08 +01:00
Hieu Hoang
eed865bd35 Merge ../mosesdecoder into hieu 2014-04-21 23:05:25 +01:00
Phil Williams
ff8ac92be1 moses_chart: oops, don't output more distinct translations than requested 2014-04-21 14:46:22 +01:00
Phil Williams
b308bd5657 moses_chart: enable ChartKBestExtractor. 2014-04-21 10:19:00 +01:00
Hieu Hoang
51f020b2c4 Merge ../mosesdecoder into hieu 2014-04-20 00:20:25 +01:00
Phil Williams
568685cb66 ChartKBestExtractor: fix memory leak, clean-up code 2014-04-19 10:29:41 +01:00
Hieu Hoang
def22cef44 Merge branch 'hieu' of github.com:hieuhoang/mosesdecoder into hieu 2014-04-18 18:11:56 +01:00
Hieu Hoang
7f41ab2390 trying to count the total number of possible translations 2014-04-18 16:32:29 +01:00
Barry Haddow
7681cd4a15 Merge branch 'master' of github.com:moses-smt/mosesdecoder 2014-04-17 20:50:27 +01:00
Hieu Hoang
8d69327eb1 merge 2014-04-17 20:22:40 +01:00
Hieu Hoang
a4d32a2b09 minor compile errors in ChartKBestExtractor 2014-04-17 20:04:02 +01:00
Barry Haddow
b9c25afd45 Merge branch 'master' of github.com:moses-smt/mosesdecoder 2014-04-17 19:02:57 +01:00
Phil Williams
66d0fe81e2 moses_chart: add ChartKBestExtractor (not enabled yet) 
Implements algorithm 3 from Huang and Chiang (2005)
 2014-04-15 17:34:21 +01:00
Ulrich Germann
abfbd936d7 Merge branch 'dynamic-phrase-tables' of file:///fs/sif0/germann/git/mosesdecoder into dynamic-phrase-tables 2014-04-14 00:12:21 +01:00
Ulrich Germann
27d233796d Bug fix in Word::CreateFromString(...). 2014-04-12 22:10:40 +01:00
Hieu Hoang
75213181ae Merge branch 'master' of github.com:moses-smt/mosesdecoder 2014-04-12 17:20:49 +02:00
Ulrich Germann
f85bf72a8e Merge branch 'master' into dynamic-phrase-tables 
Conflicts:
	moses/TranslationModel/UG/mmsapt.cpp
 2014-04-11 19:24:37 +01:00
Ulrich Germann
b7f2e18efc Rewrite of Word::CreateFromString() to accommodate elimination of 
ConfusionNet::String2Word().
 2014-04-11 15:24:52 +01:00
Hieu Hoang
e347020049 add FF ReferenceComparison 2014-04-11 11:03:50 +01:00
Hieu Hoang
0ec2fe0169 add FF RuleAmbiguity 2014-04-11 10:42:38 +01:00
Hieu Hoang
1e116a21ae add FF RuleAmbiguity 2014-04-11 10:29:57 +01:00
Hieu Hoang
9644a30858 add FF RuleAmbiguity 2014-04-11 10:22:03 +01:00
Ulrich Germann
24545ba1de Changes due to new pure virtual function in PhraseTable. 2014-04-11 04:28:34 +01:00
Ulrich Germann
84a72198d1 Shortcut in Word::CreateFromString() didn't work. 2014-04-11 00:37:38 +01:00
Hieu Hoang
1686686e65 start on FF ReferenceComparison 2014-04-10 21:47:06 +01:00
Hieu Hoang
91d6bfe0d5 start on FF ReferenceComparison 2014-04-10 21:34:34 +01:00
Kenneth Heafield
40c819d285 Merge branch 'master' of github.com:moses-smt/mosesdecoder 2014-04-09 10:03:02 -07:00
Kenneth Heafield
c61231fad9 Fix BackwardTest failure which was just compiler-drive differences in floating point precision. 
BOOST_CHECK_CLOSE's tolerance is proportional to the value on the left, so 0.0 has zero tolerance.  Changed to an explicit range.
 2014-04-09 10:01:19 -07:00
Hieu Hoang
65326f205f -stack 0 means no stack pruning 2014-04-09 10:33:33 +01:00
Hieu Hoang
0e98f1bc06 bug in ControlRecombinationState 2014-04-09 09:59:52 +01:00
Hieu Hoang
4c634023ce add FF CountNonTerms 2014-04-08 17:19:26 +01:00
Hieu Hoang
7ae44a96f1 add FF CountNonTerms 2014-04-08 17:16:24 +01:00
Ulrich Germann
21f141ce0d Merge branch 'master' into dynamic-phrase-tables 
Conflicts:
	moses/InputPath.cpp
	moses/Word.cpp
 2014-04-08 14:58:29 +01:00
Ulrich Germann
ad678ac0ec Replaced calls to String2Word by calls to Word::CreateFromString(). String2Word is deprecated (code duplication) and should be removed. Some reformatting for readability. 2014-04-08 12:44:21 +01:00
Ulrich Germann
43a51bb5d7 Some reformatting for readability. Special check for factor-less operation in Word::CreateFromString(). 2014-04-08 12:43:21 +01:00
Ulrich Germann
f82e79d237 Replaced calls to String2Word by calls to Word::CreateFromString(). String2Word is deprecated (code duplication) and should be removed. 2014-04-08 12:41:30 +01:00
Hieu Hoang
bf11ff6251 merge error 2014-04-07 14:07:16 +01:00
Hieu Hoang
6b1e147cff add -default-non-term-for-empty-range-only 2014-04-07 14:00:40 +01:00
Hieu Hoang
5d61639763 Merge branch 'hieu' of github.com:hieuhoang/mosesdecoder into hieu 2014-04-07 12:01:55 +01:00
Hieu Hoang
8b7e122f3d Merge ../mosesdecoder into hieu 2014-04-07 12:00:47 +01:00
Hieu Hoang
2ea2224ba1 output source side of rule, if available 2014-04-07 12:00:13 +01:00
Hieu Hoang
59e26ff54d output source side of rule, if available 2014-04-07 11:54:57 +01:00
Hieu Hoang
daf4e57ea5 no scores for FF SetSourcePhrase 2014-04-07 11:39:34 +01:00
Hieu Hoang
b98dac2971 merge 2014-04-06 19:07:13 +01:00
Hieu Hoang
09dc0e9913 add FF/SetSourcePhrase 2014-04-06 16:57:54 +01:00
Rico Sennrich
395285d879 more logging for incremental chart search 2014-04-04 15:54:48 +01:00
Ulrich Germann
8c04078d14 Reformatting for readability. 2014-04-03 20:33:40 +01:00
Hieu Hoang
6fa561e39e comment out debugging message 2014-04-03 18:24:48 +01:00
Hieu Hoang
67b1cbce13 error in constrained decoding 2014-04-03 18:03:58 +01:00
Hieu Hoang
5c633fcbf8 error in constrained decoding 2014-04-03 18:01:52 +01:00
Hieu Hoang
7e7cc64c1d debugging 2014-04-03 17:21:31 +01:00
Hieu Hoang
0953b5caca debugging 2014-04-03 16:54:46 +01:00
Rico Sennrich
1f435340f0 faster pruning in chart decoding 2014-03-26 11:23:59 +00:00
Ulrich Germann
f63606a92b 1. Renamed PhraseDictionary member function release() to Release(). 
2. Added option to check phrase prefixes for early stopping of
   InputPath generation when translating confusion networks.
 2014-03-25 01:49:24 +00:00
Ulrich Germann
80b031769d (1) Bug fixes related to locking. (2) pstats cache disabled when translating confusion networks. 2014-03-25 01:48:49 +00:00
Ulrich Germann
84299006da Added another constructor to TSA_tree_iterator. 2014-03-25 01:43:00 +00:00
Ulrich Germann
f5d969ab37 Added missing library for custom-pt.cc 2014-03-25 01:42:06 +00:00
Ulrich Germann
344b7a0759 Changed verbosity level for "read confusion net with format ..." to 2. 2014-03-25 01:41:21 +00:00
Hieu Hoang
5e5ad96729 segfault when there's nothing in the stack 2014-03-24 16:50:43 +00:00
Ulrich Germann
728ceb4e4c Added missing #include. 2014-03-24 14:51:51 +00:00
Ulrich Germann
14e52c9b94 Errors occurring when accessing data files now throw exceptions. 2014-03-24 13:39:15 +00:00
Ulrich Germann
ed825cd234 Bug fix: dependency .../UG/generic//generic was missing for custom-pt 2014-03-24 13:33:46 +00:00
Ulrich Germann
20aa9dd189 Adapted to configurable map type in pstats. 2014-03-24 13:32:41 +00:00
Ulrich Germann
794defa238 Numerous bug fixes; implemented caching and improved life cycle 
management for TargetPhraseCollections.
 2014-03-24 13:30:22 +00:00
Ulrich Germann
7fd87943ea Reinstated pstats caching. 2014-03-24 13:29:05 +00:00
germann
2063da274d Made member Tokenindex.ridx mutable. 2014-03-24 12:34:59 +00:00
germann
5ec4436b07 Initial check-in. 2014-03-24 12:33:17 +00:00
germann
c12c25770a Added virtual member function release(TargetPhraseCollection*) to 
PhraseDictionary, which gets called by the destructor of InputPath to
release TargetPhraseCollections that aren't needed any more.
 2014-03-24 12:29:46 +00:00
germann
042e840045 Added mmlib as a dependency for FF_Factory.o 2014-03-24 12:23:18 +00:00
germann
f7ec2ee09a 1. Fixed order of initializations in constructor so gcc won't complain. 
2. Added call  to PhraseDictionary.release(TargetPhraseCollection*) to
   release used TargetPhraseCollection's.
 2014-03-24 10:55:47 +00:00
germann
ec6fd21603 Make sure m_numInputScores is initialized properly. 2014-03-24 10:53:29 +00:00
Hieu Hoang
e5a6d63cf2 Merge branch 'master' of github.com:moses-smt/mosesdecoder into hieu 2014-03-21 21:41:04 +00:00
Ulrich Germann
1f32bd0d08 Fixed header inclusions. 2014-03-21 16:49:44 +00:00
Rico Sennrich
fb16df8c00 typo in last commit 2014-03-21 11:59:41 +00:00
Rico Sennrich
45630a5851 various optimizations to make CYK+ parser several times faster and eat less memory. 
speed-up of decoding depends on how much time is spent in parser:
10-50% speed-up for string-to-tree systems observed (more on long sentences and with high max-chart-span).

if you only use hiero or string-to-tree models (but none with source syntax), use compile-option --unlabelled-source for (small) efficiency gains.
 2014-03-21 11:12:24 +00:00
Ulrich Germann
4685aeadd2 Bug fix. 2014-03-21 11:11:24 +00:00
Ulrich Germann
d5c96eff35 Minor edit of a comment. 2014-03-21 11:08:37 +00:00
Ulrich Germann
f030986f51 Bug fixes. Added some sanity checks. 2014-03-21 11:07:17 +00:00
Ulrich Germann
43e6c68586 Changed default method for PhraseScoreFwd. Added minor debugging message. 2014-03-19 20:54:49 +00:00
Ulrich Germann
7e344d1c53 1. Added variant of CreateRuleLookupManager variant as required by the new 
pure virtual function in PhraseTable.h (from the master branch).

2. Bug fix in pool_stats.
 2014-03-19 20:43:47 +00:00
Ulrich Germann
c91fb5cc84 Added mmlex-lookup utility program. 2014-03-19 20:41:29 +00:00
Hieu Hoang
172a33c53c probing pt source vocab 2014-03-19 13:20:00 +00:00
Hieu Hoang
f4befcf125 probing pt source vocab 2014-03-19 12:59:55 +00:00
Hieu Hoang
49d3564a1b changes to probing pt v2 2014-03-19 11:39:27 +00:00
Ulrich Germann
ef974cd6ad Merge branch 'master' into dynamic-phrase-tables 2014-03-18 18:25:35 +00:00
Ulrich Germann
740c9dc096 Memory-mapped stuff is now compiled only with --with-mm. 2014-03-18 18:23:59 +00:00
Ulrich Germann
ef2c012470 Merge branch 'master' into dynamic-phrase-tables 
Conflicts:
	moses/Jamfile
 2014-03-18 18:23:23 +00:00
Ulrich Germann
febb2afc4d Initial check-in. 2014-03-18 12:23:53 +00:00