moses-smt/mosesdecoder
1
1
Fork 0
mirror of https://github.com/moses-smt/mosesdecoder.git synced 2024-12-29 23:12:41 +03:00
Code Issues Packages Projects Releases Wiki Activity
17,021 Commits 136 Branches 26 Tags 147 MiB
02772c07de
Commit Graph

2764 Commits

Author SHA1 Message Date
Hieu Hoang
63d547897c exit if errors in parameter parsing 2016-08-24 13:45:55 +01:00
Hieu Hoang
5542103b54 leak in distinct nbest 2016-08-24 13:17:29 +01:00
Hieu Hoang
d5131bae27 leak 2016-08-24 12:51:44 +01:00
Hieu Hoang
07a9d3b5f5 nbest now monotonic 2016-08-24 12:31:33 +01:00
Hieu Hoang
dd09d4358a debug 2016-08-24 11:14:56 +01:00
Hieu Hoang
8dc7d5612f debug 2016-08-23 22:38:27 +01:00
Hieu Hoang
392abaf2e8 debug 2016-08-23 22:01:14 +01:00
Hieu Hoang
e0083e238d const method 2016-08-23 21:40:17 +01:00
Hieu Hoang
4b2986c578 debug non-monotonic nbest list 2016-08-23 21:25:45 +01:00
Hieu Hoang
7fa940f105 priority queue 2016-08-23 19:09:09 +01:00
Hieu Hoang
933e4d29ea create deviants 2016-08-23 18:30:28 +01:00
Hieu Hoang
3af84b6fc9 create deviants 2016-08-23 16:57:25 +01:00
Hieu Hoang
5d33d7f6e8 leaks 2016-08-23 16:37:06 +01:00
Hieu Hoang
31627f9d77 store nbest rather than arclist 2016-08-23 15:52:30 +01:00
Hieu Hoang
8c99db123f rename 2016-08-23 15:38:01 +01:00
Hieu Hoang
e1fd84b864 correct string output 2016-08-23 15:18:03 +01:00
Hieu Hoang
955e4156f6 output scores 2016-08-23 14:49:14 +01:00
Hieu Hoang
589c036199 output scores 2016-08-23 14:30:38 +01:00
Hieu Hoang
34ac59ccc6 redo nbest list. Key nbest candidate to hypo states instead of hypo collection or stack 2016-08-23 13:50:03 +01:00
Hieu Hoang
3153a61320 delete SCFG::TrellisPath 2016-08-23 11:09:23 +01:00
Hieu Hoang
848e756d46 Merge branch 'master' of github.com:moses-smt/mosesdecoder 2016-08-22 13:03:50 +01:00
Hieu Hoang
d16f47cb92 const input arg 2016-08-22 13:03:45 +01:00
Hieu Hoang
5c145f7414 get weights from param::allWeights. Need to be able to override weights when tuning 2016-08-18 22:23:21 +01:00
Hieu Hoang
2e5b61af86 consistent function name 2016-08-18 19:03:43 +01:00
Hieu Hoang
d9255131ed delete unused method 2016-08-18 18:31:34 +01:00
Hieu Hoang
54e189ab2e don't fix precision for nbest list 2016-08-18 17:41:37 +01:00
Hieu Hoang
3695e30ac0 use hash of strings and unordered set for distinct hypos 2016-08-18 17:24:53 +01:00
Hieu Hoang
757c177a7e typo 2016-08-18 10:00:20 +01:00
Hieu Hoang
3da0b1aa49 unk penalty must also have a active chart entry so that it can share symbolbin with inputPath 2016-08-17 15:17:14 +01:00
Hieu Hoang
682bba480e store tps in a list rather than map 2016-08-17 14:40:08 +01:00
Hieu Hoang
c6c10ec551 init m_hypoIndColl 2016-08-17 14:10:32 +01:00
Hieu Hoang
5054aa053f use reference rather than pointer to tps 2016-08-17 13:47:55 +01:00
Hieu Hoang
a15997cb09 m_hypoIndColl in queueitem is a pointer to that in seen position. They have the same info 2016-08-17 13:43:47 +01:00
Hieu Hoang
b1278e582e store seen position even for 1st position 2016-08-17 13:34:29 +01:00
Hieu Hoang
3ea68b4a6c add symbolbind to seen position. May have same translation rule but bind to different NT 2016-08-17 13:03:23 +01:00
Hieu Hoang
dd6bd92a21 make target phrases collection a variable, not a pointer 2016-08-16 13:19:21 +01:00
Hieu Hoang
e9ec2ddd99 delete special method for Unk penalty 2016-08-16 13:11:09 +01:00
Hieu Hoang
d6e5827e8a get rid of PhraseTableOnDisk 2016-08-16 11:42:21 +01:00
Hieu Hoang
beb96b3491 get rid of range from SymbolBindElement 2016-08-16 11:36:45 +01:00
Hieu Hoang
4f63ff2754 calc kenlm score in isolation 2016-08-15 23:14:09 +01:00
Hieu Hoang
54f0035e84 start sorting scfg target phrases by all weighted scores, not just p(e|f) 2016-08-15 22:46:29 +01:00
Hieu Hoang
b5c38e916c debug 2016-08-15 18:28:03 +01:00
Hieu Hoang
1e2cfd5526 debug 2016-08-15 14:13:57 +01:00
Hieu Hoang
f6effea29a output transopt 2016-08-15 13:21:29 +01:00
Hieu Hoang
b1ae8d133f debug 2016-08-15 11:49:10 +01:00
Hieu Hoang
6db5165038 fixed precision 2016-08-14 20:56:08 +01:00
Hieu Hoang
dffe9c2cca delete AllocateWithAlign(). Always align 2016-08-13 15:16:09 +01:00
Hieu Hoang
96a8e14abc always alignment memory to 4 bytes 2016-08-13 15:11:07 +01:00
Hieu Hoang
5c432e5349 Merge branch 'perf_moses2' of github.com:hieuhoang/mosesdecoder into perf_moses2 2016-08-13 11:42:17 +01:00
Hieu Hoang
ae654adf83 Merge ../mosesdecoder into perf_moses2 2016-08-13 11:42:05 +01:00
Hieu Hoang
3de9f8b66c Merge branch 'perf_moses2' of github.com:hieuhoang/mosesdecoder into perf_moses2 2016-08-12 14:19:48 +02:00
Hieu Hoang
4fad95b9e0 delete debug message 2016-08-12 14:19:32 +02:00
Michael Denkowski
3aedc0bf68 Standalone phrase distance feature 
(Uses input coordinates populated by XML input and target phrase coordinates
 populated by phrase dictionary implementation)
 2016-08-12 12:05:12 +02:00
Michael Denkowski
d29916bbb3 Roll back to just tracking coordinates in mmsapt 
(distance wants to be a separate FF)
 2016-08-12 12:05:12 +02:00
Michael Denkowski
5c2b8d843c Distance feature for mmsapt 
- Make ttask visible to scorers
- Only track sentence ids if using distance feature
 2016-08-12 12:05:12 +02:00
Hieu Hoang
c42bb54c04 formatting of -show-weights to make it work with mert script 2016-08-12 09:20:43 +01:00
Hieu Hoang
a8325a3e8e make probing pt work with ems 2016-08-11 15:46:43 +01:00
Hieu Hoang
5f3239963e segfault with gcc 2016-08-11 16:28:00 +02:00
Hieu Hoang
984894ffad show weights 2016-08-11 15:24:32 +02:00
Hieu Hoang
151c0e0ba5 Merge ../mosesdecoder into perf_moses2 2016-08-11 15:00:26 +02:00
Hieu Hoang
7eb759d1d9 start show weights 2016-08-11 14:00:55 +02:00
Hieu Hoang
33ec4cb7ab start show weights 2016-08-11 13:41:19 +02:00
Hieu Hoang
1416d8d290 Only build moses2 if xmlrpc-c is present 2016-08-11 07:25:24 -04:00
Hieu Hoang
8f7a184075 eclipse 2016-08-10 23:00:24 +02:00
Hieu Hoang
09c4924154 move moses2 to contrib 2016-08-10 22:40:05 +02:00
Hieu Hoang
a4df68a9fa consistent method names 2016-08-06 08:18:32 +01:00
Hieu Hoang
928061d7f1 comment out debugging message 2016-08-06 07:50:47 +01:00
Hieu Hoang
03defc35f6 support distinct nbest 2016-08-05 17:04:01 +01:00
Hieu Hoang
3fe75d3fed support distinct nbest 2016-08-05 16:38:45 +01:00
Hieu Hoang
bd4cb68d0e fixed segfault. Use calculated hypo, rather than hypo key. Usually they are same as hypo, except when scores are equal 2016-08-05 15:41:58 +01:00
Hieu Hoang
25fbb8395d leak 2016-08-05 13:29:25 +01:00
Hieu Hoang
0352c30ed4 leak 2016-08-05 13:25:46 +01:00
Hieu Hoang
512dd465cb get rid of iterator over hypo collection. Create too many errors 2016-08-05 11:35:42 +01:00
Hieu Hoang
05aa676cd6 start to get rid of iterator over hypo collection. Create too many errors 2016-08-05 11:15:03 +01:00
Hieu Hoang
051ce0b44e segfault creating nbest. Was using all hypos in coll instead of sorted and pruned hypos only 2016-08-05 11:02:52 +01:00
Hieu Hoang
16044c176b use same method to add hypos to coll 2016-08-04 22:41:29 +01:00
Hieu Hoang
18442fdacf add debugging messages 2016-08-04 22:13:38 +01:00
Hieu Hoang
99227e63b2 comment out debugging messages 2016-08-04 20:09:10 +01:00
Hieu Hoang
8477905494 comment out debugging messages 2016-08-04 19:34:53 +01:00
Hieu Hoang
a18b8607aa mem leak 2016-08-04 18:57:07 +01:00
Hieu Hoang
259cb96538 stem leak by using mempool 2016-08-04 18:49:19 +01:00
Hieu Hoang
dfabbda5aa use tmpPool to create source phrases 2016-08-04 16:10:38 +01:00
Hieu Hoang
69270a999e trime <s> and </s> from nbest 2016-08-04 15:42:13 +01:00
Hieu Hoang
c47c1a0fb9 correct nbest. yay 2016-08-04 15:32:35 +01:00
Hieu Hoang
5a3e707879 move TrellisPaths 2016-08-04 12:47:45 +01:00
Hieu Hoang
b00369007f calc scores 2016-08-03 23:39:20 +01:00
Hieu Hoang
32b9efd775 Revert "use Cast() throughout" 
This reverts commit 8b1e9f45ad.
 2016-08-03 21:34:47 +01:00
Hieu Hoang
2abc631251 create deviant paths 2016-08-03 19:29:31 +01:00
Hieu Hoang
995c8f265a use reference instead of pointers 2016-08-03 19:08:55 +01:00
Hieu Hoang
3c644c8be4 create deviant paths 2016-08-03 18:56:03 +01:00
Hieu Hoang
c4bb814cd5 templatize TrellisPaths 2016-08-03 16:46:40 +01:00
Hieu Hoang
632dee2081 templatize TrellisPaths 2016-08-03 16:23:11 +01:00
Hieu Hoang
8ee2d7dda7 move TrellisPaths 2016-08-03 15:53:28 +01:00
Hieu Hoang
8b1e9f45ad use Cast() throughout 2016-08-03 12:39:58 +01:00
Hieu Hoang
6b88cda171 output nbest in correct format 2016-08-03 12:12:39 +01:00
Hieu Hoang
8b08444ab0 return reference, not pointer 2016-08-03 11:10:35 +01:00
Hieu Hoang
5966acac02 start nbest for SCFG 2016-08-03 11:03:11 +01:00
Hieu Hoang
432355803f get rid of translationId from InputType 2016-08-03 09:45:53 +01:00
Hieu Hoang
17d53818c2 start nbest for SCFG 2016-08-03 09:37:49 +01:00
Hieu Hoang
e17c7e87cd start nbest for SCFG 2016-08-02 18:02:20 +01:00
Hieu Hoang
721eaa5489 start nbest for SCFG 2016-08-02 16:35:55 +01:00
Hieu Hoang
ac9a4e9089 Merge branch 'perf_moses2' of github.com:hieuhoang/mosesdecoder into perf_moses2 2016-08-01 14:54:11 -04:00
Hieu Hoang
417a6f8b61 comments 2016-08-01 14:53:28 -04:00
Hieu Hoang
e361ca94aa Eclipse 2016-08-01 19:02:15 +01:00
Hieu Hoang
824ef2af91 Merge branch 'perf_moses2' of github.com:hieuhoang/mosesdecoder into perf_moses2 2016-08-01 13:49:58 -04:00
Hieu Hoang
9ec372beac caching for SCFG 2016-08-01 18:41:10 +01:00
Hieu Hoang
8e7643c982 restart caching for SCFG 2016-08-01 18:29:42 +01:00
Hieu Hoang
e2a56dae39 restart caching for SCFG 2016-08-01 18:18:15 +01:00
Hieu Hoang
257a8510b4 cleanup 2016-08-01 17:42:14 +01:00
Hieu Hoang
92ee41290c bug in probing cache. Mixing up previous and current source 2016-08-01 17:33:47 +01:00
Hieu Hoang
0dfade5880 increment API_VERSION. Don't skip last source word in cache 2016-08-01 14:29:57 +01:00
Hieu Hoang
514c4a4d6b add source key and value to cache 2016-08-01 14:19:29 +01:00
Hieu Hoang
b8dff73b68 Revert "caching for SCFG" 
This reverts commit 973fc6a700.
 2016-08-01 11:07:03 +01:00
Hieu Hoang
973fc6a700 caching for SCFG 2016-07-31 23:36:42 +01:00
Hieu Hoang
5cf5a02e05 Merge branch 'perf_moses2' of github.com:hieuhoang/mosesdecoder into perf_moses2 2016-07-31 07:12:22 -04:00
Hieu Hoang
f8a11cd59b Merge branch 'perf_moses2' of github.com:hieuhoang/mosesdecoder into perf_moses2 2016-07-31 02:21:38 +01:00
Hieu Hoang
19aeb803c0 gcc compile error 2016-07-31 02:21:27 +01:00
Hieu Hoang
5d0174b08e Merge branch 'perf_moses2' of github.com:hieuhoang/mosesdecoder into perf_moses2 2016-07-31 02:19:45 +01:00
Hieu Hoang
d9a2a37e9c increment API version 2016-07-31 02:19:38 +01:00
Hieu Hoang
e7ebd6fe3f Merge branch 'perf_moses2' of github.com:hieuhoang/mosesdecoder into perf_moses2 2016-07-31 02:18:15 +01:00
Hieu Hoang
1aa158ee52 get ready to store alignments for terminals and non-terms separately 2016-07-31 02:16:44 +01:00
Hieu Hoang
05075ab700 Merge branch 'perf_moses2' of github.com:hieuhoang/mosesdecoder into perf_moses2 2016-07-30 23:28:22 +01:00
Hieu Hoang
7035a3c9cc get ready to store alignments for terminals and non-terms separately 2016-07-30 23:25:48 +01:00
Hieu Hoang
0eee296725 debug 2016-07-30 22:45:29 +01:00
Hieu Hoang
3428322045 start caching for SCFG 2016-07-30 20:31:42 +01:00
Hieu Hoang
13ecc786b8 Merge branch 'perf_moses2' of github.com:hieuhoang/mosesdecoder into perf_moses2 2016-07-30 10:13:28 -04:00
Hieu Hoang
cbc3161a4f add back mgr as class variable. Still make sure it gets deleted in Run() 2016-07-30 15:13:18 +01:00
Hieu Hoang
972667a5c8 Merge branch 'perf_moses2' of github.com:hieuhoang/mosesdecoder into perf_moses2 2016-07-30 08:56:24 -04:00
Hieu Hoang
a14cc6041d fix segfault when server processes multiple streams 2016-07-30 13:52:04 +01:00
Hieu Hoang
356b74d5e7 don't segfault if Decode() is not called 2016-07-30 12:24:47 +01:00
Hieu Hoang
39cbe71854 delete unused fn 2016-07-29 21:51:35 +01:00
Hieu Hoang
45aa4a223f compiles 2016-07-29 20:53:01 +01:00
Hieu Hoang
12e573bc00 separate array into separate file 2016-07-29 20:38:55 +01:00
Hieu Hoang
6821f113ce comments 2016-07-29 04:42:24 -04:00
Hieu Hoang
8fe1b9b527 add client-stdin.perl 2016-07-28 15:15:16 -04:00
Hieu Hoang
4afedf4a21 TargetPhrase<Moses2::Word> -> TargetPhraseImpl. Move estimated score into TargetPhrase for PB only 2016-07-15 17:25:27 +01:00
Hieu Hoang
d6000cdd3d consistent comparitor for PB and SCFG 2016-07-15 16:35:38 +01:00
Hieu Hoang
1c818b270b set sortScore even when log prob 2016-07-15 16:25:48 +01:00
Hieu Hoang
7a0df40efa *merge 2016-07-08 01:15:24 +01:00
Hieu Hoang
6d60cf33b1 submit multiple sentences to server 2016-07-08 00:50:44 +01:00
Hieu Hoang
62bee20b9d *estimatedScore -> &estimatedScore 2016-07-07 21:15:04 +01:00
Hieu Hoang
cab1adcf10 correct number of nbest output 2016-07-07 20:57:06 +01:00
Hieu Hoang
b54c4a4497 no newline when outputting to server 2016-07-07 20:51:07 +01:00
Hieu Hoang
d2704a7c81 add all tp to tps in 1 go 2016-06-30 10:43:37 +01:00
Hieu Hoang
2e25be7856 don't output <s> </s> 2016-06-29 11:50:29 +01:00
Hieu Hoang
3810c7b681 don't output <s> </s> 2016-06-29 11:45:33 +01:00
Hieu Hoang
7a094d8231 cleanup server code 2016-06-29 00:52:28 +01:00
Hieu Hoang
1c2d94a0e3 less debug 2016-06-28 15:22:51 +01:00
Hieu Hoang
a8b908174f less debug 2016-06-28 15:20:50 +01:00
Hieu Hoang
eff96da4d5 be clear about which mem pool we're using 2016-06-28 15:18:38 +01:00
Hieu Hoang
e5df298508 Merge branch 'perf_moses2' of github.com:hieuhoang/mosesdecoder into perf_moses2 2016-06-28 15:02:28 +01:00
Hieu Hoang
7bf3f175c8 use manager pool to create queue items 2016-06-28 15:02:13 +01:00
Hieu Hoang
9a089b5baa import OnDisk pt 2016-06-28 11:59:10 +01:00
Hieu Hoang
fd7c2b94aa use uint32_t for alignment id 2016-06-28 08:56:42 +01:00
Hieu Hoang
882ba33dee delete folder 2016-06-27 22:34:01 +01:00
Hieu Hoang
2377b4eb04 debug 2016-06-27 21:50:03 +01:00
Hieu Hoang
d4a6395260 recycle QueueItem 2016-06-24 20:23:18 +01:00
Hieu Hoang
ad867b6304 Revert "memory tweaks" 
This reverts commit 262c4ef799.
 2016-06-24 20:13:26 +01:00
Hieu Hoang
1ac14484ee consistent betwenn pb and hiero 2016-06-24 20:07:43 +01:00
Hieu Hoang
262c4ef799 memory tweaks 2016-06-24 20:03:51 +01:00
Hieu Hoang
ac51f24811 consistent betwenn pb and hiero 2016-06-24 19:50:26 +01:00
Hieu Hoang
f7c23b7d90 debug 2016-06-24 19:15:47 +01:00
Hieu Hoang
f1f3eb673a set sortScore = 3rd score unless there's not enough scores 2016-06-24 18:39:39 +01:00
Hieu Hoang
0ffa6205fe create trie to write source prefixes even if there's no rules 2016-06-24 16:23:55 +01:00
Hieu Hoang
a94a5787f2 reformat and sort before binarizing 2016-06-24 13:21:04 +01:00
Hieu Hoang
9e94edd6db bug fix. Matches moses 2016-06-24 01:52:40 +01:00
Hieu Hoang
192a0dcf32 use prefixes 2016-06-23 22:59:33 +01:00
Hieu Hoang
873dafc22b debug 2016-06-23 22:27:30 +01:00
Hieu Hoang
5fc4d4038f use prefixes 2016-06-23 18:33:19 +01:00
Hieu Hoang
4f2c3a1efa use prefixes 2016-06-23 18:13:27 +01:00
Hieu Hoang
5645242d7f store prefixes 2016-06-23 17:52:34 +01:00
Hieu Hoang
2c75120763 get ready to store prefixes 2016-06-23 17:45:00 +01:00
Hieu Hoang
8da63839ea share LookupGivenWord() across all pt 2016-06-23 15:12:08 +01:00
Hieu Hoang
2b39bbf7c4 share LookupNT() across all pt 2016-06-23 14:51:40 +01:00
Hieu Hoang
88524b222f share LookupUnary() across all pt 2016-06-23 14:41:54 +01:00
Hieu Hoang
36d83dd1a9 store sorted & pruned hypos instead of hypoColl. No chance of confusing the 2 and reading pruned hypos 2016-06-23 12:54:43 +01:00
Hieu Hoang
974a4915dd pass manager object down 2016-06-23 12:28:14 +01:00
Hieu Hoang
f88eb49577 debug 2016-06-22 22:23:51 +01:00
Hieu Hoang
b70f379061 comment out debug msg 2016-06-22 16:46:43 +01:00
Hieu Hoang
d3dd66cff8 use alignment. Now decode 2016-06-22 16:23:55 +01:00
Hieu Hoang
b4379490d9 use alignment 2016-06-22 15:09:52 +01:00
Hieu Hoang
08c632f8b6 save alignment 2016-06-22 12:36:33 +01:00
Hieu Hoang
ff44e141eb differentiate between non-term and terms. Still segfaults, probably due to lack of word alignments 2016-06-22 12:09:52 +01:00
Hieu Hoang
b99192677e bugs. Now successfully decode 2016-06-22 11:18:49 +01:00
Hieu Hoang
5e6ba8d2f7 get ready to move AddTargetPhrasesToPath() 2016-06-22 10:59:51 +01:00
Hieu Hoang
050bfe2993 get ready to move AddTargetPhrasesToPath() 2016-06-22 10:48:27 +01:00
Hieu Hoang
7caf02c1ea merge 2016-06-22 00:24:00 +01:00
Hieu Hoang
7d8ef1a9b4 don't set affinity by default 2016-06-22 00:14:35 +01:00
Hieu Hoang
0008440e25 use pool. Leak fixed 2016-06-21 23:59:01 +01:00
Hieu Hoang
350ba32eb7 use pool 2016-06-21 23:53:50 +01:00
Hieu Hoang
996df0c984 get ready to use pool 2016-06-21 23:36:58 +01:00
Hieu Hoang
8614486cdf merge 2016-06-21 21:28:44 +01:00
Hieu Hoang
fa2e28dd4d comment out debugging info 2016-06-21 21:26:43 +01:00
Hieu Hoang
3122574b40 don't use Debug() to output to nbest. Take out total score from nbest score breakdown 2016-06-21 21:19:25 +01:00
Hieu Hoang
5118578347 ToString() -> OutputTargetPhrase() 2016-06-21 21:07:38 +01:00
Hieu Hoang
c4daa158c2 move ReorderingConstraint to PhraseBased/ 2016-06-21 18:17:15 +01:00
Hieu Hoang
320f01c722 merge 2016-06-21 17:41:53 +01:00
Hieu Hoang
78dee8201e merge 2016-06-21 17:39:00 +01:00
Hieu Hoang
cf59f24654 xml constraint not supported 2016-06-21 16:57:16 +01:00
Hieu Hoang
6570fa969e set m_xmlCoverageMap 2016-06-21 16:52:10 +01:00