moses-smt/mosesdecoder
1
1
Fork 0
mirror of https://github.com/moses-smt/mosesdecoder.git synced 2025-01-02 17:09:36 +03:00
Code Issues Packages Projects Releases Wiki Activity
15,923 Commits 136 Branches 26 Tags 147 MiB
307742a32e
Commit Graph

755 Commits

Author SHA1 Message Date
Hieu Hoang
a3fa3ff82a Merge branch 'perf_moses2' of github.com:hieuhoang/mosesdecoder into perf_moses2 2016-01-08 14:44:00 +00:00
Hieu Hoang
3840c307fd DebugCounts() 2016-01-08 09:43:47 -05:00
Hieu Hoang
1eb3ff9d93 DebugCounts() for cube pruning 2016-01-08 14:43:14 +00:00
Hieu Hoang
51d9b4d102 DebugCounts() 2016-01-08 14:14:48 +00:00
Hieu Hoang
2cab9734be create cube pruning datastructures 1st, including ministacks 2016-01-08 13:42:16 +00:00
Hieu Hoang
361d831939 add Misc to NSCubePruningPerMiniStack 2016-01-08 12:34:35 +00:00
Hieu Hoang
8a1ee99e7a move Hypotheses to Stack.h 2016-01-08 11:50:33 +00:00
Hieu Hoang
9f6f510c45 re-added Stacks 2016-01-08 00:15:44 +00:00
Hieu Hoang
8aa298efa2 re-added Stacks 2016-01-07 19:04:22 +00:00
Hieu Hoang
d8dfda7de9 added NSCubePruningPerMiniStack 2016-01-07 18:43:55 +00:00
Hieu Hoang
5d1bc4b406 log prob 2016-01-07 12:06:49 +00:00
Hieu Hoang
bef1910c5e start log prob in binary pt 2016-01-07 10:46:54 +00:00
Hieu Hoang
08bbf273ea recycle mini stack 2016-01-06 21:27:41 +00:00
Hieu Hoang
de8b99ccc0 slight difference with Moses1 for 1st hypo 2016-01-06 17:42:23 +00:00
Hieu Hoang
4443a34052 support -tt 2016-01-06 16:35:24 +00:00
Hieu Hoang
4791e75a6b support -report-segmentation 2016-01-06 16:09:08 +00:00
Hieu Hoang
f66885166e delete Stacks class 2016-01-06 15:04:09 +00:00
Hieu Hoang
d8f65ecbc9 don't have vector of stacks. Just 1 stack 2016-01-06 14:54:40 +00:00
Hieu Hoang
5bbc0bf97d only require 1 stack. Reuse each time 2016-01-06 14:41:14 +00:00
Hieu Hoang
fbba014c8a static methods QueueItem::Create(). Recycle existing item, use 1 from recycle bin, or create new item 2016-01-06 12:13:22 +00:00
Hieu Hoang
0ee1f452db comments 2016-01-06 11:41:42 +00:00
Hieu Hoang
197fd1b903 use system pool for hypo 2016-01-05 16:58:27 +00:00
Hieu Hoang
749c531996 use system pool for hypo 2016-01-05 14:34:59 +00:00
Hieu Hoang
aebc27316d rename methods in Recyler 2016-01-05 11:53:17 +00:00
Hieu Hoang
7a92e78e3e get ready to change pool for hypo 2016-01-05 11:22:53 +00:00
Hieu Hoang
f179072902 cpu affinity offset 2016-01-04 00:19:35 +00:00
Hieu Hoang
abf41d04e9 cpu affinity offset 2016-01-03 23:22:50 +00:00
Hieu Hoang
4208115174 add Recycler class 2016-01-02 23:03:08 +00:00
Hieu Hoang
6994fc9a4b reference, not copy 2016-01-02 22:43:17 +00:00
Hieu Hoang
0c22176c66 thread-safe system pool 2016-01-02 21:44:44 +00:00
Hieu Hoang
3249d4a634 seen position set and get at the same time 2016-01-02 01:20:51 +00:00
Hieu Hoang
9bc7dd6157 cleanup 2016-01-02 00:46:48 +00:00
Hieu Hoang
42e8aadbd4 QueueItem recycler 2016-01-02 00:22:03 +00:00
Hieu Hoang
9b2cdac051 reuse QueueItem 2016-01-02 00:14:08 +00:00
Hieu Hoang
aae48b2859 add unused hypos from queue into recycler 2016-01-01 23:59:06 +00:00
Hieu Hoang
e869392051 reuse QueueItem 2016-01-01 23:45:27 +00:00
Hieu Hoang
d26f7d3a27 use pointers 2016-01-01 23:35:21 +00:00
Hieu Hoang
f73b90516a delete Recycler 2016-01-01 21:24:12 +00:00
Hieu Hoang
bce8c63f7b get ready to delet Recycler 2016-01-01 21:08:58 +00:00
Hieu Hoang
48a3a4260b leak 2016-01-01 19:10:05 +00:00
Hieu Hoang
4b0508a20b Revert "recycle query_result" 
This reverts commit b7d78d7a8b.
 2016-01-01 13:34:25 -05:00
Hieu Hoang
dcf486308e Revert "recycle query_result" 
This reverts commit aaa88d5d05.
 2016-01-01 13:34:24 -05:00
Hieu Hoang
785cc722fd Revert "leak" 
This reverts commit 4feb3e355c.
 2016-01-01 13:34:22 -05:00
Hieu Hoang
4feb3e355c leak 2016-01-01 18:07:57 +00:00
Hieu Hoang
aaa88d5d05 recycle query_result 2016-01-01 18:02:12 +00:00
Hieu Hoang
b7d78d7a8b recycle query_result 2016-01-01 17:58:19 +00:00
Hieu Hoang
e1ed7ee219 recycle vector in vbyte_decode_line() 2016-01-01 17:25:23 +00:00
Hieu Hoang
4af443d8af get ready to recycle more objects 2016-01-01 17:14:40 +00:00
Hieu Hoang
566faa7039 class scope recycler 2016-01-01 16:38:18 +00:00
Hieu Hoang
e097b4cf52 reset target_text when taken from recycler 2016-01-01 16:30:45 +00:00
Hieu Hoang
155a03df15 recycle target_text 2016-01-01 16:20:37 +00:00
Hieu Hoang
8107bcf1ea store pointer to vocab id 2016-01-01 15:12:16 +00:00
Hieu Hoang
0dec823c60 Revert "reuse query_result" 
This reverts commit 32f988be87.
 2016-01-01 08:49:57 -05:00
Hieu Hoang
32f988be87 reuse query_result 2016-01-01 13:41:44 +00:00
Hieu Hoang
7d1aeaa601 warnings 2015-12-31 17:28:11 +00:00
Hieu Hoang
6435f15a14 delete unused variable 2015-12-31 17:10:30 +00:00
Hieu Hoang
e26081e5c5 Revert "get ready to cache MiniStack" 
This reverts commit acc0fdf807.
 2015-12-31 11:42:12 -05:00
Hieu Hoang
8873d71497 Revert "cache MiniStack" 
This reverts commit b887885a04.
 2015-12-31 11:42:09 -05:00
Hieu Hoang
b887885a04 cache MiniStack 2015-12-31 16:38:18 +00:00
Hieu Hoang
acc0fdf807 get ready to cache MiniStack 2015-12-31 16:22:10 +00:00
Hieu Hoang
d39c7626f5 comment out prefetch 2015-12-31 10:01:17 -05:00
Hieu Hoang
6417924de0 use hypothesis::Prefetch() 2015-12-31 14:53:16 +00:00
Hieu Hoang
f1580def18 set methods in range class 2015-12-31 14:31:49 +00:00
Hieu Hoang
7d035c235d comment out prefetch 2015-12-31 09:26:09 -05:00
Hieu Hoang
adc38bc834 prefetch losing hypo for reuse 2015-12-31 13:56:28 +00:00
Hieu Hoang
d79d227418 return asap 2015-12-31 13:48:34 +00:00
Hieu Hoang
0915dbd472 force new hypo into place of losing hypo 2015-12-31 13:41:48 +00:00
Hieu Hoang
8f33d13ab2 move Add() into MiniStack 2015-12-31 13:30:34 +00:00
Hieu Hoang
6da80a599d output warning if no translation 2015-12-31 12:59:33 +00:00
Hieu Hoang
fbc2c60b36 no output to cerr 2015-12-31 12:56:47 +00:00
Hieu Hoang
bb60b02619 more mempool 2015-12-31 02:09:57 +00:00
Hieu Hoang
646079b8ec more mempool 2015-12-31 01:49:53 +00:00
Hieu Hoang
d1cceb1d38 more mempool 2015-12-31 01:28:21 +00:00
Hieu Hoang
844e8a16c4 anonymous objects 2015-12-31 00:58:48 +00:00
Hieu Hoang
d0951cde0a allocator for stack 2015-12-31 00:48:16 +00:00
Hieu Hoang
f4a07ef1fa allocator for stack 2015-12-31 00:29:13 +00:00
Hieu Hoang
00952efdec anonymous objects 2015-12-30 23:35:03 +00:00
Hieu Hoang
ff09020526 don't use pointers 2015-12-30 23:27:41 +00:00
Hieu Hoang
95ea1d8485 use my allocator 2015-12-30 23:15:28 +00:00
Hieu Hoang
7d4432a2d6 clear container using old method 2015-12-30 23:10:13 +00:00
Hieu Hoang
181914b822 use boost fast pool allocator 2015-12-30 22:58:12 +00:00
Hieu Hoang
08f61d69e9 use boost pool allocator 2015-12-30 22:51:50 +00:00
Hieu Hoang
95e59ac36f use pointers for mempool 2015-12-30 21:54:03 +00:00
Hieu Hoang
5f4b34b95f use pointers for mempool 2015-12-30 21:49:46 +00:00
Hieu Hoang
bc2f065f45 Mempool with Queue 2015-12-30 21:42:21 +00:00
Hieu Hoang
34946bc931 undo stack diversity 2015-12-30 19:20:00 +00:00
Hieu Hoang
599bc4855a create hypo from every edge. Increase diversity 2015-12-30 19:00:27 +00:00
Hieu Hoang
c5826035c2 Mempool with Queue 2015-12-30 16:46:42 +00:00
Hieu Hoang
448d0a3e9c Mempool with seen position 2015-12-30 16:33:48 +00:00
Hieu Hoang
654f2616e6 get ready to use Mempool with seen position 2015-12-30 16:16:36 +00:00
Hieu Hoang
2cdbad5f2d Revert "delete score breakdown pointer" 
This reverts commit 2f7e1b599b.
 2015-12-30 16:01:49 +00:00
Hieu Hoang
2f7e1b599b delete score breakdown pointer 2015-12-30 15:31:16 +00:00
Hieu Hoang
9cf5d24ef4 get ready to delete score breakdown pointer 2015-12-30 15:24:01 +00:00
Hieu Hoang
fbbd118b3e use only 1 score for estimated score, don't need Score object 2015-12-30 14:57:02 +00:00
Hieu Hoang
4e03fa5fba use only 1 score for estimated score, don't need Score object 2015-12-30 14:34:24 +00:00
Hieu Hoang
e7a5b2bafc use dequeue for recycler 2015-12-30 14:02:02 +00:00
Hieu Hoang
b2b5e5cf85 Revert "recycler unused bitmaps" 
This reverts commit 48b26851e0.
 2015-12-30 13:48:21 +00:00
Hieu Hoang
48b26851e0 recycler unused bitmaps 2015-12-30 13:40:52 +00:00
Hieu Hoang
e7247534ce Revert "use mempool in bitmap" 
This reverts commit 1bf356ee8d.
 2015-12-30 08:16:24 -05:00
Hieu Hoang
4c1e8eb1fd Revert "vector -> array" 
This reverts commit fa54628a65.
 2015-12-30 08:15:25 -05:00
Hieu Hoang
f844cde08b Revert "use mempool for bitmaps" 
This reverts commit 7eb5dc397f.
 2015-12-30 08:15:22 -05:00
Hieu Hoang
b7f9c5e858 Revert "recycler unused bitmaps" 
This reverts commit b47f51dce3.
 2015-12-30 08:15:17 -05:00
Hieu Hoang
b47f51dce3 recycler unused bitmaps 2015-12-30 13:06:49 +00:00
Hieu Hoang
7eb5dc397f use mempool for bitmaps 2015-12-30 12:25:10 +00:00
Hieu Hoang
fa54628a65 vector -> array 2015-12-30 11:42:20 +00:00
Hieu Hoang
1bf356ee8d use mempool in bitmap 2015-12-30 11:02:03 +00:00
Hieu Hoang
bb65df0062 don't use global object 2015-12-30 00:09:23 +00:00
Hieu Hoang
98c4563ffb don't use global object 2015-12-29 23:58:56 +00:00
Hieu Hoang
650f5d462d list -> vector 2015-12-29 23:50:06 +00:00
Hieu Hoang
bc9a9a2231 Revert "use allocator for queue" 
This reverts commit 3b53627525.
 2015-12-29 23:46:50 +00:00
Hieu Hoang
e61ad0474d Revert "use allocator for seen positions" 
This reverts commit cabaf142f2.
 2015-12-29 23:46:38 +00:00
Hieu Hoang
cabaf142f2 use allocator for seen positions 2015-12-29 23:10:31 +00:00
Hieu Hoang
3b53627525 use allocator for queue 2015-12-29 22:26:38 +00:00
Hieu Hoang
22f0329abc don't use allocator 2015-12-29 22:03:23 +00:00
Hieu Hoang
ff0c559425 use allocator 2015-12-29 20:37:42 +00:00
Hieu Hoang
b6f099031c fix leak 2015-12-29 19:03:46 +00:00
Hieu Hoang
88d0b1b88d revert to yesterday 2015-12-29 18:44:42 +00:00
Hieu Hoang
2b134138fb don't delete 2015-12-29 17:32:19 +00:00
Hieu Hoang
f1dff235c0 merge 2015-12-29 12:25:06 -05:00
Hieu Hoang
95d45dd7f3 use pointers for stack object 2015-12-29 17:23:21 +00:00
Hieu Hoang
f446b0e1ac Merge branch 'perf_moses2' of github.com:hieuhoang/mosesdecoder into perf_moses2 2015-12-29 12:04:57 -05:00
Hieu Hoang
7040936083 use allocator for queue objects 2015-12-29 17:04:50 +00:00
Hieu Hoang
d798ae7909 Merge branch 'perf_moses2' of github.com:hieuhoang/mosesdecoder into perf_moses2 2015-12-29 11:49:46 -05:00
Hieu Hoang
21cbbbff49 minor compile error 2015-12-29 16:49:38 +00:00
Hieu Hoang
8af0300334 Merge branch 'perf_moses2' of github.com:hieuhoang/mosesdecoder into perf_moses2 2015-12-29 11:30:58 -05:00
Hieu Hoang
1f7d9ae94f use allocator for seen position object 2015-12-29 16:30:49 +00:00
Hieu Hoang
51335251c0 Merge branch 'perf_moses2' of github.com:hieuhoang/mosesdecoder into perf_moses2 2015-12-29 11:08:06 -05:00
Hieu Hoang
0128081bb4 use allocator in stacks 2015-12-29 16:07:56 +00:00
Hieu Hoang
99bd5c1267 Revert "don't have Stack*" 
This reverts commit 75d63e4450.
 2015-12-29 09:27:50 -05:00
Hieu Hoang
75d63e4450 don't have Stack* 2015-12-29 14:23:10 +00:00
Hieu Hoang
0ae22ce6fe don't have m_alloc 2015-12-29 14:15:16 +00:00
Hieu Hoang
eb5bb41900 mempool 2015-12-29 09:11:01 -05:00
Hieu Hoang
dea29e1b5b don't use pointers 2015-12-29 13:56:03 +00:00
Hieu Hoang
b0a7d86fbd redo using allocator 2015-12-29 13:40:50 +00:00
Hieu Hoang
02b7e72766 delete debug message 2015-12-29 13:28:53 +00:00
Hieu Hoang
396290810c don't recycle stack 2015-12-29 13:27:29 +00:00
Hieu Hoang
14a8beebcb undo using allocator 2015-12-29 08:22:15 -05:00
Hieu Hoang
2ad156b939 mempool allocator for stack 2015-12-29 12:54:20 +00:00
Hieu Hoang
4b6a940421 mempool allocator for stack 2015-12-29 12:33:08 +00:00
Hieu Hoang
c093bf206e increment probing pt version 2015-12-28 21:15:43 +00:00
Hieu Hoang
c5549fc9d3 seems to be working 2015-12-28 20:41:33 +00:00
Hieu Hoang
6a8b43cf01 add Lex RO as probabilities, not scores 2015-12-28 20:32:26 +00:00
Hieu Hoang
1de4921d2a create score properties 2015-12-28 20:07:47 +00:00
Hieu Hoang
73516a34c2 append lex probs to pt probs 2015-12-28 15:50:42 +00:00
Hieu Hoang
06cd3b977b append lex probs to pt probs 2015-12-28 13:00:26 +00:00
Hieu Hoang
2461ccc175 Scan<SCORE> 2015-12-27 01:06:27 +00:00
Hieu Hoang
07b521cc07 MemPoolAllocator 2015-12-26 00:59:43 +00:00
Hieu Hoang
436fdbe438 integrated Lex RO 2015-12-24 20:20:51 +00:00
Hieu Hoang
902f1c5c97 MemPool return uint8_t* 2015-12-24 20:11:28 +00:00
Hieu Hoang
45983bd249 trying to find segfault 2015-12-24 17:15:31 +00:00
Hieu Hoang
b221423f1c use pool given to method 2015-12-24 16:32:26 +00:00
Hieu Hoang
c466bb43c1 EOS terminator 2015-12-23 14:08:45 +00:00
Hieu Hoang
50a75aef1e apply Nikolay's changes to moses2 fork 2015-12-23 13:31:23 +00:00
Hieu Hoang
73165f0b31 use properties to store lex ro data in pt 2015-12-21 18:12:12 +00:00
Hieu Hoang
eefcd29bef use properties to store lex ro data in pt 2015-12-21 18:05:37 +00:00
Hieu Hoang
4423d031a4 integrate compact lex RO 2015-12-20 17:07:06 +00:00
Hieu Hoang
4a36aa59c8 integrate compact lex RO 2015-12-20 17:03:16 +00:00
Hieu Hoang
6f25f43cf9 integrate compact lex RO 2015-12-20 16:21:34 +00:00
Hieu Hoang
b43292eb4f set HasPhraseTableInd() to true 2015-12-20 00:49:56 +00:00
Hieu Hoang
dc44436481 integrate compact lex RO 2015-12-20 00:10:28 +00:00
Hieu Hoang
3d87a4fa5f integrate compact lex RO 2015-12-19 23:12:46 +00:00
Hieu Hoang
bc5bf2d9ce integrate compact lexicalised reordering model 2015-12-19 02:52:41 +00:00
Hieu Hoang
da022f8e64 integrate compact lexicalised reordering model 2015-12-19 02:32:08 +00:00
Hieu Hoang
d06a8019c5 get ready for lex RO 2015-12-18 18:39:02 +00:00
Hieu Hoang
6e52eeddeb integrate binary lexicalised reordering model 2015-12-18 15:26:25 +00:00
Hieu Hoang
a2454cdd55 floor score 2015-12-18 12:02:02 +00:00
Hieu Hoang
3e7d2c9497 delete all memtion of shared ptr 2015-12-17 22:07:19 +00:00
Hieu Hoang
a1a380f806 use cached data 2015-12-17 21:49:46 +00:00
Hieu Hoang
75c06c3613 use cached data 2015-12-17 21:43:02 +00:00
Hieu Hoang
8d9a6e2375 use cached data 2015-12-17 21:31:00 +00:00
Hieu Hoang
56b3d0981f add MemPool arg to EvaluateInIsolation() 2015-12-17 21:03:28 +00:00
Hieu Hoang
de2397a205 minor cleanup 2015-12-17 20:51:34 +00:00
Hieu Hoang
117ea5c295 codify what mempool to be passed into EvaluateInIsolation() 2015-12-17 18:01:00 +00:00
Hieu Hoang
d3b367284e use cache FF for lexicalized reordering 2015-12-17 17:16:55 +00:00
Hieu Hoang
8028e8f600 create framework to cache FF data in rules 2015-12-17 17:08:49 +00:00
Hieu Hoang
995b668d53 replicate moses1 scoring 2015-12-17 16:24:22 +00:00
Hieu Hoang
7d612f2faa comments 2015-12-17 15:46:41 +00:00
Hieu Hoang
4b53705cf8 correct offset 2015-12-17 14:29:47 +00:00
Hieu Hoang
fc4e585f4b set state 2015-12-17 13:45:36 +00:00
Hieu Hoang
aae432627e rename HypothesisSet -> MiniStack 2015-12-16 21:54:27 +00:00
Hieu Hoang
71ee1746cc correct LR score 2015-12-16 17:19:34 +00:00
Hieu Hoang
ac52c6062f output ff name for debugging 2015-12-16 16:53:00 +00:00
Hieu Hoang
06f6db303d changes in Lookup framework broke pt memory 2015-12-16 16:42:37 +00:00
Hieu Hoang
eeb81146cc lexicalised reordering 2015-12-16 14:33:05 +00:00
Hieu Hoang
dcd059a21e get hypo to store path 2015-12-16 13:51:35 +00:00
Hieu Hoang
5fdf9d8046 lexicalised reordering 2015-12-16 13:11:11 +00:00
Hieu Hoang
007756abb3 lexicalised reordering 2015-12-15 15:46:15 +00:00
Hieu Hoang
d7c41a255e line counts 2015-12-15 14:43:55 +00:00
Hieu Hoang
72c0c585df parameters 2015-12-15 14:20:48 +00:00
Hieu Hoang
b2f0139c23 Implemented Load() 2015-12-15 13:36:44 +00:00
Hieu Hoang
aa6182522e start lexicalised re-ordering 2015-12-15 11:53:16 +00:00
Hieu Hoang
d4882a491b add EOL 2015-12-14 13:31:37 +00:00
Hieu Hoang
a2c6ccd5ec increment translationId 2015-12-14 13:28:33 +00:00
Hieu Hoang
6cea5d01e7 increment translationId 2015-12-14 13:25:29 +00:00
Hieu Hoang
b7d261088b use OutputCollector 2015-12-14 13:16:40 +00:00
Hieu Hoang
79f59abecd use OutputCollector 2015-12-14 13:15:20 +00:00
Hieu Hoang
8baf6199d1 :add InputType framework. Get ready for OutputCollector 2015-12-14 12:03:37 +00:00
Hieu Hoang
69e46c388d make ComputeDistortionDistance inline 2015-12-13 01:29:33 +00:00
Hieu Hoang
b935767dc4 recycle stack 2015-12-11 15:11:16 +00:00
Hieu Hoang
fd55279b09 recycle bitmaps 2015-12-11 13:41:33 +00:00