moses-smt/mosesdecoder
1
1
Fork 0
mirror of https://github.com/moses-smt/mosesdecoder.git synced 2024-12-29 06:52:34 +03:00
Code Issues Packages Projects Releases Wiki Activity
17,111 Commits 136 Branches 26 Tags 147 MiB
c037962feb
Commit Graph

2130 Commits

Author SHA1 Message Date
Hieu Hoang
78dee8201e merge 2016-06-21 17:39:00 +01:00
Hieu Hoang
cf59f24654 xml constraint not supported 2016-06-21 16:57:16 +01:00
Hieu Hoang
6570fa969e set m_xmlCoverageMap 2016-06-21 16:52:10 +01:00
Hieu Hoang
23677d2b65 compile errors 2016-06-21 15:24:58 +01:00
Hieu Hoang
501f9ccd8d use xml options 2016-06-21 15:08:18 +01:00
Hieu Hoang
44960ca13d Merge ../mosesdecoder.perf.public into perf_moses2 2016-06-21 14:10:20 +01:00
Hieu Hoang
eacd2cb5ae use xml options 2016-06-21 14:10:13 +01:00
Hieu Hoang
062081c2a9 use xml options 2016-06-21 12:46:46 +01:00
Hieu Hoang
b1a0ca960d parse xml forced translation 2016-06-21 12:35:16 +01:00
Hieu Hoang
ac8fbbde40 add SatisfyBackoff(). Supports XML too 2016-06-21 11:20:38 +01:00
Hieu Hoang
f40c6bd030 separate out xml processing into separate method 2016-06-20 19:11:53 +01:00
Hieu Hoang
fc02b00360 expose unkWP FF. To be used to put XML forced translation 2016-06-20 19:07:50 +01:00
Hieu Hoang
fb2bbf99fc debug 2016-06-20 16:33:04 +01:00
Hieu Hoang
19b8bc50b7 test signing 
Signed-off-by: Hieu Hoang <hieuhoang@gmail.com>
 2016-06-18 01:15:40 +01:00
Hieu Hoang
2f9d8d211f debug 2016-06-17 23:06:02 +01:00
Hieu Hoang
ad2153f264 debug 2016-06-17 22:54:32 +01:00
Hieu Hoang
70ce10a85e debug 2016-06-17 21:55:47 +01:00
Hieu Hoang
12a56efe19 GetKey can be pass pt oov 2016-06-17 21:12:17 +01:00
Hieu Hoang
eb076ed392 lookup scfg rules in probing pt 2016-06-17 20:13:41 +01:00
Hieu Hoang
9de1f72a8e get ready to hash source 2016-06-17 16:35:17 +01:00
Hieu Hoang
45790f3c38 get ready to hash source 2016-06-17 16:09:21 +01:00
Hieu Hoang
42b3f337fd get ready to hash source 2016-06-17 15:11:59 +01:00
Hieu Hoang
b64e3afd4d minor rename 2016-06-17 14:31:30 +01:00
Hieu Hoang
96158051e7 don't use last source word when storing. It's the LHS 2016-06-17 14:21:19 +01:00
Hieu Hoang
774b4dd78a cleanup 2016-06-17 13:50:43 +01:00
Hieu Hoang
7d00c6a8fe eclipse 2016-06-17 12:56:57 +01:00
Hieu Hoang
aa07fd26cf eclipse 2016-06-17 12:48:28 +01:00
Hieu Hoang
e7ec90da58 start scfg decoding with probing pt 2016-06-16 18:17:48 +01:00
Hieu Hoang
62c23a9859 start scfg decoding with probing pt 2016-06-16 17:14:18 +01:00
Hieu Hoang
cad0235159 reformat NT 2016-06-16 15:24:34 +01:00
Hieu Hoang
ca264d8fcd retrieve non-terms 2016-06-16 13:53:20 +01:00
Hieu Hoang
647a9a3cff start reformat rules for storing 2016-06-16 12:41:50 +01:00
Hieu Hoang
127c057e62 use StoreVocab for source vocab, take 2. Continue to use simple hashing 2016-06-16 11:26:30 +01:00
Hieu Hoang
101b4bfe0c modularize getKey() 2016-06-16 10:41:47 +01:00
Hieu Hoang
f8f916aeba Revert "use StoreVocab for source vocab" 
This reverts commit 7f39875c74.
 2016-06-16 05:05:17 -04:00
Hieu Hoang
7f39875c74 use StoreVocab for source vocab 2016-06-15 19:41:03 +01:00
Hieu Hoang
193c921281 use template type 2016-06-15 18:25:34 +01:00
Hieu Hoang
30bd411e17 add GetVocabIds() 2016-06-15 18:21:04 +01:00
Hieu Hoang
73ac4e43cb use StoreVocab class in StoreTarget 2016-06-15 17:40:39 +01:00
Hieu Hoang
c068365d85 add StoreVocab class 2016-06-15 17:05:11 +01:00
Hieu Hoang
6ec42385ac start scfg for probing pt 2016-06-15 15:24:30 +01:00
Hieu Hoang
d57aa3c40b better error message 2016-06-15 14:53:48 +01:00
Hieu Hoang
161f80dc92 better error message 2016-06-15 14:48:43 +01:00
Hieu Hoang
661fab4ea5 std::unordered_map -> boost::unordered_map 2016-06-15 14:45:12 +01:00
Hieu Hoang
3ed2bf1a52 update probing pt version. Change config format 2016-06-15 12:56:36 +01:00
Hieu Hoang
fbb8b30681 Everything in SeenPosition is also in QueueItem so make 1 contain the other 2016-06-15 10:49:24 +01:00
Hieu Hoang
faa92e8c5e tpInd -> tpInd + 1 2016-06-15 01:12:26 +01:00
Hieu Hoang
141f7a246c SeenPositionItem -> SeenPosition 2016-06-15 01:05:35 +01:00
Hieu Hoang
045903ff79 support output-hypo-score arg. std::vector -> Vector 2016-06-15 01:01:29 +01:00
Hieu Hoang
081639e536 index SeenPosition by tps and tpInd, not by tp. Same tp could span different non-terms 2016-06-15 00:47:00 +01:00
Hieu Hoang
c8539ea61e debug 2016-06-15 00:34:30 +01:00
Hieu Hoang
cfebaabc73 debug 2016-06-14 22:45:19 +01:00
Hieu Hoang
9942fec457 use pointer for SeenPositionItem 2016-06-14 21:34:07 +01:00
Hieu Hoang
572d7aec03 debug 2016-06-14 17:22:19 +01:00
Hieu Hoang
be23e00e13 debug 2016-06-14 17:14:48 +01:00
Hieu Hoang
1674159c43 debug info 2016-06-13 16:58:08 +01:00
Hieu Hoang
fd1c0d891f debug info 2016-06-13 16:16:51 +01:00
Hieu Hoang
416e66e9fe don't create rules with unk pt if there are already rules 2016-06-13 15:33:26 +01:00
Hieu Hoang
0e8111e422 sort by p(e|f) 2016-06-13 14:34:50 +01:00
Hieu Hoang
8f4764f1bc Change CompareFutureScore() 2016-06-13 13:59:26 +01:00
Hieu Hoang
14bfbdc5e3 change pointer to object 2016-06-13 13:47:15 +01:00
Hieu Hoang
91b01b55a0 cleanup Score class 2016-06-11 02:12:48 +01:00
Hieu Hoang
e23041f7a8 separate Debug() and OutputToStream() 2016-06-11 02:07:43 +01:00
Hieu Hoang
d90a31d8dd consistent Debug() 2016-06-11 01:51:26 +01:00
Hieu Hoang
4453aecb2b consistent Debug() 2016-06-11 01:45:26 +01:00
Hieu Hoang
04fd6630d5 consistent Debug() 2016-06-11 01:31:40 +01:00
Hieu Hoang
f3874d3945 consistent Debug() 2016-06-11 01:03:23 +01:00
Hieu Hoang
7f8c66df7a fix segfault 2016-06-11 00:55:39 +01:00
Hieu Hoang
bc77f63948 consistent debug function 2016-06-11 00:15:15 +01:00
Hieu Hoang
5a69037115 oov penalty 2016-06-09 23:12:58 +01:00
Hieu Hoang
f8b6e55bfc add scores to new hypo 2016-06-09 22:55:59 +01:00
Hieu Hoang
ea1517bd77 print memory usage 2016-06-09 17:17:20 +01:00
Hieu Hoang
482457bf2b align to 4-byte boundary. Seems to work. 2016-06-08 09:06:39 -04:00
Hieu Hoang
f8c0c25400 revert previous & fix bug 2016-06-08 13:26:40 +01:00
Hieu Hoang
0da2cbc214 AllocateWithAlign() aligns each object in array to 8 byte boundary 2016-06-08 13:21:15 +01:00
Hieu Hoang
0958d3f4f4 size_t -> bool. Use AllocateWithAlign() to allocate mem for bool 2016-06-08 13:10:14 +01:00
Hieu Hoang
e85f048ec9 add AllocateWithAlign() to MemPool 2016-06-08 13:07:09 +01:00
Hieu Hoang
dc052be4ca bool -> size_t. Prevent segfault when using mempool 2016-06-08 12:55:53 +01:00
Hieu Hoang
00618430fa std::vector -> Vector 2016-06-07 18:07:49 +01:00
Hieu Hoang
9cf2862a52 vector -> pair 2016-06-07 18:04:12 +01:00
Hieu Hoang
15b8c97fce use mempool to stem leak 2016-06-07 17:57:33 +01:00
Hieu Hoang
849b161bf8 make XMLOption an inner class of Sentence 2016-06-07 17:39:01 +01:00
Hieu Hoang
c39dbcb17b cleanup 2016-06-07 17:33:20 +01:00
Hieu Hoang
81642a82c1 debugging zones. Work the same as moses 2016-06-07 17:23:31 +01:00
Hieu Hoang
462b17de8a use zone and walls for normal and batch algorithm 2016-06-07 16:08:31 +01:00
Hieu Hoang
b63380dae4 use zone and walls 2016-06-07 15:59:18 +01:00
Hieu Hoang
36dedae35f zone 2016-06-07 14:09:50 +01:00
Hieu Hoang
7c038657aa compile error on xdata 2016-06-07 13:46:22 +01:00
Hieu Hoang
764b2b2e05 import ReorderingConstraint class 2016-06-07 13:40:32 +01:00
Hieu Hoang
4fcc10a2c2 don't create xml options if we don't need it 2016-06-06 20:42:26 +01:00
Hieu Hoang
66493f9718 start xml 2016-06-06 20:27:48 +01:00
Hieu Hoang
05a2c0c1ab recycle queue items 2016-06-05 23:37:13 +01:00
Hieu Hoang
9442e26473 recycle queue items 2016-06-05 23:29:27 +01:00
Hieu Hoang
7b328f3cdc recycle hypos 2016-06-05 21:35:38 +01:00
Hieu Hoang
81dd25ec0a seen positions 2016-06-04 01:17:46 +01:00
Hieu Hoang
5a065fa2c6 seen positions 2016-06-03 23:53:40 +01:00
Hieu Hoang
fec1319e33 consistent use of mem pool 2016-06-03 23:04:02 +01:00
Hieu Hoang
77f0551008 seen positions 2016-06-03 22:50:12 +01:00
Hieu Hoang
aab5ccd1c7 delete debug 2016-06-03 21:26:27 +01:00
Hieu Hoang
821153c75d clear out queue before using it 2016-06-03 21:21:42 +01:00
Hieu Hoang
d0acadae46 debug 2016-06-03 19:57:51 +01:00
Hieu Hoang
d0c237e2d6 don't require n-best list 2016-06-03 17:04:30 +01:00
Hieu Hoang
1a2edfbfeb fix leaks 2016-06-03 16:19:22 +01:00
Hieu Hoang
6206ee8d40 get ready to make QueueItem a recycleable object 2016-06-03 16:03:41 +01:00
Hieu Hoang
0702f9fb79 share queue item's HyposColl 2016-06-03 15:53:33 +01:00
Hieu Hoang
dbaac76c1f only create hypos with static Create() 2016-06-03 15:33:07 +01:00
Hieu Hoang
408490f626 call EvaluateWhenApplied(). No segfaults. Hallelujah 2016-06-03 15:29:11 +01:00
Hieu Hoang
2a2fcc9696 move symbol bind from hypo to queue item 2016-06-03 14:14:48 +01:00
Hieu Hoang
cfee8cde88 cube pruning - queue 2016-06-02 17:47:04 +01:00
Hieu Hoang
2f371b79b1 cube pruning 2016-06-02 16:21:15 +01:00
Hieu Hoang
c7e0221e6b cube pruning 2016-06-02 13:57:29 +01:00
Hieu Hoang
720af91638 cube pruning 2016-06-02 12:58:29 +01:00
Hieu Hoang
cc5631c469 replumb n-best arcs. Now works 2016-06-01 23:44:46 +01:00
Hieu Hoang
ffbd774afd debug 2016-06-01 23:30:26 +01:00
Hieu Hoang
14fd2e1def Merge ../mosesdecoder into perf_moses2 2016-06-01 21:24:38 +01:00
Hieu Hoang
2ad1eacef7 eclipse 2016-06-01 21:24:28 +01:00
Hieu Hoang
3b879e17e7 runs on 1 word sentence but crashes otherwise 2016-06-01 20:53:52 +01:00
Hieu Hoang
48bfb00248 change symbol bind pointer to object. May be source of data leak 2016-06-01 20:21:44 +01:00
Hieu Hoang
f7e5b0fe0d debug 2016-06-01 20:11:01 +01:00
Hieu Hoang
082b10fa11 debug 2016-06-01 19:28:49 +01:00
Hieu Hoang
38d6ab09ad debug 2016-06-01 18:47:57 +01:00
Hieu Hoang
a05dc31059 debug 2016-06-01 16:36:44 +01:00
Hieu Hoang
8ef36d2925 debug 2016-06-01 16:12:50 +01:00
Hieu Hoang
9f9cec80e3 debug 2016-06-01 15:59:35 +01:00
Hieu Hoang
7e5e7b8614 debug 2016-06-01 15:27:13 +01:00
Hieu Hoang
7913c5d09d debug 2016-06-01 14:29:48 +01:00
Hieu Hoang
f7e910b68a debug 2016-05-31 18:34:26 +01:00
Hieu Hoang
ce85cf2529 set score for SCFG model 2016-05-30 21:23:23 +02:00
Hieu Hoang
eca081976e set state 2016-05-30 19:47:19 +02:00
Hieu Hoang
11f39c2278 set scores for scfg 2016-05-30 19:42:28 +02:00
Hieu Hoang
28cbc08fcc change EvaluateWhenApplied() to more closely resemble moses 2016-05-30 19:32:56 +02:00
Hieu Hoang
dacdc28bf4 BlankState() take System arg, not Manager 2016-05-30 17:21:12 +02:00
Hieu Hoang
e1ebefd7c8 eclipse 2016-05-26 23:44:37 +01:00
Hieu Hoang
178f5a9f18 kenlm chart state 2016-05-26 21:11:56 +01:00
Hieu Hoang
da4bd6dd51 EvaluateWhenApplied() for SCFG 2016-05-26 20:32:42 +01:00
Hieu Hoang
3f73c77f58 kenlm EvaluateInIsolation() for SCFG 2016-05-26 15:11:23 +01:00
Hieu Hoang
28816cbd1f mem leak in target phrases 2016-05-26 14:46:27 +01:00
Hieu Hoang
843a3033eb mem leak in hypo 2016-05-26 12:26:48 +01:00
Hieu Hoang
9c1d258b62 mem leak in active chart 2016-05-26 12:21:37 +01:00
Hieu Hoang
8466d6aaf5 mem leak in InputPath 2016-05-26 11:42:00 +01:00
Hieu Hoang
aa865da027 mem leak 2016-05-25 22:25:10 +01:00
Hieu Hoang
2e6ef643bd adhere to phrase limit 2016-05-25 16:31:48 +01:00
Hieu Hoang
fbba1e17f8 adhere to phrase limit 2016-05-25 16:22:24 +01:00
Hieu Hoang
e4f2397d1a adhere to phrase limit 2016-05-25 15:53:19 +01:00
Hieu Hoang
8bb252ffed correctly set input sizes for loops and matrix. Doesn't crash, but doesn't adhere to phrase limit 2016-05-25 14:44:09 +01:00
Hieu Hoang
bf21dc37f1 set is-nt in unk penalty. Still segfaults 2016-05-25 00:43:26 +01:00
Hieu Hoang
0dc66e1c08 delete debug 2016-05-24 23:35:43 +01:00
Hieu Hoang
ff360f2a55 compile error on clang 6 2016-05-17 13:22:17 -04:00
Hieu Hoang
248e957471 score hypo 2016-05-11 15:47:41 +01:00
Hieu Hoang
e261980e97 start using state info to store prev ngram 2016-05-11 11:30:00 +01:00
Hieu Hoang
853ac1736f rename GLM -> GPULM 2016-05-11 11:25:04 +01:00
Hieu Hoang
6f7c429f53 create contexts 2016-05-11 10:14:41 +01:00
Hieu Hoang
9c852f0d74 add GLM 2016-05-10 19:17:58 +01:00
Hieu Hoang
d9b2855cdd reuse Batch 2016-05-10 15:33:07 +01:00
Hieu Hoang
9063ccd14e use Vector for batch 2016-05-10 15:19:36 +01:00
Hieu Hoang
b708533e22 typedef Batch in Typedef.h 2016-05-10 15:07:45 +01:00
Hieu Hoang
510c60b222 batch evaluate 2016-05-09 23:55:59 +01:00
Hieu Hoang
f6e8e3aee0 batch evaluate 2016-05-09 18:33:15 +01:00
Hieu Hoang
a0e00ded87 add batch search 2016-05-09 18:00:15 +01:00
Hieu Hoang
63e4ff419b add KenLM Batch. 2016-05-09 17:05:34 +01:00
Hieu Hoang
f3a150a05a add batch search 2016-05-09 14:59:52 +01:00
Hieu Hoang
4b9d2d9912 namespace fo Normal search 2016-05-09 13:43:43 +01:00
Hieu Hoang
362705d4f4 timer loaded 2016-05-08 21:48:17 +01:00
Hieu Hoang
1778151b7e explicit word constructor 2016-05-06 16:07:57 +01:00
Hieu Hoang
9f5f729c19 create hypos 2016-05-06 15:41:50 +01:00
Hieu Hoang
6a326e909c add hypo coll to symbol bind for nt 2016-05-06 11:23:31 +01:00
Hieu Hoang
8a46657dea set prev hypos 2016-05-05 17:41:50 +01:00
Hieu Hoang
ed6db0a6b3 simplify ActiveChartEntry constructor 2016-05-01 10:58:22 +01:00
Hieu Hoang
fb6018ba4f add hypo coll to symbol bind for nt 2016-05-01 10:54:17 +01:00
Hieu Hoang
613a8cab06 correct symbol bind 2016-05-01 09:32:49 +01:00
Hieu Hoang
dc6e9ee5ad correct symbol bind 2016-05-01 08:58:36 +01:00
Hieu Hoang
070fe0bff4 tweak 2016-04-30 23:22:29 +01:00
Hieu Hoang
b227a9222d tweak 2016-04-30 22:50:48 +01:00
Hieu Hoang
b7e85c37ff store actual words/no rather than just isNT in symbolBInd 2016-04-30 19:59:14 +01:00
Hieu Hoang
8c1883a838 Merge branch 'perf_moses2' of github.com:moses-smt/mosesdecoder into perf_moses2 2016-04-30 16:15:07 +01:00
hieu
a888c8cea7 gcc compile error on centos 6.7 2016-04-30 11:10:08 -04:00
Hieu Hoang
8c3ab7046d debugging messages 2016-04-30 11:47:51 +01:00
Hieu Hoang
0feb925061 merge 2016-04-29 19:18:22 +01:00
Michael R. Hutt
66a83ed09b adds missing extraction operators for enums 2016-04-29 11:53:59 -04:00
Hieu Hoang
afe2af6b93 get ready for scfg prev hypos 2016-04-29 13:09:49 +01:00
Hieu Hoang
9de346825f less debugging messages 2016-04-29 12:23:43 +01:00
Hieu Hoang
96e735dda1 isNT must be set correctly when creating factor 2016-04-29 12:19:53 +01:00
Hieu Hoang
88b4b23434 debugging messages 2016-04-29 10:54:58 +01:00
Hieu Hoang
cca5f17f81 debugging messages 2016-04-29 01:05:47 +01:00
Hieu Hoang
acd1ae8b6e debugging messages 2016-04-29 00:09:02 +01:00
Hieu Hoang
86243ef945 tweak 2016-04-28 23:45:23 +01:00
Hieu Hoang
895b461a85 debugging messages 2016-04-28 23:41:09 +01:00
Hieu Hoang
f12f61f157 LookupNT() 2016-04-28 20:56:26 +01:00
Hieu Hoang
29345fbbc4 debug messages 2016-04-28 20:08:55 +01:00
Hieu Hoang
389d497ddb measure time from before loading 2016-04-28 16:51:42 +01:00
Hieu Hoang
38c1872bea debug messages 2016-04-28 15:19:53 +01:00
Hieu Hoang
e899ef9e4c use pointers for root nodes to prevent them being confused with each other 2016-04-28 00:21:02 +01:00
Hieu Hoang
b353025c42 add target phrase to path 2016-04-28 00:13:25 +01:00
Hieu Hoang
aa08ce9a0a tweak 2016-04-28 00:09:19 +01:00
Hieu Hoang
aaa3f24109 debugging method for source phrase 2016-04-27 23:30:24 +01:00
Hieu Hoang
f84e9c60e0 delete debugging message 2016-04-27 23:19:19 +01:00
Hieu Hoang
758c509e3c word penalty 2016-04-27 23:30:12 +04:00
Hieu Hoang
876aad0eca EvaluateInIsolation() for SCFG 2016-04-27 23:21:32 +04:00
Hieu Hoang
efb7155e6a call correct Sentence class 2016-04-27 22:59:03 +04:00
Hieu Hoang
ad0541e0f8 templatize node 2016-04-27 22:52:05 +04:00