moses-smt/mosesdecoder
1
1
Fork 0
mirror of https://github.com/moses-smt/mosesdecoder.git synced 2024-12-26 13:23:25 +03:00
Code Issues Packages Projects Releases Wiki Activity
10,967 Commits 136 Branches 26 Tags 147 MiB
1758381c98
Commit Graph

281 Commits

Author SHA1 Message Date
Hieu Hoang
e1a86bd9c9 sometimes target phrase collection can have 0 phrases 2013-07-16 16:30:54 +01:00
Hieu Hoang
56f4344ff2 merge 2013-07-13 23:26:21 +01:00
Kenneth Heafield
789039fa2e In-line PhraseDictionaryNodeMemory with the hash table data structure 
Before:
Name:moses_chart        VmPeak:11988700 kB      VmRSS:10112684 kB
real    2m58.393s
user    2m39.673s
sys     0m12.171s

After:
Name:moses_chart        VmPeak:11952688 kB      VmRSS:10077304 kB
real    2m42.329s
user    2m31.219s
sys     0m10.261s
 2013-07-12 07:13:19 -07:00
Kenneth Heafield
1b3f769c5c Always have a TargetPhraseCollection. 
Before:
Name:moses_chart        VmPeak:12027904 kB      VmRSS:10151884 kB       real:173.000
real    2m53.608s
user    2m44.003s
sys     0m8.401s

After:
Name:moses_chart        VmPeak:11988700 kB      VmRSS:10112684 kB       real:177.500
real    2m58.393s
user    2m39.673s
sys     0m12.171s
 2013-07-12 07:01:33 -07:00
Hieu Hoang
43507b61e3 Merge github.com:moses-smt/mosesdecoder into hieu_opt_input 2013-07-12 14:44:26 +01:00
Kenneth Heafield
23c2530617 PhraseDictionaryNodeMemory doesn't need to be virtual. Slight reduction in CPU/RAM. 2013-07-12 06:32:33 -07:00
Hieu Hoang
66e7720b91 rename SetTargetPhraseFromPtMatrix() to GetTargetPhraseCollectionBatch() 2013-07-11 20:18:06 +01:00
Hieu Hoang
c6d8259f68 delete as many override of GetTargetPhraseCollection() as possible 2013-07-11 20:05:08 +01:00
Hieu Hoang
5be6d8d755 call legacy code for confusion network input with binary pt 2013-07-11 17:24:19 +01:00
Hieu Hoang
e7f6c6906c Merge github.com:moses-smt/mosesdecoder into hieu_opt_input 2013-07-11 15:42:13 +01:00
Hieu Hoang
958d7ca9b8 prefix subphrase optimization for confusion networks 2013-07-09 16:56:49 +01:00
Hieu Hoang
5f9d0a85f5 rename vector<InputPath> to InputPathList. Pass regression tests 2013-07-09 14:19:35 +01:00
Hieu Hoang
49ede97304 merge 2013-07-09 12:41:08 +01:00
Hieu Hoang
7eb5e41ea1 Merge pull request #41 from neubig/check-processphrasetable-width 
Prevented segfault on print non-existant alignment string
 2013-07-09 04:28:57 -07:00
Hieu Hoang
e07304bdd1 beautify 2013-07-08 20:56:51 +01:00
Hieu Hoang
310b26f989 beautify 2013-07-08 20:52:14 +01:00
Hieu Hoang
3eba5782c2 beautify 2013-07-08 20:25:47 +01:00
Hieu Hoang
e2dd5b2efa beautify 2013-07-08 19:12:11 +01:00
Hieu Hoang
852242b804 beautify 2013-07-08 19:02:18 +01:00
Hieu Hoang
8d857fa26e OnDisk pt passes regression 2013-07-08 19:01:28 +01:00
Hieu Hoang
b03ac10a2d error processing OOV when using OnDisk pt 2013-07-08 17:58:22 +01:00
Marcin Junczys-Dowmunt
d3b4c11be2 Fixed queryPhraseTableMin, added warnings for compacting phrase tables qithout alignment 2013-07-08 09:58:02 +02:00
Hieu Hoang
29b895a97a InputLatticeNode to InputPath 2013-07-07 02:14:51 +01:00
Hieu Hoang
ecef376aed rename InputLatticeNode.h to InputPath.h 2013-07-06 22:42:52 +01:00
Hieu Hoang
6a060abe74 a little bit more towards using prefix subphrase to optimize translation rule lookup 2013-07-05 11:24:27 +01:00
Hieu Hoang
7114b164c7 cleanup of OnDiskPt methods 2013-07-05 11:10:10 +01:00
Hieu Hoang
ee0d29f56b prefix subphrase optimization 2013-07-05 10:57:45 +01:00
Hieu Hoang
e073d22301 prefix subphrase optimization 2013-07-05 10:52:12 +01:00
Hieu Hoang
a60da7d4b7 prefix subphrase optimization 2013-07-05 09:59:50 +01:00
Graham Neubig
0cde047e6e Merge branch 'master' of github.com:neubig/mosesdecoder into check-processphrasetable-width 2013-07-05 12:38:31 +09:00
Graham Neubig
6c25048d20 Prevented segfault on print non-existant alignment 
In processPhraseTable, there is an option to print alignments.
When this is selected, processPhraseTable will print the 4th
column in the phrase table, but the check for the phrase table
width only currently goes to the 3rd column. This patch makes
sure that the phrase table must have 4 columns when alignments
are being printed, and 3 otherwise.
 2013-07-05 12:21:01 +09:00
Hieu Hoang
574d186708 beautify 2013-07-04 23:38:18 +01:00
Hieu Hoang
0efcf64c28 Merge github.com:moses-smt/mosesdecoder into hieu_opt_input2 2013-07-04 23:37:21 +01:00
Hieu Hoang
f4d77aae38 start prefix subphrase optimization with OnDisk pt 2013-07-04 23:36:17 +01:00
Marcin Junczys-Dowmunt
bc904ff61f Deleted stray "cerr << .." 2013-07-05 00:15:27 +02:00
Hieu Hoang
f0dc30f25a prefix subphrase optimization done for in-memory pt. Need to delete objects also 2013-07-04 15:13:29 +01:00
Hieu Hoang
934e24718e a little bit more towards using prefix subphrase to optimize translation rule lookup 2013-07-04 09:24:13 +01:00
Hieu Hoang
5e355c1595 move lookup of all subphrases into PhraseDictionary class 2013-07-03 19:07:36 +01:00
Hieu Hoang
ac8d10af38 move lookup of all subphrases into PhraseDictionary class 2013-07-03 16:57:23 +01:00
Hieu Hoang
d203f2bb6e single threaded build compiles & passes regression 2013-07-01 17:37:24 +01:00
Ulrich Germann
c963338476 Feature values are now returned as log probabilities. 2013-06-27 17:55:16 +01:00
Hieu Hoang
efc2c6145e beautify 2013-06-26 17:19:09 +01:00
Hieu Hoang
771e792bb4 Merge branch 'master' into dynamic-phrase-tables 2013-06-26 17:13:36 +01:00
Hieu Hoang
3833d67ec0 merge with master 2013-06-26 17:12:22 +01:00
Hieu Hoang
352e36334a Merge branch 'master' of github.com:moses-smt/mosesdecoder 2013-06-26 11:54:12 +01:00
Ulrich Germann
2f6ca0c0b8 Merge branch 'master' into dynamic-phrase-tables 
Conflicts:
	moses/Hypothesis.cpp
	moses/StaticData.cpp
	moses/TargetPhrase.cpp
	moses/TranslationModel/BilingualDynSuffixArray.cpp
	moses/TranslationModel/BilingualDynSuffixArray.h
	moses/TranslationModel/PhraseDictionary.h
	moses/TranslationModel/PhraseDictionaryDynSuffixArray.cpp
	moses/TranslationModel/PhraseDictionaryDynSuffixArray.h
 2013-06-26 03:00:27 +01:00
Hieu Hoang
ea0c7b63e7 fix argument parsing in PhraseDictionaryALSuffixArray 2013-06-24 16:10:22 +01:00
Kenneth Heafield
718e90d006 Missing header 2013-06-21 09:38:45 -04:00
Hieu Hoang
a69ad79e9c Merge branch 'master' of github.com:moses-smt/mosesdecoder 2013-06-21 14:38:14 +01:00
Kenneth Heafield
217c8ea38f Missing header / Mehndi Bhargava 2013-06-21 09:37:05 -04:00
Hieu Hoang
3d790b7226 minor cleanup 2013-06-20 17:15:54 +01:00
Hieu Hoang
d68277d369 redo parsing of feature function parameters 2013-06-20 13:25:02 +01:00
Hieu Hoang
24d2edd790 redo parsing of feature function parameters 2013-06-20 13:06:03 +01:00
Hieu Hoang
dc33fa3d3d redo parsing of feature function parameters 2013-06-20 12:50:41 +01:00
Hieu Hoang
c2f4c48758 clean up SetParameter() 2013-06-19 00:24:21 +01:00
Hieu Hoang
acb4baacaf delete 2 unused legacy functions 2013-06-17 18:46:40 +01:00
Hieu Hoang
e51c4feb6b push calling of OnlyTheseFactors() to individual PT implementations, as needed 2013-06-17 18:31:06 +01:00
Hieu Hoang
f7371cf53d calculate m_featuresToApply in Load() for PhraseDictionary. Since every phrase dictionary now has to implement it, make Load() for PhraseDictionary abstract 2013-06-14 18:34:47 +01:00
Rico Sennrich
b1b70e802d use read-write lock in multimodel 2013-06-12 15:25:53 +02:00
Rico Sennrich
9e9cbbc13b refactoring multimodel code out of StaticData; locking. 2013-06-12 14:47:40 +02:00
Hieu Hoang
5becd868a9 delete misc unused variables from PhraseDictionary 2013-06-11 16:31:43 +01:00
Hieu Hoang
61e2b7a24f refactor parsing of feature functiona args 2013-06-11 01:46:04 +01:00
Hieu Hoang
b4ad093334 refactor parsing of feature functiona args 2013-06-11 00:05:12 +01:00
Hieu Hoang
efa42b451c refactor parsing of feature functiona args 2013-06-10 21:24:31 +01:00
Hieu Hoang
abe6bb7c22 refactor parsing of feature functiona args 2013-06-10 18:11:55 +01:00
Hieu Hoang
96d116dba0 refactor parsing of feature functiona args 2013-06-10 16:29:07 +01:00
Hieu Hoang
e291c5f4a4 implement feature override as a run-time override of specific argument of a feature function. Only implemented table-limit for far 2013-06-07 17:32:01 +01:00
Hieu Hoang
23fba08ded minor 2013-06-06 21:49:38 +01:00
Hieu Hoang
df4b8e5828 m_numInputScores deleted from PhraseDictionary. 2013-06-06 18:54:44 +01:00
Ulrich Germann
7ecfb88a29 Alpha version of phrase tables based on dynamic suffix arrays. 2013-06-05 10:46:42 +01:00
Hieu Hoang
21ff6d4831 beautify 2013-06-03 12:33:18 +01:00
Ulrich Germann
570b7c6aa5 Fixed bug in ComparePosition::operator()(...). 2013-06-03 12:22:05 +01:00
Hieu Hoang
3b76143072 refactor loading in class PhraseDictionaryFuzzyMatch 2013-05-31 23:39:19 +01:00
Hieu Hoang
eaf53769ad consistent Load() for all feature functions. Can be used not to load if doing --show-weights 2013-05-31 20:21:02 +01:00
Hieu Hoang
173cfffaac Reapply Abby Levenberg's log change @4b4aba3 2013-05-31 15:06:36 +01:00
Hieu Hoang
0db5504629 port PhraseDictionaryDynSuffixArray to new ini format. Scores aren't the same. Probably something to do with Abby Levenberg's log change 2013-05-31 15:02:34 +01:00
Hieu Hoang
65d2d68bc5 beautify 2013-05-31 00:00:21 +01:00
Hieu Hoang
117eb76b0a figure out which feature function to apply at which decode step. Book-keeping 2013-05-30 18:34:10 +01:00
Hieu Hoang
6249432407 beautify 2013-05-29 18:16:15 +01:00
Hieu Hoang
66afd6611b parse scores correctly. Assume dense features, always in the same order 2013-05-28 15:54:11 +01:00
Hieu Hoang
ed220e91a9 debug PhraseDictionaryALSuffixArray for Wilker. Sparse feature score to do 2013-05-28 15:38:36 +01:00
Hieu Hoang
0580e2bd11 delete create-ini 2013-05-28 13:11:37 +01:00
Hieu Hoang
036ea4ee29 delete sourcePhrase variable in TargetPhrase class 2013-05-28 11:38:18 +01:00
Hieu Hoang
ae82d02a05 delete sourcePhrase variable in TargetPhrase class 2013-05-28 00:41:25 +01:00
Hieu Hoang
e8dadb58ce split classes in DummyScoreProducers into separate files 2013-05-25 00:57:37 +01:00
Hieu Hoang
a80d838b0f move feature functions into moses/FF 2013-05-24 18:02:49 +01:00
Hieu Hoang
6c322ba05e move and rename PhraseDictionarySCFG -> PhraseDictionaryMemory 2013-05-24 16:33:30 +01:00
Hieu Hoang
4a280c8ca7 move and rename PhraseDictionarySCFG -> PhraseDictionaryMemory 2013-05-24 16:23:31 +01:00
Hieu Hoang
d273812231 delete enum StatelessFeatureType. Features can be multiple types at once. Just overload the Evaluate() function you need 2013-05-24 16:04:38 +01:00
Hieu Hoang
ca8007be12 leak 2013-05-23 15:29:25 +01:00
Hieu Hoang
b10ecf2b44 make each node in trie a pointer. No possibility that the node is delete during build of the trie 2013-05-23 12:50:57 +01:00
Hieu Hoang
6480805635 technical memory leak due to not cleaning up StaticData. Doesn't make much difference, but annoying messages from valgrind 2013-05-22 17:32:06 +01:00
Hieu Hoang
bfe321a068 change reference to sourceLHS to pointer, it can be NULL for phrase based. TODO - sourceLHS isn't being stored in the trie 2013-05-22 15:00:32 +01:00
Hieu Hoang
541bf99b9b Make Phrase::CreateFromString() more efficient 2013-05-22 14:41:28 +01:00
Hieu Hoang
bce3b36170 Make Phrase::CreateFromString() more efficient 2013-05-22 12:28:32 +01:00
Hieu Hoang
112bcdc925 Make Phrase::CreateFromString() more efficient 2013-05-22 12:03:00 +01:00
Hieu Hoang
f4a75b30f7 Make lhsTarget a pointer 2013-05-22 11:22:17 +01:00
Ulrich Germann
7996378ffb Made InitDictionary() a pure virtual function to force explicit initialization in derived classes. 2013-05-21 19:06:39 +01:00
Hieu Hoang
2f38746f6a delete references to translation feature weights. Everything is done automatically via Evaluate() for each feature function 2013-05-21 14:00:28 +01:00
Hieu Hoang
7dc177514d delete references to WordPenaltyProducer 2013-05-21 13:50:18 +01:00