moses-smt/mosesdecoder
1
1
Fork 0
mirror of https://github.com/moses-smt/mosesdecoder.git synced 2024-12-28 22:45:50 +03:00
Code Issues Packages Projects Releases Wiki Activity
10,815 Commits 136 Branches 26 Tags 147 MiB
74aea53fd5
Commit Graph

2897 Commits

Author SHA1 Message Date
Ulrich Germann
a0c13837e0 Fixed computation of lexical scores. 2013-07-28 16:28:41 +01:00
Hieu Hoang
0b23a0e327 rollback excluding translation options that have incompatible alignments for placeholders. This should be exclude at the training step 2013-07-26 19:39:10 +01:00
Hieu Hoang
2aef439d8a Merge /home/hieu/workspace/github/mosesdecoder into hieu_opt_input 2013-07-25 15:26:38 +01:00
Hieu Hoang
76a9730ca8 Merge branch 'master' of github.com:moses-smt/mosesdecoder 2013-07-25 15:23:12 +01:00
Barry Haddow
7081f06413 Fixes to the shared build 2013-07-25 15:24:34 +01:00
Hieu Hoang
e2c2bc59f1 beautify 2013-07-25 15:23:05 +01:00
Hieu Hoang
07693a3f96 Merge /Users/hieuhoang/unison/workspace/github/mosesdecoder into hieu_opt_input 2013-07-25 08:42:24 +01:00
Hieu Hoang
c0aba71c79 bug processing unknown word with digits 2013-07-25 08:41:59 +01:00
Hieu Hoang
6ac4d4ddad Merge /home/hieu/workspace/github/mosesdecoder into hieu_opt_input 2013-07-24 19:04:54 +01:00
Hieu Hoang
8ffd53e321 create target phrases, not Translation Options, for XML. Can be used in both pb and scfg 2013-07-24 18:06:46 +01:00
Hieu Hoang
d5d4051d8b create target phrases, not Translation Options, for XML. Can be used in both pb and scfg 2013-07-24 15:55:17 +01:00
Kenneth Heafield
71ae8c9d19 LM/Factory.cpp -> FF/Factory.cpp oops 2013-07-24 12:13:11 +01:00
Hieu Hoang
7380d12af7 create target phrases, not Translation Options, for XML. Can be used in both pb and scfg 2013-07-23 16:46:30 +01:00
Hieu Hoang
26c373ca3c Merge /home/hieu/workspace/github/mosesdecoder into hieu_opt_input 2013-07-23 13:13:17 +01:00
Hieu Hoang
e6a3df7e97 Merge branch 'master' of github.com:moses-smt/mosesdecoder 2013-07-23 13:12:30 +01:00
Hieu Hoang
206b165d14 randlm compile with refactored code. No regression tests yet 2013-07-23 12:56:35 +01:00
Hieu Hoang
1e906bea73 add ControlRecombination feature function 2013-07-23 01:38:08 +01:00
Hieu Hoang
42c1c908a5 add ControlRecombination feature function 2013-07-23 01:32:25 +01:00
Hieu Hoang
2590601708 add ControlRecombination feature function 2013-07-20 23:41:49 +01:00
Hieu Hoang
a098227abe add ControlRecombination feature function 2013-07-20 23:10:50 +01:00
Hieu Hoang
96da822861 Don't deprecate lmodel-oov-feature 2013-07-20 17:20:12 +01:00
Hieu Hoang
b6f8e3c383 Don't mix old and new ini file format 2013-07-20 17:08:03 +01:00
Hieu Hoang
5b7a9af588 refactor RandLM. Compiles with eclipse but not with bjam 2013-07-20 00:19:04 +01:00
Hieu Hoang
11666a8359 RandLM is currently broken 2013-07-19 22:39:20 +01:00
Hieu Hoang
0c5618dc75 Merge /home/hieu/workspace/github/mosesdecoder into hieu_opt_input 2013-07-19 18:53:21 +01:00
Hieu Hoang
4a4b1a168d Merge branch 'master' of github.com:moses-smt/mosesdecoder 2013-07-19 18:52:54 +01:00
Kenneth Heafield
2f6e669fb9 Merge branch 'master' of github.com:moses-smt/mosesdecoder 2013-07-19 18:50:29 +01:00
Kenneth Heafield
e1a2b2f0c9 Reduce scope of lm dependency 2013-07-19 18:50:12 +01:00
Hieu Hoang
d3aa3ee65c source and target factors for placeholders 2013-07-19 18:23:38 +01:00
Hieu Hoang
04801de0ae support for placeholder in pb model 2013-07-19 17:41:52 +01:00
Hieu Hoang
e528921e44 beautify 2013-07-19 15:38:39 +01:00
Hieu Hoang
b7b23dd703 starting placeholder 2013-07-19 15:38:13 +01:00
Hieu Hoang
116f1dcd41 beautify 2013-07-19 13:58:39 +01:00
Hieu Hoang
d7ab163b0e Merge /home/hieu/workspace/github/mosesdecoder into hieu_opt_input 2013-07-19 13:56:45 +01:00
Hieu Hoang
c77ec1b904 beautfiy 2013-07-19 13:56:02 +01:00
Hieu Hoang
1234b433da minor compile error 2013-07-19 13:52:47 +01:00
Hieu Hoang
320f7d575d Merge github.com:moses-smt/mosesdecoder into hieu_opt_input 2013-07-19 13:42:21 +01:00
Hieu Hoang
a95127b972 add default weights for feature functions that aren't tuneable, eg. OOV feature 2013-07-19 13:24:05 +01:00
Hieu Hoang
8a28178339 add default weights for feature functions that aren't tuneable, eg. OOV feature 2013-07-19 11:35:50 +01:00
Hieu Hoang
7896bbce5d Merge github.com:moses-smt/mosesdecoder into hieu_opt_input 2013-07-18 23:11:13 +01:00
Hieu Hoang
ea02ef6b6e find placeholders 2013-07-18 23:09:33 +01:00
Kenneth Heafield
b5e6b9c959 Factory 2013-07-18 22:54:52 +01:00
Hieu Hoang
6088531dfe starting placeholder 2013-07-18 20:39:15 +01:00
Hieu Hoang
0b5776faf7 starting placeholder 2013-07-18 20:23:44 +01:00
Hieu Hoang
2da41f7a56 Merge github.com:moses-smt/mosesdecoder into hieu_opt_input 2013-07-18 17:01:51 +01:00
Hieu Hoang
cb46bd3c00 beautify 2013-07-18 17:00:44 +01:00
Hieu Hoang
f81d7a446f starting placeholder 2013-07-18 17:00:07 +01:00
Hieu Hoang
89f74c130d Merge github.com:moses-smt/mosesdecoder into hieu_opt_input 2013-07-18 14:22:12 +01:00
Hieu Hoang
415c5ab693 tokenize only first occurence of = char for key=value. Allow paths with = chars 2013-07-18 14:01:08 +01:00
Hieu Hoang
87c346654e merge 2013-07-16 17:11:12 +01:00
Hieu Hoang
5cbfcacccc Merge github.com:moses-smt/mosesdecoder into hieu_opt_input 2013-07-16 16:56:22 +01:00
Hieu Hoang
f3e0fd2dd4 finished converting TargetPhraseCollection* to TargetPhraseCollection 2013-07-16 16:55:56 +01:00
Hieu Hoang
e1a86bd9c9 sometimes target phrase collection can have 0 phrases 2013-07-16 16:30:54 +01:00
Hieu Hoang
56f4344ff2 merge 2013-07-13 23:26:21 +01:00
Kenneth Heafield
789039fa2e In-line PhraseDictionaryNodeMemory with the hash table data structure 
Before:
Name:moses_chart        VmPeak:11988700 kB      VmRSS:10112684 kB
real    2m58.393s
user    2m39.673s
sys     0m12.171s

After:
Name:moses_chart        VmPeak:11952688 kB      VmRSS:10077304 kB
real    2m42.329s
user    2m31.219s
sys     0m10.261s
 2013-07-12 07:13:19 -07:00
Kenneth Heafield
1b3f769c5c Always have a TargetPhraseCollection. 
Before:
Name:moses_chart        VmPeak:12027904 kB      VmRSS:10151884 kB       real:173.000
real    2m53.608s
user    2m44.003s
sys     0m8.401s

After:
Name:moses_chart        VmPeak:11988700 kB      VmRSS:10112684 kB       real:177.500
real    2m58.393s
user    2m39.673s
sys     0m12.171s
 2013-07-12 07:01:33 -07:00
Hieu Hoang
43507b61e3 Merge github.com:moses-smt/mosesdecoder into hieu_opt_input 2013-07-12 14:44:26 +01:00
Kenneth Heafield
23c2530617 PhraseDictionaryNodeMemory doesn't need to be virtual. Slight reduction in CPU/RAM. 2013-07-12 06:32:33 -07:00
Hieu Hoang
6b1ac2a3d0 Merge github.com:moses-smt/mosesdecoder into hieu_opt_input 2013-07-11 20:43:32 +01:00
Hieu Hoang
a88096f669 debug message for overriding feature function arguments 2013-07-11 20:41:42 +01:00
Hieu Hoang
66e7720b91 rename SetTargetPhraseFromPtMatrix() to GetTargetPhraseCollectionBatch() 2013-07-11 20:18:06 +01:00
Hieu Hoang
c6d8259f68 delete as many override of GetTargetPhraseCollection() as possible 2013-07-11 20:05:08 +01:00
Hieu Hoang
5be6d8d755 call legacy code for confusion network input with binary pt 2013-07-11 17:24:19 +01:00
Hieu Hoang
183de6ea77 beautify 2013-07-11 16:20:15 +01:00
Hieu Hoang
7ec09ca24f call legacy code for confusion network input with binary pt 2013-07-11 16:19:38 +01:00
Hieu Hoang
a3a652bf62 Merge github.com:moses-smt/mosesdecoder into hieu_opt_input 2013-07-11 15:42:37 +01:00
Hieu Hoang
175b540509 Merge branch 'master' of github.com:moses-smt/mosesdecoder 2013-07-11 15:42:17 +01:00
Hieu Hoang
e7f6c6906c Merge github.com:moses-smt/mosesdecoder into hieu_opt_input 2013-07-11 15:42:13 +01:00
Hieu Hoang
ddffe5e01b minor debug tweak 2013-07-11 15:42:01 +01:00
Hieu Hoang
d7fa851f19 call legacy code for confusion network input with binary pt 2013-07-11 12:46:04 +01:00
Hieu Hoang
01641bfeee call legacy code for confusion network input with binary pt 2013-07-11 12:37:20 +01:00
Hieu Hoang
e3982f9ea8 minor compile error. Strange that it doesn't see function from it's antecedent 2013-07-11 10:26:24 +01:00
Hieu Hoang
6e64169a04 move CreateTranslationOptionsForRange() from TranslationOptionCollectionText to TranslationOptionCollection so that it can be used by confusion networks too 2013-07-11 10:13:34 +01:00
Hieu Hoang
bad58530e5 minor cleanup 2013-07-11 09:28:08 +01:00
Hieu Hoang
475ebd2bbb get rid of translation option cache 2013-07-10 19:44:38 +01:00
Hieu Hoang
3c343b7cc5 prefix subphrase optimization for confusion networks 2013-07-10 18:02:38 +01:00
Hieu Hoang
958d7ca9b8 prefix subphrase optimization for confusion networks 2013-07-09 16:56:49 +01:00
Hieu Hoang
92163d5091 prefix subphrase optimization done for confusion networks 2013-07-09 15:48:36 +01:00
Hieu Hoang
ffba363961 minor cleanup 2013-07-09 14:28:03 +01:00
Hieu Hoang
5f9d0a85f5 rename vector<InputPath> to InputPathList. Pass regression tests 2013-07-09 14:19:35 +01:00
Hieu Hoang
49ede97304 merge 2013-07-09 12:41:08 +01:00
Hieu Hoang
7eb5e41ea1 Merge pull request #41 from neubig/check-processphrasetable-width 
Prevented segfault on print non-existant alignment string
 2013-07-09 04:28:57 -07:00
Nadir Durrani
418abf42fa Merge branch 'nadir_osm' 2013-07-09 11:44:14 +01:00
Hieu Hoang
fad88a1451 begin adding input score to InputPath 2013-07-08 22:47:02 +01:00
Hieu Hoang
e07304bdd1 beautify 2013-07-08 20:56:51 +01:00
Hieu Hoang
310b26f989 beautify 2013-07-08 20:52:14 +01:00
Hieu Hoang
3eba5782c2 beautify 2013-07-08 20:25:47 +01:00
Hieu Hoang
e2dd5b2efa beautify 2013-07-08 19:12:11 +01:00
Hieu Hoang
852242b804 beautify 2013-07-08 19:02:18 +01:00
Hieu Hoang
8d857fa26e OnDisk pt passes regression 2013-07-08 19:01:28 +01:00
Hieu Hoang
b03ac10a2d error processing OOV when using OnDisk pt 2013-07-08 17:58:22 +01:00
Marcin Junczys-Dowmunt
d3b4c11be2 Fixed queryPhraseTableMin, added warnings for compacting phrase tables qithout alignment 2013-07-08 09:58:02 +02:00
Hieu Hoang
29b895a97a InputLatticeNode to InputPath 2013-07-07 02:14:51 +01:00
Hieu Hoang
ecef376aed rename InputLatticeNode.h to InputPath.h 2013-07-06 22:42:52 +01:00
Hieu Hoang
6a060abe74 a little bit more towards using prefix subphrase to optimize translation rule lookup 2013-07-05 11:24:27 +01:00
Hieu Hoang
7114b164c7 cleanup of OnDiskPt methods 2013-07-05 11:10:10 +01:00
Hieu Hoang
ee0d29f56b prefix subphrase optimization 2013-07-05 10:57:45 +01:00
Hieu Hoang
e073d22301 prefix subphrase optimization 2013-07-05 10:52:12 +01:00
Hieu Hoang
a60da7d4b7 prefix subphrase optimization 2013-07-05 09:59:50 +01:00
Graham Neubig
0cde047e6e Merge branch 'master' of github.com:neubig/mosesdecoder into check-processphrasetable-width 2013-07-05 12:38:31 +09:00
Graham Neubig
6c25048d20 Prevented segfault on print non-existant alignment 
In processPhraseTable, there is an option to print alignments.
When this is selected, processPhraseTable will print the 4th
column in the phrase table, but the check for the phrase table
width only currently goes to the 3rd column. This patch makes
sure that the phrase table must have 4 columns when alignments
are being printed, and 3 otherwise.
 2013-07-05 12:21:01 +09:00
Hieu Hoang
574d186708 beautify 2013-07-04 23:38:18 +01:00
Hieu Hoang
0efcf64c28 Merge github.com:moses-smt/mosesdecoder into hieu_opt_input2 2013-07-04 23:37:21 +01:00
Hieu Hoang
f4d77aae38 start prefix subphrase optimization with OnDisk pt 2013-07-04 23:36:17 +01:00
Marcin Junczys-Dowmunt
bc904ff61f Deleted stray "cerr << .." 2013-07-05 00:15:27 +02:00
Hieu Hoang
6e56a9aed5 beautify 2013-07-04 20:29:05 +01:00
Hieu Hoang
f35750bc08 beautify 2013-07-04 20:19:51 +01:00
Hieu Hoang
ed6b5add67 free objects 2013-07-04 16:07:24 +01:00
Hieu Hoang
f0dc30f25a prefix subphrase optimization done for in-memory pt. Need to delete objects also 2013-07-04 15:13:29 +01:00
Nadir Durrani
389b7762e8 Merge branch 'nadir_osm' of github.com:moses-smt/mosesdecoder into nadir_osm 2013-07-04 12:05:00 +01:00
Nadir Durrani
e402714d2b Getting Rid of feature-path 2013-07-04 12:04:29 +01:00
Hieu Hoang
934e24718e a little bit more towards using prefix subphrase to optimize translation rule lookup 2013-07-04 09:24:13 +01:00
Hieu Hoang
5e355c1595 move lookup of all subphrases into PhraseDictionary class 2013-07-03 19:07:36 +01:00
Hieu Hoang
ac8d10af38 move lookup of all subphrases into PhraseDictionary class 2013-07-03 16:57:23 +01:00
Hieu Hoang
c38e1a7682 Merge github.com:moses-smt/mosesdecoder into hieu_opt_input2 2013-07-03 10:04:36 +01:00
Nadir Durrani
33a167ac9d Merge branch 'nadir_osm' 2013-07-02 13:24:21 +01:00
Nadir Durrani
c05317dd82 Future-Cost 2013-07-02 13:19:55 +01:00
Hieu Hoang
8123772b43 separate class InputLatticeNode into seprate file 2013-07-01 23:27:13 +01:00
Hieu Hoang
8d98d026d4 Merge github.com:moses-smt/mosesdecoder into hieu_opt_input2 2013-07-01 22:30:41 +01:00
Hieu Hoang
bff761b605 a little bit more towards using prefix subphrase to optimize translation model lookup 2013-07-01 22:30:29 +01:00
Hieu Hoang
d203f2bb6e single threaded build compiles & passes regression 2013-07-01 17:37:24 +01:00
Nadir Durrani
582d31cb16 No SRILM 2013-07-01 13:10:58 +01:00
Nadir Durrani
ba72c70c6e Merge branch 'nadir_osm' of github.com:moses-smt/mosesdecoder into nadir_osm 2013-07-01 11:07:21 +01:00
Nadir Durrani
6a915253e1 Bug Fix 2013-07-01 11:06:55 +01:00
Kenneth Heafield
51c0b6ae43 Unbreak build by disabling OSM without SRI. 
TODO(nadir): remove (or make optional) your dependence on SRILM then undo this commit and remove the if statement from the Jamfile.
 2013-06-30 13:08:12 +01:00
Hieu Hoang
4e773acd5f find all target phrases before search 2013-06-29 22:00:23 +01:00
Hieu Hoang
eb0c7b62bc Merge https://github.com/moses-smt/mosesdecoder into hieu_opt_input2 2013-06-29 19:02:53 +01:00
Hieu Hoang
3fde63c7b1 find all target phrases before search 2013-06-28 18:15:12 +01:00
Hieu Hoang
600a188a4e beautify 2013-06-28 18:06:36 +01:00
Hieu Hoang
bddbdeec04 find all target phrases before search 2013-06-28 18:05:49 +01:00
Hieu Hoang
d2e37b701b find all target phrases before search 2013-06-28 17:40:22 +01:00
Hieu Hoang
2b357f91e3 separate CreateTranslationOptionsForRange() for text and confusion network input 2013-06-28 15:43:56 +01:00
Hieu Hoang
fa4b92fc0a Merge branch 'master' into nadir_osm 2013-06-28 14:19:30 +01:00
Nadir Durrani
8c19c2ba8a KenLM 2013-06-28 14:16:41 +01:00
Hieu Hoang
fd1a70739f create matrix of subphrases in class TranslationOptionCollectionText 2013-06-28 13:21:04 +01:00
Ulrich Germann
c963338476 Feature values are now returned as log probabilities. 2013-06-27 17:55:16 +01:00
Hieu Hoang
efc2c6145e beautify 2013-06-26 17:19:09 +01:00
Hieu Hoang
771e792bb4 Merge branch 'master' into dynamic-phrase-tables 2013-06-26 17:13:36 +01:00
Hieu Hoang
3833d67ec0 merge with master 2013-06-26 17:12:22 +01:00
Hieu Hoang
352e36334a Merge branch 'master' of github.com:moses-smt/mosesdecoder 2013-06-26 11:54:12 +01:00
Ulrich Germann
2f6ca0c0b8 Merge branch 'master' into dynamic-phrase-tables 
Conflicts:
	moses/Hypothesis.cpp
	moses/StaticData.cpp
	moses/TargetPhrase.cpp
	moses/TranslationModel/BilingualDynSuffixArray.cpp
	moses/TranslationModel/BilingualDynSuffixArray.h
	moses/TranslationModel/PhraseDictionary.h
	moses/TranslationModel/PhraseDictionaryDynSuffixArray.cpp
	moses/TranslationModel/PhraseDictionaryDynSuffixArray.h
 2013-06-26 03:00:27 +01:00
Nadir Durrani
4335587698 Merge branch 'nadir_osm' of github.com:moses-smt/mosesdecoder into nadir_osm 2013-06-25 17:09:15 +01:00
Nadir Durrani
7aecb78d76 merge 2013-06-25 17:09:06 +01:00
Hieu Hoang
eefd94edb1 don't pollute namespace 2013-06-25 17:08:15 +01:00
Nadir Durrani
f66bbdd1f1 merge 2013-06-25 17:00:28 +01:00
Nadir Durrani
4f7a55e247 KenLM 2013-06-25 16:54:41 +01:00
Hieu Hoang
546d0add94 merge 2013-06-25 16:53:41 +01:00
Hieu Hoang
4b881dd74f don't pollute namespace 2013-06-25 16:51:56 +01:00
Nadir Durrani
537210b010 KenLM 2013-06-25 16:27:41 +01:00
Hieu Hoang
493d713075 Merge branch 'nadir_osm' of github.com:moses-smt/mosesdecoder into nadir_osm 2013-06-25 10:52:00 +01:00