moses-smt/mosesdecoder
1
1
Fork 0
mirror of https://github.com/moses-smt/mosesdecoder.git synced 2024-09-20 07:42:21 +03:00
Code Issues Packages Projects Releases Wiki Activity
13,509 Commits 136 Branches 26 Tags 147 MiB
6ef257a9e9
Commit Graph

4392 Commits

Author SHA1 Message Date
Matthias Huck
59fd278ef7 IOWrapper: less compiler warnings 2014-12-04 19:16:30 +00:00
Matthias Huck
ad243f3500 Timer: no log output if verbosity parameter is 0 2014-12-04 19:14:11 +00:00
Matthias Huck
0f9e8aec91 verbosity macros 2014-12-04 19:13:42 +00:00
Hieu Hoang
bb6d04a9b7 Merge ../mosesdecoder.merge-cmd 2014-12-04 18:35:55 +00:00
Hieu Hoang
b23c00989b move OutputDetailedTreeFragmentsTranslationReport() to Manager 2014-12-04 18:35:19 +00:00
Hieu Hoang
3a2f58eb2c move OutputDetailedTranslationReport() to Manager 2014-12-04 16:46:14 +00:00
Hieu Hoang
c5fdb170fb move OutputUnknowns() into Managers 2014-12-04 16:25:19 +00:00
Hieu Hoang
d5e315a599 move OutputDetailedTranslationReport() into Syntax::Manager 2014-12-04 15:28:50 +00:00
Hieu Hoang
80eada0640 delete unused methods in IOWrapper 2014-12-04 14:43:23 +00:00
Hieu Hoang
5e4b27412a move n-best output into Syntax::Manager 2014-12-04 14:11:49 +00:00
Hieu Hoang
ba33cf93ba Merge branch 'master' of github.com:moses-smt/mosesdecoder 2014-12-03 23:43:39 +00:00
Matthias Huck
24a8a6a511 PhraseOrientationFeature 2014-12-03 20:04:26 +00:00
Hieu Hoang
2b3202dbc8 make Phil's Manager class inherit from BaseManager too 2014-12-03 17:27:21 +00:00
Hieu Hoang
f4a716f676 move OutputDetailedTranslationReport() to Managers 2014-12-03 17:18:19 +00:00
Hieu Hoang
ccbcc14ed8 move OutputDetailedTranslationReport() to Managers 2014-12-03 17:04:10 +00:00
Hieu Hoang
184e79f4d6 add OutputAlignment() to BaseManager. 2014-12-03 15:00:36 +00:00
Hieu Hoang
fb25616bdd add OutputLatticeSamples() to BaseManager. Move OutputAlignment() to ChartManager 2014-12-03 12:05:35 +00:00
Hieu Hoang
49a2ff1faa Merge branch 'merge-cmd' 2014-12-02 19:09:34 +00:00
Hieu Hoang
3da8415095 add OutputNBest() as abstract method to BaseManager. Tighten up framework 2014-12-02 19:09:10 +00:00
Hieu Hoang
ba7afba9f6 move n-best code for phrase-based from IOWrapper to ChartManager 2014-12-02 17:40:53 +00:00
Hieu Hoang
08c57bce87 delete OutputNBest() in IOWrapper 2014-12-02 15:12:01 +00:00
Hieu Hoang
d0576866af move OutputLatticeSamples() for phrase-based from IOWrapper to Manager 2014-12-02 14:58:44 +00:00
Hieu Hoang
29ead5be7b move n-best code for phrase-based from IOWrapper to Manager 2014-12-02 14:38:03 +00:00
Hieu Hoang
2fdd49d973 start cleanup of IOWrapper. Constructor takes no args 2014-12-01 15:26:47 +00:00
Rico Sennrich
56921cae3b small simplification of recursive CYK+ 
(following Chris Dyer's suggestion and Phil's refactoring in S2T decoder)
 2014-12-01 11:05:17 +00:00
Hieu Hoang
46e7aebce4 move SetBooleanParameter to Parameter class. Make it a specialized version of template SetParameter 2014-11-30 20:15:15 +00:00
mliska
0a264bf5e3 DistortionFunctionScorer dyn_cast removed in FF. 2014-11-29 17:02:10 +01:00
mliska
381c3f64d8 std::unordered_set replaces std::set for m_seenPosition. 2014-11-29 16:59:10 +01:00
Hieu Hoang
b2c0fa097c make GetParam() a const method with no side effect 2014-11-28 18:33:45 +00:00
Hieu Hoang
da8b4b74eb make GetParam() a const method with no side effect 2014-11-28 18:29:03 +00:00
Hieu Hoang
79e1c12c1c make GetParam() a const method with no side effect 2014-11-28 17:15:01 +00:00
Hieu Hoang
e8cdbcd84b make GetParam() a const method with no side effect 2014-11-28 14:46:10 +00:00
Hieu Hoang
8306de019c make GetParam() a const method with no side effect 2014-11-27 23:02:41 +00:00
Hieu Hoang
5b82661163 make GetParam() a const method with no side effect 2014-11-27 21:09:05 +00:00
Hieu Hoang
cafd01854d make GetParam() a const method with no side effect 2014-11-27 18:41:40 +00:00
Hieu Hoang
b70c34a083 make GetParam() a const method with no side effect 2014-11-27 17:49:08 +00:00
Hieu Hoang
604208964a make GetParam() a const method with no side effect 2014-11-27 12:51:23 +00:00
Hieu Hoang
e2ef576f40 Merge branch 'master' of github.com:moses-smt/mosesdecoder into merge-cmd 2014-11-27 12:12:29 +00:00
Paul Baltescu
4eaf58f100 Integrate OxLM tree models. 2014-11-27 01:34:22 +00:00
Hieu Hoang
0f597bfad6 make GetParam() a const method with no side effect 2014-11-26 18:44:06 +00:00
Hieu Hoang
32869f5c7c Merge branch 'merge-cmd' of github.com:moses-smt/mosesdecoder into merge-cmd 2014-11-26 17:55:21 +00:00
Hieu Hoang
194b84eeec Parameter class cleanup 2014-11-26 17:54:52 +00:00
Hieu Hoang
ea229fd25c Merge branch 'master' of github.com:moses-smt/mosesdecoder into merge-cmd 2014-11-26 17:53:46 +00:00
Hieu Hoang
cdb5cd3e62 make GetParam() a const method with no side effect 2014-11-26 17:47:33 +00:00
Hieu Hoang
a2bb435a1a make GetParam() a const method with no side effect 2014-11-26 16:41:25 +00:00
XapaJIaMnu
6be776bcd1 Implement a TODO 2014-11-26 12:31:47 +00:00
Rico Sennrich
70c2ca0fd1 re-apply commit e0b3105 (write tree fragment string, not pointer) 2014-11-25 12:00:03 +00:00
XapaJIaMnu
5b00eb6509 Throw an exception in case no alignemnt is found in BilingualLM 2014-11-24 14:19:55 +00:00
Hieu Hoang
a2967ab206 Merge branch 'master' of github.com:moses-smt/mosesdecoder 2014-11-24 14:11:23 +00:00
Hieu Hoang
efed89d0e6 eclipse 2014-11-24 13:46:08 +00:00
Paul Baltescu
a1e7ed8217 Cache OxLM queries when loading the phrase table. 2014-11-22 19:03:43 +00:00
Hieu Hoang
cf49419013 bug when compiled --with-mm but not using simulated post-editing 2014-11-22 10:49:24 +00:00
Ulrich Germann
6a70eed481 Bug fix in a sanity check in PrefixTree.h 2014-11-21 23:58:17 +00:00
Ulrich Germann
7aa4d5d8d5 Merge branch 'master' of https://github.com/moses-smt/mosesdecoder 
Conflicts:
	moses-cmd/simulate-pe.cc
 2014-11-20 17:55:51 +00:00
Hieu Hoang
2711410712 Merge branch 'master' of github.com:moses-smt/mosesdecoder 2014-11-20 17:38:17 +00:00
Hieu Hoang
9e5be78a7d simulated post editing 2014-11-20 17:28:50 +00:00
Hieu Hoang
c87db0c141 Parameter error 2014-11-20 17:05:08 +00:00
XapaJIaMnu
52c520c042 Resolve merge conflicts 2014-11-20 15:50:32 +00:00
Hieu Hoang
c346a4b198 Add rt library 2014-11-20 15:02:05 +00:00
Hieu Hoang
8137aeae43 Small changes to Parameters and StaticData 2014-11-20 11:21:50 +00:00
Nicola Bertoldi
626c9d7a84 improvement on passthrough information 2014-11-18 15:02:26 +01:00
Rico Sennrich
6748f4106f compatibility with NPLM 0.3 (breaks earlier versions of NPLM) 2014-11-17 18:08:42 +00:00
Hieu Hoang
b1902cac4f cygwin compile 2014-11-17 17:44:46 +00:00
Rico Sennrich
4a94a41310 compatibility with NPLM 0.3 (fork on https://github.com/rsennrich/nplm) 
breaks compatibility with older versions of NPLM, and older models.
 2014-11-17 15:12:18 +00:00
Ulrich Germann
dbcad69f1d Merge branch 'master' of https://github.com/moses-smt/mosesdecoder 2014-11-16 16:33:21 +00:00
Ulrich Germann
bda7ace530 Minor changes due to changes in the Moses API. Removed from list of standard programs to be compiled and installed. May need some work to get it working again. 2014-11-16 16:31:12 +00:00
Marcin Junczys-Dowmunt
7388b19188 Update IOWrapper.cpp 
Adding back std::endl in void IOWrapper::OutputAlignment(ostream &out, const vector<const Hypothesis *> &edges), removing it breaks -alignment-output-file
 2014-11-16 14:17:28 +01:00
Paul Baltescu
35401f7dd3 Merge branch 'bilingual-lm' of github.com:moses-smt/mosesdecoder into bilingual-lm 2014-11-14 12:39:02 +00:00
Paul Baltescu
fefd2b0ada Fix merge errors. 2014-11-13 16:52:24 +00:00
XapaJIaMnu
2028b86642 BilingualNPLM requries specific source and target vocabulary lists 2014-11-13 16:14:17 +00:00
Rico Sennrich
9b161dc888 floor score of bilingualNPLM 2014-11-13 16:14:16 +00:00
XapaJIaMnu
b254fcea99 moses/LM/BilingualLM.cpp 2014-11-13 16:14:16 +00:00
XapaJIaMnu
f6c64adb92 Remove unnecessary const gimmicks 2014-11-13 16:14:16 +00:00
XapaJIaMnu
18a6d12cb0 Rework lookup and greatly speedup decoding (2x+) 2014-11-13 16:14:16 +00:00
XapaJIaMnu
cf3fe60cf6 Backoff to null token in hiero 2014-11-13 16:14:16 +00:00
XapaJIaMnu
493f05eb97 Remove old unused code from BilingualLM 2014-11-13 16:14:16 +00:00
Paul Baltescu
211878c9d3 Make OxLM extensions compile. 2014-11-13 16:14:16 +00:00
XapaJIaMnu
8e4ff790a7 Add null word support for hiero. 2014-11-13 16:14:16 +00:00
Paul Baltescu
0578a31799 Flag for setting OxLM models to unnormalized. 2014-11-13 16:14:16 +00:00
XapaJIaMnu
5851a2c2bb Prevent realocation of vectors 2014-11-13 16:13:38 +00:00
Paul Baltescu
2705a47876 Fix OxLM. 2014-11-13 16:13:38 +00:00
XapaJIaMnu
02c375ef78 Refactor the BilingualLM for chart to make it faster. Untested 2014-11-13 16:13:04 +00:00
XapaJIaMnu
7858f74e9e Rename BilingualLM_NPLM so that it is not confused with a sparse feature 2014-11-13 16:13:04 +00:00
Paul Baltescu
61826cee8a Rename OxLM features. 2014-11-13 16:13:04 +00:00
Paul Baltescu
32c169c25f Optional back-off to POS tags in OxLM. 2014-11-13 16:12:19 +00:00
Paul Baltescu
86d64b65e2 Correctly map the source unknown token. 2014-11-13 16:10:40 +00:00
Paul Baltescu
167e272818 Fix Bilingual OxLM context word order. 2014-11-13 16:10:40 +00:00
Paul Baltescu
5f9d481ee6 Make query cache sentence specific. 2014-11-13 16:10:40 +00:00
Paul Baltescu
248aa4bf8a Convert moses words to oxlm word ids. 2014-11-13 16:10:40 +00:00
Paul Baltescu
97b632e045 Clean up OxLMMapper. 2014-11-13 16:08:56 +00:00
Paul Baltescu
90ebf13789 Set BilingualLM parameters nicely. 2014-11-13 16:08:10 +00:00
Paul Baltescu
7588c4b8e3 Skeleton for source conditioned OxLM feature. 2014-11-13 16:08:10 +00:00
Paul Baltescu
6f9d59129f Rename LBLLM -> OxLM. 2014-11-13 16:07:38 +00:00
Paul Baltescu
af28063e3b Fix compilation errors introduced by new oxlm changes. 2014-11-13 15:54:54 +00:00
Paul Baltescu
4811701277 Fix broken include. 2014-11-13 15:51:48 +00:00
Paul Baltescu
cb7167f088 Fix bugs in BilingualLM for chart based decoding. 2014-11-13 15:51:48 +00:00
Paul Baltescu
3624bd776c Fix a few bugs in BilingualLM for phrase based decoding. 2014-11-13 15:51:48 +00:00
Paul Baltescu
5f87cf94d8 Move BilingualLM under LM. 2014-11-13 15:51:48 +00:00
XapaJIaMnu
a1a10a9209 Remove unused variable that likely causes crashes 2014-11-13 15:51:48 +00:00
XapaJIaMnu
97b7c766e4 fix vectors not being cleared 2014-11-13 15:51:47 +00:00
XapaJIaMnu
5d1584a3d8 Make it possible to have backend specific parameters. 2014-11-13 15:51:47 +00:00
XapaJIaMnu
c2f0419089 Fix compilation 2014-11-13 15:51:47 +00:00
XapaJIaMnu
0271b451b7 Some file renaming and fixes 2014-11-13 15:51:47 +00:00
XapaJIaMnu
af8ee9da43 Abstract BilingualLM. Working 2014-11-13 15:51:47 +00:00
XapaJIaMnu
bceff44fcb Abstract BilingualLM to Binlm_NPLM. Doesn't compile 2014-11-13 15:51:47 +00:00
XapaJIaMnu
46d8987635 Start abstracting NPLM from BilingualLM 2014-11-13 15:51:47 +00:00
XapaJIaMnu
e449df548a Fix a bunch of off by one and silly errors 2014-11-13 15:51:47 +00:00
XapaJIaMnu
38197698e0 Fix silly mistake in getTargetPhraseChart 2014-11-13 15:51:47 +00:00
XapaJIaMnu
ea03722aaa Fix a stupid mistake in TargetWords 2014-11-13 15:51:47 +00:00
XapaJIaMnu
7e26228fd7 Add support for moses_chart. Cleaning up needed. 2014-11-13 15:51:47 +00:00
Hieu Hoang
f76e1e8a03 compiles on gcc 4.6.3 2014-11-13 15:51:47 +00:00
XapaJIaMnu
62ec653df8 More work, source_ngrams missing 2014-11-13 15:51:47 +00:00
XapaJIaMnu
2979a617a9 int -> size_t 2014-11-13 15:51:47 +00:00
XapaJIaMnu
6e148defba Rename variable to better represent what it means. Start syntactic decoding 2014-11-13 15:51:47 +00:00
XapaJIaMnu
ea01ede796 Support for POS backoff in factored models. Spaghetti code needs refactoring 2014-11-13 15:51:46 +00:00
XapaJIaMnu
caf797a6a4 GetNeuralLMId requries word instead of factor 2014-11-13 15:51:46 +00:00
XapaJIaMnu
b5e5e12c9f Make some parameters of neuralLM optional 2014-11-13 15:51:46 +00:00
XapaJIaMnu
59ab036eb5 Get state, correct 2014-11-13 15:51:46 +00:00
XapaJIaMnu
4309903d6e State, almost correct or correct 2014-11-13 15:51:46 +00:00
XapaJIaMnu
c225fb27d1 Fix small indexing error 2014-11-13 15:51:46 +00:00
XapaJIaMnu
19a11b854d Fast version. Needs some cleanup, but works 2014-11-13 15:51:46 +00:00
XapaJIaMnu
1d1972330a State doesn't require whole target phrase 2014-11-13 15:51:46 +00:00
XapaJIaMnu
27ac2e49c7 Full lookup cache 2014-11-13 15:51:46 +00:00
XapaJIaMnu
e333bef6f4 Caching working 2014-11-13 15:51:46 +00:00
XapaJIaMnu
d9177cf75a Matching scores! And bugfixing 2014-11-13 15:51:46 +00:00
XapaJIaMnu
563f08c1d7 More bugfixes. Still different result from slow and fast 2014-11-13 15:51:46 +00:00
XapaJIaMnu
15a98ceadb A bunch of bugfixes and initial fast version. Still something wrong 2014-11-13 15:51:46 +00:00
Hieu Hoang
41b207e623 fix Jamfile 2014-11-13 15:51:46 +00:00
XapaJIaMnu
2a9d83b375 Compiles, works, outputs some numbers 2014-11-13 15:50:06 +00:00
XapaJIaMnu
7b00ca1115 BilingualLM FF, compiles, untested. 2014-11-13 15:50:06 +00:00
XapaJIaMnu
37abe3f14a nearly done, compiles, untested 2014-11-13 15:50:06 +00:00
XapaJIaMnu
0af522e512 Getting source and target words, compiling 2014-11-13 15:50:06 +00:00
XapaJIaMnu
d1d38b4aa8 Started implementing evaluate when applied 2014-11-13 15:50:06 +00:00
XapaJIaMnu
b2b1d352d4 Don't do anything if we don't have enough ngrams 2014-11-13 15:50:06 +00:00
XapaJIaMnu
85db12549c EvaluateWithSource compiles 2014-11-13 15:50:06 +00:00
XapaJIaMnu
16eecbd35f Add scores for target side 2014-11-13 15:50:06 +00:00
XapaJIaMnu
598f4c194d Some neural integration 2014-11-13 15:50:06 +00:00
XapaJIaMnu
8f0ce75c33 my bad 2014-11-13 15:50:06 +00:00
XapaJIaMnu
683783d519 Compiles 2014-11-13 15:50:06 +00:00
XapaJIaMnu
4bea830188 doesn't work 2014-11-13 15:50:05 +00:00
Hieu Hoang
a7abd02fc6 add Bilingual LM class 2014-11-13 15:50:05 +00:00
Hieu Hoang
6dda0d19cf add Bilingual LM class 2014-11-13 15:50:05 +00:00
Rico Sennrich
00c35fb565 fix LMOOVFeature and implement it for NeuralLMWrapper 2014-11-13 10:36:26 +00:00
Paul Baltescu
d4b76d3945 Allow unnormalized scores in oxlm. 2014-11-12 15:19:45 +00:00
Phil Williams
33677ab862 Minor clean-up. 2014-11-10 16:54:14 +00:00
Hieu Hoang
8d15287eee added phil's functions 2014-11-08 17:36:32 +00:00
Hieu Hoang
18d2d56cc7 added phil's functions 2014-11-07 19:51:18 +00:00
XapaJIaMnu
ca06d05b66 BilingualNPLM requries specific source and target vocabulary lists 2014-11-07 15:28:49 +00:00
Hieu Hoang
e1092c0dad merge 2014-11-07 14:35:36 +00:00
Rico Sennrich
3dbf2b556b floor score of bilingualNPLM 2014-11-06 14:34:05 +00:00
XapaJIaMnu
43a036351c moses/LM/BilingualLM.cpp 2014-11-05 16:08:34 +00:00
Hieu Hoang
1615d56b69 Merge branch 'master' of github.com:moses-smt/mosesdecoder 2014-11-05 13:06:49 +00:00
Hieu Hoang
c8e49177a6 race condition in moses server 2014-11-05 13:06:33 +00:00
Laura Kieras
ecae85e9a8 mm2dTable now opens its data file read-only, using mapped_file_source, so that we don't need write permissions on the file 2014-11-04 16:30:46 -05:00
Phil Williams
5240c430ce Merge s2t branch 
This adds a new string-to-tree decoder, which can be enabled with the -s2t
option.  It's intended to be faster and simpler than the generic chart
decoder, and is designed to support lattice input (still WIP).  For a en-de
system trained on WMT14 data, it's approximately 40% faster in practice.

For background information, see the decoding section of the EMNLP tutorial
on syntax-based MT:

  http://www.emnlp2014.org/tutorials/5_notes.pdf

Some features are not implemented yet, including support for internal tree
structure and soft source-syntactic constraints.
 2014-11-04 13:13:56 +00:00
Nikolay Bogoychev
35a007e232 Merge branch 'master' into bilingual-lm 2014-10-27 11:56:56 +00:00
Ulrich Germann
07202c544c Added ptable-describe-features to list features used by PhraseDictionaryBitextSampling. 2014-10-25 12:06:38 -07:00
Ulrich Germann
44215b79c0 Added ptable-describe-features to list features used by PhraseDictionaryBitextSampling. 2014-10-25 12:06:24 -07:00
Ulrich Germann
53ef6c5c38 Added demo program for use of suffix arrays. 2014-10-23 11:11:28 -07:00
Barry Haddow
562cf7e007 Merge branch 'master' of github.com:moses-smt/mosesdecoder 2014-10-21 15:11:22 +01:00
Rico Sennrich
a128ad1847 n-best-tree output needs trees for unknown words 2014-10-21 09:51:20 +01:00
Rico Sennrich
b42591d585 NPLM could implement shared or thread-local cache; wrapper works with either. 2014-10-16 14:33:35 +01:00
XapaJIaMnu
74e90343d6 Remove unnecessary const gimmicks 2014-10-16 12:36:19 +01:00
XapaJIaMnu
bb06091713 Rework lookup and greatly speedup decoding (2x+) 2014-10-16 11:53:49 +01:00
XapaJIaMnu
250e376036 Backoff to null token in hiero 2014-10-15 17:00:09 +01:00
Rico Sennrich
ef2c9dc92d hiero model compatibility 2014-10-15 14:20:40 +01:00
XapaJIaMnu
c479ccfe89 Remove old unused code from BilingualLM 2014-10-15 13:13:15 +01:00
Paul Baltescu
2d7ae17e2d Make OxLM extensions compile. 2014-10-14 19:29:46 +01:00
XapaJIaMnu
7b7bfbc8f0 Add null word support for hiero. 2014-10-14 18:22:04 +01:00
Paul Baltescu
8b4ab21324 Flag for setting OxLM models to unnormalized. 2014-10-14 02:09:47 +01:00
Hieu Hoang
90e4eca0a3 cleaning up IOWrapper. Make all functions member of the class 2014-10-13 12:46:58 +01:00
Hieu Hoang
5078192a1c Merge ../mosesdecoder into merge-cmd 2014-10-13 10:42:46 +01:00
Hieu Hoang
855d972852 refactor some stateless feature functions to avoid using EvaluateWhenApplied(). Not kosher 2014-10-13 10:42:25 +01:00
Hieu Hoang
cce818015d Merge ../mosesdecoder into merge-cmd 2014-10-10 15:50:12 +01:00
Phil Williams
05ecc914c2 Fix a few more compiler warnings (from Clang mostly). 2014-10-10 15:47:53 +01:00
Phil Williams
ee57e59f2b Fix a few compiler warnings (from Clang mostly). 2014-10-10 14:22:53 +01:00
Hieu Hoang
d507af5b1d TranslationTaskChart --> TranslationTask 2014-10-10 13:19:02 +01:00
Hieu Hoang
4403bd5b3c get ready to merge IOWrapper 2014-10-10 11:50:32 +01:00
Hieu Hoang
e4489cbdde get ready to merge IOWrapper 2014-10-10 11:35:14 +01:00
Hieu Hoang
89f8c7d284 get ready to merge IOWrapper 2014-10-09 17:28:57 +01:00
Hieu Hoang
23d0afcde1 get ready to merge IOWrapper 2014-10-09 16:42:16 +01:00
Hieu Hoang
3f019a528d Merge ../mosesdecoder into merge-cmd 2014-10-09 16:28:43 +01:00
XapaJIaMnu
3fdda76e9c Merge branch 'bilingual-lm' of https://github.com/moses-smt/mosesdecoder into bilingual-lm 2014-10-09 14:19:24 +01:00
XapaJIaMnu
b787ae8d7d Prevent realocation of vectors 2014-10-09 14:19:11 +01:00
Hieu Hoang
849eadf41b consistent namespace 2014-10-09 13:33:16 +01:00
Hieu Hoang
bf089b56ac consistent namespace 2014-10-09 12:52:06 +01:00
Hieu Hoang
e7916239d5 delete m_translationId variable 2014-10-09 12:13:32 +01:00
Hieu Hoang
d744400a33 delete m_translationId variable 2014-10-09 11:12:59 +01:00
Paul Baltescu
f1a220ede9 Fix OxLM. 2014-10-08 22:53:05 +01:00
Paul Baltescu
8f74ecd8f3 Fix OxLM. 2014-10-08 22:08:42 +01:00
XapaJIaMnu
7306182337 Refactor the BilingualLM for chart to make it faster. Untested 2014-10-08 19:53:31 +01:00
XapaJIaMnu
e4d7779108 Rename BilingualLM_NPLM so that it is not confused with a sparse feature 2014-10-08 19:52:49 +01:00
Hieu Hoang
85eae6b9f2 delete lineNumber variable from ChartManager. It should always be carried by the input sentence itself 2014-10-08 19:23:26 +01:00
Hieu Hoang
1743f7eeb2 Merge ../mosesdecoder into merge-cmd 2014-10-08 17:55:07 +01:00
Hieu Hoang
deb2187428 delete lineNumber variable from Manager and TranslationTask. It should always be carried by the input sentence itself 2014-10-07 19:28:44 +01:00
Hieu Hoang
433186ae25 delete lineNumber variable from Manager and TranslationTask. It should always be carried by the input sentence itself 2014-10-07 19:16:30 +01:00
Ulrich Germann
576931b088 Mmsapt now adds word alignment info to target phrases. 2014-10-07 18:08:31 +01:00
Hieu Hoang
2f290f1d6d standardize GetIOWrapper(). 2014-10-03 16:08:15 +01:00
Hieu Hoang
65bd64ca6a move lattice sample collector from Main to IOWrapper 2014-10-03 15:30:06 +01:00
Hieu Hoang
62fe00b1a5 delete m_latticeSamplesCollector 2014-10-03 15:04:55 +01:00
Hieu Hoang
25db86d16b delete m_latticeSamplesCollector 2014-10-03 14:57:27 +01:00
Hieu Hoang
056913df57 delete nbestOut 2014-10-03 12:30:18 +01:00
Hieu Hoang
fe9a2561a3 Merge ../mosesdecoder into merge-cmd 2014-10-02 18:42:44 +01:00
Hieu Hoang
2468e2108a duplicate n-best stream processing 2014-10-02 18:39:40 +01:00
Hieu Hoang
e41d2ba08b sort out suppressSingleBestOutput variable 2014-10-02 17:08:29 +01:00
Michael Denkowski
fc4388e19c Merge branch 'master' of https://github.com/moses-smt/mosesdecoder 2014-10-02 11:15:31 -04:00
Michael Denkowski
5013e11dc5 XML option for runtime weight-overwrite 2014-10-02 11:15:03 -04:00
Hieu Hoang
af225a6d55 only 1 constructor for IOWrapper 2014-10-02 16:00:36 +01:00
Hieu Hoang
b907fb7492 merge TranslationTask and IOWrapper. Move m_wordGraphCollector into IOWrapper 2014-10-02 14:57:04 +01:00
Hieu Hoang
f8b762a6ec merge TranslationTask and IOWrapper. Move m_detailedTranslationCollector into IOWrapper 2014-10-02 14:18:12 +01:00
Hieu Hoang
c45967c9b4 merge TranslationTask and IOWrapper. Move m_searchGraphOutputCollector into IOWrapper 2014-10-02 12:20:49 +01:00
Hieu Hoang
0843a2a901 merge TranslationTask and IOWrapper. Move m_alignmentInfoCollector into IOWrapper 2014-10-02 11:22:52 +01:00
Hieu Hoang
00365257fa merge TranslationTask and IOWrapper. Move m_unknownsStream into IOWrapper 2014-10-01 19:24:58 +01:00
Hieu Hoang
8611dc8cdf merge TranslationTask and IOWrapper. Move m_nBestOutputCollector into IOWrapper 2014-10-01 18:21:57 +01:00
Hieu Hoang
8d0f74c6e4 merge TranslationTask and IOWrapper. Move m_singleBestOutputCollector into IOWrapper 2014-10-01 17:43:51 +01:00
Paul Baltescu
1d28f5186b Rename OxLM features. 2014-09-30 15:57:20 +01:00
Hieu Hoang
33ed15ef19 move misc common functions into moses/ 2014-09-30 14:22:38 +01:00
Ulrich Germann
a01b6d6257 Merge branch 'master' of https://github.com/moses-smt/mosesdecoder 2014-09-30 13:24:34 +01:00
Hieu Hoang
c20af584e7 move TranslationTask into moses/ 2014-09-30 12:59:31 +01:00
Hieu Hoang
c40faed0d6 separate out class TranslationTask into it's on file 2014-09-30 12:25:36 +01:00
Hieu Hoang
85261aa190 Merge ../mosesdecoder into merge-cmd 2014-09-30 10:32:29 +01:00
Hieu Hoang
87afd853e5 warn against using SRILM 1.7.1 2014-09-30 10:06:52 +01:00
Hieu Hoang
55db902f5f move IOWrapper into moses/ 2014-09-29 17:58:20 +01:00
Hieu Hoang
d60571341d move LatticeMBR into moses/ 2014-09-29 17:34:11 +01:00
Paul Baltescu
26ef9fddaa Optional back-off to POS tags in OxLM. 2014-09-29 17:25:26 +01:00
Hieu Hoang
08d6f77cea move TranslationAnalysis for both moses-cmd and moses-char-cmd into moses/ 2014-09-29 17:16:11 +01:00
Paul Baltescu
c5ad6f9839 Correctly map the source unknown token. 2014-09-29 01:05:43 +01:00
Paul Baltescu
edde0324dd Fix Bilingual OxLM context word order. 2014-09-28 16:53:41 +01:00
Paul Baltescu
c8ac4532ff Make query cache sentence specific. 2014-09-27 15:28:42 +01:00
Paul Baltescu
822aaa7dae Convert moses words to oxlm word ids. 2014-09-26 23:20:26 +01:00
Paul Baltescu
5c3b3f6057 Clean up OxLMMapper. 2014-09-26 16:46:04 +01:00
Paul Baltescu
3babd6dbb9 Set BilingualLM parameters nicely. 2014-09-26 16:25:48 +01:00
Paul Baltescu
a36ae66345 Skeleton for source conditioned OxLM feature. 2014-09-26 15:50:25 +01:00
Paul Baltescu
92f03662ec Rename LBLLM -> OxLM. 2014-09-26 15:18:14 +01:00
Paul Baltescu
30c3616a83 Fix compilation errors introduced by new oxlm changes. 2014-09-26 14:43:16 +01:00
Paul Baltescu
b6fa358427 Fix broken include. 2014-09-26 13:25:06 +01:00
Paul Baltescu
220c95ed9f Fix bugs in BilingualLM for chart based decoding. 2014-09-26 12:32:18 +01:00
Paul Baltescu
d162c47587 Fix a few bugs in BilingualLM for phrase based decoding. 2014-09-26 12:32:18 +01:00
Paul Baltescu
46b388885a Move BilingualLM under LM. 2014-09-26 12:32:17 +01:00
XapaJIaMnu
238e79e045 Remove unused variable that likely causes crashes 2014-09-22 13:43:55 +01:00
Philipp Koehn
33fbbc5faa allow specification of default scores for lexicalized reordering model 2014-09-21 07:43:44 +01:00
Barry Haddow
091948bff0 Improved debug 2014-09-18 17:03:19 +01:00
XapaJIaMnu
7c4301af09 fix vectors not being cleared 2014-09-16 11:24:50 +01:00
XapaJIaMnu
b0e0a6a148 Make it possible to have backend specific parameters. 2014-09-15 15:42:48 +01:00
Rico Sennrich
0861b464c5 use square brackets with output format '--brackets' (for cleaner escaping and consistency with decoder tree output) 2014-09-15 14:37:52 +01:00
XapaJIaMnu
920ce9ae1c Fix compilation 2014-09-13 09:27:46 +01:00
XapaJIaMnu
1d65a74d62 Some file renaming and fixes 2014-09-13 09:13:10 +01:00
XapaJIaMnu
9c41a01102 Abstract BilingualLM. Working 2014-09-13 08:54:58 +01:00
Matthias Huck
7e359a8892 Merge branch 'master' of github.com:moses-smt/mosesdecoder 2014-09-13 00:20:52 +01:00
Matthias Huck
5ac6c42508 PhraseOrientationFeature: bugfixes 2014-09-13 00:20:17 +01:00
XapaJIaMnu
e6bda71d2e Abstract BilingualLM to Binlm_NPLM. Doesn't compile 2014-09-12 19:01:10 +01:00
Rico Sennrich
61c00ed636 n-best tree output 2014-09-12 19:17:23 +02:00
Rico Sennrich
0a127cb0f1 refactoring 2014-09-12 19:17:23 +02:00
Matthias Huck
152ddb7513 PhraseOrientationFeature 2014-09-12 18:05:27 +01:00
Matthias Huck
0ddea954f2 PhraseOrientationFeature: TODOs 
(1) For efficiancy reasons, the initialization of the phrase orientation scoring object should be done as a precomputation step
(2) Handle degenerate cases with boundary non-terminals
 2014-09-12 17:50:02 +01:00
Matthias Huck
90e6ec8d11 PhraseOrientationFeature: remove debug output 2014-09-12 17:17:40 +01:00
XapaJIaMnu
c8f17dd6f2 Start abstracting NPLM from BilingualLM 2014-09-12 17:11:19 +01:00
Matthias Huck
85a441ae11 Jamfile update for PhraseOrientationFeature 2014-09-12 13:52:46 +01:00
Matthias Huck
1523f3315d PhraseOrientationFeature for chart-based decoding: a first simple version, 
with lots of log output
 2014-09-12 13:51:04 +01:00
XapaJIaMnu
0c6e593c49 Fix a bunch of off by one and silly errors 2014-09-12 10:23:46 +01:00
XapaJIaMnu
52a85e3f22 Fix silly mistake in getTargetPhraseChart 2014-09-10 18:27:44 +01:00
XapaJIaMnu
1235e5bc27 Fix a stupid mistake in TargetWords 2014-09-10 18:04:39 +01:00
Ulrich Germann
aff577748f Merge branch 'biased-bitext-sampling' of /fs/sif0/germann/git/mosesdecoder 2014-09-10 17:52:05 +01:00
Ulrich Germann
1d834e2b48 Fixed bug with respect to adding check option to Mmsapt::Load(). 2014-09-10 18:51:20 +02:00
Ulrich Germann
a58c7ceb18 Fixed issues with ambiguity in typedef of uint64_t (conflict between boost typedef and stdint typedef). 2014-09-10 12:07:57 +02:00
Matthias Huck
58842b8ffb PhraseOrientationFeature dummy. Does not do anything yet. 2014-09-09 18:17:05 +01:00
Ulrich Germann
31578d4915 Finished code for bias loading from Mmsapt config file. 2014-09-09 18:07:26 +01:00
Ulrich Germann
cda94c7d85 Fix in biased sampling. Started code on loading and using bias in Mmsapt. 2014-09-09 17:45:48 +01:00
XapaJIaMnu
e7afe5701e Add support for moses_chart. Cleaning up needed. 2014-09-09 17:17:51 +01:00
Ulrich Germann
f86fa65a6f Added utility count-ptable-features to count features in Mmsapt given a moses.ini config line. 2014-09-08 16:56:45 +01:00
Ulrich Germann
db6e5de641 Added initial code for utility to count features of PhraseDictionaryBitextSampling. 2014-09-08 11:03:05 +01:00
Ulrich Germann
5571ec91c6 Code cleanup. 2014-09-08 09:26:09 +01:00
Ulrich Germann
a86d49fc88 Added bias to bitext sampling. 2014-09-08 09:26:08 +01:00
Ulrich Germann
cef6460981 Initial check-in. 2014-09-08 09:26:08 +01:00
Ulrich Germann
a87a9ff207 Moved class PhrasePair back to ug_bitext. 
Moved function expand() from mmsapt.cc to ug_bitext.h.
Added new lookup function to class Bitext.
Bug fixes related to inverse lookup in class Bitext.
 2014-09-08 09:26:08 +01:00
Ulrich Germann
b588df77f0 Bug fix related to threading. 2014-09-08 09:26:08 +01:00
Ulrich Germann
2405293aaa Fiddling around with the code. Not for production. 2014-09-08 09:26:08 +01:00
Ulrich Germann
90c91ae9bb Added fakelib stringdist. 2014-09-08 09:26:08 +01:00
Ulrich Germann
9af3a61678 Added try-align2. 2014-09-08 09:26:08 +01:00
Ulrich Germann
a028fec7af Work in progress. 2014-09-08 09:26:08 +01:00
Ulrich Germann
25b733e35d Merge branch 'master' of https://github.com/moses-smt/mosesdecoder 2014-09-08 09:25:41 +01:00
Michael Denkowski
3304030a4e Merge branch 'master' of https://github.com/moses-smt/mosesdecoder 2014-09-04 11:19:32 -04:00
Michael Denkowski
6c33bc99dc Option to add TM-specific word and phrase counts 2014-09-04 11:17:42 -04:00
Michael Denkowski
756bcf0f15 Option to add TM-specific word and phrase counts 2014-09-04 01:49:26 -04:00
Hieu Hoang
02b11bfd6d compiles on gcc 4.6.3 2014-09-02 23:27:16 +01:00
Hieu Hoang
b6428f412b eclipse 2014-09-02 22:42:07 +01:00
XapaJIaMnu
6d39032a01 More work, source_ngrams missing 2014-09-02 17:05:46 +01:00
XapaJIaMnu
b5070d8d13 int -> size_t 2014-09-02 16:07:18 +01:00
XapaJIaMnu
a15aa1dd1f Rename variable to better represent what it means. Start syntactic decoding 2014-09-02 16:05:25 +01:00
Rico Sennrich
2a46e8ccea parse chart compression for faster CYK+ parsing with syntax systems. 2014-09-01 18:16:22 +01:00
XapaJIaMnu
dd43c287f6 Support for POS backoff in factored models. Spaghetti code needs refactoring 2014-09-01 16:04:49 +01:00
XapaJIaMnu
6df221966d GetNeuralLMId requries word instead of factor 2014-09-01 15:36:39 +01:00
mnadejde
8ba6853dcd Delete InternalStructStatelessFF.h 2014-09-01 13:58:11 +01:00
mnadejde
76cd3df393 Delete InternalStructStatelessFF.cpp 2014-09-01 13:57:56 +01:00
XapaJIaMnu
d747543916 Make some parameters of neuralLM optional 2014-09-01 13:57:08 +01:00
Ulrich Germann
ab35f6d4a3 Back to the standard default max phrase length of 20 with --with-mm. 2014-08-31 13:27:17 +01:00
XapaJIaMnu
b3ff8005f5 Get state, correct 2014-08-29 19:43:13 +01:00
XapaJIaMnu
310581100e State, almost correct or correct 2014-08-29 19:32:19 +01:00
XapaJIaMnu
b05a2b8b3a Fix small indexing error 2014-08-29 18:38:46 +01:00
Hieu Hoang
b76f0ef1b9 Merge branch 'bilingual-lm' of github.com:moses-smt/mosesdecoder into bilingual-lm 2014-08-29 17:05:03 +01:00