Commit Graph

16 Commits

Author SHA1 Message Date
chardmeier
837a667a95 Cleaned up language modelling code by disentangling the decoder's LM feature
function from the LM toolkit abstraction layer. There are two different groups
of classes now:
- LanguageModel, which inherits from StatefulFeatureFunction and contains
  the n-gram model feature function.
- LanguageModelImplementation, which is the base class of the individual
  LM implementations (SRI, IRST, RandLM, KenLM) and provides methods to
  query LM probabilities and states.
Each LanguageModel controls a LanguageModelImplementation. Implementations can
be shared by more than one LanguageModel.
This should make it easier to use the LM libraries as a backend for other
feature functions while retaining the flexibility to use different LM toolkits.


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3719 1f5c12ca-751b-0410-a591-d2e778427230
2010-11-17 14:06:21 +00:00
heafield
614d6002a6 Integrate heafield-refactorlm. Faster kenlm with new binary format. Stateful language model
framework.  



git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3671 1f5c12ca-751b-0410-a591-d2e778427230
2010-10-27 17:50:40 +00:00
bhaddow
904133fcb7 Merge in the multiple models branch. These changes allow the moses server
to support multiple translation, language and generation models within the
same process. The main design change is the introduction of a TranslationSystem
object to manage the models, which have been moved out of StaticData.
The changes should have no effect on existing systems.


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3394 1f5c12ca-751b-0410-a591-d2e778427230
2010-08-10 13:12:00 +00:00
jfouet
242846f07c Use portable include guard instead of pragma once
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2939 1f5c12ca-751b-0410-a591-d2e778427230
2010-02-24 11:15:44 +00:00
hieuhoang1972
928d771085 create namespace
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1897 1f5c12ca-751b-0410-a591-d2e778427230
2008-10-08 23:51:26 +00:00
hieuhoang1972
4f642808f1 move cube pruning moses lib to trunk
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1848 1f5c12ca-751b-0410-a591-d2e778427230
2008-06-11 10:52:57 +00:00
hieuhoang1972
6615fe0302 delete old moses lib
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1847 1f5c12ca-751b-0410-a591-d2e778427230
2008-06-11 10:52:01 +00:00
hieuhoang1972
c58393a4b4 verbose=0 nothing goes to stderr except for real, aborting errors
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1232 1f5c12ca-751b-0410-a591-d2e778427230
2007-02-22 23:44:38 +00:00
hieuhoang1972
7ecb0ce66e change unknown word processing to be closer to the way pharaoh does it - create unknown word whenever single word is not in translation table but penalise hypothesis for using it.
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1227 1f5c12ca-751b-0410-a591-d2e778427230
2007-02-21 19:48:53 +00:00
hieuhoang1972
f3cbacba3e code cleanup - make FactorCollection and StaticData totally accessible only globally
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1218 1f5c12ca-751b-0410-a591-d2e778427230
2007-02-16 18:08:37 +00:00
hieuhoang1972
a1f39c3ce7 lots of small changes and code clean up:
error catching/fail more gracefully on tables/lm load error & consistent user output
consistent debugging output
cleaned up timing functions
cleaned up mose/moses-cmd api calls/interaction
split up loading of all data in StaticData into separate fns
got binary phrase table to work under WIN32 & passed regression !!
added more comments
deleted phrase table filtering code
deleted mysql support
change calls to ToString() which might affect decoding to a call to a non-debugging fn instead, eg GetString()


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@988 1f5c12ca-751b-0410-a591-d2e778427230
2006-11-21 19:35:37 +00:00
hieuhoang1972
6b1c0837b0 added comments
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@907 1f5c12ca-751b-0410-a591-d2e778427230
2006-10-21 19:00:58 +00:00
redpony
441b147b35 Get rid of FactorArrayWrapper/FactorArray and use only Word. Memory pool is currently disabled, but the net cleanup resulted in better performance despite this (there are fewer copies of FactorArray -> Word now).
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@827 1f5c12ca-751b-0410-a591-d2e778427230
2006-09-25 21:35:10 +00:00
hieuhoang1972
acd36570df visual studio issues
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@759 1f5c12ca-751b-0410-a591-d2e778427230
2006-08-15 20:04:19 +00:00
redpony
2fc0a82c92 enable language model stats computation. enable with '-L true'
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@739 1f5c12ca-751b-0410-a591-d2e778427230
2006-08-14 22:51:56 +00:00
hieuhoang1972
93af8a2780 joint multi-factor LM
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@733 1f5c12ca-751b-0410-a591-d2e778427230
2006-08-14 21:30:19 +00:00