moses-smt/mosesdecoder
1
1
Fork 0
mirror of https://github.com/moses-smt/mosesdecoder.git synced 2025-01-05 19:23:13 +03:00
Code Issues Packages Projects Releases Wiki Activity
2,560 Commits 136 Branches 26 Tags 147 MiB
1ec58e7edf
Commit Graph

1210 Commits

Author SHA1 Message Date
bhaddow
d9d1c9440a Fix prehistoric memory leak. This was affecting the loading of 
alignment info from binarised phrase tables.


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3936 1f5c12ca-751b-0410-a591-d2e778427230
 2011-03-25 17:42:21 +00:00
hieuhoang1972
6cb1993541 xcode 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3932 1f5c12ca-751b-0410-a591-d2e778427230
 2011-03-21 06:32:06 +00:00
oliver-wilson
e168875599 Fix broken refactoring. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3931 1f5c12ca-751b-0410-a591-d2e778427230
 2011-03-18 15:58:12 +00:00
oliver-wilson
888e551aae Init thread specific data in Load() so the LM is useable in the main 
thread.


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3929 1f5c12ca-751b-0410-a591-d2e778427230
 2011-03-16 21:04:34 +00:00
hieuhoang1972
a4d22f7714 vs.net 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3928 1f5c12ca-751b-0410-a591-d2e778427230
 2011-03-11 19:26:01 +00:00
bhaddow
4a652200e4 Fix compile error in boost version 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3927 1f5c12ca-751b-0410-a591-d2e778427230
 2011-03-11 17:35:31 +00:00
hieuhoang1972
77c7a27f13 xcode 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3926 1f5c12ca-751b-0410-a591-d2e778427230
 2011-03-11 17:05:05 +00:00
phkoehn
ff8a624344 re-organized syntax code for clarity - part II 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3925 1f5c12ca-751b-0410-a591-d2e778427230
 2011-03-11 16:28:36 +00:00
phkoehn
1cbe245919 re-organized syntax code for clarity - part I 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3924 1f5c12ca-751b-0410-a591-d2e778427230
 2011-03-11 13:08:43 +00:00
pjwilliams
26d125d3cd documentation of chart decoder 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3923 1f5c12ca-751b-0410-a591-d2e778427230
 2011-03-10 12:58:49 +00:00
phkoehn
ad140593b0 documentation of chart decoder 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3920 1f5c12ca-751b-0410-a591-d2e778427230
 2011-03-09 17:55:13 +00:00
heafield
9d4d2a413e Have language models also return whether the word was OOV or not. This isn't used yet, but will make it into a feature. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3916 1f5c12ca-751b-0410-a591-d2e778427230
 2011-03-07 23:21:09 +00:00
bhaddow
3c9112bc74 remove useless debug 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3912 1f5c12ca-751b-0410-a591-d2e778427230
 2011-03-03 17:05:28 +00:00
phkoehn
4ee7e5f673 added some comments 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3911 1f5c12ca-751b-0410-a591-d2e778427230
 2011-03-02 19:02:07 +00:00
oliver-wilson
106c4e0fc2 Init thread specific data on RandLM before sentence processing. Needed 
for multithreaded RandLM.


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3910 1f5c12ca-751b-0410-a591-d2e778427230
 2011-03-02 16:07:20 +00:00
pjwilliams
75709a6c87 Memory efficiency: make the reserveSize argument non-optional in 
Moses::Phrase's constructor.  The default used to be ARRAY_SIZE_INCR = 10,
which will be excessive in many cases.  Where the default was used, I've
set the exact size where that was obvious and explicitly used ARRAY_SIZE_INCR 
otherwise.  If you know the code involved, it's probably worth reviewing.



git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3908 1f5c12ca-751b-0410-a591-d2e778427230
 2011-02-28 11:41:08 +00:00
leven101
a5a860ad1f optimized some parameters 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3907 1f5c12ca-751b-0410-a591-d2e778427230
 2011-02-25 14:53:34 +00:00
hieuhoang1972
251011df9e irstlm variable 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3906 1f5c12ca-751b-0410-a591-d2e778427230
 2011-02-25 02:26:05 +00:00
phkoehn
93ed0ca1a9 get rid of "unused variable" warnings 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3903 1f5c12ca-751b-0410-a591-d2e778427230
 2011-02-24 16:17:38 +00:00
hieuhoang1972
a0b6abdfd3 run beautify.perl. Consistent formatting for .h & .cpp files 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3901 1f5c12ca-751b-0410-a591-d2e778427230
 2011-02-24 13:14:42 +00:00
oliver-wilson
06df9d1770 Changed white space to test effect on svn blame. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3892 1f5c12ca-751b-0410-a591-d2e778427230
 2011-02-22 16:48:51 +00:00
bhaddow
e7a8ba3b7f check for correct number of factors 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3891 1f5c12ca-751b-0410-a591-d2e778427230
 2011-02-22 13:06:16 +00:00
maurocettolo
4c6dfbddc3 minor changes to make Moses compliant with IRSTLM toolkit (release 5.60.01) 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3878 1f5c12ca-751b-0410-a591-d2e778427230
 2011-02-11 11:32:35 +00:00
ales-t
e922c159b6 Alignment points are also created for unknown source words. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3876 1f5c12ca-751b-0410-a591-d2e778427230
 2011-02-08 18:04:09 +00:00
pjwilliams
d4359f9875 If Boost is available, use per-sentence object pools to allocate ProcessedRule 
and WordConsumed objects (which are used to store rule table lookup state).
Large numbers of these objects are used during decoding and this can
significantly improve performance, especially for multithreaded decoding,
though at the cost of increased total memory use.

The ./configure option --disable-boost-pool can be used to disable this
feature if memory is tight.  This currently only affects moses_chart with
in-memory rule tables.



git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3873 1f5c12ca-751b-0410-a591-d2e778427230
 2011-02-07 15:43:19 +00:00
bhaddow
6221d2a558 Patch to add covered to osgx from Dennis Mehay 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3872 1f5c12ca-751b-0410-a591-d2e778427230
 2011-02-04 16:21:57 +00:00
maurocettolo
8fcd76f2fc made handling of chunk LM compatible with recent efficiency updates to IRSTLM toolkit by Nicola 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3870 1f5c12ca-751b-0410-a591-d2e778427230
 2011-02-04 07:42:56 +00:00
phkoehn
4e72cd91be added decoding-graph-backoff, still experimenting with it 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3869 1f5c12ca-751b-0410-a591-d2e778427230
 2011-02-03 13:41:44 +00:00
bojar
0bc0ece594 Ales Tamchyna's printing of alignments (-print-alignment-info did nothing) 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3867 1f5c12ca-751b-0410-a591-d2e778427230
 2011-02-03 09:08:42 +00:00
hieuhoang1972
0eed5716b7 get rid of linked trans opt 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3864 1f5c12ca-751b-0410-a591-d2e778427230
 2011-02-02 11:24:19 +00:00
hieuhoang1972
e087e78df9 get rid of linked trans opt 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3863 1f5c12ca-751b-0410-a591-d2e778427230
 2011-02-02 11:06:19 +00:00
hieuhoang1972
abacb9166a xcode 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3857 1f5c12ca-751b-0410-a591-d2e778427230
 2011-01-28 14:57:55 +00:00
heafield
87f15593da Remove vestigial len parameter from language model calls 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3856 1f5c12ca-751b-0410-a591-d2e778427230
 2011-01-27 19:01:45 +00:00
pjwilliams
967b7be213 Support for multithreading in moses_chart (-threads option). This hasn't 
been thoroughly tested yet, so don't be surprised if it breaks.  Verbose
output will be scrambled.



git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3853 1f5c12ca-751b-0410-a591-d2e778427230
 2011-01-26 13:15:13 +00:00
redpony
eddb28e0ce facilitate programmatic creation of word lattices 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3848 1f5c12ca-751b-0410-a591-d2e778427230
 2011-01-25 20:08:29 +00:00
pjwilliams
8051c5ad35 Use TranslationTask objects to perform sentence decoding in moses-chart. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3846 1f5c12ca-751b-0410-a591-d2e778427230
 2011-01-25 17:15:30 +00:00
pjwilliams
99bbfe938b Use OutputCollector to write moses-chart output. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3843 1f5c12ca-751b-0410-a591-d2e778427230
 2011-01-25 15:17:17 +00:00
pjwilliams
67b30ea0c7 Move sentence-specific rule lookup state out of PhraseDictionarySCFG and 
PhraseDictionaryOnDisk and into ChartRuleLookupManager.



git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3842 1f5c12ca-751b-0410-a591-d2e778427230
 2011-01-24 19:14:19 +00:00
bhaddow
7b6503680a Shortcut when trans opts cache is size 0. Avoids potential uninitialised read. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3825 1f5c12ca-751b-0410-a591-d2e778427230
 2011-01-21 21:25:14 +00:00
hieuhoang1972
96bd3a164d vs.net 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3802 1f5c12ca-751b-0410-a591-d2e778427230
 2011-01-17 16:19:33 +00:00
bhaddow
a9cd71628a Change of boost macros - please make sure you favourite configuration still works 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3799 1f5c12ca-751b-0410-a591-d2e778427230
 2011-01-13 23:38:48 +00:00
pjwilliams
d20667a46d Faster lookup for rules with source and/or target syntax labels (in-memory rule 
table only).


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3797 1f5c12ca-751b-0410-a591-d2e778427230
 2011-01-13 00:25:10 +00:00
hieuhoang1972
41c5b3a1c2 xcode 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3769 1f5c12ca-751b-0410-a591-d2e778427230
 2010-12-10 13:04:06 +00:00
heafield
5e9df58a3c Respect -v 0 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3768 1f5c12ca-751b-0410-a591-d2e778427230
 2010-12-09 22:13:09 +00:00
bhaddow
50f0e6c07d Add a show-weights option. It prints out the moses features and exits. May 
load tables as a side-effect.


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3744 1f5c12ca-751b-0410-a591-d2e778427230
 2010-11-29 16:44:28 +00:00
bhaddow
6255216b6a Remove gnu-specific typeof 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3725 1f5c12ca-751b-0410-a591-d2e778427230
 2010-11-22 10:05:17 +00:00
bhaddow
a7e0977eea Fix compile error by using correct macro. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3720 1f5c12ca-751b-0410-a591-d2e778427230
 2010-11-18 10:27:30 +00:00
chardmeier
837a667a95 Cleaned up language modelling code by disentangling the decoder's LM feature 
function from the LM toolkit abstraction layer. There are two different groups
of classes now:
- LanguageModel, which inherits from StatefulFeatureFunction and contains
  the n-gram model feature function.
- LanguageModelImplementation, which is the base class of the individual
  LM implementations (SRI, IRST, RandLM, KenLM) and provides methods to
  query LM probabilities and states.
Each LanguageModel controls a LanguageModelImplementation. Implementations can
be shared by more than one LanguageModel.
This should make it easier to use the LM libraries as a backend for other
feature functions while retaining the flexibility to use different LM toolkits.


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3719 1f5c12ca-751b-0410-a591-d2e778427230
 2010-11-17 14:06:21 +00:00
chardmeier
d18ff948f5 Bugfixes in srilm adaptor. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3718 1f5c12ca-751b-0410-a591-d2e778427230
 2010-11-17 13:23:44 +00:00
leven101
84d83480b6 function name changes 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3711 1f5c12ca-751b-0410-a591-d2e778427230
 2010-11-15 11:32:02 +00:00
leven101
5251a2823a separated source and target vocab in suffixarrays to support unequal factors 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3710 1f5c12ca-751b-0410-a591-d2e778427230
 2010-11-15 11:28:27 +00:00
bojar
2ea140062b don't warn about probs outside [0,1] in -verbose 0 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3698 1f5c12ca-751b-0410-a591-d2e778427230
 2010-11-10 11:51:26 +00:00
heafield
2784923899 Rename a bunch of kenlm files. A ./regenerate-makefiles.sh is required. 
Make loading with MAP_POPULATE on Linux and read on other OSes the default.
Use LM #9 for lazy loading, as recommended by other devs.  
Slightly faster trie.  



git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3688 1f5c12ca-751b-0410-a591-d2e778427230
 2010-11-06 00:40:16 +00:00
leven101
34b45c0480 removed debug messages from BilingualDynSuffixArray.cpp 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3687 1f5c12ca-751b-0410-a591-d2e778427230
 2010-11-04 18:41:04 +00:00
bhaddow
3aee6fab5d Use correct conditional compilation flag for threaded moses 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3686 1f5c12ca-751b-0410-a591-d2e778427230
 2010-11-03 18:43:18 +00:00
heafield
c12c2c59d2 Autodetect model from binary format. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3675 1f5c12ca-751b-0410-a591-d2e778427230
 2010-10-28 01:05:04 +00:00
hieuhoang1972
735d5b682f xcode 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3673 1f5c12ca-751b-0410-a591-d2e778427230
 2010-10-27 18:54:50 +00:00
heafield
614d6002a6 Integrate heafield-refactorlm. Faster kenlm with new binary format. Stateful language model 
framework.  



git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3671 1f5c12ca-751b-0410-a591-d2e778427230
 2010-10-27 17:50:40 +00:00
nicolabertoldi
bb08dcb5b6 made code compliant with the enhanced IRSTLM library; IRSTLM release 5.50.01 is needed; back compatibility is not assured; 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3662 1f5c12ca-751b-0410-a591-d2e778427230
 2010-10-26 16:06:01 +00:00
nicolabertoldi
3d7f391ae5 remove inefficient function calls 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3638 1f5c12ca-751b-0410-a591-d2e778427230
 2010-10-22 13:30:50 +00:00
hieuhoang1972
564a9ad371 word alignment info in binary pt, code provided by Christof Pintaske 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3637 1f5c12ca-751b-0410-a591-d2e778427230
 2010-10-22 10:37:19 +00:00
hieuhoang1972
ac875ffc81 mem leak 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3617 1f5c12ca-751b-0410-a591-d2e778427230
 2010-10-12 14:47:58 +00:00
hieuhoang1972
620d92c68a bug in on-disk pt due to refactoring 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3616 1f5c12ca-751b-0410-a591-d2e778427230
 2010-10-12 14:10:19 +00:00
rsennrich
6e032b6ab5 XML spans parameter now uses '-' as token segmentation everywhere, as documented on the website. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3609 1f5c12ca-751b-0410-a591-d2e778427230
 2010-10-07 08:35:08 +00:00
pjwilliams
be8dbd20e9 Remove unused Phrase::m_arity member variable. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3600 1f5c12ca-751b-0410-a591-d2e778427230
 2010-10-01 10:37:10 +00:00
hieuhoang1972
e504b797b2 xcode 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3599 1f5c12ca-751b-0410-a591-d2e778427230
 2010-10-01 00:21:27 +00:00
pjwilliams
2761762434 If Boost is available, use boost::unordered_map instead of std::map for 
storing child nodes in PhraseDictionaryNodeSCFG.


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3598 1f5c12ca-751b-0410-a591-d2e778427230
 2010-09-30 21:28:30 +00:00
heafield
770df2a92d Unbodge kenlm by moving compilation to kenlm/ instead of kenlm/lm. Changing the headers every 
time I copied to Moses was getting annoying.  



git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3587 1f5c12ca-751b-0410-a591-d2e778427230
 2010-09-28 16:26:55 +00:00
heafield
d99066e4e9 Hieu wanted GetLmID 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3585 1f5c12ca-751b-0410-a591-d2e778427230
 2010-09-28 15:21:50 +00:00
hieuhoang1972
8fa18b50a7 xcode. And don't invoke internal LM when sri is specified, even if sri isn't compiled 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3583 1f5c12ca-751b-0410-a591-d2e778427230
 2010-09-28 11:02:58 +00:00
heafield
c2b49c9d97 Now returning the same probabilities as SRI. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3582 1f5c12ca-751b-0410-a591-d2e778427230
 2010-09-28 00:22:33 +00:00
heafield
331283f040 LanguageModelKen now without segfaults. The load function was missing some undocumented initialization. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3581 1f5c12ca-751b-0410-a591-d2e778427230
 2010-09-27 21:52:15 +00:00
pjwilliams
34767c4d0a Oops, don't prune rule table if ttable-limit=0. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3580 1f5c12ca-751b-0410-a591-d2e778427230
 2010-09-27 20:49:42 +00:00
heafield
5b74b38527 Remove vestigial dub parameter. Surrender to tab-based whitespace. More passive-aggressive message about nGramOrder. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3575 1f5c12ca-751b-0410-a591-d2e778427230
 2010-09-27 16:01:58 +00:00
heafield
cac7600662 Make --with-kenlm=$PWD/kenlm compile mine instead of IRST. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3574 1f5c12ca-751b-0410-a591-d2e778427230
 2010-09-27 15:07:28 +00:00
hieuhoang1972
375ed9b7fe ken lm integration 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3573 1f5c12ca-751b-0410-a591-d2e778427230
 2010-09-27 10:46:39 +00:00
heafield
e6184ae947 Updates to kenlm: 
Kludged and slow interface requested by Hieu because apparently Moses can't store language model state.  
Separate files for ARPA reading, vocabulary, and weights.  
Remove build shell scripts that won't work after Hieu changed the header file layout.  



git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3572 1f5c12ca-751b-0410-a591-d2e778427230
 2010-09-27 03:46:44 +00:00
hieuhoang1972
559a5e3ece ken lm integration 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3570 1f5c12ca-751b-0410-a591-d2e778427230
 2010-09-26 22:29:01 +00:00
pjwilliams
f5a096e2fa In ChartTranslationOptionList::Add(), always consider every target phrase as 
the TargetPhraseCollection won't be ordered.


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3567 1f5c12ca-751b-0410-a591-d2e778427230
 2010-09-26 12:09:29 +00:00
pjwilliams
81372704de Enforce ttable-limit for memory-based rule table. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3566 1f5c12ca-751b-0410-a591-d2e778427230
 2010-09-26 11:38:09 +00:00
hieuhoang1972
031617a8db delete unused function 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3563 1f5c12ca-751b-0410-a591-d2e778427230
 2010-09-25 20:30:19 +00:00
hieuhoang1972
ea04dc5209 vs.net 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3562 1f5c12ca-751b-0410-a591-d2e778427230
 2010-09-24 21:01:11 +00:00
hieuhoang1972
465d1cef0a make class name consistent for chart & pb decoders 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3561 1f5c12ca-751b-0410-a591-d2e778427230
 2010-09-23 17:39:32 +00:00
hieuhoang1972
4d34e5155e make class name consistent for chart & pb decoders 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3560 1f5c12ca-751b-0410-a591-d2e778427230
 2010-09-23 16:54:32 +00:00
hieuhoang1972
0b6a29327c make class name consistent for chart & pb decoders 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3559 1f5c12ca-751b-0410-a591-d2e778427230
 2010-09-23 16:32:44 +00:00
hieuhoang1972
cafc5046c7 make class name consistent for chart & pb decoders 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3558 1f5c12ca-751b-0410-a591-d2e778427230
 2010-09-23 15:48:33 +00:00
hieuhoang1972
73ece4f859 make class name consistent for chart & pb decoders 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3557 1f5c12ca-751b-0410-a591-d2e778427230
 2010-09-23 15:18:11 +00:00
hieuhoang1972
1920fc007c makefile for ken lm 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3556 1f5c12ca-751b-0410-a591-d2e778427230
 2010-09-23 14:37:06 +00:00
hieuhoang1972
af228d7e93 make class name consistent for chart & pb decoders 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3555 1f5c12ca-751b-0410-a591-d2e778427230
 2010-09-23 12:34:35 +00:00
hieuhoang1972
34873a1ba4 make class name consistent for chart & pb decoders 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3554 1f5c12ca-751b-0410-a591-d2e778427230
 2010-09-23 12:12:49 +00:00
hieuhoang1972
b83fc49a1f make class name consistent for chart & pb decoders 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3553 1f5c12ca-751b-0410-a591-d2e778427230
 2010-09-23 11:05:38 +00:00
hieuhoang1972
989eb55449 ken lm integration 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3544 1f5c12ca-751b-0410-a591-d2e778427230
 2010-09-21 23:10:15 +00:00
hieuhoang1972
32d3565b04 ken lm integration 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3543 1f5c12ca-751b-0410-a591-d2e778427230
 2010-09-21 22:43:29 +00:00
hieuhoang1972
e9b355ed8c get rid of weights in translation model 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3471 1f5c12ca-751b-0410-a591-d2e778427230
 2010-09-15 16:49:49 +00:00
hieuhoang1972
d0aecca9fe get rid of weight in LM class 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3466 1f5c12ca-751b-0410-a591-d2e778427230
 2010-09-15 16:02:41 +00:00
pjwilliams
130ae457dd In PhraseDictionarySCFG, only add non-leaf nodes to the running node lists. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3424 1f5c12ca-751b-0410-a591-d2e778427230
 2010-09-10 15:19:25 +00:00
hieuhoang1972
bdfa2beab8 delete eclipse build 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3422 1f5c12ca-751b-0410-a591-d2e778427230
 2010-09-10 09:16:50 +00:00
bhaddow
d3680b4255 Make sure distortion feature are adjacent 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3418 1f5c12ca-751b-0410-a591-d2e778427230
 2010-09-07 10:54:04 +00:00
pjwilliams
2deb68af84 In PhraseDictionaryNodeSCFG, use separate maps for children with terminal and non-terminal keys. This removes the need to look up source terminals twice. The chart decoder spends a *lot* of time in PhraseDictionaryNodeSCFG::GetChild() (approx 38% of post-startup decoding time in my target syntax test, according to callgrind), so this makes a significant difference. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3409 1f5c12ca-751b-0410-a591-d2e778427230
 2010-08-17 13:41:46 +00:00
pjwilliams
98383c1393 Pare down PhraseDictionaryNodeSCFG. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3408 1f5c12ca-751b-0410-a591-d2e778427230
 2010-08-17 11:01:03 +00:00
hieuhoang1972
ef09298824 move function calls with side effects out of asserts 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3403 1f5c12ca-751b-0410-a591-d2e778427230
 2010-08-12 12:54:55 +00:00
hieuhoang1972
8616a2bdee visual studio 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3396 1f5c12ca-751b-0410-a591-d2e778427230
 2010-08-10 13:51:20 +00:00
hieuhoang1972
8fc72ee74a xcode 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3395 1f5c12ca-751b-0410-a591-d2e778427230
 2010-08-10 13:35:08 +00:00
bhaddow
904133fcb7 Merge in the multiple models branch. These changes allow the moses server 
to support multiple translation, language and generation models within the
same process. The main design change is the introduction of a TranslationSystem
object to manage the models, which have been moved out of StaticData.
The changes should have no effect on existing systems.


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3394 1f5c12ca-751b-0410-a591-d2e778427230
 2010-08-10 13:12:00 +00:00
hieuhoang1972
277ba483e1 alignment info in the decoder 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3381 1f5c12ca-751b-0410-a591-d2e778427230
 2010-08-02 16:32:36 +00:00
phkoehn
56447ed42c bug fix for nested zones 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3379 1f5c12ca-751b-0410-a591-d2e778427230
 2010-07-30 22:17:08 +00:00
hieuhoang1972
7221bf2dd4 alignment info, for chart decoding, updated regression 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3368 1f5c12ca-751b-0410-a591-d2e778427230
 2010-07-28 09:53:21 +00:00
hieuhoang1972
7c6007f018 alignment info, new format 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3365 1f5c12ca-751b-0410-a591-d2e778427230
 2010-07-27 11:21:12 +00:00
hieuhoang1972
fc56e031d4 alignment info, new format 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3364 1f5c12ca-751b-0410-a591-d2e778427230
 2010-07-27 11:10:13 +00:00
hieuhoang1972
b9339bdf0e svn properties 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3360 1f5c12ca-751b-0410-a591-d2e778427230
 2010-07-17 23:23:09 +00:00
hieuhoang1972
dd7d3d1b56 vs.net 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3359 1f5c12ca-751b-0410-a591-d2e778427230
 2010-07-17 23:11:55 +00:00
hieuhoang1972
31930eb6fc alignment info in pt 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3358 1f5c12ca-751b-0410-a591-d2e778427230
 2010-07-17 22:29:06 +00:00
hieuhoang1972
a21c9bff68 debug output 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3334 1f5c12ca-751b-0410-a591-d2e778427230
 2010-06-13 21:06:34 +00:00
hieuhoang1972
f24fb6449e delete pragma once when using #define 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3324 1f5c12ca-751b-0410-a591-d2e778427230
 2010-06-10 15:36:08 +00:00
nicolabertoldi
6e67edd11f sorted lif of headers and sources 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3323 1f5c12ca-751b-0410-a591-d2e778427230
 2010-06-10 15:32:33 +00:00
nicolabertoldi
e6d39bf83a minor fix 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3322 1f5c12ca-751b-0410-a591-d2e778427230
 2010-06-10 15:19:07 +00:00
nicolabertoldi
79d91a572d memory-unmap moved into the destructor 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3316 1f5c12ca-751b-0410-a591-d2e778427230
 2010-06-09 14:34:15 +00:00
nicolabertoldi
f38d220b67 moving compilation of LanguageModelParallelBackoff in the SRILM-dependent region 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3315 1f5c12ca-751b-0410-a591-d2e778427230
 2010-06-09 14:08:55 +00:00
hieuhoang1972
7db72471ce xcode 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3313 1f5c12ca-751b-0410-a591-d2e778427230
 2010-06-09 13:00:54 +00:00
hieuhoang1972
0d6d78b5a9 xcode 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3308 1f5c12ca-751b-0410-a591-d2e778427230
 2010-06-08 11:42:28 +00:00
hieuhoang1972
c620d1a5f7 xcode 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3307 1f5c12ca-751b-0410-a591-d2e778427230
 2010-06-08 01:26:34 +00:00
hieuhoang1972
fdfd4907f5 xcode 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3306 1f5c12ca-751b-0410-a591-d2e778427230
 2010-06-08 00:59:48 +00:00
hieuhoang1972
dee9be66f8 don't check for files exists in Parameter class. May be a folder for OnDisk format 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3304 1f5c12ca-751b-0410-a591-d2e778427230
 2010-06-07 17:19:51 +00:00
hieuhoang1972
c9eb463fe6 vs.net 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3301 1f5c12ca-751b-0410-a591-d2e778427230
 2010-06-05 13:10:26 +00:00
leven101
69d92bfec3 git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3281 1f5c12ca-751b-0410-a591-d2e778427230 2010-05-27 18:53:39 +00:00
leven101
da9814fcf9 fixed bug 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3278 1f5c12ca-751b-0410-a591-d2e778427230
 2010-05-27 15:44:23 +00:00
bhaddow
f62d9a9be1 Remove trailing space on GetStringRep() 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3273 1f5c12ca-751b-0410-a591-d2e778427230
 2010-05-27 08:14:21 +00:00
leven101
4338889c71 removed debug statments 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3270 1f5c12ca-751b-0410-a591-d2e778427230
 2010-05-25 13:11:42 +00:00
leven101
142a05c64d suffix array operates on single thread. bugs in dynamic suffix array 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3269 1f5c12ca-751b-0410-a591-d2e778427230
 2010-05-25 13:06:56 +00:00
bhaddow
4874f711ca Update mosesserver so that it can return the search graph and/or 
translation options. Also add a sample client to demonstrate
these new features.


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3265 1f5c12ca-751b-0410-a591-d2e778427230
 2010-05-20 16:52:46 +00:00
bhaddow
c9baabe2ea Merging of moses mains. mosesmt now does single and multi-threeaded 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3264 1f5c12ca-751b-0410-a591-d2e778427230
 2010-05-19 16:42:18 +00:00
leven101
c2f35f614b fixed bug 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3263 1f5c12ca-751b-0410-a591-d2e778427230
 2010-05-19 16:37:57 +00:00
chardmeier
531598684b Removed PhraseDictionaryTree's dependency on StaticData to facilitate using the object outside the decoder. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3258 1f5c12ca-751b-0410-a591-d2e778427230
 2010-05-16 19:13:32 +00:00
bhaddow
bb2431331c Fix to consensus decoding to stop it discarding hypos with zero score. 
Move hypothesis id management into Manager to make it thread-safe.


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3257 1f5c12ca-751b-0410-a591-d2e778427230
 2010-05-14 09:33:03 +00:00
leven101
ff6d734b60 small fix for regex use 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3256 1f5c12ca-751b-0410-a591-d2e778427230
 2010-05-13 16:05:10 +00:00
leven101
f0e8f86501 fixed double free bug 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3255 1f5c12ca-751b-0410-a591-d2e778427230
 2010-05-13 16:03:56 +00:00
chardmeier
6f8d9fa0fb Added limited backwards compatibility with old ini file format for phrase table specs. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3253 1f5c12ca-751b-0410-a591-d2e778427230
 2010-05-13 14:12:05 +00:00
hieuhoang1972
f36759478b don't spew out param list if there's a genuine error 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3250 1f5c12ca-751b-0410-a591-d2e778427230
 2010-05-11 21:44:35 +00:00
phkoehn
f52a2cc55d code cleanup & enable trace ("-t") for MBR decoding ("-mbr") 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3246 1f5c12ca-751b-0410-a591-d2e778427230
 2010-05-10 21:18:47 +00:00
pjwilliams
2a5b262503 Write translation details to a named file instead of cerr. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3243 1f5c12ca-751b-0410-a591-d2e778427230
 2010-05-08 15:51:59 +00:00
leven101
ed0a9a369b git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3241 1f5c12ca-751b-0410-a591-d2e778427230 2010-05-07 21:25:53 +00:00
hieuhoang1972
e67670ad5b add alignment info for -T output 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3240 1f5c12ca-751b-0410-a591-d2e778427230
 2010-05-07 19:33:39 +00:00
pjwilliams
01f4ca892a Remove the lmstats parameter since it doesn't have any effect except as 
a synonym for translation-details.


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3237 1f5c12ca-751b-0410-a591-d2e778427230
 2010-05-07 17:38:20 +00:00
leven101
f4696695f8 git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3236 1f5c12ca-751b-0410-a591-d2e778427230 2010-05-07 16:55:48 +00:00
leven101
ad554c7d6d git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3235 1f5c12ca-751b-0410-a591-d2e778427230 2010-05-07 16:54:25 +00:00
leven101
e1ea1e8301 added suffix array phrase table updater service to moses server 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3232 1f5c12ca-751b-0410-a591-d2e778427230
 2010-05-07 09:50:19 +00:00
hieuhoang1972
034528988d better -T output for chart decoding 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3231 1f5c12ca-751b-0410-a591-d2e778427230
 2010-05-06 19:46:23 +00:00
hieuhoang1972
44e2247086 minor 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3229 1f5c12ca-751b-0410-a591-d2e778427230
 2010-05-06 11:35:25 +00:00
hieuhoang1972
b6a92ba8af delete param object after decoding 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3228 1f5c12ca-751b-0410-a591-d2e778427230
 2010-05-06 11:26:52 +00:00
hieuhoang1972
a6b7807b2e less debugging info from OnDisk pt 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3227 1f5c12ca-751b-0410-a591-d2e778427230
 2010-05-06 11:19:53 +00:00
bhaddow
ffea53717e Refactoring of output search graph code to enable it to be used 
from moses server.


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3226 1f5c12ca-751b-0410-a591-d2e778427230
 2010-05-05 16:53:41 +00:00
bojar
0118733a45 fixing traditional distortion penalty with -continue-partial-translation 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3222 1f5c12ca-751b-0410-a591-d2e778427230
 2010-05-03 21:41:28 +00:00
bojar
0a21c35fef fixed to initialize m_currSourceWordsRange correctly with -continue-partial-translation 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3221 1f5c12ca-751b-0410-a591-d2e778427230
 2010-05-03 21:41:00 +00:00
bojar
65eb118d62 fixed Ondrej Odchazel's code to include partial output in LM scoring, renamed to match style 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3219 1f5c12ca-751b-0410-a591-d2e778427230
 2010-05-03 21:40:17 +00:00
bojar
c9ba824c70 applying Ondrej Odchazel's patch 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3218 1f5c12ca-751b-0410-a591-d2e778427230
 2010-05-03 21:39:43 +00:00
bojar
30921793a4 Ondrej Odchazel's update for AJAX-CAT 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3217 1f5c12ca-751b-0410-a591-d2e778427230
 2010-05-03 21:39:23 +00:00
hieuhoang1972
2bf009008b move sentence-based cache cleaning into IRST LM 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3203 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-26 14:56:06 +00:00
nicolabertoldi
1c615ea751 re-introduced the initialization and cleaning up into the constructor and destructor of Managers object. And I moved the check for LM cache cleanup into LanguageModelIRST 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3202 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-26 13:38:56 +00:00
hieuhoang1972
1a638746f3 SRI compile error in Ondrej's parallel backoff LM 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3201 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-26 12:02:28 +00:00
nicolabertoldi
cf468273bc Safer way to cleanup LM caches after a given number of processed sentences. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3197 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-24 00:06:49 +00:00
nicolabertoldi
476528c05f I added a new parameter (-clean-lm-cache <value>) in order to control how often the LMs clean up their caches (if available). 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3195 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-23 15:01:06 +00:00
chardmeier
ca353d3e68 - word lattice distortion calculation and distortion limit checks revisited 
- crash in a controlled way if a jump in a word lattice exceeds max-phrase-length
- check if a sufficient number of ttable-limits is specified in the configuration

Thanks to Arianna Bisazza for help with sorting out lattice distortion and
to Suzy Howlett for help with debugging the other issues.


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3193 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-22 08:42:32 +00:00
hieuhoang1972
44368da77d xcode 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3153 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-21 11:27:45 +00:00
bojar
34a0e9b3a2 support for SRILM's factored language models, implemented by Michal Richter 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3147 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-20 15:25:52 +00:00
bojar
323d48a821 factored version of PhraseDictionaryDynSuffixArray; squashed cmt of following: 
commit b8a86018e1100efede28023c1b351a4e27da9b75
Author: Kamil Kos <kamil.kos@matfyz.cz>
Date:   Tue Apr 20 14:28:40 2010 +0200

    added comments and  minor bug fixes

    Conflicts:

    	moses/src/BilingualDynSuffixArray.cpp
    	moses/src/PhraseDictionary.cpp
    	moses/src/PhraseDictionaryDynSuffixArray.h

commit 7d09ec19bb49f516a98bf6e6e2cd86c19d428e73
Author: Kamil Kos <kamil.kos@matfyz.cz>
Date:   Thu Apr 15 18:29:08 2010 +0200

    factored version of PhraseDictionaryDynSuffixArray


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3146 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-20 14:09:53 +00:00
hieuhoang1972
0e01ebaadb lex reordering bug 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3142 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-19 17:26:54 +00:00
bhaddow
b35ab24bb8 Fix mosesmt phrase table loading (it got squashed in the chart merge) 
Fix some of the unused warnings.


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3138 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-19 13:52:56 +00:00
hieuhoang1972
b96bac6106 vs studio 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3124 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-12 13:50:11 +00:00
hieuhoang1972
3029aaccc3 delete using std namespace in .h files 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3123 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-12 12:29:37 +00:00
hieuhoang1972
f0933ad883 LGPL license 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3117 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-12 10:15:49 +00:00
hieuhoang1972
45ecd333f9 delete std namespace in .h files 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3116 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-12 10:04:35 +00:00
hieuhoang1972
49d3780bcc re-merge lattice mbr & consensus decoding 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3115 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-12 09:51:29 +00:00
leven101
8839474d3d changed assertion on ttable entry in moses.ini 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3081 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-08 20:50:22 +00:00
leven101
ce4192d2d6 fixed bug 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3080 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-08 18:57:35 +00:00
hieuhoang1972
5bab778f02 merge 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3079 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-08 17:57:38 +00:00
hieuhoang1972
c117ef7c17 Copy in changes from the chart_merge branch (doing it manually because the 
server doesn't seem to support subversion's --reintegrate option).


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3078 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-08 17:16:10 +00:00
bojar
5f1fd96111 Merge branch 'bilingualSA' into moses-svn 
Conflicts:

	moses/src/DynSAInclude/file.h


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3074 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-08 14:52:35 +00:00
bhaddow
521d50fe63 Implementation of consensus decoding - first cut. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3071 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-07 15:47:58 +00:00
bhaddow
639c8e5187 Fix compile errors in dynamic suffix array code 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3065 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-07 11:02:04 +00:00
leven101
e894097edf added dependencies to suffix array classes 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3062 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-07 10:21:48 +00:00
leven101
a47e6b7bee incorporate suffix array classes (not final solution) 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3053 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-06 11:37:50 +00:00
hieuhoang1972
202bcf2911 eclipse build 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3015 1f5c12ca-751b-0410-a591-d2e778427230
 2010-03-31 20:03:54 +00:00
bhaddow
742355266d Fix leak which was affecting mbr/lmbr 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3013 1f5c12ca-751b-0410-a591-d2e778427230
 2010-03-31 13:16:39 +00:00
hieuhoang1972
853c443375 delete old lex reordering code 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3010 1f5c12ca-751b-0410-a591-d2e778427230
 2010-03-30 22:48:26 +00:00
hieuhoang1972
4bb021d0ce ide project files 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3008 1f5c12ca-751b-0410-a591-d2e778427230
 2010-03-30 10:51:12 +00:00
bhaddow
5f734d3b9f Merged r2670-3001 from hierarchical-reo branch 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3002 1f5c12ca-751b-0410-a591-d2e778427230
 2010-03-25 11:43:18 +00:00
bhaddow
795224736b Merge revisions 2670-2988 from track. Passes all regression except lexicalised 
reordering


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/branches/hierarchical-reo@2989 1f5c12ca-751b-0410-a591-d2e778427230
 2010-03-19 17:52:51 +00:00
bhaddow
ee2ae991e5 Roll-back to non-reproducible, but transitive Compare operation 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/branches/hierarchical-reo@2988 1f5c12ca-751b-0410-a591-d2e778427230
 2010-03-19 16:59:08 +00:00
bhaddow
22b74b56f7 Output of nbest from lattice mbr 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2980 1f5c12ca-751b-0410-a591-d2e778427230
 2010-03-16 11:34:50 +00:00
bhaddow
0208bddb70 Added parameters for lmbr n-best 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2971 1f5c12ca-751b-0410-a591-d2e778427230
 2010-03-14 20:23:17 +00:00
abarun
154b01a2ef More efficient implementation of print all derivations 2 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2958 1f5c12ca-751b-0410-a591-d2e778427230
 2010-03-08 16:30:31 +00:00
abarun
e09be2bfd5 More efficient implementation of print all derivations 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2957 1f5c12ca-751b-0410-a591-d2e778427230
 2010-03-08 15:28:40 +00:00
abarun
86ef292af1 Writing out sentence number when printing all derivations 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2954 1f5c12ca-751b-0410-a591-d2e778427230
 2010-03-07 08:28:16 +00:00
abarun
3434d15cda Lattice MBR pruning based on target length of MAP hypothesis 
Added option (-disable-discarding) to disable discarding of hypos during search
Added option (-print-all-derivations) to output (in unsorted order) ALL derivations found in search graph 


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2953 1f5c12ca-751b-0410-a591-d2e778427230
 2010-03-07 07:57:48 +00:00
jfouet
242846f07c Use portable include guard instead of pragma once 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2939 1f5c12ca-751b-0410-a591-d2e778427230
 2010-02-24 11:15:44 +00:00
jfouet
e14dc33dc8 enable trace with a configure options and fix compilation error when trace are disabled 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2938 1f5c12ca-751b-0410-a591-d2e778427230
 2010-02-24 10:37:49 +00:00
jfouet
087b2021c5 Link libmoses with boost-thread 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2936 1f5c12ca-751b-0410-a591-d2e778427230
 2010-02-24 10:34:00 +00:00
jfouet
5403d34faf pragma only valid with visual studio 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2935 1f5c12ca-751b-0410-a591-d2e778427230
 2010-02-24 10:30:24 +00:00
jfouet
133f18bd90 Use stringstream operation instead of atoi and atof to prevent error and overflow 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2934 1f5c12ca-751b-0410-a591-d2e778427230
 2010-02-24 10:03:22 +00:00
jfouet
50c22f1e42 Fix seg fault when libmoses is used ad a dynamic library on windows, MagicWord was initialized to a wrong value 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2933 1f5c12ca-751b-0410-a591-d2e778427230
 2010-02-24 10:02:17 +00:00