moses-smt/mosesdecoder
1
1
Fork 0
mirror of https://github.com/moses-smt/mosesdecoder.git synced 2024-09-20 07:42:21 +03:00
Code Issues Packages Projects Releases Wiki Activity
2,144 Commits 136 Branches 26 Tags 147 MiB
1cbe245919
Commit Graph

2144 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
phkoehn
1cbe245919 re-organized syntax code for clarity - part I 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3924 1f5c12ca-751b-0410-a591-d2e778427230
 2011-03-11 13:08:43 +00:00
pjwilliams
26d125d3cd documentation of chart decoder 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3923 1f5c12ca-751b-0410-a591-d2e778427230
 2011-03-10 12:58:49 +00:00
hieuhoang1972
c93569c906 #include according to Ceslav 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3922 1f5c12ca-751b-0410-a591-d2e778427230
 2011-03-10 07:10:48 +00:00
phkoehn
aead480bea documentation of chart decoder 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3921 1f5c12ca-751b-0410-a591-d2e778427230
 2011-03-09 17:55:27 +00:00
phkoehn
ad140593b0 documentation of chart decoder 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3920 1f5c12ca-751b-0410-a591-d2e778427230
 2011-03-09 17:55:13 +00:00
bojar
65048a3714 zero jobs means serial 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3919 1f5c12ca-751b-0410-a591-d2e778427230
 2011-03-09 10:43:34 +00:00
hieuhoang1972
1e76baa978 #include for Ubuntu build 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3918 1f5c12ca-751b-0410-a591-d2e778427230
 2011-03-08 15:45:03 +00:00
bojar
4bb2cd5994 support for --alignment-output-file (pass it to moses and later concatenate 
outputs)


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3917 1f5c12ca-751b-0410-a591-d2e778427230
 2011-03-08 07:02:49 +00:00
heafield
9d4d2a413e Have language models also return whether the word was OOV or not. This isn't used yet, but will make it into a feature. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3916 1f5c12ca-751b-0410-a591-d2e778427230
 2011-03-07 23:21:09 +00:00
heafield
98d4d36a49 Fix compiler warning 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3915 1f5c12ca-751b-0410-a591-d2e778427230
 2011-03-07 22:51:25 +00:00
hieuhoang1972
2880656d8d option of outputting scoring to stdout 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3914 1f5c12ca-751b-0410-a591-d2e778427230
 2011-03-07 02:44:34 +00:00
hieuhoang1972
cd384a1fbc option of outputting scoring to stdout 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3913 1f5c12ca-751b-0410-a591-d2e778427230
 2011-03-05 15:38:50 +00:00
bhaddow
3c9112bc74 remove useless debug 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3912 1f5c12ca-751b-0410-a591-d2e778427230
 2011-03-03 17:05:28 +00:00
phkoehn
4ee7e5f673 added some comments 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3911 1f5c12ca-751b-0410-a591-d2e778427230
 2011-03-02 19:02:07 +00:00
oliver-wilson
106c4e0fc2 Init thread specific data on RandLM before sentence processing. Needed 
for multithreaded RandLM.


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3910 1f5c12ca-751b-0410-a591-d2e778427230
 2011-03-02 16:07:20 +00:00
heafield
b2b2688a74 Make kenlm compile with icc by changing exception handling 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3909 1f5c12ca-751b-0410-a591-d2e778427230
 2011-03-01 17:29:37 +00:00
pjwilliams
75709a6c87 Memory efficiency: make the reserveSize argument non-optional in 
Moses::Phrase's constructor.  The default used to be ARRAY_SIZE_INCR = 10,
which will be excessive in many cases.  Where the default was used, I've
set the exact size where that was obvious and explicitly used ARRAY_SIZE_INCR 
otherwise.  If you know the code involved, it's probably worth reviewing.



git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3908 1f5c12ca-751b-0410-a591-d2e778427230
 2011-02-28 11:41:08 +00:00
leven101
a5a860ad1f optimized some parameters 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3907 1f5c12ca-751b-0410-a591-d2e778427230
 2011-02-25 14:53:34 +00:00
hieuhoang1972
251011df9e irstlm variable 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3906 1f5c12ca-751b-0410-a591-d2e778427230
 2011-02-25 02:26:05 +00:00
heafield
7b5d0234c6 More error handling: 
<s> and </s> throw up is optional, but default.  
If a binary file makes it to the ARPA parser (somebody gzipped a binary file or passed it build binary), the message is more informative.  



git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3905 1f5c12ca-751b-0410-a591-d2e778427230
 2011-02-24 19:37:39 +00:00
heafield
6f63bb4161 Prevent people from loading partially built binary files. Partially build files made with the old build_binary will still load, but any partial files with the new build_binary (this revision) will throw an error on load. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3904 1f5c12ca-751b-0410-a591-d2e778427230
 2011-02-24 17:11:53 +00:00
phkoehn
93ed0ca1a9 get rid of "unused variable" warnings 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3903 1f5c12ca-751b-0410-a591-d2e778427230
 2011-02-24 16:17:38 +00:00
hieuhoang1972
a3d97584a9 run beautify.perl. Consistent formatting for .h & .cpp files 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3902 1f5c12ca-751b-0410-a591-d2e778427230
 2011-02-24 13:57:11 +00:00
hieuhoang1972
a0b6abdfd3 run beautify.perl. Consistent formatting for .h & .cpp files 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3901 1f5c12ca-751b-0410-a591-d2e778427230
 2011-02-24 13:14:42 +00:00
hieuhoang1972
2572046803 run beautify.perl. Consistent formatting for .h & .cpp files 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3900 1f5c12ca-751b-0410-a591-d2e778427230
 2011-02-24 12:50:06 +00:00
hieuhoang1972
148c1e8305 run beautify.perl. Consistent formatting for .h & .cpp files 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3899 1f5c12ca-751b-0410-a591-d2e778427230
 2011-02-24 12:42:19 +00:00
hieuhoang1972
a59ad11b58 run beautify.perl. Consistent formatting for .h & .cpp files 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3898 1f5c12ca-751b-0410-a591-d2e778427230
 2011-02-24 12:40:21 +00:00
hieuhoang1972
67dd80fb7b run beautify.perl. Consistent formatting for .h & .cpp files 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3897 1f5c12ca-751b-0410-a591-d2e778427230
 2011-02-24 12:39:29 +00:00
hieuhoang1972
508d89eda8 run beautify.perl. Consistent formatting for .h & .cpp files 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3896 1f5c12ca-751b-0410-a591-d2e778427230
 2011-02-24 12:36:50 +00:00
heafield
5f0eacce4b Apparently some systems (including those at IRST) don't print exceptions that work their way up 
to main.  Do this.  


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3895 1f5c12ca-751b-0410-a591-d2e778427230
 2011-02-23 17:59:56 +00:00
hieuhoang1972
4eb32d3f76 avoid mangling *.hh in kenlm 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3894 1f5c12ca-751b-0410-a591-d2e778427230
 2011-02-23 15:28:10 +00:00
phkoehn
4c11bcd617 extensions to phrase table scoring options 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3893 1f5c12ca-751b-0410-a591-d2e778427230
 2011-02-23 10:27:54 +00:00
oliver-wilson
06df9d1770 Changed white space to test effect on svn blame. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3892 1f5c12ca-751b-0410-a591-d2e778427230
 2011-02-22 16:48:51 +00:00
bhaddow
e7a8ba3b7f check for correct number of factors 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3891 1f5c12ca-751b-0410-a591-d2e778427230
 2011-02-22 13:06:16 +00:00
heafield
cb848f41b3 Fix corner case in trie builder context merging 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3890 1f5c12ca-751b-0410-a591-d2e778427230
 2011-02-21 17:15:24 +00:00
hieuhoang1972
948d916ca0 xcode 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3889 1f5c12ca-751b-0410-a591-d2e778427230
 2011-02-20 07:28:01 +00:00
bhaddow
c86e6b38b3 add new nonbreaking prefixes 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3884 1f5c12ca-751b-0410-a591-d2e778427230
 2011-02-17 21:51:17 +00:00
bhaddow
6b8415bffb Write alignment info through OutputCollector so it gets ordered 
correctly when run with multiple threads.


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3882 1f5c12ca-751b-0410-a591-d2e778427230
 2011-02-16 16:50:55 +00:00
phkoehn
df901e7ce6 added files from Tom Hoar 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3881 1f5c12ca-751b-0410-a591-d2e778427230
 2011-02-16 10:44:26 +00:00
bojar
76174ccd4b mark web/bin/detokenizer.perl as outdated 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3880 1f5c12ca-751b-0410-a591-d2e778427230
 2011-02-14 13:35:04 +00:00
bojar
26ccace946 Czech detokenization 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3879 1f5c12ca-751b-0410-a591-d2e778427230
 2011-02-14 13:32:41 +00:00
maurocettolo
4c6dfbddc3 minor changes to make Moses compliant with IRSTLM toolkit (release 5.60.01) 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3878 1f5c12ca-751b-0410-a591-d2e778427230
 2011-02-11 11:32:35 +00:00
heafield
fb02a67afb Fix segfaults (or at least one of them) 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3877 1f5c12ca-751b-0410-a591-d2e778427230
 2011-02-11 01:51:30 +00:00
ales-t
e922c159b6 Alignment points are also created for unknown source words. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3876 1f5c12ca-751b-0410-a591-d2e778427230
 2011-02-08 18:04:09 +00:00
ales-t
83e2406f42 Word alignment output also works with MBR decoding. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3875 1f5c12ca-751b-0410-a591-d2e778427230
 2011-02-08 17:15:50 +00:00
bhaddow
47df5fd51c Add triples with default values if insufficient number are supplied. Note 
that min and max are no longer used, and should be removed at some point.


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3874 1f5c12ca-751b-0410-a591-d2e778427230
 2011-02-08 16:26:17 +00:00
pjwilliams
d4359f9875 If Boost is available, use per-sentence object pools to allocate ProcessedRule 
and WordConsumed objects (which are used to store rule table lookup state).
Large numbers of these objects are used during decoding and this can
significantly improve performance, especially for multithreaded decoding,
though at the cost of increased total memory use.

The ./configure option --disable-boost-pool can be used to disable this
feature if memory is tight.  This currently only affects moses_chart with
in-memory rule tables.



git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3873 1f5c12ca-751b-0410-a591-d2e778427230
 2011-02-07 15:43:19 +00:00
bhaddow
6221d2a558 Patch to add covered to osgx from Dennis Mehay 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3872 1f5c12ca-751b-0410-a591-d2e778427230
 2011-02-04 16:21:57 +00:00
heafield
fccfd85c6e Option for null context in n-gram query, use tab for delimiter 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3871 1f5c12ca-751b-0410-a591-d2e778427230
 2011-02-04 15:38:47 +00:00
maurocettolo
8fcd76f2fc made handling of chunk LM compatible with recent efficiency updates to IRSTLM toolkit by Nicola 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3870 1f5c12ca-751b-0410-a591-d2e778427230
 2011-02-04 07:42:56 +00:00