mosesdecoder/kenlm
heafield 6dae77c3eb Fix segfault withe trie on models without <unk>. Problem was that trie writes correct counts to
the binary file header, including <unk>.  But the vocabulary was sized based on the ARPA file 
count (excluding <unk>).  Then when the binary file was loaded, the vocabulary size was based on 
the count including <unk>.  Fix this by pre-padding vocabulary to the count including <unk>.  

Also, some minor cleanups: remove a debug message and change some always-true returns to void.  



git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4145 1f5c12ca-751b-0410-a591-d2e778427230
2011-08-16 12:57:21 +00:00
..
kenlm.xcodeproj xcode 2011-07-15 12:09:33 +00:00
lm Fix segfault withe trie on models without <unk>. Problem was that trie writes correct counts to 2011-08-16 12:57:21 +00:00
util Optional compression for trie. Also, some better error handling. 2011-07-13 20:53:18 +00:00
clean.sh Hieu's opinion is to keep the standalone test shell scripts. 2011-03-29 15:07:57 +00:00
compile.sh Minor fixes. One for David Chiang who has files without initial newlines. 2011-08-03 19:46:19 +00:00
COPYING Ken's LM 2010-09-10 00:36:07 +00:00
COPYING.LESSER Ken's LM 2010-09-10 00:36:07 +00:00
LICENSE kenlm update 2010-09-14 21:33:11 +00:00
Makefile.am Optional compression for trie. Also, some better error handling. 2011-07-13 20:53:18 +00:00
README Hieu's opinion is to keep the standalone test shell scripts. 2011-03-29 15:07:57 +00:00
test.sh Hieu's opinion is to keep the standalone test shell scripts. 2011-03-29 15:07:57 +00:00

Language model inference code by Kenneth Heafield <infer at kheafield.com>
The official website is http://kheafield.com/code/kenlm/ .  If you're a decoder developer, please download the latest version from there instead of copying from Moses.  

While the primary means of building kenlm for use in Moses is the Moses build system, you can also compile independently using:
./compile.sh to compile the code
./test.sh to compile and run tests; requires Boost
./clean.sh to clean

The rest of the documentation is directed at decoder developers.  

Binary format via mmap is supported.  Run ./build_binary to make one then pass the binary file name instead.  

Currently, it assumes POSIX APIs for errno, sterror_r, open, close, mmap, munmap, ftruncate, fstat, and read.  This is tested on Linux and the non-UNIX Mac OS X.  I welcome submissions porting (via #ifdef) to other systems (e.g. Windows) but proudly have no machine on which to test it.  

A brief note to Mac OS X users: your gcc is too old to recognize the pack pragma.  The warning effectively means that, on 64-bit machines, the model will use 16 bytes instead of 12 bytes per n-gram of maximum order (those of lower order are already 16 bytes) in the probing and sorted models.  The trie is not impacted by this.  

It does not depend on Boost or ICU.  However, if you use Boost and/or ICU in the rest of your code, you should define HAVE_BOOST and/or HAVE_ICU in util/have.hh.  Defining HAVE_BOOST will let you hash StringPiece.  Defining HAVE_ICU will use ICU's StringPiece to prevent a conflict with the one provided here.  

The recommend way to use this:
Copy the code and distribute with your decoder.  
Set HAVE_ICU and HAVE_BOOST at the top of util/have.hh as instructed above.  
Look at compile.sh and reimplement using your build system.  
Use either the interface in lm/model.hh or lm/virtual_interface.hh
Interface documentation is in comments of lm/virtual_interface.hh (including for lm/model.hh).  

I recommend copying the code and distributing it with your decoder.  However, please send improvements to me so that they can be integrated into the package.  

Also included:
A wrapper to SRI with the same interface.