moses-smt/mosesdecoder
1
1
Fork 0
mirror of https://github.com/moses-smt/mosesdecoder.git synced 2024-09-20 15:48:05 +03:00
Code Issues Packages Projects Releases Wiki Activity
6,107 Commits 136 Branches 26 Tags 147 MiB
ac8b86f0c8
Commit Graph

33 Commits

Author SHA1 Message Date
heafield
915cb22b6a Make tests run on OS X. This was an issue with the test (and its use of popen) not with the code. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3767 1f5c12ca-751b-0410-a591-d2e778427230
 2010-12-09 21:58:54 +00:00
heafield
9062e3b73b KenLM update: allow user to specify data structure and parameters on command line to 
build_binary.  Also some minor bugfixes.  



git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3762 1f5c12ca-751b-0410-a591-d2e778427230
 2010-12-08 03:15:37 +00:00
heafield
eabc137306 Make kenlm tests compile on more systems by adding headers. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3738 1f5c12ca-751b-0410-a591-d2e778427230
 2010-11-28 21:16:06 +00:00
heafield
0d8b62791e kenlm update. 
Improved portability:
Hopefully handle big endian architectures (trie will fail at runtime with if this isn't working yet).  
Remove dependence on err.h.
Handle some Solaris weirdness wrt mmap and strerror.  
Clean up murmur_hash header.  

Add comparison and ZeroRemaining requested by Chris Dyer.  

More number parsing in FilePiece.  



git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3737 1f5c12ca-751b-0410-a591-d2e778427230
 2010-11-28 02:54:56 +00:00
heafield
82f29bfc16 Chris Dyer says this should make things compile better on OS X. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3694 1f5c12ca-751b-0410-a591-d2e778427230
 2010-11-10 02:05:51 +00:00
hieuhoang1972
3b6b002df8 xcode 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3691 1f5c12ca-751b-0410-a591-d2e778427230
 2010-11-09 13:25:09 +00:00
hieuhoang1972
9a72825d29 mac compile 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3689 1f5c12ca-751b-0410-a591-d2e778427230
 2010-11-08 16:09:04 +00:00
heafield
2784923899 Rename a bunch of kenlm files. A ./regenerate-makefiles.sh is required. 
Make loading with MAP_POPULATE on Linux and read on other OSes the default.
Use LM #9 for lazy loading, as recommended by other devs.  
Slightly faster trie.  



git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3688 1f5c12ca-751b-0410-a591-d2e778427230
 2010-11-06 00:40:16 +00:00
heafield
bf88f87d78 Fix return value of FilePiece::ReadLine at end of file. Did not impact existing kenlm (since 
they don't read to the end of file) but will impact future versions.  


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3682 1f5c12ca-751b-0410-a591-d2e778427230
 2010-10-29 17:53:19 +00:00
heafield
c12c2c59d2 Autodetect model from binary format. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3675 1f5c12ca-751b-0410-a591-d2e778427230
 2010-10-28 01:05:04 +00:00
hieuhoang1972
eb374bf082 cygwin build 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3674 1f5c12ca-751b-0410-a591-d2e778427230
 2010-10-27 20:47:28 +00:00
hieuhoang1972
735d5b682f xcode 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3673 1f5c12ca-751b-0410-a591-d2e778427230
 2010-10-27 18:54:50 +00:00
heafield
614d6002a6 Integrate heafield-refactorlm. Faster kenlm with new binary format. Stateful language model 
framework.  



git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3671 1f5c12ca-751b-0410-a591-d2e778427230
 2010-10-27 17:50:40 +00:00
hieuhoang1972
46b59cbdd7 xcode 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3667 1f5c12ca-751b-0410-a591-d2e778427230
 2010-10-27 10:20:33 +00:00
heafield
64cfacd1bd Backporting FilePiece leaked scoped_FILE, but only into the test. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3665 1f5c12ca-751b-0410-a591-d2e778427230
 2010-10-27 04:04:23 +00:00
hieuhoang1972
34e7c43114 xcode 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3664 1f5c12ca-751b-0410-a591-d2e778427230
 2010-10-27 03:14:11 +00:00
heafield
d1b1b4f34c Tom from precision translation tools reports that IRST doesn't generate a blank line after each block. Removed this 
requirement from the parser.  


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3657 1f5c12ca-751b-0410-a591-d2e778427230
 2010-10-26 14:04:32 +00:00
heafield
8d0d44f5cd Support gzipped ARPA files. Progress bar tweak. Test fixes. Holding off on the big change for now. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3643 1f5c12ca-751b-0410-a591-d2e778427230
 2010-10-23 05:21:10 +00:00
heafield
e65ecd0632 Put official website in README 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3631 1f5c12ca-751b-0410-a591-d2e778427230
 2010-10-19 15:55:49 +00:00
hieuhoang1972
5f6baa9021 xcode 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3627 1f5c12ca-751b-0410-a591-d2e778427230
 2010-10-18 05:47:07 +00:00
hieuhoang1972
7463257be5 gcc 3.4 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3613 1f5c12ca-751b-0410-a591-d2e778427230
 2010-10-11 13:36:40 +00:00
hieuhoang1972
e504b797b2 xcode 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3599 1f5c12ca-751b-0410-a591-d2e778427230
 2010-10-01 00:21:27 +00:00
heafield
770df2a92d Unbodge kenlm by moving compilation to kenlm/ instead of kenlm/lm. Changing the headers every 
time I copied to Moses was getting annoying.  



git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3587 1f5c12ca-751b-0410-a591-d2e778427230
 2010-09-28 16:26:55 +00:00
heafield
d1a7c636ac Jon Clark complained that IRSTLM puts 0.0 backoff for n-grams of longest order and that I throw 
an error.  



git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3578 1f5c12ca-751b-0410-a591-d2e778427230
 2010-09-27 18:38:21 +00:00
heafield
e6184ae947 Updates to kenlm: 
Kludged and slow interface requested by Hieu because apparently Moses can't store language model state.  
Separate files for ARPA reading, vocabulary, and weights.  
Remove build shell scripts that won't work after Hieu changed the header file layout.  



git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3572 1f5c12ca-751b-0410-a591-d2e778427230
 2010-09-27 03:46:44 +00:00
heafield
61f5472f1c Avoid some unused parameter complaints and force automake dependencies to fix parallel make. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3571 1f5c12ca-751b-0410-a591-d2e778427230
 2010-09-27 00:57:11 +00:00
hieuhoang1972
a82c2d5531 ken lm integration 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3569 1f5c12ca-751b-0410-a591-d2e778427230
 2010-09-26 17:02:53 +00:00
hieuhoang1972
f7ceddd0a9 ken lm integration 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3568 1f5c12ca-751b-0410-a591-d2e778427230
 2010-09-26 16:49:07 +00:00
hieuhoang1972
52c823fec4 ken lm integration 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3551 1f5c12ca-751b-0410-a591-d2e778427230
 2010-09-23 10:21:29 +00:00
hieuhoang1972
32d3565b04 ken lm integration 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3543 1f5c12ca-751b-0410-a591-d2e778427230
 2010-09-21 22:43:29 +00:00
heafield
a02268a7c1 Fix memory corruption with exceptions. 
Fix compilation with -m64 in murmur_hash.  
Extract most mmap calls.  



git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3494 1f5c12ca-751b-0410-a591-d2e778427230
 2010-09-16 19:53:33 +00:00
heafield
d00c788760 kenlm update 
mmap works; utility to build binary format included.  
Configuration struct (including unknown handling options). 
config option to build a binary format while loading an ARPA.  
Doesn't require Boost or ICU. 
Works on 32 and 64 bit. 
query appends </s>. 
Reduced memory consumption: 12 bytes per 5-gram instead of 16 bytes on 64-bit machines.  
Reduced memory consumption: vocabulary takes 8 bytes/word instead of 12 bytes/word if sorted is 
used. 
Removed some cruft that wasn't needed by this code.  
Compiles on Mac OS X.  
Add script to run tests; these depend on Boost.  
SRI wrapper works again, is slightly faster, no longer depends on Boost, and has a test.
Debugging code only appears with -DDEBUG, so the default is fast.  



git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3447 1f5c12ca-751b-0410-a591-d2e778427230
 2010-09-14 21:33:11 +00:00
hieuhoang1972
473e0e3e96 Ken's LM 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3421 1f5c12ca-751b-0410-a591-d2e778427230
 2010-09-10 00:36:07 +00:00