Marcin Junczys-Dowmunt
061aac9e3b
First attempt at fixing missing-backoffs bug in lmplz
2014-07-27 20:35:15 +02:00
Kenneth Heafield
c83c5a3ee6
D'ph forgot to copy util
2014-07-19 06:54:01 +08:00
Hieu Hoang
07d325876c
dump_counts doesn't compile
2014-07-18 21:30:49 +01:00
Kenneth Heafield
ff1d5ca0d6
KenLM 8386117fdc026b8532adac85c5f921388754d569
2014-07-18 22:24:28 +08:00
Rico Sennrich
eb5336ad9c
thread-safety, precomputation and caching for NeuralLMWrapper
...
doesn't work with default nplm, but with the fork at https://github.com/rsennrich/nplm
2014-07-17 16:50:08 +01:00
Rico Sennrich
0d8d77e3da
add nplm wrapper from Kenneth's kenlm repo
2014-07-17 16:10:30 +01:00
Kenneth Heafield
6b22097cbe
KenLM e94d0b3ea9082d8aa443e15e33aee6fb9c2af990 including possible deadlock fix
2014-06-22 05:54:03 -07:00
Kenneth Heafield
6d9173ba72
KenLM f81d02792087a837ea17e6ce2b33f9b7aaecca68 should fix segfault with ArrayTrie
2014-06-04 16:03:39 -07:00
Kenneth Heafield
dd03f9fb69
KenLM 5a7efd8fe1db88ee0a9f7e9479b24ac3ca348221 with Hieu's patch to exception.hh
2014-06-02 10:29:40 -07:00
Kenneth Heafield
14e02978fc
KenLM 5cc905bc2d214efa7de2db56a9a672b749a95591
...
Avoid unspecified behavior of mmap when a file is resized reported by Christian Hardmeier
Fixes for Mavericks and a workaround for Boost's broken semaphore
Clean clang compile (of kenlm)
Merged some of 744376b3fb
2014-01-27 16:51:35 -08:00
jiejiang
5f1217d793
merged upstream with origin for mingw
2014-01-15 18:16:56 +00:00
Kenneth Heafield
732a1b0744
KenLM 85c82bd, revamp Moses timer to have more precision
2014-01-01 13:19:06 -08:00
jiejiang
744376b3fb
moses windows build, with some TODO list
2013-12-18 20:15:39 +00:00
Kenneth Heafield
5cca6fafcb
KenLM 6f7913cc7ca0f7672c6d899358365f047a742bbb
...
Mostly fixes from Tetsuo Kiso and Jonathan Graehl
2013-09-09 10:43:20 +01:00
Lane Schwartz
226031ecf6
Checking in examples of CMake files.
...
The example in contrib works, as does the one in biconcor.
The examples in lm and moses don't work at the moment,
but checking them in anyway as a starting point
for any potential work on them next week during the
MT Marathon.
2013-09-06 13:39:46 -04:00
Kenneth Heafield
1a8977d890
EmptyContextState -> NullContextState / Nadir
2013-07-01 16:22:25 +01:00
Kenneth Heafield
794867c555
KenLM 6b4a1c7940a36026de1d96693ccb6ec0f16de8dc
2013-06-24 16:05:47 +01:00
Kenneth Heafield
59bd7deb4b
Tetsuo's two patches
2013-05-29 05:30:39 -04:00
Kenneth Heafield
50652382e9
KenLM 10ddf7d923355b35a7de9a5219673eca9e18be98 except Hieu's slow string_piece_hash
2013-05-19 10:12:06 -04:00
Xiang Li
1b63277e66
Update search_trie.hh
...
There is no array compression header for order 2 models
2013-05-16 10:27:46 +08:00
Kenneth Heafield
03b077364a
KenLM 31a6644 resizable probing hash table, build fixes
2013-01-24 12:07:46 +00:00
Kenneth Heafield
22bf1c77e9
Ales Tamchyna patch for ancient Boost.
2013-01-23 13:28:15 +00:00
Kenneth Heafield
e2fc306231
KenLM 58da33: compiler warnings, correctly handle bare K, kill HAVE_BOOST
2013-01-22 20:23:35 +00:00
Kenneth Heafield
cfe7d00ea2
Compiler warnings, rename to ${binary}_main.cc
2013-01-20 13:43:19 +00:00
Kenneth Heafield
a52d97f99f
Cross platform fixes
2013-01-20 12:25:58 +00:00
Hieu Hoang
8a3b93c72e
compile error on mac
2013-01-20 11:52:18 +00:00
Kenneth Heafield
a35299497f
Tolerate odd characters in corpora.
2013-01-19 13:35:00 +00:00
Kenneth Heafield
fc5868d0ff
KenLM df5be22 lmplz for estimation
2013-01-18 15:59:51 +00:00
Kenneth Heafield
5f7b91e702
Workaround for bjam inverting argument order
2013-01-18 13:40:56 +00:00
Kenneth Heafield
a7731ce432
KenLM 014bced6
2013-01-17 11:58:58 +00:00
Kenneth Heafield
1530ae4f5f
Fix state comparison (impacted 32-bit)
2013-01-09 13:15:04 +00:00
Kenneth Heafield
f9ee7ae4b3
KenLM 0e5d259 including read_compressed fix
2013-01-04 21:02:47 +00:00
Ales Tamchyna
35cb7e68be
correct parameter in comment for KenLM maximum order
2012-11-15 16:06:51 +01:00
Kenneth Heafield
d653d45abc
KenLM 85e5fe4: mostly fixes for cygwin/windows
2012-11-15 12:00:15 +00:00
Kenneth Heafield
d4f433c7f4
Put core Moses into one library libmoses.
2012-11-12 18:07:53 +00:00
Kenneth Heafield
14392acc8f
KenLM 36b746 including -w mmap <unk> offset fix
2012-11-04 15:36:42 -05:00
Kenneth Heafield
9a1ac30889
Increase paranoia about --kenlm-max-order and --max-factors
2012-10-31 14:48:29 +00:00
Kenneth Heafield
4b98b0937c
Add bzip and xz input for FilePiece KenLM 06f78db2
2012-10-30 19:33:22 +00:00
Kenneth Heafield
5f67527143
Bugfix vocabulary corruption on building
2012-10-19 12:00:10 +01:00
Kenneth Heafield
791a7b68fc
Make max-factors a compile option and add recompilation safety
2012-10-19 10:16:45 +01:00
Kenneth Heafield
de5276630b
Restore kenlm max order
2012-10-19 10:00:01 +01:00
Kenneth Heafield
0eb98df0fe
Submodules are evil. Undo it.
2012-10-15 13:58:33 +01:00
Kenneth Heafield
4faab6c68f
Extract kenlm into a submodule, referencing lazy
2012-09-24 18:30:51 +01:00
Kenneth Heafield
b317522563
Move max-order to lm directory and direct dependencies.
2012-08-18 12:07:53 -04:00
Kenneth Heafield
128829ebee
Don't segfault trie building when there is no n-gram of a given order. Jon Clark.
2012-08-16 16:01:43 -04:00
Kenneth Heafield
6efb6c0ffe
Throw exception for binary files with too large order / Ondrej Bojar
2012-08-15 11:08:08 -04:00
Lane Schwartz
a2afc98665
Add program to query the KenLM maximum n-gram order
2012-08-08 16:41:29 -04:00
Lane Schwartz
da5429318b
KenLM maximum n-gram order can now be set via a compile-time flag
2012-08-08 16:22:13 -04:00
Kenneth Heafield
650fcd6919
Cast everything to double before BOOST_CHECK_CLOSE for Yvette
2012-08-06 08:31:02 -04:00
Kenneth Heafield
d5c2868f97
Fix slow backoff lookup skip path
2012-07-16 10:47:47 -04:00
