hieuhoang1972
1c342ffefd
Deal with updated file format. Thanks to Christof Pintaske
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3576 1f5c12ca-751b-0410-a591-d2e778427230
2010-09-27 18:03:22 +00:00
heafield
5b74b38527
Remove vestigial dub parameter. Surrender to tab-based whitespace. More passive-aggressive message about nGramOrder.
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3575 1f5c12ca-751b-0410-a591-d2e778427230
2010-09-27 16:01:58 +00:00
heafield
cac7600662
Make --with-kenlm=$PWD/kenlm compile mine instead of IRST.
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3574 1f5c12ca-751b-0410-a591-d2e778427230
2010-09-27 15:07:28 +00:00
hieuhoang1972
375ed9b7fe
ken lm integration
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3573 1f5c12ca-751b-0410-a591-d2e778427230
2010-09-27 10:46:39 +00:00
heafield
e6184ae947
Updates to kenlm:
...
Kludged and slow interface requested by Hieu because apparently Moses can't store language model state.
Separate files for ARPA reading, vocabulary, and weights.
Remove build shell scripts that won't work after Hieu changed the header file layout.
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3572 1f5c12ca-751b-0410-a591-d2e778427230
2010-09-27 03:46:44 +00:00
heafield
61f5472f1c
Avoid some unused parameter complaints and force automake dependencies to fix parallel make.
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3571 1f5c12ca-751b-0410-a591-d2e778427230
2010-09-27 00:57:11 +00:00
hieuhoang1972
559a5e3ece
ken lm integration
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3570 1f5c12ca-751b-0410-a591-d2e778427230
2010-09-26 22:29:01 +00:00
hieuhoang1972
a82c2d5531
ken lm integration
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3569 1f5c12ca-751b-0410-a591-d2e778427230
2010-09-26 17:02:53 +00:00
hieuhoang1972
f7ceddd0a9
ken lm integration
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3568 1f5c12ca-751b-0410-a591-d2e778427230
2010-09-26 16:49:07 +00:00
pjwilliams
f5a096e2fa
In ChartTranslationOptionList::Add(), always consider every target phrase as
...
the TargetPhraseCollection won't be ordered.
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3567 1f5c12ca-751b-0410-a591-d2e778427230
2010-09-26 12:09:29 +00:00
pjwilliams
81372704de
Enforce ttable-limit for memory-based rule table.
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3566 1f5c12ca-751b-0410-a591-d2e778427230
2010-09-26 11:38:09 +00:00
pjwilliams
ab62fe2ed3
QueueEntry::CreateChildEntry() should always succeed, so remove isOK check
...
and add assertion to test for failure.
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3565 1f5c12ca-751b-0410-a591-d2e778427230
2010-09-26 10:41:57 +00:00
pjwilliams
d7a61c751f
Don't bother popping QueueEntrys off the priority queue when destructing
...
MosesChart::Cube.
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3564 1f5c12ca-751b-0410-a591-d2e778427230
2010-09-26 10:30:28 +00:00
hieuhoang1972
031617a8db
delete unused function
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3563 1f5c12ca-751b-0410-a591-d2e778427230
2010-09-25 20:30:19 +00:00
hieuhoang1972
ea04dc5209
vs.net
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3562 1f5c12ca-751b-0410-a591-d2e778427230
2010-09-24 21:01:11 +00:00
hieuhoang1972
465d1cef0a
make class name consistent for chart & pb decoders
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3561 1f5c12ca-751b-0410-a591-d2e778427230
2010-09-23 17:39:32 +00:00
hieuhoang1972
4d34e5155e
make class name consistent for chart & pb decoders
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3560 1f5c12ca-751b-0410-a591-d2e778427230
2010-09-23 16:54:32 +00:00
hieuhoang1972
0b6a29327c
make class name consistent for chart & pb decoders
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3559 1f5c12ca-751b-0410-a591-d2e778427230
2010-09-23 16:32:44 +00:00
hieuhoang1972
cafc5046c7
make class name consistent for chart & pb decoders
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3558 1f5c12ca-751b-0410-a591-d2e778427230
2010-09-23 15:48:33 +00:00
hieuhoang1972
73ece4f859
make class name consistent for chart & pb decoders
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3557 1f5c12ca-751b-0410-a591-d2e778427230
2010-09-23 15:18:11 +00:00
hieuhoang1972
1920fc007c
makefile for ken lm
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3556 1f5c12ca-751b-0410-a591-d2e778427230
2010-09-23 14:37:06 +00:00
hieuhoang1972
af228d7e93
make class name consistent for chart & pb decoders
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3555 1f5c12ca-751b-0410-a591-d2e778427230
2010-09-23 12:34:35 +00:00
hieuhoang1972
34873a1ba4
make class name consistent for chart & pb decoders
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3554 1f5c12ca-751b-0410-a591-d2e778427230
2010-09-23 12:12:49 +00:00
hieuhoang1972
b83fc49a1f
make class name consistent for chart & pb decoders
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3553 1f5c12ca-751b-0410-a591-d2e778427230
2010-09-23 11:05:38 +00:00
hieuhoang1972
9f5d530910
make class name consistent for chart & pb decoders
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3552 1f5c12ca-751b-0410-a591-d2e778427230
2010-09-23 10:43:31 +00:00
hieuhoang1972
52c823fec4
ken lm integration
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3551 1f5c12ca-751b-0410-a591-d2e778427230
2010-09-23 10:21:29 +00:00
rsennrich
7929e4624e
more informative error message when hierarchical phrase extraction fails.
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3550 1f5c12ca-751b-0410-a591-d2e778427230
2010-09-22 12:56:11 +00:00
rosasjolu
8746482d04
Change data files location
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3549 1f5c12ca-751b-0410-a591-d2e778427230
2010-09-22 11:03:36 +00:00
rosasjolu
16302a45a8
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3548 1f5c12ca-751b-0410-a591-d2e778427230
2010-09-22 11:02:29 +00:00
rosasjolu
d2fd75ac49
Change data files location
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3547 1f5c12ca-751b-0410-a591-d2e778427230
2010-09-22 10:27:07 +00:00
rosasjolu
ad62e27c90
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3546 1f5c12ca-751b-0410-a591-d2e778427230
2010-09-22 10:24:52 +00:00
hieuhoang1972
ee842f578c
delete data files
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3545 1f5c12ca-751b-0410-a591-d2e778427230
2010-09-22 10:05:11 +00:00
hieuhoang1972
989eb55449
ken lm integration
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3544 1f5c12ca-751b-0410-a591-d2e778427230
2010-09-21 23:10:15 +00:00
hieuhoang1972
32d3565b04
ken lm integration
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3543 1f5c12ca-751b-0410-a591-d2e778427230
2010-09-21 22:43:29 +00:00
phkoehn
f34b37bad3
added hierarchical alignment view to web analysis tool
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3514 1f5c12ca-751b-0410-a591-d2e778427230
2010-09-17 13:28:04 +00:00
heafield
a02268a7c1
Fix memory corruption with exceptions.
...
Fix compilation with -m64 in murmur_hash.
Extract most mmap calls.
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3494 1f5c12ca-751b-0410-a591-d2e778427230
2010-09-16 19:53:33 +00:00
hieuhoang1972
a582d483cc
add lowercase script
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3475 1f5c12ca-751b-0410-a591-d2e778427230
2010-09-16 07:32:32 +00:00
hieuhoang1972
e9b355ed8c
get rid of weights in translation model
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3471 1f5c12ca-751b-0410-a591-d2e778427230
2010-09-15 16:49:49 +00:00
phkoehn
fb8b0eb180
new prefix files for tokenizer
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3467 1f5c12ca-751b-0410-a591-d2e778427230
2010-09-15 16:06:04 +00:00
hieuhoang1972
d0aecca9fe
get rid of weight in LM class
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3466 1f5c12ca-751b-0410-a591-d2e778427230
2010-09-15 16:02:41 +00:00
heafield
d00c788760
kenlm update
...
mmap works; utility to build binary format included.
Configuration struct (including unknown handling options).
config option to build a binary format while loading an ARPA.
Doesn't require Boost or ICU.
Works on 32 and 64 bit.
query appends </s>.
Reduced memory consumption: 12 bytes per 5-gram instead of 16 bytes on 64-bit machines.
Reduced memory consumption: vocabulary takes 8 bytes/word instead of 12 bytes/word if sorted is
used.
Removed some cruft that wasn't needed by this code.
Compiles on Mac OS X.
Add script to run tests; these depend on Boost.
SRI wrapper works again, is slightly faster, no longer depends on Boost, and has a test.
Debugging code only appears with -DDEBUG, so the default is fast.
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3447 1f5c12ca-751b-0410-a591-d2e778427230
2010-09-14 21:33:11 +00:00
rosasjolu
128a885406
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3446 1f5c12ca-751b-0410-a591-d2e778427230
2010-09-14 19:27:15 +00:00
pjwilliams
130ae457dd
In PhraseDictionarySCFG, only add non-leaf nodes to the running node lists.
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3424 1f5c12ca-751b-0410-a591-d2e778427230
2010-09-10 15:19:25 +00:00
hieuhoang1972
c73dddfb5b
delete eclipse build
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3423 1f5c12ca-751b-0410-a591-d2e778427230
2010-09-10 09:34:17 +00:00
hieuhoang1972
bdfa2beab8
delete eclipse build
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3422 1f5c12ca-751b-0410-a591-d2e778427230
2010-09-10 09:16:50 +00:00
hieuhoang1972
473e0e3e96
Ken's LM
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3421 1f5c12ca-751b-0410-a591-d2e778427230
2010-09-10 00:36:07 +00:00
suzyh
fa4eca6ccb
Added loop to check_if_crashed in EMS experiment.perl to wait in case the .STDERR file is slow in appearing after the step has completed. Reinstated --old-sge and --filterfile command-line arguments to mert-moses.pl.
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3420 1f5c12ca-751b-0410-a591-d2e778427230
2010-09-09 11:40:40 +00:00
bhaddow
fd7997dbf5
Fix for mert script from Yu Chen, to make sure it reads the correct
...
feature order from the nbest list.
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3419 1f5c12ca-751b-0410-a591-d2e778427230
2010-09-08 08:17:05 +00:00
bhaddow
d3680b4255
Make sure distortion feature are adjacent
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3418 1f5c12ca-751b-0410-a591-d2e778427230
2010-09-07 10:54:04 +00:00
hieuhoang1972
e53aeb903c
need non-empty arg in mert-moses.pl otherwise it crashes. The wonders of a non-typechecked, arg checking scripting language
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3417 1f5c12ca-751b-0410-a591-d2e778427230
2010-09-04 00:16:26 +00:00