moses-smt/mosesdecoder
1
1
Fork 0
mirror of https://github.com/moses-smt/mosesdecoder.git synced 2024-12-26 13:23:25 +03:00
Code Issues Packages Projects Releases Wiki Activity
11,300 Commits 136 Branches 26 Tags 147 MiB
732a1b0744
Commit Graph

11300 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Kenneth Heafield
732a1b0744 KenLM 85c82bd, revamp Moses timer to have more precision 2014-01-01 13:19:06 -08:00
Hieu Hoang
18aaf4750a destroy feature functions at end. Otherwise it shows up on valgrind as memory leak 2013-12-31 20:15:43 +00:00
Hieu Hoang
02b78d72d9 update sample query to read ini file with relative file paths 2013-12-31 16:52:47 +00:00
Hieu Hoang
a9c81857c1 minor addition of non-const method 2013-12-29 23:33:46 +00:00
Hieu Hoang
ad617e0c86 minor change from UINT32 to UINT64 2013-12-29 23:18:16 +00:00
Lane Schwartz
867eabd699 Re-enabled unit test for backward LM. 
The test is compiling and running with no errors.
 2013-12-20 15:49:55 -05:00
Lane Schwartz
f57ac23be9 Added comment in Jamroot on how to compile against NPLM 2013-12-20 15:06:49 -05:00
Lane Schwartz
f40e1ac8f8 Added support for hierarchical maximum entropy LM through SRILM. 
Currently, the code compiles, but has not been run or tested.

    Steps to compile with maxent LM are below:

    - Download SRILM 1.6.0

    - Download the maxent patch from
      http://www.phon.ioc.ee/~tanela/srilm-me/srilm-1.6.0-me.patch

    - Download the required libLBFGS library from
      http://www.chokkan.org/software/liblbfgs

    - Edit SRILM's sbin/machine-type script
      to return MACHINE_TYPE=i686-m64 for case x86_64

    - Compile libLBFGS and install it in the /tools/SRILM/SRILM-1.7.0 directory.
      Make sure that the liblbfgs.* library files are in /tools/SRILM/SRILM-1.7.0/lib//

    - In the /tools/SRILM/SRILM-1.7.0 dir, apply the maxent patch to SRILM:
      patch -p1 < srilm-1.6.0-me.patch

    - Point the appropriate SRILM makefile
      to see libLBFGS's include and lib dirs, following the instructions at:
      http://www.phon.ioc.ee/dokuwiki/doku.php?id=people:tanel:srilm-me.en

    - Compile SRILM

    - Compile Moses using --with-srilm and --with-maxent-srilm:
      ./bjam --with-srilm=/tools/SRILM/SRILM-1.7.0 --with-maxent-srilm=true

    NOTE: The above steps were for my original integration.
          Current versions of SRILM (such as 1.7.1) include the maxent patch.
          As such, it should be sufficient to compile a current version of SRILM
          with support form maxent enabled,
          and then compile Moses using --with-srilm and --with-maxent-srilm:
          ./bjam --with-srilm=/tools/SRILM/SRILM-1.7.0 --with-maxent-srilm=true
 2013-12-20 15:04:09 -05:00
Lane Schwartz
83f89ac2b2 Updated Jamfile for compiling and linking against NPLM 2013-12-20 13:49:56 -05:00
Hieu Hoang
dea1fe995f default factor to 0 for language models that don't specify what factor type 2013-12-18 20:54:53 +00:00
Nadir Durrani
a92a46a66a Merge branch 'master' of https://github.com/moses-smt/mosesdecoder 2013-12-18 16:11:28 +00:00
Nadir Durrani
7f75018349 Post-decoding Transliteration Script 2013-12-18 16:10:57 +00:00
Hieu Hoang
1555bf3b9e Merge branch 'master' of github.com:moses-smt/mosesdecoder 2013-12-17 15:55:44 +00:00
Hieu Hoang
e8b2d2a269 bug - should get score for translation rule from the translation option, not the target phrase. 2013-12-17 14:54:12 +00:00
Hieu Hoang
f7cff6d692 Merge pull request #55 from jnory/master 
DALMWrapper using more efficient state class in DALM.
 2013-12-17 03:28:40 -08:00
Jun-ya NORIMATSU
da4e2719b2 modify DALMState to use the DALM compare function. 2013-12-17 14:36:19 +09:00
Nadir Durrani
c291f859a0 Transliteration Mining 2013-12-16 18:19:44 +00:00
Jun-ya NORIMATSU
dbf90ce8e8 more efficient state for DALM. 2013-12-16 23:17:56 +09:00
Ian Johnson
c74d85f58b Fixed mistake in README 2013-12-16 13:21:19 +00:00
Ian Johnson
513e0f1081 Merge branch 'master' of github.com:moses-smt/mosesdecoder 2013-12-16 13:17:43 +00:00
Ian Johnson
fb5545ed2c Updated PCL to v1.1.0-beta 2013-12-16 13:17:34 +00:00
Hieu Hoang
597cb4a20f added negate=true. Only create translation which DON'T match reference 2013-12-15 01:42:22 +00:00
Hieu Hoang
ea8837b5b9 delete debugging message 2013-12-14 16:15:27 +00:00
phikoehn
7763b4c41f sparse feature support for alternate weight setting 2013-12-14 02:56:01 +00:00
Hieu Hoang
ff3c8c195a compare translation option dimension using pointers to translation option, rather than comparing the phrase. Comparinf the phrase is incorrect - need to compare LHS and possibly alignments too. Unnecessary 2013-12-13 17:23:39 +00:00
Hieu Hoang
3bd5583976 more debug functions 2013-12-13 15:47:26 +00:00
Hieu Hoang
95a41c44fa more debug functions 2013-12-12 01:20:12 +00:00
Hieu Hoang
06b0b6ae87 c++ acting like a dumbass untyped scripting language and silently cast variables. Hashing of some obscure object didn't work on old boost. Broke target syntax when linking against those old boost libraries 2013-12-12 01:13:23 +00:00
Hieu Hoang
7c237bf389 delete -#if defined(BOOST... Trying to find target syntax bug when using old boost 2013-12-11 21:45:30 +00:00
Hieu Hoang
bc9ce334d0 more debug functions 2013-12-11 20:03:30 +00:00
Hieu Hoang
fb356606e9 debug function for ChartTranslationOptionList and ChartTranslationOptions 2013-12-11 19:19:25 +00:00
Hieu Hoang
ebc6e59b17 consistent global variable g_mosesDebug 2013-12-11 19:04:42 +00:00
Ian Johnson
b2d074a191 Added missing Makefiles. 2013-12-11 17:23:08 +00:00
Ian Johnson
ce6e2b1953 Fixed mert phase 2013-12-11 17:22:00 +00:00
Ian Johnson
54c6bbdd9e Replaced the tokenizer with a PCL implementation. 2013-12-11 17:00:07 +00:00
phikoehn
8222a764bc Merge branch 'master' of ssh://github.com/moses-smt/mosesdecoder 2013-12-10 19:19:30 +00:00
phikoehn
34b73cd57e added scripts for threshold filtering phrase table 2013-12-10 19:19:12 +00:00
Hieu Hoang
295c07e884 bug in constrained decoding 2013-12-10 18:47:13 +00:00
Hieu Hoang
e64368e7e4 Forgot to call SetFeaturesToApply() for SkeletonPT and Transliteration phrase table 2013-12-10 12:42:05 +00:00
Jason Riesa
2b882b6e71 Updated Picaro to include missing sample alignment files. 2013-12-09 18:41:26 -08:00
Hieu Hoang
2ebb2cd5d4 added Picaro /Jason Riesa 2013-12-09 23:24:48 +00:00
Hieu Hoang
7bae70a2bc minor. More description of -feature-overwrite 2013-12-09 11:53:46 +00:00
Hieu Hoang
ba209202ef move all feature functions out of StaticData 2013-12-07 00:21:06 +00:00
Hieu Hoang
a5d0296699 make transliteration phrase table thread safe 2013-12-06 16:25:17 +00:00
Hieu Hoang
09d50f87b1 move word penalty feature out of StaticData 2013-12-06 15:57:49 +00:00
Hieu Hoang
f77fd7fb45 set parameters for fuzzy match. Not tested 2013-12-06 14:23:22 +00:00
Hieu Hoang
78eba93a6b minor change in argument key for consistency 2013-12-06 13:09:34 +00:00
Hieu Hoang
0e156afd5b Merge branch 'master' of github.com:moses-smt/mosesdecoder 2013-12-06 13:07:40 +00:00
Hieu Hoang
fc8f8ba657 minor change in argument key for consistency 2013-12-06 13:07:33 +00:00
Rico Sennrich
96ca814b86 special handling of partial rules that can only have terminal symbol next; 
we only need to search once for matching terminals.

speed: 336s -> 309s
VmPeak: 2914132kB -> 2783064kB
RSSMax: 2766976kB -> 2699148kB
 2013-12-06 12:54:58 +00:00