moses-smt/mosesdecoder
1
1
Fork 0
mirror of https://github.com/moses-smt/mosesdecoder.git synced 2024-12-27 22:14:57 +03:00
Code Issues Packages Projects Releases Wiki Activity
11,762 Commits 136 Branches 26 Tags 147 MiB
ce853731ae
Commit Graph

161 Commits

Author SHA1 Message Date
Ulrich Germann
ce853731ae Added mmsapt lookup utility. 2014-06-05 01:38:55 +01:00
Ulrich Germann
21f141ce0d Merge branch 'master' into dynamic-phrase-tables 
Conflicts:
	moses/InputPath.cpp
	moses/Word.cpp
 2014-04-08 14:58:29 +01:00
Rico Sennrich
45630a5851 various optimizations to make CYK+ parser several times faster and eat less memory. 
speed-up of decoding depends on how much time is spent in parser:
10-50% speed-up for string-to-tree systems observed (more on long sentences and with high max-chart-span).

if you only use hiero or string-to-tree models (but none with source syntax), use compile-option --unlabelled-source for (small) efficiency gains.
 2014-03-21 11:12:24 +00:00
Ulrich Germann
c91fb5cc84 Added mmlex-lookup utility program. 2014-03-19 20:41:29 +00:00
Ulrich Germann
394a40b934 Added program calc-coverage. 2014-03-18 12:17:07 +00:00
Ulrich Germann
9025ac065f Added utilities: 
- mam2symal converts memory-mapped word alignments to symal format
- mam_verify performs a sanity check on memory-mapped word alignments
 2014-03-12 08:06:55 +00:00
Ulrich Germann
a7c85780ee Merge branch 'master' into dynamic-phrase-tables 
Conflicts:
	phrase-extract/score-main.cpp
 2014-03-10 14:25:45 +00:00
Kenneth Heafield
952bb76b77 We weren't linking against tcmalloc... 2014-03-01 07:42:32 +00:00
Ulrich Germann
4b95c3a906 Merge branch 'dynamic-phrase-tables' of ssh://thor//home/germann/git/mosesdecoder into dynamic-phrase-tables 
due to resetting the location of the remote repository.
 2014-02-21 01:09:38 +00:00
Ulrich Germann
b410026804 Added test program try-align. 2014-02-20 23:56:45 +00:00
Ulrich Germann
af007c4cf0 Minor fixes related to mm. 2014-02-08 18:18:45 +00:00
Rico Sennrich
c1d8f6e267 Revert "testing the waters for C++11 adoption" 
This reverts commit d2d508184e.

there's problems with gcc 4.5, and apparently different problems with new boost versions; sticking with C++03 for the time being.
 2014-01-15 16:16:11 +00:00
Rico Sennrich
d2d508184e testing the waters for C++11 adoption 2014-01-14 17:01:46 +00:00
Hieu Hoang
ca42b9b01f tcmalloc /buhomecanico 2014-01-06 11:52:21 +00:00
Lane Schwartz
f57ac23be9 Added comment in Jamroot on how to compile against NPLM 2013-12-20 15:06:49 -05:00
Lane Schwartz
f40e1ac8f8 Added support for hierarchical maximum entropy LM through SRILM. 
Currently, the code compiles, but has not been run or tested.

    Steps to compile with maxent LM are below:

    - Download SRILM 1.6.0

    - Download the maxent patch from
      http://www.phon.ioc.ee/~tanela/srilm-me/srilm-1.6.0-me.patch

    - Download the required libLBFGS library from
      http://www.chokkan.org/software/liblbfgs

    - Edit SRILM's sbin/machine-type script
      to return MACHINE_TYPE=i686-m64 for case x86_64

    - Compile libLBFGS and install it in the /tools/SRILM/SRILM-1.7.0 directory.
      Make sure that the liblbfgs.* library files are in /tools/SRILM/SRILM-1.7.0/lib//

    - In the /tools/SRILM/SRILM-1.7.0 dir, apply the maxent patch to SRILM:
      patch -p1 < srilm-1.6.0-me.patch

    - Point the appropriate SRILM makefile
      to see libLBFGS's include and lib dirs, following the instructions at:
      http://www.phon.ioc.ee/dokuwiki/doku.php?id=people:tanel:srilm-me.en

    - Compile SRILM

    - Compile Moses using --with-srilm and --with-maxent-srilm:
      ./bjam --with-srilm=/tools/SRILM/SRILM-1.7.0 --with-maxent-srilm=true

    NOTE: The above steps were for my original integration.
          Current versions of SRILM (such as 1.7.1) include the maxent patch.
          As such, it should be sufficient to compile a current version of SRILM
          with support form maxent enabled,
          and then compile Moses using --with-srilm and --with-maxent-srilm:
          ./bjam --with-srilm=/tools/SRILM/SRILM-1.7.0 --with-maxent-srilm=true
 2013-12-20 15:04:09 -05:00
Ulrich Germann
290ae0e763 Added bjam command line switch to compile with memory-mapped data structures. Aperently this doesn't work on MacOS X. 2013-10-28 21:49:36 +00:00
Hieu Hoang
0eff7344d7 move Uli's new pt into it's own folder. Comment out bjam compile. Doesn't work on MacOSX yet 2013-09-26 10:28:52 +01:00
=
d8686f79db Reformatting. 2013-09-25 00:11:11 +01:00
Kenneth Heafield
5e506ed914 Make dl have the same linkage as glibc 2013-09-12 09:27:01 +01:00
Hieu Hoang
b534e8c4b7 f*cking link error 2013-09-12 09:17:18 +01:00
Kenneth Heafield
6b5c1a09e4 unwind library, cleanup 2013-08-22 10:16:12 +01:00
Kenneth Heafield
37c22cd4da Also use unwind for for tcmalloc 2013-08-21 14:29:00 +01:00
Kenneth Heafield
b1d7340b78 Add --full-tcmalloc to link against the non-minimal tcmalloc library 2013-08-21 10:16:33 +01:00
Hieu Hoang
5fa6e506b1 Jamroot error 2013-08-15 15:39:40 +01:00
phikoehn
b368085609 xml constraint 2013-08-15 11:46:45 +01:00
Barry Haddow
c127c58e9b fix to single thread build 2013-07-25 15:56:20 +01:00
Hieu Hoang
418652fe1d iconv for SRILM 1.7.0 on MacOSX 2013-07-01 10:57:06 +01:00
Kenneth Heafield
215e93377c Move iconv to everything but for OS X only 2013-07-01 09:01:56 +01:00
Kenneth Heafield
0d9020fefe Add message for missing tcmalloc and improve install directions 2013-06-18 13:31:37 -04:00
Lane Schwartz
aacc936204 Adding unit tests to Backward LM 2013-04-29 09:18:46 -04:00
Kenneth Heafield
030e2d40b2 Use tcmalloc_minimal and tell people how to build it 2012-11-14 20:29:55 +00:00
Kenneth Heafield
be7e78c6bf Use tcmalloc with multithread if detected. 
Change option and documentation to --without-tcmalloc.
 2012-11-14 20:01:07 +00:00
Hieu Hoang
f96b33de83 only include moses root when compiling 2012-11-14 13:43:04 +00:00
Hieu Hoang
0bc0b85358 resolved merge conflicts 2012-11-12 21:12:01 +00:00
Hieu Hoang
125fdbcff6 move moses-cmd/src/* to moses-cmd/ 2012-11-12 20:42:29 +00:00
Hieu Hoang
5e3ef23cef move moses/src/* to moses/ 2012-11-12 19:56:18 +00:00
Barry Haddow
62fa6d6f28 Feature function interface for use in scoring 2012-11-02 23:30:51 +00:00
Hieu Hoang
87b6520d82 add moses server back in. Changed ScoreComponentCollection to expose dense feature array 2012-10-22 14:41:23 +01:00
Hieu Hoang
6647ec4a41 don't compile moses server for the moment. Doesn't compile on some gcc/boost combinations 2012-10-22 11:40:26 +01:00
Kenneth Heafield
5f67527143 Bugfix vocabulary corruption on building 2012-10-19 12:00:10 +01:00
Barry Haddow
365e680115 Merge remote-tracking branch 'origin/master' into miramerge 
NB Untested

Conflicts:
	Jamroot
	moses-chart-cmd/src/Main.cpp
	moses/src/ChartManager.cpp
	moses/src/RuleTable/LoaderStandard.cpp
	moses/src/RuleTable/PhraseDictionaryALSuffixArray.cpp
	moses/src/Word.cpp
 2012-10-15 21:35:56 +01:00
Kenneth Heafield
0eb98df0fe Submodules are evil. Undo it. 2012-10-15 13:58:33 +01:00
Barry Haddow
a638736722 Reinstate Moses unit tests 2012-10-11 21:33:11 +01:00
Barry Haddow
0a950ee9f4 Merge remote branch 'github/master' into miramerge 
Compiles, but not tested. Had to disable relent filter. Strangely, it seems to contain the
whole of moses-cmd.

Conflicts:
	Jamroot
	OnDiskPt/TargetPhrase.cpp
	moses-cmd/src/Main.cpp
	moses/src/AlignmentInfo.cpp
	moses/src/AlignmentInfo.h
	moses/src/ChartTranslationOptionCollection.cpp
	moses/src/ChartTranslationOptionCollection.h
	moses/src/GenerationDictionary.cpp
	moses/src/Jamfile
	moses/src/Parameter.cpp
	moses/src/PhraseDictionary.cpp
	moses/src/StaticData.cpp
	moses/src/StaticData.h
	moses/src/TargetPhrase.h
	moses/src/TranslationSystem.cpp
	moses/src/TranslationSystem.h
	moses/src/Word.cpp
	phrase-extract/score.cpp
	regression-testing/Jamfile
	scripts/ems/experiment.meta
	scripts/ems/experiment.perl
	scripts/training/train-model.perl
 2012-09-26 22:49:33 +01:00
Kenneth Heafield
4faab6c68f Extract kenlm into a submodule, referencing lazy 2012-09-24 18:30:51 +01:00
wlin12
6202c7cc60 adding the code for relative entropy pruning to contrib/relent-filter 2012-09-07 17:33:44 +01:00
Kenneth Heafield
b317522563 Move max-order to lm directory and direct dependencies. 2012-08-18 12:07:53 -04:00
Lane Schwartz
2556dfc0e5 If --install-scripts is provided with no argument, install to $prefix/scripts 
If --prefix is specified, but --install-scripts is not, do not install scripts
 2012-08-08 16:50:33 -04:00
Lane Schwartz
a2afc98665 Add program to query the KenLM maximum n-gram order 2012-08-08 16:41:29 -04:00