moses-smt/mosesdecoder
1
1
Fork 0
mirror of https://github.com/moses-smt/mosesdecoder.git synced 2024-12-28 14:32:38 +03:00
Code Issues Packages Projects Releases Wiki Activity
14,280 Commits 136 Branches 26 Tags 147 MiB
1dabb7c8fc
Commit Graph

224 Commits

Author SHA1 Message Date
Ulrich Germann
a86d49fc88 Added bias to bitext sampling. 2014-09-08 09:26:08 +01:00
Ulrich Germann
015d690b6f Added try-align2 to --with-mm and new switch --with-icu. 2014-09-08 09:26:08 +01:00
Hieu Hoang
b51062672f Roll out mixed syntax 2014-08-29 15:55:48 +01:00
Ulrich Germann
215041dfff Bug fix in Jamroot with respect to compiling without --with-cmph. 2014-08-05 02:19:43 +01:00
Barry Haddow
2826ccc001 Boost version update, specify filesystem v3 
Shout if it breaks your favourite platform!
 2014-07-23 17:22:14 +01:00
Ulrich Germann
5915e2b1cb Merge branch 'master' of https://github.com/moses-smt/mosesdecoder 2014-07-17 15:53:07 +01:00
Hieu Hoang
53073726d6 compiles with bjam 2014-07-15 15:26:50 +01:00
Ulrich Germann
4d41211c2c Major overhaul of Mmsapt. Reorganization of old and addition of new features in phrase tables. Many critical bug fixes. 2014-07-09 02:41:28 +01:00
Ulrich Germann
73081786bc Name change: lookup_mmsapt -> ptable-lookup 2014-06-28 20:08:41 +01:00
Hieu Hoang
2c6cdb5889 bjam for ProbingPT 2014-06-13 09:48:32 +01:00
Hieu Hoang
ad4825ad44 integrated probing pt into decoder 2014-06-11 14:25:18 +01:00
Ulrich Germann
ce853731ae Added mmsapt lookup utility. 2014-06-05 01:38:55 +01:00
Ulrich Germann
21f141ce0d Merge branch 'master' into dynamic-phrase-tables 
Conflicts:
	moses/InputPath.cpp
	moses/Word.cpp
 2014-04-08 14:58:29 +01:00
Rico Sennrich
45630a5851 various optimizations to make CYK+ parser several times faster and eat less memory. 
speed-up of decoding depends on how much time is spent in parser:
10-50% speed-up for string-to-tree systems observed (more on long sentences and with high max-chart-span).

if you only use hiero or string-to-tree models (but none with source syntax), use compile-option --unlabelled-source for (small) efficiency gains.
 2014-03-21 11:12:24 +00:00
Ulrich Germann
c91fb5cc84 Added mmlex-lookup utility program. 2014-03-19 20:41:29 +00:00
Ulrich Germann
394a40b934 Added program calc-coverage. 2014-03-18 12:17:07 +00:00
Ulrich Germann
9025ac065f Added utilities: 
- mam2symal converts memory-mapped word alignments to symal format
- mam_verify performs a sanity check on memory-mapped word alignments
 2014-03-12 08:06:55 +00:00
Ulrich Germann
a7c85780ee Merge branch 'master' into dynamic-phrase-tables 
Conflicts:
	phrase-extract/score-main.cpp
 2014-03-10 14:25:45 +00:00
Kenneth Heafield
952bb76b77 We weren't linking against tcmalloc... 2014-03-01 07:42:32 +00:00
Ulrich Germann
4b95c3a906 Merge branch 'dynamic-phrase-tables' of ssh://thor//home/germann/git/mosesdecoder into dynamic-phrase-tables 
due to resetting the location of the remote repository.
 2014-02-21 01:09:38 +00:00
Ulrich Germann
b410026804 Added test program try-align. 2014-02-20 23:56:45 +00:00
Ulrich Germann
af007c4cf0 Minor fixes related to mm. 2014-02-08 18:18:45 +00:00
Hieu Hoang
2387f3123c bjam 2014-01-23 13:14:59 +00:00
Hieu Hoang
5800ff848c bjam 2014-01-22 20:57:06 +00:00
Rico Sennrich
c1d8f6e267 Revert "testing the waters for C++11 adoption" 
This reverts commit d2d508184e.

there's problems with gcc 4.5, and apparently different problems with new boost versions; sticking with C++03 for the time being.
 2014-01-15 16:16:11 +00:00
Rico Sennrich
d2d508184e testing the waters for C++11 adoption 2014-01-14 17:01:46 +00:00
Hieu Hoang
ca42b9b01f tcmalloc /buhomecanico 2014-01-06 11:52:21 +00:00
Lane Schwartz
f57ac23be9 Added comment in Jamroot on how to compile against NPLM 2013-12-20 15:06:49 -05:00
Lane Schwartz
f40e1ac8f8 Added support for hierarchical maximum entropy LM through SRILM. 
Currently, the code compiles, but has not been run or tested.

    Steps to compile with maxent LM are below:

    - Download SRILM 1.6.0

    - Download the maxent patch from
      http://www.phon.ioc.ee/~tanela/srilm-me/srilm-1.6.0-me.patch

    - Download the required libLBFGS library from
      http://www.chokkan.org/software/liblbfgs

    - Edit SRILM's sbin/machine-type script
      to return MACHINE_TYPE=i686-m64 for case x86_64

    - Compile libLBFGS and install it in the /tools/SRILM/SRILM-1.7.0 directory.
      Make sure that the liblbfgs.* library files are in /tools/SRILM/SRILM-1.7.0/lib//

    - In the /tools/SRILM/SRILM-1.7.0 dir, apply the maxent patch to SRILM:
      patch -p1 < srilm-1.6.0-me.patch

    - Point the appropriate SRILM makefile
      to see libLBFGS's include and lib dirs, following the instructions at:
      http://www.phon.ioc.ee/dokuwiki/doku.php?id=people:tanel:srilm-me.en

    - Compile SRILM

    - Compile Moses using --with-srilm and --with-maxent-srilm:
      ./bjam --with-srilm=/tools/SRILM/SRILM-1.7.0 --with-maxent-srilm=true

    NOTE: The above steps were for my original integration.
          Current versions of SRILM (such as 1.7.1) include the maxent patch.
          As such, it should be sufficient to compile a current version of SRILM
          with support form maxent enabled,
          and then compile Moses using --with-srilm and --with-maxent-srilm:
          ./bjam --with-srilm=/tools/SRILM/SRILM-1.7.0 --with-maxent-srilm=true
 2013-12-20 15:04:09 -05:00
Ulrich Germann
290ae0e763 Added bjam command line switch to compile with memory-mapped data structures. Aperently this doesn't work on MacOS X. 2013-10-28 21:49:36 +00:00
Hieu Hoang
0eff7344d7 move Uli's new pt into it's own folder. Comment out bjam compile. Doesn't work on MacOSX yet 2013-09-26 10:28:52 +01:00
=
d8686f79db Reformatting. 2013-09-25 00:11:11 +01:00
Kenneth Heafield
5e506ed914 Make dl have the same linkage as glibc 2013-09-12 09:27:01 +01:00
Hieu Hoang
b534e8c4b7 f*cking link error 2013-09-12 09:17:18 +01:00
Kenneth Heafield
6b5c1a09e4 unwind library, cleanup 2013-08-22 10:16:12 +01:00
Kenneth Heafield
37c22cd4da Also use unwind for for tcmalloc 2013-08-21 14:29:00 +01:00
Kenneth Heafield
b1d7340b78 Add --full-tcmalloc to link against the non-minimal tcmalloc library 2013-08-21 10:16:33 +01:00
Hieu Hoang
5fa6e506b1 Jamroot error 2013-08-15 15:39:40 +01:00
phikoehn
b368085609 xml constraint 2013-08-15 11:46:45 +01:00
Barry Haddow
c127c58e9b fix to single thread build 2013-07-25 15:56:20 +01:00
Hieu Hoang
418652fe1d iconv for SRILM 1.7.0 on MacOSX 2013-07-01 10:57:06 +01:00
Kenneth Heafield
215e93377c Move iconv to everything but for OS X only 2013-07-01 09:01:56 +01:00
Kenneth Heafield
0d9020fefe Add message for missing tcmalloc and improve install directions 2013-06-18 13:31:37 -04:00
Lane Schwartz
aacc936204 Adding unit tests to Backward LM 2013-04-29 09:18:46 -04:00
Kenneth Heafield
030e2d40b2 Use tcmalloc_minimal and tell people how to build it 2012-11-14 20:29:55 +00:00
Kenneth Heafield
be7e78c6bf Use tcmalloc with multithread if detected. 
Change option and documentation to --without-tcmalloc.
 2012-11-14 20:01:07 +00:00
Hieu Hoang
f96b33de83 only include moses root when compiling 2012-11-14 13:43:04 +00:00
Hieu Hoang
0bc0b85358 resolved merge conflicts 2012-11-12 21:12:01 +00:00
Hieu Hoang
125fdbcff6 move moses-cmd/src/* to moses-cmd/ 2012-11-12 20:42:29 +00:00
Hieu Hoang
5e3ef23cef move moses/src/* to moses/ 2012-11-12 19:56:18 +00:00
Barry Haddow
62fa6d6f28 Feature function interface for use in scoring 2012-11-02 23:30:51 +00:00
Hieu Hoang
87b6520d82 add moses server back in. Changed ScoreComponentCollection to expose dense feature array 2012-10-22 14:41:23 +01:00
Hieu Hoang
6647ec4a41 don't compile moses server for the moment. Doesn't compile on some gcc/boost combinations 2012-10-22 11:40:26 +01:00
Kenneth Heafield
5f67527143 Bugfix vocabulary corruption on building 2012-10-19 12:00:10 +01:00
Barry Haddow
365e680115 Merge remote-tracking branch 'origin/master' into miramerge 
NB Untested

Conflicts:
	Jamroot
	moses-chart-cmd/src/Main.cpp
	moses/src/ChartManager.cpp
	moses/src/RuleTable/LoaderStandard.cpp
	moses/src/RuleTable/PhraseDictionaryALSuffixArray.cpp
	moses/src/Word.cpp
 2012-10-15 21:35:56 +01:00
Kenneth Heafield
0eb98df0fe Submodules are evil. Undo it. 2012-10-15 13:58:33 +01:00
Barry Haddow
a638736722 Reinstate Moses unit tests 2012-10-11 21:33:11 +01:00
Barry Haddow
0a950ee9f4 Merge remote branch 'github/master' into miramerge 
Compiles, but not tested. Had to disable relent filter. Strangely, it seems to contain the
whole of moses-cmd.

Conflicts:
	Jamroot
	OnDiskPt/TargetPhrase.cpp
	moses-cmd/src/Main.cpp
	moses/src/AlignmentInfo.cpp
	moses/src/AlignmentInfo.h
	moses/src/ChartTranslationOptionCollection.cpp
	moses/src/ChartTranslationOptionCollection.h
	moses/src/GenerationDictionary.cpp
	moses/src/Jamfile
	moses/src/Parameter.cpp
	moses/src/PhraseDictionary.cpp
	moses/src/StaticData.cpp
	moses/src/StaticData.h
	moses/src/TargetPhrase.h
	moses/src/TranslationSystem.cpp
	moses/src/TranslationSystem.h
	moses/src/Word.cpp
	phrase-extract/score.cpp
	regression-testing/Jamfile
	scripts/ems/experiment.meta
	scripts/ems/experiment.perl
	scripts/training/train-model.perl
 2012-09-26 22:49:33 +01:00
Kenneth Heafield
4faab6c68f Extract kenlm into a submodule, referencing lazy 2012-09-24 18:30:51 +01:00
wlin12
6202c7cc60 adding the code for relative entropy pruning to contrib/relent-filter 2012-09-07 17:33:44 +01:00
Kenneth Heafield
b317522563 Move max-order to lm directory and direct dependencies. 2012-08-18 12:07:53 -04:00
Lane Schwartz
2556dfc0e5 If --install-scripts is provided with no argument, install to $prefix/scripts 
If --prefix is specified, but --install-scripts is not, do not install scripts
 2012-08-08 16:50:33 -04:00
Lane Schwartz
a2afc98665 Add program to query the KenLM maximum n-gram order 2012-08-08 16:41:29 -04:00
Lane Schwartz
da5429318b KenLM maximum n-gram order can now be set via a compile-time flag 2012-08-08 16:22:13 -04:00
Kenneth Heafield
b52ed110b0 Build changes. --static will force static linkage. 
--without-libsegfault will skip libSegFault.
 2012-08-06 10:41:49 -04:00
Marcin Junczys-Dowmunt
767969af6d Jamroot 2012-08-04 22:49:42 +02:00
Marcin Junczys-Dowmunt
a245e9ea55 Correction of jamfiles etc. 2012-08-04 15:39:30 +02:00
Marcin Junczys-Dowmunt
88a860d776 Warnings, build issues 2012-08-03 19:29:21 +02:00
Marcin Junczys-Dowmunt
a35791893f Updated Jamsfiles, build instructions etc. 2012-08-03 16:38:45 +02:00
Kenneth Heafield
141861842d Change header file glob from an opt-out to an opt-in for Ondrej Bojar 2012-07-30 15:50:30 -04:00
Barry Haddow
2b4e61d826 Merge branch 'trunk' into miramerge 
Compiles, not tested.

Conflicts:
	Jamroot
	OnDiskPt/PhraseNode.h
	OnDiskPt/TargetPhrase.cpp
	OnDiskPt/TargetPhrase.h
	OnDiskPt/TargetPhraseCollection.cpp
	mert/BleuScorer.cpp
	mert/Data.cpp
	mert/FeatureData.cpp
	moses-chart-cmd/src/Main.cpp
	moses/src/AlignmentInfo.h
	moses/src/ChartManager.cpp
	moses/src/LM/Ken.cpp
	moses/src/LM/Ken.h
	moses/src/LMList.h
	moses/src/LexicalReordering.h
	moses/src/PhraseDictionaryTree.h
	moses/src/ScoreIndexManager.h
	moses/src/StaticData.h
	moses/src/TargetPhrase.h
	moses/src/Word.cpp
	scripts/ems/experiment.meta
	scripts/ems/experiment.perl
	scripts/training/train-model.perl
 2012-07-17 13:36:50 +01:00
Kenneth Heafield
272aeaa67e Merge some build system changes that were made for cdec 2012-06-21 09:52:25 -04:00
Colin Cherry
07a5c67ebc Merge branch 'master' into miramerge 
Conflicts:
	Jamroot
	misc/queryPhraseTable.cpp
	scripts/training/train-model.perl
 2012-06-14 17:08:16 -04:00
Barry Haddow
06f7615779 Merge branch 'miramerge' of github.com:moses-smt/mosesdecoder into miramerge 2012-06-07 00:26:47 +01:00
Barry Haddow
e53aee1275 Fix mpi build 2012-06-07 00:26:22 +01:00
Hieu Hoang
86a1886cb4 get rid of mention of --with-giza arg 2012-06-05 18:15:43 +01:00
Kenneth Heafield
833435561d Warning about dist directory. 2012-05-31 19:13:01 -04:00
Hieu Hoang
417d5ee57b move biconcor to / 2012-05-31 23:32:24 +01:00
Hieu Hoang
a5ca652a76 move c++ code out of /script/ to / 2012-05-31 17:58:10 +01:00
Hieu Hoang
4eef94b121 move c++ code out of /script/ to / 2012-05-31 17:24:06 +01:00
Hieu Hoang
4c88377cbe change CreateOnDisk to CreateOnDiskPt 2012-05-31 14:28:58 +01:00
Hieu Hoang
465c5cbf97 move all executables into bin 2012-05-31 12:55:05 +01:00
Colin Cherry
3c44d04baf Merge branch 'master' into miramerge 
Conflicts:
	Jamroot
	mert/FeatureStats.cpp
	moses-cmd/src/IOWrapper.h
	scripts/training/mert-moses.pl
	scripts/training/train-model.perl.missing_bin_dir
 2012-05-30 12:39:53 -04:00
Hieu Hoang
8844be6f25 Move queryOnDiskPt from contrib to OnDiskPt folder. Easier to amend Jam file 2012-05-25 13:41:06 +01:00
Barry Haddow
c397d2068b Merge branch 'trunk' into miramerge. Still to fix build. 
Conflicts:
	Jamroot
	mert/Data.cpp
	mert/Data.h
	mert/FeatureArray.cpp
	mert/FeatureArray.h
	mert/FeatureData.cpp
	mert/FeatureData.h
	mert/FeatureStats.cpp
	mert/FeatureStats.h
	mert/mert.cpp
	moses-chart-cmd/src/IOWrapper.h
	moses-chart-cmd/src/Main.cpp
	moses-cmd/src/IOWrapper.cpp
	moses-cmd/src/IOWrapper.h
	moses-cmd/src/Main.cpp
	moses/src/GlobalLexicalModel.cpp
	moses/src/Jamfile
	moses/src/Parameter.cpp
	moses/src/PhraseDictionary.cpp
	moses/src/ScoreIndexManager.h
	moses/src/TargetPhrase.h
	regression-testing/tests/phrase.lexicalized-reordering-bin/truth/results.txt
	regression-testing/tests/phrase.lexicalized-reordering-cn/truth/results.txt
	regression-testing/tests/phrase.lexicalized-reordering/truth/results.txt
	regression-testing/tests/phrase.multiple-translation-system-lr/truth/results.txt
	regression-testing/tests/phrase.show-weights.lex-reorder/truth/results.txt
	regression-testing/tests/phrase.show-weights/truth/results.txt
	scripts/ems/experiment.meta
	scripts/ems/experiment.perl
	scripts/training/filter-model-given-input.pl
	scripts/training/mert-moses.pl
 2012-05-24 21:11:35 +01:00
Kenneth Heafield
e3a155494b Fix bjam from any directory. Add iostreams dependency to score for Hieu. 2012-05-10 12:18:49 -04:00
Kenneth Heafield
952f4b2523 Fix bjam from any directory. Add iostreams dependency to score for Hieu. 2012-05-10 12:18:49 -04:00
Kenneth Heafield
36f31a017b Refactor Jamroot to separate Moses high-level and common low-level. 2012-05-07 12:58:34 -04:00
Kenneth Heafield
8fc321cb7b Refactor Jamroot to separate Moses high-level and common low-level. 2012-05-07 12:58:34 -04:00
Kenneth Heafield
265f2207fc Only add -static to clang on linux 2012-05-06 20:13:19 -04:00
Kenneth Heafield
28b1fbcf5d Only add -static to clang on linux 2012-05-06 20:13:19 -04:00
Kenneth Heafield
32e5d5e170 Apparently bjam doesn't pass -static to clang 2012-05-04 19:06:03 -04:00
Kenneth Heafield
2e7e28092c Apparently bjam doesn't pass -static to clang 2012-05-04 19:06:03 -04:00
Kenneth Heafield
d3a8d281a9 Fix previous.sh for people who don't have bjam installed 2012-04-02 15:20:56 -04:00
Kenneth Heafield
1b858ef085 Fix previous.sh for people who don't have bjam installed 2012-04-02 15:20:56 -04:00
Kenneth Heafield
a54f7f855f Save bjam arguments to previous.sh. 2012-03-30 19:57:05 -04:00
Kenneth Heafield
40c5888723 Save bjam arguments to previous.sh. 2012-03-30 19:57:05 -04:00
Kenneth Heafield
05909cc7f4 Optional header installation 2012-02-13 14:31:37 -05:00
Kenneth Heafield
d62f301345 Optional header installation 2012-02-13 14:31:37 -05:00
Phil Williams
8462f24b72 Restore the "--with-tcmalloc" configuration option that was lost 
in the transition from autotools to bjam.
 2012-01-23 11:34:46 +00:00