moses-smt/mosesdecoder
1
1
Fork 0
mirror of https://github.com/moses-smt/mosesdecoder.git synced 2024-08-17 23:40:50 +03:00
Code Issues Packages Projects Releases Wiki Activity
17,477 Commits 136 Branches 26 Tags 147 MiB
master
Commit Graph

602 Commits

Author SHA1 Message Date
Hieu Hoang
0a8390981e delete includes for util/check.hh 2013-11-21 18:58:28 +00:00
Hieu Hoang
3ffc5e7a8f replace CHECK with UTIL_THROW_IF in moses-cmd 2013-11-18 18:27:01 +00:00
Hieu Hoang
3f8972d67b move collection of generation-table from Staticdata to GenerationDictionary 2013-11-15 17:43:41 +00:00
Hieu Hoang
2f00ae936b move collection of phrase-table from Staticdata to PhraseDictionary 2013-11-15 17:26:26 +00:00
phikoehn
ecc30bcfe2 also report lm scores, and matched n-gram length (thanks Ken!) 2013-10-13 22:05:07 +01:00
phikoehn
e6257ddc43 one more fix 2013-10-13 09:04:03 +01:00
phikoehn
8e23a3f2c1 more enriched trace 2013-10-13 06:59:05 +01:00
Hieu Hoang
6ccc17b804 speed up compiling by removing headers from StaticData.h 2013-10-03 11:05:53 +01:00
Hieu Hoang
31ce9b510e beautify 2013-09-27 09:35:24 +01:00
Hieu Hoang
deafd5922d skeleton PT 2013-09-25 16:57:01 +01:00
Hieu Hoang
00b21b4f53 placeholder for chart decoding 2013-09-22 15:36:50 +02:00
Hieu Hoang
574d716f23 placeholder for chart decoding 2013-09-22 15:09:49 +02:00
Hieu Hoang
ffbfcb5577 placeholders work in pb decoder. God i'm bored 2013-09-19 16:33:21 +02:00
Hieu Hoang
a40d9082cd more placeholder code and 'NO BEST TRANSLATION' to stderr for pb 2013-09-18 23:47:50 +02:00
Hieu Hoang
4dfb625536 beautify 2013-08-23 13:53:30 +01:00
Hieu Hoang
55bd87d38c Merge pull request #43 from jgwinnup/master 
Added option to output unknown words with prefix 'UNK'
 2013-08-15 14:20:48 -07:00
Hieu Hoang
9f92642e59 beautify 2013-08-15 21:14:04 +01:00
Hieu Hoang
841ce108a5 add phrase table caching 2013-08-15 20:20:44 +01:00
Jeremy Gwinnup
091bed03af Added option to output unknown words with prefix 'UNK' 2013-08-15 11:30:08 -04:00
Hieu Hoang
efc765ddff implement Evaluate(inputpath, input) for chart translation options. Plumbing for lattice input 2013-08-13 20:36:32 +01:00
Hieu Hoang
c50cc18f36 use input path for unknown word processing in phrase-based 2013-08-13 20:01:13 +01:00
Hieu Hoang
8756ad8ffd beautify 2013-08-09 18:17:18 +01:00
Hieu Hoang
6f7d239f13 Merge github.com:moses-smt/mosesdecoder into hieu_opt_input 2013-08-07 15:19:22 +01:00
Hieu Hoang
441acf15e6 beautify 2013-08-07 14:18:12 +01:00
Hieu Hoang
a1f5167f85 get ready to delete source phrase from TargetPhrase class 2013-08-07 11:37:40 +01:00
Hieu Hoang
66d4c2b0be source phrase is a pointer in translation options. 2013-08-07 10:28:28 +01:00
phikoehn
ab4e3c63a6 enriched trace 2013-08-07 05:31:45 +01:00
Hieu Hoang
a83c166678 delete Hypothesis::GetSourcePhrase(). Take it directly from TranslationOption::GetSourcePhrase() 2013-08-05 13:46:54 +01:00
Hieu Hoang
ea67f29b8f segfault outputting placeholder 2013-08-01 14:26:54 +01:00
Hieu Hoang
77872f7521 beautify 2013-07-30 15:04:37 +01:00
Hieu Hoang
d3aa3ee65c source and target factors for placeholders 2013-07-19 18:23:38 +01:00
Hieu Hoang
beda0767f9 support for placeholder in pb model 2013-07-19 17:58:52 +01:00
Hieu Hoang
82224a7a46 feature functions can be both sparse AND dense 2013-06-09 18:48:09 +01:00
Hieu Hoang
6249432407 beautify 2013-05-29 18:16:15 +01:00
Hieu Hoang
06a5c86695 delete LMList 2013-05-27 16:50:21 +01:00
Hieu Hoang
e8dadb58ce split classes in DummyScoreProducers into separate files 2013-05-25 00:57:37 +01:00
Hieu Hoang
a80d838b0f move feature functions into moses/FF 2013-05-24 18:02:49 +01:00
Hieu Hoang
f04ec4c56d 1. remove all code for MetaFeature from mira. 
2. in ShowWeights(), all print out dense feature weights. Don't print 'sparse' for sparse feature functions. All features functions can contains dense and sparse
 2013-05-16 19:05:08 +01:00
Hieu Hoang
0f75865faa 1. Get rid of GetSparseFeatureReporting(). Always report detailed sparse features, if they exist. 
2. Name sparse features properly in class WordTranslationFeature
 2013-05-15 11:37:21 +01:00
Hieu Hoang
981351758a delete class TranslationSystem 2013-05-11 14:13:26 +01:00
Hieu Hoang
4890e4673a get rid of FeatureFunction::unlimited. Every ff can have sparse AND dense features 2013-05-08 15:34:56 +01:00
Hieu Hoang
0425083c12 use GetWeightedScore() instead of InnerProduct(...) 2013-05-02 10:30:20 +01:00
Hieu Hoang
4894df6494 merge 2013-04-29 19:04:00 +01:00
Lane Schwartz
ef3d30146a Re-enabled commented out code. 
Hopefully this commit will now compile on all versions of Boost.
 2013-04-29 12:24:44 -04:00
Christian Buck
046e031723 Merge branch 'master' of git://github.com/moses-smt/mosesdecoder 2013-04-20 20:33:16 +01:00
Hieu Hoang
b1da4dbe0e merged 2013-04-19 15:03:34 +01:00
Barry Haddow
a8fe122e4b Merge branch 'master' of github.com:moses-smt/mosesdecoder 2013-04-19 08:35:01 +01:00
Hieu Hoang
9749ca5fd9 Fixed blank lines when using -print-alignment-info 2013-04-18 16:38:52 +01:00
Christian Buck
464b752128 Merge branch 'master' of git://github.com/moses-smt/mosesdecoder 2013-04-12 19:44:02 +01:00
Christian Buck
9c2c6c603b typo 2013-04-12 19:39:04 +01:00
Barry Haddow
c6e36f4dd7 Fix compile error 2013-04-12 16:29:52 +01:00
Hieu Hoang
44a0e52e30 fixed ShowWeights() for confusion networks. This is a reason why we should get rid of ShortNames and move to refactored moses pdq 2013-04-09 14:44:32 +01:00
Hieu Hoang
72aa0a80bd Merge github.com:moses-smt/mosesdecoder into weight-new 2013-03-25 13:51:47 +00:00
Joan Puigcerver
4a954f2be7 Fixes compilation error using libboost_1_50 
Problem spotted in https://github.com/moses-smt/mosesdecoder/issues/32 
fixed.

According to the Boost doc, nbestPath.parent_path().filename() 
returns a path object, to get the correspondent std::string 
representation, one must call one of the methods listed in:
http://www.boost.org/doc/libs/1_53_0/libs/filesystem/doc/reference.html#path-native-format-observers

native() is supposed to return the path in the specific OS
path format (using backslashes for Windows). Anyway, since we 
are considering only the filename here, the result is the same.
 2013-03-24 20:01:59 +01:00
Hieu Hoang
fd4e954322 merge 2013-03-24 09:57:36 +00:00
Hieu Hoang
db005f6503 compile error caused by different versions of boost 2013-03-23 15:58:07 +00:00
Lane Schwartz
3a4e63c558 Ensure directory exists for outputting hypergraphs 2013-03-22 12:14:28 -04:00
Lane Schwartz
b2bba0bae3 Work on compression and defaults for outputting hypergraphs. 2013-03-21 16:48:47 -04:00
Lane Schwartz
28c980d58e Allow hypergraph output to be in plain text, gzip, or bzip2. 
The output-search-graph-hypergraph flag now takes two params:

* The first param must be "none", "gzip", or "bzip2"
* The second param is the hypergraph directory,
  which must already exist
 2013-03-21 15:19:31 -04:00
Lane Schwartz
22c77f7331 Work on decreasing memory requirement for outputting hypergraph 2013-03-21 12:17:49 -04:00
Hieu Hoang
7888c7ba23 eclipse project files 2013-03-15 22:02:00 +00:00
Hieu Hoang
2d252d2dd0 memory leak 2013-03-15 16:11:15 +00:00
Hieu Hoang
3da09b921c memory leak 2013-03-15 12:30:39 +00:00
Hieu Hoang
cedb41be2a change : back to = 2013-03-14 19:06:01 +00:00
Hieu Hoang
ca43a88bd2 change feature weight delimiter from = to : 2013-03-13 18:26:24 +00:00
Hieu Hoang
2f78fe5fe5 Merge github.com:moses-smt/mosesdecoder into weight-new 2013-03-13 17:54:03 +00:00
Hieu Hoang
21c51194fa add -print-alignment-info 2013-03-13 12:12:33 +00:00
Hieu Hoang
35bffae402 Merge github.com:moses-smt/mosesdecoder into weight-new 2013-03-04 18:19:28 +00:00
Lane Schwartz
ec69acf3d4 Don't open all hypergraph output files at once. 2013-03-04 12:07:37 -05:00
Lane Schwartz
8573a66da0 Close hypergraph output files. 2013-03-04 09:15:46 -05:00
Hieu Hoang
bcecfe55db merge with Lane's search graph output 2013-02-24 00:31:29 +00:00
Hieu Hoang
30e0d1e0fa Merge github.com:moses-smt/mosesdecoder into weight-new 2013-02-22 22:39:24 +00:00
Lane Schwartz
4adeb7e338 Output feature weights to a separate file when producing hypergraph 2013-02-22 16:28:48 -05:00
Lane Schwartz
04f107fbb0 Add flag to output search graph in Kenneth's hypergraph format. 2013-02-22 16:28:48 -05:00
Lane Schwartz
774ed64f2e Work to allow output search graph in HTK standard lattice format 2013-02-22 16:28:47 -05:00
Hieu Hoang
01af82310b delete ScoreProducer 2013-02-21 20:03:35 +00:00
Hieu Hoang
ebd7b85aa6 Merge github.com:moses-smt/mosesdecoder into weight-new 2013-02-20 10:11:21 +00:00
Lane Schwartz
2e6b5e7a23 Merge branch 'master' of www:/repos/git/Decoders/mosesdecoder 2013-02-14 12:03:35 -05:00
Lane Schwartz
f9a46adc31 Specify -start-translation-id to tell Moses the line number of the first sentence 2013-02-13 15:52:40 -05:00
Hieu Hoang
f00d781c37 merge with master 2013-02-13 14:17:20 +00:00
Hieu Hoang
1a8cd20b51 -print-id for christian buck and phi 2013-02-11 18:01:33 +00:00
Hieu Hoang
6885e04b42 move feature function variables into FeatureFunction.h 2012-12-31 00:57:21 +00:00
Hieu Hoang
efbc8bb6c4 move language model from TranslationSystem back to StaticData 2012-12-20 16:38:15 +00:00
Hieu Hoang
c2d9d84ba0 move feature function variables from TranslationSystem back to StaticData 2012-12-19 14:37:00 +00:00
Hieu Hoang
667b8a495f decoder -show-weight to output consistent new format 2012-12-17 17:17:44 +00:00
hieu
05045d574c don't display unknown weight penalty when showing weight, don't usually tune. Also, change delimiter in mert extractor from : to = 2012-12-16 18:29:53 +00:00
hieu
57b33f40cf don't display unknown weight penalty when showing weight. Don't usually tune 2012-12-15 19:20:07 +00:00
hieu
5ab3c3f081 output weight formatted according to discussions with Ken & Barry 2012-12-12 12:22:13 +00:00
Hieu Hoang
d1fcb37b06 make output of feature scores the same in moses-cmd as in moses-chart-cmd 2012-12-07 13:34:44 +00:00
Hieu Hoang
3d6d53bf49 delete hardcoded if() statements to show scores in n-best list. Excluded UnknownWordPenalty and made sure PhraseModel & Generation is in particular order 2012-12-06 17:28:56 +00:00
Hieu Hoang
08ca44b34e delete hardcoded if() statements for show weights. Excluded UnknownWordPenalty and made sure PhraseModel & Generation is in particular order 2012-12-06 17:13:00 +00:00
Hieu Hoang
28b70a5697 delete hardcoded if() statements for show weights. Excluded UnknownWordPenalty and made sure PhraseModel & Generation is in particular order 2012-12-06 16:59:54 +00:00
Hieu Hoang
4f3805a0d7 change \!UnknownWordPenalty to UnknownWordPenalty 2012-12-05 11:58:21 +00:00
Hieu Hoang
cb270a76be all regression tests passed 2012-12-04 18:39:06 +00:00
Hieu Hoang
08b508d686 indentation 2012-12-04 17:54:39 +00:00
Hieu Hoang
9fe742ce52 get rid function GetScoreProducerWeightShortName(). Fails 1 regression test 2012-12-04 17:09:23 +00:00
Hieu Hoang
33105a7ba7 get rid of int argument from GetScoreProducerWeightShortName() 2012-12-04 13:08:00 +00:00
Hieu Hoang
623289465c change argument -include-alignment-in-n-best to -include-segmentation-in-n-best. Avoids confusion with word alignment 2012-11-14 16:57:33 +00:00
Hieu Hoang
f96b33de83 only include moses root when compiling 2012-11-14 13:43:04 +00:00
Hieu Hoang
0bc0b85358 resolved merge conflicts 2012-11-12 21:12:01 +00:00
Hieu Hoang
125fdbcff6 move moses-cmd/src/* to moses-cmd/ 2012-11-12 20:42:29 +00:00
Hieu Hoang
44f24bd673 resolved merge conflicts 2012-11-12 20:35:32 +00:00
Hieu Hoang
6b354f1e90 move moses/src/* to moses/ 2012-11-12 20:21:32 +00:00
Hieu Hoang
5e3ef23cef move moses/src/* to moses/ 2012-11-12 19:56:18 +00:00
Kenneth Heafield
1bae8e1701 Why was mira pulling in IOWrapper? 2012-11-12 18:27:38 +00:00
Kenneth Heafield
b9b2874abc There's no need for the _lib suffix on IOWrapper. Also, why does this library exist? 2012-11-12 18:13:29 +00:00
Barry Haddow
0d26c2f521 Sync showweights order with nbest 2012-11-09 12:11:49 +00:00
Hieu Hoang
88b2f05e03 process empty lines regardless 2012-11-09 01:26:21 -08:00
Hieu Hoang
397e704b49 terminal and non-terminal word alignments stored in 2 separate objects 2012-10-19 15:10:10 +01:00
Barry Haddow
848aafb644 Merge remote branch 'github/master' into miramerge 
Conflicts:
	moses/src/AlignmentInfo.cpp
	moses/src/AlignmentInfo.h
	moses/src/ChartHypothesis.cpp
	moses/src/ChartTrellisNode.cpp
	moses/src/LM/Implementation.cpp
	moses/src/LM/Ken.cpp
	moses/src/TargetPhrase.cpp
	moses/src/TargetPhrase.h
 2012-10-08 17:54:59 +01:00
Kenneth Heafield
483e6b666b Print usage information before exit 2012-10-05 17:49:52 +01:00
Barry Haddow
0a950ee9f4 Merge remote branch 'github/master' into miramerge 
Compiles, but not tested. Had to disable relent filter. Strangely, it seems to contain the
whole of moses-cmd.

Conflicts:
	Jamroot
	OnDiskPt/TargetPhrase.cpp
	moses-cmd/src/Main.cpp
	moses/src/AlignmentInfo.cpp
	moses/src/AlignmentInfo.h
	moses/src/ChartTranslationOptionCollection.cpp
	moses/src/ChartTranslationOptionCollection.h
	moses/src/GenerationDictionary.cpp
	moses/src/Jamfile
	moses/src/Parameter.cpp
	moses/src/PhraseDictionary.cpp
	moses/src/StaticData.cpp
	moses/src/StaticData.h
	moses/src/TargetPhrase.h
	moses/src/TranslationSystem.cpp
	moses/src/TranslationSystem.h
	moses/src/Word.cpp
	phrase-extract/score.cpp
	regression-testing/Jamfile
	scripts/ems/experiment.meta
	scripts/ems/experiment.perl
	scripts/training/train-model.perl
 2012-09-26 22:49:33 +01:00
Barry Haddow
1ce788e2b8 Fixes for regression tests 2012-09-26 14:40:06 +01:00
Kenneth Heafield
15b0ae1aef Single-threaded compilation fixes 2012-09-21 18:34:48 -04:00
Barry Haddow
bfae09725c Add option to dump OOVs to file 2012-09-21 08:55:37 +01:00
Barry Haddow
54433bb221 Take notice of calc in t-option flag 2012-09-04 17:50:30 +01:00
Lane Schwartz
357845c1a4 Removed duplicate usage printout when moses is called with no parameters. 2012-09-04 17:32:55 +01:00
Jonathan Clark
6f008096ea Use Moses' Timer class, which is based on wall-time instead of the C clock() function, which measures clock ticks used within the *process* rather than each thread. This makes timings more meaningful in multi-threaded mode. However, these timings will still be inaccurate on Windows (second-level only). 2012-08-10 15:32:00 -04:00
Hieu Hoang
97f46537b7 wrap phi's suffix array implementation and extraction method in a wrapper class. Compiles 2012-07-30 21:07:19 +01:00
Barry Haddow
2b4e61d826 Merge branch 'trunk' into miramerge 
Compiles, not tested.

Conflicts:
	Jamroot
	OnDiskPt/PhraseNode.h
	OnDiskPt/TargetPhrase.cpp
	OnDiskPt/TargetPhrase.h
	OnDiskPt/TargetPhraseCollection.cpp
	mert/BleuScorer.cpp
	mert/Data.cpp
	mert/FeatureData.cpp
	moses-chart-cmd/src/Main.cpp
	moses/src/AlignmentInfo.h
	moses/src/ChartManager.cpp
	moses/src/LM/Ken.cpp
	moses/src/LM/Ken.h
	moses/src/LMList.h
	moses/src/LexicalReordering.h
	moses/src/PhraseDictionaryTree.h
	moses/src/ScoreIndexManager.h
	moses/src/StaticData.h
	moses/src/TargetPhrase.h
	moses/src/Word.cpp
	scripts/ems/experiment.meta
	scripts/ems/experiment.perl
	scripts/training/train-model.perl
 2012-07-17 13:36:50 +01:00
Eva Hasler
b00e4b3115 move deps into IOWrapper_lib 2012-07-03 16:22:59 +01:00
Hieu Hoang
75e038f4cf create namespace for all classes 2012-07-02 17:05:11 +01:00
Hieu Hoang
121e258e84 namespace all classes in mert directory 2012-06-30 21:39:10 +01:00
Hieu Hoang
b1ce27d01f commented all classes in OnDiskPt and moses-cmd 2012-06-28 23:19:28 -04:00
Eva Hasler
e1c1a5343c merge 2012-06-07 11:16:52 +01:00
Eva Hasler
6a6a35c65e fix start weights in experiment.perl, add hypothesis queue for picking hope and fear translations, add variations to 1slack formulation 2012-06-01 01:49:42 +01:00
Hieu Hoang
465c5cbf97 move all executables into bin 2012-05-31 12:55:05 +01:00
Colin Cherry
3c44d04baf Merge branch 'master' into miramerge 
Conflicts:
	Jamroot
	mert/FeatureStats.cpp
	moses-cmd/src/IOWrapper.h
	scripts/training/mert-moses.pl
	scripts/training/train-model.perl.missing_bin_dir
 2012-05-30 12:39:53 -04:00
Hieu Hoang
45870348ff xcode build supports threads. move 'using namespace' out from .h file to stop namespace pollution 2012-05-30 12:47:20 +01:00
Hieu Hoang
ef26388aff eclipse project 2012-05-28 17:29:46 +01:00
Barry Haddow
c397d2068b Merge branch 'trunk' into miramerge. Still to fix build. 
Conflicts:
	Jamroot
	mert/Data.cpp
	mert/Data.h
	mert/FeatureArray.cpp
	mert/FeatureArray.h
	mert/FeatureData.cpp
	mert/FeatureData.h
	mert/FeatureStats.cpp
	mert/FeatureStats.h
	mert/mert.cpp
	moses-chart-cmd/src/IOWrapper.h
	moses-chart-cmd/src/Main.cpp
	moses-cmd/src/IOWrapper.cpp
	moses-cmd/src/IOWrapper.h
	moses-cmd/src/Main.cpp
	moses/src/GlobalLexicalModel.cpp
	moses/src/Jamfile
	moses/src/Parameter.cpp
	moses/src/PhraseDictionary.cpp
	moses/src/ScoreIndexManager.h
	moses/src/TargetPhrase.h
	regression-testing/tests/phrase.lexicalized-reordering-bin/truth/results.txt
	regression-testing/tests/phrase.lexicalized-reordering-cn/truth/results.txt
	regression-testing/tests/phrase.lexicalized-reordering/truth/results.txt
	regression-testing/tests/phrase.multiple-translation-system-lr/truth/results.txt
	regression-testing/tests/phrase.show-weights.lex-reorder/truth/results.txt
	regression-testing/tests/phrase.show-weights/truth/results.txt
	scripts/ems/experiment.meta
	scripts/ems/experiment.perl
	scripts/training/filter-model-given-input.pl
	scripts/training/mert-moses.pl
 2012-05-24 21:11:35 +01:00
Hieu Hoang
4888e114ed Check factor is not null before outputting 2012-05-17 16:25:53 +01:00
Hieu Hoang
22ca90bd8a Check factor is not null before outputting 2012-05-17 16:25:53 +01:00
Eva Hasler
30deedde9f changed permission, everything changed.. 2012-05-10 18:54:24 +01:00
Jonathan Clark
19328231ff Add phrase segmentation information to n-best list 2012-04-18 17:09:02 -04:00
Jonathan Clark
bb36565e78 Add phrase segmentation information to n-best list 2012-04-18 17:09:02 -04:00
Eva Hasler
56f1101dfc ignore empty input lines 2012-04-07 21:03:30 +01:00
Lane Schwartz
1c455a2877 Fixed memory leak for single-threaded mode. 2012-03-30 15:25:42 -04:00
Lane Schwartz
8a03fd63ba Fixed memory leak for single-threaded mode. 2012-03-30 15:25:42 -04:00
Eva Hasler
03d3cb6a7a adapt Jamfiles, don't output bleu score feature in nbest 2012-03-29 20:58:53 +01:00
Eva Hasler
f5a4d61fd4 resolving conflicts 2012-03-27 14:24:49 +01:00
Eva Hasler
c64b8bc37f revert changes to nbest output for sparse features 2012-03-20 13:13:30 +00:00
Eva Hasler
da538961d4 nbest output for sparse features: report non-zero weights instead of scores 2012-03-11 16:26:25 +00:00
Eva Hasler
cb4a0d36bf change output format for sparse features 2012-03-08 14:58:12 +00:00
Barry Haddow
5a17ef82b3 Merge remote branch 'github/miramerge' into bjam 
Conflicts:
	mira/Decoder.h
	mira/Main.cpp
	mira/Main.h
	moses-chart-cmd/src/IOWrapper.cpp
	moses-chart-cmd/src/Main.cpp
	moses-cmd/src/Main.cpp
	moses/src/BleuScoreFeature.cpp
	moses/src/BleuScoreFeature.h
	moses/src/TargetNgramFeature.h
 2012-02-01 10:12:16 +00:00
Michal Hrusecky
ade1aa4acc Use make_pair without type 
Replacing typed make_pair with non-typed version to fix compilation
issues.
 2012-01-31 17:00:01 +01:00
Michal Hrusecky
c4397c1491 Use make_pair without type 
Replacing typed make_pair with non-typed version to fix compilation
issues.
 2012-01-31 17:00:01 +01:00
Barry Haddow
1e10bb7ef7 Merge from miramerge. 
Reverted ChartHypothesis as it breaks chart-decoding.
 2012-01-20 15:35:55 +00:00
Eva Hasler
b704381a42 restore order of scores returned by decoder and in nbestlist 2012-01-18 12:26:51 +00:00
Kenneth Heafield
06648fd832 Surround mains in try-catch blocks. 2012-01-13 10:20:42 -05:00