moses-smt/mosesdecoder
1
1
Fork 0
mirror of https://github.com/moses-smt/mosesdecoder.git synced 2024-12-28 22:45:50 +03:00
Code Issues Packages Projects Releases Wiki Activity
9,484 Commits 136 Branches 26 Tags 147 MiB
1ffae8010d
Commit Graph

557 Commits

Author SHA1 Message Date
Tetsuo Kiso
8fdec9bf30 Use boost::unordered_map instead of std::map. 
For storing the word vocabulary used in computation of
BLEU scores. This change will reduce the running time
of extractor about 2-3 seconds (9% reduction).
 2012-12-07 05:12:24 +09:00
Tetsuo Kiso
6c04c4ad9c Add more tests to the Data class. 2012-12-07 02:46:59 +09:00
Tetsuo Kiso
c7f6e38326 Use FilePiece to load N-best lists. 
Since FilePiece is friendly with StringPiece.
 2012-12-07 02:39:02 +09:00
Tetsuo Kiso
38e145e556 Use util::TokenIter to tokenize n-best lists. 
Reduce creating std::string objects, too. In both ScoreArray
and FeatureArray classes, the private members to track sentence
indices (namely, "m_index") were unnecessarily declared as
std::string, but it's better to directly declare them as 'int'.
 2012-12-07 01:39:22 +09:00
Tetsuo Kiso
cd3fb3b831 Untabify. 2012-12-06 23:46:22 +09:00
Tetsuo Kiso
ac045a11c1 Speed up N-gram counts when running extractor. 
By replacing std::map with boost::unordered_map.

Runtime of extractor on 100-best lists of 2679 sentences:

Before:
real    0m35.314s
user    0m34.030s
sys     0m1.280s

Ater:
real    0m26.729s
user    0m25.420s
sys     0m1.310s
 2012-12-06 22:08:33 +09:00
Hieu Hoang
f96b33de83 only include moses root when compiling 2012-11-14 13:43:04 +00:00
Hieu Hoang
f8438f80cc move moses/src/* to moses/ 2012-11-12 20:30:39 +00:00
Hieu Hoang
5e3ef23cef move moses/src/* to moses/ 2012-11-12 19:56:18 +00:00
Kenneth Heafield
e9eb7dd021 More shared build fixes 2012-11-07 23:28:42 +01:00
Kenneth Heafield
d7ecd0be1a Remove bleu_lib target. 2012-11-07 23:21:59 +01:00
Tetsuo Kiso
96f7b42eb9 Move implementation details from the header to .cpp file. 
Also add const to variables that we don't want to change.
 2012-11-05 01:24:16 +09:00
Tetsuo Kiso
cccfb9a0c9 Using namespace std in a header file pollutes the global namespace. 
Using directives should be put into the implementation files.
 2012-11-05 00:43:36 +09:00
Kenneth Heafield
dc31857bbc Reduce header pollution 2012-10-30 20:25:05 +00:00
Barry Haddow
848aafb644 Merge remote branch 'github/master' into miramerge 
Conflicts:
	moses/src/AlignmentInfo.cpp
	moses/src/AlignmentInfo.h
	moses/src/ChartHypothesis.cpp
	moses/src/ChartTrellisNode.cpp
	moses/src/LM/Implementation.cpp
	moses/src/LM/Ken.cpp
	moses/src/TargetPhrase.cpp
	moses/src/TargetPhrase.h
 2012-10-08 17:54:59 +01:00
Arianna Bisazza
c78d571561 fixed bug in permutation scores due to scientific notation 2012-10-08 16:15:59 +01:00
Barry Haddow
0a950ee9f4 Merge remote branch 'github/master' into miramerge 
Compiles, but not tested. Had to disable relent filter. Strangely, it seems to contain the
whole of moses-cmd.

Conflicts:
	Jamroot
	OnDiskPt/TargetPhrase.cpp
	moses-cmd/src/Main.cpp
	moses/src/AlignmentInfo.cpp
	moses/src/AlignmentInfo.h
	moses/src/ChartTranslationOptionCollection.cpp
	moses/src/ChartTranslationOptionCollection.h
	moses/src/GenerationDictionary.cpp
	moses/src/Jamfile
	moses/src/Parameter.cpp
	moses/src/PhraseDictionary.cpp
	moses/src/StaticData.cpp
	moses/src/StaticData.h
	moses/src/TargetPhrase.h
	moses/src/TranslationSystem.cpp
	moses/src/TranslationSystem.h
	moses/src/Word.cpp
	phrase-extract/score.cpp
	regression-testing/Jamfile
	scripts/ems/experiment.meta
	scripts/ems/experiment.perl
	scripts/training/train-model.perl
 2012-09-26 22:49:33 +01:00
Arianna Bisazza
ff276e9911 Fixed several bugs in LRscore-MERT. Namely, solved a float-to-int conversion; added hypothesis counter to the scores file to enable later computation of average reordering score; fixed special case of 1-word hypothesis; enabled reading of word-based alignments from n-best-list. 2012-09-24 15:40:18 +02:00
Colin Cherry
ae6ac1c2ae Merge branch 'master' of https://github.com/moses-smt/mosesdecoder 2012-09-14 14:07:53 -04:00
Colin Cherry
3fa95c022b Addad a "--safe-hope" option to kbmira. 
This will limit the influence of model score on oracle (hope) selection.
Good for cases whith extremely large feature values. May make it the defult.
 2012-09-14 13:58:28 -04:00
Barry Haddow
021f5702a7 remove obsolete file 2012-09-13 22:21:31 +01:00
Barry Haddow
2b4e61d826 Merge branch 'trunk' into miramerge 
Compiles, not tested.

Conflicts:
	Jamroot
	OnDiskPt/PhraseNode.h
	OnDiskPt/TargetPhrase.cpp
	OnDiskPt/TargetPhrase.h
	OnDiskPt/TargetPhraseCollection.cpp
	mert/BleuScorer.cpp
	mert/Data.cpp
	mert/FeatureData.cpp
	moses-chart-cmd/src/Main.cpp
	moses/src/AlignmentInfo.h
	moses/src/ChartManager.cpp
	moses/src/LM/Ken.cpp
	moses/src/LM/Ken.h
	moses/src/LMList.h
	moses/src/LexicalReordering.h
	moses/src/PhraseDictionaryTree.h
	moses/src/ScoreIndexManager.h
	moses/src/StaticData.h
	moses/src/TargetPhrase.h
	moses/src/Word.cpp
	scripts/ems/experiment.meta
	scripts/ems/experiment.perl
	scripts/training/train-model.perl
 2012-07-17 13:36:50 +01:00
bhaddow
d0f1c15105 enable single character option 2012-07-12 19:47:57 +01:00
Barry Haddow
c303142ab2 option to skip duplicate removal 2012-07-12 19:08:55 +01:00
Colin Cherry
662e7e7f64 As requested by my bosses: added NRC copyright to kbmira. 2012-07-10 13:13:50 -04:00
Hieu Hoang
7d664b745e Integrate Lexi's LR Score into tuning 2012-07-10 09:25:00 +01:00
Eva Hasler
027a20730e merge Jamfiles 2012-07-04 11:49:07 +01:00
Hieu Hoang
75e038f4cf create namespace for all classes 2012-07-02 17:05:11 +01:00
Hieu Hoang
b5aa04feb7 compile error 2012-07-02 10:23:26 +01:00
Hieu Hoang
121e258e84 namespace all classes in mert directory 2012-06-30 21:39:10 +01:00
Hieu Hoang
e3dd3a8d2c namespace all classes in mert directory 2012-06-30 20:23:45 +01:00
Colin Cherry
65df386581 Merge branch 'master' of https://github.com/moses-smt/mosesdecoder 2012-06-26 17:07:27 -04:00
Colin Cherry
58c3280c2c HypPackEnumerator now stores MiraFeatureVectors, as opposed to 
FeatureDataItems. Uses roughly half the memory.
 2012-06-26 17:02:32 -04:00
Hieu Hoang
3c7b7ac9f5 rollback 2012-06-26 16:31:38 -04:00
Colin Cherry
32299593fa Added debugging info to kbmira. 2012-06-26 16:29:20 -04:00
Hieu Hoang
153e80053c lock m_vocab variable access in Encode() and Lookup(). Other functions are still not threadsafe 2012-06-26 13:33:50 -04:00
Hieu Hoang
00f018a477 Merge https://github.com/moses-smt/mosesdecoder into lrscore 2012-06-25 16:57:17 -04:00
Hieu Hoang
2a03f275a3 change regression data download to git instead of download from edin server. 
Minor change in mert/trimStr() function to prevent warning
 2012-06-25 16:03:11 -04:00
Hieu Hoang
8498b17a41 gcc version-specific error 2012-06-25 14:45:45 +01:00
Hieu Hoang
0fd0adc1f6 merge Lexi Birch's LRScore from mert_mtm5 branch. Compiles and run. Hack, must double check with barry or lexi 2012-06-23 22:58:18 -04:00
Hieu Hoang
0cb63edcb9 merge Lexi Birch's LRScore from mert_mtm5 branch. Compiles and run. Hack, must double check with barry or lexi 2012-06-23 22:51:48 -04:00
Hieu Hoang
f48c348508 typo 2012-06-22 22:23:14 -04:00
Hieu Hoang
b1ca36387f mert now compiles with PermScorer. However, didn't implement score() - assert(false). Update Jamfile 2012-06-22 21:07:05 -04:00
Hieu Hoang
7d19fe13ae merge Lexi Birch's LRScore from mert_mtm5 branch 2012-06-22 18:19:16 +01:00
Colin Cherry
07a5c67ebc Merge branch 'master' into miramerge 
Conflicts:
	Jamroot
	misc/queryPhraseTable.cpp
	scripts/training/train-model.perl
 2012-06-14 17:08:16 -04:00
Colin Cherry
5932800489 Spurious space disagreed with master 2012-06-14 14:15:06 -04:00
Colin Cherry
a8a5f896db Fixed some bugs in BatchMira's sparse feature handling. 2012-06-14 14:09:06 -04:00
Colin Cherry
a901fc9f50 Fixed some bugs in BatchMira's sparse feature handling. 2012-06-14 13:41:47 -04:00
Tetsuo Kiso
1dbd8e5ec5 Merge branch 'master' of github.com:moses-smt/mosesdecoder 2012-06-09 19:33:31 +09:00
Tetsuo Kiso
2599ef6dc3 Bug fix: kbmira failed to load dense weights. 2012-06-09 18:03:12 +09:00