moses-smt/mosesdecoder
1
1
Fork 0
mirror of https://github.com/moses-smt/mosesdecoder.git synced 2024-09-11 19:27:11 +03:00
Code Issues Packages Projects Releases Wiki Activity
9,608 Commits 136 Branches 26 Tags 147 MiB
7dc4faa97e
Commit Graph

562 Commits

Author SHA1 Message Date
Barry Haddow
9ca364fb22 Implement brevity penalty smoothing for PRO 
As in Nakov et al (Coling 2012)
 2013-02-18 11:11:20 +00:00
hieu
1dfbe1113c delete MergeScorer in mert/ 2013-01-03 15:01:30 +00:00
Tetsuo Kiso
ce1b650b53 Fix memory leak. 
The object was allocated with new, but it was not deleted.
This may not be a serious problem because the program mostly runs
a short time. However, it is not a good practice.
 2012-12-21 03:06:41 +09:00
Ales Tamchyna
598d65bcfd adding a simple command-line utility for computing sentence-level BLEU (+1) 2012-12-10 13:12:34 +01:00
Tetsuo Kiso
2a3c9fc679 Further optimization for extractor. 
Fixes inefficient updating N-gram counts.

NOTE: Using '--binary' option (this option is not enabled by default yet)
for saving outputs would lead to significant speed up.
 2012-12-07 08:45:47 +09:00
Tetsuo Kiso
8fdec9bf30 Use boost::unordered_map instead of std::map. 
For storing the word vocabulary used in computation of
BLEU scores. This change will reduce the running time
of extractor about 2-3 seconds (9% reduction).
 2012-12-07 05:12:24 +09:00
Tetsuo Kiso
6c04c4ad9c Add more tests to the Data class. 2012-12-07 02:46:59 +09:00
Tetsuo Kiso
c7f6e38326 Use FilePiece to load N-best lists. 
Since FilePiece is friendly with StringPiece.
 2012-12-07 02:39:02 +09:00
Tetsuo Kiso
38e145e556 Use util::TokenIter to tokenize n-best lists. 
Reduce creating std::string objects, too. In both ScoreArray
and FeatureArray classes, the private members to track sentence
indices (namely, "m_index") were unnecessarily declared as
std::string, but it's better to directly declare them as 'int'.
 2012-12-07 01:39:22 +09:00
Tetsuo Kiso
cd3fb3b831 Untabify. 2012-12-06 23:46:22 +09:00
Tetsuo Kiso
ac045a11c1 Speed up N-gram counts when running extractor. 
By replacing std::map with boost::unordered_map.

Runtime of extractor on 100-best lists of 2679 sentences:

Before:
real    0m35.314s
user    0m34.030s
sys     0m1.280s

Ater:
real    0m26.729s
user    0m25.420s
sys     0m1.310s
 2012-12-06 22:08:33 +09:00
Hieu Hoang
f96b33de83 only include moses root when compiling 2012-11-14 13:43:04 +00:00
Hieu Hoang
f8438f80cc move moses/src/* to moses/ 2012-11-12 20:30:39 +00:00
Hieu Hoang
5e3ef23cef move moses/src/* to moses/ 2012-11-12 19:56:18 +00:00
Kenneth Heafield
e9eb7dd021 More shared build fixes 2012-11-07 23:28:42 +01:00
Kenneth Heafield
d7ecd0be1a Remove bleu_lib target. 2012-11-07 23:21:59 +01:00
Tetsuo Kiso
96f7b42eb9 Move implementation details from the header to .cpp file. 
Also add const to variables that we don't want to change.
 2012-11-05 01:24:16 +09:00
Tetsuo Kiso
cccfb9a0c9 Using namespace std in a header file pollutes the global namespace. 
Using directives should be put into the implementation files.
 2012-11-05 00:43:36 +09:00
Kenneth Heafield
dc31857bbc Reduce header pollution 2012-10-30 20:25:05 +00:00
Barry Haddow
848aafb644 Merge remote branch 'github/master' into miramerge 
Conflicts:
	moses/src/AlignmentInfo.cpp
	moses/src/AlignmentInfo.h
	moses/src/ChartHypothesis.cpp
	moses/src/ChartTrellisNode.cpp
	moses/src/LM/Implementation.cpp
	moses/src/LM/Ken.cpp
	moses/src/TargetPhrase.cpp
	moses/src/TargetPhrase.h
 2012-10-08 17:54:59 +01:00
Arianna Bisazza
c78d571561 fixed bug in permutation scores due to scientific notation 2012-10-08 16:15:59 +01:00
Barry Haddow
0a950ee9f4 Merge remote branch 'github/master' into miramerge 
Compiles, but not tested. Had to disable relent filter. Strangely, it seems to contain the
whole of moses-cmd.

Conflicts:
	Jamroot
	OnDiskPt/TargetPhrase.cpp
	moses-cmd/src/Main.cpp
	moses/src/AlignmentInfo.cpp
	moses/src/AlignmentInfo.h
	moses/src/ChartTranslationOptionCollection.cpp
	moses/src/ChartTranslationOptionCollection.h
	moses/src/GenerationDictionary.cpp
	moses/src/Jamfile
	moses/src/Parameter.cpp
	moses/src/PhraseDictionary.cpp
	moses/src/StaticData.cpp
	moses/src/StaticData.h
	moses/src/TargetPhrase.h
	moses/src/TranslationSystem.cpp
	moses/src/TranslationSystem.h
	moses/src/Word.cpp
	phrase-extract/score.cpp
	regression-testing/Jamfile
	scripts/ems/experiment.meta
	scripts/ems/experiment.perl
	scripts/training/train-model.perl
 2012-09-26 22:49:33 +01:00
Arianna Bisazza
ff276e9911 Fixed several bugs in LRscore-MERT. Namely, solved a float-to-int conversion; added hypothesis counter to the scores file to enable later computation of average reordering score; fixed special case of 1-word hypothesis; enabled reading of word-based alignments from n-best-list. 2012-09-24 15:40:18 +02:00
Colin Cherry
ae6ac1c2ae Merge branch 'master' of https://github.com/moses-smt/mosesdecoder 2012-09-14 14:07:53 -04:00
Colin Cherry
3fa95c022b Addad a "--safe-hope" option to kbmira. 
This will limit the influence of model score on oracle (hope) selection.
Good for cases whith extremely large feature values. May make it the defult.
 2012-09-14 13:58:28 -04:00
Barry Haddow
021f5702a7 remove obsolete file 2012-09-13 22:21:31 +01:00
Barry Haddow
2b4e61d826 Merge branch 'trunk' into miramerge 
Compiles, not tested.

Conflicts:
	Jamroot
	OnDiskPt/PhraseNode.h
	OnDiskPt/TargetPhrase.cpp
	OnDiskPt/TargetPhrase.h
	OnDiskPt/TargetPhraseCollection.cpp
	mert/BleuScorer.cpp
	mert/Data.cpp
	mert/FeatureData.cpp
	moses-chart-cmd/src/Main.cpp
	moses/src/AlignmentInfo.h
	moses/src/ChartManager.cpp
	moses/src/LM/Ken.cpp
	moses/src/LM/Ken.h
	moses/src/LMList.h
	moses/src/LexicalReordering.h
	moses/src/PhraseDictionaryTree.h
	moses/src/ScoreIndexManager.h
	moses/src/StaticData.h
	moses/src/TargetPhrase.h
	moses/src/Word.cpp
	scripts/ems/experiment.meta
	scripts/ems/experiment.perl
	scripts/training/train-model.perl
 2012-07-17 13:36:50 +01:00
bhaddow
d0f1c15105 enable single character option 2012-07-12 19:47:57 +01:00
Barry Haddow
c303142ab2 option to skip duplicate removal 2012-07-12 19:08:55 +01:00
Colin Cherry
662e7e7f64 As requested by my bosses: added NRC copyright to kbmira. 2012-07-10 13:13:50 -04:00
Hieu Hoang
7d664b745e Integrate Lexi's LR Score into tuning 2012-07-10 09:25:00 +01:00
Eva Hasler
027a20730e merge Jamfiles 2012-07-04 11:49:07 +01:00
Hieu Hoang
75e038f4cf create namespace for all classes 2012-07-02 17:05:11 +01:00
Hieu Hoang
b5aa04feb7 compile error 2012-07-02 10:23:26 +01:00
Hieu Hoang
121e258e84 namespace all classes in mert directory 2012-06-30 21:39:10 +01:00
Hieu Hoang
e3dd3a8d2c namespace all classes in mert directory 2012-06-30 20:23:45 +01:00
Colin Cherry
65df386581 Merge branch 'master' of https://github.com/moses-smt/mosesdecoder 2012-06-26 17:07:27 -04:00
Colin Cherry
58c3280c2c HypPackEnumerator now stores MiraFeatureVectors, as opposed to 
FeatureDataItems. Uses roughly half the memory.
 2012-06-26 17:02:32 -04:00
Hieu Hoang
3c7b7ac9f5 rollback 2012-06-26 16:31:38 -04:00
Colin Cherry
32299593fa Added debugging info to kbmira. 2012-06-26 16:29:20 -04:00
Hieu Hoang
153e80053c lock m_vocab variable access in Encode() and Lookup(). Other functions are still not threadsafe 2012-06-26 13:33:50 -04:00
Hieu Hoang
00f018a477 Merge https://github.com/moses-smt/mosesdecoder into lrscore 2012-06-25 16:57:17 -04:00
Hieu Hoang
2a03f275a3 change regression data download to git instead of download from edin server. 
Minor change in mert/trimStr() function to prevent warning
 2012-06-25 16:03:11 -04:00
Hieu Hoang
8498b17a41 gcc version-specific error 2012-06-25 14:45:45 +01:00
Hieu Hoang
0fd0adc1f6 merge Lexi Birch's LRScore from mert_mtm5 branch. Compiles and run. Hack, must double check with barry or lexi 2012-06-23 22:58:18 -04:00
Hieu Hoang
0cb63edcb9 merge Lexi Birch's LRScore from mert_mtm5 branch. Compiles and run. Hack, must double check with barry or lexi 2012-06-23 22:51:48 -04:00
Hieu Hoang
f48c348508 typo 2012-06-22 22:23:14 -04:00
Hieu Hoang
b1ca36387f mert now compiles with PermScorer. However, didn't implement score() - assert(false). Update Jamfile 2012-06-22 21:07:05 -04:00
Hieu Hoang
7d19fe13ae merge Lexi Birch's LRScore from mert_mtm5 branch 2012-06-22 18:19:16 +01:00
Colin Cherry
07a5c67ebc Merge branch 'master' into miramerge 
Conflicts:
	Jamroot
	misc/queryPhraseTable.cpp
	scripts/training/train-model.perl
 2012-06-14 17:08:16 -04:00