Barry Haddow
9ca364fb22
Implement brevity penalty smoothing for PRO
...
As in Nakov et al (Coling 2012)
2013-02-18 11:11:20 +00:00
hieu
1dfbe1113c
delete MergeScorer in mert/
2013-01-03 15:01:30 +00:00
Tetsuo Kiso
ce1b650b53
Fix memory leak.
...
The object was allocated with new, but it was not deleted.
This may not be a serious problem because the program mostly runs
a short time. However, it is not a good practice.
2012-12-21 03:06:41 +09:00
Ales Tamchyna
598d65bcfd
adding a simple command-line utility for computing sentence-level BLEU (+1)
2012-12-10 13:12:34 +01:00
Tetsuo Kiso
2a3c9fc679
Further optimization for extractor.
...
Fixes inefficient updating N-gram counts.
NOTE: Using '--binary' option (this option is not enabled by default yet)
for saving outputs would lead to significant speed up.
2012-12-07 08:45:47 +09:00
Tetsuo Kiso
8fdec9bf30
Use boost::unordered_map instead of std::map.
...
For storing the word vocabulary used in computation of
BLEU scores. This change will reduce the running time
of extractor about 2-3 seconds (9% reduction).
2012-12-07 05:12:24 +09:00
Tetsuo Kiso
6c04c4ad9c
Add more tests to the Data class.
2012-12-07 02:46:59 +09:00
Tetsuo Kiso
c7f6e38326
Use FilePiece to load N-best lists.
...
Since FilePiece is friendly with StringPiece.
2012-12-07 02:39:02 +09:00
Tetsuo Kiso
38e145e556
Use util::TokenIter to tokenize n-best lists.
...
Reduce creating std::string objects, too. In both ScoreArray
and FeatureArray classes, the private members to track sentence
indices (namely, "m_index") were unnecessarily declared as
std::string, but it's better to directly declare them as 'int'.
2012-12-07 01:39:22 +09:00
Tetsuo Kiso
cd3fb3b831
Untabify.
2012-12-06 23:46:22 +09:00
Tetsuo Kiso
ac045a11c1
Speed up N-gram counts when running extractor.
...
By replacing std::map with boost::unordered_map.
Runtime of extractor on 100-best lists of 2679 sentences:
Before:
real 0m35.314s
user 0m34.030s
sys 0m1.280s
Ater:
real 0m26.729s
user 0m25.420s
sys 0m1.310s
2012-12-06 22:08:33 +09:00
Hieu Hoang
f96b33de83
only include moses root when compiling
2012-11-14 13:43:04 +00:00
Hieu Hoang
f8438f80cc
move moses/src/* to moses/
2012-11-12 20:30:39 +00:00
Hieu Hoang
5e3ef23cef
move moses/src/* to moses/
2012-11-12 19:56:18 +00:00
Kenneth Heafield
e9eb7dd021
More shared build fixes
2012-11-07 23:28:42 +01:00
Kenneth Heafield
d7ecd0be1a
Remove bleu_lib target.
2012-11-07 23:21:59 +01:00
Tetsuo Kiso
96f7b42eb9
Move implementation details from the header to .cpp file.
...
Also add const to variables that we don't want to change.
2012-11-05 01:24:16 +09:00
Tetsuo Kiso
cccfb9a0c9
Using namespace std in a header file pollutes the global namespace.
...
Using directives should be put into the implementation files.
2012-11-05 00:43:36 +09:00
Kenneth Heafield
dc31857bbc
Reduce header pollution
2012-10-30 20:25:05 +00:00
Barry Haddow
848aafb644
Merge remote branch 'github/master' into miramerge
...
Conflicts:
moses/src/AlignmentInfo.cpp
moses/src/AlignmentInfo.h
moses/src/ChartHypothesis.cpp
moses/src/ChartTrellisNode.cpp
moses/src/LM/Implementation.cpp
moses/src/LM/Ken.cpp
moses/src/TargetPhrase.cpp
moses/src/TargetPhrase.h
2012-10-08 17:54:59 +01:00
Arianna Bisazza
c78d571561
fixed bug in permutation scores due to scientific notation
2012-10-08 16:15:59 +01:00
Barry Haddow
0a950ee9f4
Merge remote branch 'github/master' into miramerge
...
Compiles, but not tested. Had to disable relent filter. Strangely, it seems to contain the
whole of moses-cmd.
Conflicts:
Jamroot
OnDiskPt/TargetPhrase.cpp
moses-cmd/src/Main.cpp
moses/src/AlignmentInfo.cpp
moses/src/AlignmentInfo.h
moses/src/ChartTranslationOptionCollection.cpp
moses/src/ChartTranslationOptionCollection.h
moses/src/GenerationDictionary.cpp
moses/src/Jamfile
moses/src/Parameter.cpp
moses/src/PhraseDictionary.cpp
moses/src/StaticData.cpp
moses/src/StaticData.h
moses/src/TargetPhrase.h
moses/src/TranslationSystem.cpp
moses/src/TranslationSystem.h
moses/src/Word.cpp
phrase-extract/score.cpp
regression-testing/Jamfile
scripts/ems/experiment.meta
scripts/ems/experiment.perl
scripts/training/train-model.perl
2012-09-26 22:49:33 +01:00
Arianna Bisazza
ff276e9911
Fixed several bugs in LRscore-MERT. Namely, solved a float-to-int conversion; added hypothesis counter to the scores file to enable later computation of average reordering score; fixed special case of 1-word hypothesis; enabled reading of word-based alignments from n-best-list.
2012-09-24 15:40:18 +02:00
Colin Cherry
ae6ac1c2ae
Merge branch 'master' of https://github.com/moses-smt/mosesdecoder
2012-09-14 14:07:53 -04:00
Colin Cherry
3fa95c022b
Addad a "--safe-hope" option to kbmira.
...
This will limit the influence of model score on oracle (hope) selection.
Good for cases whith extremely large feature values. May make it the defult.
2012-09-14 13:58:28 -04:00
Barry Haddow
021f5702a7
remove obsolete file
2012-09-13 22:21:31 +01:00
Barry Haddow
2b4e61d826
Merge branch 'trunk' into miramerge
...
Compiles, not tested.
Conflicts:
Jamroot
OnDiskPt/PhraseNode.h
OnDiskPt/TargetPhrase.cpp
OnDiskPt/TargetPhrase.h
OnDiskPt/TargetPhraseCollection.cpp
mert/BleuScorer.cpp
mert/Data.cpp
mert/FeatureData.cpp
moses-chart-cmd/src/Main.cpp
moses/src/AlignmentInfo.h
moses/src/ChartManager.cpp
moses/src/LM/Ken.cpp
moses/src/LM/Ken.h
moses/src/LMList.h
moses/src/LexicalReordering.h
moses/src/PhraseDictionaryTree.h
moses/src/ScoreIndexManager.h
moses/src/StaticData.h
moses/src/TargetPhrase.h
moses/src/Word.cpp
scripts/ems/experiment.meta
scripts/ems/experiment.perl
scripts/training/train-model.perl
2012-07-17 13:36:50 +01:00
bhaddow
d0f1c15105
enable single character option
2012-07-12 19:47:57 +01:00
Barry Haddow
c303142ab2
option to skip duplicate removal
2012-07-12 19:08:55 +01:00
Colin Cherry
662e7e7f64
As requested by my bosses: added NRC copyright to kbmira.
2012-07-10 13:13:50 -04:00
Hieu Hoang
7d664b745e
Integrate Lexi's LR Score into tuning
2012-07-10 09:25:00 +01:00
Eva Hasler
027a20730e
merge Jamfiles
2012-07-04 11:49:07 +01:00
Hieu Hoang
75e038f4cf
create namespace for all classes
2012-07-02 17:05:11 +01:00
Hieu Hoang
b5aa04feb7
compile error
2012-07-02 10:23:26 +01:00
Hieu Hoang
121e258e84
namespace all classes in mert directory
2012-06-30 21:39:10 +01:00
Hieu Hoang
e3dd3a8d2c
namespace all classes in mert directory
2012-06-30 20:23:45 +01:00
Colin Cherry
65df386581
Merge branch 'master' of https://github.com/moses-smt/mosesdecoder
2012-06-26 17:07:27 -04:00
Colin Cherry
58c3280c2c
HypPackEnumerator now stores MiraFeatureVectors, as opposed to
...
FeatureDataItems. Uses roughly half the memory.
2012-06-26 17:02:32 -04:00
Hieu Hoang
3c7b7ac9f5
rollback
2012-06-26 16:31:38 -04:00
Colin Cherry
32299593fa
Added debugging info to kbmira.
2012-06-26 16:29:20 -04:00
Hieu Hoang
153e80053c
lock m_vocab variable access in Encode() and Lookup(). Other functions are still not threadsafe
2012-06-26 13:33:50 -04:00
Hieu Hoang
00f018a477
Merge https://github.com/moses-smt/mosesdecoder into lrscore
2012-06-25 16:57:17 -04:00
Hieu Hoang
2a03f275a3
change regression data download to git instead of download from edin server.
...
Minor change in mert/trimStr() function to prevent warning
2012-06-25 16:03:11 -04:00
Hieu Hoang
8498b17a41
gcc version-specific error
2012-06-25 14:45:45 +01:00
Hieu Hoang
0fd0adc1f6
merge Lexi Birch's LRScore from mert_mtm5 branch. Compiles and run. Hack, must double check with barry or lexi
2012-06-23 22:58:18 -04:00
Hieu Hoang
0cb63edcb9
merge Lexi Birch's LRScore from mert_mtm5 branch. Compiles and run. Hack, must double check with barry or lexi
2012-06-23 22:51:48 -04:00
Hieu Hoang
f48c348508
typo
2012-06-22 22:23:14 -04:00
Hieu Hoang
b1ca36387f
mert now compiles with PermScorer. However, didn't implement score() - assert(false). Update Jamfile
2012-06-22 21:07:05 -04:00
Hieu Hoang
7d19fe13ae
merge Lexi Birch's LRScore from mert_mtm5 branch
2012-06-22 18:19:16 +01:00
Colin Cherry
07a5c67ebc
Merge branch 'master' into miramerge
...
Conflicts:
Jamroot
misc/queryPhraseTable.cpp
scripts/training/train-model.perl
2012-06-14 17:08:16 -04:00