Rico Sennrich
d39cbca0b9
(optionally) use n-best file for evaluator/return-best-dev
...
this adds support for metrics that rely on alignment / trees
2014-09-22 10:49:20 +01:00
Hieu Hoang
6249432407
beautify
2013-05-29 18:16:15 +01:00
Tetsuo Kiso
38e145e556
Use util::TokenIter to tokenize n-best lists.
...
Reduce creating std::string objects, too. In both ScoreArray
and FeatureArray classes, the private members to track sentence
indices (namely, "m_index") were unnecessarily declared as
std::string, but it's better to directly declare them as 'int'.
2012-12-07 01:39:22 +09:00
Barry Haddow
2b4e61d826
Merge branch 'trunk' into miramerge
...
Compiles, not tested.
Conflicts:
Jamroot
OnDiskPt/PhraseNode.h
OnDiskPt/TargetPhrase.cpp
OnDiskPt/TargetPhrase.h
OnDiskPt/TargetPhraseCollection.cpp
mert/BleuScorer.cpp
mert/Data.cpp
mert/FeatureData.cpp
moses-chart-cmd/src/Main.cpp
moses/src/AlignmentInfo.h
moses/src/ChartManager.cpp
moses/src/LM/Ken.cpp
moses/src/LM/Ken.h
moses/src/LMList.h
moses/src/LexicalReordering.h
moses/src/PhraseDictionaryTree.h
moses/src/ScoreIndexManager.h
moses/src/StaticData.h
moses/src/TargetPhrase.h
moses/src/Word.cpp
scripts/ems/experiment.meta
scripts/ems/experiment.perl
scripts/training/train-model.perl
2012-07-17 13:36:50 +01:00
Hieu Hoang
e3dd3a8d2c
namespace all classes in mert directory
2012-06-30 20:23:45 +01:00
Barry Haddow
c397d2068b
Merge branch 'trunk' into miramerge. Still to fix build.
...
Conflicts:
Jamroot
mert/Data.cpp
mert/Data.h
mert/FeatureArray.cpp
mert/FeatureArray.h
mert/FeatureData.cpp
mert/FeatureData.h
mert/FeatureStats.cpp
mert/FeatureStats.h
mert/mert.cpp
moses-chart-cmd/src/IOWrapper.h
moses-chart-cmd/src/Main.cpp
moses-cmd/src/IOWrapper.cpp
moses-cmd/src/IOWrapper.h
moses-cmd/src/Main.cpp
moses/src/GlobalLexicalModel.cpp
moses/src/Jamfile
moses/src/Parameter.cpp
moses/src/PhraseDictionary.cpp
moses/src/ScoreIndexManager.h
moses/src/TargetPhrase.h
regression-testing/tests/phrase.lexicalized-reordering-bin/truth/results.txt
regression-testing/tests/phrase.lexicalized-reordering-cn/truth/results.txt
regression-testing/tests/phrase.lexicalized-reordering/truth/results.txt
regression-testing/tests/phrase.multiple-translation-system-lr/truth/results.txt
regression-testing/tests/phrase.show-weights.lex-reorder/truth/results.txt
regression-testing/tests/phrase.show-weights/truth/results.txt
scripts/ems/experiment.meta
scripts/ems/experiment.perl
scripts/training/filter-model-given-input.pl
scripts/training/mert-moses.pl
2012-05-24 21:11:35 +01:00
Tetsuo Kiso
dbfe766f2c
Fix using directive refers to implicitly-defined namespace 'std'.
2012-05-06 05:27:04 +09:00
Tetsuo Kiso
df4586740d
Fix using directive refers to implicitly-defined namespace 'std'.
2012-05-06 05:27:04 +09:00
Tetsuo Kiso
27515f5de1
Add a function to check whether a string ends with a suffix.
...
- Use the function in Data::InitFeatureMap().
- Add an unit test for InitFeatureMap().
- Move helper functions for Data::loadnbest() to public for unit testing.
2012-04-04 22:04:51 +09:00
Tetsuo Kiso
1ade69a546
Add a function to check whether a string ends with a suffix.
...
- Use the function in Data::InitFeatureMap().
- Add an unit test for InitFeatureMap().
- Move helper functions for Data::loadnbest() to public for unit testing.
2012-04-04 22:04:51 +09:00
Tetsuo Kiso
3ce46da4cd
Clean up Data; add TODOs.
2012-03-10 17:47:01 +09:00
Tetsuo Kiso
81309bdb2d
Clean up Data; add TODOs.
2012-03-10 17:47:01 +09:00
Tetsuo Kiso
b5bcf48b17
Pass by pointers to Scorer instead of references.
2012-03-10 17:28:38 +09:00
Tetsuo Kiso
a1ab79c7fc
Pass by pointers to Scorer instead of references.
2012-03-10 17:28:38 +09:00
Tetsuo Kiso
e7a2483b22
mert: Prefix private members with "m_" except TER.
...
Squashed commit of the following:
- Clean up PRO.
- Clean up ScoreStats.
- Clean up ScoreData.
- Clean up ScoreArray.
- Remove unnecessary headers.
- Clean up ScopedVector.
- Clean up Point.
- Clean up PerScorer.
- Clean up Optimizer.
- Clean up MergeScorer.
- Clean up InterpolatedScorer.
- Clean up FileStream.
- Clean up FeatureStats.
- Remove inefficient string concatenation.
- Clean up FeatureData.
- Clean up FeatureArray.
- Clean up Data.
2012-03-10 17:12:34 +09:00
Tetsuo Kiso
eb2c9ee5e3
mert: Prefix private members with "m_" except TER.
...
Squashed commit of the following:
- Clean up PRO.
- Clean up ScoreStats.
- Clean up ScoreData.
- Clean up ScoreArray.
- Remove unnecessary headers.
- Clean up ScopedVector.
- Clean up Point.
- Clean up PerScorer.
- Clean up Optimizer.
- Clean up MergeScorer.
- Clean up InterpolatedScorer.
- Clean up FileStream.
- Clean up FeatureStats.
- Remove inefficient string concatenation.
- Clean up FeatureData.
- Clean up FeatureArray.
- Clean up Data.
2012-03-10 17:12:34 +09:00
Tetsuo Kiso
6b1dfa3434
Clean up Data::loadnbest().
...
Add helper functions.
2012-03-07 07:01:28 +09:00
Tetsuo Kiso
2bdeee9caa
Clean up Data::loadnbest().
...
Add helper functions.
2012-03-07 07:01:28 +09:00
Tetsuo Kiso
47ac8a474d
Change the naming conventions for the guard macros; Rename TER directory.
...
This change might be useful to avoid duplicating the names.
The reason is that although MERT programs are standalone
applications, some header files such as data.h and
point.h have common guard macro names like "DATA_H" and
"POINT_H", and this is not good naming conventions
when you want to include external headers.
Some files actually include headers in Moses and KenLM's util.
2012-02-20 09:46:08 +09:00
Tetsuo Kiso
8c3b82e596
Change the naming conventions for the guard macros; Rename TER directory.
...
This change might be useful to avoid duplicating the names.
The reason is that although MERT programs are standalone
applications, some header files such as data.h and
point.h have common guard macro names like "DATA_H" and
"POINT_H", and this is not good naming conventions
when you want to include external headers.
Some files actually include headers in Moses and KenLM's util.
2012-02-20 09:46:08 +09:00
Tetsuo Kiso
82da44b030
Fix typo.
2012-02-20 08:29:53 +09:00
Tetsuo Kiso
94888b258d
Fix typo.
2012-02-20 08:29:53 +09:00
Barry Haddow
62d7d034bb
Fix sharding bug
2012-02-08 17:11:56 +00:00
Barry Haddow
752724594e
Fix sharding bug
2012-02-08 17:11:56 +00:00
Barry Haddow
ced24a881d
Implementation of feature-merging for pro-mert
2012-01-13 16:52:15 +00:00
Hieu Hoang
753eebd959
revert
2011-12-12 20:48:42 +07:00
Hieu Hoang
21009b5d1e
revert
2011-12-12 20:48:42 +07:00
Barry Haddow
ba5633c7b5
Merge remote branch 'origin/master' into pro
...
Conflicts:
kenlm
mert/Data.cpp
mert/Data.h
mert/Makefile.am
mert/mert.cpp
2011-11-17 15:58:56 +00:00
Barry Haddow
71c777f01d
Remove old PRO. Fix running of PRO from mert script.
2011-11-16 17:43:54 +00:00
Tetsuo Kiso
be1506e759
Use passing objects by const references not passing by their values.
2011-11-14 14:00:47 +09:00
Tetsuo Kiso
625fe118e0
Fix bugs, private members were not initialized when the instances are created.
...
When the objects of Data and FeatureData classes are created,
the primitive private members were not initialized properly.
2011-11-14 12:20:04 +09:00
Tetsuo Kiso
3d70b2e1a5
Small change: modify initialization of the Data class.
2011-11-12 22:04:22 +09:00
Tetsuo Kiso
fb3b0f9f62
Add const to Data and Feature class members.
2011-11-12 11:26:13 +09:00
Tetsuo Kiso
dfb714296f
Add 'explicit' for constructors with one argument.
2011-11-12 09:51:27 +09:00
Tetsuo Kiso
ce9a628ed0
Remove unnecessary semicolons used in end of member functions.
2011-11-12 09:40:01 +09:00
Tetsuo Kiso
664ffe0130
Fix indentation.
2011-11-12 09:24:19 +09:00
Tetsuo Kiso
68315d6407
Fix class, function, and implementation comments format.
...
Functions comments should be placed in their declarations.
2011-11-12 08:58:23 +09:00
Tetsuo Kiso
4f6d022fe7
Add comments to mark the end of #define guards.
2011-11-12 07:59:50 +09:00
Tetsuo Kiso
087756b8c3
Fix memory leaks in extractor.
2011-11-11 20:02:26 +09:00
bhaddow
fc695c38a7
Implementation of sharding and resampling in mert.
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4226 1f5c12ca-751b-0410-a591-d2e778427230
2011-09-15 17:45:35 +00:00
phkoehn
41a1849437
support for sparse feature functions (mert support only when using PRO)
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4184 1f5c12ca-751b-0410-a591-d2e778427230
2011-09-07 16:37:33 +00:00
phkoehn
36db0ffe48
added pairwise ranked optimization (PRO) as proposd by [Hopkins&May,2011], just use switch --pairwise-ranked
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4106 1f5c12ca-751b-0410-a591-d2e778427230
2011-08-03 17:00:17 +00:00
hieuhoang1972
148c1e8305
run beautify.perl. Consistent formatting for .h & .cpp files
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3899 1f5c12ca-751b-0410-a591-d2e778427230
2011-02-24 12:42:19 +00:00
nicolabertoldi
e94834012d
added facilities to read and write score statistics in binary format
...
moved facilities for feature names in FeatureData object
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1824 1f5c12ca-751b-0410-a591-d2e778427230
2008-06-05 17:03:54 +00:00
nicolabertoldi
930e67c5e3
fixed another bug related to the handling of feature names
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1822 1f5c12ca-751b-0410-a591-d2e778427230
2008-06-05 11:30:37 +00:00
nicolabertoldi
44d7e0e0f7
fixed a bug related to the handling of feature names
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1821 1f5c12ca-751b-0410-a591-d2e778427230
2008-06-05 11:26:54 +00:00
nicolabertoldi
281bf610b8
added binary read/load facility for feature data
...
added names of features in the header
added methods to access the features by name
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1819 1f5c12ca-751b-0410-a591-d2e778427230
2008-06-05 07:23:34 +00:00
nicolabertoldi
af585bc492
nbest can be read from stdin, too
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1797 1f5c12ca-751b-0410-a591-d2e778427230
2008-05-27 17:20:01 +00:00
nicolabertoldi
291260abf7
- made output more compliant with old version
...
- added PerSCorer.h and BleuScorer.h
- stored feature names
- fixed bug about output of best Point
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1796 1f5c12ca-751b-0410-a591-d2e778427230
2008-05-27 16:50:52 +00:00
nicolabertoldi
8cf59edcdc
remove loadnbest from FeatureData and Scoredata; change test_scorer accordingly;
...
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1787 1f5c12ca-751b-0410-a591-d2e778427230
2008-05-20 14:33:47 +00:00