moses-smt/mosesdecoder
1
1
Fork 0
mirror of https://github.com/moses-smt/mosesdecoder.git synced 2024-12-26 05:14:36 +03:00
Code Issues Packages Projects Releases Wiki Activity
17,477 Commits 136 Branches 26 Tags 147 MiB
master
Commit Graph

670 Commits

Author SHA1 Message Date
Tetsuo Kiso
df4586740d Fix using directive refers to implicitly-defined namespace 'std'. 2012-05-06 05:27:04 +09:00
Tetsuo Kiso
b6f55f6bec Fix using namespace "std" before including headers in *.h. 2012-05-06 01:31:04 +09:00
Tetsuo Kiso
4b5fe180ed Fix using namespace "std" before including headers in *.h. 2012-05-06 01:31:04 +09:00
Tetsuo Kiso
1b634488fe Prefix a private member of mert/CderScorer with "m_". 2012-05-03 05:38:35 +09:00
Tetsuo Kiso
fc2f4d4ba1 Prefix a private member of mert/CderScorer with "m_". 2012-05-03 05:38:35 +09:00
Matous Machacek
933f25b3b2 CderScorer compute both CDER and WER metric 2012-05-02 22:03:58 +02:00
Matous Machacek
370bf4e697 CderScorer compute both CDER and WER metric 2012-05-02 22:03:58 +02:00
Eva
6c2a58a48e clean up mira, add sampling from hope/model/fear 2012-04-29 21:29:18 -07:00
Eva
6f39ad0b3e test 2012-04-28 23:11:30 -07:00
Tetsuo Kiso
7d457f5049 Add TODO to mert/Reference.h. 2012-04-26 05:44:05 +09:00
Tetsuo Kiso
66992ed497 Add TODO to mert/Reference.h. 2012-04-26 05:44:05 +09:00
Tetsuo Kiso
91a6725081 Add a test case for setting up BLEU's reference length. 
To make sure the type of effective reference length is
set to the specified arguments.
 2012-04-22 20:11:05 +09:00
Tetsuo Kiso
cee8b2e357 Add a test case for setting up BLEU's reference length. 
To make sure the type of effective reference length is
set to the specified arguments.
 2012-04-22 20:11:05 +09:00
Tetsuo Kiso
fe79b96328 Use std::stringstream instead of using snprintf() for Windows. 
This commit fixes compilation problems related to
snprintf() for Windows users.

Thanks to Raka Prasetya for reporting the errors.
Thanks also to Kenneth Heafield and Barry Haddow for suggestions.
 2012-04-18 23:47:48 +09:00
Tetsuo Kiso
bd79fc2c13 Use std::stringstream instead of using snprintf() for Windows. 
This commit fixes compilation problems related to
snprintf() for Windows users.

Thanks to Raka Prasetya for reporting the errors.
Thanks also to Kenneth Heafield and Barry Haddow for suggestions.
 2012-04-18 23:47:48 +09:00
Tetsuo Kiso
408dd72f9d Pass by reference instead of returning objects. 
In the previous implementation, copy-constructor got called, which
was not necessary. This commit prevents it, making code more efficient.
 2012-04-12 09:40:52 +09:00
Tetsuo Kiso
b25e9c2915 Pass by reference instead of returning objects. 
In the previous implementation, copy-constructor got called, which
was not necessary. This commit prevents it, making code more efficient.
 2012-04-12 09:40:52 +09:00
Tetsuo Kiso
0e9b5fd9d0 Add const to return values of overloaded operators. 
* This commit prevents developers from doing mistakes like:

  Point p1, p2, p4;
  if (p1 + p2 = p4) { // Bang! We actually wanted to compare
                      // the result of two points: (p1 + p2 == p4).
    // do something.
  }

See, e.g., http://www.gotw.ca/gotw/006.htm for details.

* Add more test cases according to this change.
* Move a helper function to compare floating point numbers
  to Util.h.
 2012-04-12 09:16:27 +09:00
Tetsuo Kiso
48c1d712b8 Add const to return values of overloaded operators. 
* This commit prevents developers from doing mistakes like:

  Point p1, p2, p4;
  if (p1 + p2 = p4) { // Bang! We actually wanted to compare
                      // the result of two points: (p1 + p2 == p4).
    // do something.
  }

See, e.g., http://www.gotw.ca/gotw/006.htm for details.

* Add more test cases according to this change.
* Move a helper function to compare floating point numbers
  to Util.h.
 2012-04-12 09:16:27 +09:00
Tetsuo Kiso
49a4be2899 Reduce the number of calling the numeric_limits constructor. 
- Create a test module for Point.
- Add test cases.
 2012-04-12 05:19:11 +09:00
Tetsuo Kiso
70800c2012 Reduce the number of calling the numeric_limits constructor. 
- Create a test module for Point.
- Add test cases.
 2012-04-12 05:19:11 +09:00
Tetsuo Kiso
65dc5a391f Update mert/TODO 2012-04-08 22:47:20 +09:00
Tetsuo Kiso
fa98111b6f Update mert/TODO 2012-04-08 22:47:20 +09:00
Tetsuo Kiso
4c38bbb434 Add things to TODO. 2012-04-08 22:45:04 +09:00
Tetsuo Kiso
1b8a72a093 Add things to TODO. 2012-04-08 22:45:04 +09:00
Tetsuo Kiso
d034eeb703 Add test cases for BLEU and sentence-level BLEU+1. 
- Move a definition of sentenceLevelBleuPlusOne() from pro.cpp
  to BleuScorer.cpp.
- Add check for the length of an input vector.
 2012-04-07 01:02:32 +09:00
Tetsuo Kiso
87ec82d853 Add test cases for BLEU and sentence-level BLEU+1. 
- Move a definition of sentenceLevelBleuPlusOne() from pro.cpp
  to BleuScorer.cpp.
- Add check for the length of an input vector.
 2012-04-07 01:02:32 +09:00
Tetsuo Kiso
bcc1958d94 Fix a warning; remove duplicate function calls. 2012-04-05 12:49:49 +09:00
Tetsuo Kiso
a0e787384f Fix a warning; remove duplicate function calls. 2012-04-05 12:49:49 +09:00
Tetsuo Kiso
fecee1c4fc Fix a grammar error; add more test case. 
- Fix a warning, too.
 2012-04-05 10:10:16 +09:00
Tetsuo Kiso
2bfeb39ba6 Fix a grammar error; add more test case. 
- Fix a warning, too.
 2012-04-05 10:10:16 +09:00
Tetsuo Kiso
e2a92c0f91 Use EndsWith(). 2012-04-05 00:03:13 +09:00
Tetsuo Kiso
8a2495c966 Use EndsWith(). 2012-04-05 00:03:13 +09:00
Tetsuo Kiso
20b99ed9e9 Add a test case for EndsWith(). 2012-04-04 22:43:51 +09:00
Tetsuo Kiso
8c5e69bb70 Add a test case for EndsWith(). 2012-04-04 22:43:51 +09:00
Tetsuo Kiso
a6cec240c0 Fix typo. 2012-04-04 22:36:22 +09:00
Tetsuo Kiso
685f4081b2 Fix typo. 2012-04-04 22:36:22 +09:00
Tetsuo Kiso
eaa0ab486a Add a test case for BLEU's clipped counts. 
- Make BleuScorer::setReferenceFiles() more testable by
  adding OpenReference() and OpenReferenceStream().
 2012-04-04 22:33:30 +09:00
Tetsuo Kiso
9d10b46ca5 Add a test case for BLEU's clipped counts. 
- Make BleuScorer::setReferenceFiles() more testable by
  adding OpenReference() and OpenReferenceStream().
 2012-04-04 22:33:30 +09:00
Tetsuo Kiso
6c69a97cbb Fix a compilation error. 2012-04-04 22:11:43 +09:00
Tetsuo Kiso
2ca4513e29 Fix a compilation error. 2012-04-04 22:11:43 +09:00
Tetsuo Kiso
27515f5de1 Add a function to check whether a string ends with a suffix. 
- Use the function in Data::InitFeatureMap().
- Add an unit test for InitFeatureMap().
- Move helper functions for Data::loadnbest() to public for unit testing.
 2012-04-04 22:04:51 +09:00
Tetsuo Kiso
1ade69a546 Add a function to check whether a string ends with a suffix. 
- Use the function in Data::InitFeatureMap().
- Add an unit test for InitFeatureMap().
- Move helper functions for Data::loadnbest() to public for unit testing.
 2012-04-04 22:04:51 +09:00
Tetsuo Kiso
9686e86c70 Fix a warning: comparison between signed and unsigned integers. 2012-03-31 10:47:28 +09:00
Tetsuo Kiso
422a199a97 Fix a warning: comparison between signed and unsigned integers. 2012-03-31 10:47:28 +09:00
Tetsuo Kiso
24b1314d90 Fix indentation. 2012-03-31 10:43:54 +09:00
Tetsuo Kiso
16fc3443dd Fix indentation. 2012-03-31 10:43:54 +09:00
Tetsuo Kiso
59f755587a Prefix a private member of Optimizer with "m_". 2012-03-31 10:39:59 +09:00
Tetsuo Kiso
850402d6ba Prefix a private member of Optimizer with "m_". 2012-03-31 10:39:59 +09:00
Hieu Hoang
fba75a885e typedef error on mac 2012-03-30 18:33:50 +07:00
Hieu Hoang
cebf88f39e typedef error on mac 2012-03-30 18:33:50 +07:00
tosch
fab03f5a6e added an (optional) positivity constraint to MERT 2012-03-30 11:50:23 +02:00
tosch
47ee26951e added an (optional) positivity constraint to MERT 2012-03-30 11:50:23 +02:00
Eva Hasler
f5a4d61fd4 resolving conflicts 2012-03-27 14:24:49 +01:00
Eva Hasler
614c98e682 fix feature name splitting to split on last colon 2012-03-26 21:17:10 +01:00
Matous Machacek
9e5e502687 Merge branch 'master' of github.com:moses-smt/mosesdecoder 2012-03-25 15:51:22 +02:00
Matous Machacek
81eab37970 Merge branch 'master' of github.com:moses-smt/mosesdecoder 2012-03-25 15:51:22 +02:00
Tetsuo Kiso
377d971c5d Use assertions in the test case. 2012-03-25 21:31:44 +09:00
Tetsuo Kiso
d2ff30791c Use assertions in the test case. 2012-03-25 21:31:44 +09:00
Matous Machacek
1665a23fe1 Added posibility to weight words in sempos 2012-03-24 16:07:47 +01:00
Matous Machacek
d9c7b86ccb Added posibility to weight words in sempos 2012-03-24 16:07:47 +01:00
Matous Machacek
ac5ba83ec8 Changed maximum scores constant to be able to experiment with more sempos classes 2012-03-23 23:42:00 +01:00
Matous Machacek
8ba0aed8f7 Changed maximum scores constant to be able to experiment with more sempos classes 2012-03-23 23:42:00 +01:00
Tetsuo Kiso
184b24b456 Remove unecessary headers; fix indentation. 2012-03-24 04:12:33 +09:00
Tetsuo Kiso
e0c29f414a Remove unecessary headers; fix indentation. 2012-03-24 04:12:33 +09:00
Matous Machacek
fbc726dd49 Fixed compilation error 2012-03-23 19:09:04 +01:00
Matous Machacek
e0408d80fd Fixed compilation error 2012-03-23 19:09:04 +01:00
Tetsuo Kiso
5e5f500254 Remove unnecessary headers; prefix private members with "m_". 
- Add const.
- Add a virtual destructor to abstract class.
- Add some TODOs to some constant values.

Create files for "overlapping" classes.
 2012-03-24 02:39:04 +09:00
Tetsuo Kiso
f0ed7779d7 Remove unnecessary headers; prefix private members with "m_". 
- Add const.
- Add a virtual destructor to abstract class.
- Add some TODOs to some constant values.

Create files for "overlapping" classes.
 2012-03-24 02:39:04 +09:00
Matous Machacek
3319805102 Added SemposScorer to mert and evaluator 2012-03-23 14:39:24 +01:00
Matous Machacek
ca4ba7eb79 Added SemposScorer to mert and evaluator 2012-03-23 14:39:24 +01:00
Tetsuo Kiso
2f899c9adb Fix a compile error I missed when developing on OS X. 2012-03-21 00:18:44 +09:00
Tetsuo Kiso
ae60572123 Fix a compile error I missed when developing on OS X. 2012-03-21 00:18:44 +09:00
Tetsuo Kiso
0a852b60d1 Move the GzFileBuf implementation into .cpp file. 
Add error check when gzopen() fails.
 2012-03-21 00:11:54 +09:00
Tetsuo Kiso
6540c4d8ac Move the GzFileBuf implementation into .cpp file. 
Add error check when gzopen() fails.
 2012-03-21 00:11:54 +09:00
Tetsuo Kiso
91df03e2ed Rename gzfilebuf.h; apply coding style. 2012-03-20 23:48:52 +09:00
Tetsuo Kiso
ccae7bae6e Rename gzfilebuf.h; apply coding style. 2012-03-20 23:48:52 +09:00
Tetsuo Kiso
8e79cd5692 Use assertions in the test case. 
It gets easier to detect the point of failure in the test.
 2012-03-20 16:57:57 +09:00
Tetsuo Kiso
3a73c5990d Use assertions in the test case. 
It gets easier to detect the point of failure in the test.
 2012-03-20 16:57:57 +09:00
Tetsuo Kiso
2fdb47fe67 Add an unit test for BleuScorer. 
The test contains only ngram counting. More tests for calculating
BLEU score are reuiqred.
 2012-03-20 05:52:35 +09:00
Tetsuo Kiso
82ae12249c Add an unit test for BleuScorer. 
The test contains only ngram counting. More tests for calculating
BLEU score are reuiqred.
 2012-03-20 05:52:35 +09:00
Tetsuo Kiso
8987fed667 Add thread unsafe Singleton class. 
- Add Vocabulary factory and the unit test.
- Remove Scorer::ClearVocabulary().
 2012-03-20 05:49:10 +09:00
Tetsuo Kiso
6c64d94b29 Add thread unsafe Singleton class. 
- Add Vocabulary factory and the unit test.
- Remove Scorer::ClearVocabulary().
 2012-03-20 05:49:10 +09:00
Tetsuo Kiso
1678ee3769 Merge branch 'master' of github.com:moses-smt/mosesdecoder into cleanup-mert 2012-03-20 03:46:17 +09:00
Tetsuo Kiso
ebf87d6154 Merge branch 'master' of github.com:moses-smt/mosesdecoder into cleanup-mert 2012-03-20 03:46:17 +09:00
Tetsuo Kiso
525f06452c Change the Encoder class to Vocabulary. 
- Introduce the namespace to avoid naming collisions. The class name
  is used in KenLM.
- Add the unit test.
 2012-03-20 03:43:04 +09:00
Tetsuo Kiso
3b77ef5006 Change the Encoder class to Vocabulary. 
- Introduce the namespace to avoid naming collisions. The class name
  is used in KenLM.
- Add the unit test.
 2012-03-20 03:43:04 +09:00
Christian Buck
20f2e703bf changed size_t to std::size_t for gcc4.6 2012-03-19 17:32:55 +00:00
Christian Buck
d0b5c09b9e changed size_t to std::size_t for gcc4.6 2012-03-19 17:32:55 +00:00
Tetsuo Kiso
2b28072f7a Move Encoder class from Scorer.h to Ngram.h. 
To add unit tests.
 2012-03-19 23:21:02 +09:00
Tetsuo Kiso
a65165f227 Move Encoder class from Scorer.h to Ngram.h. 
To add unit tests.
 2012-03-19 23:21:02 +09:00
Tetsuo Kiso
f686e8771a Add some functions to BleuScorer for unit testing. 
This commit also includes
- Fix typo.
- Fix indentations.
- Add 'const' to Scorer::applyFactors().
 2012-03-19 22:45:15 +09:00
Tetsuo Kiso
0edf7d3774 Add some functions to BleuScorer for unit testing. 
This commit also includes
- Fix typo.
- Fix indentations.
- Add 'const' to Scorer::applyFactors().
 2012-03-19 22:45:15 +09:00
Tetsuo Kiso
54233d3270 Create files for OptimizerFactory class; add the unit test. 2012-03-18 13:08:38 +09:00
Tetsuo Kiso
724b5cbb37 Create files for OptimizerFactory class; add the unit test. 2012-03-18 13:08:38 +09:00
Tetsuo Kiso
38c662d4e0 Pass parameters by const reference. 2012-03-18 12:32:11 +09:00
Tetsuo Kiso
82c683758f Pass parameters by const reference. 2012-03-18 12:32:11 +09:00
Tetsuo Kiso
0e8184c1c1 Fix a file permission. 2012-03-18 10:52:40 +09:00
Tetsuo Kiso
e08392962a Fix a file permission. 2012-03-18 10:52:40 +09:00
Matous Machacek
73d5c9ea4e Fixed bug in CderScorer.cpp 2012-03-17 23:35:56 +01:00
Matous Machacek
b89d02b4a9 Fixed bug in CderScorer.cpp 2012-03-17 23:35:56 +01:00
Tetsuo Kiso
6b95a19eda Create Reference class to clean up BleuScorer. 
- Add an unit test for Reference.
- Move functions to calculate the reference length from
  BleuScorer to Reference.
 2012-03-18 05:58:40 +09:00
Tetsuo Kiso
812140274b Create Reference class to clean up BleuScorer. 
- Add an unit test for Reference.
- Move functions to calculate the reference length from
  BleuScorer to Reference.
 2012-03-18 05:58:40 +09:00
Tetsuo Kiso
c6536a134b Clean up BleuScorer. 2012-03-14 22:44:51 +09:00
Tetsuo Kiso
8a9385ad09 Clean up BleuScorer. 2012-03-14 22:44:51 +09:00
Tetsuo Kiso
5007f129d8 Clean up BleuScorer with lookup(). 2012-03-14 22:41:29 +09:00
Tetsuo Kiso
519abd7675 Clean up BleuScorer with lookup(). 2012-03-14 22:41:29 +09:00
Tetsuo Kiso
fba01c7cdf Create a header file for NgramCounts class. 
The reason is that we want to add the unit test.
 2012-03-14 22:14:11 +09:00
Tetsuo Kiso
52d8053b0e Create a header file for NgramCounts class. 
The reason is that we want to add the unit test.
 2012-03-14 22:14:11 +09:00
Tetsuo Kiso
b354da4c65 Clean up accessors and mutators. 2012-03-10 19:27:52 +09:00
Tetsuo Kiso
0c88a1ec2e Clean up accessors and mutators. 2012-03-10 19:27:52 +09:00
Tetsuo Kiso
5d600f2b50 Remove hard-coded "/dev/stdout". 
This will improve the portability.
We also change the interface of I/O functions for ease of the
development unit tests.
 2012-03-10 19:04:43 +09:00
Tetsuo Kiso
3f8d8d7842 Remove hard-coded "/dev/stdout". 
This will improve the portability.
We also change the interface of I/O functions for ease of the
development unit tests.
 2012-03-10 19:04:43 +09:00
Tetsuo Kiso
3ce46da4cd Clean up Data; add TODOs. 2012-03-10 17:47:01 +09:00
Tetsuo Kiso
81309bdb2d Clean up Data; add TODOs. 2012-03-10 17:47:01 +09:00
Tetsuo Kiso
b5bcf48b17 Pass by pointers to Scorer instead of references. 2012-03-10 17:28:38 +09:00
Tetsuo Kiso
a1ab79c7fc Pass by pointers to Scorer instead of references. 2012-03-10 17:28:38 +09:00
Tetsuo Kiso
e7a2483b22 mert: Prefix private members with "m_" except TER. 
Squashed commit of the following:

- Clean up PRO.
- Clean up ScoreStats.
- Clean up ScoreData.
- Clean up ScoreArray.
- Remove unnecessary headers.
- Clean up ScopedVector.
- Clean up Point.
- Clean up PerScorer.
- Clean up Optimizer.
- Clean up MergeScorer.
- Clean up InterpolatedScorer.
- Clean up FileStream.
- Clean up FeatureStats.
- Remove inefficient string concatenation.
- Clean up FeatureData.
- Clean up FeatureArray.
- Clean up Data.
 2012-03-10 17:12:34 +09:00
Tetsuo Kiso
eb2c9ee5e3 mert: Prefix private members with "m_" except TER. 
Squashed commit of the following:

- Clean up PRO.
- Clean up ScoreStats.
- Clean up ScoreData.
- Clean up ScoreArray.
- Remove unnecessary headers.
- Clean up ScopedVector.
- Clean up Point.
- Clean up PerScorer.
- Clean up Optimizer.
- Clean up MergeScorer.
- Clean up InterpolatedScorer.
- Clean up FileStream.
- Clean up FeatureStats.
- Remove inefficient string concatenation.
- Clean up FeatureData.
- Clean up FeatureArray.
- Clean up Data.
 2012-03-10 17:12:34 +09:00
Tetsuo Kiso
ed6e6f00b1 Minor change for calculating BLEU. 
To avoid defining the similar variables twice to calculate
document-wise BLEU and sentence-wise BLEU scores.
 2012-03-10 02:49:31 +09:00
Tetsuo Kiso
e4063b0f3b Minor change for calculating BLEU. 
To avoid defining the similar variables twice to calculate
document-wise BLEU and sentence-wise BLEU scores.
 2012-03-10 02:49:31 +09:00
Tetsuo Kiso
127f958bed Remove an unused variable and unnecessary 'std::'. 2012-03-07 07:19:24 +09:00
Tetsuo Kiso
851a1835b6 Remove an unused variable and unnecessary 'std::'. 2012-03-07 07:19:24 +09:00
Tetsuo Kiso
07d42f7614 Remove an unused variable. 2012-03-07 07:07:29 +09:00
Tetsuo Kiso
6ada41576c Remove an unused variable. 2012-03-07 07:07:29 +09:00
Tetsuo Kiso
6b1dfa3434 Clean up Data::loadnbest(). 
Add helper functions.
 2012-03-07 07:01:28 +09:00
Tetsuo Kiso
2bdeee9caa Clean up Data::loadnbest(). 
Add helper functions.
 2012-03-07 07:01:28 +09:00
Tetsuo Kiso
d6c1abe6bb Rewrite FeatureData::setFeatureMap(); add the unit test. 2012-03-07 06:32:38 +09:00
Tetsuo Kiso
19ff397aac Rewrite FeatureData::setFeatureMap(); add the unit test. 2012-03-07 06:32:38 +09:00
Tetsuo Kiso
5c4e2a8c8d Use boost::scoped_ptr to avoid resource leaks. 2012-03-05 00:35:07 +09:00
Tetsuo Kiso
79b71ed5cd Use boost::scoped_ptr to avoid resource leaks. 2012-03-05 00:35:07 +09:00
Tetsuo Kiso
c8800f3822 Change the private member function in mert/Timer. 2012-03-03 23:49:17 +09:00
Tetsuo Kiso
2351045f18 Change the private member function in mert/Timer. 2012-03-03 23:49:17 +09:00
Tetsuo Kiso
ee5174de58 Delete assertions to check elapsed CPU time. 
The accuracy of getrusage() is limited by the resolution
of software clock as described in
http://www.kernel.org/doc/man-pages/online/pages/man7/time.7.html

The assertions required a timer with microsecond accuracy.
However, we don't necessarily want the timer, and we don't
want to add some time-consuming processes to the test code because
we normally build programs again and again, which means
we want to run unit tests as quickly as possible.
 2012-03-03 23:24:08 +09:00
Tetsuo Kiso
ea3bfb5d4c Delete assertions to check elapsed CPU time. 
The accuracy of getrusage() is limited by the resolution
of software clock as described in
http://www.kernel.org/doc/man-pages/online/pages/man7/time.7.html

The assertions required a timer with microsecond accuracy.
However, we don't necessarily want the timer, and we don't
want to add some time-consuming processes to the test code because
we normally build programs again and again, which means
we want to run unit tests as quickly as possible.
 2012-03-03 23:24:08 +09:00
Tetsuo Kiso
9a46c5cd7f Disable undesirable copying Timer objects. 2012-03-03 21:12:40 +09:00
Tetsuo Kiso
88b1c7e738 Disable undesirable copying Timer objects. 2012-03-03 21:12:40 +09:00
Matous Machacek
f196a87763 Fix mert.cpp to work with InterpolatedScorer 2012-03-02 14:16:05 +01:00
Matous Machacek
01f8fee238 Fix mert.cpp to work with InterpolatedScorer 2012-03-02 14:16:05 +01:00
Tetsuo Kiso
7735670a57 Disable failed assertions of TimerTest anyway. 
This commit is kludgy. A better solution to the problem will be pushed.
Note that the assertions have no impact on the MERT process.
 2012-02-29 12:38:02 +09:00
Tetsuo Kiso
4d8a1e0fe6 Disable failed assertions of TimerTest anyway. 
This commit is kludgy. A better solution to the problem will be pushed.
Note that the assertions have no impact on the MERT process.
 2012-02-29 12:38:02 +09:00
Tetsuo Kiso
b99ebb7a19 Fix failure of the Timer unit test. 2012-02-28 12:34:40 +09:00
Tetsuo Kiso
37a9888a46 Fix failure of the Timer unit test. 2012-02-28 12:34:40 +09:00
Matous Machacek
ba987c94ba Support for using factors in mert and evaluator 
example:
Use --factor "0|2" to use only first and third factor from nbest list and from reference.
If you use interpolated scorer, separate records with comma (e.g. --factor "0|2,1").
 2012-02-28 02:27:23 +01:00
Matous Machacek
457d004368 Support for using factors in mert and evaluator 
example:
Use --factor "0|2" to use only first and third factor from nbest list and from reference.
If you use interpolated scorer, separate records with comma (e.g. --factor "0|2,1").
 2012-02-28 02:27:23 +01:00
Tetsuo Kiso
6d6fb4383d Fix a mistake in a previous commit: tuning on a subset of features. 
In the commit 4b6232b757,
I thought I had fixed the bug around the tuning on a subset of
features by checking whether pdim and the length of the
active features which you want to optimize in the tuning.

However, it was wrong. I should set Point::optindices
appropriately according to specified the subset.
 2012-02-28 00:35:42 +09:00
Tetsuo Kiso
3878135f8b Fix a mistake in a previous commit: tuning on a subset of features. 
In the commit 4b6232b757,
I thought I had fixed the bug around the tuning on a subset of
features by checking whether pdim and the length of the
active features which you want to optimize in the tuning.

However, it was wrong. I should set Point::optindices
appropriately according to specified the subset.
 2012-02-28 00:35:42 +09:00
Tetsuo Kiso
c3bb4c7abd Fix compiling mert: add a missed header. 2012-02-27 18:50:27 +09:00
Tetsuo Kiso
cd7855cd58 Fix compiling mert: add a missed header. 2012-02-27 18:50:27 +09:00
Tetsuo Kiso
5e74e87da0 Fix memory leaks. 
- The Scorer and ScoreData objects allocated by the new
  operator are now released using the ScopedVector class.

- Add 'virtual' to inherited functions from the Scorer
  class.
 2012-02-27 14:30:37 +09:00
Tetsuo Kiso
361a9f8e2c Fix memory leaks. 
- The Scorer and ScoreData objects allocated by the new
  operator are now released using the ScopedVector class.

- Add 'virtual' to inherited functions from the Scorer
  class.
 2012-02-27 14:30:37 +09:00
Tetsuo Kiso
04a717be2b Merge branch 'master' of github.com:moses-smt/mosesdecoder 2012-02-27 08:35:09 +09:00
Tetsuo Kiso
3b847f2025 Merge branch 'master' of github.com:moses-smt/mosesdecoder 2012-02-27 08:35:09 +09:00
Tetsuo Kiso
7093d2e2cd Change mert/Timer. 
- Add a high resolution timing function to measure the
  wall-clock time by gettimeofday().

- Now the Timer class use getrusage() to measure the elapsed
  CPU time as KenLM does.

- Revive Timer::restart().

- Add Timer::ToString() for reporting the detail statistics
  as well as for debugging.

- Add a simple unit test for Timer.
 2012-02-27 08:34:51 +09:00
Tetsuo Kiso
52acd9c3a8 Change mert/Timer. 
- Add a high resolution timing function to measure the
  wall-clock time by gettimeofday().

- Now the Timer class use getrusage() to measure the elapsed
  CPU time as KenLM does.

- Revive Timer::restart().

- Add Timer::ToString() for reporting the detail statistics
  as well as for debugging.

- Add a simple unit test for Timer.
 2012-02-27 08:34:51 +09:00
Matous Machacek
e3f0280f27 Change of evaluator usage (see mert/evaluator --help). 2012-02-26 23:04:02 +01:00
Matous Machacek
19f0f4ebe9 Change of evaluator usage (see mert/evaluator --help). 2012-02-26 23:04:02 +01:00
Matous Machacek
99a98a336b Check of the number of weights in InterpolatedScorer 2012-02-26 22:52:20 +01:00
Matous Machacek
29a1b790bc Check of the number of weights in InterpolatedScorer 2012-02-26 22:52:20 +01:00
Matous Machacek
bd92b0634a Fix small bugs (info is printed to cerr) 2012-02-26 22:23:57 +01:00
Matous Machacek
ceb70ec00c Fix small bugs (info is printed to cerr) 2012-02-26 22:23:57 +01:00
Matous Machacek
e8a94a7bd2 Added interpolated scorer 
example: to interpolate BLEU and CDER use --sctype=BLEU,CDER
to specify weights use --scconfig=weights:0.3+0.7

This scorer should replace MergeScorer (which requires mert-moses-multi.pl) soon.
Interpolated scorer is more universal and is used in the same way as other scorers.
 2012-02-26 18:53:08 +01:00
Matous Machacek
fa2eb79977 Added interpolated scorer 
example: to interpolate BLEU and CDER use --sctype=BLEU,CDER
to specify weights use --scconfig=weights:0.3+0.7

This scorer should replace MergeScorer (which requires mert-moses-multi.pl) soon.
Interpolated scorer is more universal and is used in the same way as other scorers.
 2012-02-26 18:53:08 +01:00
Tetsuo Kiso
3b47348550 Cleanup the Timer class in mert. 2012-02-26 14:40:17 +09:00
Tetsuo Kiso
a4d8dbcd02 Cleanup the Timer class in mert. 2012-02-26 14:40:17 +09:00
Tetsuo Kiso
0c24f7e10b Remove unused members. 2012-02-26 13:58:48 +09:00
Tetsuo Kiso
0dae6e337e Remove unused members. 2012-02-26 13:58:48 +09:00
Tetsuo Kiso
c62365b419 Prefix private members with "m_". 2012-02-26 13:52:47 +09:00
Tetsuo Kiso
b5f4d6729c Prefix private members with "m_". 2012-02-26 13:52:47 +09:00
Tetsuo Kiso
ff5ae511b1 Clean up ScoreStats::set(); Remove a constructor which has a string. 2012-02-26 13:44:47 +09:00
Tetsuo Kiso
830d89994b Clean up ScoreStats::set(); Remove a constructor which has a string. 2012-02-26 13:44:47 +09:00
Tetsuo Kiso
9d6263d337 Remove unnecessary conversions using ostream_iterator. 2012-02-26 13:14:34 +09:00
Tetsuo Kiso
4e270ccd3d Remove unnecessary conversions using ostream_iterator. 2012-02-26 13:14:34 +09:00
Tetsuo Kiso
c913effe13 Clean up. 2012-02-26 13:04:27 +09:00
Tetsuo Kiso
194472d831 Clean up. 2012-02-26 13:04:27 +09:00
Tetsuo Kiso
c26e83fd09 Remove obsolete and unused logging statements. 2012-02-26 02:19:40 +09:00
Tetsuo Kiso
4291677066 Remove obsolete and unused logging statements. 2012-02-26 02:19:40 +09:00
Tetsuo Kiso
224c654fa5 Don't repeat calling functions many times. 
Consider using constants the result if it is possible.
 2012-02-26 02:12:59 +09:00
Tetsuo Kiso
82c948e0d3 Don't repeat calling functions many times. 
Consider using constants the result if it is possible.
 2012-02-26 02:12:59 +09:00
Tetsuo Kiso
669b9d9c7a Minor change the logging utility for n-gram counts. 
Use std::ostream instead of directly using std::cerr.
 2012-02-26 02:01:03 +09:00
Tetsuo Kiso
37c19feebd Minor change the logging utility for n-gram counts. 
Use std::ostream instead of directly using std::cerr.
 2012-02-26 02:01:03 +09:00
Tetsuo Kiso
8e0a61d0d7 Clean up calculation effective reference length. 2012-02-26 01:54:51 +09:00
Tetsuo Kiso
4a63846f82 Clean up calculation effective reference length. 2012-02-26 01:54:51 +09:00
Tetsuo Kiso
c4fa8a3865 Add a more efficient member to set up ScoreStats. 
- Remove unnecessary conversions.

- Add 'const' to local variables.
 2012-02-26 01:41:17 +09:00
Tetsuo Kiso
51f86de1b6 Add a more efficient member to set up ScoreStats. 
- Remove unnecessary conversions.

- Add 'const' to local variables.
 2012-02-26 01:41:17 +09:00
Tetsuo Kiso
2c2bd63bbd Replace string objects with const char[]. 2012-02-26 01:18:08 +09:00
Tetsuo Kiso
28cc3631cb Replace string objects with const char[]. 2012-02-26 01:18:08 +09:00
Tetsuo Kiso
17f06a3250 Hide the implementation details of Ngram counts from the header. 2012-02-26 01:11:56 +09:00
Tetsuo Kiso
40d0ff0434 Hide the implementation details of Ngram counts from the header. 2012-02-26 01:11:56 +09:00
Tetsuo Kiso
0c9023abc6 Clean up commented out code snippets for debugging purposes. 2012-02-25 18:14:00 +09:00
Tetsuo Kiso
a455b06f2f Clean up commented out code snippets for debugging purposes. 2012-02-25 18:14:00 +09:00
Matous Machacek
16376eabcc Fixed quadratic time when adding ScoreStats to ScoreData 2012-02-21 10:39:04 +01:00
Matous Machacek
85f9303bd1 Fixed quadratic time when adding ScoreStats to ScoreData 2012-02-21 10:39:04 +01:00
Tetsuo Kiso
aefa6e1000 Fix a memory leak. 2012-02-20 11:04:21 +09:00
Tetsuo Kiso
fa43a88d46 Fix a memory leak. 2012-02-20 11:04:21 +09:00
Tetsuo Kiso
c2ef7093ed Add 'virtual' to destructors. 2012-02-20 10:23:59 +09:00
Tetsuo Kiso
e749924706 Add 'virtual' to destructors. 2012-02-20 10:23:59 +09:00
Tetsuo Kiso
47ac8a474d Change the naming conventions for the guard macros; Rename TER directory. 
This change might be useful to avoid duplicating the names.
The reason is that although MERT programs are standalone
applications, some header files such as data.h and
point.h have common guard macro names like "DATA_H" and
"POINT_H", and this is not good naming conventions
when you want to include external headers.
Some files actually include headers in Moses and KenLM's util.
 2012-02-20 09:46:08 +09:00
Tetsuo Kiso
8c3b82e596 Change the naming conventions for the guard macros; Rename TER directory. 
This change might be useful to avoid duplicating the names.
The reason is that although MERT programs are standalone
applications, some header files such as data.h and
point.h have common guard macro names like "DATA_H" and
"POINT_H", and this is not good naming conventions
when you want to include external headers.
Some files actually include headers in Moses and KenLM's util.
 2012-02-20 09:46:08 +09:00
Tetsuo Kiso
82da44b030 Fix typo. 2012-02-20 08:29:53 +09:00
Tetsuo Kiso
94888b258d Fix typo. 2012-02-20 08:29:53 +09:00
Tetsuo Kiso
ce7b136994 Add comments; remove unused macros. 2012-02-20 08:20:44 +09:00
Tetsuo Kiso
232e514774 Add comments; remove unused macros. 2012-02-20 08:20:44 +09:00
Tetsuo Kiso
a70925317e Put global variables in mert/util.cpp in anonymous space. 
We do not allow clients to access the following variables.
Instead, use the APIs which we provide for that.

Also, remove the unused function, and fix smoke tests.
 2012-02-20 08:02:23 +09:00
Tetsuo Kiso
faab4b214d Put global variables in mert/util.cpp in anonymous space. 
We do not allow clients to access the following variables.
Instead, use the APIs which we provide for that.

Also, remove the unused function, and fix smoke tests.
 2012-02-20 08:02:23 +09:00
Tetsuo Kiso
5d1cfa0ebb Bug fix: tokenizer used in mert; add unit tests for that. 
When tokenizing a string delimited by spaces (say, "9 9 8 7 ")
with Tokenize(), resulting a sequence of strings are
{"9", "9", "8", "7", "" }, which is different
from we have expected. We are not interested in empty strings.

This commit fix this issue, and add unit tests for
the tokenize functions.
 2012-02-20 07:39:24 +09:00
Tetsuo Kiso
8c7dfe04e7 Bug fix: tokenizer used in mert; add unit tests for that. 
When tokenizing a string delimited by spaces (say, "9 9 8 7 ")
with Tokenize(), resulting a sequence of strings are
{"9", "9", "8", "7", "" }, which is different
from we have expected. We are not interested in empty strings.

This commit fix this issue, and add unit tests for
the tokenize functions.
 2012-02-20 07:39:24 +09:00
Tetsuo Kiso
a7666735b5 Add error checking to setup 'to_optimize'. 
mert will check whether the dimension and the number of
fetures are equal.
 2012-02-17 09:16:10 +09:00
Tetsuo Kiso
4b6232b757 Add error checking to setup 'to_optimize'. 
mert will check whether the dimension and the number of
fetures are equal.
 2012-02-17 09:16:10 +09:00
Tetsuo Kiso
6c003e544a Bug fix mert: when you want to optimize fewer features. 
This commit is a temporary bug fix.
 2012-02-17 08:25:18 +09:00
Tetsuo Kiso
c5e7e4cea7 Bug fix mert: when you want to optimize fewer features. 
This commit is a temporary bug fix.
 2012-02-17 08:25:18 +09:00
Tetsuo Kiso
819dc9e0f9 Add a utility function to FeatureData for debugging. 2012-02-17 07:27:07 +09:00
Tetsuo Kiso
47b535ee0a Add a utility function to FeatureData for debugging. 2012-02-17 07:27:07 +09:00
Tetsuo Kiso
c1b85b480c Delete mert/sample/README; Add smoke tests. 
Replace README with a bunch of shell script
for smoke testing of MERT.

The README file was not a typical README file.
It was like a sample script to run mert and
extractor, so I renamed it as smoke tests stuff.
 2012-02-17 03:53:52 +09:00
Tetsuo Kiso
91645503e8 Delete mert/sample/README; Add smoke tests. 
Replace README with a bunch of shell script
for smoke testing of MERT.

The README file was not a typical README file.
It was like a sample script to run mert and
extractor, so I renamed it as smoke tests stuff.
 2012-02-17 03:53:52 +09:00
Barry Haddow
7091555cd6 Merge branch 'master' of github.com:moses-smt/mosesdecoder 2012-02-08 17:48:19 +00:00
Barry Haddow
69afc63fb0 Merge branch 'master' of github.com:moses-smt/mosesdecoder 2012-02-08 17:48:19 +00:00
Barry Haddow
fa6753b0f3 Really simple sharding test 2012-02-08 17:47:54 +00:00
Barry Haddow
757f08a141 Really simple sharding test 2012-02-08 17:47:54 +00:00
Barry Haddow
62d7d034bb Fix sharding bug 2012-02-08 17:11:56 +00:00
Barry Haddow
752724594e Fix sharding bug 2012-02-08 17:11:56 +00:00
Tetsuo Kiso
905f959d83 Move funcions defined in a header into .cpp file. 2012-02-01 21:44:37 +09:00
Tetsuo Kiso
270809fa97 Move funcions defined in a header into .cpp file. 2012-02-01 21:44:37 +09:00
Tetsuo Kiso
b2987337d8 Remove virtual keyword from whoami() function. 
The funtion is neither inherited from Scorer nor
StatisticsBasedScorer.
 2012-02-01 21:36:25 +09:00
Tetsuo Kiso
4e3dd11f53 Remove virtual keyword from whoami() function. 
The funtion is neither inherited from Scorer nor
StatisticsBasedScorer.
 2012-02-01 21:36:25 +09:00
Tetsuo Kiso
923efa0a4c Move funcionts used in StatisticsBasedScorer into anonymous namespace. 
And add 'inline' keyword to the functions.
 2012-02-01 21:30:50 +09:00
Tetsuo Kiso
4051f723f7 Move funcionts used in StatisticsBasedScorer into anonymous namespace. 
And add 'inline' keyword to the functions.
 2012-02-01 21:30:50 +09:00
Tetsuo Kiso
5cd5b90d0d Create a initialize function. 2012-02-01 21:26:47 +09:00
Tetsuo Kiso
ae5a58da89 Create a initialize function. 2012-02-01 21:26:47 +09:00
Tetsuo Kiso
17e864e446 Create private class to encapssulate encoding process. 
Instead of using typedefs inside a class only,
it might be better to create a private class to do same things.
 2012-02-01 21:19:25 +09:00
Tetsuo Kiso
92434a2293 Create private class to encapssulate encoding process. 
Instead of using typedefs inside a class only,
it might be better to create a private class to do same things.
 2012-02-01 21:19:25 +09:00
Tetsuo Kiso
a351a74c18 Move regularizaion type into StatisticsBasedScorer. 
The type is used as internal purpose.
 2012-02-01 20:58:49 +09:00
Tetsuo Kiso
c5f8c7d33a Move regularizaion type into StatisticsBasedScorer. 
The type is used as internal purpose.
 2012-02-01 20:58:49 +09:00
Tetsuo Kiso
b19e7777ce Add prefix 'm_' to private and protected members in Scorer classes. 2012-02-01 20:54:20 +09:00
Tetsuo Kiso
cb729ad3a5 Add prefix 'm_' to private and protected members in Scorer classes. 2012-02-01 20:54:20 +09:00
Tetsuo Kiso
33cea0f1dc Delete unused enums. 2012-02-01 20:26:25 +09:00
Tetsuo Kiso
947e30ac3b Delete unused enums. 2012-02-01 20:26:25 +09:00
Tetsuo Kiso
30fa97e404 Move reference length type into a private member of BleuScorer. 
The reason is that the type is used as internal purpose.
 2012-02-01 20:24:48 +09:00
Tetsuo Kiso
3ac41ff2d7 Move reference length type into a private member of BleuScorer. 
The reason is that the type is used as internal purpose.
 2012-02-01 20:24:48 +09:00
Tetsuo Kiso
d742341176 Bug fix: memory leak issues when using MergeScorer. 2012-02-01 18:13:32 +09:00
Tetsuo Kiso
671a2f8c33 Bug fix: memory leak issues when using MergeScorer. 2012-02-01 18:13:32 +09:00
Tetsuo Kiso
3ef03a77c4 Change casts to C++ style casts. 2012-02-01 18:13:00 +09:00
Tetsuo Kiso
4373c07cf0 Change casts to C++ style casts. 2012-02-01 18:13:00 +09:00
Tetsuo Kiso
142342f8be Change casts to C++ style casts, and delete unnecessary casts. 2012-02-01 17:17:58 +09:00
Tetsuo Kiso
194e24115a Change casts to C++ style casts, and delete unnecessary casts. 2012-02-01 17:17:58 +09:00
Tetsuo Kiso
3c04b7e826 Create a function to save final weights to a file. 2012-02-01 16:59:48 +09:00
Tetsuo Kiso
380f550cf3 Create a function to save final weights to a file. 2012-02-01 16:59:48 +09:00
Tetsuo Kiso
1bafc99ea2 Use a constant instead of hard-coding. 2012-02-01 16:51:25 +09:00
Tetsuo Kiso
2a7d2a395f Use a constant instead of hard-coding. 2012-02-01 16:51:25 +09:00
Tetsuo Kiso
8e1bf6d7d1 Pass std::string objects by const reference instead of by value. 2012-02-01 16:46:36 +09:00