Commit Graph

3038 Commits

Author SHA1 Message Date
Tetsuo Kiso
2bdeee9caa Clean up Data::loadnbest().
Add helper functions.
2012-03-07 07:01:28 +09:00
Tetsuo Kiso
19ff397aac Rewrite FeatureData::setFeatureMap(); add the unit test. 2012-03-07 06:32:38 +09:00
Kenneth Heafield
022cf38f7d Attempt to fix randlm compilation 2012-03-05 12:18:27 -05:00
Tetsuo Kiso
9e6d3c24b7 Merge branch 'prepare-for-resource-leaks' 2012-03-05 00:47:31 +09:00
Tetsuo Kiso
79b71ed5cd Use boost::scoped_ptr to avoid resource leaks. 2012-03-05 00:35:07 +09:00
Tetsuo Kiso
2351045f18 Change the private member function in mert/Timer. 2012-03-03 23:49:17 +09:00
Tetsuo Kiso
ea3bfb5d4c Delete assertions to check elapsed CPU time.
The accuracy of getrusage() is limited by the resolution
of software clock as described in
http://www.kernel.org/doc/man-pages/online/pages/man7/time.7.html

The assertions required a timer with microsecond accuracy.
However, we don't necessarily want the timer, and we don't
want to add some time-consuming processes to the test code because
we normally build programs again and again, which means
we want to run unit tests as quickly as possible.
2012-03-03 23:24:08 +09:00
Tetsuo Kiso
88b1c7e738 Disable undesirable copying Timer objects. 2012-03-03 21:12:40 +09:00
Matous Machacek
01f8fee238 Fix mert.cpp to work with InterpolatedScorer 2012-03-02 14:16:05 +01:00
Matous Machacek
d5c78d19da Added extractorargs parameter 2012-02-29 15:53:44 +01:00
Tetsuo Kiso
4d8a1e0fe6 Disable failed assertions of TimerTest anyway.
This commit is kludgy. A better solution to the problem will be pushed.
Note that the assertions have no impact on the MERT process.
2012-02-29 12:38:02 +09:00
Kenneth Heafield
1be424bcb3 KenLM 98814b2 including faster malloc-backed building and portability improvements 2012-02-28 13:58:00 -05:00
Rico Sennrich
bf54a5f38b use specified tmpdir when sorting 2012-02-28 10:05:30 +01:00
Tetsuo Kiso
37a9888a46 Fix failure of the Timer unit test. 2012-02-28 12:34:40 +09:00
Matous Machacek
457d004368 Support for using factors in mert and evaluator
example:
Use --factor "0|2" to use only first and third factor from nbest list and from reference.
If you use interpolated scorer, separate records with comma (e.g. --factor "0|2,1").
2012-02-28 02:27:23 +01:00
Tetsuo Kiso
d1292d4d11 Merge branch 'fix-tuning-subset-features' 2012-02-28 00:36:34 +09:00
Tetsuo Kiso
3878135f8b Fix a mistake in a previous commit: tuning on a subset of features.
In the commit 4b6232b757,
I thought I had fixed the bug around the tuning on a subset of
features by checking whether pdim and the length of the
active features which you want to optimize in the tuning.

However, it was wrong. I should set Point::optindices
appropriately according to specified the subset.
2012-02-28 00:35:42 +09:00
Rico Sennrich
230b93148c tmpdir option and bugfix to reordering 2012-02-27 14:11:47 +01:00
Hieu Hoang
205c1a868b Merge branch 'master' of github.com:moses-smt/mosesdecoder 2012-02-27 13:08:41 +00:00
Tetsuo Kiso
cd7855cd58 Fix compiling mert: add a missed header. 2012-02-27 18:50:27 +09:00
Tetsuo Kiso
361a9f8e2c Fix memory leaks.
- The Scorer and ScoreData objects allocated by the new
  operator are now released using the ScopedVector class.

- Add 'virtual' to inherited functions from the Scorer
  class.
2012-02-27 14:30:37 +09:00
Tetsuo Kiso
3b847f2025 Merge branch 'master' of github.com:moses-smt/mosesdecoder 2012-02-27 08:35:09 +09:00
Tetsuo Kiso
52acd9c3a8 Change mert/Timer.
- Add a high resolution timing function to measure the
  wall-clock time by gettimeofday().

- Now the Timer class use getrusage() to measure the elapsed
  CPU time as KenLM does.

- Revive Timer::restart().

- Add Timer::ToString() for reporting the detail statistics
  as well as for debugging.

- Add a simple unit test for Timer.
2012-02-27 08:34:51 +09:00
Matous Machacek
19f0f4ebe9 Change of evaluator usage (see mert/evaluator --help). 2012-02-26 23:04:02 +01:00
Matous Machacek
29a1b790bc Check of the number of weights in InterpolatedScorer 2012-02-26 22:52:20 +01:00
Matous Machacek
ceb70ec00c Fix small bugs (info is printed to cerr) 2012-02-26 22:23:57 +01:00
Matous Machacek
fa2eb79977 Added interpolated scorer
example: to interpolate BLEU and CDER use --sctype=BLEU,CDER
to specify weights use --scconfig=weights:0.3+0.7

This scorer should replace MergeScorer (which requires mert-moses-multi.pl) soon.
Interpolated scorer is more universal and is used in the same way as other scorers.
2012-02-26 18:53:08 +01:00
Hieu Hoang
c542a7b77f compile error from latest zlib header 2012-02-26 17:33:23 +00:00
Tetsuo Kiso
a4d8dbcd02 Cleanup the Timer class in mert. 2012-02-26 14:40:17 +09:00
Tetsuo Kiso
0dae6e337e Remove unused members. 2012-02-26 13:58:48 +09:00
Tetsuo Kiso
b5f4d6729c Prefix private members with "m_". 2012-02-26 13:52:47 +09:00
Tetsuo Kiso
830d89994b Clean up ScoreStats::set(); Remove a constructor which has a string. 2012-02-26 13:44:47 +09:00
Tetsuo Kiso
4e270ccd3d Remove unnecessary conversions using ostream_iterator. 2012-02-26 13:14:34 +09:00
Tetsuo Kiso
194472d831 Clean up. 2012-02-26 13:04:27 +09:00
Tetsuo Kiso
4291677066 Remove obsolete and unused logging statements. 2012-02-26 02:19:40 +09:00
Tetsuo Kiso
82c948e0d3 Don't repeat calling functions many times.
Consider using constants the result if it is possible.
2012-02-26 02:12:59 +09:00
Tetsuo Kiso
37c19feebd Minor change the logging utility for n-gram counts.
Use std::ostream instead of directly using std::cerr.
2012-02-26 02:01:03 +09:00
Tetsuo Kiso
4a63846f82 Clean up calculation effective reference length. 2012-02-26 01:54:51 +09:00
Tetsuo Kiso
51f86de1b6 Add a more efficient member to set up ScoreStats.
- Remove unnecessary conversions.

- Add 'const' to local variables.
2012-02-26 01:41:17 +09:00
Tetsuo Kiso
28cc3631cb Replace string objects with const char[]. 2012-02-26 01:18:08 +09:00
Tetsuo Kiso
40d0ff0434 Hide the implementation details of Ngram counts from the header. 2012-02-26 01:11:56 +09:00
Tetsuo Kiso
a455b06f2f Clean up commented out code snippets for debugging purposes. 2012-02-25 18:14:00 +09:00
Ondrej Bojar
5f84b6e074 accept gzipped input files (tested for non-factored phrase-based) 2012-02-24 18:02:50 +01:00
Hieu Hoang
dbb7980a72 cygwin 2012-02-24 14:18:22 +00:00
Hieu Hoang
b63308f163 zlib changes. More strongly type gzFile variables. HAVE_ZLIB prob no longer works 2012-02-24 00:04:08 +00:00
Hieu Hoang
70d4e01bde zlib changes 2012-02-23 23:54:07 +00:00
Hieu Hoang
9365723fb1 compile error when libs are updated by macports 2012-02-23 22:34:30 +00:00
Hieu Hoang
de9eeab7e9 mac osx compatible split & sort 2012-02-23 13:26:19 +00:00
phikoehn
d3e17f0ebd fix bug with < and > 2012-02-22 00:49:29 +00:00
Matous Machacek
85f9303bd1 Fixed quadratic time when adding ScoreStats to ScoreData 2012-02-21 10:39:04 +01:00