moses-smt/mosesdecoder
1
1
Fork 0
mirror of https://github.com/moses-smt/mosesdecoder.git synced 2024-12-28 14:32:38 +03:00
Code Issues Packages Projects Releases Wiki Activity
12,619 Commits 136 Branches 26 Tags 147 MiB
44b22ece69
Commit Graph

612 Commits

Author SHA1 Message Date
Matous Machacek
29a1b790bc Check of the number of weights in InterpolatedScorer 2012-02-26 22:52:20 +01:00
Matous Machacek
bd92b0634a Fix small bugs (info is printed to cerr) 2012-02-26 22:23:57 +01:00
Matous Machacek
ceb70ec00c Fix small bugs (info is printed to cerr) 2012-02-26 22:23:57 +01:00
Matous Machacek
e8a94a7bd2 Added interpolated scorer 
example: to interpolate BLEU and CDER use --sctype=BLEU,CDER
to specify weights use --scconfig=weights:0.3+0.7

This scorer should replace MergeScorer (which requires mert-moses-multi.pl) soon.
Interpolated scorer is more universal and is used in the same way as other scorers.
 2012-02-26 18:53:08 +01:00
Matous Machacek
fa2eb79977 Added interpolated scorer 
example: to interpolate BLEU and CDER use --sctype=BLEU,CDER
to specify weights use --scconfig=weights:0.3+0.7

This scorer should replace MergeScorer (which requires mert-moses-multi.pl) soon.
Interpolated scorer is more universal and is used in the same way as other scorers.
 2012-02-26 18:53:08 +01:00
Tetsuo Kiso
3b47348550 Cleanup the Timer class in mert. 2012-02-26 14:40:17 +09:00
Tetsuo Kiso
a4d8dbcd02 Cleanup the Timer class in mert. 2012-02-26 14:40:17 +09:00
Tetsuo Kiso
0c24f7e10b Remove unused members. 2012-02-26 13:58:48 +09:00
Tetsuo Kiso
0dae6e337e Remove unused members. 2012-02-26 13:58:48 +09:00
Tetsuo Kiso
c62365b419 Prefix private members with "m_". 2012-02-26 13:52:47 +09:00
Tetsuo Kiso
b5f4d6729c Prefix private members with "m_". 2012-02-26 13:52:47 +09:00
Tetsuo Kiso
ff5ae511b1 Clean up ScoreStats::set(); Remove a constructor which has a string. 2012-02-26 13:44:47 +09:00
Tetsuo Kiso
830d89994b Clean up ScoreStats::set(); Remove a constructor which has a string. 2012-02-26 13:44:47 +09:00
Tetsuo Kiso
9d6263d337 Remove unnecessary conversions using ostream_iterator. 2012-02-26 13:14:34 +09:00
Tetsuo Kiso
4e270ccd3d Remove unnecessary conversions using ostream_iterator. 2012-02-26 13:14:34 +09:00
Tetsuo Kiso
c913effe13 Clean up. 2012-02-26 13:04:27 +09:00
Tetsuo Kiso
194472d831 Clean up. 2012-02-26 13:04:27 +09:00
Tetsuo Kiso
c26e83fd09 Remove obsolete and unused logging statements. 2012-02-26 02:19:40 +09:00
Tetsuo Kiso
4291677066 Remove obsolete and unused logging statements. 2012-02-26 02:19:40 +09:00
Tetsuo Kiso
224c654fa5 Don't repeat calling functions many times. 
Consider using constants the result if it is possible.
 2012-02-26 02:12:59 +09:00
Tetsuo Kiso
82c948e0d3 Don't repeat calling functions many times. 
Consider using constants the result if it is possible.
 2012-02-26 02:12:59 +09:00
Tetsuo Kiso
669b9d9c7a Minor change the logging utility for n-gram counts. 
Use std::ostream instead of directly using std::cerr.
 2012-02-26 02:01:03 +09:00
Tetsuo Kiso
37c19feebd Minor change the logging utility for n-gram counts. 
Use std::ostream instead of directly using std::cerr.
 2012-02-26 02:01:03 +09:00
Tetsuo Kiso
8e0a61d0d7 Clean up calculation effective reference length. 2012-02-26 01:54:51 +09:00
Tetsuo Kiso
4a63846f82 Clean up calculation effective reference length. 2012-02-26 01:54:51 +09:00
Tetsuo Kiso
c4fa8a3865 Add a more efficient member to set up ScoreStats. 
- Remove unnecessary conversions.

- Add 'const' to local variables.
 2012-02-26 01:41:17 +09:00
Tetsuo Kiso
51f86de1b6 Add a more efficient member to set up ScoreStats. 
- Remove unnecessary conversions.

- Add 'const' to local variables.
 2012-02-26 01:41:17 +09:00
Tetsuo Kiso
2c2bd63bbd Replace string objects with const char[]. 2012-02-26 01:18:08 +09:00
Tetsuo Kiso
28cc3631cb Replace string objects with const char[]. 2012-02-26 01:18:08 +09:00
Tetsuo Kiso
17f06a3250 Hide the implementation details of Ngram counts from the header. 2012-02-26 01:11:56 +09:00
Tetsuo Kiso
40d0ff0434 Hide the implementation details of Ngram counts from the header. 2012-02-26 01:11:56 +09:00
Tetsuo Kiso
0c9023abc6 Clean up commented out code snippets for debugging purposes. 2012-02-25 18:14:00 +09:00
Tetsuo Kiso
a455b06f2f Clean up commented out code snippets for debugging purposes. 2012-02-25 18:14:00 +09:00
Matous Machacek
16376eabcc Fixed quadratic time when adding ScoreStats to ScoreData 2012-02-21 10:39:04 +01:00
Matous Machacek
85f9303bd1 Fixed quadratic time when adding ScoreStats to ScoreData 2012-02-21 10:39:04 +01:00
Tetsuo Kiso
aefa6e1000 Fix a memory leak. 2012-02-20 11:04:21 +09:00
Tetsuo Kiso
fa43a88d46 Fix a memory leak. 2012-02-20 11:04:21 +09:00
Tetsuo Kiso
c2ef7093ed Add 'virtual' to destructors. 2012-02-20 10:23:59 +09:00
Tetsuo Kiso
e749924706 Add 'virtual' to destructors. 2012-02-20 10:23:59 +09:00
Tetsuo Kiso
47ac8a474d Change the naming conventions for the guard macros; Rename TER directory. 
This change might be useful to avoid duplicating the names.
The reason is that although MERT programs are standalone
applications, some header files such as data.h and
point.h have common guard macro names like "DATA_H" and
"POINT_H", and this is not good naming conventions
when you want to include external headers.
Some files actually include headers in Moses and KenLM's util.
 2012-02-20 09:46:08 +09:00
Tetsuo Kiso
8c3b82e596 Change the naming conventions for the guard macros; Rename TER directory. 
This change might be useful to avoid duplicating the names.
The reason is that although MERT programs are standalone
applications, some header files such as data.h and
point.h have common guard macro names like "DATA_H" and
"POINT_H", and this is not good naming conventions
when you want to include external headers.
Some files actually include headers in Moses and KenLM's util.
 2012-02-20 09:46:08 +09:00
Tetsuo Kiso
82da44b030 Fix typo. 2012-02-20 08:29:53 +09:00
Tetsuo Kiso
94888b258d Fix typo. 2012-02-20 08:29:53 +09:00
Tetsuo Kiso
ce7b136994 Add comments; remove unused macros. 2012-02-20 08:20:44 +09:00
Tetsuo Kiso
232e514774 Add comments; remove unused macros. 2012-02-20 08:20:44 +09:00
Tetsuo Kiso
a70925317e Put global variables in mert/util.cpp in anonymous space. 
We do not allow clients to access the following variables.
Instead, use the APIs which we provide for that.

Also, remove the unused function, and fix smoke tests.
 2012-02-20 08:02:23 +09:00
Tetsuo Kiso
faab4b214d Put global variables in mert/util.cpp in anonymous space. 
We do not allow clients to access the following variables.
Instead, use the APIs which we provide for that.

Also, remove the unused function, and fix smoke tests.
 2012-02-20 08:02:23 +09:00
Tetsuo Kiso
5d1cfa0ebb Bug fix: tokenizer used in mert; add unit tests for that. 
When tokenizing a string delimited by spaces (say, "9 9 8 7 ")
with Tokenize(), resulting a sequence of strings are
{"9", "9", "8", "7", "" }, which is different
from we have expected. We are not interested in empty strings.

This commit fix this issue, and add unit tests for
the tokenize functions.
 2012-02-20 07:39:24 +09:00
Tetsuo Kiso
8c7dfe04e7 Bug fix: tokenizer used in mert; add unit tests for that. 
When tokenizing a string delimited by spaces (say, "9 9 8 7 ")
with Tokenize(), resulting a sequence of strings are
{"9", "9", "8", "7", "" }, which is different
from we have expected. We are not interested in empty strings.

This commit fix this issue, and add unit tests for
the tokenize functions.
 2012-02-20 07:39:24 +09:00
Tetsuo Kiso
a7666735b5 Add error checking to setup 'to_optimize'. 
mert will check whether the dimension and the number of
fetures are equal.
 2012-02-17 09:16:10 +09:00
Tetsuo Kiso
4b6232b757 Add error checking to setup 'to_optimize'. 
mert will check whether the dimension and the number of
fetures are equal.
 2012-02-17 09:16:10 +09:00
Tetsuo Kiso
6c003e544a Bug fix mert: when you want to optimize fewer features. 
This commit is a temporary bug fix.
 2012-02-17 08:25:18 +09:00
Tetsuo Kiso
c5e7e4cea7 Bug fix mert: when you want to optimize fewer features. 
This commit is a temporary bug fix.
 2012-02-17 08:25:18 +09:00
Tetsuo Kiso
819dc9e0f9 Add a utility function to FeatureData for debugging. 2012-02-17 07:27:07 +09:00
Tetsuo Kiso
47b535ee0a Add a utility function to FeatureData for debugging. 2012-02-17 07:27:07 +09:00
Tetsuo Kiso
c1b85b480c Delete mert/sample/README; Add smoke tests. 
Replace README with a bunch of shell script
for smoke testing of MERT.

The README file was not a typical README file.
It was like a sample script to run mert and
extractor, so I renamed it as smoke tests stuff.
 2012-02-17 03:53:52 +09:00
Tetsuo Kiso
91645503e8 Delete mert/sample/README; Add smoke tests. 
Replace README with a bunch of shell script
for smoke testing of MERT.

The README file was not a typical README file.
It was like a sample script to run mert and
extractor, so I renamed it as smoke tests stuff.
 2012-02-17 03:53:52 +09:00
Barry Haddow
7091555cd6 Merge branch 'master' of github.com:moses-smt/mosesdecoder 2012-02-08 17:48:19 +00:00
Barry Haddow
69afc63fb0 Merge branch 'master' of github.com:moses-smt/mosesdecoder 2012-02-08 17:48:19 +00:00
Barry Haddow
fa6753b0f3 Really simple sharding test 2012-02-08 17:47:54 +00:00
Barry Haddow
757f08a141 Really simple sharding test 2012-02-08 17:47:54 +00:00
Barry Haddow
62d7d034bb Fix sharding bug 2012-02-08 17:11:56 +00:00
Barry Haddow
752724594e Fix sharding bug 2012-02-08 17:11:56 +00:00
Tetsuo Kiso
905f959d83 Move funcions defined in a header into .cpp file. 2012-02-01 21:44:37 +09:00
Tetsuo Kiso
270809fa97 Move funcions defined in a header into .cpp file. 2012-02-01 21:44:37 +09:00
Tetsuo Kiso
b2987337d8 Remove virtual keyword from whoami() function. 
The funtion is neither inherited from Scorer nor
StatisticsBasedScorer.
 2012-02-01 21:36:25 +09:00
Tetsuo Kiso
4e3dd11f53 Remove virtual keyword from whoami() function. 
The funtion is neither inherited from Scorer nor
StatisticsBasedScorer.
 2012-02-01 21:36:25 +09:00
Tetsuo Kiso
923efa0a4c Move funcionts used in StatisticsBasedScorer into anonymous namespace. 
And add 'inline' keyword to the functions.
 2012-02-01 21:30:50 +09:00
Tetsuo Kiso
4051f723f7 Move funcionts used in StatisticsBasedScorer into anonymous namespace. 
And add 'inline' keyword to the functions.
 2012-02-01 21:30:50 +09:00
Tetsuo Kiso
5cd5b90d0d Create a initialize function. 2012-02-01 21:26:47 +09:00
Tetsuo Kiso
ae5a58da89 Create a initialize function. 2012-02-01 21:26:47 +09:00
Tetsuo Kiso
17e864e446 Create private class to encapssulate encoding process. 
Instead of using typedefs inside a class only,
it might be better to create a private class to do same things.
 2012-02-01 21:19:25 +09:00
Tetsuo Kiso
92434a2293 Create private class to encapssulate encoding process. 
Instead of using typedefs inside a class only,
it might be better to create a private class to do same things.
 2012-02-01 21:19:25 +09:00
Tetsuo Kiso
a351a74c18 Move regularizaion type into StatisticsBasedScorer. 
The type is used as internal purpose.
 2012-02-01 20:58:49 +09:00
Tetsuo Kiso
c5f8c7d33a Move regularizaion type into StatisticsBasedScorer. 
The type is used as internal purpose.
 2012-02-01 20:58:49 +09:00
Tetsuo Kiso
b19e7777ce Add prefix 'm_' to private and protected members in Scorer classes. 2012-02-01 20:54:20 +09:00
Tetsuo Kiso
cb729ad3a5 Add prefix 'm_' to private and protected members in Scorer classes. 2012-02-01 20:54:20 +09:00
Tetsuo Kiso
33cea0f1dc Delete unused enums. 2012-02-01 20:26:25 +09:00
Tetsuo Kiso
947e30ac3b Delete unused enums. 2012-02-01 20:26:25 +09:00
Tetsuo Kiso
30fa97e404 Move reference length type into a private member of BleuScorer. 
The reason is that the type is used as internal purpose.
 2012-02-01 20:24:48 +09:00
Tetsuo Kiso
3ac41ff2d7 Move reference length type into a private member of BleuScorer. 
The reason is that the type is used as internal purpose.
 2012-02-01 20:24:48 +09:00
Tetsuo Kiso
d742341176 Bug fix: memory leak issues when using MergeScorer. 2012-02-01 18:13:32 +09:00
Tetsuo Kiso
671a2f8c33 Bug fix: memory leak issues when using MergeScorer. 2012-02-01 18:13:32 +09:00
Tetsuo Kiso
3ef03a77c4 Change casts to C++ style casts. 2012-02-01 18:13:00 +09:00
Tetsuo Kiso
4373c07cf0 Change casts to C++ style casts. 2012-02-01 18:13:00 +09:00
Tetsuo Kiso
142342f8be Change casts to C++ style casts, and delete unnecessary casts. 2012-02-01 17:17:58 +09:00
Tetsuo Kiso
194e24115a Change casts to C++ style casts, and delete unnecessary casts. 2012-02-01 17:17:58 +09:00
Tetsuo Kiso
3c04b7e826 Create a function to save final weights to a file. 2012-02-01 16:59:48 +09:00
Tetsuo Kiso
380f550cf3 Create a function to save final weights to a file. 2012-02-01 16:59:48 +09:00
Tetsuo Kiso
1bafc99ea2 Use a constant instead of hard-coding. 2012-02-01 16:51:25 +09:00
Tetsuo Kiso
2a7d2a395f Use a constant instead of hard-coding. 2012-02-01 16:51:25 +09:00
Tetsuo Kiso
8e1bf6d7d1 Pass std::string objects by const reference instead of by value. 2012-02-01 16:46:36 +09:00
Tetsuo Kiso
79fd25d0c4 Pass std::string objects by const reference instead of by value. 2012-02-01 16:46:36 +09:00
Tetsuo Kiso
b399dfc906 Rename too short variables. 
'D' and 'O' is hard to find when you might want to
search by command line tools such as grep.
 2012-02-01 16:39:38 +09:00
Tetsuo Kiso
12ccef6d50 Rename too short variables. 
'D' and 'O' is hard to find when you might want to
search by command line tools such as grep.
 2012-02-01 16:39:38 +09:00
Tetsuo Kiso
8007150db4 Add constans to prevent avoid hard coding many times. 
Default filenames in usage() were differen from the actual ones.
 2012-02-01 16:33:23 +09:00
Tetsuo Kiso
b7507a84b2 Add constans to prevent avoid hard coding many times. 
Default filenames in usage() were differen from the actual ones.
 2012-02-01 16:33:23 +09:00
Tetsuo Kiso
c3fbe5081e Add whitespaces. 2012-02-01 16:15:22 +09:00
Tetsuo Kiso
5b30f52fbe Add whitespaces. 2012-02-01 16:15:22 +09:00
Tetsuo Kiso
e8823d5015 Add 'virtual' keyword for functions in OptimizationTask. 
Adding 'virtual' keyword for an inherited virtual function is good
practice because it will let the users know where the function
comes from.
 2012-02-01 15:49:33 +09:00
Tetsuo Kiso
cac1e14d50 Add 'virtual' keyword for functions in OptimizationTask. 
Adding 'virtual' keyword for an inherited virtual function is good
practice because it will let the users know where the function
comes from.
 2012-02-01 15:49:33 +09:00
Tetsuo Kiso
c1c9c1c8bb Create a struct for command line options in mert. 2012-02-01 15:21:49 +09:00
Tetsuo Kiso
7c843dbe3f Create a struct for command line options in mert. 2012-02-01 15:21:49 +09:00
Tetsuo Kiso
616b014554 Delete unused variables. 2012-02-01 12:37:41 +09:00
Tetsuo Kiso
3c9f44494e Delete unused variables. 2012-02-01 12:37:41 +09:00
Tetsuo Kiso
bb9a6b9184 Introduce anonymous namespace. 2012-02-01 12:35:50 +09:00
Tetsuo Kiso
d4b9536d43 Introduce anonymous namespace. 2012-02-01 12:35:50 +09:00
Tetsuo Kiso
c94b1f8ff0 Add whitespaces. 2012-02-01 12:29:45 +09:00
Tetsuo Kiso
345448f950 Add whitespaces. 2012-02-01 12:29:45 +09:00
Tetsuo Kiso
cad03f7a03 Create a struct for command line options in extractor. 2012-02-01 12:23:15 +09:00
Tetsuo Kiso
99f4158d44 Create a struct for command line options in extractor. 2012-02-01 12:23:15 +09:00
Tetsuo Kiso
4d3fd9fd4b Create a wrapper function to init seed. 
Move g_bootstrap from a global variable to
a member of struct ProgramOption.
 2012-02-01 11:49:26 +09:00
Tetsuo Kiso
4fb90867b0 Create a wrapper function to init seed. 
Move g_bootstrap from a global variable to
a member of struct ProgramOption.
 2012-02-01 11:49:26 +09:00
Tetsuo Kiso
1452f88ed5 Create a struct for command line options. 
Add a wrapper function to parse the options, too.
 2012-02-01 11:27:17 +09:00
Tetsuo Kiso
ad24930547 Create a struct for command line options. 
Add a wrapper function to parse the options, too.
 2012-02-01 11:27:17 +09:00
Tetsuo Kiso
037af96a6e Create a utility class for mert/evaluator.cpp to avoid name collisions, just in case. 
And introduce anonymous namespace for the class and global variables
as well.
 2012-01-27 04:06:36 +09:00
Tetsuo Kiso
8d5e0cbe02 Create a utility class for mert/evaluator.cpp to avoid name collisions, just in case. 
And introduce anonymous namespace for the class and global variables
as well.
 2012-01-27 04:06:36 +09:00
Tetsuo Kiso
940dadaa4c Add whitespaces. 2012-01-27 03:39:13 +09:00
Tetsuo Kiso
0bf69e840a Add whitespaces. 2012-01-27 03:39:13 +09:00
Tetsuo Kiso
f9eac588e7 Add prefixe 'g_' to global variables in mert/evaluator.cpp 
While the size of mert/evaluator.cpp is still relatively small,
adding the marker to the variables allows us to easily distinguish
them from local variables.
 2012-01-27 03:24:51 +09:00
Tetsuo Kiso
cd292a2449 Add prefixe 'g_' to global variables in mert/evaluator.cpp 
While the size of mert/evaluator.cpp is still relatively small,
adding the marker to the variables allows us to easily distinguish
them from local variables.
 2012-01-27 03:24:51 +09:00
Matous Machacek
5254e7917b mert/evaluator should now compute confidence interval correctly 2012-01-24 21:25:15 +01:00
Matous Machacek
b3096acdec mert/evaluator should now compute confidence interval correctly 2012-01-24 21:25:15 +01:00
Matous Machacek
6cbdfc513b fixed bugs in mert/evaluator, nicer printing of results 2012-01-24 19:18:44 +01:00
Matous Machacek
40f720aee6 fixed bugs in mert/evaluator, nicer printing of results 2012-01-24 19:18:44 +01:00
Matous Machacek
b4a50ec50b mert/evaluator can compute more metrics at once 2012-01-22 01:01:08 +01:00
Matous Machacek
05aa103d5f mert/evaluator can compute more metrics at once 2012-01-22 01:01:08 +01:00
Barry Haddow
ced24a881d Implementation of feature-merging for pro-mert 2012-01-13 16:52:15 +00:00
cservan
fa107dda85 Fix the memory overflow 2011-12-23 11:14:34 +01:00
cservan
9ad507d19e Fix the memory overflow 2011-12-23 11:14:34 +01:00
Hieu Hoang
575168c277 uint -> size_t 2011-12-12 23:27:27 +07:00
Hieu Hoang
ca0a3ea870 uint -> size_t 2011-12-12 23:27:27 +07:00
Hieu Hoang
753eebd959 revert 2011-12-12 20:48:42 +07:00
Hieu Hoang
21009b5d1e revert 2011-12-12 20:48:42 +07:00
Tetsuo Kiso
442ddb7458 Pass by reference. 2011-12-08 14:18:37 +09:00
Tetsuo Kiso
95653828f9 Pass by reference. 2011-12-08 14:18:37 +09:00
Tetsuo Kiso
61568a602d Add explicit 2011-12-08 14:17:39 +09:00
Tetsuo Kiso
62d03a80f4 Add explicit 2011-12-08 14:17:39 +09:00
Kenneth Heafield
c53453b266 No more autotools (with bhaddow in the room) 2011-12-01 17:17:28 +00:00
Kenneth Heafield
42975b0635 No more autotools (with bhaddow in the room) 2011-12-01 17:17:28 +00:00
Tetsuo Kiso
4d189eb14d Fix a typedef for comparing N-grams. 
Declared const_iterator was not *const* actually.
 2011-11-30 00:27:57 +09:00
Tetsuo Kiso
a639116847 Fix a typedef for comparing N-grams. 
Declared const_iterator was not *const* actually.
 2011-11-30 00:27:57 +09:00
Kenneth Heafield
f564876e5e Avoid naming conflict with glibc's libutil 2011-11-28 09:19:45 +00:00
Kenneth Heafield
58054214b7 Avoid naming conflict with glibc's libutil 2011-11-28 09:19:45 +00:00
Kenneth Heafield
a7222a322d Regression tests expect mert in mert/ 2011-11-25 22:14:04 +00:00
Kenneth Heafield
6edf5fe9cc Regression tests expect mert in mert/ 2011-11-25 22:14:04 +00:00
Kenneth Heafield
463f2e232e Link dynamically against libz if we have to 2011-11-21 15:09:38 +00:00
Kenneth Heafield
f15eee0abf Jam: Clean up boost library handling, assimilate mert 2011-11-18 11:27:57 -05:00
Kenneth Heafield
bf78f7a1ac Replace assert with CHECK until people learn how to use assert properly 2011-11-18 12:07:41 +00:00
Rico Sennrich
57bbf82c27 broken/obsolete 2011-11-18 10:49:34 +01:00
Barry Haddow
ba5633c7b5 Merge remote branch 'origin/master' into pro 
Conflicts:
	kenlm
	mert/Data.cpp
	mert/Data.h
	mert/Makefile.am
	mert/mert.cpp
 2011-11-17 15:58:56 +00:00
Barry Haddow
71c777f01d Remove old PRO. Fix running of PRO from mert script. 2011-11-16 17:43:54 +00:00
Barry Haddow
aeead91b54 Output to file. Remove debug. 2011-11-16 16:26:01 +00:00
Barry Haddow
79de3c8699 Complete initial version of pro extractor 2011-11-16 14:54:23 +00:00
Tetsuo Kiso
d7bd58c6f5 Resolved conflicts 2011-11-15 22:41:21 +09:00
Barry Haddow
0a2e0f44a6 Finish and test feature and score data iterators. 2011-11-15 13:12:14 +00:00
Tetsuo Kiso
1ba9a4e868 Add error checking for file existence. 2011-11-15 10:17:00 +09:00
Tetsuo Kiso
9b8cb6a6d4 Fix typo. 2011-11-15 10:15:09 +09:00
bhaddow
3a6c0e0680 iterate through feature file 2011-11-14 22:18:22 +00:00
bhaddow
4cf6e0320a pro extraction mainline and stub of feature data iterator 2011-11-14 18:32:36 +00:00
bhaddow
cba190bee3 start new pro extractor 2011-11-14 16:21:16 +00:00
Barry Haddow
58268cc3dd Test for pro 2011-11-14 16:03:56 +00:00
Tetsuo Kiso
b7dd5ecfce Add shell script to run memcheck. 2011-11-14 20:17:58 +09:00
Tetsuo Kiso
284f37d09e Add missed code 2011-11-14 20:06:10 +09:00
Tetsuo Kiso
2fde1cab0e Add missing headers. 2011-11-14 19:52:21 +09:00
Tetsuo Kiso
29c16d252a Minimize using #include headers in headers. 
Should use it in .cpp files.
 2011-11-14 15:15:30 +09:00
Tetsuo Kiso
be1506e759 Use passing objects by const references not passing by their values. 2011-11-14 14:00:47 +09:00
Tetsuo Kiso
0c7a38d9d2 Prohibit the client from calling some constructors without arguments. 
The reason is to avoid the misuse of constructors that should be called
with arguments.
 2011-11-14 12:58:42 +09:00
Tetsuo Kiso
20afe4babb Fix uninitialized private members when constructors are called. 2011-11-14 12:53:20 +09:00
Tetsuo Kiso
625fe118e0 Fix bugs, private members were not initialized when the instances are created. 
When the objects of Data and FeatureData classes are created,
the primitive private members were not initialized properly.
 2011-11-14 12:20:04 +09:00
Tetsuo Kiso
8f0ba037f3 Add comments. 2011-11-13 21:13:44 +09:00
Tetsuo Kiso
3d70b2e1a5 Small change: modify initialization of the Data class. 2011-11-12 22:04:22 +09:00
Tetsuo Kiso
eecfb171da Fix usage of whitespaces for Optimizers. 2011-11-12 18:47:31 +09:00
Tetsuo Kiso
6c845323ac Fix the order of initialization when the constructor will be called. 2011-11-12 18:17:31 +09:00
Tetsuo Kiso
20e27aab92 Refactoring: constructors and copying objects. 2011-11-12 18:12:07 +09:00
Tetsuo Kiso
87a5c16b55 Replace macros with const variables and inline functions. 2011-11-12 17:30:33 +09:00
Tetsuo Kiso
ae9701ff77 Fix initialization of score_, and add destructor. 2011-11-12 16:50:18 +09:00
Tetsuo Kiso
df0874df89 Make Point::score a private by defining accessor/mutator. 2011-11-12 16:39:57 +09:00
Tetsuo Kiso
fdcd148cac Replace macros with const char arrays used in XXXArray classes. 2011-11-12 12:13:35 +09:00
Tetsuo Kiso
03abf54d21 Add a utility function to make sure the filename with ".gz" extension. 2011-11-12 11:50:39 +09:00
Tetsuo Kiso
ee0345b9b6 Move filestream wrapper classes to FileStream.{h,cpp} 2011-11-12 11:44:39 +09:00
Tetsuo Kiso
8e7693076c Add const to stream wrapper classes. 2011-11-12 11:35:20 +09:00
Tetsuo Kiso
fb3b0f9f62 Add const to Data and Feature class members. 2011-11-12 11:26:13 +09:00
Tetsuo Kiso
54b3b846c7 Add const member functions in Scorer classes. 2011-11-12 10:58:14 +09:00
Tetsuo Kiso
00b8c6d768 Use const Scorer::calculateScore(). 2011-11-12 10:40:54 +09:00
Tetsuo Kiso
1ec58e7edf Fix constructors and initializing const members. 2011-11-12 10:28:08 +09:00
Tetsuo Kiso
d776281b8b Simple refactoring of BLEU scorer. 2011-11-12 10:21:08 +09:00
Tetsuo Kiso
43beb88df5 Fix constructors of scorer classes and optimizer classes. 
Using public const members is not good idea.
It should be initialized in private by constructors.
 2011-11-12 10:16:31 +09:00
Tetsuo Kiso
fdb83b0f6d Put constructor and destructor into private for Factory class 2011-11-12 09:56:21 +09:00
Tetsuo Kiso
dfb714296f Add 'explicit' for constructors with one argument. 2011-11-12 09:51:27 +09:00
Tetsuo Kiso
ce9a628ed0 Remove unnecessary semicolons used in end of member functions. 2011-11-12 09:40:01 +09:00
Tetsuo Kiso
664ffe0130 Fix indentation. 2011-11-12 09:24:19 +09:00
Tetsuo Kiso
0fc56ae869 Add a comment to mark the end of #define guard. 2011-11-12 09:16:05 +09:00
Tetsuo Kiso
68315d6407 Fix class, function, and implementation comments format. 
Functions comments should be placed in their declarations.
 2011-11-12 08:58:23 +09:00
Tetsuo Kiso
4f6d022fe7 Add comments to mark the end of #define guards. 2011-11-12 07:59:50 +09:00
Tetsuo Kiso
888c47d921 Fix splitting strings from a string. 2011-11-11 22:00:30 +09:00
Tetsuo Kiso
c2121695c2 Fix memory leaks in mert. 2011-11-11 20:40:59 +09:00
Tetsuo Kiso
087756b8c3 Fix memory leaks in extractor. 2011-11-11 20:02:26 +09:00
Kenneth Heafield
1c4be5675c Lines are better when they end 2011-11-07 17:08:19 +00:00
bhaddow
7c0d9c34da Fix formatting of pro input file 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4360 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-14 14:01:15 +00:00
bhaddow
718c55d3aa Reducing memory usage for sparse features 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4355 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-14 07:40:53 +00:00
servan
eef0f213e9 A mert/MergeScorer.h 
A    mert/MergeScorer.cpp


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4303 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-06 11:00:48 +00:00
servan
f223f5a276 M mert/TerScorer.cpp 
M    mert/BleuScorer.h
M    mert/ScorerFactory.h
M    mert/Scorer.h
M    mert/PerScorer.h
M    mert/TerScorer.h
M    mert/Makefile.am
AM   scripts/training/mert-moses-multi.pl


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4299 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-05 13:36:17 +00:00
bhaddow
fc695c38a7 Implementation of sharding and resampling in mert. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4226 1f5c12ca-751b-0410-a591-d2e778427230
 2011-09-15 17:45:35 +00:00
servan
c55b03dd66 M TerScorer.cpp 
M    TerScorer.h


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4206 1f5c12ca-751b-0410-a591-d2e778427230
 2011-09-09 13:26:45 +00:00
bhaddow
0955474dea remove old tests 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4195 1f5c12ca-751b-0410-a591-d2e778427230
 2011-09-08 18:04:13 +00:00
bhaddow
ed8d85acc1 moving mert tests 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4193 1f5c12ca-751b-0410-a591-d2e778427230
 2011-09-08 17:51:09 +00:00
phkoehn
41a1849437 support for sparse feature functions (mert support only when using PRO) 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4184 1f5c12ca-751b-0410-a591-d2e778427230
 2011-09-07 16:37:33 +00:00
bhaddow
9fee4a97f2 clean-up 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4183 1f5c12ca-751b-0410-a591-d2e778427230
 2011-09-07 08:36:08 +00:00
bhaddow
ca5c0f19b7 Multi-threading of mert, for random restarts. 
Fix mert tests.


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4182 1f5c12ca-751b-0410-a591-d2e778427230
 2011-09-07 08:08:35 +00:00
bhaddow
37ebe7a62e Build common files into archive so we don't have to compile them 3 times 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4179 1f5c12ca-751b-0410-a591-d2e778427230
 2011-09-06 12:29:18 +00:00
machacekmatous
1daacfbb72 TER Scorer sources cleaned, compilable without boost now. TER Scorer added back to the default built. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4173 1f5c12ca-751b-0410-a591-d2e778427230
 2011-09-05 22:26:18 +00:00
bhaddow
1786d33e0f Make ter scoring optional 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4172 1f5c12ca-751b-0410-a591-d2e778427230
 2011-09-05 16:23:46 +00:00
machacekmatous
642e8dce95 Added evaluator to MERT directory. This tool computes a metric score for given candidate and reference files: 
evaluator --sctype PER --reference ref.file --candidate cand.file

usage: evaluator [options] --reference ref1[,ref2[,ref3...]] --candidate cand1[,cand2[,cand3...]]
[--sctype|-s] the scorer type (default BLEU)
[--scconfig|-c] configuration string passed to scorer
        This is of the form NAME1:VAL1,NAME2:VAL2 etc
[--reference|-R] comma separated list of reference files
[--candidate|-C] comma separated list of candidate files
[--bootstrap|-b] number of booststraped samples (default 0 - no bootstraping)
[--rseed|-r] the random seed for bootstraping (defaults to system clock)
[--help|-h] print this message and exit


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4153 1f5c12ca-751b-0410-a591-d2e778427230
 2011-08-20 15:25:19 +00:00
machacekmatous
63fd490a51 Added CDER metric to use in MERT. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4152 1f5c12ca-751b-0410-a591-d2e778427230
 2011-08-18 21:35:16 +00:00
bojar
998b86f639 addind a TODO list for anyone, esp. Matous Machacek 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4150 1f5c12ca-751b-0410-a591-d2e778427230
 2011-08-17 10:01:14 +00:00
machacekmatous
3ef02eb7e6 merged in TER Scorer from mert-other_metrics (at r4140) 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4146 1f5c12ca-751b-0410-a591-d2e778427230
 2011-08-16 16:21:31 +00:00
phkoehn
36db0ffe48 added pairwise ranked optimization (PRO) as proposd by [Hopkins&May,2011], just use switch --pairwise-ranked 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4106 1f5c12ca-751b-0410-a591-d2e778427230
 2011-08-03 17:00:17 +00:00
phkoehn
1bd74fc87f added random directions [Cer&al.,2008] and historic best as starting points [Foster&Kuhn,2009] to MERT 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4086 1f5c12ca-751b-0410-a591-d2e778427230
 2011-07-23 00:24:45 +00:00
bojar
1ba2de3c02 - cmert: added support for passing min and max values for weights 
(used to be in old cmert but not in new cmert, i.e. moses/mert/)
- modified mert-moses.pl accordingly, esp. set min&max to 0&1 as it used to be
  hardwired in the new cmert
- adding mert-moses-ondrej.pl, a simplification of mert-moses.pl, please test it


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4066 1f5c12ca-751b-0410-a591-d2e778427230
 2011-07-03 21:01:16 +00:00
hieuhoang1972
148c1e8305 run beautify.perl. Consistent formatting for .h & .cpp files 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3899 1f5c12ca-751b-0410-a591-d2e778427230
 2011-02-24 12:42:19 +00:00
rsennrich
ec00f9a916 fix to MERT: disable normaliziation when optimizing subset of features. 
before, active features were normalized to 1; optimizing one feature would always set it to 1, preventing any real optimization.

git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3845 1f5c12ca-751b-0410-a591-d2e778427230
 2011-01-25 16:10:47 +00:00
bhaddow
2e77dce57e improvement to error message 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3763 1f5c12ca-751b-0410-a591-d2e778427230
 2010-12-08 10:13:19 +00:00
bhaddow
6e8856edc2 Prevent intersections going to infinity 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3125 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-12 14:31:10 +00:00
jfouet
79a8f0f133 restore trace by default 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2950 1f5c12ca-751b-0410-a591-d2e778427230
 2010-03-01 11:32:15 +00:00
jfouet
e14dc33dc8 enable trace with a configure options and fix compilation error when trace are disabled 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2938 1f5c12ca-751b-0410-a591-d2e778427230
 2010-02-24 10:37:49 +00:00
nicolabertoldi
e0ae1ad605 now it is possible to perform mert on a subset of features 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2645 1f5c12ca-751b-0410-a591-d2e778427230
 2010-01-08 15:12:28 +00:00
nicolabertoldi
53ab10d620 minor changes to extractor-bin regr-test to make it compliant with both mac and linux 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2556 1f5c12ca-751b-0410-a591-d2e778427230
 2009-10-01 14:33:08 +00:00
nicolabertoldi
1a90b6a915 changing the randomize function -from rand() to random()- to have the same behavior of linux machine also on mac; changing the regression test accordingly 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2554 1f5c12ca-751b-0410-a591-d2e778427230
 2009-10-01 14:08:22 +00:00
nicolabertoldi
fa59f147b9 minor changes to extractor-bin regr-test to be compliant with both mac and linux 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2553 1f5c12ca-751b-0410-a591-d2e778427230
 2009-10-01 14:05:01 +00:00
nicolabertoldi
820e3dfe9c adding regression tests for mert 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2551 1f5c12ca-751b-0410-a591-d2e778427230
 2009-10-01 12:53:44 +00:00
nicolabertoldi
0393183eb4 mert software now works with different reference length policies: shortest, average, closest (default) and with case information (default is preserving case). Pay attention that both defaults are different from the previous version (which were shortest reflen and case-insensitive). 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2459 1f5c12ca-751b-0410-a591-d2e778427230
 2009-08-05 15:38:35 +00:00
bhaddow
1ef0f3844d Connect new mert to the moses build system 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2221 1f5c12ca-751b-0410-a591-d2e778427230
 2009-02-26 11:17:12 +00:00
bhaddow
bb9e2f427d Fix random restarts so that they are actually random, using the system clock 
for seeding.

Previously, new mert followed old mert in using the same set of "random" points 
for every iteration of the outer loop.


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2126 1f5c12ca-751b-0410-a591-d2e778427230
 2009-02-12 12:58:27 +00:00
bhaddow
6c8c8e9dc4 initial weights for toy example 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1989 1f5c12ca-751b-0410-a591-d2e778427230
 2009-01-22 10:25:14 +00:00
nicolabertoldi
830d9f3404 small change to reduce few useless computations 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1970 1f5c12ca-751b-0410-a591-d2e778427230
 2009-01-07 13:47:38 +00:00
nicolabertoldi
4b4c1b3973 imported utilities for timing from Moses 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1969 1f5c12ca-751b-0410-a591-d2e778427230
 2009-01-07 13:30:06 +00:00
nicolabertoldi
32029561da mert can now load more data files 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1942 1f5c12ca-751b-0410-a591-d2e778427230
 2008-11-18 18:51:02 +00:00
phkoehn
1b5d99ad26 added headers for standard compliance (gcc 4.3 on 64 bit linux) 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1905 1f5c12ca-751b-0410-a591-d2e778427230
 2008-10-16 21:14:38 +00:00
bhaddow
cd28f119c6 mert tests 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1881 1f5c12ca-751b-0410-a591-d2e778427230
 2008-09-01 17:18:51 +00:00
bhaddow
83f234cf17 Implementation of Cer et al mert regularisation. Use with argument such 
as --scconfig regtype:min,regwin:3 in extractor and mert. Only tested
on toy example so far.


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1860 1f5c12ca-751b-0410-a591-d2e778427230
 2008-06-24 19:27:18 +00:00
nicolabertoldi
e94834012d added facilities to read and write score statistics in binary format 
moved facilities for feature names in FeatureData object


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1824 1f5c12ca-751b-0410-a591-d2e778427230
 2008-06-05 17:03:54 +00:00
nicolabertoldi
8e96e68476 overall change of a variable name: array_ instead of array2_ 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1823 1f5c12ca-751b-0410-a591-d2e778427230
 2008-06-05 11:42:00 +00:00
nicolabertoldi
930e67c5e3 fixed another bug related to the handling of feature names 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1822 1f5c12ca-751b-0410-a591-d2e778427230
 2008-06-05 11:30:37 +00:00
nicolabertoldi
44d7e0e0f7 fixed a bug related to the handling of feature names 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1821 1f5c12ca-751b-0410-a591-d2e778427230
 2008-06-05 11:26:54 +00:00
bhaddow
37cf805139 Fix bug in output of scorestats to text file 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1820 1f5c12ca-751b-0410-a591-d2e778427230
 2008-06-05 11:08:29 +00:00
nicolabertoldi
281bf610b8 added binary read/load facility for feature data 
added names of features in the header
added methods to access the features by name


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1819 1f5c12ca-751b-0410-a591-d2e778427230
 2008-06-05 07:23:34 +00:00
jfouet
f849154a73 fix FeatureStats::savetxt 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1815 1f5c12ca-751b-0410-a591-d2e778427230
 2008-06-04 08:33:37 +00:00
jfouet
8e54557e9f fix crash in getNextPound 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1814 1f5c12ca-751b-0410-a591-d2e778427230
 2008-06-04 08:32:19 +00:00
nicolabertoldi
1402f3036b added ScorerFactory.h 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1811 1f5c12ca-751b-0410-a591-d2e778427230
 2008-06-03 08:56:37 +00:00
nicolabertoldi
af585bc492 nbest can be read from stdin, too 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1797 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-27 17:20:01 +00:00
nicolabertoldi
291260abf7 - made output more compliant with old version 
- added PerSCorer.h and BleuScorer.h
- stored feature names
- fixed bug about output of best Point


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1796 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-27 16:50:52 +00:00
jfouet
89194be5eb safer(hopefully) insertion of threshold in LineOptimize 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1795 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-23 12:02:14 +00:00
nicolabertoldi
c9593648bb change from int to unsigned where needed 
add some debugging output (to remove later)

git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1794 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-23 11:48:16 +00:00
bhaddow
e8d26f4f5c Fixes to optimiser to stop it crashing and failing assert 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1793 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-22 17:19:41 +00:00
nicolabertoldi
09b5a0a282 modified log outputs to e comliant with old mert 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1792 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-21 16:52:00 +00:00
bhaddow
f1fa67d15e Fixed small bug in initialisation of 1bests 
removed failing assert


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1791 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-21 16:22:43 +00:00
nicolabertoldi
f87b4261cf I fixed a small bug when reading parameters 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1789 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-21 10:03:48 +00:00
nicolabertoldi
b611428b2b added missing file for managing gzipped files 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1788 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-21 09:33:19 +00:00
nicolabertoldi
8cf59edcdc remove loadnbest from FeatureData and Scoredata; change test_scorer accordingly; 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1787 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-20 14:33:47 +00:00
nicolabertoldi
8a594fc254 reading from textual gzipped file is now possible 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1786 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-20 14:15:30 +00:00
jfouet
f30000b875 add type threshold 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1785 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-20 12:57:32 +00:00
jfouet
6ee4084797 simplification of LineOptimizer code (same behaviour) 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1784 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-20 12:42:11 +00:00
bhaddow
39cf94984c Fix bug in finding of leftmost, higher line in optimizer 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1783 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-20 10:21:20 +00:00
bhaddow
86772c27e2 Fixed return of non-initialised variable 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1782 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-17 13:24:09 +00:00
bhaddow
74fb532e00 Option to use random optimiser 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1781 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-17 12:19:02 +00:00
bhaddow
83300050f7 Remove extraneous debug messages 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1766 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-16 20:24:00 +00:00
bhaddow
4ec448b3f2 Command line arg tidy up in extractor 
Remove buffers from feature


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1761 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-16 19:57:01 +00:00
bhaddow
597c960933 Remove extra debug 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1752 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-16 16:58:15 +00:00
bhaddow
1512abb7bd Save correct weight set! 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1750 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-16 16:51:39 +00:00
bhaddow
61e7ca1437 Fixes to optimiser: correct calculation of intersection, choice of bestx and prevent it 
from deleting best point


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1749 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-16 16:47:01 +00:00
jfouet
2b07443683 bugfixes 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1740 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-16 14:21:24 +00:00
jfouet
044a43b512 add init.opt file 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1731 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-16 12:35:17 +00:00
jfouet
e9df80a05c bugfixes 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1726 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-16 10:57:24 +00:00
jfouet
bb42cb0dd5 bugfixes+ introduction of verbose level 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1723 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-16 07:09:15 +00:00
jfouet
ad79af41df bug fixes 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1719 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-15 20:49:49 +00:00
bhaddow
165f7d049f pass by reference 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1717 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-15 20:44:24 +00:00
bhaddow
1a6dcf5e36 remove buffers 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1715 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-15 20:32:37 +00:00
bhaddow
6112215974 Fix crash in parameter reading 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1714 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-15 19:26:35 +00:00
jfouet
b231ffc8b1 add Types.h to unify the typedefs 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1713 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-15 19:09:01 +00:00
bhaddow
ad2ee87603 Fix makefile 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1711 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-15 17:16:46 +00:00
bhaddow
933400a503 extractor uses scorer factory 
remove feature_extractor


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1710 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-15 17:14:11 +00:00
nicolabertoldi
cc1d95272a add READMEs 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1707 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-15 16:20:23 +00:00
bhaddow
c0643d47f2 Add scorer factory. Fix compile error in Optimizer 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1706 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-15 16:03:49 +00:00
bhaddow
f320cf5174 Refactor PerScorer and BleuScorer to remove common code 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1704 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-15 14:48:11 +00:00
jfouet
b0ee845d7e various small fixes 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1702 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-15 14:17:34 +00:00
bhaddow
777ea5f754 Finish per scorer 
Command line args for extractor


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1700 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-15 14:13:32 +00:00
jfouet
c66aec9e83 implementation of the main 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1698 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-15 14:04:42 +00:00
jfouet
94d47101ca add (almost empty) main 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1696 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-15 11:06:32 +00:00
jfouet
bfe3661110 implementation of optimization on a subset of the parameters + debug mode in the makefile 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1695 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-15 10:57:20 +00:00
nicolabertoldi
2ef6f3e2d8 main command for managing feature and error statistics is ready; small example fortesting is available in directory example 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1692 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-15 08:35:56 +00:00
bhaddow
1f84c9eab7 Get rid of warnings about lack of newlines 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1691 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-15 06:44:36 +00:00
bhaddow
f98de25e70 Stub out per scorer. some refactoring to make this possible 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1681 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-14 20:36:11 +00:00
bhaddow
e2921af063 Incremental interface for scorer 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1678 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-14 19:47:34 +00:00
bhaddow
69c6164b82 normalisation script 
normalisation of test data


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1673 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-14 17:17:44 +00:00
bhaddow
48533d1021 Fixes to bleu scorer 
add method get ref length


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1669 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-14 16:31:22 +00:00
jfouet
cff8d4702b add some sanity checks 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1668 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-14 16:19:00 +00:00
jfouet
13f6a427c5 buxfixes on lineOptimize 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1665 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-14 15:44:59 +00:00
jfouet
cb5305ab46 implementation of LineOptimizer 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1659 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-14 14:25:07 +00:00
nicolabertoldi
9202738527 support for mergin; other small changes 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1657 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-14 14:16:05 +00:00
bhaddow
87efff7d66 bleu scorer 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1653 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-14 13:36:55 +00:00
jfouet
558c8bd9b7 fix compilation of Optimizer 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1651 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-14 12:49:45 +00:00
bhaddow
b672d9f6df Stubbing out of Scorer. 
Scorer test data.


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1646 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-14 12:23:58 +00:00
jfouet
43ba7835bb class to inplement a set of lambda parameter and their stat score 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1642 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-14 11:32:17 +00:00
nicolabertoldi
36b25323c9 small changes 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1641 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-14 11:28:54 +00:00
jfouet
40c93618fc add const get accesor to FeatureDataand Array 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1639 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-14 11:11:55 +00:00
nicolabertoldi
b68b41f7e2 add Parameter 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1638 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-14 11:05:32 +00:00
jfouet
f9448b76a0 objet code for the Optimzer algorithm 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1637 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-14 11:00:59 +00:00
nicolabertoldi
eb2d6e971d changes 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1636 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-14 10:51:42 +00:00
bhaddow
d794418121 First version of scorer 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1634 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-14 08:46:15 +00:00
nicolabertoldi
9c12330785 I am adding new object for handling statistics forerror measures 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1633 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-14 08:14:13 +00:00
nicolabertoldi
49891adeed I am starting the improvement of Minimum Error Training scritps 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1632 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-14 07:57:45 +00:00