moses-smt/mosesdecoder
1
1
Fork 0
mirror of https://github.com/moses-smt/mosesdecoder.git synced 2024-12-27 22:14:57 +03:00
Code Issues Packages Projects Releases Wiki Activity
3,138 Commits 136 Branches 26 Tags 147 MiB
fe79b96328
Commit Graph

310 Commits

Author SHA1 Message Date
Barry Haddow
7091555cd6 Merge branch 'master' of github.com:moses-smt/mosesdecoder 2012-02-08 17:48:19 +00:00
Barry Haddow
fa6753b0f3 Really simple sharding test 2012-02-08 17:47:54 +00:00
Barry Haddow
62d7d034bb Fix sharding bug 2012-02-08 17:11:56 +00:00
Tetsuo Kiso
905f959d83 Move funcions defined in a header into .cpp file. 2012-02-01 21:44:37 +09:00
Tetsuo Kiso
b2987337d8 Remove virtual keyword from whoami() function. 
The funtion is neither inherited from Scorer nor
StatisticsBasedScorer.
 2012-02-01 21:36:25 +09:00
Tetsuo Kiso
923efa0a4c Move funcionts used in StatisticsBasedScorer into anonymous namespace. 
And add 'inline' keyword to the functions.
 2012-02-01 21:30:50 +09:00
Tetsuo Kiso
5cd5b90d0d Create a initialize function. 2012-02-01 21:26:47 +09:00
Tetsuo Kiso
17e864e446 Create private class to encapssulate encoding process. 
Instead of using typedefs inside a class only,
it might be better to create a private class to do same things.
 2012-02-01 21:19:25 +09:00
Tetsuo Kiso
a351a74c18 Move regularizaion type into StatisticsBasedScorer. 
The type is used as internal purpose.
 2012-02-01 20:58:49 +09:00
Tetsuo Kiso
b19e7777ce Add prefix 'm_' to private and protected members in Scorer classes. 2012-02-01 20:54:20 +09:00
Tetsuo Kiso
33cea0f1dc Delete unused enums. 2012-02-01 20:26:25 +09:00
Tetsuo Kiso
30fa97e404 Move reference length type into a private member of BleuScorer. 
The reason is that the type is used as internal purpose.
 2012-02-01 20:24:48 +09:00
Tetsuo Kiso
d742341176 Bug fix: memory leak issues when using MergeScorer. 2012-02-01 18:13:32 +09:00
Tetsuo Kiso
3ef03a77c4 Change casts to C++ style casts. 2012-02-01 18:13:00 +09:00
Tetsuo Kiso
142342f8be Change casts to C++ style casts, and delete unnecessary casts. 2012-02-01 17:17:58 +09:00
Tetsuo Kiso
3c04b7e826 Create a function to save final weights to a file. 2012-02-01 16:59:48 +09:00
Tetsuo Kiso
1bafc99ea2 Use a constant instead of hard-coding. 2012-02-01 16:51:25 +09:00
Tetsuo Kiso
8e1bf6d7d1 Pass std::string objects by const reference instead of by value. 2012-02-01 16:46:36 +09:00
Tetsuo Kiso
b399dfc906 Rename too short variables. 
'D' and 'O' is hard to find when you might want to
search by command line tools such as grep.
 2012-02-01 16:39:38 +09:00
Tetsuo Kiso
8007150db4 Add constans to prevent avoid hard coding many times. 
Default filenames in usage() were differen from the actual ones.
 2012-02-01 16:33:23 +09:00
Tetsuo Kiso
c3fbe5081e Add whitespaces. 2012-02-01 16:15:22 +09:00
Tetsuo Kiso
e8823d5015 Add 'virtual' keyword for functions in OptimizationTask. 
Adding 'virtual' keyword for an inherited virtual function is good
practice because it will let the users know where the function
comes from.
 2012-02-01 15:49:33 +09:00
Tetsuo Kiso
c1c9c1c8bb Create a struct for command line options in mert. 2012-02-01 15:21:49 +09:00
Tetsuo Kiso
616b014554 Delete unused variables. 2012-02-01 12:37:41 +09:00
Tetsuo Kiso
bb9a6b9184 Introduce anonymous namespace. 2012-02-01 12:35:50 +09:00
Tetsuo Kiso
c94b1f8ff0 Add whitespaces. 2012-02-01 12:29:45 +09:00
Tetsuo Kiso
cad03f7a03 Create a struct for command line options in extractor. 2012-02-01 12:23:15 +09:00
Tetsuo Kiso
4d3fd9fd4b Create a wrapper function to init seed. 
Move g_bootstrap from a global variable to
a member of struct ProgramOption.
 2012-02-01 11:49:26 +09:00
Tetsuo Kiso
1452f88ed5 Create a struct for command line options. 
Add a wrapper function to parse the options, too.
 2012-02-01 11:27:17 +09:00
Tetsuo Kiso
037af96a6e Create a utility class for mert/evaluator.cpp to avoid name collisions, just in case. 
And introduce anonymous namespace for the class and global variables
as well.
 2012-01-27 04:06:36 +09:00
Tetsuo Kiso
940dadaa4c Add whitespaces. 2012-01-27 03:39:13 +09:00
Tetsuo Kiso
f9eac588e7 Add prefixe 'g_' to global variables in mert/evaluator.cpp 
While the size of mert/evaluator.cpp is still relatively small,
adding the marker to the variables allows us to easily distinguish
them from local variables.
 2012-01-27 03:24:51 +09:00
Matous Machacek
5254e7917b mert/evaluator should now compute confidence interval correctly 2012-01-24 21:25:15 +01:00
Matous Machacek
6cbdfc513b fixed bugs in mert/evaluator, nicer printing of results 2012-01-24 19:18:44 +01:00
Matous Machacek
b4a50ec50b mert/evaluator can compute more metrics at once 2012-01-22 01:01:08 +01:00
cservan
fa107dda85 Fix the memory overflow 2011-12-23 11:14:34 +01:00
Hieu Hoang
575168c277 uint -> size_t 2011-12-12 23:27:27 +07:00
Hieu Hoang
753eebd959 revert 2011-12-12 20:48:42 +07:00
Tetsuo Kiso
442ddb7458 Pass by reference. 2011-12-08 14:18:37 +09:00
Tetsuo Kiso
61568a602d Add explicit 2011-12-08 14:17:39 +09:00
Kenneth Heafield
c53453b266 No more autotools (with bhaddow in the room) 2011-12-01 17:17:28 +00:00
Tetsuo Kiso
4d189eb14d Fix a typedef for comparing N-grams. 
Declared const_iterator was not *const* actually.
 2011-11-30 00:27:57 +09:00
Kenneth Heafield
f564876e5e Avoid naming conflict with glibc's libutil 2011-11-28 09:19:45 +00:00
Kenneth Heafield
a7222a322d Regression tests expect mert in mert/ 2011-11-25 22:14:04 +00:00
Kenneth Heafield
463f2e232e Link dynamically against libz if we have to 2011-11-21 15:09:38 +00:00
Kenneth Heafield
f15eee0abf Jam: Clean up boost library handling, assimilate mert 2011-11-18 11:27:57 -05:00
Kenneth Heafield
bf78f7a1ac Replace assert with CHECK until people learn how to use assert properly 2011-11-18 12:07:41 +00:00
Rico Sennrich
57bbf82c27 broken/obsolete 2011-11-18 10:49:34 +01:00
Barry Haddow
ba5633c7b5 Merge remote branch 'origin/master' into pro 
Conflicts:
	kenlm
	mert/Data.cpp
	mert/Data.h
	mert/Makefile.am
	mert/mert.cpp
 2011-11-17 15:58:56 +00:00
Barry Haddow
71c777f01d Remove old PRO. Fix running of PRO from mert script. 2011-11-16 17:43:54 +00:00
Barry Haddow
aeead91b54 Output to file. Remove debug. 2011-11-16 16:26:01 +00:00
Barry Haddow
79de3c8699 Complete initial version of pro extractor 2011-11-16 14:54:23 +00:00
Tetsuo Kiso
d7bd58c6f5 Resolved conflicts 2011-11-15 22:41:21 +09:00
Barry Haddow
0a2e0f44a6 Finish and test feature and score data iterators. 2011-11-15 13:12:14 +00:00
Tetsuo Kiso
1ba9a4e868 Add error checking for file existence. 2011-11-15 10:17:00 +09:00
Tetsuo Kiso
9b8cb6a6d4 Fix typo. 2011-11-15 10:15:09 +09:00
bhaddow
3a6c0e0680 iterate through feature file 2011-11-14 22:18:22 +00:00
bhaddow
4cf6e0320a pro extraction mainline and stub of feature data iterator 2011-11-14 18:32:36 +00:00
bhaddow
cba190bee3 start new pro extractor 2011-11-14 16:21:16 +00:00
Barry Haddow
58268cc3dd Test for pro 2011-11-14 16:03:56 +00:00
Tetsuo Kiso
b7dd5ecfce Add shell script to run memcheck. 2011-11-14 20:17:58 +09:00
Tetsuo Kiso
284f37d09e Add missed code 2011-11-14 20:06:10 +09:00
Tetsuo Kiso
2fde1cab0e Add missing headers. 2011-11-14 19:52:21 +09:00
Tetsuo Kiso
29c16d252a Minimize using #include headers in headers. 
Should use it in .cpp files.
 2011-11-14 15:15:30 +09:00
Tetsuo Kiso
be1506e759 Use passing objects by const references not passing by their values. 2011-11-14 14:00:47 +09:00
Tetsuo Kiso
0c7a38d9d2 Prohibit the client from calling some constructors without arguments. 
The reason is to avoid the misuse of constructors that should be called
with arguments.
 2011-11-14 12:58:42 +09:00
Tetsuo Kiso
20afe4babb Fix uninitialized private members when constructors are called. 2011-11-14 12:53:20 +09:00
Tetsuo Kiso
625fe118e0 Fix bugs, private members were not initialized when the instances are created. 
When the objects of Data and FeatureData classes are created,
the primitive private members were not initialized properly.
 2011-11-14 12:20:04 +09:00
Tetsuo Kiso
8f0ba037f3 Add comments. 2011-11-13 21:13:44 +09:00
Tetsuo Kiso
3d70b2e1a5 Small change: modify initialization of the Data class. 2011-11-12 22:04:22 +09:00
Tetsuo Kiso
eecfb171da Fix usage of whitespaces for Optimizers. 2011-11-12 18:47:31 +09:00
Tetsuo Kiso
6c845323ac Fix the order of initialization when the constructor will be called. 2011-11-12 18:17:31 +09:00
Tetsuo Kiso
20e27aab92 Refactoring: constructors and copying objects. 2011-11-12 18:12:07 +09:00
Tetsuo Kiso
87a5c16b55 Replace macros with const variables and inline functions. 2011-11-12 17:30:33 +09:00
Tetsuo Kiso
ae9701ff77 Fix initialization of score_, and add destructor. 2011-11-12 16:50:18 +09:00
Tetsuo Kiso
df0874df89 Make Point::score a private by defining accessor/mutator. 2011-11-12 16:39:57 +09:00
Tetsuo Kiso
fdcd148cac Replace macros with const char arrays used in XXXArray classes. 2011-11-12 12:13:35 +09:00
Tetsuo Kiso
03abf54d21 Add a utility function to make sure the filename with ".gz" extension. 2011-11-12 11:50:39 +09:00
Tetsuo Kiso
ee0345b9b6 Move filestream wrapper classes to FileStream.{h,cpp} 2011-11-12 11:44:39 +09:00
Tetsuo Kiso
8e7693076c Add const to stream wrapper classes. 2011-11-12 11:35:20 +09:00
Tetsuo Kiso
fb3b0f9f62 Add const to Data and Feature class members. 2011-11-12 11:26:13 +09:00
Tetsuo Kiso
54b3b846c7 Add const member functions in Scorer classes. 2011-11-12 10:58:14 +09:00
Tetsuo Kiso
00b8c6d768 Use const Scorer::calculateScore(). 2011-11-12 10:40:54 +09:00
Tetsuo Kiso
1ec58e7edf Fix constructors and initializing const members. 2011-11-12 10:28:08 +09:00
Tetsuo Kiso
d776281b8b Simple refactoring of BLEU scorer. 2011-11-12 10:21:08 +09:00
Tetsuo Kiso
43beb88df5 Fix constructors of scorer classes and optimizer classes. 
Using public const members is not good idea.
It should be initialized in private by constructors.
 2011-11-12 10:16:31 +09:00
Tetsuo Kiso
fdb83b0f6d Put constructor and destructor into private for Factory class 2011-11-12 09:56:21 +09:00
Tetsuo Kiso
dfb714296f Add 'explicit' for constructors with one argument. 2011-11-12 09:51:27 +09:00
Tetsuo Kiso
ce9a628ed0 Remove unnecessary semicolons used in end of member functions. 2011-11-12 09:40:01 +09:00
Tetsuo Kiso
664ffe0130 Fix indentation. 2011-11-12 09:24:19 +09:00
Tetsuo Kiso
0fc56ae869 Add a comment to mark the end of #define guard. 2011-11-12 09:16:05 +09:00
Tetsuo Kiso
68315d6407 Fix class, function, and implementation comments format. 
Functions comments should be placed in their declarations.
 2011-11-12 08:58:23 +09:00
Tetsuo Kiso
4f6d022fe7 Add comments to mark the end of #define guards. 2011-11-12 07:59:50 +09:00
Tetsuo Kiso
888c47d921 Fix splitting strings from a string. 2011-11-11 22:00:30 +09:00
Tetsuo Kiso
c2121695c2 Fix memory leaks in mert. 2011-11-11 20:40:59 +09:00
Tetsuo Kiso
087756b8c3 Fix memory leaks in extractor. 2011-11-11 20:02:26 +09:00
Kenneth Heafield
1c4be5675c Lines are better when they end 2011-11-07 17:08:19 +00:00
bhaddow
7c0d9c34da Fix formatting of pro input file 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4360 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-14 14:01:15 +00:00
bhaddow
718c55d3aa Reducing memory usage for sparse features 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4355 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-14 07:40:53 +00:00
servan
eef0f213e9 A mert/MergeScorer.h 
A    mert/MergeScorer.cpp


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4303 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-06 11:00:48 +00:00
servan
f223f5a276 M mert/TerScorer.cpp 
M    mert/BleuScorer.h
M    mert/ScorerFactory.h
M    mert/Scorer.h
M    mert/PerScorer.h
M    mert/TerScorer.h
M    mert/Makefile.am
AM   scripts/training/mert-moses-multi.pl


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4299 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-05 13:36:17 +00:00
bhaddow
fc695c38a7 Implementation of sharding and resampling in mert. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4226 1f5c12ca-751b-0410-a591-d2e778427230
 2011-09-15 17:45:35 +00:00
servan
c55b03dd66 M TerScorer.cpp 
M    TerScorer.h


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4206 1f5c12ca-751b-0410-a591-d2e778427230
 2011-09-09 13:26:45 +00:00
bhaddow
0955474dea remove old tests 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4195 1f5c12ca-751b-0410-a591-d2e778427230
 2011-09-08 18:04:13 +00:00
bhaddow
ed8d85acc1 moving mert tests 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4193 1f5c12ca-751b-0410-a591-d2e778427230
 2011-09-08 17:51:09 +00:00
phkoehn
41a1849437 support for sparse feature functions (mert support only when using PRO) 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4184 1f5c12ca-751b-0410-a591-d2e778427230
 2011-09-07 16:37:33 +00:00
bhaddow
9fee4a97f2 clean-up 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4183 1f5c12ca-751b-0410-a591-d2e778427230
 2011-09-07 08:36:08 +00:00
bhaddow
ca5c0f19b7 Multi-threading of mert, for random restarts. 
Fix mert tests.


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4182 1f5c12ca-751b-0410-a591-d2e778427230
 2011-09-07 08:08:35 +00:00
bhaddow
37ebe7a62e Build common files into archive so we don't have to compile them 3 times 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4179 1f5c12ca-751b-0410-a591-d2e778427230
 2011-09-06 12:29:18 +00:00
machacekmatous
1daacfbb72 TER Scorer sources cleaned, compilable without boost now. TER Scorer added back to the default built. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4173 1f5c12ca-751b-0410-a591-d2e778427230
 2011-09-05 22:26:18 +00:00
bhaddow
1786d33e0f Make ter scoring optional 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4172 1f5c12ca-751b-0410-a591-d2e778427230
 2011-09-05 16:23:46 +00:00
machacekmatous
642e8dce95 Added evaluator to MERT directory. This tool computes a metric score for given candidate and reference files: 
evaluator --sctype PER --reference ref.file --candidate cand.file

usage: evaluator [options] --reference ref1[,ref2[,ref3...]] --candidate cand1[,cand2[,cand3...]]
[--sctype|-s] the scorer type (default BLEU)
[--scconfig|-c] configuration string passed to scorer
        This is of the form NAME1:VAL1,NAME2:VAL2 etc
[--reference|-R] comma separated list of reference files
[--candidate|-C] comma separated list of candidate files
[--bootstrap|-b] number of booststraped samples (default 0 - no bootstraping)
[--rseed|-r] the random seed for bootstraping (defaults to system clock)
[--help|-h] print this message and exit


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4153 1f5c12ca-751b-0410-a591-d2e778427230
 2011-08-20 15:25:19 +00:00
machacekmatous
63fd490a51 Added CDER metric to use in MERT. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4152 1f5c12ca-751b-0410-a591-d2e778427230
 2011-08-18 21:35:16 +00:00
bojar
998b86f639 addind a TODO list for anyone, esp. Matous Machacek 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4150 1f5c12ca-751b-0410-a591-d2e778427230
 2011-08-17 10:01:14 +00:00
machacekmatous
3ef02eb7e6 merged in TER Scorer from mert-other_metrics (at r4140) 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4146 1f5c12ca-751b-0410-a591-d2e778427230
 2011-08-16 16:21:31 +00:00
phkoehn
36db0ffe48 added pairwise ranked optimization (PRO) as proposd by [Hopkins&May,2011], just use switch --pairwise-ranked 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4106 1f5c12ca-751b-0410-a591-d2e778427230
 2011-08-03 17:00:17 +00:00
phkoehn
1bd74fc87f added random directions [Cer&al.,2008] and historic best as starting points [Foster&Kuhn,2009] to MERT 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4086 1f5c12ca-751b-0410-a591-d2e778427230
 2011-07-23 00:24:45 +00:00
bojar
1ba2de3c02 - cmert: added support for passing min and max values for weights 
(used to be in old cmert but not in new cmert, i.e. moses/mert/)
- modified mert-moses.pl accordingly, esp. set min&max to 0&1 as it used to be
  hardwired in the new cmert
- adding mert-moses-ondrej.pl, a simplification of mert-moses.pl, please test it


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4066 1f5c12ca-751b-0410-a591-d2e778427230
 2011-07-03 21:01:16 +00:00
hieuhoang1972
148c1e8305 run beautify.perl. Consistent formatting for .h & .cpp files 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3899 1f5c12ca-751b-0410-a591-d2e778427230
 2011-02-24 12:42:19 +00:00
rsennrich
ec00f9a916 fix to MERT: disable normaliziation when optimizing subset of features. 
before, active features were normalized to 1; optimizing one feature would always set it to 1, preventing any real optimization.

git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3845 1f5c12ca-751b-0410-a591-d2e778427230
 2011-01-25 16:10:47 +00:00
bhaddow
2e77dce57e improvement to error message 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3763 1f5c12ca-751b-0410-a591-d2e778427230
 2010-12-08 10:13:19 +00:00
bhaddow
6e8856edc2 Prevent intersections going to infinity 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3125 1f5c12ca-751b-0410-a591-d2e778427230
 2010-04-12 14:31:10 +00:00
jfouet
79a8f0f133 restore trace by default 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2950 1f5c12ca-751b-0410-a591-d2e778427230
 2010-03-01 11:32:15 +00:00
jfouet
e14dc33dc8 enable trace with a configure options and fix compilation error when trace are disabled 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2938 1f5c12ca-751b-0410-a591-d2e778427230
 2010-02-24 10:37:49 +00:00
nicolabertoldi
e0ae1ad605 now it is possible to perform mert on a subset of features 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2645 1f5c12ca-751b-0410-a591-d2e778427230
 2010-01-08 15:12:28 +00:00
nicolabertoldi
53ab10d620 minor changes to extractor-bin regr-test to make it compliant with both mac and linux 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2556 1f5c12ca-751b-0410-a591-d2e778427230
 2009-10-01 14:33:08 +00:00
nicolabertoldi
1a90b6a915 changing the randomize function -from rand() to random()- to have the same behavior of linux machine also on mac; changing the regression test accordingly 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2554 1f5c12ca-751b-0410-a591-d2e778427230
 2009-10-01 14:08:22 +00:00
nicolabertoldi
fa59f147b9 minor changes to extractor-bin regr-test to be compliant with both mac and linux 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2553 1f5c12ca-751b-0410-a591-d2e778427230
 2009-10-01 14:05:01 +00:00
nicolabertoldi
820e3dfe9c adding regression tests for mert 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2551 1f5c12ca-751b-0410-a591-d2e778427230
 2009-10-01 12:53:44 +00:00
nicolabertoldi
0393183eb4 mert software now works with different reference length policies: shortest, average, closest (default) and with case information (default is preserving case). Pay attention that both defaults are different from the previous version (which were shortest reflen and case-insensitive). 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2459 1f5c12ca-751b-0410-a591-d2e778427230
 2009-08-05 15:38:35 +00:00
bhaddow
1ef0f3844d Connect new mert to the moses build system 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2221 1f5c12ca-751b-0410-a591-d2e778427230
 2009-02-26 11:17:12 +00:00
bhaddow
bb9e2f427d Fix random restarts so that they are actually random, using the system clock 
for seeding.

Previously, new mert followed old mert in using the same set of "random" points 
for every iteration of the outer loop.


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2126 1f5c12ca-751b-0410-a591-d2e778427230
 2009-02-12 12:58:27 +00:00
bhaddow
6c8c8e9dc4 initial weights for toy example 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1989 1f5c12ca-751b-0410-a591-d2e778427230
 2009-01-22 10:25:14 +00:00
nicolabertoldi
830d9f3404 small change to reduce few useless computations 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1970 1f5c12ca-751b-0410-a591-d2e778427230
 2009-01-07 13:47:38 +00:00
nicolabertoldi
4b4c1b3973 imported utilities for timing from Moses 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1969 1f5c12ca-751b-0410-a591-d2e778427230
 2009-01-07 13:30:06 +00:00
nicolabertoldi
32029561da mert can now load more data files 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1942 1f5c12ca-751b-0410-a591-d2e778427230
 2008-11-18 18:51:02 +00:00
phkoehn
1b5d99ad26 added headers for standard compliance (gcc 4.3 on 64 bit linux) 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1905 1f5c12ca-751b-0410-a591-d2e778427230
 2008-10-16 21:14:38 +00:00
bhaddow
cd28f119c6 mert tests 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1881 1f5c12ca-751b-0410-a591-d2e778427230
 2008-09-01 17:18:51 +00:00
bhaddow
83f234cf17 Implementation of Cer et al mert regularisation. Use with argument such 
as --scconfig regtype:min,regwin:3 in extractor and mert. Only tested
on toy example so far.


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1860 1f5c12ca-751b-0410-a591-d2e778427230
 2008-06-24 19:27:18 +00:00
nicolabertoldi
e94834012d added facilities to read and write score statistics in binary format 
moved facilities for feature names in FeatureData object


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1824 1f5c12ca-751b-0410-a591-d2e778427230
 2008-06-05 17:03:54 +00:00
nicolabertoldi
8e96e68476 overall change of a variable name: array_ instead of array2_ 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1823 1f5c12ca-751b-0410-a591-d2e778427230
 2008-06-05 11:42:00 +00:00
nicolabertoldi
930e67c5e3 fixed another bug related to the handling of feature names 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1822 1f5c12ca-751b-0410-a591-d2e778427230
 2008-06-05 11:30:37 +00:00
nicolabertoldi
44d7e0e0f7 fixed a bug related to the handling of feature names 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1821 1f5c12ca-751b-0410-a591-d2e778427230
 2008-06-05 11:26:54 +00:00
bhaddow
37cf805139 Fix bug in output of scorestats to text file 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1820 1f5c12ca-751b-0410-a591-d2e778427230
 2008-06-05 11:08:29 +00:00
nicolabertoldi
281bf610b8 added binary read/load facility for feature data 
added names of features in the header
added methods to access the features by name


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1819 1f5c12ca-751b-0410-a591-d2e778427230
 2008-06-05 07:23:34 +00:00
jfouet
f849154a73 fix FeatureStats::savetxt 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1815 1f5c12ca-751b-0410-a591-d2e778427230
 2008-06-04 08:33:37 +00:00
jfouet
8e54557e9f fix crash in getNextPound 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1814 1f5c12ca-751b-0410-a591-d2e778427230
 2008-06-04 08:32:19 +00:00
nicolabertoldi
1402f3036b added ScorerFactory.h 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1811 1f5c12ca-751b-0410-a591-d2e778427230
 2008-06-03 08:56:37 +00:00
nicolabertoldi
af585bc492 nbest can be read from stdin, too 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1797 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-27 17:20:01 +00:00
nicolabertoldi
291260abf7 - made output more compliant with old version 
- added PerSCorer.h and BleuScorer.h
- stored feature names
- fixed bug about output of best Point


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1796 1f5c12ca-751b-0410-a591-d2e778427230
 2008-05-27 16:50:52 +00:00