moses-smt/mosesdecoder
1
1
Fork 0
mirror of https://github.com/moses-smt/mosesdecoder.git synced 2024-12-30 23:42:30 +03:00
Code Issues Packages Projects Releases Wiki Activity
10,674 Commits 136 Branches 26 Tags 147 MiB
4fde5f7ea2
Commit Graph

577 Commits

Author SHA1 Message Date
Hieu Hoang
e3dd3a8d2c namespace all classes in mert directory 2012-06-30 20:23:45 +01:00
Colin Cherry
65df386581 Merge branch 'master' of https://github.com/moses-smt/mosesdecoder 2012-06-26 17:07:27 -04:00
Colin Cherry
58c3280c2c HypPackEnumerator now stores MiraFeatureVectors, as opposed to 
FeatureDataItems. Uses roughly half the memory.
 2012-06-26 17:02:32 -04:00
Hieu Hoang
3c7b7ac9f5 rollback 2012-06-26 16:31:38 -04:00
Colin Cherry
32299593fa Added debugging info to kbmira. 2012-06-26 16:29:20 -04:00
Hieu Hoang
153e80053c lock m_vocab variable access in Encode() and Lookup(). Other functions are still not threadsafe 2012-06-26 13:33:50 -04:00
Hieu Hoang
00f018a477 Merge https://github.com/moses-smt/mosesdecoder into lrscore 2012-06-25 16:57:17 -04:00
Hieu Hoang
2a03f275a3 change regression data download to git instead of download from edin server. 
Minor change in mert/trimStr() function to prevent warning
 2012-06-25 16:03:11 -04:00
Hieu Hoang
8498b17a41 gcc version-specific error 2012-06-25 14:45:45 +01:00
Hieu Hoang
0fd0adc1f6 merge Lexi Birch's LRScore from mert_mtm5 branch. Compiles and run. Hack, must double check with barry or lexi 2012-06-23 22:58:18 -04:00
Hieu Hoang
0cb63edcb9 merge Lexi Birch's LRScore from mert_mtm5 branch. Compiles and run. Hack, must double check with barry or lexi 2012-06-23 22:51:48 -04:00
Hieu Hoang
f48c348508 typo 2012-06-22 22:23:14 -04:00
Hieu Hoang
b1ca36387f mert now compiles with PermScorer. However, didn't implement score() - assert(false). Update Jamfile 2012-06-22 21:07:05 -04:00
Hieu Hoang
7d19fe13ae merge Lexi Birch's LRScore from mert_mtm5 branch 2012-06-22 18:19:16 +01:00
Colin Cherry
07a5c67ebc Merge branch 'master' into miramerge 
Conflicts:
	Jamroot
	misc/queryPhraseTable.cpp
	scripts/training/train-model.perl
 2012-06-14 17:08:16 -04:00
Colin Cherry
5932800489 Spurious space disagreed with master 2012-06-14 14:15:06 -04:00
Colin Cherry
a8a5f896db Fixed some bugs in BatchMira's sparse feature handling. 2012-06-14 14:09:06 -04:00
Colin Cherry
a901fc9f50 Fixed some bugs in BatchMira's sparse feature handling. 2012-06-14 13:41:47 -04:00
Tetsuo Kiso
1dbd8e5ec5 Merge branch 'master' of github.com:moses-smt/mosesdecoder 2012-06-09 19:33:31 +09:00
Tetsuo Kiso
2599ef6dc3 Bug fix: kbmira failed to load dense weights. 2012-06-09 18:03:12 +09:00
Eva Hasler
e1c1a5343c merge 2012-06-07 11:16:52 +01:00
Eva Hasler
6a6a35c65e fix start weights in experiment.perl, add hypothesis queue for picking hope and fear translations, add variations to 1slack formulation 2012-06-01 01:49:42 +01:00
Tetsuo Kiso
713ff8c5e2 Delete mert/init.opt. 
It looks like the file was no longer used.
 2012-06-01 02:25:25 +09:00
Hieu Hoang
465c5cbf97 move all executables into bin 2012-05-31 12:55:05 +01:00
Colin Cherry
3c44d04baf Merge branch 'master' into miramerge 
Conflicts:
	Jamroot
	mert/FeatureStats.cpp
	moses-cmd/src/IOWrapper.h
	scripts/training/mert-moses.pl
	scripts/training/train-model.perl.missing_bin_dir
 2012-05-30 12:39:53 -04:00
Tetsuo Kiso
beb2256dba Move 'using namespace std' out from .h. 
Add "std" to size_t, too.
 2012-05-30 23:11:09 +09:00
Tetsuo Kiso
01eb60f350 Add "virtual" destructor to the HypPackEnumerator class. 2012-05-30 22:59:23 +09:00
Hieu Hoang
d25805858d xcode build supports threads. move 'using namespace' out from .h file to stop namespace pollution 2012-05-30 13:04:02 +01:00
Hieu Hoang
45870348ff xcode build supports threads. move 'using namespace' out from .h file to stop namespace pollution 2012-05-30 12:47:20 +01:00
Colin Cherry
fd577d7a65 Batch k-best MIRA is written and integrated into mert-moses.pl 
Regression tests all check out, and kbmira seems to work fine
on a Hansard French->English task.

HypPackEnumerator class may be of interest to pro.cpp and future
optimizers, as it abstracts a lot of the boilerplate involved in
enumerating multiple k-best lists.

MiraWeightVector is not really mira-specific - just a weight vector
that enables efficient averaging. Could be useful to a perceptron
as well. Same goes for MiraFeatureVector.

Interaction with sparse features is written, but untested.
 2012-05-29 13:38:57 -04:00
Barry Haddow
c397d2068b Merge branch 'trunk' into miramerge. Still to fix build. 
Conflicts:
	Jamroot
	mert/Data.cpp
	mert/Data.h
	mert/FeatureArray.cpp
	mert/FeatureArray.h
	mert/FeatureData.cpp
	mert/FeatureData.h
	mert/FeatureStats.cpp
	mert/FeatureStats.h
	mert/mert.cpp
	moses-chart-cmd/src/IOWrapper.h
	moses-chart-cmd/src/Main.cpp
	moses-cmd/src/IOWrapper.cpp
	moses-cmd/src/IOWrapper.h
	moses-cmd/src/Main.cpp
	moses/src/GlobalLexicalModel.cpp
	moses/src/Jamfile
	moses/src/Parameter.cpp
	moses/src/PhraseDictionary.cpp
	moses/src/ScoreIndexManager.h
	moses/src/TargetPhrase.h
	regression-testing/tests/phrase.lexicalized-reordering-bin/truth/results.txt
	regression-testing/tests/phrase.lexicalized-reordering-cn/truth/results.txt
	regression-testing/tests/phrase.lexicalized-reordering/truth/results.txt
	regression-testing/tests/phrase.multiple-translation-system-lr/truth/results.txt
	regression-testing/tests/phrase.show-weights.lex-reorder/truth/results.txt
	regression-testing/tests/phrase.show-weights/truth/results.txt
	scripts/ems/experiment.meta
	scripts/ems/experiment.perl
	scripts/training/filter-model-given-input.pl
	scripts/training/mert-moses.pl
 2012-05-24 21:11:35 +01:00
Matous Machacek
a77cca4f86 Fixed CderScorer name bug 2012-05-15 00:35:08 +02:00
Matous Machacek
7da028e240 Fixed CderScorer name bug 2012-05-15 00:35:08 +02:00
Matous Machacek
3943112eb3 Fixed bug in SemposScorer.cpp 2012-05-13 11:11:13 +02:00
Matous Machacek
7a0c42b1bb Fixed bug in SemposScorer.cpp 2012-05-13 11:11:13 +02:00
Matous Machacek
97f82a3e4d Fixed interpolated scorer 2012-05-12 16:11:33 +02:00
Matous Machacek
8343a469e0 Fixed interpolated scorer 2012-05-12 16:11:33 +02:00
Eva Hasler
30deedde9f changed permission, everything changed.. 2012-05-10 18:54:24 +01:00
Christian Federmann
c360b5a329 Merge remote-tracking branch 'upstream/master' 2012-05-10 09:49:57 +02:00
Christian Federmann
60dde0b06b Merge remote-tracking branch 'upstream/master' 2012-05-10 09:49:57 +02:00
Tetsuo Kiso
9c9d88a78a Avoid "using namespace std" in headers. 2012-05-10 07:51:05 +09:00
Tetsuo Kiso
7bd6243c7f Avoid "using namespace std" in headers. 2012-05-10 07:51:05 +09:00
Tetsuo Kiso
aa269bdb4d Merge branch 'master' of github.com:moses-smt/mosesdecoder 2012-05-10 07:31:47 +09:00
Tetsuo Kiso
96e08d0ac1 Merge branch 'master' of github.com:moses-smt/mosesdecoder 2012-05-10 07:31:47 +09:00
Tetsuo Kiso
5f7967402a Reduce compilation dependencies. 2012-05-10 07:16:38 +09:00
Tetsuo Kiso
18bd11aca6 Reduce compilation dependencies. 2012-05-10 07:16:38 +09:00
Matous Machacek
31edaec4c2 Bigger buffer 2012-05-10 00:08:54 +02:00
Matous Machacek
a7040c2a5d Bigger buffer 2012-05-10 00:08:54 +02:00
Tetsuo Kiso
afa356aec4 Small changes to just improve the quality. 
- Use forward declaration to reduce dependencies.
- Add "virtual" to the destructor of _fdstream class.
- Avoid using namespace std in header.
  - We have already used in mert a lot, though. It should be fixed.
- Fix warnings "-Wreorder".
- Fix the usage of enum.
 2012-05-10 06:57:44 +09:00
Tetsuo Kiso
60df4ba931 Small changes to just improve the quality. 
- Use forward declaration to reduce dependencies.
- Add "virtual" to the destructor of _fdstream class.
- Avoid using namespace std in header.
  - We have already used in mert a lot, though. It should be fixed.
- Fix warnings "-Wreorder".
- Fix the usage of enum.
 2012-05-10 06:57:44 +09:00
Christian Federmann
c1cf0faf14 GzFileBuf.h got lost in the last merge ;) 2012-05-09 21:52:59 +02:00
Christian Federmann
0628e5c54c GzFileBuf.h got lost in the last merge ;) 2012-05-09 21:52:59 +02:00
Christian Federmann
25f43d13b8 Merged in upstream. 2012-05-09 21:10:52 +02:00
Christian Federmann
f9d747ed5e Merged in upstream. 2012-05-09 21:10:52 +02:00
Matous Machacek
440650bd6e Added support for external unix filters to preprocess sentences in mert and evaluator 2012-05-09 19:21:41 +02:00
Matous Machacek
f48a6665af Added support for external unix filters to preprocess sentences in mert and evaluator 2012-05-09 19:21:41 +02:00
Tetsuo Kiso
dbfe766f2c Fix using directive refers to implicitly-defined namespace 'std'. 2012-05-06 05:27:04 +09:00
Tetsuo Kiso
df4586740d Fix using directive refers to implicitly-defined namespace 'std'. 2012-05-06 05:27:04 +09:00
Tetsuo Kiso
b6f55f6bec Fix using namespace "std" before including headers in *.h. 2012-05-06 01:31:04 +09:00
Tetsuo Kiso
4b5fe180ed Fix using namespace "std" before including headers in *.h. 2012-05-06 01:31:04 +09:00
Tetsuo Kiso
1b634488fe Prefix a private member of mert/CderScorer with "m_". 2012-05-03 05:38:35 +09:00
Tetsuo Kiso
fc2f4d4ba1 Prefix a private member of mert/CderScorer with "m_". 2012-05-03 05:38:35 +09:00
Matous Machacek
933f25b3b2 CderScorer compute both CDER and WER metric 2012-05-02 22:03:58 +02:00
Matous Machacek
370bf4e697 CderScorer compute both CDER and WER metric 2012-05-02 22:03:58 +02:00
Eva
6c2a58a48e clean up mira, add sampling from hope/model/fear 2012-04-29 21:29:18 -07:00
Eva
6f39ad0b3e test 2012-04-28 23:11:30 -07:00
Tetsuo Kiso
7d457f5049 Add TODO to mert/Reference.h. 2012-04-26 05:44:05 +09:00
Tetsuo Kiso
66992ed497 Add TODO to mert/Reference.h. 2012-04-26 05:44:05 +09:00
Tetsuo Kiso
91a6725081 Add a test case for setting up BLEU's reference length. 
To make sure the type of effective reference length is
set to the specified arguments.
 2012-04-22 20:11:05 +09:00
Tetsuo Kiso
cee8b2e357 Add a test case for setting up BLEU's reference length. 
To make sure the type of effective reference length is
set to the specified arguments.
 2012-04-22 20:11:05 +09:00
Tetsuo Kiso
fe79b96328 Use std::stringstream instead of using snprintf() for Windows. 
This commit fixes compilation problems related to
snprintf() for Windows users.

Thanks to Raka Prasetya for reporting the errors.
Thanks also to Kenneth Heafield and Barry Haddow for suggestions.
 2012-04-18 23:47:48 +09:00
Tetsuo Kiso
bd79fc2c13 Use std::stringstream instead of using snprintf() for Windows. 
This commit fixes compilation problems related to
snprintf() for Windows users.

Thanks to Raka Prasetya for reporting the errors.
Thanks also to Kenneth Heafield and Barry Haddow for suggestions.
 2012-04-18 23:47:48 +09:00
Tetsuo Kiso
408dd72f9d Pass by reference instead of returning objects. 
In the previous implementation, copy-constructor got called, which
was not necessary. This commit prevents it, making code more efficient.
 2012-04-12 09:40:52 +09:00
Tetsuo Kiso
b25e9c2915 Pass by reference instead of returning objects. 
In the previous implementation, copy-constructor got called, which
was not necessary. This commit prevents it, making code more efficient.
 2012-04-12 09:40:52 +09:00
Tetsuo Kiso
0e9b5fd9d0 Add const to return values of overloaded operators. 
* This commit prevents developers from doing mistakes like:

  Point p1, p2, p4;
  if (p1 + p2 = p4) { // Bang! We actually wanted to compare
                      // the result of two points: (p1 + p2 == p4).
    // do something.
  }

See, e.g., http://www.gotw.ca/gotw/006.htm for details.

* Add more test cases according to this change.
* Move a helper function to compare floating point numbers
  to Util.h.
 2012-04-12 09:16:27 +09:00
Tetsuo Kiso
48c1d712b8 Add const to return values of overloaded operators. 
* This commit prevents developers from doing mistakes like:

  Point p1, p2, p4;
  if (p1 + p2 = p4) { // Bang! We actually wanted to compare
                      // the result of two points: (p1 + p2 == p4).
    // do something.
  }

See, e.g., http://www.gotw.ca/gotw/006.htm for details.

* Add more test cases according to this change.
* Move a helper function to compare floating point numbers
  to Util.h.
 2012-04-12 09:16:27 +09:00
Tetsuo Kiso
49a4be2899 Reduce the number of calling the numeric_limits constructor. 
- Create a test module for Point.
- Add test cases.
 2012-04-12 05:19:11 +09:00
Tetsuo Kiso
70800c2012 Reduce the number of calling the numeric_limits constructor. 
- Create a test module for Point.
- Add test cases.
 2012-04-12 05:19:11 +09:00
Tetsuo Kiso
65dc5a391f Update mert/TODO 2012-04-08 22:47:20 +09:00
Tetsuo Kiso
fa98111b6f Update mert/TODO 2012-04-08 22:47:20 +09:00
Tetsuo Kiso
4c38bbb434 Add things to TODO. 2012-04-08 22:45:04 +09:00
Tetsuo Kiso
1b8a72a093 Add things to TODO. 2012-04-08 22:45:04 +09:00
Tetsuo Kiso
d034eeb703 Add test cases for BLEU and sentence-level BLEU+1. 
- Move a definition of sentenceLevelBleuPlusOne() from pro.cpp
  to BleuScorer.cpp.
- Add check for the length of an input vector.
 2012-04-07 01:02:32 +09:00
Tetsuo Kiso
87ec82d853 Add test cases for BLEU and sentence-level BLEU+1. 
- Move a definition of sentenceLevelBleuPlusOne() from pro.cpp
  to BleuScorer.cpp.
- Add check for the length of an input vector.
 2012-04-07 01:02:32 +09:00
Tetsuo Kiso
bcc1958d94 Fix a warning; remove duplicate function calls. 2012-04-05 12:49:49 +09:00
Tetsuo Kiso
a0e787384f Fix a warning; remove duplicate function calls. 2012-04-05 12:49:49 +09:00
Tetsuo Kiso
fecee1c4fc Fix a grammar error; add more test case. 
- Fix a warning, too.
 2012-04-05 10:10:16 +09:00
Tetsuo Kiso
2bfeb39ba6 Fix a grammar error; add more test case. 
- Fix a warning, too.
 2012-04-05 10:10:16 +09:00
Tetsuo Kiso
e2a92c0f91 Use EndsWith(). 2012-04-05 00:03:13 +09:00
Tetsuo Kiso
8a2495c966 Use EndsWith(). 2012-04-05 00:03:13 +09:00
Tetsuo Kiso
20b99ed9e9 Add a test case for EndsWith(). 2012-04-04 22:43:51 +09:00
Tetsuo Kiso
8c5e69bb70 Add a test case for EndsWith(). 2012-04-04 22:43:51 +09:00
Tetsuo Kiso
a6cec240c0 Fix typo. 2012-04-04 22:36:22 +09:00
Tetsuo Kiso
685f4081b2 Fix typo. 2012-04-04 22:36:22 +09:00
Tetsuo Kiso
eaa0ab486a Add a test case for BLEU's clipped counts. 
- Make BleuScorer::setReferenceFiles() more testable by
  adding OpenReference() and OpenReferenceStream().
 2012-04-04 22:33:30 +09:00
Tetsuo Kiso
9d10b46ca5 Add a test case for BLEU's clipped counts. 
- Make BleuScorer::setReferenceFiles() more testable by
  adding OpenReference() and OpenReferenceStream().
 2012-04-04 22:33:30 +09:00
Tetsuo Kiso
6c69a97cbb Fix a compilation error. 2012-04-04 22:11:43 +09:00
Tetsuo Kiso
2ca4513e29 Fix a compilation error. 2012-04-04 22:11:43 +09:00
Tetsuo Kiso
27515f5de1 Add a function to check whether a string ends with a suffix. 
- Use the function in Data::InitFeatureMap().
- Add an unit test for InitFeatureMap().
- Move helper functions for Data::loadnbest() to public for unit testing.
 2012-04-04 22:04:51 +09:00
Tetsuo Kiso
1ade69a546 Add a function to check whether a string ends with a suffix. 
- Use the function in Data::InitFeatureMap().
- Add an unit test for InitFeatureMap().
- Move helper functions for Data::loadnbest() to public for unit testing.
 2012-04-04 22:04:51 +09:00