moses-smt/mosesdecoder
1
1
Fork 0
mirror of https://github.com/moses-smt/mosesdecoder.git synced 2024-08-17 07:20:48 +03:00
Code Issues Packages Projects Releases Wiki Activity
9,498 Commits 136 Branches 26 Tags 147 MiB
GHK
Commit Graph

9498 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Kenneth Heafield
1530ae4f5f Fix state comparison (impacted 32-bit) 2013-01-09 13:15:04 +00:00
Barry Haddow
936dbf6516 Instance weighting 2013-01-08 16:40:00 +00:00
Barry Haddow
c86c11abbe instance weighting of lex weights 2013-01-08 15:34:29 +00:00
Barry Haddow
a55a936182 remove warning 2013-01-08 14:28:16 +00:00
Hieu Hoang
1b0f9f2e88 re-enable some score regressions tests. Was failing due to rounding errors 2013-01-08 13:54:35 +00:00
Hieu
5fe686888f Merge branch 'master' of https://github.com/moses-smt/mosesdecoder 2013-01-07 18:04:41 +00:00
Hieu
8e4f5a927d eclipse 2013-01-07 18:03:40 +00:00
Hieu
9973832624 eclipse 2013-01-07 18:03:32 +00:00
Hieu
86327b1e77 eclipse 2013-01-07 18:03:20 +00:00
Hieu
3bf83af8d1 eclipse 2013-01-07 18:03:11 +00:00
Hieu
18b162266c eclipse 2013-01-07 18:02:55 +00:00
Hieu
133e375e24 eclipse 2013-01-07 18:02:40 +00:00
Kenneth Heafield
366ca8afff Remove if that doesn't do anything, define _FILE_OFFSET_BITS 64 2013-01-07 17:38:04 +00:00
Hieu Hoang
506cb8e9d1 change lseek64() to lseek() and off64_t to off_t. Cygwin doesn't have those functions/types. They should be 64-bit if _FILE_OFFSET_BITS == 64 is set 2013-01-07 14:14:39 +00:00
Kenneth Heafield
ad209c8944 Collapse lseek macro 2013-01-04 22:42:32 +00:00
Hieu Hoang
77a3f32445 mac doesn't have lseek64() 2013-01-04 22:32:48 +00:00
Kenneth Heafield
f9ee7ae4b3 KenLM 0e5d259 including read_compressed fix 2013-01-04 21:02:47 +00:00
Kenneth Heafield
3203f7c92d If statically linking, check static libs for -mt 2013-01-04 21:02:13 +00:00
Hieu Hoang
c8448ded97 Merge branch 'master' of git://github.com/moses-smt/mosesdecoder 2013-01-04 20:00:23 +00:00
Hieu Hoang
43337dfde3 get rid of HAVE_BOOST macro. Always has boost now 2013-01-04 20:00:04 +00:00
Barry Haddow
459acf87b1 Add support for instance weights file 2013-01-04 14:55:24 +00:00
hieu
1dfbe1113c delete MergeScorer in mert/ 2013-01-03 15:01:30 +00:00
hieu
15c776eda4 delete check and exit of Suffix Array phrase table implementation erroneously checked in 2013-01-03 14:35:39 +00:00
Barry Haddow
861792bfc5 extract can read an instance weights file. 
Still have to parallelise.
 2012-12-21 15:39:25 +00:00
Barry Haddow
8fe900d312 Testing of phrase length feature 2012-12-21 12:35:11 +00:00
Tetsuo Kiso
ce1b650b53 Fix memory leak. 
The object was allocated with new, but it was not deleted.
This may not be a serious problem because the program mostly runs
a short time. However, it is not a good practice.
 2012-12-21 03:06:41 +09:00
Phil Williams
139148bc8f extract-ghkm and friends: don't unescape special characters 
Don't unescape special characters when reading XML parse trees in
extract-ghkm, extract-rules, and relax-parse.
 2012-12-17 20:08:02 +00:00
Phil Williams
0ca5b8932a extract-ghkm: tweak label collection for unknown words 
Produce a better label set when unary rule elimination is enabled.
 2012-12-17 19:43:42 +00:00
Phil Williams
fb8d20a22f extract-ghkm: --UnknownWordMinRelFreq, --UnknownWordUniform 2012-12-17 19:02:30 +00:00
Phil Williams
06081f7ddb extract-target-trees.py: minor fixes, code style 2012-12-17 18:49:50 +00:00
phikoehn
b275c94dbf allow for inclusion of extract from previous run 2012-12-12 07:02:59 +00:00
phikoehn
24e1df7520 support for use of baseline alignment model 2012-12-12 03:59:14 +00:00
phikoehn
438dcb1a34 bug fix in experiment.perl wrt. get-corpus-script 2012-12-10 23:50:14 +00:00
Barry Haddow
16ea68f55f Fix bug in mml scoring 
Line length calculation was out of step with LM scoring.
 2012-12-10 15:54:24 +00:00
Ales Tamchyna
598d65bcfd adding a simple command-line utility for computing sentence-level BLEU (+1) 2012-12-10 13:12:34 +01:00
phikoehn
ed2d191821 allow specification of end point for experiment.perl 2012-12-10 05:56:51 +00:00
phikoehn
ccf9e13d8e bug fix with multicore parallelizer 2012-12-09 22:27:02 +00:00
phikoehn
466b502ae0 minor bug fixes with MML 2012-12-09 20:31:20 +00:00
Tetsuo Kiso
2a3c9fc679 Further optimization for extractor. 
Fixes inefficient updating N-gram counts.

NOTE: Using '--binary' option (this option is not enabled by default yet)
for saving outputs would lead to significant speed up.
 2012-12-07 08:45:47 +09:00
Tetsuo Kiso
8fdec9bf30 Use boost::unordered_map instead of std::map. 
For storing the word vocabulary used in computation of
BLEU scores. This change will reduce the running time
of extractor about 2-3 seconds (9% reduction).
 2012-12-07 05:12:24 +09:00
Tetsuo Kiso
6c04c4ad9c Add more tests to the Data class. 2012-12-07 02:46:59 +09:00
Tetsuo Kiso
c7f6e38326 Use FilePiece to load N-best lists. 
Since FilePiece is friendly with StringPiece.
 2012-12-07 02:39:02 +09:00
Tetsuo Kiso
38e145e556 Use util::TokenIter to tokenize n-best lists. 
Reduce creating std::string objects, too. In both ScoreArray
and FeatureArray classes, the private members to track sentence
indices (namely, "m_index") were unnecessarily declared as
std::string, but it's better to directly declare them as 'int'.
 2012-12-07 01:39:22 +09:00
Tetsuo Kiso
cd3fb3b831 Untabify. 2012-12-06 23:46:22 +09:00
Tetsuo Kiso
ac045a11c1 Speed up N-gram counts when running extractor. 
By replacing std::map with boost::unordered_map.

Runtime of extractor on 100-best lists of 2679 sentences:

Before:
real    0m35.314s
user    0m34.030s
sys     0m1.280s

Ater:
real    0m26.729s
user    0m25.420s
sys     0m1.310s
 2012-12-06 22:08:33 +09:00
Hieu Hoang
55f65c3104 race condition in chart decoding with -T arg 2012-12-03 14:57:33 +00:00
phikoehn
ab2effb6fe train MML in-/out-of-domain language models with same vocabulary 2012-12-01 13:46:59 +00:00
phikoehn
269883fedd Merge branch 'master' of git://github.com/moses-smt/mosesdecoder 2012-12-01 13:45:00 +00:00
phikoehn
0c5d000192 my change to weight-wt 2012-12-01 13:44:57 +00:00
Marcin Junczys-Dowmunt
205cea8644 Allow .minlexr suffix and bugfix 2012-12-01 00:38:20 +01:00