moses-smt/mosesdecoder
1
1
Fork 0
mirror of https://github.com/moses-smt/mosesdecoder.git synced 2025-01-03 17:33:52 +03:00
Code Issues Packages Projects Releases Wiki Activity
2,560 Commits 136 Branches 26 Tags 147 MiB
1ec58e7edf
Commit Graph

1210 Commits

Author SHA1 Message Date
Phil Williams
9903a239ea Assume HAVE_BOOST 2011-11-08 15:43:58 +00:00
Phil Williams
aa46d2eca0 moses_chart: speed up n-best list generation by deferring creation of 
ChartTrellisPath objects until a detour is selected.  The output
should be unchanged except in the case of ties and rounding differences
in score calculations.

This doesn't make much difference at n = 100 but helps for larger lists:
example real times for decoding the first 100 sentences of the
new-test2008 tuning set with four threads:

  n            before       after
  1         4m32.955s   4m28.584s
  100       4m42.375s   4m36.311s
  1500     13m17.681s   4m34.807s

And with the 'distinct' option:

               before       after
  1         4m36.656s   4m32.883s
  100      11m04.236s   4m35.221s
  1500    129m21.593s   5m06.320s
 2011-11-08 11:28:02 +00:00
Hieu Hoang
ae5e39fa86 delete premature lattice for scfg 2011-11-07 20:45:05 +07:00
Hieu Hoang
e0016712be Merge github.com:hieuhoang/mosesdecoder 2011-11-07 20:23:52 +07:00
Hieu Hoang
dbba840e10 Merge github.com:moses-smt/mosesdecoder 2011-11-07 20:18:47 +07:00
Hieu Hoang
0548a0ec3c load hiero format for reading suffix arrays 2011-11-06 20:55:57 +07:00
Hieu Hoang
24f7065826 load hiero format for reading suffix arrays 2011-11-06 20:46:46 +07:00
Hieu Hoang
bb2385657e load hiero format for reading suffix arrays 2011-11-06 16:08:37 +07:00
Hieu Hoang
86b9bf8bed load hiero format for reading suffix arrays 2011-11-06 15:35:05 +07:00
Hieu Hoang
311b5a6244 load hiero format for reading suffix arrays 2011-11-06 15:31:11 +07:00
Hieu Hoang
a257803610 load hiero format for reading suffix arrays 2011-11-05 01:06:31 +07:00
Hieu Hoang
1afbc7a5ee load hiero format for reading suffix arrays 2011-11-05 00:20:12 +07:00
Hieu Hoang
9b9ca34302 load hiero format for reading suffix arrays 2011-11-04 22:43:42 +07:00
Kenneth Heafield
1c72f79698 More informative error message about <s> errors 2011-11-01 10:24:40 +00:00
Kenneth Heafield
314313ad89 Report null context estimates to correct score breakdown 2011-10-30 17:56:42 +00:00
Hieu Hoang
df949b87de lattice for scfg decoding 2011-10-30 19:34:36 +07:00
Hieu Hoang
b6b697c545 xcode 2011-10-30 19:17:15 +07:00
Hieu Hoang
aac12ed58f xcode 2011-10-30 12:51:08 +07:00
Hieu Hoang
42924144fd Merge branch 'master' of github.com:moses-smt/mosesdecoder 2011-10-28 19:13:14 +07:00
Hieu Hoang
62c901eb52 TMX extraction by Tom Hoar and Hilario Leal Fontes 2011-10-28 19:12:20 +07:00
Germán
899293243a Modified code in Manager.cpp so that option -osgx outputs a superset of -osg. 2011-10-27 13:29:46 +02:00
Hieu Hoang
d98780c062 Just get ready but didn't fix bug yet 2011-10-24 19:13:56 +07:00
Hieu Hoang
ae3ecbc105 fix bug for tree-to-string. Didn't check source LHS 2011-10-24 19:13:17 +07:00
Hieu Hoang
82e2e094ff fix bug for tree-to-string. Didn't check sourceLHS 2011-10-24 18:54:42 +07:00
Hieu Hoang
cb87f251b2 xcode 2011-10-24 14:44:36 +07:00
Hieu Hoang
f8b6387642 xcode 2011-10-24 01:54:25 +07:00
Hieu Hoang
a93f4691f6 win32 2011-10-23 09:37:47 +07:00
Hieu Hoang
1120754a9b xcode compiles 2011-10-18 19:45:52 +07:00
Hieu Hoang
f2814c65b9 xcode. kenlm fails for some reason 2011-10-18 19:28:50 +07:00
Kenneth Heafield
8ec57a4c32 Now PhraseDictionaryMemory loads in 36% of the original time. 2011-10-17 12:38:53 +01:00
heafield
f0be9d9cf0 More Boost is allowed partying 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4381 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-17 09:40:46 +00:00
heafield
68a4626a49 Remove reference counts now that we can use boost 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4380 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-17 09:30:30 +00:00
heafield
2bb2d6dc4a Reduce text phrase table loading time by 49.5%. Add a progress bar too. StringPiece is good for you. 
This change introduces a dependency on Boost, which is now permitted in Moses.  



git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4365 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-14 16:40:30 +00:00
hieuhoang1972
897fe0f88b visual studio 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4356 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-14 10:50:08 +00:00
heafield
7ead82ba41 Remove extraneous header 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4353 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-13 16:22:04 +00:00
heafield
6b153c67f8 (16:51:52) Heafield: Does anybody use LanguageModelSkip? 
(16:52:12) Hieu Hoang: not since jhu 2006
(16:52:17) Heafield: svn rm?  
(16:52:34) Hieu Hoang: aye. & see if anyone complains
(16:52:49) Hieu Hoang: & internal if u want to



git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4352 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-13 16:01:00 +00:00
heafield
6bded791e6 Remove some virtual tags 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4351 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-13 15:34:37 +00:00
heafield
07e611ebcb Organize language models into an LM directory. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4350 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-13 14:27:01 +00:00
heafield
a95e791056 Back to using StringPiece 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4349 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-13 13:32:14 +00:00
heafield
f084248405 Cut the middle men out of the language model interface. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4348 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-13 12:33:05 +00:00
heafield
7d9bc523a6 Remove unused code 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4347 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-13 09:44:51 +00:00
heafield
541f776198 Remove unused calls 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4346 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-12 20:04:02 +00:00
heafield
e5d15a537e KenLM-specific Evaluate function 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4345 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-12 19:49:27 +00:00
heafield
cd19f14826 Faster CalcScore implementation for KenLM 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4339 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-12 13:04:12 +00:00
heafield
81acd0ffa2 Dear Hieu, a StringPiece is not necessairly null-terminated. When loading ARPA files directly, it was copying the ARPA file as 
part of the vocabulary word and breaking everything.  



git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4338 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-12 11:45:46 +00:00
heafield
c3f2ef7b25 Fix bhaddow's oovCount. Should be all words, not just the first in the phrase. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4337 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-12 10:22:45 +00:00
heafield
15adb17e35 Move EnumerateVocab to namespace lm 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4335 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-12 10:18:23 +00:00
hieuhoang1972
a65efa5a60 relax overly harsh assert 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4334 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-12 10:12:49 +00:00
heafield
ba41862d37 Source files are not executables. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4330 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-11 16:07:16 +00:00
heafield
16e37adbe0 Move phrase scoring from LanguageModel to LanguageModelImplementation. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4324 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-11 13:50:44 +00:00