moses-smt/mosesdecoder
1
1
Fork 0
mirror of https://github.com/moses-smt/mosesdecoder.git synced 2025-01-01 08:21:47 +03:00
Code Issues Packages Projects Releases Wiki Activity
2,452 Commits 136 Branches 26 Tags 147 MiB
6b153c67f8
Commit Graph

1175 Commits

Author SHA1 Message Date
heafield
6b153c67f8 (16:51:52) Heafield: Does anybody use LanguageModelSkip? 
(16:52:12) Hieu Hoang: not since jhu 2006
(16:52:17) Heafield: svn rm?  
(16:52:34) Hieu Hoang: aye. & see if anyone complains
(16:52:49) Hieu Hoang: & internal if u want to



git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4352 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-13 16:01:00 +00:00
heafield
6bded791e6 Remove some virtual tags 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4351 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-13 15:34:37 +00:00
heafield
07e611ebcb Organize language models into an LM directory. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4350 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-13 14:27:01 +00:00
heafield
a95e791056 Back to using StringPiece 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4349 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-13 13:32:14 +00:00
heafield
f084248405 Cut the middle men out of the language model interface. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4348 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-13 12:33:05 +00:00
heafield
7d9bc523a6 Remove unused code 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4347 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-13 09:44:51 +00:00
heafield
541f776198 Remove unused calls 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4346 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-12 20:04:02 +00:00
heafield
e5d15a537e KenLM-specific Evaluate function 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4345 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-12 19:49:27 +00:00
heafield
cd19f14826 Faster CalcScore implementation for KenLM 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4339 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-12 13:04:12 +00:00
heafield
81acd0ffa2 Dear Hieu, a StringPiece is not necessairly null-terminated. When loading ARPA files directly, it was copying the ARPA file as 
part of the vocabulary word and breaking everything.  



git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4338 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-12 11:45:46 +00:00
heafield
c3f2ef7b25 Fix bhaddow's oovCount. Should be all words, not just the first in the phrase. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4337 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-12 10:22:45 +00:00
heafield
15adb17e35 Move EnumerateVocab to namespace lm 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4335 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-12 10:18:23 +00:00
hieuhoang1972
a65efa5a60 relax overly harsh assert 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4334 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-12 10:12:49 +00:00
heafield
ba41862d37 Source files are not executables. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4330 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-11 16:07:16 +00:00
heafield
16e37adbe0 Move phrase scoring from LanguageModel to LanguageModelImplementation. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4324 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-11 13:50:44 +00:00
hieuhoang1972
b0e5d6c005 delete align info flag in target phrase. Not used 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4322 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-11 08:55:19 +00:00
heafield
8f0c841d28 Move ChartHypothesis stuff to LanguageModelImplementation. Ran the 
regression tests. . . the passes and fails are the same FWIW.  



git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4319 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-10 16:25:56 +00:00
heafield
5a0d84da9a Move LanguageModelChartState into LanguageModelImplementation in preparation for moving responsibility for boundary word tracking 
from ChartHypothesis to LanguageModelChartState.  



git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4316 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-10 11:15:13 +00:00
hieuhoang1972
96c8ff4b15 last checkin was for the oldest bug found in moses! Goes back to svn version 4, and prob earlier, but svn can't diff that far. 
Should have crapped out whenever there is a blank line in the ini file, which is basically every ini file. Only visual studio 2010 complained, and only recently. Very strange, and a bit worrying. Horaay anyway

git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4304 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-06 15:06:59 +00:00
heafield
9ba5460e53 Apparently we wanted a sequential id after all. . . get one in a thread-safe way from the manager. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4302 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-06 10:31:09 +00:00
hieuhoang1972
6f22c2ae29 bug reading over string size 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4301 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-06 09:16:21 +00:00
bhaddow
84d73700af Implementation of Lattice sampling (Chatterjee and Cancedda, emnlp 2010) 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4295 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-04 15:46:24 +00:00
nicolabertoldi
47e452a076 made LM interface compliant with IRSTLM 5.70.02; fixed a bug related to word encoding 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4292 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-03 16:02:02 +00:00
hieuhoang1972
1ea3acde3d compile error due to last commit 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4291 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-03 08:31:05 +00:00
hieuhoang1972
3dafb3589c visual studio 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4290 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-03 07:18:14 +00:00
nicolabertoldi
2838970fc0 changed the interface towards IRSTLM according to the recent changes in FactorCollection 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4280 1f5c12ca-751b-0410-a591-d2e778427230
 2011-09-28 18:21:55 +00:00
hieuhoang1972
7538f5406a xcode 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4278 1f5c12ca-751b-0410-a591-d2e778427230
 2011-09-28 00:30:18 +00:00
pjwilliams
ea272dc198 Move SCFG rule table loading code out of PhraseDictionarySCFG and into a 
separate RuleTableLoader class.  Start adding support for a faster-loading
rule table format.

git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4273 1f5c12ca-751b-0410-a591-d2e778427230
 2011-09-27 00:34:46 +00:00
hieuhoang1972
9762a65bd8 xcode 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4264 1f5c12ca-751b-0410-a591-d2e778427230
 2011-09-25 10:44:47 +00:00
heafield
8edf53dcf3 StringPiece. Does a body good. And avoids string copying. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4263 1f5c12ca-751b-0410-a591-d2e778427230
 2011-09-24 15:58:23 +00:00
heafield
d40ba48da3 Replace hypothesis ids with pointers to ChartHypothesis. This avoids locking. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4261 1f5c12ca-751b-0410-a591-d2e778427230
 2011-09-24 14:48:52 +00:00
pjwilliams
efddd0f390 Remove race condition from ChartHypothesis ID allocation in moses_chart. 
Thanks to Kenneth Heafield for pointing that out.

git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4259 1f5c12ca-751b-0410-a591-d2e778427230
 2011-09-24 10:06:21 +00:00
heafield
19879a8b1e -threads all option 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4252 1f5c12ca-751b-0410-a591-d2e778427230
 2011-09-22 22:29:56 +00:00
heafield
ddbfee788c Sorry about the compiler error. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4251 1f5c12ca-751b-0410-a591-d2e778427230
 2011-09-22 08:55:54 +00:00
hieuhoang1972
d2245390e0 visual studio build 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4250 1f5c12ca-751b-0410-a591-d2e778427230
 2011-09-22 05:39:32 +00:00
heafield
daf0e37175 Barry wanted to insure Factor is non-copyable. But STL containers want it to be copyable. Here's some friend magic to meet both demands. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4249 1f5c12ca-751b-0410-a591-d2e778427230
 2011-09-21 20:35:12 +00:00
heafield
3402bdfe7a Merge mtm_lm into trunk. 
There's a fair number of files with no change that somebody must have touched in the branch so metadata is being recorded. 
Updates kenlm binary file format, sorry. 
It looks like OOV isn't being computed in EvaluateChart anyway, just phrasal.  
  


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4247 1f5c12ca-751b-0410-a591-d2e778427230
 2011-09-21 16:06:48 +00:00
hieuhoang1972
659e34735d xcode 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4245 1f5c12ca-751b-0410-a591-d2e778427230
 2011-09-21 11:01:19 +00:00
heafield
fdc6c37ca4 Store the string in the factor itself. Use MurmurHash because it beats Boost's hash. Also, apparently nobody calls Exists. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4244 1f5c12ca-751b-0410-a591-d2e778427230
 2011-09-21 10:26:04 +00:00
heafield
4c8552b16f Cruft removal 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4243 1f5c12ca-751b-0410-a591-d2e778427230
 2011-09-20 19:29:24 +00:00
heafield
b9e433977d Redo FactorCollection including better locking 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4242 1f5c12ca-751b-0410-a591-d2e778427230
 2011-09-20 19:08:42 +00:00
heafield
ed30a602af Remove deprecated, and uncalled LoadVocab function 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4241 1f5c12ca-751b-0410-a591-d2e778427230
 2011-09-20 17:12:27 +00:00
nicolabertoldi
c2de897cd2 added support for a new class-based lm type provided by IRSTLM 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4240 1f5c12ca-751b-0410-a591-d2e778427230
 2011-09-20 14:46:50 +00:00
phkoehn
7927439ce5 option to get unpruned search graph for chart decoder 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4228 1f5c12ca-751b-0410-a591-d2e778427230
 2011-09-16 11:58:53 +00:00
leven101
3824ca3ea2 fixed operator [] adding bug 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4216 1f5c12ca-751b-0410-a591-d2e778427230
 2011-09-12 18:04:56 +00:00
bhaddow
e95c004145 Implement LM OOV feature 
- still have to run regression
 - doesn't work for moses_chart (awaiting H & K LM state merge)


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4211 1f5c12ca-751b-0410-a591-d2e778427230
 2011-09-09 18:03:00 +00:00
bhaddow
ca5c0f19b7 Multi-threading of mert, for random restarts. 
Fix mert tests.


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4182 1f5c12ca-751b-0410-a591-d2e778427230
 2011-09-07 08:08:35 +00:00
heafield
c4d6471c3c Print exceptions because some peopls' machines don't. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4175 1f5c12ca-751b-0410-a591-d2e778427230
 2011-09-06 09:47:08 +00:00
leven101
762c47d8c9 added checks for loopy data in dynamic suffix array 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4169 1f5c12ca-751b-0410-a591-d2e778427230
 2011-09-01 09:57:16 +00:00
nicolabertoldi
75edc2eddd change to print the corrett name of the features with InputScores 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4168 1f5c12ca-751b-0410-a591-d2e778427230
 2011-08-30 12:25:50 +00:00