moses-smt/mosesdecoder
1
1
Fork 0
mirror of https://github.com/moses-smt/mosesdecoder.git synced 2024-11-10 10:59:21 +03:00
Code Issues Packages Projects Releases Wiki Activity
2,538 Commits 136 Branches 26 Tags 147 MiB
864001b910
Commit Graph

2538 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Kenneth Heafield
81dbd6574e Reduce header pollution 2011-10-19 14:32:20 +01:00
Kenneth Heafield
bcc036c587 Minor code cleanup 2011-10-19 11:00:57 +01:00
Kenneth Heafield
7ead8ecaf9 Add more build products to .gitignore 2011-10-19 10:08:04 +01:00
Kenneth Heafield
f1a309773b Unbreak ltmain.sh 2011-10-19 10:05:48 +01:00
Hieu Hoang
1120754a9b xcode compiles 2011-10-18 19:45:52 +07:00
Hieu Hoang
01da665df5 xcode. kenlm fails for some reason 2011-10-18 19:31:08 +07:00
Hieu Hoang
f2814c65b9 xcode. kenlm fails for some reason 2011-10-18 19:28:50 +07:00
Kenneth Heafield
8ec57a4c32 Now PhraseDictionaryMemory loads in 36% of the original time. 2011-10-17 12:38:53 +01:00
bhaddow
abed9cec03 testing 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4383 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-17 10:17:51 +00:00
heafield
b13c341bc1 Remove some gcc-only variable sized arrays 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4382 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-17 09:42:36 +00:00
heafield
f0be9d9cf0 More Boost is allowed partying 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4381 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-17 09:40:46 +00:00
heafield
68a4626a49 Remove reference counts now that we can use boost 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4380 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-17 09:30:30 +00:00
bhaddow
27dd30122e boost on 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4369 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-15 09:46:43 +00:00
hieuhoang1972
9b00ae425c add regression test binary models with lex reordering. Fix other regression tests - changed to account for new LM prob from IRST or use KenLM 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4367 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-15 03:51:26 +00:00
hieuhoang1972
195cc1a2c1 add regression test binary models with lex reordering. Fix other regression tests - changed to account for new LM prob from IRST or use KenLM 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4366 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-15 03:50:00 +00:00
heafield
2bb2d6dc4a Reduce text phrase table loading time by 49.5%. Add a progress bar too. StringPiece is good for you. 
This change introduces a dependency on Boost, which is now permitted in Moses.  



git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4365 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-14 16:40:30 +00:00
bhaddow
7c0d9c34da Fix formatting of pro input file 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4360 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-14 14:01:15 +00:00
heafield
967b725d73 Make Chris Dyer feel safer about compile time if statement handling 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4358 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-14 11:52:05 +00:00
heafield
7b129fa461 Add a test and a multi-token breaker 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4357 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-14 11:51:15 +00:00
hieuhoang1972
897fe0f88b visual studio 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4356 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-14 10:50:08 +00:00
bhaddow
718c55d3aa Reducing memory usage for sparse features 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4355 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-14 07:40:53 +00:00
hieuhoang1972
57bf51fd05 all programs in training can take in gzipped file 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4354 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-13 18:57:23 +00:00
heafield
7ead82ba41 Remove extraneous header 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4353 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-13 16:22:04 +00:00
heafield
6b153c67f8 (16:51:52) Heafield: Does anybody use LanguageModelSkip? 
(16:52:12) Hieu Hoang: not since jhu 2006
(16:52:17) Heafield: svn rm?  
(16:52:34) Hieu Hoang: aye. & see if anyone complains
(16:52:49) Hieu Hoang: & internal if u want to



git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4352 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-13 16:01:00 +00:00
heafield
6bded791e6 Remove some virtual tags 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4351 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-13 15:34:37 +00:00
heafield
07e611ebcb Organize language models into an LM directory. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4350 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-13 14:27:01 +00:00
heafield
a95e791056 Back to using StringPiece 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4349 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-13 13:32:14 +00:00
heafield
f084248405 Cut the middle men out of the language model interface. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4348 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-13 12:33:05 +00:00
heafield
7d9bc523a6 Remove unused code 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4347 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-13 09:44:51 +00:00
heafield
541f776198 Remove unused calls 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4346 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-12 20:04:02 +00:00
heafield
e5d15a537e KenLM-specific Evaluate function 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4345 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-12 19:49:27 +00:00
heafield
cd19f14826 Faster CalcScore implementation for KenLM 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4339 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-12 13:04:12 +00:00
heafield
81acd0ffa2 Dear Hieu, a StringPiece is not necessairly null-terminated. When loading ARPA files directly, it was copying the ARPA file as 
part of the vocabulary word and breaking everything.  



git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4338 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-12 11:45:46 +00:00
heafield
c3f2ef7b25 Fix bhaddow's oovCount. Should be all words, not just the first in the phrase. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4337 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-12 10:22:45 +00:00
hieuhoang1972
b88fad16f8 create valid html header, according to Tomas Hudik 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4336 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-12 10:18:36 +00:00
heafield
15adb17e35 Move EnumerateVocab to namespace lm 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4335 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-12 10:18:23 +00:00
hieuhoang1972
a65efa5a60 relax overly harsh assert 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4334 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-12 10:12:49 +00:00
heafield
19f3f09a39 Updated left state minimization makes all states of length N-1 full 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4332 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-11 18:40:00 +00:00
heafield
86f1d3ec71 Fix trie for ARPAs from SRILM. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4331 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-11 18:27:36 +00:00
heafield
ba41862d37 Source files are not executables. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4330 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-11 16:07:16 +00:00
heafield
16e37adbe0 Move phrase scoring from LanguageModel to LanguageModelImplementation. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4324 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-11 13:50:44 +00:00
heafield
c9995dc44c Trie building bug fix 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4323 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-11 10:12:17 +00:00
hieuhoang1972
b0e5d6c005 delete align info flag in target phrase. Not used 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4322 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-11 08:55:19 +00:00
hieuhoang1972
ea4db80473 extract lex probability from gzip files 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4321 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-11 06:49:19 +00:00
heafield
8f0c841d28 Move ChartHypothesis stuff to LanguageModelImplementation. Ran the 
regression tests. . . the passes and fails are the same FWIW.  



git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4319 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-10 16:25:56 +00:00
heafield
5a0d84da9a Move LanguageModelChartState into LanguageModelImplementation in preparation for moving responsibility for boundary word tracking 
from ChartHypothesis to LanguageModelChartState.  



git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4316 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-10 11:15:13 +00:00
hieuhoang1972
235dda25e7 extractor wrapper to make it work on SGE 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4315 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-10 04:28:55 +00:00
heafield
71d0d389c5 Fix silly bug in merging 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4314 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-08 10:59:54 +00:00
hieuhoang1972
96c8ff4b15 last checkin was for the oldest bug found in moses! Goes back to svn version 4, and prob earlier, but svn can't diff that far. 
Should have crapped out whenever there is a blank line in the ini file, which is basically every ini file. Only visual studio 2010 complained, and only recently. Very strange, and a bit worrying. Horaay anyway

git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4304 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-06 15:06:59 +00:00
servan
eef0f213e9 A mert/MergeScorer.h 
A    mert/MergeScorer.cpp


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4303 1f5c12ca-751b-0410-a591-d2e778427230
 2011-10-06 11:00:48 +00:00