moses-smt/mosesdecoder
1
1
Fork 0
mirror of https://github.com/moses-smt/mosesdecoder.git synced 2024-10-06 08:18:02 +03:00
Code Issues Packages Projects Releases Wiki Activity
6,522 Commits 136 Branches 26 Tags 147 MiB
5b7c5ebdb5
Commit Graph

275 Commits

Author SHA1 Message Date
hieuhoang1972
e247f1da6f fixed regression test failing. Number of features for generation models MUST be specified in ini file, no backward compatability hack 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1209 1f5c12ca-751b-0410-a591-d2e778427230
 2007-02-13 19:15:34 +00:00
phkoehn
de9a5e96dd look for gziped generation file, if basefile does not exist, 
this should be done for all model files (lm, phrase table, reordering table)


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1183 1f5c12ca-751b-0410-a591-d2e778427230
 2007-02-07 19:02:42 +00:00
hieuhoang1972
46e01f9f46 unitiliased variable 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1148 1f5c12ca-751b-0410-a591-d2e778427230
 2007-01-24 20:54:37 +00:00
hieuhoang1972
b87b28d076 add n-best-factor 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@1135 1f5c12ca-751b-0410-a591-d2e778427230
 2007-01-22 12:39:28 +00:00
hieuhoang1972
7af5bb462e dismantle IODevoce framework 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@992 1f5c12ca-751b-0410-a591-d2e778427230
 2006-11-21 23:06:30 +00:00
hieuhoang1972
7e00b6ee8b fixed merge error 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@991 1f5c12ca-751b-0410-a591-d2e778427230
 2006-11-21 22:54:35 +00:00
hieuhoang1972
a1f39c3ce7 lots of small changes and code clean up: 
error catching/fail more gracefully on tables/lm load error & consistent user output
consistent debugging output
cleaned up timing functions
cleaned up mose/moses-cmd api calls/interaction
split up loading of all data in StaticData into separate fns
got binary phrase table to work under WIN32 & passed regression !!
added more comments
deleted phrase table filtering code
deleted mysql support
change calls to ToString() which might affect decoding to a call to a non-debugging fn instead, eg GetString()


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@988 1f5c12ca-751b-0410-a591-d2e778427230
 2006-11-21 19:35:37 +00:00
nicolabertoldi
8e06834ec1 Modified output of feature scores. 
Scores from Confusion Network input are output BEFORE any lexicon scores.
In nbest file, they are NOW labelled with "I:". Before they were "hidden" among other translation scores.
This change affect MERT, which will be changed ASAP.

git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@979 1f5c12ca-751b-0410-a591-d2e778427230
 2006-11-16 10:43:40 +00:00
phkoehn
28ca9b57fd minor bug fixes for training and using lexicalized reordering 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@978 1f5c12ca-751b-0410-a591-d2e778427230
 2006-11-15 17:04:19 +00:00
hieuhoang1972
8d97bce927 solved bug in n-best list creation 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@948 1f5c12ca-751b-0410-a591-d2e778427230
 2006-11-03 18:44:14 +00:00
hieuhoang1972
6e89d041bb added comments 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@909 1f5c12ca-751b-0410-a591-d2e778427230
 2006-10-21 21:10:46 +00:00
hieuhoang1972
6b1c0837b0 added comments 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@907 1f5c12ca-751b-0410-a591-d2e778427230
 2006-10-21 19:00:58 +00:00
hieuhoang1972
69d378e67d consistent variable names 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@898 1f5c12ca-751b-0410-a591-d2e778427230
 2006-10-17 23:36:43 +00:00
hieuhoang1972
d6f9458d59 code cleanup and commenting brought about when documenting for jhu report 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@886 1f5c12ca-751b-0410-a591-d2e778427230
 2006-10-17 11:07:17 +00:00
hieuhoang1972
605bbcfc54 getting rid of warnings 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@854 1f5c12ca-751b-0410-a591-d2e778427230
 2006-10-05 07:42:06 +00:00
phkoehn
3e7e4570b5 bug fix to lexicalized reordering model 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@834 1f5c12ca-751b-0410-a591-d2e778427230
 2006-09-28 18:36:38 +00:00
redpony
441b147b35 Get rid of FactorArrayWrapper/FactorArray and use only Word. Memory pool is currently disabled, but the net cleanup resulted in better performance despite this (there are fewer copies of FactorArray -> Word now). 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@827 1f5c12ca-751b-0410-a591-d2e778427230
 2006-09-25 21:35:10 +00:00
redpony
c69cfaf33e Allow the factor delimiter, that is, the string that separates the factors in a 'word' to be specified to moses and to train-factored-phrase-model.perl. The default is still to use '|'. Multi-character delimiters are allowed (for example, '+++'). Added a regression test for multi-character delimiters. 
Remove JHU dependencies on make release.  It now looks for GIZA++ and sets the BINDIR inside train-factored-phrase-model.perl at installation time (note: because of this, this script MUST BE released before it can be run now).



git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@812 1f5c12ca-751b-0410-a591-d2e778427230
 2006-09-12 15:53:50 +00:00
hieuhoang1972
e4c96d8209 rename files for consistency 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@810 1f5c12ca-751b-0410-a591-d2e778427230
 2006-09-09 17:01:00 +00:00
phkoehn
6d54aca44a minor fixes, boolean switches can now be applied without value, e.g. "-t" 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@803 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-30 19:52:16 +00:00
phkoehn
bafe7fc2ab cleanup of logging and reporting, now 4 verbose levels (0-3) 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@798 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-30 17:00:41 +00:00
redpony
3fb417f914 remove references to boost 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@777 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-19 20:09:43 +00:00
ccorbett
cdde0a6e11 final format of lexicalize reordering ini file 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@773 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-16 20:56:12 +00:00
ccorbett
69babe34bf bug fix for lexical reordering involving edge case 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@770 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-16 16:22:53 +00:00
ccorbett
0aaf45fd98 crucial bug fix for lexicalreordering, score reporting is done appropriately. change of format for lexicalreordering in the .ini file 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@765 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-15 23:59:25 +00:00
ccorbett
b4a6e05cbd distortion models now support specifying one weight per slot in the distortion table, or one weight for the entire model. useful if one wants to reduce the number of paramaters to mert. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@762 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-15 20:28:55 +00:00
ccorbett
afd9dbd0de can now use multiple distortion models at the same time, keep in mind that the .ini file has changed to accomodate this. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@755 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-15 16:23:43 +00:00
redpony
2fc0a82c92 enable language model stats computation. enable with '-L true' 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@739 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-14 22:51:56 +00:00
hieuhoang1972
b7a7b4286f code cleanup 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@736 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-14 22:37:24 +00:00
hieuhoang1972
2a7179a8fa choice of not labelling n-best output for wade shen's mert script 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@692 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-13 19:08:39 +00:00
zens
61294d90ad - generate distinct nbest surface strings (if 3rd token of n-best-list is 'distinct') 
- some sanity checks


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@674 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-12 00:06:55 +00:00
eherbst
5cb683cd7e moved class InputType to InputType.h; 
added more detailed statistics-keeping to SentenceStats;
added custom malloc/realloc (see Util.h);
some commenting


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@666 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-11 21:04:38 +00:00
mfederico
2e5c12e810 IRST LM refreshes caches for logprobs and LM state after each sentence. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@643 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-11 06:10:45 +00:00
redpony
0783a03f3a performance fixes for multi-factored models (basically, only copy the features that you really need, and only filter if you really might see a conflict) 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@641 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-11 06:03:11 +00:00
ccorbett
f6f7ee3dbb LexicalReordering model now functional with both surface forms and factors. Only one reordering model can be used at a time, for now. This will change so that an arbitrary number of reordering models may be used. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@638 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-11 02:12:53 +00:00
redpony
b4bdc45c17 compute the set of potential conflicting factors for every decode step 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@630 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-10 22:40:47 +00:00
hieuhoang1972
e88ba116b9 clean up creating language models 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@621 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-10 18:04:29 +00:00
hieuhoang1972
8fe3826a52 clean up creating language models 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@620 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-10 17:39:32 +00:00
phkoehn
525a181f12 re-organized parameter handling, now includes check for invalid parameters, reporting of parameter values, and a "help" function 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@616 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-10 04:20:06 +00:00
phkoehn
6cd263a591 eliminated compile time warning over types 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@598 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-09 20:06:31 +00:00
hieuhoang1972
b867c0e866 factored lm 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@591 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-09 03:18:37 +00:00
hieuhoang1972
e1a3565dd6 factored lm 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@590 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-09 02:41:24 +00:00
phkoehn
dfadc16043 added parameter that limits the number of partial translation options per span 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@577 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-08 21:32:33 +00:00
redpony
5d7a766aae enable optional logging of the details of the 1-best translation hypothesis, including dropped words, alignment, scores 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@549 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-07 23:43:48 +00:00
ccorbett
92fdc3e9f1 infrastructure now supports multiple reordering models; this functionality cannot be read from the configuration file yet, that's upnext. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@537 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-07 21:43:09 +00:00
ccorbett
4a88275261 lexical reordering is now functional, Forward, Backward and Bidirectional 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@532 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-07 18:18:26 +00:00
hieuhoang1972
f304d8ae29 factored lm 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@526 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-07 03:18:57 +00:00
hieuhoang1972
283c44a95a factored lm 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@525 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-07 02:48:16 +00:00
hieuhoang1972
ad5e23440b factored lm 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@524 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-07 02:01:59 +00:00
hieuhoang1972
95ad2a1609 comment out phrase dictionary filtering 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@493 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-04 15:08:06 +00:00
hieuhoang1972
f0957d6fbe minor cleanup 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@482 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-03 14:57:32 +00:00
zens
828fad58fc - parameter for distortion future costs 
- bug fix in source phrase tracking for CN


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@479 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-03 00:44:47 +00:00
redpony
0ac37af814 support for generation dictionaries having multiple feature values. code cleanup 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@478 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-03 00:21:42 +00:00
hieuhoang1972
425e1673ff allow different LM implementations to be used 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@475 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-02 23:22:42 +00:00
hieuhoang1972
d6377d7754 get rid of id in LM class 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@452 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-01 20:37:52 +00:00
hieuhoang1972
15536efa03 took out FactorType 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@450 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-01 20:02:39 +00:00
zens
ff15ed0732 - remove some obsolete code 
- optional future costs for distortion model


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@444 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-01 14:14:39 +00:00
hieuhoang1972
6bdf7eba18 memory pool cleanup 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@440 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-01 03:36:36 +00:00
hieuhoang1972
b2339c85fe remove LM types 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@432 1f5c12ca-751b-0410-a591-d2e778427230
 2006-08-01 00:10:28 +00:00
redpony
dfe989fbd2 fix error 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@426 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-31 21:47:30 +00:00
redpony
472abe01a3 add score to set weights in the right position in the global score vector so things aren't so brittle 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@423 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-31 21:14:50 +00:00
ccorbett
60af7b2844 integrated some of the confusion network code into LexicalReordering; fixed a bug in hashing the hypothesis conditioned on both the source and the target; started integrated some of the new scoring system into LexicalReordering. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@420 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-31 20:49:51 +00:00
hieuhoang1972
80423289ad rollback 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@418 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-31 20:26:18 +00:00
hieuhoang1972
66aeb9fe27 remove LM types 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@415 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-31 19:59:56 +00:00
redpony
ac9dffb069 Use selection algorithm (O(N)) instead of sort (O(NlogN)) to filter translation options, fix segv 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@407 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-31 16:25:10 +00:00
zens
a351147ba1 - cleaned up weighting of input models (confusion net) 
- added penalty/bonus for real words in confusion net


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@381 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-29 02:46:13 +00:00
zens
b4c9cccf6b - better handling of confusion net scores 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@379 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-29 02:16:04 +00:00
hieuhoang1972
38446f8def minor cleanup to enable non-N-Best decoding. small mem leak 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@375 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-29 01:39:22 +00:00
redpony
ec776dd2b7 major score refactoring 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@371 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-29 01:11:45 +00:00
zens
dec0f5f703 print confusion net posterior in nbest list (if input is confusion net) 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@368 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-28 22:57:31 +00:00
redpony
fb827cb544 preparation for the second round of score-keeping cleanups 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@366 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-28 22:41:44 +00:00
redpony
fdff6911d6 enable statistics gathering for # hypotheses pruned, recombined. add code for hash-based recombination. it works, but it's a little different than the tree-based impl so it's not enabled 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@363 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-28 21:18:51 +00:00
ccorbett
a4fe78d425 LexicalReordering incorporates weights read from config file in calculating the score of a hypothesis. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@361 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-28 19:28:56 +00:00
redpony
a404c447df getting ready for hash-based implementation of recombination. fix an existing recombination bug 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@360 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-28 19:08:24 +00:00
zens
0e6517b428 - fixing already fixed bugs that have been re-introduced by someone 
- moving some stuff from PhraseDictionaryTree to GenerateTuples
 - removing some obsolete code


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@359 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-28 18:14:20 +00:00
ccorbett
07db63fbf1 committing some lexreordering changes to enable the calculation of a bidirectional lexical reordering score. regression tests passed. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@357 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-28 16:30:41 +00:00
hieuhoang1972
2a4cb09a6c can set max trans option per coverage 
turned off profiling


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@350 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-28 03:20:09 +00:00
redpony
60b27b1bb1 Make score bookkeeping faster and easier to understand. Fix bug in unlimited distortion use-cases. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@345 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-28 01:14:42 +00:00
redpony
14d342508e preparation for new ScoreComponentCollection, which will be much more efficient 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@329 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-27 13:47:43 +00:00
zens
a2a3f5a4c2 - future costs estimation for distortion model 
(working, but further testing needed -> commented)


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@322 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-27 01:39:43 +00:00
redpony
88e62f919b add a global accessor for StaticData 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@308 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-26 19:32:04 +00:00
redpony
f6bdb0532d rename a few ambiguous function names. make ScoreComponentCollection not inherit from a stl container 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@301 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-26 14:29:32 +00:00
bojar
b0f7cb32cb Added -report-all-factors, so that stdout is more verbose. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@300 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-26 00:01:05 +00:00
hieuhoang1972
404c3ad663 fixed recombination bug 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@295 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-25 22:58:31 +00:00
hieuhoang1972
c99e2c1786 Visual Studio issues 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@289 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-25 21:41:04 +00:00
hieuhoang1972
fd87d305b5 fixed abort when 2 processes run at once. check whether filtered file exists just before renaming 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@281 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-25 19:01:35 +00:00
ccorbett
ee3e1e1576 Lexical reordering framework; confusion net still not working but Chris implied that was ok. All other regression tests passed 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@276 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-25 15:52:09 +00:00
hieuhoang1972
dd8a983149 merged up of chaching algorithm. lot6s of changed made to manager.cpp, TranslationOptColl.cpp, amongst other. 
other code may be broken as a result


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@267 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-25 04:53:29 +00:00
zens
287d01fab1 - weight for confusion net scores 
- bug fix in ranking of target phrases for confusion nets
 - code cleanup


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@266 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-25 00:24:18 +00:00
bojar
2e83536ef1 Moses accepts -report-source-span 1 for a mose verbose output. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@265 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-24 23:48:18 +00:00
zens
422418008e - confusion net: 
* more robust read functions
    * correct treatment of epsilons
    * code cleanup

 - parameter: fixed check for binary phrase table
 - staticData: do not read input phrases in case of binary phrase table


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@260 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-24 18:33:08 +00:00
zens
47edc0599d - removed some obsolete parts 
- avoided some code duplication
 - binary ttable is no longer a compiler switch:
   use binary ttable if exists, otherwise use standard ttable


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@253 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-21 22:03:03 +00:00
zens
98ef97e803 - translation of confusion nets integrated (works only with binary phrase table) 
- removed some obsolete stuff


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@250 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-21 21:09:23 +00:00
zens
70682b48a1 - prepare I/O for Confusion Nets 
- remove some of the dynamic casting stuff


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@245 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-21 17:43:42 +00:00
redpony
e6ae8ceee5 Add hook for word insertion task 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@243 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-21 15:30:01 +00:00
zens
c220c86501 - binary phrase table now fully supported 
-> define preprocessor variable USEBINTTABLE
      e.g. uncommenting the line in StaticData.cpp
           or on the make  command line
 - some clean up


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@241 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-21 13:50:41 +00:00
zens
325353b0f4 - support for binary ttable in moses almost completed 
- moved common part of CreateTranslationOptionCollection() for text and confusion net input
   to base class


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@237 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-21 13:22:00 +00:00
redpony
2c60d6b430 use pull LanguageModel creation logic into a LMFactory 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@217 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-20 15:30:27 +00:00
eherbst
7a6600617e minor cleaning-up 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@196 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-19 22:05:52 +00:00
redpony
507b8a7df7 remove overhead for word deletion 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@177 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-18 19:12:42 +00:00
eherbst
5c5d971895 added unknown-word handling code (currently commented out) and source-word deletion (also currently commented out) 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@176 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-18 18:13:45 +00:00
hieuhoang1972
f4de7e5e0e table limit for each phrase table 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@157 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-17 21:18:32 +00:00
redpony
605206c535 split LanguageModel into polymorphic classes to make it easier to add support for IRST LM. add comments. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@142 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-17 12:48:13 +00:00
zens
34b6f1d349 prefix tree phrase table: 
- access to tree structure
  - tool for conversion
  - some code cleaning / reformatting


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@129 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-14 23:50:58 +00:00
hieuhoang1972
4e757cc398 bug fixes & starting on translation option caching 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@117 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-14 18:43:48 +00:00
fetidadder
b941b343ff Basic structure for lexical reordering structure implemented: mechanism for reading pertinent options from config file, added new LexicalReordering class. Regression tests run and passed. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@116 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-14 18:20:07 +00:00
zens
45588167bd almost no modification to existing system, only additions: 
- prefix tree structured phrase table
 - binary phrase table format with on-demand loading


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@101 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-14 02:06:09 +00:00
redpony
f7481d1262 check in better logging. test commit email functionality 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@99 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-14 00:48:08 +00:00
alex_const
874983ded4 Added information about which stack a hypothesis gets added on (verbose option).Detects whether a newly added hypothesis is the best one on the stack. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@85 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-12 21:32:42 +00:00
redpony
57e3eda797 fix PhraseDictionary filtering bug. moved future cost estimation from 
TranslationOption to TargetPhrase.cpp and had TO reference the latter


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@84 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-12 21:01:53 +00:00
alex_const
d0526330d4 Added command line argument (-v / -verbose), which prints out information about each hypothesis for debugging. 
Changed toString method for "Word" to remove parantheses around words.


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@83 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-12 20:31:25 +00:00
hieuhoang1972
6da62624ff visual studio 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@65 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-12 01:46:30 +00:00
hieuhoang1972
1243ae7146 get rid of id's in dictionary 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@49 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-11 01:30:42 +00:00
eherbst
0a9d179442 changed name of max-hypo-stack-size parameter to match pharaoh 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@46 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-11 00:35:10 +00:00
eherbst
bb2f1e7123 added default value for max hypo-stack size 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@44 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-11 00:10:08 +00:00
hieuhoang1972
8f0751291f erase dictionary Id 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@43 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-11 00:03:20 +00:00
eherbst
e014f90b3a added max hypothesis-stack size as a runtime parameter (-hypmax) 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@42 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-10 22:07:34 +00:00
hieuhoang1972
09e46ab211 variable number of translation component scores 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@21 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-09 01:35:22 +00:00
hieuhoang1972
df0d858673 variable number of translation component scores 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@18 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-08 23:42:26 +00:00
hieuhoang1972
da55c9f5e2 variable number of translation component scores 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@13 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-08 03:15:31 +00:00
hieuhoang1972
1954227b76 variable number of translation component scores 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@10 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-08 01:18:35 +00:00
hieuhoang1972
7a79e349a2 variable number of translation component scores 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@9 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-08 00:28:51 +00:00
hieuhoang1972
1138c0c0f7 variable number of translation component scores 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@8 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-07 16:46:30 +00:00
hieuhoang1972
b9955c5bbd variable number of translation component scores 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@7 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-07 16:25:04 +00:00
hieuhoang1972
32edb3d668 start of sourceforge 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4 1f5c12ca-751b-0410-a591-d2e778427230
 2006-07-04 17:44:36 +00:00