moses-smt/mosesdecoder
1
1
Fork 0
mirror of https://github.com/moses-smt/mosesdecoder.git synced 2024-09-20 23:58:15 +03:00
Code Issues Packages Projects Releases Wiki Activity
6,705 Commits 136 Branches 26 Tags 147 MiB
f99c35395c
Commit Graph

1371 Commits

Author SHA1 Message Date
bhaddow
4d8f9a0716 Remove excessive debug 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4162 1f5c12ca-751b-0410-a591-d2e778427230
 2011-08-24 14:00:21 +00:00
mlegendr
3f0d83531f Part 3 of n-gram thing: added LanguageKenLM.h to public library headers 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4161 1f5c12ca-751b-0410-a591-d2e778427230
 2011-08-24 11:26:41 +00:00
heafield
6f391a7dbd Part 2 of Marc LEGENDRE's changes to expose n-gram length. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4160 1f5c12ca-751b-0410-a591-d2e778427230
 2011-08-24 10:45:41 +00:00
heafield
b3c06822ed Fix memory leak reported by Marc LEGENDRE. Also make the FFState for begin and null context const. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4158 1f5c12ca-751b-0410-a591-d2e778427230
 2011-08-24 10:15:29 +00:00
bhaddow
1f3c11b441 Merge branch 'master' into miramerge 
Passes all regression except for the score tests, as I still have to merge
in the missing files for these tests.

Conflicts:
	.gitignore
	Makefile.am
	configure.in
	moses-chart/moses-chart.xcodeproj/project.pbxproj
	moses/src/ChartHypothesis.cpp
	moses/src/ChartTranslationOptionCollection.cpp
	moses/src/ChartTrellisPath.cpp
	moses/src/DotChart.cpp
	moses/src/LMList.h
	moses/src/LanguageModel.cpp
	moses/src/LanguageModel.h
	moses/src/LanguageModelFactory.cpp
	moses/src/LanguageModelRemote.h
	moses/src/LexicalReordering.h
	moses/src/Makefile.am
	moses/src/Manager.cpp
	moses/src/Parameter.cpp
	moses/src/RuleCubeQueue.h
	moses/src/ScoreIndexManager.cpp
	moses/src/StaticData.cpp
	moses/src/StaticData.h
	moses/src/TargetPhrase.cpp
	moses/src/TargetPhrase.h
	moses/src/TranslationOptionCollection.cpp
 2011-08-22 14:52:02 +01:00
bhaddow
1282500392 Merge commit 'a0b6abdfd3599e7fbdc6aac76fcd2cb4483d63ce' into miramerge 
Conflicts:
	moses/src/ConfusionNet.cpp
	moses/src/DecodeFeature.cpp
	moses/src/DecodeFeature.h
	moses/src/DotChartOnDisk.cpp
	moses/src/DummyScoreProducers.cpp
	moses/src/DummyScoreProducers.h
	moses/src/DynSAInclude/vocab.h
	moses/src/FeatureFunction.h
	moses/src/GenerationDictionary.cpp
	moses/src/GenerationDictionary.h
	moses/src/GlobalLexicalModel.cpp
	moses/src/GlobalLexicalModel.h
	moses/src/LMList.cpp
	moses/src/LMList.h
	moses/src/LanguageModel.cpp
	moses/src/LanguageModel.h
	moses/src/LanguageModelFactory.cpp
	moses/src/LanguageModelFactory.h
	moses/src/LanguageModelImplementation.h
	moses/src/LanguageModelKen.h
	moses/src/LanguageModelMultiFactor.cpp
	moses/src/LanguageModelMultiFactor.h
	moses/src/LanguageModelParallelBackoff.h
	moses/src/LanguageModelRemote.h
	moses/src/LanguageModelSingleFactor.cpp
	moses/src/LanguageModelSingleFactor.h
	moses/src/LexicalReordering.cpp
	moses/src/LexicalReordering.h
	moses/src/LexicalReorderingState.cpp
	moses/src/LexicalReorderingState.h
	moses/src/Manager.cpp
	moses/src/PDTAimp.h
	moses/src/Parameter.cpp
	moses/src/Parameter.h
	moses/src/Phrase.h
	moses/src/PhraseDictionary.cpp
	moses/src/PhraseDictionary.h
	moses/src/PhraseDictionaryMemory.cpp
	moses/src/ScoreComponentCollection.cpp
	moses/src/ScoreComponentCollection.h
	moses/src/ScoreIndexManager.cpp
	moses/src/ScoreIndexManager.h
	moses/src/ScoreProducer.h
	moses/src/StaticData.cpp
	moses/src/StaticData.h
	moses/src/TargetPhrase.cpp
	moses/src/TargetPhrase.h
	moses/src/TranslationOption.cpp
	moses/src/TranslationOptionCollection.cpp
	moses/src/TranslationSystem.cpp
	moses/src/TranslationSystem.h
	moses/src/TrellisPath.h
 2011-08-19 17:09:36 +01:00
Ondrej Bojar
76648c840c Merge commit '5f0eacce4bd1781356e63f94c0d1d49e0ea81c82' into miramerge 
redone what Barry did before, i.e. including the commit 5f0e (aka r3895) of
trunk into the miramerge branch

status after the merge:
  moses compiles
  even the regression test phrase.basic-surface-only FAILS
    (it did fail identically on the mira branch alone)

I may be accidentally adding some files that should not be here, e.g. boost.m4.

Conflicts:
	Makefile.am
	config.h.in
	configure.in
	m4/boost.m4
	moses/src/Makefile.am
	moses/src/Parameter.cpp
	moses/src/PhraseDictionary.cpp
	moses/src/StaticData.cpp
	moses/src/StaticData.h
	scripts/generic/fsa2plf.pl
 2011-08-18 17:13:46 +02:00
oliver-wilson
96417949c2 Keep track of the order at which the last ngram request succeeded and 
use it to inform the next request.


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4151 1f5c12ca-751b-0410-a591-d2e778427230
 2011-08-18 12:36:02 +00:00
bhaddow
cf81f9188f Merged up to 3895 from trunk 
git-svn-id: http://svn.statmt.org/repository/mira@3934 cc96ff50-19ce-11e0-b349-13d7f0bd23df
 2011-08-18 12:59:38 +02:00
bhaddow
cc437739a1 Thread safety of feature vector. 
git-svn-id: http://svn.statmt.org/repository/mira@3933 cc96ff50-19ce-11e0-b349-13d7f0bd23df
 2011-08-18 12:59:38 +02:00
bhaddow
f19961c6a6 cleanup. move threshold into FeatureVector. restore encapsulation. 
git-svn-id: http://svn.statmt.org/repository/mira@3932 cc96ff50-19ce-11e0-b349-13d7f0bd23df
 2011-08-18 12:59:38 +02:00
pkoehn
7d3a6103a8 added PhraseLengthFeature 
git-svn-id: http://svn.statmt.org/repository/mira@3931 cc96ff50-19ce-11e0-b349-13d7f0bd23df
 2011-08-18 12:59:38 +02:00
pkoehn
69fdd15792 added WordTranslationFeature 
git-svn-id: http://svn.statmt.org/repository/mira@3930 cc96ff50-19ce-11e0-b349-13d7f0bd23df
 2011-08-18 12:59:38 +02:00
pkoehn
c815741145 added SourceWordDeletionFeature 
git-svn-id: http://svn.statmt.org/repository/mira@3929 cc96ff50-19ce-11e0-b349-13d7f0bd23df
 2011-08-18 12:59:38 +02:00
pkoehn
56a9a26c06 added TargetWordInsertionFeature 
git-svn-id: http://svn.statmt.org/repository/mira@3928 cc96ff50-19ce-11e0-b349-13d7f0bd23df
 2011-08-18 12:59:38 +02:00
pkoehn
7c4255c6f7 fix 
git-svn-id: http://svn.statmt.org/repository/mira@3927 cc96ff50-19ce-11e0-b349-13d7f0bd23df
 2011-08-18 12:59:38 +02:00
pkoehn
42e10d010b added reporting of sparse features in n-best list 
git-svn-id: http://svn.statmt.org/repository/mira@3926 cc96ff50-19ce-11e0-b349-13d7f0bd23df
 2011-08-18 12:59:38 +02:00
pkoehn
b919b862b8 added phrase length feature 
git-svn-id: http://svn.statmt.org/repository/mira@3925 cc96ff50-19ce-11e0-b349-13d7f0bd23df
 2011-08-18 12:59:38 +02:00
ehasler
120be1df4f code clean-up, step 1 
git-svn-id: http://svn.statmt.org/repository/mira@3918 cc96ff50-19ce-11e0-b349-13d7f0bd23df
 2011-08-18 12:59:38 +02:00
ehasler
2ada4d56e5 change verbosity for cerr messages, remove some unwanted options, introduce --margin-slack, --margin-incr 
git-svn-id: http://svn.statmt.org/repository/mira@3914 cc96ff50-19ce-11e0-b349-13d7f0bd23df
 2011-08-18 12:59:38 +02:00
bhaddow
0585646b2d Fix source side version 
git-svn-id: http://svn.statmt.org/repository/mira@3912 cc96ff50-19ce-11e0-b349-13d7f0bd23df
 2011-08-18 12:59:37 +02:00
ehasler
14a0d2c8da enable target bigrams of POS tags 
git-svn-id: http://svn.statmt.org/repository/mira@3909 cc96ff50-19ce-11e0-b349-13d7f0bd23df
 2011-08-18 12:59:37 +02:00
ehasler
c8c3f763ec fix setting core features for multi-score features 
git-svn-id: http://svn.statmt.org/repository/mira@3905 cc96ff50-19ce-11e0-b349-13d7f0bd23df
 2011-08-18 12:59:37 +02:00
ehasler
dedd322025 code cleanup 
git-svn-id: http://svn.statmt.org/repository/mira@3902 cc96ff50-19ce-11e0-b349-13d7f0bd23df
 2011-08-18 12:59:37 +02:00
ehasler
d598fc4c85 print oracle/reference length, implement --core-weights 
git-svn-id: http://svn.statmt.org/repository/mira@3901 cc96ff50-19ce-11e0-b349-13d7f0bd23df
 2011-08-18 12:59:37 +02:00
bhaddow
08c3efdf88 phrase boundary feature 
git-svn-id: http://svn.statmt.org/repository/mira@3893 cc96ff50-19ce-11e0-b349-13d7f0bd23df
 2011-08-18 12:59:37 +02:00
ehasler
1964eaf98a refactor handling of accumulated constraints, constraint checking, introduce burn-in for bleu history, sentence-level bleu, bleu score weight 
git-svn-id: http://svn.statmt.org/repository/mira@3882 cc96ff50-19ce-11e0-b349-13d7f0bd23df
 2011-08-18 12:59:36 +02:00
ehasler
eaada140a4 update ScoreComponentCollection 
git-svn-id: http://svn.statmt.org/repository/mira@3881 cc96ff50-19ce-11e0-b349-13d7f0bd23df
 2011-08-18 12:59:36 +02:00
ehasler
3266ff924d update FVector for linear classifier to work 
git-svn-id: http://svn.statmt.org/repository/mira@3880 cc96ff50-19ce-11e0-b349-13d7f0bd23df
 2011-08-18 12:59:36 +02:00
ehasler
1338ccb718 monitor changes in bleu feature history 
git-svn-id: http://svn.statmt.org/repository/mira@3875 cc96ff50-19ce-11e0-b349-13d7f0bd23df
 2011-08-18 12:59:36 +02:00
bhaddow
e664b4a4b3 Merge 3791-3842 from trunk 
git-svn-id: http://svn.statmt.org/repository/mira@3873 cc96ff50-19ce-11e0-b349-13d7f0bd23df
 2011-08-18 12:59:36 +02:00
ehasler
4bf98c9169 dont print name of feature in vector 
git-svn-id: http://svn.statmt.org/repository/mira@3854 cc96ff50-19ce-11e0-b349-13d7f0bd23df
 2011-08-18 12:59:35 +02:00
ehasler
269f1018c3 introduce option to use average weights for pruning 
git-svn-id: http://svn.statmt.org/repository/mira@3852 cc96ff50-19ce-11e0-b349-13d7f0bd23df
 2011-08-18 12:59:35 +02:00
bhaddow
c8e7c5da05 initialisation of source phrase member variable in TargetPhrase 
git-svn-id: http://svn.statmt.org/repository/mira@3849 cc96ff50-19ce-11e0-b349-13d7f0bd23df
 2011-08-18 12:59:35 +02:00
bhaddow
367ac683a3 Implementation of phrase pair feature 
git-svn-id: http://svn.statmt.org/repository/mira@3842 cc96ff50-19ce-11e0-b349-13d7f0bd23df
 2011-08-18 12:59:34 +02:00
bhaddow
5f4ac522e8 Print names as well as values in feature vector printout 
git-svn-id: http://svn.statmt.org/repository/mira@3840 cc96ff50-19ce-11e0-b349-13d7f0bd23df
 2011-08-18 12:59:34 +02:00
bhaddow
9545a2e068 don't try to get feature list for sparse features 
git-svn-id: http://svn.statmt.org/repository/mira@3839 cc96ff50-19ce-11e0-b349-13d7f0bd23df
 2011-08-18 12:59:34 +02:00
bhaddow
14b35072bc Avoid bad_alloc for variable length features 
git-svn-id: http://svn.statmt.org/repository/mira@3837 cc96ff50-19ce-11e0-b349-13d7f0bd23df
 2011-08-18 12:59:34 +02:00
ehasler
b7971689cb adapt BP factor 
git-svn-id: http://svn.statmt.org/repository/mira@3836 cc96ff50-19ce-11e0-b349-13d7f0bd23df
 2011-08-18 12:59:34 +02:00
ehasler
8d95753d0c back to dumping cumulative weights, fix learning rate and max-sentence-update 
git-svn-id: http://svn.statmt.org/repository/mira@3831 cc96ff50-19ce-11e0-b349-13d7f0bd23df
 2011-08-18 12:59:34 +02:00
ehasler
ffcd03dd51 change max-sentence-update to scale weights instead of clipping them, fix mpi commands 
git-svn-id: http://svn.statmt.org/repository/mira@3823 cc96ff50-19ce-11e0-b349-13d7f0bd23df
 2011-08-18 12:59:34 +02:00
ehasler
75565119d2 introduce parameters --max-sentence-update, --check-updates, --decr-learning-rate, change some default values, code cleanup 
git-svn-id: http://svn.statmt.org/repository/mira@3819 cc96ff50-19ce-11e0-b349-13d7f0bd23df
 2011-08-18 12:59:34 +02:00
bhaddow
3348dcd03e Configurable factor id for target bigram 
git-svn-id: http://svn.statmt.org/repository/mira@3812 cc96ff50-19ce-11e0-b349-13d7f0bd23df
 2011-08-18 12:59:33 +02:00
ehasler
7ceae4581a return unaltered state if no new words are translated 
git-svn-id: http://svn.statmt.org/repository/mira@3809 cc96ff50-19ce-11e0-b349-13d7f0bd23df
 2011-08-18 12:59:33 +02:00
ehasler
0993dde7ce fix precision 
git-svn-id: http://svn.statmt.org/repository/mira@3808 cc96ff50-19ce-11e0-b349-13d7f0bd23df
 2011-08-18 12:59:33 +02:00
ehasler
2f54b8f61f remove smoothing from function CalculateBleuOfCorpus 
git-svn-id: http://svn.statmt.org/repository/mira@3805 cc96ff50-19ce-11e0-b349-13d7f0bd23df
 2011-08-18 12:59:33 +02:00
ehasler
5312e8fc22 compute bleu on oracles translations of dev set, introduce params --min-weight-change, --msf-step, --msf-min, --slack-step, --slack-max, --decoder-settings 
git-svn-id: http://svn.statmt.org/repository/mira@3804 cc96ff50-19ce-11e0-b349-13d7f0bd23df
 2011-08-18 12:59:33 +02:00
ehasler
0db01446b7 fix bug introduced in last CL 
git-svn-id: http://svn.statmt.org/repository/mira@3802 cc96ff50-19ce-11e0-b349-13d7f0bd23df
 2011-08-18 12:59:33 +02:00
ehasler
df66bf2816 extend lexical reordering component to allow for more weights 
git-svn-id: http://svn.statmt.org/repository/mira@3801 cc96ff50-19ce-11e0-b349-13d7f0bd23df
 2011-08-18 12:59:33 +02:00
ehasler
606355deba Enable taking logs of feature values 
git-svn-id: http://svn.statmt.org/repository/mira@3798 cc96ff50-19ce-11e0-b349-13d7f0bd23df
 2011-08-18 12:59:33 +02:00
theleopardess
d7752b44fc I tested check-in by adding a test line in moses/src/StaticData.cpp, producing a trivial moses revision 4122. Now I have removed that line. Everything ok but sorry for the confusion. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4129 1f5c12ca-751b-0410-a591-d2e778427230
 2011-08-05 18:57:09 +00:00
theleopardess
f8a99e5d6d yanggao-softdep-v0 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4122 1f5c12ca-751b-0410-a591-d2e778427230
 2011-08-05 15:11:43 +00:00
hieuhoang1972
e7b97c1b1a vs build 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4109 1f5c12ca-751b-0410-a591-d2e778427230
 2011-08-04 04:53:21 +00:00
hieuhoang1972
a79651d239 fixed backoff phrase table. Allow backoff of unigrams 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4089 1f5c12ca-751b-0410-a591-d2e778427230
 2011-07-25 12:23:49 +00:00
hieuhoang1972
b0ec298ce2 vs.net build 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4088 1f5c12ca-751b-0410-a591-d2e778427230
 2011-07-23 23:52:34 +00:00
chesio
1b9d99a5ad BilingualDynSuffixArray corpus may now be loaded from gzipped file as well (removed needless call to seekg()). 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4083 1f5c12ca-751b-0410-a591-d2e778427230
 2011-07-21 23:29:11 +00:00
hieuhoang1972
06af5d40d4 Improved error message 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4081 1f5c12ca-751b-0410-a591-d2e778427230
 2011-07-21 02:41:23 +00:00
pjwilliams
113d0f24dd moses_chart: avoid doing some std::map retrievals during rule lookup 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4080 1f5c12ca-751b-0410-a591-d2e778427230
 2011-07-19 12:57:02 +00:00
hieuhoang1972
9c0d725cde visual studio 2010 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4079 1f5c12ca-751b-0410-a591-d2e778427230
 2011-07-19 03:07:15 +00:00
pjwilliams
beba4b475f moses_chart: merge DottedRule and CoveredChartSpan classes. This saves 
some memory for models that require a lot of lookup state (generally
grammars with lots of target categories).

git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4078 1f5c12ca-751b-0410-a591-d2e778427230
 2011-07-18 21:44:27 +00:00
heafield
954dfd7d5e Optional compression for trie. Also, some better error handling. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4074 1f5c12ca-751b-0410-a591-d2e778427230
 2011-07-13 20:53:18 +00:00
leven101
cd96c02748 bug fixes 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4068 1f5c12ca-751b-0410-a591-d2e778427230
 2011-07-06 17:25:54 +00:00
leven101
52ce926901 added ClearWordInCache() to clear (nonfrequent) lexical word pair probs after suffix array updates 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4065 1f5c12ca-751b-0410-a591-d2e778427230
 2011-07-01 15:40:48 +00:00
leven101
e0174b413c added CacheFreqWords() to speed up decoding with suffix array PTs 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4064 1f5c12ca-751b-0410-a591-d2e778427230
 2011-07-01 14:36:28 +00:00
hieuhoang1972
3585d8bae2 xcode 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4057 1f5c12ca-751b-0410-a591-d2e778427230
 2011-06-30 18:11:04 +00:00
hieuhoang1972
12d16af0bb nothing important 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4056 1f5c12ca-751b-0410-a591-d2e778427230
 2011-06-30 17:55:16 +00:00
oliver-wilson
e49144f49d Only include LanguageModelDMapLM.h if compiling with DMap. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4053 1f5c12ca-751b-0410-a591-d2e778427230
 2011-06-30 15:42:06 +00:00
oliver-wilson
fbe8f1467c Add new language model class for DMapLM but do not link it to the build. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4052 1f5c12ca-751b-0410-a591-d2e778427230
 2011-06-30 15:15:24 +00:00
hieuhoang1972
ed7ecd5ce2 compile on gcc 4.6 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4051 1f5c12ca-751b-0410-a591-d2e778427230
 2011-06-30 04:20:53 +00:00
pjwilliams
7e288fae98 moses_chart: reduce memory use for rule lookup by decreasing the amount 
of state information duplicated between CoveredChartSpan objects.

git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4050 1f5c12ca-751b-0410-a591-d2e778427230
 2011-06-29 13:38:11 +00:00
bhaddow
7fe3143feb Improve debug 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4049 1f5c12ca-751b-0410-a591-d2e778427230
 2011-06-29 08:27:43 +00:00
hieuhoang1972
024b5f9100 vs.net build 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4048 1f5c12ca-751b-0410-a591-d2e778427230
 2011-06-28 19:38:57 +00:00
hieuhoang1972
b9ef46972c vs.net 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4047 1f5c12ca-751b-0410-a591-d2e778427230
 2011-06-28 19:26:12 +00:00
hieuhoang1972
f7d534bcdd xcode 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4046 1f5c12ca-751b-0410-a591-d2e778427230
 2011-06-28 19:02:09 +00:00
pjwilliams
2451371ca2 Changes to chart decoder cube pruning: create one cube per dotted rule 
instead of one per translation and do 'non-lazy' scoring, i.e. fully
score the corner and neighbor hypotheses inside the rule cube instead
of waiting until an item is popped.  The old behaviour -- faster but
with more search errors -- is available via the
cube-pruning-lazy-scoring option.

git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4039 1f5c12ca-751b-0410-a591-d2e778427230
 2011-06-27 15:13:15 +00:00
phkoehn
c7cc79a20e output no dead end hypotheses in search graph, note recombination (chart decoder) 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4038 1f5c12ca-751b-0410-a591-d2e778427230
 2011-06-27 00:38:43 +00:00
heafield
5e70e3bd40 Quantization. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4037 1f5c12ca-751b-0410-a591-d2e778427230
 2011-06-26 22:21:44 +00:00
pjwilliams
913f339dd0 Remove unused m_ngramScore and m_countInfo variables from TargetPhrase. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4035 1f5c12ca-751b-0410-a591-d2e778427230
 2011-06-24 18:29:50 +00:00
leven101
5acb99d76f ClearTransOptionCache() causes segfault when translating next sentence 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4030 1f5c12ca-751b-0410-a591-d2e778427230
 2011-06-24 08:12:50 +00:00
hieuhoang1972
13c1855e8f xcode 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4029 1f5c12ca-751b-0410-a591-d2e778427230
 2011-06-23 14:33:43 +00:00
pjwilliams
ab3460591c Share AlignmentInfo objects instead of storing one per TargetPhrase. 
This can save a significant amount of memory used on rule table storage,
though may increase loading time slightly.

git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4017 1f5c12ca-751b-0410-a591-d2e778427230
 2011-06-16 21:20:20 +00:00
hieuhoang1972
85283f5bee vs.net build 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4015 1f5c12ca-751b-0410-a591-d2e778427230
 2011-06-16 00:55:45 +00:00
phkoehn
4a6fec7613 chart decoder recombination now based on lm state, not suffix anymore; same feature function handling as in the phrase decoder 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4014 1f5c12ca-751b-0410-a591-d2e778427230
 2011-06-15 21:31:27 +00:00
hieuhoang1972
e5955ef1b3 make sure each parameter in ini file is known 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4011 1f5c12ca-751b-0410-a591-d2e778427230
 2011-06-11 04:59:55 +00:00
hieuhoang1972
1a29541243 don't use counts for desperation pruning 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4007 1f5c12ca-751b-0410-a591-d2e778427230
 2011-06-10 06:39:03 +00:00
hieuhoang1972
21df1feb26 xcode 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4006 1f5c12ca-751b-0410-a591-d2e778427230
 2011-06-10 01:01:09 +00:00
leven101
894b49a5b2 added LM updates to mosesserver (only for LanguageModelORLM) 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4005 1f5c12ca-751b-0410-a591-d2e778427230
 2011-06-09 17:27:48 +00:00
hieuhoang1972
4bf85266d8 dont process unknown words for 1st or last place. They're the <s> & </s> and should only be handled by the glue rules 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4004 1f5c12ca-751b-0410-a591-d2e778427230
 2011-06-08 16:22:56 +00:00
leven101
ec04285270 hash file 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4003 1f5c12ca-751b-0410-a591-d2e778427230
 2011-06-08 15:45:08 +00:00
leven101
4ea818f34a Added wrapper files for online randomised LM prototype 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4002 1f5c12ca-751b-0410-a591-d2e778427230
 2011-06-08 15:05:19 +00:00
hieuhoang1972
0a04cb11bf added std::size_t for gcc 4.6 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4001 1f5c12ca-751b-0410-a591-d2e778427230
 2011-06-08 05:49:47 +00:00
hieuhoang1972
46c4dd441e loosen assert in case of rounding errors 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@4000 1f5c12ca-751b-0410-a591-d2e778427230
 2011-06-06 16:06:51 +00:00
leven101
bcbc0c6e09 bug fix 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3998 1f5c12ca-751b-0410-a591-d2e778427230
 2011-05-31 12:04:40 +00:00
leven101
547a6cbf62 added ClearTransOptCache() to StaticData class and call it when updating suffix PTs 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3997 1f5c12ca-751b-0410-a591-d2e778427230
 2011-05-31 09:43:17 +00:00
leven101
f64885599d added ClearTransOptCache() to StaticData class and call it when updating suffix PTs 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3996 1f5c12ca-751b-0410-a591-d2e778427230
 2011-05-31 09:42:27 +00:00
dowobeha
d12dd2b4d7 Added additional level 3 logging to help verify that Syntactic LM is being loaded and used 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3992 1f5c12ca-751b-0410-a591-d2e778427230
 2011-05-24 15:47:47 +00:00
dowobeha
e04e155986 Syntactic LM can now be loaded properly by StaticData. This change should have been merged earlier, but was omitted. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3991 1f5c12ca-751b-0410-a591-d2e778427230
 2011-05-24 15:47:35 +00:00
heafield
c74f5446c8 Unbreak the build. Sorry about that. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3981 1f5c12ca-751b-0410-a591-d2e778427230
 2011-05-17 17:02:10 +00:00
naditomeh
5019e98abe A minor issue when trying to output the search graph with one of these 
flags: output-search-graph[-extended].
An empty input sentence results in a segfault.
Fix:
after the line
const Hypothesis *prevHypo = hypo->GetPrevHypo();
test if prevHypo is null before using it.


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3978 1f5c12ca-751b-0410-a591-d2e778427230
 2011-05-16 13:41:15 +00:00
dowobeha
cd017c1448 Added missing syntactic LM function declaration. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3976 1f5c12ca-751b-0410-a591-d2e778427230
 2011-05-13 20:58:03 +00:00
dowobeha
9581b6aae2 Updated syntactic classes (cherry-picking from commit bf7a7bb8e7) 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3973 1f5c12ca-751b-0410-a591-d2e778427230
 2011-05-13 19:28:23 +00:00
dowobeha
9375aa8846 Reverting changes. Revision 3971 was a bad commit. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3972 1f5c12ca-751b-0410-a591-d2e778427230
 2011-05-13 18:27:30 +00:00
dowobeha
bb941c01f6 Merge branch 'master' into local-trunk 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3971 1f5c12ca-751b-0410-a591-d2e778427230
 2011-05-13 18:07:21 +00:00
pjwilliams
70d67ffd46 Chart decoder: include child hypothesis costs in estimates when 
pruning the ChartTranslationOptionList.

git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3943 1f5c12ca-751b-0410-a591-d2e778427230
 2011-04-04 20:43:02 +00:00
hieuhoang1972
8c4692e581 warnings 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3941 1f5c12ca-751b-0410-a591-d2e778427230
 2011-04-01 23:37:20 +00:00
hieuhoang1972
6acbdcdd28 derivation of n-best list output 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3939 1f5c12ca-751b-0410-a591-d2e778427230
 2011-03-30 21:31:15 +00:00
hieuhoang1972
adc2ac2c6a xcode 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3938 1f5c12ca-751b-0410-a591-d2e778427230
 2011-03-30 20:31:09 +00:00
bhaddow
d9d1c9440a Fix prehistoric memory leak. This was affecting the loading of 
alignment info from binarised phrase tables.


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3936 1f5c12ca-751b-0410-a591-d2e778427230
 2011-03-25 17:42:21 +00:00
hieuhoang1972
6cb1993541 xcode 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3932 1f5c12ca-751b-0410-a591-d2e778427230
 2011-03-21 06:32:06 +00:00
oliver-wilson
e168875599 Fix broken refactoring. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3931 1f5c12ca-751b-0410-a591-d2e778427230
 2011-03-18 15:58:12 +00:00
oliver-wilson
888e551aae Init thread specific data in Load() so the LM is useable in the main 
thread.


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3929 1f5c12ca-751b-0410-a591-d2e778427230
 2011-03-16 21:04:34 +00:00
hieuhoang1972
a4d22f7714 vs.net 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3928 1f5c12ca-751b-0410-a591-d2e778427230
 2011-03-11 19:26:01 +00:00
bhaddow
4a652200e4 Fix compile error in boost version 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3927 1f5c12ca-751b-0410-a591-d2e778427230
 2011-03-11 17:35:31 +00:00
hieuhoang1972
77c7a27f13 xcode 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3926 1f5c12ca-751b-0410-a591-d2e778427230
 2011-03-11 17:05:05 +00:00
phkoehn
ff8a624344 re-organized syntax code for clarity - part II 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3925 1f5c12ca-751b-0410-a591-d2e778427230
 2011-03-11 16:28:36 +00:00
phkoehn
1cbe245919 re-organized syntax code for clarity - part I 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3924 1f5c12ca-751b-0410-a591-d2e778427230
 2011-03-11 13:08:43 +00:00
pjwilliams
26d125d3cd documentation of chart decoder 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3923 1f5c12ca-751b-0410-a591-d2e778427230
 2011-03-10 12:58:49 +00:00
phkoehn
ad140593b0 documentation of chart decoder 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3920 1f5c12ca-751b-0410-a591-d2e778427230
 2011-03-09 17:55:13 +00:00
heafield
9d4d2a413e Have language models also return whether the word was OOV or not. This isn't used yet, but will make it into a feature. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3916 1f5c12ca-751b-0410-a591-d2e778427230
 2011-03-07 23:21:09 +00:00
bhaddow
3c9112bc74 remove useless debug 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3912 1f5c12ca-751b-0410-a591-d2e778427230
 2011-03-03 17:05:28 +00:00
phkoehn
4ee7e5f673 added some comments 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3911 1f5c12ca-751b-0410-a591-d2e778427230
 2011-03-02 19:02:07 +00:00
oliver-wilson
106c4e0fc2 Init thread specific data on RandLM before sentence processing. Needed 
for multithreaded RandLM.


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3910 1f5c12ca-751b-0410-a591-d2e778427230
 2011-03-02 16:07:20 +00:00
pjwilliams
75709a6c87 Memory efficiency: make the reserveSize argument non-optional in 
Moses::Phrase's constructor.  The default used to be ARRAY_SIZE_INCR = 10,
which will be excessive in many cases.  Where the default was used, I've
set the exact size where that was obvious and explicitly used ARRAY_SIZE_INCR 
otherwise.  If you know the code involved, it's probably worth reviewing.



git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3908 1f5c12ca-751b-0410-a591-d2e778427230
 2011-02-28 11:41:08 +00:00
leven101
a5a860ad1f optimized some parameters 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3907 1f5c12ca-751b-0410-a591-d2e778427230
 2011-02-25 14:53:34 +00:00
hieuhoang1972
251011df9e irstlm variable 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3906 1f5c12ca-751b-0410-a591-d2e778427230
 2011-02-25 02:26:05 +00:00
phkoehn
93ed0ca1a9 get rid of "unused variable" warnings 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3903 1f5c12ca-751b-0410-a591-d2e778427230
 2011-02-24 16:17:38 +00:00
hieuhoang1972
a0b6abdfd3 run beautify.perl. Consistent formatting for .h & .cpp files 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3901 1f5c12ca-751b-0410-a591-d2e778427230
 2011-02-24 13:14:42 +00:00
oliver-wilson
06df9d1770 Changed white space to test effect on svn blame. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3892 1f5c12ca-751b-0410-a591-d2e778427230
 2011-02-22 16:48:51 +00:00
bhaddow
e7a8ba3b7f check for correct number of factors 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3891 1f5c12ca-751b-0410-a591-d2e778427230
 2011-02-22 13:06:16 +00:00
maurocettolo
4c6dfbddc3 minor changes to make Moses compliant with IRSTLM toolkit (release 5.60.01) 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3878 1f5c12ca-751b-0410-a591-d2e778427230
 2011-02-11 11:32:35 +00:00
ales-t
e922c159b6 Alignment points are also created for unknown source words. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3876 1f5c12ca-751b-0410-a591-d2e778427230
 2011-02-08 18:04:09 +00:00
pjwilliams
d4359f9875 If Boost is available, use per-sentence object pools to allocate ProcessedRule 
and WordConsumed objects (which are used to store rule table lookup state).
Large numbers of these objects are used during decoding and this can
significantly improve performance, especially for multithreaded decoding,
though at the cost of increased total memory use.

The ./configure option --disable-boost-pool can be used to disable this
feature if memory is tight.  This currently only affects moses_chart with
in-memory rule tables.



git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3873 1f5c12ca-751b-0410-a591-d2e778427230
 2011-02-07 15:43:19 +00:00
bhaddow
6221d2a558 Patch to add covered to osgx from Dennis Mehay 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3872 1f5c12ca-751b-0410-a591-d2e778427230
 2011-02-04 16:21:57 +00:00
maurocettolo
8fcd76f2fc made handling of chunk LM compatible with recent efficiency updates to IRSTLM toolkit by Nicola 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3870 1f5c12ca-751b-0410-a591-d2e778427230
 2011-02-04 07:42:56 +00:00
phkoehn
4e72cd91be added decoding-graph-backoff, still experimenting with it 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3869 1f5c12ca-751b-0410-a591-d2e778427230
 2011-02-03 13:41:44 +00:00
bojar
0bc0ece594 Ales Tamchyna's printing of alignments (-print-alignment-info did nothing) 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3867 1f5c12ca-751b-0410-a591-d2e778427230
 2011-02-03 09:08:42 +00:00
hieuhoang1972
0eed5716b7 get rid of linked trans opt 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3864 1f5c12ca-751b-0410-a591-d2e778427230
 2011-02-02 11:24:19 +00:00
hieuhoang1972
e087e78df9 get rid of linked trans opt 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3863 1f5c12ca-751b-0410-a591-d2e778427230
 2011-02-02 11:06:19 +00:00
hieuhoang1972
abacb9166a xcode 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3857 1f5c12ca-751b-0410-a591-d2e778427230
 2011-01-28 14:57:55 +00:00
heafield
87f15593da Remove vestigial len parameter from language model calls 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3856 1f5c12ca-751b-0410-a591-d2e778427230
 2011-01-27 19:01:45 +00:00
pjwilliams
967b7be213 Support for multithreading in moses_chart (-threads option). This hasn't 
been thoroughly tested yet, so don't be surprised if it breaks.  Verbose
output will be scrambled.



git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3853 1f5c12ca-751b-0410-a591-d2e778427230
 2011-01-26 13:15:13 +00:00
redpony
eddb28e0ce facilitate programmatic creation of word lattices 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3848 1f5c12ca-751b-0410-a591-d2e778427230
 2011-01-25 20:08:29 +00:00
pjwilliams
8051c5ad35 Use TranslationTask objects to perform sentence decoding in moses-chart. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3846 1f5c12ca-751b-0410-a591-d2e778427230
 2011-01-25 17:15:30 +00:00
pjwilliams
99bbfe938b Use OutputCollector to write moses-chart output. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3843 1f5c12ca-751b-0410-a591-d2e778427230
 2011-01-25 15:17:17 +00:00
pjwilliams
67b30ea0c7 Move sentence-specific rule lookup state out of PhraseDictionarySCFG and 
PhraseDictionaryOnDisk and into ChartRuleLookupManager.



git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3842 1f5c12ca-751b-0410-a591-d2e778427230
 2011-01-24 19:14:19 +00:00
bhaddow
7b6503680a Shortcut when trans opts cache is size 0. Avoids potential uninitialised read. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3825 1f5c12ca-751b-0410-a591-d2e778427230
 2011-01-21 21:25:14 +00:00
hieuhoang1972
96bd3a164d vs.net 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3802 1f5c12ca-751b-0410-a591-d2e778427230
 2011-01-17 16:19:33 +00:00
bhaddow
a9cd71628a Change of boost macros - please make sure you favourite configuration still works 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3799 1f5c12ca-751b-0410-a591-d2e778427230
 2011-01-13 23:38:48 +00:00
pjwilliams
d20667a46d Faster lookup for rules with source and/or target syntax labels (in-memory rule 
table only).


git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@3797 1f5c12ca-751b-0410-a591-d2e778427230
 2011-01-13 00:25:10 +00:00
bhaddow
a2730c445d Merge up to 3791 from trunk. 
git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/branches/mira-mtm5@3792 1f5c12ca-751b-0410-a591-d2e778427230
 2011-01-05 13:49:44 +00:00