Phil Williams
5240c430ce
Merge s2t branch
...
This adds a new string-to-tree decoder, which can be enabled with the -s2t
option. It's intended to be faster and simpler than the generic chart
decoder, and is designed to support lattice input (still WIP). For a en-de
system trained on WMT14 data, it's approximately 40% faster in practice.
For background information, see the decoding section of the EMNLP tutorial
on syntax-based MT:
http://www.emnlp2014.org/tutorials/5_notes.pdf
Some features are not implemented yet, including support for internal tree
structure and soft source-syntactic constraints.
2014-11-04 13:13:56 +00:00
Rico Sennrich
61c00ed636
n-best tree output
2014-09-12 19:17:23 +02:00
Barry Haddow
11d5fc3fef
Remove, don't just comment out
2014-07-24 16:25:53 +01:00
Barry Haddow
d43f0e40fe
Unused code
2014-07-24 16:20:49 +01:00
Nicola Bertoldi
1fe4eb0528
beautify
2014-06-08 09:44:59 +02:00
Nicola Bertoldi
4d75c889f1
merged master into dynamic-models
2014-06-08 09:39:37 +02:00
Nicola Bertoldi
c47d956e4c
added support for segment id
2014-06-08 09:16:03 +02:00
Nicola Bertoldi
306dc26edd
the management of passthrough xml-tags is introduced
2014-06-06 00:53:36 +02:00
Hieu Hoang
409c8d5349
Merge ../mosesdecoder into hieu
2014-05-20 00:16:58 +01:00
Matthias Huck
1740478238
Framework for additional phrase properties in decoding.
...
Derive your property class from PhraseProperty. Do any expensive string
processing of the property value in there, not in the feature
implementation, and provide methods to access the information in
appropriate data formats. The property value string will thus have to
be processed only once (on loading) rather than each time the respective
phrase is applied and your feature needs to access the property value.
2014-05-19 21:54:08 +01:00
Nicola Bertoldi
2f3cd5e2fe
beautify
2014-05-19 15:34:27 +02:00
Nicola Bertoldi
20381cbf89
merged master into dynamic-models and solved conflicts
2014-04-28 19:18:38 +02:00
Hieu Hoang
6b1e147cff
add -default-non-term-for-empty-range-only
2014-04-07 14:00:40 +01:00
Hieu Hoang
e5a6d63cf2
Merge branch 'master' of github.com:moses-smt/mosesdecoder into hieu
2014-03-21 21:41:04 +00:00
Rico Sennrich
45630a5851
various optimizations to make CYK+ parser several times faster and eat less memory.
...
speed-up of decoding depends on how much time is spent in parser:
10-50% speed-up for string-to-tree systems observed (more on long sentences and with high max-chart-span).
if you only use hiero or string-to-tree models (but none with source syntax), use compile-option --unlabelled-source for (small) efficiency gains.
2014-03-21 11:12:24 +00:00
Hieu Hoang
5e93c3f804
conflicts
2014-03-07 12:18:18 +00:00
Rico Sennrich
c27ecbe5c6
rename SyntaxConstraintFeature to TreeStructureFeature
...
(makes it clearer what it does; build/print the internal tree structure in string-to-tree decoding)
no longer rely on name of FF for printing internal trees
2014-03-03 13:56:19 +00:00
Hieu Hoang
1e5cd16fbf
xml parsing for source syntax
2014-02-26 18:09:27 +00:00
Hieu Hoang
1716514eda
create ResetWeights(). Only for dense weights
2014-02-23 20:07:28 +00:00
Hieu Hoang
cb2805151b
delete -adjacent-only. Rubbish
2014-01-27 12:25:02 +00:00
Hieu Hoang
9745924937
output list of available feature functions, separated by space, rather than lines
2014-01-25 10:39:47 +00:00
Rico Sennrich
5d2b4d8efc
sparse feature weights can now be set in moses.ini; overrides weights in weight-file.
2014-01-20 15:54:17 +00:00
Hieu Hoang
20e7d078a6
add feature function HyperParameterAsWeight
2014-01-17 17:59:36 +00:00
Nicola Bertoldi
4b072f2097
merge master into this branch
2014-01-17 14:04:15 +01:00
Rico Sennrich
ed25bb2b99
soft matching of target-side nonterminals
2014-01-16 18:34:33 +00:00
Nicola Bertoldi
bd83999264
beautify
2014-01-15 16:42:02 +01:00
Nicola Bertoldi
47bece6eac
code cleanup; fixings to others' code/test
2014-01-15 16:16:37 +01:00
Nicola Bertoldi
c13bb8f8e8
partial code cleanup (not working)
2014-01-14 18:27:11 +01:00
Nicola Bertoldi
50970b2b59
merge master into this branch
2014-01-14 08:50:18 +01:00
Nicola Bertoldi
c2a053bd80
added a dynamic cache-based PhraseDitctionary; applied changes to the cache-based LanguageModel ; a lot of verbose message; some code cleanup
2014-01-13 15:41:52 +01:00
Hieu Hoang
ba209202ef
move all feature functions out of StaticData
2013-12-07 00:21:06 +00:00
Hieu Hoang
09d50f87b1
move word penalty feature out of StaticData
2013-12-06 15:57:49 +00:00
Hieu Hoang
d3301fa1a3
move backoff parameter into decode graph class
2013-12-05 12:19:55 +00:00
Hieu Hoang
028edf8550
add -feature-name-overwrite. Specifically to replace PhraseDictionaryMemory with PhraseDictionaryScope3 for scope-3 decoding
2013-11-22 19:13:09 +00:00
Hieu Hoang
a57c9d6fe8
delete -parsing-algorithm. Scope-3 parsing depends on the phrase-table implementation used, not this argument
2013-11-22 17:35:22 +00:00
Hieu Hoang
749bd7fc8f
add decoder arg -no-cache for mira tuning
2013-11-21 11:52:20 +00:00
Hieu Hoang
3f8972d67b
move collection of generation-table from Staticdata to GenerationDictionary
2013-11-15 17:43:41 +00:00
Hieu Hoang
2f00ae936b
move collection of phrase-table from Staticdata to PhraseDictionary
2013-11-15 17:26:26 +00:00
Hieu Hoang
d51702f387
output list of available feature functions
2013-11-01 15:16:47 +00:00
Phil Williams
69f5e4bec3
Merge branch 'master' into GHKMStruct
2013-10-09 16:21:49 +01:00
Hieu Hoang
6759af3821
beautify
2013-10-03 11:33:48 +01:00
Hieu Hoang
52215fdf79
speed up compiling by removing headers from StaticData.h
2013-10-03 11:16:06 +01:00
Hieu Hoang
6ccc17b804
speed up compiling by removing headers from StaticData.h
2013-10-03 11:05:53 +01:00
Hieu Hoang
f2d218f8ee
move checking of legacy phrase-table to StaticData
2013-10-02 20:02:05 +01:00
Phil Williams
2a28d1a73e
Merge branch 'master' into GHKMStruct
...
Conflicts:
moses-chart-cmd/IOWrapper.cpp
moses-chart-cmd/IOWrapper.h
moses/FF/Factory.cpp
moses/Parameter.cpp
moses/StaticData.h
phrase-extract/extract-ghkm/ScfgRuleWriter.cpp
phrase-extract/score-main.cpp
2013-09-29 15:27:09 +01:00
Hieu Hoang
31ce9b510e
beautify
2013-09-27 09:35:24 +01:00
braunefe
f23c1d7769
Added All Translation Options for DIMw
2013-09-23 16:30:28 +02:00
Hieu Hoang
574d716f23
placeholder for chart decoding
2013-09-22 15:09:49 +02:00
Hieu Hoang
a4f5879fef
take out -constraint from Parameter. Now handled exclusively in FF
2013-09-18 06:23:22 +02:00
Nicola Bertoldi
7dc6ad4255
beautify
2013-09-14 10:13:24 +02:00
Matthias Huck
c39bed60c0
Tree fragments in GHKM glue rules;
...
output of LHS tag in tree fragments for UNKs;
GHKMParse info is now denoted as Tree info
2013-09-13 17:10:21 +02:00
Phil Williams
f3320c1ca0
Add -Tghkm parameter
...
Writes internal tree structure (if available) for 1-best derivation.
Code contributed by Nina Seemann.
2013-09-12 11:31:12 +02:00
Hieu Hoang
769f2fe00c
start on forced decoding for SCFG
2013-08-30 17:45:56 +01:00
Hieu Hoang
21c91dede2
delete GetWordPenaltyProducer(). Need non-const version only for mira
2013-08-22 09:24:46 +01:00
Nicola Bertoldi
dc9353fd9e
rpeparation for Dynamic cache-based LM
2013-08-17 22:10:15 +02:00
Jeremy Gwinnup
091bed03af
Added option to output unknown words with prefix 'UNK'
2013-08-15 11:30:08 -04:00
Nicola Bertoldi
048f269e8e
preparation for dynamic cache-based phrase dictionary
2013-08-14 16:11:43 +02:00
Nicola Bertoldi
5868653bd6
beautify
2013-08-11 23:41:23 +02:00
Nicola Bertoldi
7411227305
clean up related to the PhrrasePenalty producer
...
transform the PhrasePenalty basic feature functions into a FF like
WordPenalty
2013-08-11 23:32:54 +02:00
Hieu Hoang
6f7d239f13
Merge github.com:moses-smt/mosesdecoder into hieu_opt_input
2013-08-07 15:19:22 +01:00
phikoehn
67c3063574
Merge branch 'master' of ssh://github.com/moses-smt/mosesdecoder
2013-08-07 05:32:59 +01:00
phikoehn
ab4e3c63a6
enriched trace
2013-08-07 05:31:45 +01:00
Hieu Hoang
d3aa3ee65c
source and target factors for placeholders
2013-07-19 18:23:38 +01:00
Hieu Hoang
2da41f7a56
Merge github.com:moses-smt/mosesdecoder into hieu_opt_input
2013-07-18 17:01:51 +01:00
Hieu Hoang
cb46bd3c00
beautify
2013-07-18 17:00:44 +01:00
Hieu Hoang
f81d7a446f
starting placeholder
2013-07-18 17:00:07 +01:00
Hieu Hoang
475ebd2bbb
get rid of translation option cache
2013-07-10 19:44:38 +01:00
Hieu Hoang
310b26f989
beautify
2013-07-08 20:52:14 +01:00
Hieu Hoang
3eba5782c2
beautify
2013-07-08 20:25:47 +01:00
Hieu Hoang
dc33fa3d3d
redo parsing of feature function parameters
2013-06-20 12:50:41 +01:00
Rico Sennrich
9e9cbbc13b
refactoring multimodel code out of StaticData; locking.
2013-06-12 14:47:40 +02:00
Hieu Hoang
abe6bb7c22
refactor parsing of feature functiona args
2013-06-10 18:11:55 +01:00
phikoehn
0538ac84d6
nuke silly debug message
2013-06-09 14:54:13 +01:00
Hieu Hoang
e291c5f4a4
implement feature override as a run-time override of specific argument of a feature function. Only implemented table-limit for far
2013-06-07 17:32:01 +01:00
Hieu Hoang
cf910955d0
cleanup LoadFeatureFunctions()
2013-06-06 15:51:31 +01:00
phikoehn
d0699b177d
alternate weight settings can now ignore decoding graphs
2013-06-06 14:48:52 +01:00
phikoehn
081710ed4c
alternate weight settings: added option to turn off features completely
2013-06-05 19:44:43 +01:00
phikoehn
d261106571
alternate weight setting now with correct caching
2013-06-05 14:06:04 +01:00
Hieu Hoang
0c48e687d1
delete information on input scores from StaticData
2013-06-05 01:41:52 +01:00
Hieu Hoang
fc7e4f4550
minor cleanup of InputFeature
2013-06-04 22:09:21 +01:00
Hieu Hoang
21ff6d4831
beautify
2013-06-03 12:33:18 +01:00
Hieu Hoang
eaf53769ad
consistent Load() for all feature functions. Can be used not to load if doing --show-weights
2013-05-31 20:21:02 +01:00
phikoehn
d1650a5aa7
basic support for alternate weight settings
2013-05-31 12:28:57 +01:00
Hieu Hoang
6249432407
beautify
2013-05-29 18:16:15 +01:00
Hieu Hoang
06a5c86695
delete LMList
2013-05-27 16:50:21 +01:00
Hieu Hoang
195186b335
delete references to WordPenaltyProducer
2013-05-21 13:39:31 +01:00
Hieu Hoang
64b4a8805a
remove all mention of MetaFeature
2013-05-16 19:41:35 +01:00
Hieu Hoang
f04ec4c56d
1. remove all code for MetaFeature from mira.
...
2. in ShowWeights(), all print out dense feature weights. Don't print 'sparse' for sparse feature functions. All features functions can contains dense and sparse
2013-05-16 19:05:08 +01:00
Hieu Hoang
981351758a
delete class TranslationSystem
2013-05-11 14:13:26 +01:00
Hieu Hoang
7e398d3bca
port PhraseDictionaryMultiModel to new format
2013-05-10 15:33:46 +01:00
Hieu Hoang
ed7ab8146f
port PhraseDictionaryMultiModel to new format
2013-05-10 12:30:01 +01:00
Hieu Hoang
ce80626c9a
remove SparsePhraseDictionaryFeature
2013-05-01 13:17:09 +01:00
Hieu Hoang
83494418e2
delete PhraseDictionaryMemory. Fix word penalty in compact phrase table
2013-04-26 15:55:32 +01:00
Hieu Hoang
2f78fe5fe5
Merge github.com:moses-smt/mosesdecoder into weight-new
2013-03-13 17:54:03 +00:00
Hieu Hoang
21c51194fa
add -print-alignment-info
2013-03-13 12:12:33 +00:00
Hieu Hoang
bcecfe55db
merge with Lane's search graph output
2013-02-24 00:31:29 +00:00
Hieu Hoang
30e0d1e0fa
Merge github.com:moses-smt/mosesdecoder into weight-new
2013-02-22 22:39:24 +00:00
Lane Schwartz
764ce06726
More work on outputting search graph as hypergraph
2013-02-22 16:28:48 -05:00
Lane Schwartz
04f107fbb0
Add flag to output search graph in Kenneth's hypergraph format.
2013-02-22 16:28:48 -05:00
Lane Schwartz
774ed64f2e
Work to allow output search graph in HTK standard lattice format
2013-02-22 16:28:47 -05:00