* drop global lmodel-oov-feature option, and add it to LM FF config line instead
use oov-feature=1 (bool) option instead
* drop LanguageModel::GetWeight()
* KenLM: use m_verbosity of FF instead of IFVERBOSE macro which uses StaticData
* train-model.perl: move language model OOV feature onto LM feature spec line
This adds a new string-to-tree decoder, which can be enabled with the -s2t
option. It's intended to be faster and simpler than the generic chart
decoder, and is designed to support lattice input (still WIP). For a en-de
system trained on WMT14 data, it's approximately 40% faster in practice.
For background information, see the decoding section of the EMNLP tutorial
on syntax-based MT:
http://www.emnlp2014.org/tutorials/5_notes.pdf
Some features are not implemented yet, including support for internal tree
structure and soft source-syntactic constraints.
This commit updates that code to take into account Hieu's major refactoring.
Additionally, this commit re-instates changes from
commit 8459a86 (moses/LM/Jamfile, moses/LM/Ken.h, moses/LM/Ken.cpp),
commit d340218 (moses/LM/Ken.h),
commit 95e3a37 (moses/LM/Ken.h, moses/LM/Ken.cpp), and
commit aacc936 (moses/LM/Jamfile, moses/LM/Ken.cpp)
The above changes appear to have been improperly discarded
during the merge in commit 4894df6 and then again in commit 3ed17bb.
