mosesdecoder/moses/FeatureFunction.cpp

183 lines
4.7 KiB
C++
Raw Normal View History

#include <stdexcept>
Feature function overhaul. Each feature function is computed in one of three ways: 1) Stateless feature functions from the phrase table/generation table: these are computed when the TranslationOption is created. They become part of the ScoreBreakdown object contained in the TranslationOption and are added to the feature value vector when a hypothesis is extended. 2) Stateless feature functions that are computed during state exploration. Currently, only WordPenalty falls into this category, but these functions implement a method Evaluate which do does not receive a Hypothesis or any contextual information. 3) Stateful feature functions: these features receive the arc information (translation option), compute some value and then return some context information. The context information created by a particular feature function is passed back to it as the previous context when a hypothesis originating at the node where the previous edge terminates is created. States in the search space may be recombined if the context information is identical. The context information must be stored in an object implementing the FFState interface. TODO: 1) the command line interface / MERT interface needs to go to named parameters that are otherwise opaque 2) StatefulFeatureFunction's Evaluate method should just take a TranslationOption and a context object. It is not good that it takes a hypothesis, because then people may be tempted to access information about the "previous" hypothesis without "declaring" this dependency. 3) Future cost estimates should be handled using feature functions. All stateful feature functions need some kind of future cost estimate. 4) Philipp's poor-man's cube pruning is broken. git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2087 1f5c12ca-751b-0410-a591-d2e778427230
2009-02-06 18:43:06 +03:00
#include "util/check.hh"
Feature function overhaul. Each feature function is computed in one of three ways: 1) Stateless feature functions from the phrase table/generation table: these are computed when the TranslationOption is created. They become part of the ScoreBreakdown object contained in the TranslationOption and are added to the feature value vector when a hypothesis is extended. 2) Stateless feature functions that are computed during state exploration. Currently, only WordPenalty falls into this category, but these functions implement a method Evaluate which do does not receive a Hypothesis or any contextual information. 3) Stateful feature functions: these features receive the arc information (translation option), compute some value and then return some context information. The context information created by a particular feature function is passed back to it as the previous context when a hypothesis originating at the node where the previous edge terminates is created. States in the search space may be recombined if the context information is identical. The context information must be stored in an object implementing the FFState interface. TODO: 1) the command line interface / MERT interface needs to go to named parameters that are otherwise opaque 2) StatefulFeatureFunction's Evaluate method should just take a TranslationOption and a context object. It is not good that it takes a hypothesis, because then people may be tempted to access information about the "previous" hypothesis without "declaring" this dependency. 3) Future cost estimates should be handled using feature functions. All stateful feature functions need some kind of future cost estimate. 4) Philipp's poor-man's cube pruning is broken. git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2087 1f5c12ca-751b-0410-a591-d2e778427230
2009-02-06 18:43:06 +03:00
2012-09-21 14:56:01 +04:00
#include "ChartHypothesis.h"
#include "ChartManager.h"
#include "FeatureFunction.h"
#include "Hypothesis.h"
#include "Manager.h"
#include "TranslationOption.h"
2013-02-22 00:03:35 +04:00
using namespace std;
namespace Moses
{
Feature function overhaul. Each feature function is computed in one of three ways: 1) Stateless feature functions from the phrase table/generation table: these are computed when the TranslationOption is created. They become part of the ScoreBreakdown object contained in the TranslationOption and are added to the feature value vector when a hypothesis is extended. 2) Stateless feature functions that are computed during state exploration. Currently, only WordPenalty falls into this category, but these functions implement a method Evaluate which do does not receive a Hypothesis or any contextual information. 3) Stateful feature functions: these features receive the arc information (translation option), compute some value and then return some context information. The context information created by a particular feature function is passed back to it as the previous context when a hypothesis originating at the node where the previous edge terminates is created. States in the search space may be recombined if the context information is identical. The context information must be stored in an object implementing the FFState interface. TODO: 1) the command line interface / MERT interface needs to go to named parameters that are otherwise opaque 2) StatefulFeatureFunction's Evaluate method should just take a TranslationOption and a context object. It is not good that it takes a hypothesis, because then people may be tempted to access information about the "previous" hypothesis without "declaring" this dependency. 3) Future cost estimates should be handled using feature functions. All stateful feature functions need some kind of future cost estimate. 4) Philipp's poor-man's cube pruning is broken. git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2087 1f5c12ca-751b-0410-a591-d2e778427230
2009-02-06 18:43:06 +03:00
PhraseBasedFeatureContext::PhraseBasedFeatureContext(const Hypothesis* hypothesis) :
m_hypothesis(hypothesis),
m_translationOption(m_hypothesis->GetTranslationOption()),
m_source(m_hypothesis->GetManager().GetSource()) {}
PhraseBasedFeatureContext::PhraseBasedFeatureContext
(const TranslationOption& translationOption, const InputType& source) :
m_hypothesis(NULL),
m_translationOption(translationOption),
m_source(source) {}
const TranslationOption& PhraseBasedFeatureContext::GetTranslationOption() const
{
return m_translationOption;
}
const InputType& PhraseBasedFeatureContext::GetSource() const
{
return m_source;
}
const TargetPhrase& PhraseBasedFeatureContext::GetTargetPhrase() const
{
return m_translationOption.GetTargetPhrase();
}
const WordsBitmap& PhraseBasedFeatureContext::GetWordsBitmap() const
{
if (!m_hypothesis) {
throw std::logic_error("Coverage vector not available during pre-calculation");
}
return m_hypothesis->GetWordsBitmap();
}
2012-09-21 14:56:01 +04:00
ChartBasedFeatureContext::ChartBasedFeatureContext
(const ChartHypothesis* hypothesis):
m_hypothesis(hypothesis),
m_targetPhrase(hypothesis->GetCurrTargetPhrase()),
m_source(hypothesis->GetManager().GetSource()) {}
ChartBasedFeatureContext::ChartBasedFeatureContext(
const TargetPhrase& targetPhrase,
const InputType& source):
m_hypothesis(NULL),
m_targetPhrase(targetPhrase),
m_source(source) {}
const InputType& ChartBasedFeatureContext::GetSource() const
{
return m_source;
}
const TargetPhrase& ChartBasedFeatureContext::GetTargetPhrase() const
{
return m_targetPhrase;
}
2013-02-22 00:03:35 +04:00
multiset<string> FeatureFunction::description_counts;
2013-01-18 22:22:06 +04:00
std::vector<FeatureFunction*> FeatureFunction::m_producers;
std::vector<const StatelessFeatureFunction*> StatelessFeatureFunction::m_statelessFFs;
std::vector<const StatefulFeatureFunction*> StatefulFeatureFunction::m_statefulFFs;
FeatureFunction::FeatureFunction(const std::string& description, const std::string &line)
: m_tuneable(true)
{
2013-02-22 00:03:35 +04:00
ParseLine(description, line);
if (m_description == "") {
// not been given a name. Make a unique name
size_t index = description_counts.count(description);
2013-02-22 00:03:35 +04:00
ostringstream dstream;
dstream << description;
dstream << index;
2013-02-22 00:03:35 +04:00
description_counts.insert(description);
m_description = dstream.str();
}
ScoreComponentCollection::RegisterScoreProducer(this);
m_producers.push_back(this);
}
FeatureFunction::FeatureFunction(const std::string& description, size_t numScoreComponents, const std::string &line)
: m_numScoreComponents(numScoreComponents)
, m_tuneable(true)
{
2013-02-22 00:03:35 +04:00
ParseLine(description, line);
if (m_description == "") {
size_t index = description_counts.count(description);
2013-02-22 00:03:35 +04:00
ostringstream dstream;
dstream << description;
dstream << index;
2013-02-22 00:03:35 +04:00
description_counts.insert(description);
m_description = dstream.str();
}
2013-02-22 00:03:35 +04:00
ScoreComponentCollection::RegisterScoreProducer(this);
m_producers.push_back(this);
}
2012-09-21 14:56:01 +04:00
Feature function overhaul. Each feature function is computed in one of three ways: 1) Stateless feature functions from the phrase table/generation table: these are computed when the TranslationOption is created. They become part of the ScoreBreakdown object contained in the TranslationOption and are added to the feature value vector when a hypothesis is extended. 2) Stateless feature functions that are computed during state exploration. Currently, only WordPenalty falls into this category, but these functions implement a method Evaluate which do does not receive a Hypothesis or any contextual information. 3) Stateful feature functions: these features receive the arc information (translation option), compute some value and then return some context information. The context information created by a particular feature function is passed back to it as the previous context when a hypothesis originating at the node where the previous edge terminates is created. States in the search space may be recombined if the context information is identical. The context information must be stored in an object implementing the FFState interface. TODO: 1) the command line interface / MERT interface needs to go to named parameters that are otherwise opaque 2) StatefulFeatureFunction's Evaluate method should just take a TranslationOption and a context object. It is not good that it takes a hypothesis, because then people may be tempted to access information about the "previous" hypothesis without "declaring" this dependency. 3) Future cost estimates should be handled using feature functions. All stateful feature functions need some kind of future cost estimate. 4) Philipp's poor-man's cube pruning is broken. git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2087 1f5c12ca-751b-0410-a591-d2e778427230
2009-02-06 18:43:06 +03:00
FeatureFunction::~FeatureFunction() {}
2013-02-22 00:03:35 +04:00
void FeatureFunction::ParseLine(const std::string& description, const std::string &line)
{
vector<string> toks = Tokenize(line);
CHECK(toks.size());
//CHECK(toks[0] == description);
for (size_t i = 1; i < toks.size(); ++i) {
2013-03-14 23:06:01 +04:00
vector<string> args = Tokenize(toks[i], "=");
2013-02-22 00:03:35 +04:00
CHECK(args.size() == 2);
if (args[0] == "num-features") {
m_numScoreComponents = Scan<size_t>(args[1]);
2013-02-22 00:03:35 +04:00
}
else if (args[0] == "name") {
m_description = args[1];
}
else if (args[0] == "tuneable") {
m_tuneable = Scan<bool>(args[1]);
}
else {
m_args.push_back(args);
}
}
2013-02-22 00:03:35 +04:00
}
2013-02-04 05:09:15 +04:00
StatelessFeatureFunction::StatelessFeatureFunction(const std::string& description, const std::string &line)
:FeatureFunction(description, line)
{
m_statelessFFs.push_back(this);
}
StatelessFeatureFunction::StatelessFeatureFunction(const std::string& description, size_t numScoreComponents, const std::string &line)
:FeatureFunction(description, numScoreComponents, line)
{
m_statelessFFs.push_back(this);
}
StatefulFeatureFunction::StatefulFeatureFunction(const std::string& description, const std::string &line)
: FeatureFunction(description, line)
{
m_statefulFFs.push_back(this);
}
StatefulFeatureFunction::StatefulFeatureFunction(const std::string& description, size_t numScoreComponents, const std::string &line)
: FeatureFunction(description,numScoreComponents, line)
{
m_statefulFFs.push_back(this);
}
bool StatefulFeatureFunction::IsStateless() const
{
return false;
}
Feature function overhaul. Each feature function is computed in one of three ways: 1) Stateless feature functions from the phrase table/generation table: these are computed when the TranslationOption is created. They become part of the ScoreBreakdown object contained in the TranslationOption and are added to the feature value vector when a hypothesis is extended. 2) Stateless feature functions that are computed during state exploration. Currently, only WordPenalty falls into this category, but these functions implement a method Evaluate which do does not receive a Hypothesis or any contextual information. 3) Stateful feature functions: these features receive the arc information (translation option), compute some value and then return some context information. The context information created by a particular feature function is passed back to it as the previous context when a hypothesis originating at the node where the previous edge terminates is created. States in the search space may be recombined if the context information is identical. The context information must be stored in an object implementing the FFState interface. TODO: 1) the command line interface / MERT interface needs to go to named parameters that are otherwise opaque 2) StatefulFeatureFunction's Evaluate method should just take a TranslationOption and a context object. It is not good that it takes a hypothesis, because then people may be tempted to access information about the "previous" hypothesis without "declaring" this dependency. 3) Future cost estimates should be handled using feature functions. All stateful feature functions need some kind of future cost estimate. 4) Philipp's poor-man's cube pruning is broken. git-svn-id: https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk@2087 1f5c12ca-751b-0410-a591-d2e778427230
2009-02-06 18:43:06 +03:00
}