Jeroen Vermeulen
32722ab5b1
Support tokenize(const std::string &) as well.
...
Convenience wrapper: the actual function takes a const char[], but many of
the call sites want to pass a string and have to call its c_str() first.
2015-04-22 10:35:18 +07:00
Jeroen Vermeulen
b2d821a141
Unify tokenize() into util, and unit-test it.
...
The duplicate definition works fine in environments where the inline
definition becomes a weak symbol in the object file, but if it gets
generated as a regular definition, the duplicate definition causes link
problems.
In most call sites the return value could easily be made const, which
gives both the reader and the compiler a bit more certainty about the code's
intentions. In theory this may help performance, but it's mainly for clarity.
The comments are based on reverse-engineering, and the unit tests are based
on the comments. It's possible that some of what's in there is not essential,
in which case, don't feel bad about changing it!
I left a third identical definition in place, though I updated it with my
changes to avoid creeping divergence, and noted the duplication in a comment.
It would be nice to get rid of this definition as well, but it'd introduce
headers from the main Moses tree into biconcor, which may be against policy.
2015-04-22 09:59:05 +07:00
Nicola Bertoldi
e4eb201c52
merged master into dynamic-models and solved conflicts
2014-12-13 12:52:47 +01:00
Hieu Hoang
33ed15ef19
move misc common functions into moses/
2014-09-30 14:22:38 +01:00
Hieu Hoang
efa5befb16
method rename
2014-08-08 15:59:34 +01:00
Rico Sennrich
bdf8d1a405
CreateFromString no longer requires factorDelimiter
2014-07-04 10:28:24 +01:00
Hieu Hoang
d68257c34d
use standard c++ getline instead of old Moses SAFE_GETLINE
2014-06-08 16:37:12 +01:00
Nicola Bertoldi
bd83999264
beautify
2014-01-15 16:42:02 +01:00
Rico Sennrich
58ed19ce6a
remove hacks for phrase penalty from MultiModel phrase type.
2014-01-02 12:29:38 +01:00
Hieu Hoang
f13d0c8244
UTIL_THROW_IF -> UTIL_THROW_IF2
2013-11-22 20:27:46 +00:00
Hieu Hoang
16cf3bb4c7
replace CHECK with UTIL_THROW_IF in Moses
2013-11-20 17:19:57 +00:00
Hieu Hoang
4cc4913aa6
remove description argument from feature function constructor. Redundant
2013-10-29 18:09:42 +00:00
Hieu Hoang
de9decc30d
mark GetTargetPhraseCollection with LEGACY
2013-08-23 21:02:03 +01:00
Hieu Hoang
88f888f534
const TargetPhrase in TargetPhraseCollection
2013-08-08 14:38:41 +01:00
Rico Sennrich
b32366ab8c
fix future and total cost in multimodel(counts). (was broken since merge of branch weight-new in May)
2013-07-31 14:18:18 +02:00
Rico Sennrich
7b6239b663
multimodelcounts: use Word objects instead of strings in map (avoid costly conversion and string comparison)
2013-07-30 15:03:25 +02:00
Rico Sennrich
ccdcecc86f
multimodel and mosesserver: instead of optimizing first model, select model by name.
2013-07-30 13:54:50 +02:00
Hieu Hoang
e1a86bd9c9
sometimes target phrase collection can have 0 phrases
2013-07-16 16:30:54 +01:00
Hieu Hoang
d68277d369
redo parsing of feature function parameters
2013-06-20 13:25:02 +01:00
Hieu Hoang
24d2edd790
redo parsing of feature function parameters
2013-06-20 13:06:03 +01:00
Hieu Hoang
dc33fa3d3d
redo parsing of feature function parameters
2013-06-20 12:50:41 +01:00
Hieu Hoang
c2f4c48758
clean up SetParameter()
2013-06-19 00:24:21 +01:00
Hieu Hoang
f7371cf53d
calculate m_featuresToApply in Load() for PhraseDictionary. Since every phrase dictionary now has to implement it, make Load() for PhraseDictionary abstract
2013-06-14 18:34:47 +01:00
Hieu Hoang
61e2b7a24f
refactor parsing of feature functiona args
2013-06-11 01:46:04 +01:00
Hieu Hoang
b4ad093334
refactor parsing of feature functiona args
2013-06-11 00:05:12 +01:00
Hieu Hoang
efa42b451c
refactor parsing of feature functiona args
2013-06-10 21:24:31 +01:00
Hieu Hoang
abe6bb7c22
refactor parsing of feature functiona args
2013-06-10 18:11:55 +01:00
Hieu Hoang
96d116dba0
refactor parsing of feature functiona args
2013-06-10 16:29:07 +01:00
Hieu Hoang
eaf53769ad
consistent Load() for all feature functions. Can be used not to load if doing --show-weights
2013-05-31 20:21:02 +01:00
Hieu Hoang
117eb76b0a
figure out which feature function to apply at which decode step. Book-keeping
2013-05-30 18:34:10 +01:00
Hieu Hoang
6249432407
beautify
2013-05-29 18:16:15 +01:00
Hieu Hoang
ae82d02a05
delete sourcePhrase variable in TargetPhrase class
2013-05-28 00:41:25 +01:00
Hieu Hoang
6480805635
technical memory leak due to not cleaning up StaticData. Doesn't make much difference, but annoying messages from valgrind
2013-05-22 17:32:06 +01:00
Hieu Hoang
541bf99b9b
Make Phrase::CreateFromString() more efficient
2013-05-22 14:41:28 +01:00
Rico Sennrich
876de13eba
restore support for multimodel(counts) in moses server:
...
set sentence-level weights and optimize weights
2013-05-16 13:40:17 +02:00
Hieu Hoang
3876c47d1d
port PhraseDictionaryMultiModelCounts to new format. Runs but doesn't pass regression test
2013-05-15 15:52:08 +01:00
Hieu Hoang
b2488f83c3
port PhraseDictionaryMultiModelCounts to new format. Runs but doesn't pass regression test
2013-05-15 14:34:31 +01:00
Hieu Hoang
81ffe84dbc
port PhraseDictionaryMultiModelCounts to new format. Runs but doesn't pass regression test
2013-05-14 17:35:22 +01:00
Hieu Hoang
66ad7c3c18
port PhraseDictionaryMultiModelCounts to new format
2013-05-14 15:16:09 +01:00
Hieu Hoang
0c3fa9942b
port PhraseDictionaryMultiModelCounts to new format
2013-05-14 13:11:55 +01:00
Hieu Hoang
d5ebe61791
port PhraseDictionaryMultiModelCount to new format
2013-05-13 18:40:12 +01:00
Hieu Hoang
b8efef6667
port PhraseDictionaryMultiModelCount to new format
2013-05-13 18:39:06 +01:00
Hieu Hoang
dcafe74892
port PhraseDictionaryMultiModelCount to new format
2013-05-13 17:20:14 +01:00
