mosesdecoder/biconcor
Jeroen Vermeulen b2d821a141 Unify tokenize() into util, and unit-test it.
The duplicate definition works fine in environments where the inline
definition becomes a weak symbol in the object file, but if it gets
generated as a regular definition, the duplicate definition causes link
problems.

In most call sites the return value could easily be made const, which
gives both the reader and the compiler a bit more certainty about the code's
intentions.  In theory this may help performance, but it's mainly for clarity.

The comments are based on reverse-engineering, and the unit tests are based
on the comments.  It's possible that some of what's in there is not essential,
in which case, don't feel bad about changing it!

I left a third identical definition in place, though I updated it with my
changes to avoid creeping divergence, and noted the duplication in a comment.
It would be nice to get rid of this definition as well, but it'd introduce
headers from the main Moses tree into biconcor, which may be against policy.
2015-04-22 09:59:05 +07:00
..
Alignment.cpp Modernize "C" includes in biconcor. 2015-03-28 21:00:30 +07:00
Alignment.h move biconcor to / 2012-05-31 23:32:24 +01:00
base64.cpp beautify 2013-05-29 18:16:15 +01:00
base64.h move biconcor to / 2012-05-31 23:32:24 +01:00
biconcor.cpp beautify 2013-05-29 18:16:15 +01:00
CMakeLists.txt Checking in examples of CMake files. 2013-09-06 13:39:46 -04:00
Jamfile add tool for phrase lookup with biconcor 2014-09-21 06:03:51 +01:00
Mismatch.cpp Modernize "C" includes in biconcor. 2015-03-28 21:00:30 +07:00
Mismatch.h beautify 2013-05-29 18:16:15 +01:00
phrase-lookup.cpp Unify tokenize() into util, and unit-test it. 2015-04-22 09:59:05 +07:00
PhrasePair.cpp beautify 2013-05-29 18:16:15 +01:00
PhrasePair.h refactor parsing of feature functiona args 2013-06-10 18:11:55 +01:00
PhrasePairCollection.cpp Modernize "C" includes in biconcor. 2015-03-28 21:00:30 +07:00
PhrasePairCollection.h extended display options for biconcor 2013-04-05 11:26:00 +01:00
SuffixArray.cpp Modernize "C" includes in biconcor. 2015-03-28 21:00:30 +07:00
SuffixArray.h move biconcor to / 2012-05-31 23:32:24 +01:00
TargetCorpus.cpp Modernize "C" includes in biconcor. 2015-03-28 21:00:30 +07:00
TargetCorpus.h move biconcor to / 2012-05-31 23:32:24 +01:00
Vocabulary.cpp beautify 2013-05-29 18:16:15 +01:00
Vocabulary.h move biconcor to / 2012-05-31 23:32:24 +01:00