Commit Graph

7 Commits

Author SHA1 Message Date
Jeroen Vermeulen
32722ab5b1 Support tokenize(const std::string &) as well.
Convenience wrapper: the actual function takes a const char[], but many of
the call sites want to pass a string and have to call its c_str() first.
2015-04-22 10:35:18 +07:00
Jeroen Vermeulen
b2d821a141 Unify tokenize() into util, and unit-test it.
The duplicate definition works fine in environments where the inline
definition becomes a weak symbol in the object file, but if it gets
generated as a regular definition, the duplicate definition causes link
problems.

In most call sites the return value could easily be made const, which
gives both the reader and the compiler a bit more certainty about the code's
intentions.  In theory this may help performance, but it's mainly for clarity.

The comments are based on reverse-engineering, and the unit tests are based
on the comments.  It's possible that some of what's in there is not essential,
in which case, don't feel bad about changing it!

I left a third identical definition in place, though I updated it with my
changes to avoid creeping divergence, and noted the duplication in a comment.
It would be nice to get rid of this definition as well, but it'd introduce
headers from the main Moses tree into biconcor, which may be against policy.
2015-04-22 09:59:05 +07:00
Hieu Hoang
05ead45e71 beautify 2015-01-14 11:07:42 +00:00
Hieu Hoang
1b667e3e24 delete any mention of SAFE_GETLINE so it doesn't reappear 2014-06-08 17:07:12 +01:00
Hieu Hoang
cb94a3181b use standard c++ getline instead of old Moses SAFE_GETLINE 2014-06-08 16:23:14 +01:00
Hieu Hoang
6249432407 beautify 2013-05-29 18:16:15 +01:00
Kenneth Heafield
62d37fa2b6 Refactor phrase-extract/Jamfile 2012-11-12 14:17:48 +00:00