10 Commits

Author	SHA1	Message	Date
Jeroen Vermeulen	b2d821a141	Unify tokenize() into util, and unit-test it. ... The duplicate definition works fine in environments where the inline definition becomes a weak symbol in the object file, but if it gets generated as a regular definition, the duplicate definition causes link problems. In most call sites the return value could easily be made const, which gives both the reader and the compiler a bit more certainty about the code's intentions. In theory this may help performance, but it's mainly for clarity. The comments are based on reverse-engineering, and the unit tests are based on the comments. It's possible that some of what's in there is not essential, in which case, don't feel bad about changing it! I left a third identical definition in place, though I updated it with my changes to avoid creeping divergence, and noted the duplication in a comment. It would be nice to get rid of this definition as well, but it'd introduce headers from the main Moses tree into biconcor, which may be against policy.	2015-04-22 09:59:05 +07:00
Hieu Hoang	05ead45e71	beautify	2015-01-14 11:07:42 +00:00
Hieu Hoang	23ba0de224	use standard c++ getline instead of old Moses SAFE_GETLINE	2014-06-08 15:41:27 +01:00
Hieu Hoang	31ce9b510e	beautify	2013-09-27 09:35:24 +01:00
Hieu Hoang	98bb4fa1c7	placeholders work in extract	2013-09-19 12:24:57 +02:00
Hieu Hoang	6249432407	beautify	2013-05-29 18:16:15 +01:00
Barry Haddow	861792bfc5	extract can read an instance weights file. ... Still have to parallelise.	2012-12-21 15:39:25 +00:00
Hieu Hoang	5dbb0e66ce	option to produce rules that have boundary <s> & </s> words. Like Chris Dyer's extraction	2012-08-23 19:40:09 +01:00
Hieu Hoang	ef9db932aa	add namespace to phrase-extract	2012-06-30 15:43:47 +01:00
Hieu Hoang	4eef94b121	move c++ code out of /script/ to /	2012-05-31 17:24:06 +01:00