Nominatim/lib-php/tokenizer
Sarah Hoffmann 3cd85eaaf1 remove Token from explicit input for SearchDescription extension
The token string is only required by the PartialToken type, so
it can simply save the token string internally. No need to pass
it to every type.

Also moves the check for multi-word partials to the token loader
code in the tokenizer. Multi-word partials can only happen with
the legacy tokenizer and when the database was loaded with an
older version of Nominatim. No need to keep the check for
everybody.
2021-07-17 18:18:31 +02:00
..
legacy_icu_tokenizer.php remove Token from explicit input for SearchDescription extension 2021-07-17 18:18:31 +02:00
legacy_tokenizer.php remove Token from explicit input for SearchDescription extension 2021-07-17 18:18:31 +02:00