osm-search/Nominatim
1
1
Fork 0
mirror of https://github.com/osm-search/Nominatim.git synced 2024-11-23 13:44:36 +03:00
Code Issues Packages Projects Releases Wiki Activity
143ff14466
Nominatim/lib-php/tokenizer
History
Sarah Hoffmann 143ff14466 remove special status of partial tokens 
Full-word tokens are no longer marked by a space at the
beginning of the token. Use the new Partial token category
instead. This removes a couple of special casing, we don't
really need.

The word table still has the space for compatibility reasons,
so the tokenizer code needs to get rid of it when loading the
tokens.
2021-07-14 22:17:17 +02:00
..
legacy_icu_tokenizer.php remove special status of partial tokens 2021-07-14 22:17:17 +02:00
legacy_tokenizer.php remove special status of partial tokens 2021-07-14 22:17:17 +02:00