Sarah Hoffmann bc981d0261 fix insertion of special terms and countries into word table ... Special terms need to be prefixed by a space because they are full terms. For countries avoid duplicate entries of word tokens. Adds tests for adding country terms.		2021-06-02 20:22:39 +02:00
..
__init__.py	introduce tokenizer modules	2021-04-30 11:29:57 +02:00
factory.py	do not hide errors when importing tokenizer	2021-05-18 16:28:21 +02:00
legacy_icu_tokenizer.py	add tests for new full name computation with ICU	2021-05-24 10:41:42 +02:00
legacy_tokenizer.py	fix insertion of special terms and countries into word table	2021-06-02 20:22:39 +02:00