Sarah Hoffmann 32ca631b74 fix full term token in special phrases		2021-07-04 10:28:20 +02:00
..
__init__.py	introduce tokenizer modules	2021-04-30 11:29:57 +02:00
factory.py	do not hide errors when importing tokenizer	2021-05-18 16:28:21 +02:00
icu_name_processor.py	update unit tests for adapted abbreviation code	2021-07-04 10:28:20 +02:00
icu_rule_loader.py	correctly quote strings when copying in data	2021-07-04 10:28:20 +02:00
legacy_icu_tokenizer.py	fix full term token in special phrases	2021-07-04 10:28:20 +02:00
legacy_tokenizer.py	adapt tests for ICU tokenizer	2021-07-04 10:28:20 +02:00