Sarah Hoffmann 118858a55e rename legacy_icu tokenizer to icu tokenizer ... The new icu tokenizer is now no longer compatible with the old legacy tokenizer in terms of data structures. Therefore there is also no longer a need to refer to the legacy tokenizer in the name.		2021-08-17 23:11:47 +02:00
..
__init__.py	introduce tokenizer modules	2021-04-30 11:29:57 +02:00
base.py	define formal public Python interface for tokenizer	2021-08-16 11:41:54 +02:00
factory.py	do not hide errors when importing tokenizer	2021-05-18 16:28:21 +02:00
icu_name_processor.py	split up variant computation for better readability	2021-07-12 14:43:50 +02:00
icu_rule_loader.py	switch to a more flexible variant description format	2021-07-04 10:28:20 +02:00
icu_tokenizer.py	rename legacy_icu tokenizer to icu tokenizer	2021-08-17 23:11:47 +02:00
icu_variants.py	remove default parameter for namedtuple	2021-07-06 22:57:42 +02:00
legacy_tokenizer.py	define formal public Python interface for tokenizer	2021-08-16 11:41:54 +02:00