Sarah Hoffmann 8377528952 new word table layout for icu tokenizer ... The table now directly reflects the different token types. Extra information is saved in a json structure that may be dynamically extended in the future without affecting the table layout.		2021-07-28 11:31:47 +02:00
..
__init__.py	introduce tokenizer modules	2021-04-30 11:29:57 +02:00
factory.py	do not hide errors when importing tokenizer	2021-05-18 16:28:21 +02:00
icu_name_processor.py	split up variant computation for better readability	2021-07-12 14:43:50 +02:00
icu_rule_loader.py	switch to a more flexible variant description format	2021-07-04 10:28:20 +02:00
icu_variants.py	remove default parameter for namedtuple	2021-07-06 22:57:42 +02:00
legacy_icu_tokenizer.py	new word table layout for icu tokenizer	2021-07-28 11:31:47 +02:00
legacy_tokenizer.py	add helper function for execute_values	2021-07-12 21:08:20 +02:00