Sarah Hoffmann 7beccb7997 remove unnecessary pass statements		2021-12-02 15:54:24 +01:00
..
sanitizers	add documentation for new configuration of ICU tokenizer	2021-10-07 11:55:53 +02:00
token_analysis	reorganize and complete tests around generic token analysis	2021-10-06 17:03:37 +02:00
__init__.py	introduce tokenizer modules	2021-04-30 11:29:57 +02:00
base.py	remove unnecessary pass statements	2021-12-02 15:54:24 +01:00
factory.py	unify ICUNameProcessorRules and ICURuleLoader	2021-10-01 12:27:24 +02:00
icu_rule_loader.py	add tests for sanitizer tagging language	2021-10-06 12:29:25 +02:00
icu_token_analysis.py	use analyser provided in the 'analyzer' property	2021-10-05 14:10:32 +02:00
icu_tokenizer.py	ICU: use normalization from config in PHP	2021-10-27 11:32:44 +02:00
legacy_tokenizer.py	more unit tests for tokenizers	2021-12-02 15:46:36 +01:00
place_sanitizer.py	introduce sanitizer step before token analysis	2021-10-01 12:27:24 +02:00