Sarah Hoffmann c171d88194 move parsing of token analysis config to analyzer ... Adds a second callback for the analyzer which is responsible for parsing the configuration rules and converting it to whatever format necessary. This way, each analyzer implementation can define its own configuration rules.		2021-10-04 18:31:58 +02:00
..
sanitizers	add unit tests for new sanatizer functions	2021-10-01 12:27:24 +02:00
token_analysis	move parsing of token analysis config to analyzer	2021-10-04 18:31:58 +02:00
__init__.py	introduce tokenizer modules	2021-04-30 11:29:57 +02:00
base.py	unify ICUNameProcessorRules and ICURuleLoader	2021-10-01 12:27:24 +02:00
factory.py	unify ICUNameProcessorRules and ICURuleLoader	2021-10-01 12:27:24 +02:00
icu_rule_loader.py	move parsing of token analysis config to analyzer	2021-10-04 18:31:58 +02:00
icu_tokenizer.py	introduce sanitizer step before token analysis	2021-10-01 12:27:24 +02:00
icu_variants.py	unify ICUNameProcessorRules and ICURuleLoader	2021-10-01 12:27:24 +02:00
legacy_tokenizer.py	unify ICUNameProcessorRules and ICURuleLoader	2021-10-01 12:27:24 +02:00
place_sanitizer.py	introduce sanitizer step before token analysis	2021-10-01 12:27:24 +02:00