Tareq Al-Ahdal
d32a7c1888
initialize an empty dictionary for nested name key
2022-03-18 02:50:33 +08:00
Tareq Al-Ahdal
6be2077d92
Merge branch 'master' into country-names-yaml-configuration
2022-03-18 02:36:12 +08:00
Tareq Al-Ahdal
456d439e97
Reformatting of country keys
2022-03-18 02:23:11 +08:00
Tareq Al-Ahdal
b4bd4ff67d
fix linting error
2022-03-15 19:14:04 +08:00
Sandor Nagy
7e3701b64a
Fix typo in log message on replication initialisation
2022-03-15 07:50:47 +01:00
Tareq Al-Ahdal
165d17f7f7
reintroduce 'name:' prefix to country name keys
2022-03-13 18:58:27 +08:00
Tareq Al-Ahdal
377cf36be3
modify data import logic to load country names from yaml
2022-03-12 15:20:57 +08:00
Sarah Hoffmann
15beeef6ce
do not expand records in select list
...
An expression of the form 'SELECT (func()).*' will be expanded
by Postgresql _before_ execution with the result that the function
will be called as many times as there are fields in the record.
This is not what we want. The function call needs to go into
the FROM clause instead.
2022-03-01 09:34:32 +01:00
Sarah Hoffmann
92bc3cd0a7
fix linting issue
2022-03-01 09:34:32 +01:00
Sarah Hoffmann
4a3bbd0319
adapt housenumber cleanup to new word table structure
2022-03-01 09:34:32 +01:00
Sarah Hoffmann
13ed184efd
housenumber analyzer: avoid creating too many variants
...
Housenumber fields with lots of text are likely bad data. So is
data with many changes from letter to digit. Exclude them from adding
optional spaces.
2022-03-01 09:34:32 +01:00
Sarah Hoffmann
f03a05f6bb
add new analyser for houenumbers
...
This analyser makes spaces optional.
2022-03-01 09:34:32 +01:00
Sarah Hoffmann
a6903651fc
add framework for analysing housenumbers
...
This lays the groundwork for adding variants for housenumbers.
When analysis is enabled, then the 'word' field in the word table
is used as usual, so that variants can be created. There will be
only one analyser allowed which must have the fixed name
'@housenumber'.
2022-03-01 09:34:32 +01:00
Sarah Hoffmann
b8c544cc98
icu: move token deduplication into TokenInfo
...
Puts collection into one common place.
2022-03-01 09:34:32 +01:00
Sarah Hoffmann
243725aae1
icu: move housenumber token computation out of TokenInfo
...
This was the last function to use the cache. There is a more clean
separation of responsibility now.
2022-03-01 09:34:32 +01:00
Sarah Hoffmann
0bb59b2e22
handle unknown analyzer
...
When changing something in the default configuration of the sanatizers
that refers to an analyzer that is not yet loaded, there shouldn't be
any errors.
2022-03-01 09:34:32 +01:00
Sarah Hoffmann
837d44391c
move generation of normalized token form to analyzer
...
This gives the analyzer more flexibility in choosing the normalized
form. In particular, an analyzer creating different variants can choose
the variant that will be used as the canonical form.
2022-03-01 09:34:32 +01:00
Sarah Hoffmann
5425394654
add migration to add new derived_names column
2022-02-24 20:50:33 +01:00
Sarah Hoffmann
f74228830d
bdd: run full import on tests
...
This uncovered a couple of outdated/wrong tests which have been
fixed, too.
2022-02-24 14:27:51 +01:00
Sarah Hoffmann
a3e4e8e5cd
delete unused country name tokens
2022-02-23 09:23:06 +01:00
Sarah Hoffmann
38c3ef3da0
add tests for get_string_list()
...
Renaming test file for sanitizer config because pytest requires
unique names for test files.
2022-02-07 11:22:24 +01:00
Sarah Hoffmann
610f2cc254
sanitizer: move helpers into a configuration class
2022-02-07 10:48:00 +01:00
Sarah Hoffmann
a79a3210e6
implement is-a-name option for housenumbers
2022-02-07 09:27:11 +01:00
Sarah Hoffmann
98432395c3
add migration for upcoming change to tiger tables
2022-01-27 11:48:27 +01:00
Sarah Hoffmann
83d2c440d5
add migration for new interpolation table layout
2022-01-27 11:14:55 +01:00
Sarah Hoffmann
e6d855b954
add migration for new lookup index
2022-01-27 11:14:55 +01:00
Sarah Hoffmann
c170d323d9
add tests for cleaning housenumbers
2022-01-20 23:47:20 +01:00
Sarah Hoffmann
3ce123ab69
do not clean housenumbers in reverse-only mode
2022-01-20 20:21:13 +01:00
Sarah Hoffmann
d8b7a51ab6
add actual removal of housenumber tokens
2022-01-20 20:18:15 +01:00
Sarah Hoffmann
344a2bfc1a
add new command for cleaning word tokens
...
Just pulls outdated housenumbers for the moment.
2022-01-20 20:05:15 +01:00
Sarah Hoffmann
1e5a8561c0
fix linting issues
2022-01-20 16:00:23 +01:00
Sarah Hoffmann
f3c9578bca
complete documentation for new clean-houseunubmers sanatizer
2022-01-20 15:49:32 +01:00
Sarah Hoffmann
3741afa6dc
generalize filter-kind parameter for sanatizers
...
Now behaves the same for tag_analyzer_by_language and
clean_housenumbers. Adds tests.
2022-01-20 15:42:42 +01:00
Sarah Hoffmann
4774e45218
clean_housenumbers: make kinds and delimiters configurable
...
Also adds unit tests for various options.
2022-01-20 12:07:12 +01:00
Sarah Hoffmann
206ee87188
factor out housenumber splitting into sanitizer
2022-01-19 17:27:50 +01:00
Sarah Hoffmann
3df560ea38
fix linting error
2022-01-18 11:09:21 +01:00
Sarah Hoffmann
adbaf700cd
move parsing of mutation config to setup phase
2022-01-18 11:09:21 +01:00
Sarah Hoffmann
b453b0ea95
introduce mutation variants to generic token analyser
...
Mutations are regular-expression-based replacements that are applied
after variants have been computed. They are meant to be used for
variations on character level.
Add spelling variations for German umlauts.
2022-01-18 11:09:21 +01:00
Sarah Hoffmann
0192a7af96
move variant configuration reading in separate file
2022-01-18 11:09:21 +01:00
Sarah Hoffmann
630ad38a67
refactor variant production to use generators
2022-01-18 11:09:21 +01:00
Sarah Hoffmann
c3788d765e
add consistent SPDX copyright headers
2022-01-03 16:23:58 +01:00
Sarah Hoffmann
f9b56a8581
correctly match abbreviated addr:street
...
This only works when addr:street is abbreviated and the street
name isn't. It does not work the other way around.
2021-12-08 21:58:43 +01:00
Sarah Hoffmann
7f7d2fd5b3
skip most addr: tags with suffixes
...
Only one addr: tag can be processed currently, so make
sure it is the one without suffixes to not get odd data.
addr:street is the exception because it uses a different
matching mechanism.
2021-12-06 14:55:10 +01:00
Sarah Hoffmann
44cfce1ca4
revert to using full names for street name matching
...
Using partial names turned out to not work well because there are
often similarly named streets next to each other. It also
prevents us from being able to take into account all addr:street:*
tags.
This change gets all the full term tokens for the addr:street tags
from the DB. As they are used for matching only, we can assume that
the term must already be there or there will be no match. This
avoid creating unused full name tags.
2021-12-06 11:38:38 +01:00
Sarah Hoffmann
54d35ddfe9
split cli tests by subcommand and extend coverage
2021-12-02 23:45:48 +01:00
Sarah Hoffmann
7beccb7997
remove unnecessary pass statements
2021-12-02 15:54:24 +01:00
Sarah Hoffmann
14a78f55cd
more unit tests for tokenizers
2021-12-02 15:46:36 +01:00
Sarah Hoffmann
a52ed366e4
add tests for migration
2021-12-01 20:27:40 +01:00
Sarah Hoffmann
810056349f
add migration for inclusive housenumber Tiger index
2021-11-24 12:03:20 +01:00
Sarah Hoffmann
10e979e841
only instantiate indexer once for replication
...
Also makes sure that indexer object exists everywhere were needed.
See #2518 .
2021-11-19 14:48:58 +01:00
