osm-search/Nominatim
1
1
Fork 0
mirror of https://github.com/osm-search/Nominatim.git synced 2024-11-29 08:36:24 +03:00
Code Issues Packages Projects Releases Wiki Activity
4,785 Commits 28 Branches 41 Tags 164 MiB
882f182374
Commit Graph

899 Commits

Author SHA1 Message Date
Sarah Hoffmann
00e3a752c9 split SearchResult type 
Use adapted types for the different result types. This makes it
easier to have adapted output formatting and means there are only
result fields that are filled.
 2023-03-23 10:16:50 +01:00
Sarah Hoffmann
81430bd3bd bdd: be more fuzzy with coordinate comparisons 2023-03-09 22:37:45 +01:00
Sarah Hoffmann
93203f355a avoid recent Python dialect 2023-03-09 20:57:43 +01:00
Sarah Hoffmann
3f2296e3ea bdd: extend reverse API tests for format checks 
Reorganise the API reverse tests and extend the checks for the
output format, testing for all expected fields.
 2023-03-09 20:20:50 +01:00
Sarah Hoffmann
2b7eb4906a bdd: add tests for valid debug output 2023-03-09 20:10:51 +01:00
Sarah Hoffmann
db1aa4d02e bdd: replace old formatting strings 2023-03-09 19:49:55 +01:00
Sarah Hoffmann
ad88d7a3e0 bdd: more format checks for reverse XML 2023-03-09 19:40:24 +01:00
Sarah Hoffmann
e42c1c9c7a bdd: new step variant 'result contains in field' 
This replaces the + notation for recursing into result dictionaries.
 2023-03-09 19:31:21 +01:00
Sarah Hoffmann
556bb2386d bdd: factor out computation of result to-check lists 2023-03-09 18:01:45 +01:00
Sarah Hoffmann
1e58cef174 bdd: replace property_list construct with standard check functions 2023-03-09 17:56:28 +01:00
Sarah Hoffmann
01010e443f bdd: remove special case for osm_type field 
The fuzzy field check hide cover formatting errors. Use 'osm' when
only caring about the conent.
 2023-03-09 17:44:34 +01:00
Sarah Hoffmann
da0a7a765e bdd: reorganise field comparisons 
Move comparision on Field values from assert_field() into a
comparator class. Replace BadRowValueAssert with a simpler
check_row() function.
 2023-03-09 17:05:05 +01:00
Sarah Hoffmann
9769a0dcdb bdd: use new check_for_attributes() function also in steps 2023-03-09 16:44:07 +01:00
Sarah Hoffmann
fbff4fa218 bdd: fully check correctness of geojson and geocodejson 
Parse code now checks presence of all required fields and exports
all fields for inspection.
 2023-03-09 16:36:46 +01:00
Sarah Hoffmann
d17ec56e54 bdd: remove OrderedDict 
dicts are guaranteed to keep insertion order by since Python 3.7, making
use of ORderedDict mute.
 2023-03-09 16:08:39 +01:00
biswajit-k
ca149fb796 Adds sanitizer for preventing certain tags to enter search index based on parameters 
fix: pylint error

added docs for delete tags sanitizer

fixed typos in docs and code comments

fix: python typechecking error

fixed rank address type

Revert "fixed typos in docs and code comments"

This reverts commit 6839eea755a87f557895f30524fb5c03dd983d60.

added default parameters and refactored code

added test for all parameters
 2023-03-09 14:18:39 +05:30
Sarah Hoffmann
412ead5f2d adapt PHP tests for debug output 2023-02-20 16:23:28 +01:00
Sarah Hoffmann
d574ceb598 restrict place rank inheritance to address items 
Place tags must have no influence on street- or POI-level
objects.
 2023-02-17 16:25:26 +01:00
Sarah Hoffmann
ee0c5e24bb add a WKB decoder for the Point class 
This allows to return point geometries from the database and makes
the SQL a bit simpler.
 2023-02-16 17:29:56 +01:00
Sarah Hoffmann
8557105c40 add debug output for unit tests 
This uses the debug output facility meant for pretty HTML output
to give us debugging output for the unit tests.
 2023-02-14 11:57:37 +01:00
Sarah Hoffmann
42c3754dcd add tests for details result formatting and trim results 
Values that are None are no longer included in the output to save
a bit of bandwidth.
 2023-02-04 21:22:22 +01:00
Sarah Hoffmann
b742200442 expand details BDD tests 
There are now minor differences in the output between PHP and
Python versions, so introduce specific tests.
 2023-02-04 21:22:22 +01:00
Sarah Hoffmann
3ac70f7cc2 implement details endpoint in Python servers 2023-02-04 21:22:22 +01:00
Sarah Hoffmann
104722a56a switch details cli command to new Python implementation 2023-02-04 21:22:22 +01:00
Sarah Hoffmann
1924beeb20 add lookup of postcdoe data 2023-02-04 21:22:22 +01:00
Sarah Hoffmann
70f6f9a711 add lookup of tiger data 2023-02-04 21:22:22 +01:00
Sarah Hoffmann
f1ceefe9a6 add lookup of address interpolations 2023-02-04 21:22:22 +01:00
Sarah Hoffmann
189f74a40d add unit tests for lookup function 2023-02-04 21:22:22 +01:00
Sarah Hoffmann
370c9b38c0 improve scaffolding for API unit tests 
Use the static table definition to create the test database.
Add helper function to simplify filling the tables.
 2023-02-04 21:22:22 +01:00
Sarah Hoffmann
16b6484c65 add property cache for API 
This caches results from querying nominatim_properties.
 2023-01-30 09:36:17 +01:00
Sarah Hoffmann
77bec1261e add streaming json writer for JSON output 2023-01-25 15:05:33 +01:00
Sarah Hoffmann
8f4426fbc8 reorganize code around result formatting 
Code is now organized by api version. So formatting has moved to
the api.v1 module. Instead of holding a separate ResultFormatter
object per result format, simply move the functions to the
formater collector and hand in the requested format as a parameter.
Thus reorganized, the api.v1 module can export three simple functions
for result formatting which in turn makes the code that uses
the formatters much simpler.
 2023-01-24 17:20:51 +01:00
Sarah Hoffmann
32c1e59622 reorganize api submodule 
Use a directory for the submodule where the __init__ file contains
the public API. This makes it easier to separate public interface
from the internal implementation.
 2023-01-24 13:28:04 +01:00
Sarah Hoffmann
3cc357bffa
Merge pull request #2955 from lonvia/fix-importance-refresh 
Fix importance recalculation
 2023-01-23 09:07:43 +01:00
Sarah Hoffmann
ce9ed993c8 fix importance recalculation 
The signature of the compute_importance() function has changed.
 2023-01-22 22:32:16 +01:00
Sarah Hoffmann
929a13d4cd remove comma as name separator 
Commas are most of the time used as a part of a name, not to
separate multiple names.

See also #2950.
 2023-01-22 22:29:36 +01:00
Sarah Hoffmann
5f4e98e0d9 update Makefile in test directory 2023-01-09 20:49:33 +01:00
Sarah Hoffmann
a72e2ecb3f update dependencies for Actions 2023-01-03 10:03:00 +01:00
Sarah Hoffmann
0c47558729 convert version to named tuple 
Also return the new NominatimVersion rather than a string in the
status result.
 2023-01-03 10:03:00 +01:00
Sarah Hoffmann
93b9288c30 fix error message for non-existing database 2023-01-03 10:03:00 +01:00
Sarah Hoffmann
9d31a67116 add unit tests for new Python API 2023-01-03 10:03:00 +01:00
Sarah Hoffmann
7219ee6532 extend BDD API tests to query via Python frameworks 
A new config option ENGINE allows to choose between php and any of the
supported Python engines.
 2023-01-03 10:03:00 +01:00
Sarah Hoffmann
018ef5bd53 bdd: recreate project directory for every run 2022-12-23 18:36:41 +01:00
Sarah Hoffmann
200eae3bc0 add tests for examples in lua style documentation 
And fix all the errors the tests have found.
 2022-12-23 17:35:28 +01:00
Sarah Hoffmann
89a34e7508 adapt tests for new lua styles 2022-12-19 17:32:28 +01:00
Sarah Hoffmann
a915815e4d explicit export for functions in flex-base 2022-12-18 10:10:58 +01:00
Sarah Hoffmann
2231401483 clean up uses of cli.nominatim() 
They should not hand in data paths anymore.
 2022-11-27 15:27:04 +01:00
Sarah Hoffmann
2abe9e6fd9 use data paths from new nominatim.paths 2022-11-27 12:15:41 +01:00
Sarah Hoffmann
0ed60d29cb remove NOMINATIM_NOMINATIM_TOOL variable 
This was used by the old PHP scripts to call the Python tool.
With the scripts now gone, the variable can be removed.
 2022-11-26 16:40:20 +01:00
Sarah Hoffmann
41e8bddaa9 remove BDD test for tiger:county 
We no longer rely on the import to strip the tag.
 2022-11-23 10:37:27 +01:00
Sarah Hoffmann
fd3dec8efe add sanitizer for TIGER tags 
Currently only takes over cleaning the tiger:county data. This was
done by the import until now.
 2022-11-23 10:37:27 +01:00
Sarah Hoffmann
c9ff7d2130 drop illegal values for addr:interpolation on update 2022-11-18 17:26:56 +01:00
Sarah Hoffmann
52456230cc
Merge pull request #2887 from lonvia/lookup-linked-places 
Add support for lookup of linked places
 2022-11-17 13:35:53 +01:00
Sarah Hoffmann
c4b13f2b7f add support for lookup of linked places 2022-11-16 21:34:45 +01:00
Sarah Hoffmann
4f05a03d13 handle associatedStreet relations with multiple streets 
When a associatedStreet relation has multiple street members
always take the closest one. Avoid geometry operations for
the frequent case that there is only one street.
 2022-11-16 17:25:51 +01:00
Sarah Hoffmann
93ada250f7 bdd: add tests for osm2pgsql update of postcode nodes 2022-11-14 17:27:04 +01:00
Sarah Hoffmann
d8e3ba3b54 bdd: add osm2pgsql tests for updating interpolations 2022-11-14 16:57:31 +01:00
Sarah Hoffmann
a46348da38 bdd: test placex content when updating with osm2pgsql 2022-11-14 14:48:44 +01:00
Sarah Hoffmann
36cf0eb922 reorganize handling of place type changes 
Always replace existing entries in place, never delete them because
a direct delete will cause conflicts.
 2022-11-14 13:57:26 +01:00
Sarah Hoffmann
63a9bc94f7 fix country handling in flex style 
If the country tag does not match a 2-letter code, it needs to
be dropped.
 2022-11-10 15:52:13 +01:00
Sarah Hoffmann
2dafc4cf4f remove tests that differ between lua and gazetteer versions 2022-11-10 15:51:55 +01:00
Sarah Hoffmann
68d09f9cad node locations must be stable for osm2pgsql update tests 2022-11-10 11:11:45 +01:00
Sarah Hoffmann
b98d3d3f00 bdd: extend osm2pgsql update tests 
Now also checks for correct indexing state of placex table.
 2022-11-10 09:38:25 +01:00
Sarah Hoffmann
2fac507453 change updates to handle delete/insert workflow 
This makes Nominatim compatible with osm2pgsql's default update
modus operandi of deleting and reinserting data. Deletes are diverted
into a TODO table instead of executing them. When data is reinserted,
the corresponding entry in the TODO table is deleted. After updates are
finished, the remaining entries in the TODO table are executed, doing
the same work as the delete trigger did before.

The new behaviour also works against the gazetteer output with its
insert-only mechanism.
 2022-11-10 09:38:23 +01:00
Sarah Hoffmann
51ed55cc32 initial flex import scripts 
Only implements the extratags style for the moment. Tests pass
for the same behaviour as the gazetteer output. Updates still need
to be done.
 2022-11-10 09:37:38 +01:00
Sarah Hoffmann
de2a3bd5f8 bdd tests: make import style configurable 
The switch is for development. Tests are not guaranteed to still
work when run with anything but the 'extratags' style.
 2022-11-10 09:37:38 +01:00
Sarah Hoffmann
981e9700be add osm2pgsql gazetteer tests 
This ports the gazetteer tests from osm2pgsql to BDD tests.
 2022-11-10 09:37:38 +01:00
Sarah Hoffmann
5f6dcd36ed fix flaky API test 
The search 'landstr' produces many duplicates so that with
some bad luck 4 or less results may appear. Disable deduplication
to make it more predictable.
 2022-10-05 15:16:14 +02:00
Sarah Hoffmann
5877b69d51 do not run unit test when postgis_raster is not available 2022-10-01 11:01:49 +02:00
Sarah Hoffmann
5ec2c1b712 adapt unit tests to changed function names 2022-10-01 11:01:49 +02:00
Sarah Hoffmann
0a73ed7d64 add secondary importance to API BDD tests 
Also fixes a path issue during API test DB creation that could
never possibly have worked.
 2022-10-01 11:01:49 +02:00
Tareq Al-Ahdal
0ab0f0ea44 Integrated OSM views into importance computation 2022-10-01 11:01:49 +02:00
Tareq Al-Ahdal
ac467c7a2d Enhanced the implementation of OSM views GeoTIFF import functionality 2022-10-01 11:01:49 +02:00
Tareq Al-Ahdal
c85b74497b Initial implementation of GeoTIFF import functionality 2022-10-01 11:01:49 +02:00
Sarah Hoffmann
f4d3ae6f70 consolidate indexes over geometry_sectors 
The index over geometry_sectors are mainly used for ordering
the places which need indexing. That means they function effectively
as a TODO list. Consolodate them so that they always only contain
the places which are still to do. Also add the appropriate index
for the boundary indexing phase.
 2022-09-21 10:38:58 +02:00
Sarah Hoffmann
dddfa3a075 ignore irrelevant extra tags on address interpolations 
When deciding if an address interpolation has address information, only
look for addr:street and addr:place. If they are not there go looking
for the address on the address nodes. Ignores irrelevant tags like
addr:inclusion.

Fixes #2797.
 2022-08-13 14:07:06 +02:00
Sarah Hoffmann
487e81fe3c more invalidations when boundary changes rank 
When a boundary or place changes its address rank, all places where
it participates as address need to be potentially reindexed.
Also use the computed rank when testing place nodes against
boundaries. Boundaries are computed earlier.

Fixes #2794.
 2022-08-12 09:48:46 +02:00
Sarah Hoffmann
51b6d16dc6 overhaul the token analysis interface 
The functional split betweenthe two functions is now that the
first one creates the ID that is used in the word table and
the second one creates the variants. There no longer is a
requirement that the ID is the normalized version. We might
later reintroduce the requirement that a normalized version be available
but it doesn't necessarily need to be through the ID.

The function that creates the ID now gets the full PlaceName. That way
it might take into account attributes that were set by the sanitizers.

Finally rename both functions to something more sane.
 2022-07-29 15:14:11 +02:00
Sarah Hoffmann
c8873d34af harmonize interface of token analysis module 
The configure() function now receives a Transliterator object instead
of the ICU rules. This harmonizes the parameters with the create
function.
 2022-07-29 10:43:07 +02:00
Sarah Hoffmann
6d41046b15 add support for external sanitizer modules 2022-07-25 16:10:19 +02:00
Sarah Hoffmann
7b7203c149 add function for loading plugin modules 
Loads modules for configurable code like tokenizers, sanitizers, etc.
Supports internal modules, external libraries and code from the
project directory.
 2022-07-25 16:10:10 +02:00
Sarah Hoffmann
cd4bcea894 ignore API parameters in array notation 
PHP automatically parses parameters in an array notation(foo[]) into
array types. Ignore these parameters as 'unknown'.

Fixes #2763.
 2022-07-23 10:51:44 +02:00
Kian-Meng Ang
f5e52e748f docs: fix typos 2022-07-20 22:05:31 +08:00
Sarah Hoffmann
9963261d8d add type annotations to special phrase importer 2022-07-18 09:54:29 +02:00
Sarah Hoffmann
62eedbb8f6 add type hints for sanitizers 2022-07-18 09:47:57 +02:00
Sarah Hoffmann
fc254fc744 adapt use of Connection in bdd tests to name change 2022-07-18 09:47:57 +02:00
Sarah Hoffmann
aaf2b6032e fix uses of config.get_path() to expect None 2022-07-18 09:47:57 +02:00
Sarah Hoffmann
b1903f0fbf
Merge pull request #2761 from lonvia/repair-index-analysis 
Repair `admin --analyse-indexing`
 2022-07-18 09:38:08 +02:00
marc tobias
c70ca7f57b In tests for PHP 8 disable Just-in-time, it conflicts with tools that determine coverage 2022-07-09 22:03:48 +02:00
Sarah Hoffmann
4b12d52ef5 convert admin --analyse-indexing to new indexing method 
A proper run of indexing requires the place information from the
analyzer. Add the pre-processing of place data, so the right
information is handed into the update function.
 2022-07-07 16:20:08 +02:00
Sarah Hoffmann
cbbcbb1fd7 move country_info into data submodule 2022-07-06 11:08:36 +02:00
Sarah Hoffmann
bce93d60bd move PlaceInfo into data submodule 
This data structure is shared between indexer and tokenizer.
 2022-07-06 10:54:47 +02:00
Sarah Hoffmann
69e51aebab test: avoid column names with upper-case letters 
This may cause problems when the column names get quoted.
 2022-07-05 09:12:55 +02:00
Marc Tobias
ccf119206d PHP 8 behaves slightly different with in_array and usort 2022-07-03 10:55:34 +02:00
Sarah Hoffmann
3dd7410bb7 bdd: correctly skip postcode tests for legacy 2022-06-23 23:42:31 +02:00
Sarah Hoffmann
93d5be097a bdd: do not expect legacy word table to be without empty tokens 
It can happen for bogus names and this will not get fixed anymore.
 2022-06-23 23:42:31 +02:00
Sarah Hoffmann
6eb9044353 adapt search algorithm to new postcode format in word 2022-06-23 23:42:31 +02:00
Sarah Hoffmann
612d34930b handle postcodes properly on word table updates 
update_postcodes_from_db() needs to do the full postcode treatment
in order to derive the correct word table entries.
 2022-06-23 23:42:31 +02:00
Sarah Hoffmann
0f00f4968c fix up BDD tests for postcode changes 
Includes smaller code fixes found by the tests.
 2022-06-23 23:42:31 +02:00
Sarah Hoffmann
7b6ec4fc6c add tests for discarding bad postcodes 2022-06-23 23:42:31 +02:00
Sarah Hoffmann
80ea13437d move postcode matcher in a separate file 2022-06-23 23:42:31 +02:00
Sarah Hoffmann
4885fdf0f9 add class for online centroid computation 2022-06-23 23:42:31 +02:00
Sarah Hoffmann
18864afa8a postcodes: introduce a default pattern for countries without postcodes 2022-06-23 23:42:31 +02:00
Sarah Hoffmann
9172696324 postcodes: add support for optional spaces 2022-06-23 23:42:31 +02:00
Sarah Hoffmann
baee6f3de0 postcodes: strip leading country codes 2022-06-23 23:42:31 +02:00
Sarah Hoffmann
28ab2f6048 add postcodes patterns without optional spaces 2022-06-23 23:42:31 +02:00
Sarah Hoffmann
90d4d339db initial postcode cleaner for simple patterns 
Moves postcodes that are either in countries without a postcode
system or don't correspond to the local pattern for postcodes into
a field for a normal address part. Makes them searchable but not as
a special address. This has two consequences: they are no longer a
skippable part of the address and the postcodes cannot be searched
on their own.
 2022-06-23 23:42:31 +02:00
Sarah Hoffmann
8080625747 remove postcodes from countries that don't have them 
The postcodes will only be removed as a 'computed postcode' they
are still searchable for the given object.
 2022-06-23 23:42:31 +02:00
Sarah Hoffmann
d8623d6818 bdd: remove support for scenes 
Only keep support for the special point geometry 'country:xx'.
 2022-06-17 11:54:18 +02:00
Sarah Hoffmann
6c58a4c46c bdd: move query tests from scene to grid description 2022-06-17 11:54:18 +02:00
Sarah Hoffmann
19f67e167c bdd: remove step for scene setup 2022-06-17 11:54:18 +02:00
Sarah Hoffmann
00d8df6fc3 bdd: move update tests from scenes to grid descriptions 2022-06-17 11:54:18 +02:00
Sarah Hoffmann
02068aec7f bdd: move import tests from scenes to grid descriptions 2022-06-17 11:54:18 +02:00
Sarah Hoffmann
3493d317e4 bdd: clear lof buffer after a successful import run 2022-06-17 11:54:18 +02:00
Sarah Hoffmann
a2b486a5b0 bdd: allow to set an origin of the grid 2022-06-17 11:54:18 +02:00
Sarah Hoffmann
df0142678a improve address ordering with mixes of place and admin areas 
Resolves a couple of situations where a mixed use of places areas and
administrative boundaries would result in a hierarchy that did not
properly respect the contains relation.
 2022-06-16 10:44:16 +02:00
Sarah Hoffmann
15cf7dd416 add testcase for #2551 
This test proves that places that are linked need to be reindexed.
 2022-06-05 21:39:17 +02:00
Sarah Hoffmann
cbb4749996 change indexing order for interpolations 
Interpolations are now indexed after rank 30 objects. The housenumber
nodes no longer need information from the interpolations while the
interpolations can make use of precomputed postcodes.
 2022-06-02 15:16:46 +02:00
Sarah Hoffmann
8a0e3e2f3d
Merge pull request #2732 from lonvia/fix-ordering-address-parts 
Fix order when searching for addr:* components
 2022-05-31 20:26:05 +02:00
Sarah Hoffmann
bd0e157b91 fix order when searching for addr:* components 
When matching addr:* components the preference was given to
matches that do not intersect with the place.
 2022-05-31 16:57:37 +02:00
Sarah Hoffmann
46689df668 custom comparison for SpecialPhrase 
Duplicate elemination only works when a custom hash/equal function
is implemented that is based on the members.
 2022-05-30 16:30:41 +02:00
Sarah Hoffmann
e828d0d3f7 move quoting hack to wiki loader 
The bad quotes around the type for special phrases
specifically occure in the Wiki pages, so it should be
removed by the loader and not in the generic SpecialPhrase
object.
 2022-05-30 14:40:33 +02:00
Sarah Hoffmann
cce0e5ea38 convert special phrase loaders to generators 
Generators simplify the code quite a bit compared to the previous
Iterator approach.
 2022-05-30 14:12:46 +02:00
Sarah Hoffmann
042e314589 remove the language parameter in the SPWikiLoader 
Languages must always be configured through config or environment.
Also use monkeypatched environment in tests.
 2022-05-30 10:26:20 +02:00
Sarah Hoffmann
61d813bfef add get_str_list() for config 
Converts a config value written as a comma-sparated list into
a Python list of strings.
 2022-05-29 13:53:50 +02:00
Sarah Hoffmann
1d203fdb3c fix bug with keeping linking on updates 
When moving the finding of linked places to the precomputation stage,
it was also moved before the statement where the linked_place_id was
removed from the linkee. The result was that the current linkee was
excluded when looking for a linked place on updates because it was
still linked to the boundary to be updated.

Fixed by allowing to either keep the linkage or change to an unlinked
place.
 2022-05-23 10:55:10 +02:00
Sarah Hoffmann
f314abcfe1 bdd: restrict imports to four languages 
This mainly restricts the number of country names that are loaded.
 2022-05-11 16:40:53 +02:00
Sarah Hoffmann
e74e577029 bdd: recreate functions on template DB 
Avoids calling function refresh on every scenario. The content won't
change between runs.
 2022-05-11 15:50:22 +02:00
Sarah Hoffmann
aa0ae610c6 avoid calling OSM servers during bdd tests 2022-05-11 15:33:01 +02:00
Sarah Hoffmann
5ff35d9984
Merge pull request #2707 from lonvia/make-icu-tokenizer-the-default 
Make ICU tokenizer the default
 2022-05-11 08:52:49 +02:00
Marc Tobias
99fa23040a PHPUnit 9 changed configuration schema slightly 2022-05-10 15:20:43 +02:00
Sarah Hoffmann
adeebec32a switch tests to ICU tokenizer as default 2022-05-10 14:54:50 +02:00
Sarah Hoffmann
ed6fda6968
Merge pull request #2702 from lonvia/move-country-names-into-includes 
Clean up country name settings
 2022-05-10 09:21:16 +02:00
Marc Tobias
821dabb138 add git commit hash to --version output 2022-05-09 23:56:13 +02:00
Sarah Hoffmann
9d468f6da0 support arbitrary prefixes in country name list 
This means we can now get rid of the last special cases for names.
 2022-05-09 11:55:26 +02:00
Marc Tobias
0de83c4a51 fix typos of name Nominatim 2022-05-05 01:04:47 +02:00
Marc Tobias
a79ab41782 new nominatim --version CLI argument 2022-05-04 01:33:25 +02:00
Sarah Hoffmann
372874e89a accept any OSM type in street member of associatedStreet 
This is needed for pedestrian areas mapped as multipolygons
and consequently as relations. The lookup in placex guarantees
that the referenced OSM object is indeed a street.

Fixes #2669.
 2022-05-02 09:48:51 +02:00
Sarah Hoffmann
3c68b12176 keep inherited address parts after indexing 
The inherited housenumber is needed for display output. We can't
take the one from the housenumber field because it is already
normalized. Remove the inherited address only when reindexing.

Fixes #2683.
 2022-04-28 21:38:00 +02:00
Sarah Hoffmann
4f59644cc2 add tests for new data invalidation functions 2022-04-14 14:52:13 +02:00
Artem Ziablytskyi
d1479072ae fix bdd tests and docs 2022-04-07 16:37:51 +02:00
Artem Ziablytskyi
9a56e53d50 use ISO3166-2-lvl<admin_level> instead of typeLabel prefix 2022-04-07 16:37:51 +02:00
Artem Ziablytskyi
6bee188f24 Change the key to <addresspart_type>-ISO3166-2 to support xml response correctly 2022-04-07 16:37:51 +02:00
Artem Ziablytskyi
82dbcbb12a add <addresspart_type>:ISO3166-2 field to response address details 2022-04-07 16:37:51 +02:00
Artem Ziablytskyi
76c146f326 add state_code field to response address details 2022-04-07 16:37:51 +02:00
Sarah Hoffmann
fd4ab3f262
Merge pull request #2629 from tareqpi/country-names-yaml-configuration 
Move default country names into yaml configuration
 2022-04-04 09:04:25 +02:00
Tareq Al-Ahdal
e9f979b67b 'read_config' is no longer a fixture 
add 'read_config' to test cases that need it
 2022-04-01 22:52:17 +08:00
Tareq Al-Ahdal
a323b8f63a test for loading special characters from country_settings.yaml 2022-04-01 21:58:57 +08:00
Tareq Al-Ahdal
9411c14fd2 fix reset country info before loading custom data 2022-04-01 21:55:34 +08:00
Tareq Al-Ahdal
8525e7542f custom country config loads correctly 2022-04-01 21:46:56 +08:00
Sarah Hoffmann
de18cd1523 add test for new table_has_column function 2022-03-31 15:55:20 +02:00
Sarah Hoffmann
36a1560117 add migration to mark internal country names 2022-03-31 15:55:20 +02:00
Tareq Al-Ahdal
b5f311d6bc separate unit test function into three functions 2022-03-30 22:06:59 +08:00
Tareq Al-Ahdal
9db13aac72 Added unit tests for loading country info from yaml file 2022-03-25 22:22:44 +08:00
Sarah Hoffmann
a0ed80d821 restore the tokenizer directory when missing 
Automatically repopulate the tokenizer/ directory with the PHP stub
and the postgresql module, when the directory is missing. This allows
to switch working directories and in particular run the service
from a different maschine then where it was installed.
Users still need to make sure that .env files are set up correctly
or they will shoot themselves in the foot.

See #2515.
 2022-03-20 11:31:42 +01:00
Tareq Al-Ahdal
943e5fe699 Revert the removal of new line at the end of the file 2022-03-18 06:07:48 +08:00
Tareq Al-Ahdal
83b4b8d9c1 reattach 'name:' prefix to keys 2022-03-18 05:46:23 +08:00
Tareq Al-Ahdal
d0c1b73fb3 remove duplicate values 2022-03-18 02:43:42 +08:00
Tareq Al-Ahdal
6be2077d92
Merge branch 'master' into country-names-yaml-configuration 2022-03-18 02:36:12 +08:00
Tareq Al-Ahdal
456d439e97 Reformatting of country keys 2022-03-18 02:23:11 +08:00
Sarah Hoffmann
23de4c7aca adapt ParameterParser tests to new key list 2022-03-17 11:45:05 +01:00
Sarah Hoffmann
e133476c35 merge linked names correctly into namedetails 
Convert the '_place_*' entries back to normal entries before
returning them in the 'namedetails' section. If the name field is
duplicated, kept the '_place_*' notation. This preserves the previous
behaviour before _place_ names were introduces but adds the additional
names from the linked place for reference.
 2022-03-17 11:02:02 +01:00
Sarah Hoffmann
524dc64ab7 make sure outputs take into account linked place names 2022-03-16 21:44:52 +01:00
Sarah Hoffmann
42cd021d04 save differing linked polace names in extra fields 
This keeps the names tracable and ensures that all names are searchable
when they differ. Do not keep names when they are exactly the same
to save some space. Linked names are cleaned out before relinking.
 2022-03-16 16:38:52 +01:00
Sarah Hoffmann
ef98a85b05 correctly handle single-point interpolations in reverse 
Lookup in location_property_osmline needs to be special cased
for startnumber = endnumber. Also adds tests for the case.

Fixes #2680.
 2022-03-16 11:19:09 +01:00
Sarah Hoffmann
0a9f971e44 add tests for new analyzed housenumbers 2022-03-01 09:34:32 +01:00
Sarah Hoffmann
89e1446131 bdd: disable some housenumber tests for legacy 
Optional spaces in housenumbers are not supported by legacy tokenizer,
so disable those tests.
 2022-03-01 09:34:32 +01:00
Sarah Hoffmann
f03a05f6bb add new analyser for houenumbers 
This analyser makes spaces optional.
 2022-03-01 09:34:32 +01:00
Sarah Hoffmann
837d44391c move generation of normalized token form to analyzer 
This gives the analyzer more flexibility in choosing the normalized
form. In particular, an analyzer creating different variants can choose
the variant that will be used as the canonical form.
 2022-03-01 09:34:32 +01:00
Sarah Hoffmann
1d82569f6d add tests for country updates 2022-02-24 16:18:49 +01:00
Sarah Hoffmann
f74228830d bdd: run full import on tests 
This uncovered a couple of outdated/wrong tests which have been
fixed, too.
 2022-02-24 14:27:51 +01:00
Sarah Hoffmann
0e11ca9b76 add test that interpolations are found by odd/even 2022-02-10 11:23:51 +01:00
Sarah Hoffmann
a6b4e8ff67 add tests for housenumber-as-name feature 2022-02-07 11:45:12 +01:00
Sarah Hoffmann
38c3ef3da0 add tests for get_string_list() 
Renaming test file for sanitizer config because pytest requires
unique names for test files.
 2022-02-07 11:22:24 +01:00
Sarah Hoffmann
610f2cc254 sanitizer: move helpers into a configuration class 2022-02-07 10:48:00 +01:00
Sarah Hoffmann
a79a3210e6 implement is-a-name option for housenumbers 2022-02-07 09:27:11 +01:00
Sarah Hoffmann
b6fa121f53 remove tests for closest housenumber function 2022-01-27 16:21:45 +01:00
Sarah Hoffmann
64abc90d30 use new tiger step column for queries 2022-01-27 14:08:08 +01:00
Sarah Hoffmann
6b89624f33 adapt frontend to new interpolation table layout 2022-01-27 11:14:55 +01:00
Sarah Hoffmann
4b28b4fed4 adapt BDD tests for new interpolation style 2022-01-27 11:14:55 +01:00
Sarah Hoffmann
c170d323d9 add tests for cleaning housenumbers 2022-01-20 23:47:20 +01:00
Sarah Hoffmann
d09db09849 adapt ICU tets to new housenumber sanitizer 
Restrict tests to making sure that handing in multiple housenumbers
works.
 2022-01-20 16:05:49 +01:00
Sarah Hoffmann
3741afa6dc generalize filter-kind parameter for sanatizers 
Now behaves the same for tag_analyzer_by_language and
clean_housenumbers. Adds tests.
 2022-01-20 15:42:42 +01:00
Sarah Hoffmann
560a006892 add pytest config 
We are using custom marks now which need to be registered to avoid
warnings.
 2022-01-20 15:38:02 +01:00
Sarah Hoffmann
4774e45218 clean_housenumbers: make kinds and delimiters configurable 
Also adds unit tests for various options.
 2022-01-20 12:07:12 +01:00
Sarah Hoffmann
206ee87188 factor out housenumber splitting into sanitizer 2022-01-19 17:27:50 +01:00
Sarah Hoffmann
b453b0ea95 introduce mutation variants to generic token analyser 
Mutations are regular-expression-based replacements that are applied
after variants have been computed. They are meant to be used for
variations on character level.

Add spelling variations for German umlauts.
 2022-01-18 11:09:21 +01:00
Sarah Hoffmann
c3788d765e add consistent SPDX copyright headers 2022-01-03 16:23:58 +01:00
Sarah Hoffmann
ab6f35d83a
Merge pull request #2553 from lonvia/revert-street-matching-to-full-names 
Revert street matching to full names
 2021-12-14 15:52:34 +01:00
Sarah Hoffmann
f9b56a8581 correctly match abbreviated addr:street 
This only works when addr:street is abbreviated and the street
name isn't. It does not work the other way around.
 2021-12-08 21:58:43 +01:00
Sarah Hoffmann
04857d32cd enable PHPUnit 9 for coverage 
A couple of functions have been renamed.
 2021-12-07 12:07:17 +01:00
Sarah Hoffmann
109cdce92c php unit: replace deprecated regex assert 
The regEx assertion has been renamed in PHPUnit 9.5
and causes deprecation warnings.
 2021-12-07 11:34:21 +01:00
Sarah Hoffmann
b7554d9ed8 php unit: don't enforce a name on the test database 
Also gets rid of a PHPUnit deprecation warning.
 2021-12-07 11:31:45 +01:00
Sarah Hoffmann
6106f1a32e php test: class must be called like the file 2021-12-07 11:20:38 +01:00
Sarah Hoffmann
7f7d2fd5b3 skip most addr: tags with suffixes 
Only one addr: tag can be processed currently, so make
sure it is the one without suffixes to not get odd data.
addr:street is the exception because it uses a different
matching mechanism.
 2021-12-06 14:55:10 +01:00
Sarah Hoffmann
5e435b41ba ICU: matching any street name will do again 2021-12-06 14:26:08 +01:00
Sarah Hoffmann
44cfce1ca4 revert to using full names for street name matching 
Using partial names turned out to not work well because there are
often similarly named streets next to each other. It also
prevents us from being able to take into account all addr:street:*
tags.

This change gets all the full term tokens for the addr:street tags
from the DB. As they are used for matching only, we can assume that
the term must already be there or there will be no match. This
avoid creating unused full name tags.
 2021-12-06 11:38:38 +01:00
Sarah Hoffmann
5a9fb6eaf7 specify text type in test SQL 
Older version of postgres fail otherwise.
 2021-12-03 13:56:23 +01:00
Sarah Hoffmann
54d35ddfe9 split cli tests by subcommand and extend coverage 2021-12-02 23:45:48 +01:00
Sarah Hoffmann
14a78f55cd more unit tests for tokenizers 2021-12-02 15:46:36 +01:00