osm-search/Nominatim
1
1
Fork 0
mirror of https://github.com/osm-search/Nominatim.git synced 2024-11-23 13:44:36 +03:00
Code Issues Packages Projects Releases Wiki Activity
4,667 Commits 28 Branches 41 Tags 164 MiB
4315debff5
Commit Graph

968 Commits

Author SHA1 Message Date
Sarah Hoffmann
af85ad390f
Merge pull request #3273 from lonvia/search-with-sqlite 
Add forward search capability for SQLite databases
 2023-12-12 12:15:22 +01:00
Sarah Hoffmann
89094cf92e error out when a SQLite database does not exist 
Requires to mark the databse r/w when it is newly created in the
convert function.
 2023-12-07 10:24:53 +01:00
Sarah Hoffmann
3f5484f48f enable search for sqlite conversion by default 2023-12-07 09:33:42 +01:00
Sarah Hoffmann
6d39563b87 enable all API tests for sqlite and port missing features 2023-12-07 09:32:02 +01:00
Sarah Hoffmann
0d840c8d4e extend sqlite converter for search tables 2023-12-07 09:31:00 +01:00
Sarah Hoffmann
381bd0b576 remove unused function 2023-12-07 09:31:00 +01:00
Sarah Hoffmann
b5c61e0b5b improve typing for @compiles constructs 
The first parameter is in fact the self parameter referring to
the function class.
 2023-12-07 09:31:00 +01:00
Sarah Hoffmann
df6eddebcd void unnecessary aliases 2023-12-07 09:31:00 +01:00
Sarah Hoffmann
b6c8c0e72b factor out SQL for filtering by location 
Also improves on the decision if an indexed is used or not.
 2023-12-07 09:31:00 +01:00
Sarah Hoffmann
b06f5fddcb simplify handling of SQL lookup code for search_name 
Use function classes which can be instantiated directly.
 2023-12-07 09:31:00 +01:00
Sarah Hoffmann
8791c6cb69 correctly close API objects during testing 2023-12-07 09:31:00 +01:00
Sarah Hoffmann
615b166c68 clean up ST_DWithin and intersects() functions 
A non-index version of ST_DWithin is not necessary. ST_Distance
can be used for that purpose. Index use for intersects can be
covered with a simple parameter.
 2023-12-07 09:31:00 +01:00
Sarah Hoffmann
c41f2fed21 simplify weigh_search() function 
Use JSON arrays which can have mixed types and therefore have
a more logical structure than separate arrays. Avoid JSON dicts
because of their verboseness.
 2023-12-07 09:31:00 +01:00
Sarah Hoffmann
05e47fbb28 fix parameter formatting in sqlite debug output 2023-12-07 09:31:00 +01:00
Sarah Hoffmann
1b7c8240ba enable connection pools for sqlite 
Connecting is reasonably expensive because the spatialite extension
needs to be loaded. Disable pooling for tests because there is some
memory leak when quickly opening and closing QueuePools with sqlite
connections.
 2023-12-07 09:31:00 +01:00
Sarah Hoffmann
c4fd3ab97f hide type differences between Postgres and Sqlite in custom types 
Also define a custom set of operators in preparation of differences
in implementation.
 2023-12-07 09:31:00 +01:00
Sarah Hoffmann
3969ce0f55
Merge pull request #3268 from mtmail/wikipedia-file-path-warning 
Improve error message when Wikipedia importance file is not found
 2023-12-07 09:21:59 +01:00
Robbe Haesendonck
4f5f5ea8fc Removed unnecessary check for --prepare-database flag 2023-12-07 09:04:33 +01:00
Robbe Haesendonck
53d2050dc5 Fixed typechecking error 2023-12-07 09:04:33 +01:00
Robbe Haesendonck
97ac036df5 Added missing return types to functions 2023-12-07 09:04:33 +01:00
Robbe Haesendonck
567c31ab6a Fixed legacy import command 
Check whether prepare-database is true, if so exit early
 2023-12-07 09:04:33 +01:00
Robbe Haesendonck
7d28fc35d1 Disabled pylint too-many-branches 2023-12-07 09:04:33 +01:00
Robbe Haesendonck
c06f902398 Fixed setting tokenizer property 2023-12-07 09:04:33 +01:00
Robbe Haesendonck
9c7d947fd1 Updated check to see if osm_file is set 2023-12-07 09:04:33 +01:00
Robbe Haesendonck
db917cb0d4 Made sure legacy import command still works 2023-12-07 09:04:33 +01:00
Robbe Haesendonck
ba6cdd875d Removed unused variable, fixed connection 2023-12-07 09:04:33 +01:00
Robbe Haesendonck
d231ff60ed Removed _is_complete_import check 2023-12-07 09:04:33 +01:00
Robbe Haesendonck
c74904d075 Improved logic. 
Fixed small oversight in mutually exclusiveness of arguments
 2023-12-07 09:04:33 +01:00
Robbe Haesendonck
667197a47e Changed naming of flags. 
Made sure legacy import is also still ok
 2023-12-07 09:04:33 +01:00
Robbe Haesendonck
e8b866aa88 Added check to see if hstore is loaded 2023-12-07 09:04:33 +01:00
Robbe Haesendonck
e7b8e1a2c2 Fixed ci-tests, osm-file flag 
Preparing database should work without osm-file
 2023-12-07 09:04:33 +01:00
Robbe Haesendonck
279b4fd6d2 Renamed flags 2023-12-07 09:04:33 +01:00
Robbe Haesendonck
b7c83d3580 Fixed pylint warnings 2023-12-07 09:04:33 +01:00
Robbe Haesendonck
d4018f2e3b Added check for hstore extension 2023-12-07 09:04:33 +01:00
Robbe Haesendonck
38369ca3cf Fixed typo 2023-12-07 09:04:33 +01:00
Robbe Haesendonck
cc0bdd34e9 Fixed linting errors 2023-12-07 09:04:33 +01:00
Robbe Haesendonck
8e71ff329c Added version check for PostGis and Postgres 2023-12-07 09:04:33 +01:00
Robbe Haesendonck
992703b15e Added --prepare-database flag 2023-12-07 09:04:33 +01:00
Robbe Haesendonck
ba5ec80611 Added --no-superuser flag 
To allow import into already existing databases.
Refs: #2719
 2023-12-07 09:04:33 +01:00
marc tobias
1c1447e709 Improve error message when Wikipedia importance file is not found 2023-12-06 14:31:07 +01:00
Sarah Hoffmann
8a2c6067a2 skip lookup with full names when there are none 2023-12-01 12:11:58 +01:00
Sarah Hoffmann
3c7a28dab0 further restrict stop search criterion 2023-11-29 11:28:54 +01:00
Sarah Hoffmann
0c72a434e0 use restrict for housenumber lookups with few numbers 2023-11-29 11:28:54 +01:00
Sarah Hoffmann
32e7b59b1f NearSearch needs to inherit penalty from inner search 2023-11-29 11:28:52 +01:00
Sarah Hoffmann
b2319e52ff correctly exclude streets with housenumber searches 
Street result are not subject to the full filtering in the SQL
query, so recheck.
 2023-11-28 17:53:37 +01:00
Sarah Hoffmann
25279d009a add tests for interaction of category parameter with category terms 2023-11-28 16:56:08 +01:00
Sarah Hoffmann
3f72ca4bca rename use of category as POI search to near_item 
Use the term category only as a short-cut for "tuple of key and value".
 2023-11-28 16:27:05 +01:00
Sarah Hoffmann
70dc4957dc the category parameter in search should result in a qualifier 2023-11-28 12:01:49 +01:00
Sarah Hoffmann
a7f5c6c8f5 drop category tokens when they make up a full phrase 2023-11-26 20:58:50 +01:00
Sarah Hoffmann
a8b023e57e restrict base results in near search by rank 
This avoids in particular that roads or POIs are used as base
for the near search when a place result is present.
 2023-11-26 17:41:29 +01:00
Sarah Hoffmann
47ca56f21b deduplicate categories/qualifiers 2023-11-26 17:11:15 +01:00
Sarah Hoffmann
580a7b032f order near searches by distance instead of importance 2023-11-26 16:48:04 +01:00
Sarah Hoffmann
8fcc2bb7f5 avoid duplicate lines during category search 2023-11-26 14:53:20 +01:00
Sarah Hoffmann
d6fe58f84e fix polygon selection for classtable lookups 
Polygons should be used preferably with higher address ranks
where the areas are smaller.
 2023-11-25 21:01:27 +01:00
Sarah Hoffmann
4e4d29f653 increase penalty for one-letter words 2023-11-23 10:51:58 +01:00
Sarah Hoffmann
195c13ee8a more preference for name-only queries in search 2023-11-22 23:57:23 +01:00
Sarah Hoffmann
ac5ef64701 avoid index use when filtering by layer 2023-11-22 20:54:04 +01:00
Sarah Hoffmann
e7dc24c026 add timestamps to text logging 2023-11-22 17:38:32 +01:00
Sarah Hoffmann
155f26060d avoid index on rank_address in near search 2023-11-22 17:33:17 +01:00
Sarah Hoffmann
a87fe8d8bf exclude country-level searches with non-address layers 2023-11-22 17:01:41 +01:00
Sarah Hoffmann
6478409b05 improve code to collect the PostGIS version 
The SQL contained an unchecked string literal, which may in theory be
used to attack the database.
 2023-11-16 11:14:29 +01:00
Sarah Hoffmann
9a1b8a67d6 adapt typing to newest version of SQLAlchemy 2023-11-15 19:59:26 +01:00
Sarah Hoffmann
b4ce1fb599 remove now unnecessary type igonre comment 2023-11-01 11:57:57 +01:00
Sarah Hoffmann
2bf8e62580 fix assertion on address list, it may be empty 
Fixes #3237.
 2023-10-31 21:10:54 +01:00
Sarah Hoffmann
a9ac68a729 restrict geometry size for SQLite 2023-10-23 17:19:12 +02:00
Sarah Hoffmann
84d6b481ae convert sqlite: add index on parent_place_id 2023-10-23 17:19:12 +02:00
Sarah Hoffmann
613c8635a8 remove type info from SQLALchemy condition functions 
A boolean type makes the SQLite dialect produce a costruct like
'func() = 1' in WHERE condition. While syntactically correct, it tends
to confuse the query planer.
 2023-10-23 17:19:12 +02:00
Sarah Hoffmann
899a04ad26 make code work with Spatialite 4.3 
Transfer is_address_point into SQLAlchemy function, so that
json.has_key() can use the older json_extract() function.
And work around broken Distance function.
 2023-10-23 17:19:12 +02:00
Sarah Hoffmann
d8dca2a3a9 enable BDD tests for sqlite databases 
The database must currently be created by hand and the name handed
in via -DAPI_TEST_DB='sqlite:...'.
 2023-10-23 17:19:12 +02:00
Sarah Hoffmann
8216899a9a trim all coordinate output to 7 digits 2023-10-23 17:19:12 +02:00
Sarah Hoffmann
b1d419f458 add indexing support for DWithin and intersects for sqlite 2023-10-23 17:19:12 +02:00
Sarah Hoffmann
0417946153 make reverse API work with sqlite 2023-10-23 17:19:12 +02:00
Sarah Hoffmann
37488ee82b try future annotations to resolve Generics compatibility 2023-10-23 17:19:12 +02:00
Sarah Hoffmann
06bbd501fd make status API work with sqlite incl. unit tests 2023-10-23 17:19:12 +02:00
Sarah Hoffmann
07e6c5cf69 make details API work with sqlite incl. unit tests 2023-10-23 17:19:12 +02:00
Sarah Hoffmann
d0c91e4acf make lookup call work with sqlite 
Includes porting unit tests.
 2023-10-23 17:19:12 +02:00
Sarah Hoffmann
114cdafe7e add exporting of SQLite table 2023-10-23 17:19:12 +02:00
Sarah Hoffmann
837bdecde8 add skeleton code for convert function 2023-10-23 17:19:12 +02:00
Sarah Hoffmann
d9d0e70e5b streamline SQLAlchemy DB schema 
Remove all information not strictly used by the frontend as well as
any index information. This will make it easier to create a SQLite
database from the schema.
 2023-10-23 17:19:12 +02:00
Sarah Hoffmann
1255efba7f remove unused code 2023-10-23 11:55:18 +02:00
Sarah Hoffmann
6ad397d4a9 Merge branch 'clean-deleted-relations' of https://github.com/lujoh/Nominatim into lujoh-clean-deleted-relations 2023-10-23 10:47:31 +02:00
lujoh
418f381b49 made age a required argument for the -clean-deleted command 2023-10-20 15:31:55 -04:00
lujoh
9ec26c60ff adjusted tests for --clean-deleted-relations command 2023-10-17 23:03:37 -04:00
Paweł Wroniszewski
fbe40e005d Properly validate postcodes with country code 
Include postcode pattern in postcode normalisation regex, instead of
removing it from postcode pattern in config.

It properly handles postcode validation and normalization when country code
is part of the postcode, e.g. for Isle of Man, Jersey, Anguilla, Andorra,
Cayman Islands and more.

Fixes #3227.
 2023-10-17 01:04:07 +02:00
lujoh
e9efef9095 added subcommand to clean deleted relations for issue # 2444 2023-10-16 11:30:58 -04:00
Sarah Hoffmann
95c3181a35 adapt typing for newer version of mypy 2023-10-16 17:03:48 +02:00
Sarah Hoffmann
12dbfb0777
Merge pull request #3222 from lonvia/fix-river-output 
Make sure the place name always comes first in output
 2023-10-10 11:20:12 +02:00
Sarah Hoffmann
b62dbd1f92 reduce influence of viewbox 
Perfectly matching city names should still get priority.
 2023-10-07 22:00:52 +02:00
Sarah Hoffmann
5011fde176 make sure the place name always comes first in output 
Also deleted some now unused code.
 2023-10-05 16:37:28 +02:00
Sarah Hoffmann
54cb9a33b1
Merge pull request #3212 from lonvia/more-tests 
Add more unit tests for search
 2023-10-05 16:36:53 +02:00
Sarah Hoffmann
f1fbcd863d switch back meaning of reverse and search in warm 
Also do'n try to warm up searches on reverse-only databases.

Fixes #3213.
 2023-10-04 15:19:24 +02:00
Sarah Hoffmann
b00b16aa3a more unit tests for search 2023-09-27 15:00:05 +02:00
Sarah Hoffmann
7fcbe13669 move get_addressdata() implementation to Python 
The pgsql function get_addressdata() does a lookup of a lot of data
that is already available in Python.
 2023-09-26 11:21:36 +02:00
Sarah Hoffmann
21df87dedc filter duplicate results after DB query 2023-09-20 14:58:54 +02:00
Sarah Hoffmann
fd26310d6a rerank results by query 
The algorithm is similar to the PHP reranking and uses the terms from
the display name to check against the query terms. However instead of
exact matching it uses a per-word-edit-distance, so that it is less
strict when it comes to mismatching accents or other one letter
differences.

Country names get a higher penalty because they don't receive a
penalty during token matching right now.

This will work badly with the legacy tokenizer. Given that it is
marked for removal, it is simply not worth optimising for it.
 2023-09-20 14:52:05 +02:00
Sarah Hoffmann
5762a5bc80 move localization into add_result_details 
This means that the locale now needs to be handed in into the search
functions already. At least search needs them for reranking.
 2023-09-19 11:17:04 +02:00
Sarah Hoffmann
f029fb3c65 'fix' issues with recent changes to psycopg2 typing 
Some of the changes just make the warnings go away. The typing info
is still incorrect on the stub side, as far as I can determine.
 2023-09-17 15:09:34 +02:00
Sarah Hoffmann
44da684d1d reduce expected count for multi-part words 
Fixes #3196.
 2023-09-11 17:45:34 +02:00
Sarah Hoffmann
ec47459410 prepare release 4.3.0 2023-09-06 20:08:28 +02:00
Sarah Hoffmann
c55c3657c3 reenable logging of details calls 2023-09-06 11:35:53 +02:00