osm-search/Nominatim
1
1
Fork 0
mirror of https://github.com/osm-search/Nominatim.git synced 2024-12-26 06:22:13 +03:00
Code Issues Packages Projects Releases Wiki Activity
3,111 Commits 31 Branches 41 Tags 164 MiB
d7f9d2bde9
Commit Graph

276 Commits

Author SHA1 Message Date
Sarah Hoffmann
7cc4c53adb always return 0 for updates unless there is an error 
This is more in line with previous behavioru than returning
a status code when no updates are available.
 2021-02-11 10:33:49 +01:00
Sarah Hoffmann
de37dc9300 forgot to replace one occurence of sql_dir 2021-02-09 19:32:05 +01:00
Sarah Hoffmann
8ffd7d9243 remove unused BINDIR constant 2021-02-09 19:30:31 +01:00
Sarah Hoffmann
298ed11261 introduce constant for configuration directory 
This replaces {data_dir}/settings throughout the code, so that
the configuration may be placed somewhere else in the directory
structure (e.g. in /etc).
 2021-02-09 18:45:45 +01:00
Sarah Hoffmann
b9517c99ae rename sql directory to lib-sql 
Also introduces a separate constant for the sql directory, so that
it can be put separately from the rest of the data if required.
 2021-02-09 15:26:56 +01:00
Sarah Hoffmann
d81e152804 integrate analyse of indexing into nominatim tool 2021-02-08 22:22:49 +01:00
Sarah Hoffmann
0cbf98c020 consolidate warm and db-check into single admin command 2021-02-08 21:05:06 +01:00
Sarah Hoffmann
195f9f5ef3 split cli.py by subcommands 
Reduces file size below 1000 lines.
 2021-02-08 17:23:05 +01:00
Sarah Hoffmann
861e67dfe8 fix off-by-one error in replication download 2021-02-04 17:04:04 +01:00
Sarah Hoffmann
948217d5e9 reintroduce timeout for replication file download 
This ports the --socket-timeout parameter from
pyosmium-get-changes which ensures that the update
process eventually times out on hanging network connections.
 2021-02-04 11:47:11 +01:00
Sarah Hoffmann
0b2abfb115 replace make serve with nominatim serve command 
With the website directory now tied to the project directory instead
of the build directory, it is no longer possible to use make for
running the web server.
 2021-02-03 16:34:31 +01:00
Sarah Hoffmann
cb06d1f4ca do not overwrite custom set module paths 
Given that the module is now copied to the project directory
when no module path is set, we need the information that the
module path is empty. Therefore hand in the default module path
in a separate variable.
 2021-02-02 18:31:25 +01:00
Sarah Hoffmann
36447c488a print project directory before running any command 2021-02-02 11:19:31 +01:00
Sarah Hoffmann
5f63d4ca1f print nice summary after updates 2021-02-01 10:34:31 +01:00
Sarah Hoffmann
90aaab77fc fix linting issues 2021-01-30 16:42:25 +01:00
Sarah Hoffmann
7158433cd3 disable warning about non-toplevel import 
They are needed here so nominatim can be run when osmium
is not installed. Everything except replication will work fine.
 2021-01-30 16:29:28 +01:00
Sarah Hoffmann
e629a175ed introduce custom UsageError 
This is a exception to be thrown when the error occures because
of bad user data. We don't want to print a full stack trace in
these cases but just tell the user what went wrong.
 2021-01-30 16:20:10 +01:00
Sarah Hoffmann
4cb6dc01f3 port replication update function to python 2021-01-30 15:50:34 +01:00
Sarah Hoffmann
8f0885f6cb port check-for-update function to python 2021-01-28 14:50:14 +01:00
Sarah Hoffmann
d78f0ba804 port replication initialisation to Python 2021-01-26 22:50:54 +01:00
Sarah Hoffmann
5b46fcad8e convert functon creation to python 
The new functions always creates normal and partitioned functions.
Also adds specialised connection and cursor classes for adding
frequently used helper functions.
 2021-01-26 22:50:54 +01:00
Sarah Hoffmann
94fa7162be port address level computation to Python 
Also adds simple tests for correct table creation.
 2021-01-26 22:50:54 +01:00
Sarah Hoffmann
e6c2842b66 move update code for postcode and word count to Python 
Adds also tests for the new function to execute a SQL script.
 2021-01-26 22:50:54 +01:00
Sarah Hoffmann
e6d9485c4a cli: import python modules for commands on demand 
Given that only one command will be executed in the end, it is
not necessary to import what amounts to the whole library. This
becomes in particular important for update functions that have
a dependency on pyosmium. The dependency can remain optional for
people not using updates.
 2021-01-26 22:50:54 +01:00
Sarah Hoffmann
42ec67f63c add more tests for CLI parameter parser 2021-01-20 21:30:27 +01:00
Sarah Hoffmann
8c02786820 add tests for indexer 2021-01-20 21:30:27 +01:00
Sarah Hoffmann
c26f323bf5 add simple tests for CLI parsing 2021-01-20 21:30:27 +01:00
Sarah Hoffmann
041ae67fd9 optionally hand in command line arguments to CLI functions 
Allows easier testing.
 2021-01-20 21:30:27 +01:00
Sarah Hoffmann
52b76d1d01 add tests for Python exec_utils 2021-01-20 21:30:27 +01:00
marc tobias
f62c784102 correct parameter name in query CLI 2021-01-20 21:09:41 +01:00
Sarah Hoffmann
8cf54a1317 add API functions to nominatim tool 2021-01-19 19:38:46 +01:00
Sarah Hoffmann
77e287f669 rename nominatim.admin to nominatim.tools 2021-01-19 19:38:46 +01:00
Sarah Hoffmann
5d95a72758 probe for php_cgi in cmake to be used for querying 2021-01-19 19:38:46 +01:00
Sarah Hoffmann
504922ffbe remove old nominatim.py in favour of 'nominatim index' 
The PHP scripts need to know the position of the nominatim
tool in order to call it. This is handed in as environment
variable, so it can be set by the Python script.
 2021-01-18 15:43:27 +01:00
Sarah Hoffmann
c77877a934 implementaion of 'nominatim index' 2021-01-18 15:43:27 +01:00
Sarah Hoffmann
27977411e9 move indexing function into its own Python module 
This makes it mow a standard function of our new Python
library instead of a stand-alone program.
 2021-01-18 15:43:27 +01:00
Sarah Hoffmann
b79c79fa73 add function to get a DSN for psycopg 
Converts the PHP DSN syntax into psycopg syntax when necessary.
 2021-01-18 15:43:27 +01:00
Sarah Hoffmann
7cf9d459d6 use check parameter of subprocess.run 
...instead of checking on our own.

Also increase required version of Python to 3.5 because of
subprocess.run().
 2021-01-15 10:43:04 +01:00
Sarah Hoffmann
8e53f63036 fix errors reported by pylint 2021-01-15 08:57:00 +01:00
Sarah Hoffmann
eda0900c8e fix typo 2021-01-14 20:30:27 +01:00
Sarah Hoffmann
2f73bb3643 bdd: directly call utility scripts in lib 
This removes the dependency on php-symfony-dotenv for the tests.
 2021-01-14 18:19:22 +01:00
Sarah Hoffmann
88c57b4dc8 maller command execution fixes 2021-01-14 12:03:49 +01:00
Sarah Hoffmann
ba13cfd9ff make sure that environment variables have highest prio 2021-01-14 11:12:45 +01:00
Sarah Hoffmann
1ff8751caa liniting of new python code 2021-01-14 10:19:21 +01:00
Sarah Hoffmann
98dbc84836 add wrapper calls for all nominatim tool functions 2021-01-14 09:37:47 +01:00
Sarah Hoffmann
04690ad8c4 implement warming in new cli tool 
Adds infrastructure for calling the legacy PHP scripts. As the
CONST_* values cannot be set from the python script, hand the values
in via secret environment variables instead. These are all
temporary hacks for the transition phase to python code.
 2021-01-13 18:25:15 +01:00
Sarah Hoffmann
d6bcb7c8b7 consolidate cli interface to single tool 2021-01-13 10:11:58 +01:00
Sarah Hoffmann
57f5e6d898 create skeleton for new CLI tools 2021-01-12 22:21:20 +01:00
Sarah Hoffmann
612fd50612 add skeleton for new Nominatim executables 2021-01-12 10:17:28 +01:00
Sarah Hoffmann
5016eace34 improve progress logging during indexing 
Wait for 2 seconds before logging the first progress, so that we
have numbers that are a bit more reliable statistically speaking.

Also provides an actual implementation for the log_interval
parameter and fixes some small style issues.
 2020-11-30 10:59:29 +01:00
Sarah Hoffmann
4ac29fb525 only index larger batches for rank 30 
Fixes #2045.
 2020-11-05 22:14:49 +01:00
Sarah Hoffmann
13dba94307 do not run rank 0 objects in parallel 
Waterways are at address rank 0 and do linking. This might lead to
deadlocks.
 2020-08-22 19:51:19 +02:00
Sarah Hoffmann
73c449b97b switch indexind to address rank 
A place needs all lower address rank object indexed to make up
the address. The search rank no longer ensures that as it can have
a different ordering than the address rank.

This switches indexing rank order to address ranks. Non-address
objects (with address rank 0) are indexed together with POIs.
 2020-08-18 16:58:58 +02:00
Sarah Hoffmann
3816b86a9e nominatim: also index boundaries by rank 
We need to make sure that the entry in serach_name from a lower rank
is indeed available.
 2020-08-18 15:17:09 +02:00
Sarah Hoffmann
a4b30fc649 index admin boundaries before everything else 
Avoids irregularities that might happen because the address
rank of a boundary is changed through linking.
 2020-08-18 15:17:09 +02:00
Sarah Hoffmann
fc50eb8688 nominatim: move DBConnection class into its own file 2020-08-18 15:17:09 +02:00
Sarah Hoffmann
5be084e0f5 indexer: allow batch processing of places 
Request and process multiple place_ids at once so that
Postgres can make better use of caching and there are less
transactions running.
 2020-08-03 10:32:39 +02:00
Sarah Hoffmann
2323923bec indexer: move progress tracker into separate class 2020-08-03 10:32:39 +02:00
Sarah Hoffmann
0f54d42863 indexer: get rid of special handling of few places 
Given that we do not distiribute geometry sectors to threads anymore,
there is no point in this kind of special handling.
 2020-08-03 10:32:39 +02:00
Sarah Hoffmann
cca366196d Disable JIT and parallel workers when indexing 
Locally disable jit and parallel workers in the connection that
do indexing. The query planner tends to be overenthusiatic about
using JIT. But with the rather less complex queries we have, the
overhead tends to be larger than the performance gain.

Fixes #1677.
 2020-05-30 11:20:16 +02:00
Sarah Hoffmann
431948d768 nominatim: always use deadlock-protected wait 
Fixes #1785.
 2020-05-15 18:49:27 +02:00
Sarah Hoffmann
5469d02d03 nominatim.py: fix wrong use of assert 
Fixes #1762.
 2020-04-19 17:59:49 +02:00
Sarah Hoffmann
d1eeaa59a6 nominatim.py: use async in connect() function 
The _async parameter name is only supported since psycopg 2.7.
However, async is a keyword in Python >= 3.7, so using this
gives us a syntax error. Working around this by defining the
parameters in a dict and handing that into the connect function.
 2020-02-11 22:16:17 +01:00
Sarah Hoffmann
882f496e0a nominatim.py: also catch deadlocks on final wait 2020-02-11 22:16:17 +01:00
Sarah Hoffmann
8b8aa1b4e6 regularly close connection while indexing 
Postgres sooner or later runs out of memory when the connection
is used for too long.
 2020-02-11 22:16:17 +01:00
Sarah Hoffmann
1801db523b fix typo 2020-01-29 11:50:30 +01:00
Sarah Hoffmann
8f6fdfeb0b forgot to index last rank 2020-01-24 22:06:30 +01:00
Sarah Hoffmann
b4e6d72fde replace nominatim C program 2020-01-24 22:06:30 +01:00
Sarah Hoffmann
a338ebfce0 fix log levels 2020-01-24 22:06:30 +01:00
Sarah Hoffmann
4144364a15 add time display for nominatim.py 2020-01-24 22:06:30 +01:00
Sarah Hoffmann
11c0dd235b clean up and document script 2020-01-24 22:06:30 +01:00
Sarah Hoffmann
4a9502bf88 fix SQL and some other stuff 2020-01-24 22:06:30 +01:00
Sarah Hoffmann
6c0d6d3178 Revert "switch to threading" 
This reverts commit 8b1c2181be5aa5335c68d36a49cab9c4e2cd8bef.
 2020-01-24 22:06:30 +01:00
Sarah Hoffmann
0a26ca7104 switch to threading 2020-01-24 22:06:30 +01:00
Sarah Hoffmann
2a15b2522f use generator for thread choice 2020-01-24 22:06:30 +01:00
Sarah Hoffmann
c11d1d78e9 add prepared statement 2020-01-24 22:06:30 +01:00
Sarah Hoffmann
7e51aa4cef simple implementation 2020-01-24 22:06:30 +01:00
Eric Stadtherr
62747c934d Work on setup/update scripts, unit tests, and documentation to enable Postgres server to be optionally configured on a remote host 2018-07-21 12:09:47 -06:00
Sarah Hoffmann
4ac1bf2d47 clean up byte order detection 
Check for existence of the expected functions and macros
and error out if nothing appropriate can be found.
 2018-03-16 23:09:40 +01:00
Sarah Hoffmann
8f23ba076b replace non-standard uint type with unsigned 
See #879.
 2018-01-10 23:27:49 +01:00
Jonathan Montane
c54fc44b33 feat(export): added linked_place_id as an attribute to feature element 2017-12-18 10:34:05 +01:00
Sarah Hoffmann
9a47e1834f reduce message frequency during indexing 2017-09-17 20:13:05 +02:00
Edward Betts
7e3af2debc correct spelling mistakes 2017-03-08 15:06:50 +00:00
Melvyn Sopacua
13ab03a03a Fix warnings: 
- be consistent with (const char *) casts when assigning
- use xmlStrlen instead of strlen when dealing with xmlChar *
 2017-02-15 10:17:43 +01:00
Melvyn Sopacua
6eb6f35f24 BSD compat: use sys/endian.h if available 
<byteswap.h> is a linuxism. On BSD-like systems this is <sys/endian.h>
 2017-02-13 14:30:48 +01:00
Sarah Hoffmann
ea5fe54c6b force language of pgsql to C 
Fixes #558.
 2017-01-15 21:31:14 +01:00
markigail
f07d620ee8 Change load-data in setup.php. 2016-05-11 10:22:03 +02:00
markigail
190a72cab5 Fix bug in index.c and remove column admin_level from location_property_osmline. 2016-05-08 16:46:42 +02:00
markigail
1a4f369e2b fix small bug in index.c 2016-05-03 12:52:08 +02:00
Markus Gail
db719d489f index on geometry of interpolation lines, and more improvements. 2016-04-27 17:42:59 +02:00
Markus Gail
7879ad44cd Remove interpolation lines from placex and save them in an extra table. 2016-04-25 09:44:01 +02:00
Sarah Hoffmann
932abeb0e2 add actual cmake file 2016-02-29 22:26:55 +01:00
Sarah Hoffmann
35a0f1b246 move from autotools to cmake and update osm2pgsql 2016-02-29 22:26:55 +01:00
Sarah Hoffmann
7822b1bc25 adapt to new configure variables for postgres 2015-08-29 12:15:19 +02:00
Sarah Hoffmann
7854075c5c hasn't been svn for a while 2013-10-02 19:03:13 +02:00
Sarah Hoffmann
89af917105 properly close connections to postgres 2012-12-16 16:23:50 +01:00
Sarah Hoffmann
b11718699e fail on error 
- stop indexing if there is any other error than deadlock from the DB
- stop import if osm2pgsql or indexing fails
 2012-07-31 00:31:38 +02:00
Sarah Hoffmann
980632053f put libraries in right place when linking 2012-07-03 16:09:00 +03:00
Sarah Hoffmann
f52038d010 yet another autogenerated file removed 2012-06-16 01:38:27 +02:00
Sarah Hoffmann
69ed04a441 remove autogenerated files 2012-06-16 01:36:13 +02:00
Sarah Hoffmann
de46b62186 create a globale configure/make 
- move the configure from nominatim out and add checks for all dependencies
- bring module under automake control
- add osm2pgsql as a submodule, configure and build with the others
 2012-06-16 01:25:16 +02:00
Sarah Hoffmann
35fdb7af0f more verbose error messages 2012-05-24 00:26:16 +02:00
Brian Quinion
b99a043c78 Use GB postcode table as definitive source. resort by pressence of search word in output address and other misc very minor changes not commited from MQ servers 2012-03-22 00:34:56 +00:00
Sarah Hoffmann
1edcd7208b don't classify normal SQL errors as deadlocks 2012-01-28 23:33:28 +00:00
Sarah Hoffmann
cc564e2f31 - remove dependency on intarray extension 
- new config options for postgresql version and 
  location of osm2pgsql and osmosis binaries
- new option --no-npi for update.php that supresses writing of NPIs in
  osmosis update mode
- add more GRANTs for www-data
 2012-01-21 10:21:42 +00:00
Brian Quinion
4d16677d2e update place_id to BIGINT 2011-06-14 13:42:46 +00:00
Brian Quinion
588a3a5702 memory leak on large imports 2011-05-16 14:46:57 +00:00
Brian Quinion
b3f2bdb822 addresses not claculated for some named poi 2011-04-08 15:10:31 +00:00
Brian Quinion
1028c2444c move partiondef config file to settings, add --create-website option to build symlinks automatically 2011-04-06 18:19:40 +00:00
Brian Quinion
1767968f7c extra validation on import 2011-04-06 18:16:55 +00:00
Brian Quinion
44aa62d899 loading rank wrong way round 2011-03-21 10:15:29 +00:00
Brian Quinion
2148d81474 calculate search position based to 'importance' rather than address rank 2011-03-18 09:52:16 +00:00
Frederik Ramm
fb6d13a1a7 log progress to stderr instead of stdout, to make use of unbuffered output 2011-02-21 12:41:44 +00:00
Brian Quinion
c71cb63f38 stop treating capital cities as villages, memory leak on deadlock 2011-02-08 12:09:11 +00:00
Brian Quinion
3ea1b35b32 add logging of broken polygons, improve address export, better diff output in export, bug in address calculation(#3522), reaply #3520, lost in migration 2011-02-07 11:13:18 +00:00
Brian Quinion
31273a42bd aux properties, memory leaks, tweaks to search order 2011-01-21 10:40:44 +00:00
Brian Quinion
2be00f37c1 finally fix adminitrative spelling error, disable fast_update on indexes by default (performance), fixes for incremental updates on new codebase 2011-01-05 14:07:26 +00:00
Brian Quinion
52cdaba0f1 clean up formatting 2010-12-10 16:13:07 +00:00
Brian Quinion
1427753846 be a bit efficient about queuing up the places to be indexed 2010-12-10 16:09:43 +00:00
Brian Quinion
631e8d09ab postcode/zipcode improvements, finish work on handling extratags 2010-12-07 13:41:02 +00:00
Brian Quinion
bfe56e6a45 change partition from char to int, partition search_name on country 2010-11-09 10:19:36 +00:00
Brian Quinion
756f328ac9 more multi-processor improvements 2010-11-02 11:37:11 +00:00
Brian Quinion
e6d983474b data partitioning 2010-10-26 15:22:41 +00:00
Brian Quinion
4c2c499860 default path fixes 2010-10-25 12:22:22 +00:00
Brian Quinion
a98bc3fae8 nominatim refactoring 2010-10-25 10:50:52 +00:00
Brian Quinion
ab6a996772 nominatim refactoring 2010-10-23 23:12:37 +00:00