Sarah Hoffmann
4cb6dc01f3
port replication update function to python
2021-01-30 15:50:34 +01:00
Sarah Hoffmann
8f0885f6cb
port check-for-update function to python
2021-01-28 14:50:14 +01:00
Sarah Hoffmann
d78f0ba804
port replication initialisation to Python
2021-01-26 22:50:54 +01:00
Sarah Hoffmann
5b46fcad8e
convert functon creation to python
...
The new functions always creates normal and partitioned functions.
Also adds specialised connection and cursor classes for adding
frequently used helper functions.
2021-01-26 22:50:54 +01:00
Sarah Hoffmann
94fa7162be
port address level computation to Python
...
Also adds simple tests for correct table creation.
2021-01-26 22:50:54 +01:00
Sarah Hoffmann
e6c2842b66
move update code for postcode and word count to Python
...
Adds also tests for the new function to execute a SQL script.
2021-01-26 22:50:54 +01:00
Sarah Hoffmann
e6d9485c4a
cli: import python modules for commands on demand
...
Given that only one command will be executed in the end, it is
not necessary to import what amounts to the whole library. This
becomes in particular important for update functions that have
a dependency on pyosmium. The dependency can remain optional for
people not using updates.
2021-01-26 22:50:54 +01:00
Sarah Hoffmann
42ec67f63c
add more tests for CLI parameter parser
2021-01-20 21:30:27 +01:00
Sarah Hoffmann
8c02786820
add tests for indexer
2021-01-20 21:30:27 +01:00
Sarah Hoffmann
c26f323bf5
add simple tests for CLI parsing
2021-01-20 21:30:27 +01:00
Sarah Hoffmann
041ae67fd9
optionally hand in command line arguments to CLI functions
...
Allows easier testing.
2021-01-20 21:30:27 +01:00
Sarah Hoffmann
52b76d1d01
add tests for Python exec_utils
2021-01-20 21:30:27 +01:00
marc tobias
f62c784102
correct parameter name in query CLI
2021-01-20 21:09:41 +01:00
Sarah Hoffmann
8cf54a1317
add API functions to nominatim tool
2021-01-19 19:38:46 +01:00
Sarah Hoffmann
77e287f669
rename nominatim.admin to nominatim.tools
2021-01-19 19:38:46 +01:00
Sarah Hoffmann
5d95a72758
probe for php_cgi in cmake to be used for querying
2021-01-19 19:38:46 +01:00
Sarah Hoffmann
504922ffbe
remove old nominatim.py in favour of 'nominatim index'
...
The PHP scripts need to know the position of the nominatim
tool in order to call it. This is handed in as environment
variable, so it can be set by the Python script.
2021-01-18 15:43:27 +01:00
Sarah Hoffmann
c77877a934
implementaion of 'nominatim index'
2021-01-18 15:43:27 +01:00
Sarah Hoffmann
27977411e9
move indexing function into its own Python module
...
This makes it mow a standard function of our new Python
library instead of a stand-alone program.
2021-01-18 15:43:27 +01:00
Sarah Hoffmann
b79c79fa73
add function to get a DSN for psycopg
...
Converts the PHP DSN syntax into psycopg syntax when necessary.
2021-01-18 15:43:27 +01:00
Sarah Hoffmann
7cf9d459d6
use check parameter of subprocess.run
...
...instead of checking on our own.
Also increase required version of Python to 3.5 because of
subprocess.run().
2021-01-15 10:43:04 +01:00
Sarah Hoffmann
8e53f63036
fix errors reported by pylint
2021-01-15 08:57:00 +01:00
Sarah Hoffmann
eda0900c8e
fix typo
2021-01-14 20:30:27 +01:00
Sarah Hoffmann
2f73bb3643
bdd: directly call utility scripts in lib
...
This removes the dependency on php-symfony-dotenv for the tests.
2021-01-14 18:19:22 +01:00
Sarah Hoffmann
88c57b4dc8
maller command execution fixes
2021-01-14 12:03:49 +01:00
Sarah Hoffmann
ba13cfd9ff
make sure that environment variables have highest prio
2021-01-14 11:12:45 +01:00
Sarah Hoffmann
1ff8751caa
liniting of new python code
2021-01-14 10:19:21 +01:00
Sarah Hoffmann
98dbc84836
add wrapper calls for all nominatim tool functions
2021-01-14 09:37:47 +01:00
Sarah Hoffmann
04690ad8c4
implement warming in new cli tool
...
Adds infrastructure for calling the legacy PHP scripts. As the
CONST_* values cannot be set from the python script, hand the values
in via secret environment variables instead. These are all
temporary hacks for the transition phase to python code.
2021-01-13 18:25:15 +01:00
Sarah Hoffmann
d6bcb7c8b7
consolidate cli interface to single tool
2021-01-13 10:11:58 +01:00
Sarah Hoffmann
57f5e6d898
create skeleton for new CLI tools
2021-01-12 22:21:20 +01:00
Sarah Hoffmann
612fd50612
add skeleton for new Nominatim executables
2021-01-12 10:17:28 +01:00
Sarah Hoffmann
5016eace34
improve progress logging during indexing
...
Wait for 2 seconds before logging the first progress, so that we
have numbers that are a bit more reliable statistically speaking.
Also provides an actual implementation for the log_interval
parameter and fixes some small style issues.
2020-11-30 10:59:29 +01:00
Sarah Hoffmann
4ac29fb525
only index larger batches for rank 30
...
Fixes #2045 .
2020-11-05 22:14:49 +01:00
Sarah Hoffmann
13dba94307
do not run rank 0 objects in parallel
...
Waterways are at address rank 0 and do linking. This might lead to
deadlocks.
2020-08-22 19:51:19 +02:00
Sarah Hoffmann
73c449b97b
switch indexind to address rank
...
A place needs all lower address rank object indexed to make up
the address. The search rank no longer ensures that as it can have
a different ordering than the address rank.
This switches indexing rank order to address ranks. Non-address
objects (with address rank 0) are indexed together with POIs.
2020-08-18 16:58:58 +02:00
Sarah Hoffmann
3816b86a9e
nominatim: also index boundaries by rank
...
We need to make sure that the entry in serach_name from a lower rank
is indeed available.
2020-08-18 15:17:09 +02:00
Sarah Hoffmann
a4b30fc649
index admin boundaries before everything else
...
Avoids irregularities that might happen because the address
rank of a boundary is changed through linking.
2020-08-18 15:17:09 +02:00
Sarah Hoffmann
fc50eb8688
nominatim: move DBConnection class into its own file
2020-08-18 15:17:09 +02:00
Sarah Hoffmann
5be084e0f5
indexer: allow batch processing of places
...
Request and process multiple place_ids at once so that
Postgres can make better use of caching and there are less
transactions running.
2020-08-03 10:32:39 +02:00
Sarah Hoffmann
2323923bec
indexer: move progress tracker into separate class
2020-08-03 10:32:39 +02:00
Sarah Hoffmann
0f54d42863
indexer: get rid of special handling of few places
...
Given that we do not distiribute geometry sectors to threads anymore,
there is no point in this kind of special handling.
2020-08-03 10:32:39 +02:00
Sarah Hoffmann
cca366196d
Disable JIT and parallel workers when indexing
...
Locally disable jit and parallel workers in the connection that
do indexing. The query planner tends to be overenthusiatic about
using JIT. But with the rather less complex queries we have, the
overhead tends to be larger than the performance gain.
Fixes #1677 .
2020-05-30 11:20:16 +02:00
Sarah Hoffmann
431948d768
nominatim: always use deadlock-protected wait
...
Fixes #1785 .
2020-05-15 18:49:27 +02:00
Sarah Hoffmann
5469d02d03
nominatim.py: fix wrong use of assert
...
Fixes #1762 .
2020-04-19 17:59:49 +02:00
Sarah Hoffmann
d1eeaa59a6
nominatim.py: use async in connect() function
...
The _async parameter name is only supported since psycopg 2.7.
However, async is a keyword in Python >= 3.7, so using this
gives us a syntax error. Working around this by defining the
parameters in a dict and handing that into the connect function.
2020-02-11 22:16:17 +01:00
Sarah Hoffmann
882f496e0a
nominatim.py: also catch deadlocks on final wait
2020-02-11 22:16:17 +01:00
Sarah Hoffmann
8b8aa1b4e6
regularly close connection while indexing
...
Postgres sooner or later runs out of memory when the connection
is used for too long.
2020-02-11 22:16:17 +01:00
Sarah Hoffmann
1801db523b
fix typo
2020-01-29 11:50:30 +01:00
Sarah Hoffmann
8f6fdfeb0b
forgot to index last rank
2020-01-24 22:06:30 +01:00