Darkshredder
b7d6ae93e3
Nominatim/cli.py rebase fixes
2021-03-29 14:16:41 +05:30
Darkshredder
21b1b75b08
Rebase with master
2021-03-29 14:00:45 +05:30
Darkshredder
51e2654cd2
Added Manual page and fixed documentation
2021-03-29 13:57:13 +05:30
Sarah Hoffmann
09b2510219
Merge pull request #2228 from AntoJvlt/import-special-phrases-porting-python
...
Import special phrases porting python
2021-03-29 09:49:35 +02:00
AntoJvlt
57ce75eb67
Change command 'import-special-phrases --from-wiki' to 'special-phrases --import-from-wiki'.
2021-03-26 02:22:38 +01:00
AntoJvlt
cde9389e75
Errors fixes, Cleaning code, Improvement and addition of tests
2021-03-26 01:53:33 +01:00
AntoJvlt
2c19bd5ea3
Encapsulation of tools/special_phrases.py into SpecialPhrasesImporter class and add new tests.
2021-03-25 21:13:57 +01:00
AntoJvlt
ff34198569
Code cleaning, tests simplification and use of python3-icu package
2021-03-23 23:56:39 +01:00
AntoJvlt
1ce8b530cd
Introduction of PyICU for transliteration in python. Reversed changes in normalization.sql.
2021-03-23 23:34:16 +01:00
AntoJvlt
6d56cbb3e8
Changed phrase_settings.py to phrase-settings.json and added migration function for old php settings file.
2021-03-23 23:30:39 +01:00
Sarah Hoffmann
4f1bdde32e
Merge pull request #2231 from mtmail/correct-cli-help-page
...
nominatim -h was printing wrong text for lookup and details
2021-03-21 16:52:20 +01:00
Sarah Hoffmann
a08ca5b1b5
avoid division by zero in progress meter
...
On Windows systems the timer may not be accurate enough to measure
the time between init() and done(). Avoid computing statistics with
a diff time of 0 in such cases.
Fixes #2230 .
2021-03-21 16:47:22 +01:00
marc tobias
87d5883ddb
nominatim -h was priting wrong text for lookup and details
2021-03-21 16:06:41 +01:00
AntoJvlt
17cb59efbd
Ported functions for the import of special phrases from php to python.
...
- the command is now --import-special-phrases
- the output is not an sql file anymore, data are directly imported to the database.
- the little part on the documentation (section data import) has been modified.
2021-03-20 19:11:50 +01:00
Sarah Hoffmann
81a6b746b8
Merge pull request #2212 from darkshredder/country-name
...
Ported createCountryNames() to python and Added tests
2021-03-15 09:36:06 +01:00
Sarah Hoffmann
7212fa8630
fix template variable name
2021-03-13 12:05:53 +01:00
Darkshredder
b108bd1c1e
Linting fix
2021-03-12 18:28:47 +05:30
Darkshredder
077a8c1f95
refactored tests and made changes to code for easy readibility
2021-03-12 18:23:20 +05:30
Darkshredder
7a874d5b97
Ported createCountryNames() to python and added tests
2021-03-12 10:28:41 +05:30
Sarah Hoffmann
9086a794a1
Merge pull request #2204 from darkshredder/tiger-data
...
Ported tiger-data-import to Python and Added Tarball Support
2021-03-11 22:48:38 +01:00
Darkshredder
e5719de657
Added fixture for sql_preprocessor and fixed some issues
2021-03-11 15:39:17 +05:30
Darkshredder
ccfad57fca
Added test and removed runlegacyscript
2021-03-10 17:18:12 +05:30
Darkshredder
64128b699a
fixed linting, refactored threaded sql handling and removed importTigerData() function
2021-03-10 13:28:29 +05:30
Darkshredder
4080fbb95c
Test fixes
2021-03-09 01:00:56 +05:30
Darkshredder
14ec83c886
Linting fixes
2021-03-08 23:10:49 +05:30
Darkshredder
122c4618b9
Linting fixes
2021-03-08 22:59:51 +05:30
Darkshredder
2af82975cd
Ported tiger-data-import to python and Added Tarball Support
2021-03-08 21:57:56 +05:30
Sarah Hoffmann
764a41b973
automatic migration from 3.6 release
...
Adds a 'admin --migrate' command that checks for the current
database version and runs any necessary migrations. Also
has migrations going back to 3.6.
2021-03-06 16:36:57 +01:00
Sarah Hoffmann
9d103503f7
Merge pull request #2197 from lonvia/use-jinja-for-sql-preprocessing
...
Use jinja2 for SQL preprocessing
2021-03-04 16:36:18 +01:00
Sarah Hoffmann
09f4d767e4
port index creation to python
...
Also switches to jinja-based preprocessing, which allows to
simplify the SQL files. Use 'if not exists' where possible
so that the step can be rerun to fix missing indexes.
2021-03-04 11:11:47 +01:00
Sarah Hoffmann
dd301cf5ac
indexer: ANALYSE must be run outside transactions
2021-03-04 11:06:33 +01:00
Sarah Hoffmann
eacabb0e96
move table creation to jinja-based preprocessing
2021-03-03 22:07:51 +01:00
Sarah Hoffmann
d2bd6aa78d
introduce jinja2 for preprocessing SQL
...
Replaces various hand-crafted replacements of varying format with
a single Jinja2 templating mechanism. Allows full access to
configuration if necessary.
2021-03-03 17:51:08 +01:00
Sarah Hoffmann
3a0a4b9175
save software version in the database
...
The version represents the software version that was used to
import the data.
2021-03-01 20:35:15 +01:00
Sarah Hoffmann
4faefe156c
report software version of status call
2021-03-01 16:47:19 +01:00
Sarah Hoffmann
86273f5e2a
introduce database patch level for version
...
This will be needed later for automatic migrations.
2021-03-01 16:46:19 +01:00
Sarah Hoffmann
b4f64aa770
make sure that calls to PHP legacy scripts are fatal on error
2021-03-01 16:10:45 +01:00
Sarah Hoffmann
b46adbad22
make sure psql always finishes
...
If an execption is raised by other means, we still have to close
the stdin pipe to psql to make sure that it exits and releases its
connection to the database.
2021-02-27 10:24:40 +01:00
Sarah Hoffmann
d14a3df10f
do not truncate search_name in reverse-only mode
2021-02-27 09:46:42 +01:00
Sarah Hoffmann
c7f40e3cee
fix verbose flag for PHP wrapper scripts
...
The flag must come after the command.
2021-02-26 16:49:32 +01:00
Sarah Hoffmann
dd03aeb966
bdd: use python library where possible
...
Replace calls to PHP scripts with direct calls into the
nominatim Python library where possible. This speed up
tests quite a bit.
2021-02-26 16:14:29 +01:00
Sarah Hoffmann
15b5906790
move setup function to python
...
There are still back-calls to PHP for some of the sub-steps.
These needs some larger refactoring to be moved to Python.
2021-02-26 15:02:39 +01:00
Sarah Hoffmann
3ee8d9fa75
properly close connections of indexer after use
2021-02-26 12:10:54 +01:00
Sarah Hoffmann
57db5819ef
prot load-data function to python
2021-02-25 21:32:40 +01:00
Sarah Hoffmann
3c186f8030
add a function for the intial indexing run
...
Also moves postcodes to fully parallel indexing.
2021-02-25 18:42:54 +01:00
Sarah Hoffmann
c7fd0a7af4
port wikipedia importance functions to python
2021-02-25 18:42:54 +01:00
Sarah Hoffmann
32683f73c7
move import-data option to native python
...
This adds a new dependecy to the Python psutil package.
2021-02-25 18:42:54 +01:00
Sarah Hoffmann
7222235579
introduce custom object for cmdline arguments
...
Allows to define special functions over the arguments.
Also splits CLI tests in two files as they have become too many.
2021-02-25 18:42:54 +01:00
Sarah Hoffmann
f6e894a53a
port database setup function to python
...
Hide the former PHP functions in a transition command until
they are removed.
2021-02-25 18:42:54 +01:00
Sarah Hoffmann
b93ec2522e
use psql for executing sql files
...
This allows to run larger files without needing to keep
them in memory.
2021-02-25 18:42:54 +01:00
Sarah Hoffmann
af7226393a
add function to set up libpq environment
...
Instead of parsing the DSN for each external libpq program we
are going to execute, provide a function that feeds them all
necessary parameters through the environment.
osm2pgsql is the first user.
2021-02-25 18:42:54 +01:00
Sarah Hoffmann
e520613362
convert connect() into a context manager
2021-02-25 18:42:54 +01:00
Sarah Hoffmann
a1f0fc1a10
improve deadlock detection for various versions of psycopg2
...
Psycopg2 has changed the kind of exception that is emitted on
deadlocks between versions 2.7 and 2.8. The code was already
trying to catch both kind of errors but because the
psycopg2.errors package is unknown in 2.7 and below, the
code would throw an exception on anything but a deadlock error.
This commit wraps the deadlock handling into a context manager
to avoid code duplication and uses module imports to detect if
the new error codes are available.
Also sets the required psycopg2 version to 2.7 or bigger as
versions below are difficult to test.
2021-02-25 18:11:16 +01:00
Sarah Hoffmann
971df231b0
avoid os.environ as default valie
2021-02-19 19:29:57 +01:00
Sarah Hoffmann
4b32cbe518
fix return code for check database run with 'not applicable'
2021-02-19 18:32:00 +01:00
Sarah Hoffmann
f08078ccca
bdd tests: directly call python code for setup-website
2021-02-19 18:20:55 +01:00
Sarah Hoffmann
389138abfe
port setup-website to python
2021-02-19 17:51:06 +01:00
Sarah Hoffmann
a0ae4945cd
add unit tests for new check_database code
2021-02-18 20:36:11 +01:00
Sarah Hoffmann
b169e4c88c
port check-database function to python
...
This change also adapts the hints to use the nominatim tool.
Slightly changed checks, so that they are just as effective on
a frozen database.
2021-02-18 17:32:30 +01:00
Sarah Hoffmann
101a1f895d
port freeze function to python
2021-02-17 21:43:15 +01:00
Sarah Hoffmann
c9838a02ce
disable JIT and parallel execution for osm2pgsql updates again
...
The gazetteer output doesn't disable these functions when
writing to the place table but the triggers may contain
operations that cause misplanning for the query planner.
2021-02-16 18:23:47 +01:00
Sarah Hoffmann
fbe7be760b
ignore failure to get replication date
2021-02-14 12:17:30 +01:00
Sarah Hoffmann
7cc4c53adb
always return 0 for updates unless there is an error
...
This is more in line with previous behavioru than returning
a status code when no updates are available.
2021-02-11 10:33:49 +01:00
Sarah Hoffmann
de37dc9300
forgot to replace one occurence of sql_dir
2021-02-09 19:32:05 +01:00
Sarah Hoffmann
8ffd7d9243
remove unused BINDIR constant
2021-02-09 19:30:31 +01:00
Sarah Hoffmann
298ed11261
introduce constant for configuration directory
...
This replaces {data_dir}/settings throughout the code, so that
the configuration may be placed somewhere else in the directory
structure (e.g. in /etc).
2021-02-09 18:45:45 +01:00
Sarah Hoffmann
b9517c99ae
rename sql directory to lib-sql
...
Also introduces a separate constant for the sql directory, so that
it can be put separately from the rest of the data if required.
2021-02-09 15:26:56 +01:00
Sarah Hoffmann
d81e152804
integrate analyse of indexing into nominatim tool
2021-02-08 22:22:49 +01:00
Sarah Hoffmann
0cbf98c020
consolidate warm and db-check into single admin command
2021-02-08 21:05:06 +01:00
Sarah Hoffmann
195f9f5ef3
split cli.py by subcommands
...
Reduces file size below 1000 lines.
2021-02-08 17:23:05 +01:00
Sarah Hoffmann
861e67dfe8
fix off-by-one error in replication download
2021-02-04 17:04:04 +01:00
Sarah Hoffmann
948217d5e9
reintroduce timeout for replication file download
...
This ports the --socket-timeout parameter from
pyosmium-get-changes which ensures that the update
process eventually times out on hanging network connections.
2021-02-04 11:47:11 +01:00
Sarah Hoffmann
0b2abfb115
replace make serve with nominatim serve command
...
With the website directory now tied to the project directory instead
of the build directory, it is no longer possible to use make for
running the web server.
2021-02-03 16:34:31 +01:00
Sarah Hoffmann
cb06d1f4ca
do not overwrite custom set module paths
...
Given that the module is now copied to the project directory
when no module path is set, we need the information that the
module path is empty. Therefore hand in the default module path
in a separate variable.
2021-02-02 18:31:25 +01:00
Sarah Hoffmann
36447c488a
print project directory before running any command
2021-02-02 11:19:31 +01:00
Sarah Hoffmann
5f63d4ca1f
print nice summary after updates
2021-02-01 10:34:31 +01:00
Sarah Hoffmann
90aaab77fc
fix linting issues
2021-01-30 16:42:25 +01:00
Sarah Hoffmann
7158433cd3
disable warning about non-toplevel import
...
They are needed here so nominatim can be run when osmium
is not installed. Everything except replication will work fine.
2021-01-30 16:29:28 +01:00
Sarah Hoffmann
e629a175ed
introduce custom UsageError
...
This is a exception to be thrown when the error occures because
of bad user data. We don't want to print a full stack trace in
these cases but just tell the user what went wrong.
2021-01-30 16:20:10 +01:00
Sarah Hoffmann
4cb6dc01f3
port replication update function to python
2021-01-30 15:50:34 +01:00
Sarah Hoffmann
8f0885f6cb
port check-for-update function to python
2021-01-28 14:50:14 +01:00
Sarah Hoffmann
d78f0ba804
port replication initialisation to Python
2021-01-26 22:50:54 +01:00
Sarah Hoffmann
5b46fcad8e
convert functon creation to python
...
The new functions always creates normal and partitioned functions.
Also adds specialised connection and cursor classes for adding
frequently used helper functions.
2021-01-26 22:50:54 +01:00
Sarah Hoffmann
94fa7162be
port address level computation to Python
...
Also adds simple tests for correct table creation.
2021-01-26 22:50:54 +01:00
Sarah Hoffmann
e6c2842b66
move update code for postcode and word count to Python
...
Adds also tests for the new function to execute a SQL script.
2021-01-26 22:50:54 +01:00
Sarah Hoffmann
e6d9485c4a
cli: import python modules for commands on demand
...
Given that only one command will be executed in the end, it is
not necessary to import what amounts to the whole library. This
becomes in particular important for update functions that have
a dependency on pyosmium. The dependency can remain optional for
people not using updates.
2021-01-26 22:50:54 +01:00
Sarah Hoffmann
42ec67f63c
add more tests for CLI parameter parser
2021-01-20 21:30:27 +01:00
Sarah Hoffmann
8c02786820
add tests for indexer
2021-01-20 21:30:27 +01:00
Sarah Hoffmann
c26f323bf5
add simple tests for CLI parsing
2021-01-20 21:30:27 +01:00
Sarah Hoffmann
041ae67fd9
optionally hand in command line arguments to CLI functions
...
Allows easier testing.
2021-01-20 21:30:27 +01:00
Sarah Hoffmann
52b76d1d01
add tests for Python exec_utils
2021-01-20 21:30:27 +01:00
marc tobias
f62c784102
correct parameter name in query CLI
2021-01-20 21:09:41 +01:00
Sarah Hoffmann
8cf54a1317
add API functions to nominatim tool
2021-01-19 19:38:46 +01:00
Sarah Hoffmann
77e287f669
rename nominatim.admin to nominatim.tools
2021-01-19 19:38:46 +01:00
Sarah Hoffmann
5d95a72758
probe for php_cgi in cmake to be used for querying
2021-01-19 19:38:46 +01:00
Sarah Hoffmann
504922ffbe
remove old nominatim.py in favour of 'nominatim index'
...
The PHP scripts need to know the position of the nominatim
tool in order to call it. This is handed in as environment
variable, so it can be set by the Python script.
2021-01-18 15:43:27 +01:00
Sarah Hoffmann
c77877a934
implementaion of 'nominatim index'
2021-01-18 15:43:27 +01:00
Sarah Hoffmann
27977411e9
move indexing function into its own Python module
...
This makes it mow a standard function of our new Python
library instead of a stand-alone program.
2021-01-18 15:43:27 +01:00
Sarah Hoffmann
b79c79fa73
add function to get a DSN for psycopg
...
Converts the PHP DSN syntax into psycopg syntax when necessary.
2021-01-18 15:43:27 +01:00
Sarah Hoffmann
7cf9d459d6
use check parameter of subprocess.run
...
...instead of checking on our own.
Also increase required version of Python to 3.5 because of
subprocess.run().
2021-01-15 10:43:04 +01:00
Sarah Hoffmann
8e53f63036
fix errors reported by pylint
2021-01-15 08:57:00 +01:00
Sarah Hoffmann
eda0900c8e
fix typo
2021-01-14 20:30:27 +01:00
Sarah Hoffmann
2f73bb3643
bdd: directly call utility scripts in lib
...
This removes the dependency on php-symfony-dotenv for the tests.
2021-01-14 18:19:22 +01:00
Sarah Hoffmann
88c57b4dc8
maller command execution fixes
2021-01-14 12:03:49 +01:00
Sarah Hoffmann
ba13cfd9ff
make sure that environment variables have highest prio
2021-01-14 11:12:45 +01:00
Sarah Hoffmann
1ff8751caa
liniting of new python code
2021-01-14 10:19:21 +01:00
Sarah Hoffmann
98dbc84836
add wrapper calls for all nominatim tool functions
2021-01-14 09:37:47 +01:00
Sarah Hoffmann
04690ad8c4
implement warming in new cli tool
...
Adds infrastructure for calling the legacy PHP scripts. As the
CONST_* values cannot be set from the python script, hand the values
in via secret environment variables instead. These are all
temporary hacks for the transition phase to python code.
2021-01-13 18:25:15 +01:00
Sarah Hoffmann
d6bcb7c8b7
consolidate cli interface to single tool
2021-01-13 10:11:58 +01:00
Sarah Hoffmann
57f5e6d898
create skeleton for new CLI tools
2021-01-12 22:21:20 +01:00
Sarah Hoffmann
612fd50612
add skeleton for new Nominatim executables
2021-01-12 10:17:28 +01:00
Sarah Hoffmann
5016eace34
improve progress logging during indexing
...
Wait for 2 seconds before logging the first progress, so that we
have numbers that are a bit more reliable statistically speaking.
Also provides an actual implementation for the log_interval
parameter and fixes some small style issues.
2020-11-30 10:59:29 +01:00
Sarah Hoffmann
4ac29fb525
only index larger batches for rank 30
...
Fixes #2045 .
2020-11-05 22:14:49 +01:00
Sarah Hoffmann
13dba94307
do not run rank 0 objects in parallel
...
Waterways are at address rank 0 and do linking. This might lead to
deadlocks.
2020-08-22 19:51:19 +02:00
Sarah Hoffmann
73c449b97b
switch indexind to address rank
...
A place needs all lower address rank object indexed to make up
the address. The search rank no longer ensures that as it can have
a different ordering than the address rank.
This switches indexing rank order to address ranks. Non-address
objects (with address rank 0) are indexed together with POIs.
2020-08-18 16:58:58 +02:00
Sarah Hoffmann
3816b86a9e
nominatim: also index boundaries by rank
...
We need to make sure that the entry in serach_name from a lower rank
is indeed available.
2020-08-18 15:17:09 +02:00
Sarah Hoffmann
a4b30fc649
index admin boundaries before everything else
...
Avoids irregularities that might happen because the address
rank of a boundary is changed through linking.
2020-08-18 15:17:09 +02:00
Sarah Hoffmann
fc50eb8688
nominatim: move DBConnection class into its own file
2020-08-18 15:17:09 +02:00
Sarah Hoffmann
5be084e0f5
indexer: allow batch processing of places
...
Request and process multiple place_ids at once so that
Postgres can make better use of caching and there are less
transactions running.
2020-08-03 10:32:39 +02:00
Sarah Hoffmann
2323923bec
indexer: move progress tracker into separate class
2020-08-03 10:32:39 +02:00
Sarah Hoffmann
0f54d42863
indexer: get rid of special handling of few places
...
Given that we do not distiribute geometry sectors to threads anymore,
there is no point in this kind of special handling.
2020-08-03 10:32:39 +02:00
Sarah Hoffmann
cca366196d
Disable JIT and parallel workers when indexing
...
Locally disable jit and parallel workers in the connection that
do indexing. The query planner tends to be overenthusiatic about
using JIT. But with the rather less complex queries we have, the
overhead tends to be larger than the performance gain.
Fixes #1677 .
2020-05-30 11:20:16 +02:00
Sarah Hoffmann
431948d768
nominatim: always use deadlock-protected wait
...
Fixes #1785 .
2020-05-15 18:49:27 +02:00
Sarah Hoffmann
5469d02d03
nominatim.py: fix wrong use of assert
...
Fixes #1762 .
2020-04-19 17:59:49 +02:00
Sarah Hoffmann
d1eeaa59a6
nominatim.py: use async in connect() function
...
The _async parameter name is only supported since psycopg 2.7.
However, async is a keyword in Python >= 3.7, so using this
gives us a syntax error. Working around this by defining the
parameters in a dict and handing that into the connect function.
2020-02-11 22:16:17 +01:00
Sarah Hoffmann
882f496e0a
nominatim.py: also catch deadlocks on final wait
2020-02-11 22:16:17 +01:00
Sarah Hoffmann
8b8aa1b4e6
regularly close connection while indexing
...
Postgres sooner or later runs out of memory when the connection
is used for too long.
2020-02-11 22:16:17 +01:00
Sarah Hoffmann
1801db523b
fix typo
2020-01-29 11:50:30 +01:00
Sarah Hoffmann
8f6fdfeb0b
forgot to index last rank
2020-01-24 22:06:30 +01:00
Sarah Hoffmann
b4e6d72fde
replace nominatim C program
2020-01-24 22:06:30 +01:00
Sarah Hoffmann
a338ebfce0
fix log levels
2020-01-24 22:06:30 +01:00
Sarah Hoffmann
4144364a15
add time display for nominatim.py
2020-01-24 22:06:30 +01:00
Sarah Hoffmann
11c0dd235b
clean up and document script
2020-01-24 22:06:30 +01:00
Sarah Hoffmann
4a9502bf88
fix SQL and some other stuff
2020-01-24 22:06:30 +01:00
Sarah Hoffmann
6c0d6d3178
Revert "switch to threading"
...
This reverts commit 8b1c2181be5aa5335c68d36a49cab9c4e2cd8bef.
2020-01-24 22:06:30 +01:00
Sarah Hoffmann
0a26ca7104
switch to threading
2020-01-24 22:06:30 +01:00
Sarah Hoffmann
2a15b2522f
use generator for thread choice
2020-01-24 22:06:30 +01:00
Sarah Hoffmann
c11d1d78e9
add prepared statement
2020-01-24 22:06:30 +01:00
Sarah Hoffmann
7e51aa4cef
simple implementation
2020-01-24 22:06:30 +01:00
Eric Stadtherr
62747c934d
Work on setup/update scripts, unit tests, and documentation to enable Postgres server to be optionally configured on a remote host
2018-07-21 12:09:47 -06:00
Sarah Hoffmann
4ac1bf2d47
clean up byte order detection
...
Check for existence of the expected functions and macros
and error out if nothing appropriate can be found.
2018-03-16 23:09:40 +01:00
Sarah Hoffmann
8f23ba076b
replace non-standard uint type with unsigned
...
See #879 .
2018-01-10 23:27:49 +01:00
Jonathan Montane
c54fc44b33
feat(export): added linked_place_id as an attribute to feature element
2017-12-18 10:34:05 +01:00
Sarah Hoffmann
9a47e1834f
reduce message frequency during indexing
2017-09-17 20:13:05 +02:00
Edward Betts
7e3af2debc
correct spelling mistakes
2017-03-08 15:06:50 +00:00
Melvyn Sopacua
13ab03a03a
Fix warnings:
...
- be consistent with (const char *) casts when assigning
- use xmlStrlen instead of strlen when dealing with xmlChar *
2017-02-15 10:17:43 +01:00
Melvyn Sopacua
6eb6f35f24
BSD compat: use sys/endian.h if available
...
<byteswap.h> is a linuxism. On BSD-like systems this is <sys/endian.h>
2017-02-13 14:30:48 +01:00
Sarah Hoffmann
ea5fe54c6b
force language of pgsql to C
...
Fixes #558 .
2017-01-15 21:31:14 +01:00
markigail
f07d620ee8
Change load-data in setup.php.
2016-05-11 10:22:03 +02:00
markigail
190a72cab5
Fix bug in index.c and remove column admin_level from location_property_osmline.
2016-05-08 16:46:42 +02:00