From 139cea5720e3b1030ec9a10c678d723752d707f4 Mon Sep 17 00:00:00 2001 From: Sarah Hoffmann Date: Tue, 25 Jun 2024 11:42:30 +0200 Subject: [PATCH] update documentation for new directory layout --- Makefile | 2 +- docs/admin/Deployment-Python.md | 40 ++++++++----- docs/admin/Installation.md | 9 ++- docs/admin/Migration.md | 16 +++++ docs/customize/Special-Phrases.md | 2 +- docs/customize/Tokenizers.md | 16 ++--- docs/develop/Development-Environment.md | 77 ++++++++++++++++--------- docs/develop/ICU-Tokenizer-Modules.md | 10 ++-- docs/develop/Tokenizers.md | 6 +- docs/library/Configuration.md | 2 +- docs/library/Getting-Started.md | 28 ++++----- docs/library/Input-Parameter-Types.md | 18 +++--- docs/library/Low-Level-DB-Access.md | 4 +- docs/library/NominatimAPI.md | 4 +- docs/library/Result-Handling.md | 8 +-- docs/mkdocs.yml | 2 +- 16 files changed, 145 insertions(+), 99 deletions(-) diff --git a/Makefile b/Makefile index bea0e6c9..0e09a884 100644 --- a/Makefile +++ b/Makefile @@ -15,7 +15,7 @@ build-api: # Tests -tests: mypy lint pytest +tests: mypy lint pytest bdd mypy: mypy --strict src diff --git a/docs/admin/Deployment-Python.md b/docs/admin/Deployment-Python.md index e95df00d..821a2c06 100644 --- a/docs/admin/Deployment-Python.md +++ b/docs/admin/Deployment-Python.md @@ -1,6 +1,6 @@ # Deploying the Nominatim Python frontend -The Nominatim can be run as a Python-based +Nominatim can be run as a Python-based [ASGI web application](https://asgi.readthedocs.io/en/latest/). You have the choice between [Falcon](https://falcon.readthedocs.io/en/stable/) and [Starlette](https://www.starlette.io/) as the ASGI framework. @@ -12,30 +12,39 @@ to configure it. !!! Note Throughout this page, we assume your Nominatim project directory is - located in `/srv/nominatim-project` and you have installed Nominatim - using the default installation prefix `/usr/local`. If you have put it - somewhere else, you need to adjust the commands and configuration - accordingly. + located in `/srv/nominatim-project`. If you have put it somewhere else, + you need to adjust the commands and configuration accordingly. - We further assume that your web server runs as user `www-data`. Older - versions of CentOS may still use the user name `apache`. You also need - to adapt the instructions in this case. ### Installing the required packages +The Nominatim frontend is best run from its own virtual environment. If +you have already created one for the database backend during +[installation](Installation.md#Building-Nominatim), you can use that. Otherwise +create one now with: + +```sh +sudo apt-get install virtualenv +virtualenv /srv/nominatim-venv +``` + +The Nominatim frontend is contained in the 'nominatim-api' package. To +install directly from the source tree run: + +```sh +cd Nominatim +/srv/nominatim-venv/bin/pip install packaging/nominatim-api +``` + The recommended way to deploy a Python ASGI application is to run the ASGI runner [uvicorn](https://uvicorn.org/) together with [gunicorn](https://gunicorn.org/) HTTP server. We use Falcon here as the web framework. -Create a virtual environment for the Python packages and install the necessary -dependencies: +Add the necessary packages to your virtual environment: ``` sh -sudo apt install virtualenv -virtualenv /srv/nominatim-venv -/srv/nominatim-venv/bin/pip install SQLAlchemy PyICU psycopg[binary] \ - psycopg2-binary python-dotenv PyYAML falcon uvicorn gunicorn +/srv/nominatim-venv/bin/pip install falcon uvicorn gunicorn ``` ### Setting up Nominatim as a systemd job @@ -69,11 +78,10 @@ Requires=nominatim.socket [Service] Type=simple -Environment="PYTHONPATH=/usr/local/lib/nominatim/lib-python/" User=www-data Group=www-data WorkingDirectory=/srv/nominatim-project -ExecStart=/srv/nominatim-venv/bin/gunicorn -b unix:/run/nominatim.sock -w 4 -k uvicorn.workers.UvicornWorker nominatim.server.falcon.server:run_wsgi +ExecStart=/srv/nominatim-venv/bin/gunicorn -b unix:/run/nominatim.sock -w 4 -k uvicorn.workers.UvicornWorker nominatim_api.server.falcon.server:run_wsgi ExecReload=/bin/kill -s HUP $MAINPID StandardOutput=append:/var/log/gunicorn-nominatim.log StandardError=inherit diff --git a/docs/admin/Installation.md b/docs/admin/Installation.md index d6cdf751..cb0df5c1 100644 --- a/docs/admin/Installation.md +++ b/docs/admin/Installation.md @@ -155,9 +155,14 @@ wget -O Nominatim/data/country_osm_grid.sql.gz https://nominatim.org/data/countr #### Building the latest development version with pip -To install Nominatim directly from the source tree, run: +Nominatim is easiest to run from its own virtual environment. To create one, run: - pip install packaging/nominatim-{core,db,api} + sudo apt-get install virtualenv + virtualenv /srv/nominatim-venv + +To install Nominatim directly from the source tree into the virtual environment, run: + + /srv/nominatim-venv/bin/pip install packaging/nominatim-{core,db,api} #### Building in legacy CMake mode diff --git a/docs/admin/Migration.md b/docs/admin/Migration.md index 3e62d219..59702e7b 100644 --- a/docs/admin/Migration.md +++ b/docs/admin/Migration.md @@ -15,6 +15,22 @@ breaking changes. **Please read them before running the migration.** If you are migrating from a version <3.6, then you still have to follow the manual migration steps up to 3.6. +## 4.4.0 -> master + +### New structure for Python packages + +The nominatim Python package has been split into nominatim-db and nominatim-api. +Any imports need to be adapted accordingly. + +If you are running the Python frontend, change the server module from +`nominatim.server.falcon.server` to `nominatim_api.server.falcon.server`. + +If you are using the Nominatim library, all imports need to be changed +from `nominatim.api.` to `nominatim_api.`. + +If you have written custom tokenizers or sanitizers, the appropriate modules +are now found in `nominatim_db`. + ## 4.2.0 -> 4.3.0 ### New indexes for reverse lookup diff --git a/docs/customize/Special-Phrases.md b/docs/customize/Special-Phrases.md index abbe03d1..4824512b 100644 --- a/docs/customize/Special-Phrases.md +++ b/docs/customize/Special-Phrases.md @@ -17,7 +17,7 @@ columns: * **phrase**: the keyword to look for * **class**: key of the main tag of the place to find - (see [principal tags in import style](../Import-Styles.md#set_main_tags-principal-tags) + (see [principal tags in import style](Import-Styles.md#set_main_tags-principal-tags) * **type**: value of the main tag * **operator**: type of special phrase, may be one of: * *in*: place is within the place defined by the search term (e.g. "_Hotels in_ Berlin") diff --git a/docs/customize/Tokenizers.md b/docs/customize/Tokenizers.md index df336a71..6b811a9a 100644 --- a/docs/customize/Tokenizers.md +++ b/docs/customize/Tokenizers.md @@ -175,7 +175,7 @@ The following is a list of sanitizers that are shipped with Nominatim. ##### split-name-list -::: nominatim.tokenizer.sanitizers.split_name_list +::: nominatim_db.tokenizer.sanitizers.split_name_list options: members: False heading_level: 6 @@ -183,7 +183,7 @@ The following is a list of sanitizers that are shipped with Nominatim. ##### strip-brace-terms -::: nominatim.tokenizer.sanitizers.strip_brace_terms +::: nominatim_db.tokenizer.sanitizers.strip_brace_terms options: members: False heading_level: 6 @@ -191,7 +191,7 @@ The following is a list of sanitizers that are shipped with Nominatim. ##### tag-analyzer-by-language -::: nominatim.tokenizer.sanitizers.tag_analyzer_by_language +::: nominatim_db.tokenizer.sanitizers.tag_analyzer_by_language options: members: False heading_level: 6 @@ -199,7 +199,7 @@ The following is a list of sanitizers that are shipped with Nominatim. ##### clean-housenumbers -::: nominatim.tokenizer.sanitizers.clean_housenumbers +::: nominatim_db.tokenizer.sanitizers.clean_housenumbers options: members: False heading_level: 6 @@ -207,7 +207,7 @@ The following is a list of sanitizers that are shipped with Nominatim. ##### clean-postcodes -::: nominatim.tokenizer.sanitizers.clean_postcodes +::: nominatim_db.tokenizer.sanitizers.clean_postcodes options: members: False heading_level: 6 @@ -215,7 +215,7 @@ The following is a list of sanitizers that are shipped with Nominatim. ##### clean-tiger-tags -::: nominatim.tokenizer.sanitizers.clean_tiger_tags +::: nominatim_db.tokenizer.sanitizers.clean_tiger_tags options: members: False heading_level: 6 @@ -223,7 +223,7 @@ The following is a list of sanitizers that are shipped with Nominatim. #### delete-tags -::: nominatim.tokenizer.sanitizers.delete_tags +::: nominatim_db.tokenizer.sanitizers.delete_tags options: members: False heading_level: 6 @@ -231,7 +231,7 @@ The following is a list of sanitizers that are shipped with Nominatim. #### tag-japanese -::: nominatim.tokenizer.sanitizers.tag_japanese +::: nominatim_db.tokenizer.sanitizers.tag_japanese options: members: False heading_level: 6 diff --git a/docs/develop/Development-Environment.md b/docs/develop/Development-Environment.md index 64306454..20d02411 100644 --- a/docs/develop/Development-Environment.md +++ b/docs/develop/Development-Environment.md @@ -4,19 +4,19 @@ This chapter gives an overview how to set up Nominatim for development and how to run tests. !!! Important - This guide assumes that you develop under the latest version of Ubuntu. You - can of course also use your favourite distribution. You just might have to - adapt the commands below slightly, in particular the commands for installing - additional software. + This guide assumes that you develop under the latest version of Debain/Ubuntu. + You can of course also use your favourite distribution. You just might have + to adapt the commands below slightly, in particular the commands for + installing additional software. ## Installing Nominatim The first step is to install Nominatim itself. Please follow the installation instructions in the [Admin section](../admin/Installation.md). You don't need -to set up a webserver for development, the webserver that is included with PHP -is sufficient. +to set up a webserver for development, the webserver that can be started +via `nominatim serve` is sufficient. -If you want to run Nominatim in a VM via Vagrant, use the default `ubuntu` setup. +If you want to run Nominatim in a VM via Vagrant, use the default `ubuntu24` setup. Vagrant's libvirt provider runs out-of-the-box under Ubuntu. You also need to install an NFS daemon to enable directory sharing between host and guest. The following packages should get you started: @@ -47,32 +47,39 @@ depending on your choice of webserver framework: The documentation is built with mkdocs: * [mkdocs](https://www.mkdocs.org/) >= 1.1.2 -* [mkdocstrings](https://mkdocstrings.github.io/) >= 0.18 -* [mkdocstrings-python](https://mkdocstrings.github.io/python/) +* [mkdocstrings](https://mkdocstrings.github.io/) >= 0.25 +* [mkdocs-material](https://squidfunk.github.io/mkdocs-material/) ### Installing prerequisites on Ubuntu/Debian -Some of the Python packages require the newest version which is not yet -available with the current distributions. Therefore it is recommended to -install pip to get the newest versions. +The Python tools should always be run with the most recent version. +In particular, pylint tends to have a lot of breaking changes between versions. +The easiest way, to handle these Python dependencies is to run your +development from within a virtual environment. -To install all necessary packages run: +To set up the virtual environment with all necessary packages run: ```sh -sudo apt install php-cgi phpunit php-codesniffer \ - python3-pip python3-setuptools python3-dev - -pip3 install --user behave mkdocs mkdocstrings pytest pytest-asyncio pylint \ - mypy types-PyYAML types-jinja2 types-psycopg2 types-psutil \ - types-ujson types-requests types-Pygments typing-extensions\ - httpx asgi-lifespan +virtualenv ~/nominatim-dev-venv +~/nominatim-dev-venv/bin/pip install\ + psycopg2-binary psutil psycopg[binary] PyICU SQLAlchemy \ + python-dotenv jinja2 pyYAML datree \ + behave mkdocs mkdocstrings pytest pytest-asyncio pylint \ + types-jinja2 types-markupsafe types-psutil types-psycopg2 \ + types-pygments types-pyyaml types-requests types-ujson \ + types-urllib3 typing-extensions unicorn falcon ``` -The `mkdocs` executable will be located in `.local/bin`. You may have to add -this directory to your path, for example by running: +Now enter the virtual environment whenever you want to develop: +```sh +. ~/nominatim-dev-venv/bin/activate ``` -echo 'export PATH=~/.local/bin:$PATH' > ~/.profile + +For installing the PHP development tools, run: + +```sh +sudo apt install php-cgi phpunit php-codesniffer ``` If your distribution does not have PHPUnit 7.3+, you can install it (as well @@ -85,24 +92,40 @@ composer global require "phpunit/phpunit=8.*" ``` The binaries are found in `.config/composer/vendor/bin`. You need to add this -to your PATH as well: +to your PATH: ``` echo 'export PATH=~/.config/composer/vendor/bin:$PATH' > ~/.profile ``` +### Running Nominatim during development + +The source code for Nominatim can be found in the `src` directory and can +be run in-place. The source directory features a special script +`nominatim-cli.py` which does the same as the installed 'nominatim' binary +but executes against the code in the source tree. For example: + +``` +me@machine:~$ cd Nomiantim +me@machine:~Nomiantim$ ./nominatim-cli.py --version +Nominatim version 4.4.99-1 +``` + +Make sure you have activated the virtual environment that holds all +necessary dependencies. ## Executing Tests All tests are located in the `/test` directory. -To run all tests just go to the build directory and run make: +To run all tests, run make from the source root: ```sh -cd build -make test +make tests ``` +There are also goals for executing parts of the test suite: mypy, lint, pytest, bdd. + For more information about the structure of the tests and how to change and extend the test suite, see the [Testing chapter](Testing.md). diff --git a/docs/develop/ICU-Tokenizer-Modules.md b/docs/develop/ICU-Tokenizer-Modules.md index daadf899..d189422a 100644 --- a/docs/develop/ICU-Tokenizer-Modules.md +++ b/docs/develop/ICU-Tokenizer-Modules.md @@ -52,7 +52,7 @@ the function. ### Sanitizer configuration -::: nominatim.tokenizer.sanitizers.config.SanitizerConfig +::: nominatim_db.tokenizer.sanitizers.config.SanitizerConfig options: heading_level: 6 @@ -73,14 +73,14 @@ adding extra attributes) or completely replace the list with a different one. #### PlaceInfo - information about the place -::: nominatim.data.place_info.PlaceInfo +::: nominatim_db.data.place_info.PlaceInfo options: heading_level: 6 #### PlaceName - extended naming information -::: nominatim.data.place_name.PlaceName +::: nominatim_db.data.place_name.PlaceName options: heading_level: 6 @@ -139,12 +139,12 @@ They can be found in the directory ## Custom token analysis module -::: nominatim.tokenizer.token_analysis.base.AnalysisModule +::: nominatim_db.tokenizer.token_analysis.base.AnalysisModule options: heading_level: 6 -::: nominatim.tokenizer.token_analysis.base.Analyzer +::: nominatim_db.tokenizer.token_analysis.base.Analyzer options: heading_level: 6 diff --git a/docs/develop/Tokenizers.md b/docs/develop/Tokenizers.md index 05037177..03988ce0 100644 --- a/docs/develop/Tokenizers.md +++ b/docs/develop/Tokenizers.md @@ -130,16 +130,16 @@ class as defined below. ### Python Tokenizer Class -All tokenizers must inherit from `nominatim.tokenizer.base.AbstractTokenizer` +All tokenizers must inherit from `nominatim_db.tokenizer.base.AbstractTokenizer` and implement the abstract functions defined there. -::: nominatim.tokenizer.base.AbstractTokenizer +::: nominatim_db.tokenizer.base.AbstractTokenizer options: heading_level: 6 ### Python Analyzer Class -::: nominatim.tokenizer.base.AbstractAnalyzer +::: nominatim_db.tokenizer.base.AbstractAnalyzer options: heading_level: 6 diff --git a/docs/library/Configuration.md b/docs/library/Configuration.md index b97c2ccd..bf093166 100644 --- a/docs/library/Configuration.md +++ b/docs/library/Configuration.md @@ -20,7 +20,7 @@ configuration parameters, see the [Configuration page](../customize/Settings.md) ## `Configuration` class -::: nominatim.config.Configuration +::: nominatim_core.config.Configuration options: members: - get_bool diff --git a/docs/library/Getting-Started.md b/docs/library/Getting-Started.md index 2ae17e87..22d9681d 100644 --- a/docs/library/Getting-Started.md +++ b/docs/library/Getting-Started.md @@ -12,22 +12,18 @@ in the database. The library interface is currently in an experimental stage. There might be some smaller adjustments to the public interface until the next version. - The library also misses a proper installation routine, so some manipulation - of the PYTHONPATH is required. At the moment, use is only recommended for - developers with some experience in Python. - ## Installation To use the Nominatim library, you need access to a local Nominatim database. Follow the [installation](../admin/Installation.md) and [import](../admin/Import.md) instructions to set up your database. -It is not yet possible to install it in the usual way via pip or inside a -virtualenv. To get access to the library you need to set an appropriate -`PYTHONPATH`. With the default installation, the python library can be found -under `/usr/local/share/nominatim/lib-python`. If you have installed -Nominatim under a different prefix, adapt the `/usr/local/` part accordingly. -You can also point the `PYTHONPATH` to the Nominatim source code. +The Nominatim frontend library is contained in the 'nominatim-api' package. +To install the package from the source tree directly, run: + + pip install packaging/nominatim-api + +Usually, you would want to run this in a virtual environment. ### A simple search example @@ -43,7 +39,7 @@ This code snippet implements a simple search for the town of 'Brugge': from pathlib import Path import asyncio - import nominatim.api as napi + import nominatim_api as napi async def search(query): api = napi.NominatimAPIAsync(Path('.')) @@ -61,7 +57,7 @@ This code snippet implements a simple search for the town of 'Brugge': ``` python from pathlib import Path - import nominatim.api as napi + import nominatim_api as napi api = napi.NominatimAPI(Path('.')) @@ -116,7 +112,7 @@ standard 'nominatim' database: from pathlib import Path import asyncio - import nominatim.api as napi + import nominatim_api as napi config_params = { 'NOMINATIM_DATABASE_DSN': 'pgsql:dbname=belgium' @@ -134,7 +130,7 @@ standard 'nominatim' database: ``` python from pathlib import Path - import nominatim.api as napi + import nominatim_api as napi config_params = { 'NOMINATIM_DATABASE_DSN': 'pgsql:dbname=belgium' @@ -161,7 +157,7 @@ Again searching for 'Brugge', this time with a nicely formatted result: from pathlib import Path import asyncio - import nominatim.api as napi + import nominatim_api as napi async def search(query): api = napi.NominatimAPIAsync(Path('.')) @@ -180,7 +176,7 @@ Again searching for 'Brugge', this time with a nicely formatted result: ``` python from pathlib import Path - import nominatim.api as napi + import nominatim_api as napi api = napi.NominatimAPI(Path('.')) diff --git a/docs/library/Input-Parameter-Types.md b/docs/library/Input-Parameter-Types.md index 9227dc32..5c537732 100644 --- a/docs/library/Input-Parameter-Types.md +++ b/docs/library/Input-Parameter-Types.md @@ -5,26 +5,26 @@ in the query functions of the API object. ## Place identification -The [details](NominatimAPI.md#nominatim.api.core.NominatimAPI.details) and -[lookup](NominatimAPI.md#nominatim.api.core.NominatimAPI.lookup) functions +The [details](NominatimAPI.md#nominatim_api.core.NominatimAPI.details) and +[lookup](NominatimAPI.md#nominatim_api.core.NominatimAPI.lookup) functions require references to places in the database. Below the possible types for place identification are listed. All types are dataclasses. ### PlaceID -::: nominatim.api.PlaceID +::: nominatim_api.PlaceID options: heading_level: 6 ### OsmID -::: nominatim.api.OsmID +::: nominatim_api.OsmID options: heading_level: 6 ## Geometry types -::: nominatim.api.GeometryFormat +::: nominatim_api.GeometryFormat options: heading_level: 6 members_order: source @@ -33,14 +33,14 @@ types for place identification are listed. All types are dataclasses. ### Point -::: nominatim.api.Point +::: nominatim_api.Point options: heading_level: 6 show_signature_annotations: True ### Bbox -::: nominatim.api.Bbox +::: nominatim_api.Bbox options: heading_level: 6 show_signature_annotations: True @@ -54,9 +54,7 @@ orthogonal to restriction by address ranks, which groups places by their geographic extent. -::: nominatim.api.DataLayer +::: nominatim_api.DataLayer options: heading_level: 6 members_order: source - - diff --git a/docs/library/Low-Level-DB-Access.md b/docs/library/Low-Level-DB-Access.md index acd93fd1..84a40b9b 100644 --- a/docs/library/Low-Level-DB-Access.md +++ b/docs/library/Low-Level-DB-Access.md @@ -26,7 +26,7 @@ the placex table: import asyncio from pathlib import Path import sqlalchemy as sa -from nominatim.api import NominatimAPIAsync +from nominatim_api import NominatimAPIAsync async def print_table_size(): api = NominatimAPIAsync(Path('.')) @@ -45,7 +45,7 @@ asyncio.run(print_table_size()) ## SearchConnection class -::: nominatim.api.SearchConnection +::: nominatim_api.SearchConnection options: members: - scalar diff --git a/docs/library/NominatimAPI.md b/docs/library/NominatimAPI.md index 0fa9d659..be1f973d 100644 --- a/docs/library/NominatimAPI.md +++ b/docs/library/NominatimAPI.md @@ -6,7 +6,7 @@ to instantiate a separate instance for each thread. ### NominatimAPI -::: nominatim.api.NominatimAPI +::: nominatim_api.NominatimAPI options: members: - __init__ @@ -25,7 +25,7 @@ to instantiate a separate instance for each thread. ### NominatimAPIAsync -::: nominatim.api.NominatimAPIAsync +::: nominatim_api.NominatimAPIAsync options: members: - __init__ diff --git a/docs/library/Result-Handling.md b/docs/library/Result-Handling.md index e2803f9f..1e8043cf 100644 --- a/docs/library/Result-Handling.md +++ b/docs/library/Result-Handling.md @@ -12,7 +12,7 @@ functions that allow further processing of the results. Nominatim takes the result data from multiple sources. The `source_table` field in the result describes, from which source the result was retrieved. -::: nominatim.api.SourceTable +::: nominatim_api.SourceTable options: heading_level: 6 members_order: source @@ -31,7 +31,7 @@ contains the result place itself and some artificial entries, for example, for the house number or the country code. This makes processing and creating a full address easier. -::: nominatim.api.AddressLine +::: nominatim_api.AddressLine options: heading_level: 6 members_order: source @@ -43,7 +43,7 @@ may be used to find a place, when the `keywords` parameter is set. Search terms are split into terms for the name of the place and search terms for its address. -::: nominatim.api.WordInfo +::: nominatim_api.WordInfo options: heading_level: 6 @@ -53,6 +53,6 @@ Results are always returned with the full list of available names. ### Locale -::: nominatim.api.Locales +::: nominatim_api.Locales options: heading_level: 6 diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index cb4ce9de..fa65a11b 100644 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -75,7 +75,7 @@ plugins: - mkdocstrings: handlers: python: - paths: ["${PROJECT_SOURCE_DIR}"] + paths: ["${PROJECT_SOURCE_DIR}/src"] options: show_source: False show_bases: False