update documentation for new directory layout

Makefile

@@ -15,7 +15,7 @@ build-api:
 
 # Tests
 
-tests: mypy lint pytest
+tests: mypy lint pytest bdd
 
 mypy:
 	mypy --strict src

docs/admin/Deployment-Python.md

@@ -1,6 +1,6 @@
 # Deploying the Nominatim Python frontend
 
-The Nominatim can be run as a Python-based 
+Nominatim can be run as a Python-based
 [ASGI web application](https://asgi.readthedocs.io/en/latest/). You have the
 choice between [Falcon](https://falcon.readthedocs.io/en/stable/)
 and [Starlette](https://www.starlette.io/) as the ASGI framework.
@@ -12,30 +12,39 @@ to configure it.
 
 !!! Note
     Throughout this page, we assume your Nominatim project directory is
-    located in `/srv/nominatim-project` and you have installed Nominatim
+    located in `/srv/nominatim-project`. If you have put it somewhere else,
-    using the default installation prefix `/usr/local`. If you have put it
+    you need to adjust the commands and configuration accordingly.
-    somewhere else, you need to adjust the commands and configuration
-    accordingly.
 
-    We further assume that your web server runs as user `www-data`. Older
-    versions of CentOS may still use the user name `apache`. You also need
-    to adapt the instructions in this case.
 
 ### Installing the required packages
 
+The Nominatim frontend is best run from its own virtual environment. If
+you have already created one for the database backend during
+[installation](Installation.md#Building-Nominatim), you can use that. Otherwise
+create one now with:
+
+```sh
+sudo apt-get install virtualenv
+virtualenv /srv/nominatim-venv
+```
+
+The Nominatim frontend is contained in the 'nominatim-api' package. To
+install directly from the source tree run:
+
+```sh
+cd Nominatim
+/srv/nominatim-venv/bin/pip install packaging/nominatim-api
+```
+
 The recommended way to deploy a Python ASGI application is to run
 the ASGI runner [uvicorn](https://uvicorn.org/)
 together with [gunicorn](https://gunicorn.org/) HTTP server. We use
 Falcon here as the web framework.
 
-Create a virtual environment for the Python packages and install the necessary
+Add the necessary packages to your virtual environment:
-dependencies:
 
 ``` sh
-sudo apt install virtualenv
+/srv/nominatim-venv/bin/pip install falcon uvicorn gunicorn
-virtualenv /srv/nominatim-venv
-/srv/nominatim-venv/bin/pip install SQLAlchemy PyICU psycopg[binary] \
-   psycopg2-binary python-dotenv PyYAML falcon uvicorn gunicorn
 ```
 
 ### Setting up Nominatim as a systemd job
@@ -69,11 +78,10 @@ Requires=nominatim.socket
 
 [Service]
 Type=simple
-Environment="PYTHONPATH=/usr/local/lib/nominatim/lib-python/"
 User=www-data
 Group=www-data
 WorkingDirectory=/srv/nominatim-project
-ExecStart=/srv/nominatim-venv/bin/gunicorn -b unix:/run/nominatim.sock -w 4 -k uvicorn.workers.UvicornWorker nominatim.server.falcon.server:run_wsgi
+ExecStart=/srv/nominatim-venv/bin/gunicorn -b unix:/run/nominatim.sock -w 4 -k uvicorn.workers.UvicornWorker nominatim_api.server.falcon.server:run_wsgi
 ExecReload=/bin/kill -s HUP $MAINPID
 StandardOutput=append:/var/log/gunicorn-nominatim.log
 StandardError=inherit

docs/admin/Installation.md

@@ -155,9 +155,14 @@ wget -O Nominatim/data/country_osm_grid.sql.gz https://nominatim.org/data/countr
 
 #### Building the latest development version with pip
 
-To install Nominatim directly from the source tree, run:
+Nominatim is easiest to run from its own virtual environment. To create one, run:
 
-    pip install packaging/nominatim-{core,db,api}
+    sudo apt-get install virtualenv
+    virtualenv /srv/nominatim-venv
+
+To install Nominatim directly from the source tree into the virtual environment, run:
+
+    /srv/nominatim-venv/bin/pip install packaging/nominatim-{core,db,api}
 
 #### Building in legacy CMake mode
 

docs/admin/Migration.md

@@ -15,6 +15,22 @@ breaking changes. **Please read them before running the migration.**
     If you are migrating from a version <3.6, then you still have to follow
     the manual migration steps up to 3.6.
 
+## 4.4.0 -> master
+
+### New structure for Python packages
+
+The nominatim Python package has been split into nominatim-db and nominatim-api.
+Any imports need to be adapted accordingly.
+
+If you are running the Python frontend, change the server module from
+`nominatim.server.falcon.server` to `nominatim_api.server.falcon.server`.
+
+If you are using the Nominatim library, all imports need to be changed
+from `nominatim.api.<module>` to `nominatim_api.<module>`.
+
+If you have written custom tokenizers or sanitizers, the appropriate modules
+are now found in `nominatim_db`.
+
 ## 4.2.0 -> 4.3.0
 
 ### New indexes for reverse lookup

docs/customize/Special-Phrases.md

@@ -17,7 +17,7 @@ columns:
 
  * **phrase**: the keyword to look for
  * **class**: key of the main tag of the place to find
-   (see [principal tags in import style](../Import-Styles.md#set_main_tags-principal-tags)
+   (see [principal tags in import style](Import-Styles.md#set_main_tags-principal-tags)
  * **type**: value of the main tag
  * **operator**: type of special phrase, may be one of:
      * *in*: place is within the place defined by the search term (e.g. "_Hotels in_ Berlin")

docs/customize/Tokenizers.md

@@ -175,7 +175,7 @@ The following is a list of sanitizers that are shipped with Nominatim.
 
 ##### split-name-list
 
-::: nominatim.tokenizer.sanitizers.split_name_list
+::: nominatim_db.tokenizer.sanitizers.split_name_list
     options:
         members: False
         heading_level: 6
@@ -183,7 +183,7 @@ The following is a list of sanitizers that are shipped with Nominatim.
 
 ##### strip-brace-terms
 
-::: nominatim.tokenizer.sanitizers.strip_brace_terms
+::: nominatim_db.tokenizer.sanitizers.strip_brace_terms
     options:
         members: False
         heading_level: 6
@@ -191,7 +191,7 @@ The following is a list of sanitizers that are shipped with Nominatim.
 
 ##### tag-analyzer-by-language
 
-::: nominatim.tokenizer.sanitizers.tag_analyzer_by_language
+::: nominatim_db.tokenizer.sanitizers.tag_analyzer_by_language
     options:
         members: False
         heading_level: 6
@@ -199,7 +199,7 @@ The following is a list of sanitizers that are shipped with Nominatim.
 
 ##### clean-housenumbers
 
-::: nominatim.tokenizer.sanitizers.clean_housenumbers
+::: nominatim_db.tokenizer.sanitizers.clean_housenumbers
     options:
         members: False
         heading_level: 6
@@ -207,7 +207,7 @@ The following is a list of sanitizers that are shipped with Nominatim.
 
 ##### clean-postcodes
 
-::: nominatim.tokenizer.sanitizers.clean_postcodes
+::: nominatim_db.tokenizer.sanitizers.clean_postcodes
     options:
         members: False
         heading_level: 6
@@ -215,7 +215,7 @@ The following is a list of sanitizers that are shipped with Nominatim.
 
 ##### clean-tiger-tags
 
-::: nominatim.tokenizer.sanitizers.clean_tiger_tags
+::: nominatim_db.tokenizer.sanitizers.clean_tiger_tags
     options:
         members: False
         heading_level: 6
@@ -223,7 +223,7 @@ The following is a list of sanitizers that are shipped with Nominatim.
 
 #### delete-tags
 
-::: nominatim.tokenizer.sanitizers.delete_tags
+::: nominatim_db.tokenizer.sanitizers.delete_tags
     options:
         members: False
         heading_level: 6
@@ -231,7 +231,7 @@ The following is a list of sanitizers that are shipped with Nominatim.
 
 #### tag-japanese
 
-::: nominatim.tokenizer.sanitizers.tag_japanese
+::: nominatim_db.tokenizer.sanitizers.tag_japanese
     options:
         members: False
         heading_level: 6

docs/develop/Development-Environment.md

@@ -4,19 +4,19 @@ This chapter gives an overview how to set up Nominatim for development
 and how to run tests.
 
 !!! Important
-    This guide assumes that you develop under the latest version of Ubuntu. You
+    This guide assumes that you develop under the latest version of Debain/Ubuntu.
-    can of course also use your favourite distribution. You just might have to
+    You can of course also use your favourite distribution. You just might have
-    adapt the commands below slightly, in particular the commands for installing
+    to adapt the commands below slightly, in particular the commands for
-    additional software.
+    installing additional software.
 
 ## Installing Nominatim
 
 The first step is to install Nominatim itself. Please follow the installation
 instructions in the [Admin section](../admin/Installation.md). You don't need
-to set up a webserver for development, the webserver that is included with PHP
+to set up a webserver for development, the webserver that can be started
-is sufficient.
+via `nominatim serve` is sufficient.
 
-If you want to run Nominatim in a VM via Vagrant, use the default `ubuntu` setup.
+If you want to run Nominatim in a VM via Vagrant, use the default `ubuntu24` setup.
 Vagrant's libvirt provider runs out-of-the-box under Ubuntu. You also need to
 install an NFS daemon to enable directory sharing between host and guest. The
 following packages should get you started:
@@ -47,32 +47,39 @@ depending on your choice of webserver framework:
 The documentation is built with mkdocs:
 
 * [mkdocs](https://www.mkdocs.org/) >= 1.1.2
-* [mkdocstrings](https://mkdocstrings.github.io/) >= 0.18
+* [mkdocstrings](https://mkdocstrings.github.io/) >= 0.25
-* [mkdocstrings-python](https://mkdocstrings.github.io/python/)
+* [mkdocs-material](https://squidfunk.github.io/mkdocs-material/)
 
 ### Installing prerequisites on Ubuntu/Debian
 
-Some of the Python packages require the newest version which is not yet
+The Python tools should always be run with the most recent version.
-available with the current distributions. Therefore it is recommended to
+In particular, pylint tends to have a lot of breaking changes between versions.
-install pip to get the newest versions.
+The easiest way, to handle these Python dependencies is to run your
+development from within a virtual environment.
 
-To install all necessary packages run:
+To set up the virtual environment with all necessary packages run:
 
 ```sh
-sudo apt install php-cgi phpunit php-codesniffer \
+virtualenv ~/nominatim-dev-venv
-                 python3-pip python3-setuptools python3-dev
+~/nominatim-dev-venv/bin/pip install\
-
+    psycopg2-binary psutil psycopg[binary] PyICU SQLAlchemy \
-pip3 install --user behave mkdocs mkdocstrings pytest pytest-asyncio pylint \
+    python-dotenv jinja2 pyYAML datree \
-                    mypy types-PyYAML types-jinja2 types-psycopg2 types-psutil \
+    behave mkdocs mkdocstrings pytest pytest-asyncio pylint \
-                    types-ujson types-requests types-Pygments typing-extensions\
+    types-jinja2 types-markupsafe types-psutil types-psycopg2 \
-                    httpx asgi-lifespan
+    types-pygments types-pyyaml types-requests types-ujson \
+    types-urllib3 typing-extensions unicorn falcon
 ```
 
-The `mkdocs` executable will be located in `.local/bin`. You may have to add
+Now enter the virtual environment whenever you want to develop:
-this directory to your path, for example by running:
 
+```sh
+. ~/nominatim-dev-venv/bin/activate
 ```
-echo 'export PATH=~/.local/bin:$PATH' > ~/.profile
+
+For installing the PHP development tools, run:
+
+```sh
+sudo apt install php-cgi phpunit php-codesniffer
 ```
 
 If your distribution does not have PHPUnit 7.3+, you can install it (as well
@@ -85,24 +92,40 @@ composer global require "phpunit/phpunit=8.*"
 ```
 
 The binaries are found in `.config/composer/vendor/bin`. You need to add this
-to your PATH as well:
+to your PATH:
 
 ```
 echo 'export PATH=~/.config/composer/vendor/bin:$PATH' > ~/.profile
 ```
 
+### Running Nominatim during development
+
+The source code for Nominatim can be found in the `src` directory and can
+be run in-place. The source directory features a special script
+`nominatim-cli.py` which does the same as the installed 'nominatim' binary
+but executes against the code in the source tree. For example:
+
+```
+me@machine:~$ cd Nomiantim
+me@machine:~Nomiantim$ ./nominatim-cli.py --version
+Nominatim version 4.4.99-1
+```
+
+Make sure you have activated the virtual environment that holds all
+necessary dependencies.
 
 ## Executing Tests
 
 All tests are located in the `/test` directory.
 
-To run all tests just go to the build directory and run make:
+To run all tests, run make from the source root:
 
 ```sh
-cd build
+make tests
-make test
 ```
 
+There are also goals for executing parts of the test suite: mypy, lint, pytest, bdd.
+
 For more information about the structure of the tests and how to change and
 extend the test suite, see the [Testing chapter](Testing.md).
 

docs/develop/ICU-Tokenizer-Modules.md

@@ -52,7 +52,7 @@ the function.
 
 ### Sanitizer configuration
 
-::: nominatim.tokenizer.sanitizers.config.SanitizerConfig
+::: nominatim_db.tokenizer.sanitizers.config.SanitizerConfig
     options:
         heading_level: 6
 
@@ -73,14 +73,14 @@ adding extra attributes) or completely replace the list with a different one.
 
 #### PlaceInfo - information about the place
 
-::: nominatim.data.place_info.PlaceInfo
+::: nominatim_db.data.place_info.PlaceInfo
     options:
         heading_level: 6
 
 
 #### PlaceName - extended naming information
 
-::: nominatim.data.place_name.PlaceName
+::: nominatim_db.data.place_name.PlaceName
     options:
         heading_level: 6
 
@@ -139,12 +139,12 @@ They can be found in the directory
 
 ## Custom token analysis module
 
-::: nominatim.tokenizer.token_analysis.base.AnalysisModule
+::: nominatim_db.tokenizer.token_analysis.base.AnalysisModule
     options:
         heading_level: 6
 
 
-::: nominatim.tokenizer.token_analysis.base.Analyzer
+::: nominatim_db.tokenizer.token_analysis.base.Analyzer
     options:
         heading_level: 6
 

docs/develop/Tokenizers.md

@@ -130,16 +130,16 @@ class as defined below.
 
 ### Python Tokenizer Class
 
-All tokenizers must inherit from `nominatim.tokenizer.base.AbstractTokenizer`
+All tokenizers must inherit from `nominatim_db.tokenizer.base.AbstractTokenizer`
 and implement the abstract functions defined there.
 
-::: nominatim.tokenizer.base.AbstractTokenizer
+::: nominatim_db.tokenizer.base.AbstractTokenizer
     options:
         heading_level: 6
 
 ### Python Analyzer Class
 
-::: nominatim.tokenizer.base.AbstractAnalyzer
+::: nominatim_db.tokenizer.base.AbstractAnalyzer
     options:
         heading_level: 6
 

docs/library/Configuration.md

@@ -20,7 +20,7 @@ configuration parameters, see the [Configuration page](../customize/Settings.md)
 
 ## `Configuration` class
 
-::: nominatim.config.Configuration
+::: nominatim_core.config.Configuration
     options:
         members:
             - get_bool

docs/library/Getting-Started.md

@@ -12,22 +12,18 @@ in the database.
     The library interface is currently in an experimental stage. There might
     be some smaller adjustments to the public interface until the next version.
 
-    The library also misses a proper installation routine, so some manipulation
-    of the PYTHONPATH is required. At the moment, use is only recommended for
-    developers with some experience in Python.
-
 ## Installation
 
 To use the Nominatim library, you need access to a local Nominatim database.
 Follow the [installation](../admin/Installation.md) and
 [import](../admin/Import.md) instructions to set up your database.
 
-It is not yet possible to install it in the usual way via pip or inside a
+The Nominatim frontend library is contained in the 'nominatim-api' package.
-virtualenv. To get access to the library you need to set an appropriate
+To install the package from the source tree directly, run:
-`PYTHONPATH`. With the default installation, the python library can be found
+
-under `/usr/local/share/nominatim/lib-python`. If you have installed
+    pip install packaging/nominatim-api
-Nominatim under a different prefix, adapt the `/usr/local/` part accordingly.
+
-You can also point the `PYTHONPATH` to the Nominatim source code.
+Usually, you would want to run this in a virtual environment.
 
 ### A simple search example
 
@@ -43,7 +39,7 @@ This code snippet implements a simple search for the town of 'Brugge':
         from pathlib import Path
         import asyncio
 
-        import nominatim.api as napi
+        import nominatim_api as napi
 
         async def search(query):
             api = napi.NominatimAPIAsync(Path('.'))
@@ -61,7 +57,7 @@ This code snippet implements a simple search for the town of 'Brugge':
         ``` python
         from pathlib import Path
 
-        import nominatim.api as napi
+        import nominatim_api as napi
 
         api = napi.NominatimAPI(Path('.'))
 
@@ -116,7 +112,7 @@ standard 'nominatim' database:
         from pathlib import Path
         import asyncio
 
-        import nominatim.api as napi
+        import nominatim_api as napi
 
         config_params = {
             'NOMINATIM_DATABASE_DSN': 'pgsql:dbname=belgium'
@@ -134,7 +130,7 @@ standard 'nominatim' database:
         ``` python
         from pathlib import Path
 
-        import nominatim.api as napi
+        import nominatim_api as napi
 
         config_params = {
             'NOMINATIM_DATABASE_DSN': 'pgsql:dbname=belgium'
@@ -161,7 +157,7 @@ Again searching for 'Brugge', this time with a nicely formatted result:
         from pathlib import Path
         import asyncio
 
-        import nominatim.api as napi
+        import nominatim_api as napi
 
         async def search(query):
             api = napi.NominatimAPIAsync(Path('.'))
@@ -180,7 +176,7 @@ Again searching for 'Brugge', this time with a nicely formatted result:
         ``` python
         from pathlib import Path
 
-        import nominatim.api as napi
+        import nominatim_api as napi
 
         api = napi.NominatimAPI(Path('.'))
 

docs/library/Input-Parameter-Types.md

@@ -5,26 +5,26 @@ in the query functions of the API object.
 
 ## Place identification
 
-The [details](NominatimAPI.md#nominatim.api.core.NominatimAPI.details) and
+The [details](NominatimAPI.md#nominatim_api.core.NominatimAPI.details) and
-[lookup](NominatimAPI.md#nominatim.api.core.NominatimAPI.lookup) functions
+[lookup](NominatimAPI.md#nominatim_api.core.NominatimAPI.lookup) functions
 require references to places in the database. Below the possible
 types for place identification are listed. All types are dataclasses.
 
 ### PlaceID
 
-::: nominatim.api.PlaceID
+::: nominatim_api.PlaceID
     options:
         heading_level: 6
 
 ### OsmID
 
-::: nominatim.api.OsmID
+::: nominatim_api.OsmID
     options:
         heading_level: 6
 
 ## Geometry types
 
-::: nominatim.api.GeometryFormat
+::: nominatim_api.GeometryFormat
     options:
         heading_level: 6
         members_order: source
@@ -33,14 +33,14 @@ types for place identification are listed. All types are dataclasses.
 
 ### Point
 
-::: nominatim.api.Point
+::: nominatim_api.Point
     options:
         heading_level: 6
         show_signature_annotations: True
 
 ### Bbox
 
-::: nominatim.api.Bbox
+::: nominatim_api.Bbox
     options:
         heading_level: 6
         show_signature_annotations: True
@@ -54,9 +54,7 @@ orthogonal to restriction by address ranks, which groups places by their
 geographic extent.
 
 
-::: nominatim.api.DataLayer
+::: nominatim_api.DataLayer
     options:
         heading_level: 6
         members_order: source
-
-

docs/library/Low-Level-DB-Access.md

@@ -26,7 +26,7 @@ the placex table:
 import asyncio
 from pathlib import Path
 import sqlalchemy as sa
-from nominatim.api import NominatimAPIAsync
+from nominatim_api import NominatimAPIAsync
 
 async def print_table_size():
     api = NominatimAPIAsync(Path('.'))
@@ -45,7 +45,7 @@ asyncio.run(print_table_size())
 
 ## SearchConnection class
 
-::: nominatim.api.SearchConnection
+::: nominatim_api.SearchConnection
     options:
         members:
             - scalar

docs/library/NominatimAPI.md

@@ -6,7 +6,7 @@ to instantiate a separate instance for each thread.
 
 ### NominatimAPI
 
-::: nominatim.api.NominatimAPI
+::: nominatim_api.NominatimAPI
     options:
         members:
             - __init__
@@ -25,7 +25,7 @@ to instantiate a separate instance for each thread.
 
 ### NominatimAPIAsync
 
-::: nominatim.api.NominatimAPIAsync
+::: nominatim_api.NominatimAPIAsync
     options:
         members:
             - __init__

docs/library/Result-Handling.md

@@ -12,7 +12,7 @@ functions that allow further processing of the results.
 Nominatim takes the result data from multiple sources. The `source_table` field
 in the result describes, from which source the result was retrieved.
 
-::: nominatim.api.SourceTable
+::: nominatim_api.SourceTable
     options:
         heading_level: 6
         members_order: source
@@ -31,7 +31,7 @@ contains the result place itself and some artificial entries, for example,
 for the house number or the country code. This makes processing and creating
 a full address easier.
 
-::: nominatim.api.AddressLine
+::: nominatim_api.AddressLine
     options:
         heading_level: 6
         members_order: source
@@ -43,7 +43,7 @@ may be used to find a place, when the `keywords` parameter is set. Search
 terms are split into terms for the name of the place and search terms for
 its address.
 
-::: nominatim.api.WordInfo
+::: nominatim_api.WordInfo
     options:
         heading_level: 6
 
@@ -53,6 +53,6 @@ Results are always returned with the full list of available names.
 
 ### Locale
 
-::: nominatim.api.Locales
+::: nominatim_api.Locales
     options:
         heading_level: 6

docs/mkdocs.yml

@@ -75,7 +75,7 @@ plugins:
     - mkdocstrings:
         handlers:
           python:
-            paths: ["${PROJECT_SOURCE_DIR}"]
+            paths: ["${PROJECT_SOURCE_DIR}/src"]
             options:
               show_source: False
               show_bases: False