osm-search/Nominatim
1
1
Fork 0
mirror of https://github.com/osm-search/Nominatim.git synced 2024-07-14 22:40:38 +03:00
Code Issues Packages Projects Releases Wiki Activity

update documentation for new directory layout

Browse Source
This commit is contained in:
Sarah Hoffmann 2024-06-25 11:42:30 +02:00
parent e2416f90af
commit 139cea5720
16 changed files with 145 additions and 99 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
Makefile
View File

@ -15,7 +15,7 @@ build-api:
# Tests
tests: mypy lint pytest
tests: mypy lint pytest bdd
mypy:
mypy --strict src

40
docs/admin/Deployment-Python.md
View File

@ -1,6 +1,6 @@
# Deploying the Nominatim Python frontend
The Nominatim can be run as a Python-based
Nominatim can be run as a Python-based
[ASGI web application](https://asgi.readthedocs.io/en/latest/). You have the
choice between [Falcon](https://falcon.readthedocs.io/en/stable/)
and [Starlette](https://www.starlette.io/) as the ASGI framework.
@ -12,30 +12,39 @@ to configure it.
!!! Note
Throughout this page, we assume your Nominatim project directory is
located in `/srv/nominatim-project` and you have installed Nominatim
using the default installation prefix `/usr/local`. If you have put it
somewhere else, you need to adjust the commands and configuration
accordingly.
located in `/srv/nominatim-project`. If you have put it somewhere else,
you need to adjust the commands and configuration accordingly.
We further assume that your web server runs as user `www-data`. Older
versions of CentOS may still use the user name `apache`. You also need
to adapt the instructions in this case.
### Installing the required packages
The Nominatim frontend is best run from its own virtual environment. If
you have already created one for the database backend during
[installation](Installation.md#Building-Nominatim), you can use that. Otherwise
create one now with:
```sh
sudo apt-get install virtualenv
virtualenv /srv/nominatim-venv
```
The Nominatim frontend is contained in the 'nominatim-api' package. To
install directly from the source tree run:
```sh
cd Nominatim
/srv/nominatim-venv/bin/pip install packaging/nominatim-api
```
The recommended way to deploy a Python ASGI application is to run
the ASGI runner [uvicorn](https://uvicorn.org/)
together with [gunicorn](https://gunicorn.org/) HTTP server. We use
Falcon here as the web framework.
Create a virtual environment for the Python packages and install the necessary
dependencies:
Add the necessary packages to your virtual environment:
``` sh
sudo apt install virtualenv
virtualenv /srv/nominatim-venv
/srv/nominatim-venv/bin/pip install SQLAlchemy PyICU psycopg[binary] \
psycopg2-binary python-dotenv PyYAML falcon uvicorn gunicorn
/srv/nominatim-venv/bin/pip install falcon uvicorn gunicorn
```
### Setting up Nominatim as a systemd job
@ -69,11 +78,10 @@ Requires=nominatim.socket
[Service]
Type=simple
Environment="PYTHONPATH=/usr/local/lib/nominatim/lib-python/"
User=www-data
Group=www-data
WorkingDirectory=/srv/nominatim-project
ExecStart=/srv/nominatim-venv/bin/gunicorn -b unix:/run/nominatim.sock -w 4 -k uvicorn.workers.UvicornWorker nominatim.server.falcon.server:run_wsgi
ExecStart=/srv/nominatim-venv/bin/gunicorn -b unix:/run/nominatim.sock -w 4 -k uvicorn.workers.UvicornWorker nominatim_api.server.falcon.server:run_wsgi
ExecReload=/bin/kill -s HUP $MAINPID
StandardOutput=append:/var/log/gunicorn-nominatim.log
StandardError=inherit

9
docs/admin/Installation.md
View File

@ -155,9 +155,14 @@ wget -O Nominatim/data/country_osm_grid.sql.gz https://nominatim.org/data/countr
#### Building the latest development version with pip
To install Nominatim directly from the source tree, run:
Nominatim is easiest to run from its own virtual environment. To create one, run:
pip install packaging/nominatim-{core,db,api}
sudo apt-get install virtualenv
virtualenv /srv/nominatim-venv
To install Nominatim directly from the source tree into the virtual environment, run:
/srv/nominatim-venv/bin/pip install packaging/nominatim-{core,db,api}
#### Building in legacy CMake mode

16
docs/admin/Migration.md
View File

@ -15,6 +15,22 @@ breaking changes. **Please read them before running the migration.**
If you are migrating from a version <3.6, then you still have to follow
the manual migration steps up to 3.6.
## 4.4.0 -> master
### New structure for Python packages
The nominatim Python package has been split into nominatim-db and nominatim-api.
Any imports need to be adapted accordingly.
If you are running the Python frontend, change the server module from
`nominatim.server.falcon.server` to `nominatim_api.server.falcon.server`.
If you are using the Nominatim library, all imports need to be changed
from `nominatim.api.<module>` to `nominatim_api.<module>`.
If you have written custom tokenizers or sanitizers, the appropriate modules
are now found in `nominatim_db`.
## 4.2.0 -> 4.3.0
### New indexes for reverse lookup

2
docs/customize/Special-Phrases.md
View File

@ -17,7 +17,7 @@ columns:
* **phrase**: the keyword to look for
* **class**: key of the main tag of the place to find
(see [principal tags in import style](../Import-Styles.md#set_main_tags-principal-tags)
(see [principal tags in import style](Import-Styles.md#set_main_tags-principal-tags)
* **type**: value of the main tag
* **operator**: type of special phrase, may be one of:
* *in*: place is within the place defined by the search term (e.g. "_Hotels in_ Berlin")

16
docs/customize/Tokenizers.md
View File

@ -175,7 +175,7 @@ The following is a list of sanitizers that are shipped with Nominatim.
##### split-name-list
::: nominatim.tokenizer.sanitizers.split_name_list
::: nominatim_db.tokenizer.sanitizers.split_name_list
options:
members: False
heading_level: 6
@ -183,7 +183,7 @@ The following is a list of sanitizers that are shipped with Nominatim.
##### strip-brace-terms
::: nominatim.tokenizer.sanitizers.strip_brace_terms
::: nominatim_db.tokenizer.sanitizers.strip_brace_terms
options:
members: False
heading_level: 6
@ -191,7 +191,7 @@ The following is a list of sanitizers that are shipped with Nominatim.
##### tag-analyzer-by-language
::: nominatim.tokenizer.sanitizers.tag_analyzer_by_language
::: nominatim_db.tokenizer.sanitizers.tag_analyzer_by_language
options:
members: False
heading_level: 6
@ -199,7 +199,7 @@ The following is a list of sanitizers that are shipped with Nominatim.
##### clean-housenumbers
::: nominatim.tokenizer.sanitizers.clean_housenumbers
::: nominatim_db.tokenizer.sanitizers.clean_housenumbers
options:
members: False
heading_level: 6
@ -207,7 +207,7 @@ The following is a list of sanitizers that are shipped with Nominatim.
##### clean-postcodes
::: nominatim.tokenizer.sanitizers.clean_postcodes
::: nominatim_db.tokenizer.sanitizers.clean_postcodes
options:
members: False
heading_level: 6
@ -215,7 +215,7 @@ The following is a list of sanitizers that are shipped with Nominatim.
##### clean-tiger-tags
::: nominatim.tokenizer.sanitizers.clean_tiger_tags
::: nominatim_db.tokenizer.sanitizers.clean_tiger_tags
options:
members: False
heading_level: 6
@ -223,7 +223,7 @@ The following is a list of sanitizers that are shipped with Nominatim.
#### delete-tags
::: nominatim.tokenizer.sanitizers.delete_tags
::: nominatim_db.tokenizer.sanitizers.delete_tags
options:
members: False
heading_level: 6
@ -231,7 +231,7 @@ The following is a list of sanitizers that are shipped with Nominatim.
#### tag-japanese
::: nominatim.tokenizer.sanitizers.tag_japanese
::: nominatim_db.tokenizer.sanitizers.tag_japanese
options:
members: False
heading_level: 6

77
docs/develop/Development-Environment.md
View File

@ -4,19 +4,19 @@ This chapter gives an overview how to set up Nominatim for development
and how to run tests.
!!! Important
This guide assumes that you develop under the latest version of Ubuntu. You
can of course also use your favourite distribution. You just might have to
adapt the commands below slightly, in particular the commands for installing
additional software.
This guide assumes that you develop under the latest version of Debain/Ubuntu.
You can of course also use your favourite distribution. You just might have
to adapt the commands below slightly, in particular the commands for
installing additional software.
## Installing Nominatim
The first step is to install Nominatim itself. Please follow the installation
instructions in the [Admin section](../admin/Installation.md). You don't need
to set up a webserver for development, the webserver that is included with PHP
is sufficient.
to set up a webserver for development, the webserver that can be started
via `nominatim serve` is sufficient.
If you want to run Nominatim in a VM via Vagrant, use the default `ubuntu` setup.
If you want to run Nominatim in a VM via Vagrant, use the default `ubuntu24` setup.
Vagrant's libvirt provider runs out-of-the-box under Ubuntu. You also need to
install an NFS daemon to enable directory sharing between host and guest. The
following packages should get you started:
@ -47,32 +47,39 @@ depending on your choice of webserver framework:
The documentation is built with mkdocs:
* [mkdocs](https://www.mkdocs.org/) >= 1.1.2
* [mkdocstrings](https://mkdocstrings.github.io/) >= 0.18
* [mkdocstrings-python](https://mkdocstrings.github.io/python/)
* [mkdocstrings](https://mkdocstrings.github.io/) >= 0.25
* [mkdocs-material](https://squidfunk.github.io/mkdocs-material/)
### Installing prerequisites on Ubuntu/Debian
Some of the Python packages require the newest version which is not yet
available with the current distributions. Therefore it is recommended to
install pip to get the newest versions.
The Python tools should always be run with the most recent version.
In particular, pylint tends to have a lot of breaking changes between versions.
The easiest way, to handle these Python dependencies is to run your
development from within a virtual environment.
To install all necessary packages run:
To set up the virtual environment with all necessary packages run:
```sh
sudo apt install php-cgi phpunit php-codesniffer \
python3-pip python3-setuptools python3-dev
pip3 install --user behave mkdocs mkdocstrings pytest pytest-asyncio pylint \
mypy types-PyYAML types-jinja2 types-psycopg2 types-psutil \
types-ujson types-requests types-Pygments typing-extensions\
httpx asgi-lifespan
virtualenv ~/nominatim-dev-venv
~/nominatim-dev-venv/bin/pip install\
psycopg2-binary psutil psycopg[binary] PyICU SQLAlchemy \
python-dotenv jinja2 pyYAML datree \
behave mkdocs mkdocstrings pytest pytest-asyncio pylint \
types-jinja2 types-markupsafe types-psutil types-psycopg2 \
types-pygments types-pyyaml types-requests types-ujson \
types-urllib3 typing-extensions unicorn falcon
```
The `mkdocs` executable will be located in `.local/bin`. You may have to add
this directory to your path, for example by running:
Now enter the virtual environment whenever you want to develop:
```sh
. ~/nominatim-dev-venv/bin/activate
```
echo 'export PATH=~/.local/bin:$PATH' > ~/.profile
For installing the PHP development tools, run:
```sh
sudo apt install php-cgi phpunit php-codesniffer
```
If your distribution does not have PHPUnit 7.3+, you can install it (as well
@ -85,24 +92,40 @@ composer global require "phpunit/phpunit=8.*"
```
The binaries are found in `.config/composer/vendor/bin`. You need to add this
to your PATH as well:
to your PATH:
```
echo 'export PATH=~/.config/composer/vendor/bin:$PATH' > ~/.profile
```
### Running Nominatim during development
The source code for Nominatim can be found in the `src` directory and can
be run in-place. The source directory features a special script
`nominatim-cli.py` which does the same as the installed 'nominatim' binary
but executes against the code in the source tree. For example:
```
me@machine:~$ cd Nomiantim
me@machine:~Nomiantim$ ./nominatim-cli.py --version
Nominatim version 4.4.99-1
```
Make sure you have activated the virtual environment that holds all
necessary dependencies.
## Executing Tests
All tests are located in the `/test` directory.
To run all tests just go to the build directory and run make:
To run all tests, run make from the source root:
```sh
cd build
make test
make tests
```
There are also goals for executing parts of the test suite: mypy, lint, pytest, bdd.
For more information about the structure of the tests and how to change and
extend the test suite, see the [Testing chapter](Testing.md).

10
docs/develop/ICU-Tokenizer-Modules.md
View File

@ -52,7 +52,7 @@ the function.
### Sanitizer configuration
::: nominatim.tokenizer.sanitizers.config.SanitizerConfig
::: nominatim_db.tokenizer.sanitizers.config.SanitizerConfig
options:
heading_level: 6
@ -73,14 +73,14 @@ adding extra attributes) or completely replace the list with a different one.
#### PlaceInfo - information about the place
::: nominatim.data.place_info.PlaceInfo
::: nominatim_db.data.place_info.PlaceInfo
options:
heading_level: 6
#### PlaceName - extended naming information
::: nominatim.data.place_name.PlaceName
::: nominatim_db.data.place_name.PlaceName
options:
heading_level: 6
@ -139,12 +139,12 @@ They can be found in the directory
## Custom token analysis module
::: nominatim.tokenizer.token_analysis.base.AnalysisModule
::: nominatim_db.tokenizer.token_analysis.base.AnalysisModule
options:
heading_level: 6
::: nominatim.tokenizer.token_analysis.base.Analyzer
::: nominatim_db.tokenizer.token_analysis.base.Analyzer
options:
heading_level: 6

6
docs/develop/Tokenizers.md
View File

@ -130,16 +130,16 @@ class as defined below.
### Python Tokenizer Class
All tokenizers must inherit from `nominatim.tokenizer.base.AbstractTokenizer`
All tokenizers must inherit from `nominatim_db.tokenizer.base.AbstractTokenizer`
and implement the abstract functions defined there.
::: nominatim.tokenizer.base.AbstractTokenizer
::: nominatim_db.tokenizer.base.AbstractTokenizer
options:
heading_level: 6
### Python Analyzer Class
::: nominatim.tokenizer.base.AbstractAnalyzer
::: nominatim_db.tokenizer.base.AbstractAnalyzer
options:
heading_level: 6

2
docs/library/Configuration.md
View File

@ -20,7 +20,7 @@ configuration parameters, see the [Configuration page](../customize/Settings.md)
## `Configuration` class
::: nominatim.config.Configuration
::: nominatim_core.config.Configuration
options:
members:
- get_bool

28
docs/library/Getting-Started.md
View File

@ -12,22 +12,18 @@ in the database.
The library interface is currently in an experimental stage. There might
be some smaller adjustments to the public interface until the next version.
The library also misses a proper installation routine, so some manipulation
of the PYTHONPATH is required. At the moment, use is only recommended for
developers with some experience in Python.
## Installation
To use the Nominatim library, you need access to a local Nominatim database.
Follow the [installation](../admin/Installation.md) and
[import](../admin/Import.md) instructions to set up your database.
It is not yet possible to install it in the usual way via pip or inside a
virtualenv. To get access to the library you need to set an appropriate
`PYTHONPATH`. With the default installation, the python library can be found
under `/usr/local/share/nominatim/lib-python`. If you have installed
Nominatim under a different prefix, adapt the `/usr/local/` part accordingly.
You can also point the `PYTHONPATH` to the Nominatim source code.
The Nominatim frontend library is contained in the 'nominatim-api' package.
To install the package from the source tree directly, run:
pip install packaging/nominatim-api
Usually, you would want to run this in a virtual environment.
### A simple search example
@ -43,7 +39,7 @@ This code snippet implements a simple search for the town of 'Brugge':
from pathlib import Path
import asyncio
import nominatim.api as napi
import nominatim_api as napi
async def search(query):
api = napi.NominatimAPIAsync(Path('.'))
@ -61,7 +57,7 @@ This code snippet implements a simple search for the town of 'Brugge':
``` python
from pathlib import Path
import nominatim.api as napi
import nominatim_api as napi
api = napi.NominatimAPI(Path('.'))
@ -116,7 +112,7 @@ standard 'nominatim' database:
from pathlib import Path
import asyncio
import nominatim.api as napi
import nominatim_api as napi
config_params = {
'NOMINATIM_DATABASE_DSN': 'pgsql:dbname=belgium'
@ -134,7 +130,7 @@ standard 'nominatim' database:
``` python
from pathlib import Path
import nominatim.api as napi
import nominatim_api as napi
config_params = {
'NOMINATIM_DATABASE_DSN': 'pgsql:dbname=belgium'
@ -161,7 +157,7 @@ Again searching for 'Brugge', this time with a nicely formatted result:
from pathlib import Path
import asyncio
import nominatim.api as napi
import nominatim_api as napi
async def search(query):
api = napi.NominatimAPIAsync(Path('.'))
@ -180,7 +176,7 @@ Again searching for 'Brugge', this time with a nicely formatted result:
``` python
from pathlib import Path
import nominatim.api as napi
import nominatim_api as napi
api = napi.NominatimAPI(Path('.'))

18
docs/library/Input-Parameter-Types.md
View File

@ -5,26 +5,26 @@ in the query functions of the API object.
## Place identification
The [details](NominatimAPI.md#nominatim.api.core.NominatimAPI.details) and
[lookup](NominatimAPI.md#nominatim.api.core.NominatimAPI.lookup) functions
The [details](NominatimAPI.md#nominatim_api.core.NominatimAPI.details) and
[lookup](NominatimAPI.md#nominatim_api.core.NominatimAPI.lookup) functions
require references to places in the database. Below the possible
types for place identification are listed. All types are dataclasses.
### PlaceID
::: nominatim.api.PlaceID
::: nominatim_api.PlaceID
options:
heading_level: 6
### OsmID
::: nominatim.api.OsmID
::: nominatim_api.OsmID
options:
heading_level: 6
## Geometry types
::: nominatim.api.GeometryFormat
::: nominatim_api.GeometryFormat
options:
heading_level: 6
members_order: source
@ -33,14 +33,14 @@ types for place identification are listed. All types are dataclasses.
### Point
::: nominatim.api.Point
::: nominatim_api.Point
options:
heading_level: 6
show_signature_annotations: True
### Bbox
::: nominatim.api.Bbox
::: nominatim_api.Bbox
options:
heading_level: 6
show_signature_annotations: True
@ -54,9 +54,7 @@ orthogonal to restriction by address ranks, which groups places by their
geographic extent.
::: nominatim.api.DataLayer
::: nominatim_api.DataLayer
options:
heading_level: 6
members_order: source

4
docs/library/Low-Level-DB-Access.md
View File

@ -26,7 +26,7 @@ the placex table:
import asyncio
from pathlib import Path
import sqlalchemy as sa
from nominatim.api import NominatimAPIAsync
from nominatim_api import NominatimAPIAsync
async def print_table_size():
api = NominatimAPIAsync(Path('.'))
@ -45,7 +45,7 @@ asyncio.run(print_table_size())
## SearchConnection class
::: nominatim.api.SearchConnection
::: nominatim_api.SearchConnection
options:
members:
- scalar

4
docs/library/NominatimAPI.md
View File

@ -6,7 +6,7 @@ to instantiate a separate instance for each thread.
### NominatimAPI
::: nominatim.api.NominatimAPI
::: nominatim_api.NominatimAPI
options:
members:
- __init__
@ -25,7 +25,7 @@ to instantiate a separate instance for each thread.
### NominatimAPIAsync
::: nominatim.api.NominatimAPIAsync
::: nominatim_api.NominatimAPIAsync
options:
members:
- __init__

8
docs/library/Result-Handling.md
View File

@ -12,7 +12,7 @@ functions that allow further processing of the results.
Nominatim takes the result data from multiple sources. The `source_table` field
in the result describes, from which source the result was retrieved.
::: nominatim.api.SourceTable
::: nominatim_api.SourceTable
options:
heading_level: 6
members_order: source
@ -31,7 +31,7 @@ contains the result place itself and some artificial entries, for example,
for the house number or the country code. This makes processing and creating
a full address easier.
::: nominatim.api.AddressLine
::: nominatim_api.AddressLine
options:
heading_level: 6
members_order: source
@ -43,7 +43,7 @@ may be used to find a place, when the `keywords` parameter is set. Search
terms are split into terms for the name of the place and search terms for
its address.
::: nominatim.api.WordInfo
::: nominatim_api.WordInfo
options:
heading_level: 6
@ -53,6 +53,6 @@ Results are always returned with the full list of available names.
### Locale
::: nominatim.api.Locales
::: nominatim_api.Locales
options:
heading_level: 6

2
docs/mkdocs.yml
View File

@ -75,7 +75,7 @@ plugins:
- mkdocstrings:
handlers:
python:
paths: ["${PROJECT_SOURCE_DIR}"]
paths: ["${PROJECT_SOURCE_DIR}/src"]
options:
show_source: False
show_bases: False