mirror of
https://github.com/osm-search/Nominatim.git
synced 2024-12-25 05:52:32 +03:00
restructure developer's manual
Add a section on setting up the development environment which now also includes the former chapter on recreating the documentation. Move the README from test/ into the manual as the new Testing chapter.
This commit is contained in:
Sarah Hoffmann
parent
fb63a7418e
commit
2b11a47a2f
@ -48,11 +48,8 @@ For running continuous updates:
|
||||
|
||||
* [pyosmium](https://osmcode.org/pyosmium/) (with Python 3)
|
||||
|
||||
For running tests:
|
||||
|
||||
* [behave](http://pythonhosted.org/behave/)
|
||||
* [nose](https://nose.readthedocs.io)
|
||||
* [phpunit](https://phpunit.de) >= 7.3
|
||||
For dependencies for running tests and building documentation, see
|
||||
the [Development section](../develop/Development-Environment.md).
|
||||
|
||||
### Hardware
|
||||
|
||||
|
||||
170
docs/develop/Development-Environment.md
Normal file
170
docs/develop/Development-Environment.md
Normal file
@ -0,0 +1,170 @@
|
||||
# Setting up Nominatim for Development
|
||||
|
||||
This chapter gives an overview how to set up Nominatim for developement
|
||||
and how to run tests.
|
||||
|
||||
!!!Important
|
||||
This guide assumes that you develop under the latest version of Ubuntu. You
|
||||
can of course also use your favourite distribution. You just might have to
|
||||
adapt the commands below slightly, in particular the commands for installing
|
||||
additional software.
|
||||
|
||||
## Installing Nominatim
|
||||
|
||||
The first step is to install Nominatim itself. Please follow the installation
|
||||
instructions in the [Admin section](../admin/Installation.md). You don't need
|
||||
to set up a webserver for development, the webserver that is included with PHP
|
||||
is sufficient.
|
||||
|
||||
If you want to run Nominatim in a VM via Vagrant, use the default `ubuntu` setup.
|
||||
Vagrant's libvirt provider runs out-of-the-box under Ubuntu. You also need to
|
||||
install an NFS daemon to enable directory sharing between host and guest. The
|
||||
following packages should get you started:
|
||||
|
||||
sudo apt install vagrant vagrant-libvirt libvirt-daemon nfs-kernel-server
|
||||
|
||||
## Prerequisites for testing and documentation
|
||||
|
||||
The Nominatim tests suite consists of behavioural tests (using behave) and
|
||||
unit tests (using PHPUnit). It has the following additional requirements:
|
||||
|
||||
* [behave test framework](https://github.com/behave/behave) >= 1.2.5
|
||||
* [nose](https://nose.readthedocs.org)
|
||||
* [pytidylib](http://countergram.com/open-source/pytidylib)
|
||||
* [phpunit](https://phpunit.de) >= 7.3
|
||||
* [PHP CodeSniffer](https://github.com/squizlabs/PHP_CodeSniffer)
|
||||
|
||||
The documentation is built with mkdocs:
|
||||
|
||||
* [mkdocs](https://www.mkdocs.org/) >= 1.1.2
|
||||
|
||||
### Installing prerequisites on Ubuntu/Debian
|
||||
|
||||
Some of the Python packages require the newest version which is not yet
|
||||
available with the current distributions. Therefore it is recommended to
|
||||
install pip to get the newest versions.
|
||||
|
||||
To install all necessary packages run:
|
||||
|
||||
```sh
|
||||
sudo apt install php-cgi phpunit php-codesniffer \
|
||||
python3-pip python3-setuptools python3-dev
|
||||
|
||||
pip3 install --user behave nose mkdocs
|
||||
```
|
||||
|
||||
The `mkdocs` executable will be located in `.local/bin`. You may have to add
|
||||
this directory to your path, for example by running:
|
||||
|
||||
```
|
||||
echo 'export PATH=~/.local/bin:$PATH' > ~/.profile
|
||||
```
|
||||
|
||||
If your distribution does not have PHPUnit 7.3+, you can install it (as well
|
||||
as CodeSniffer) via composer:
|
||||
|
||||
```
|
||||
sudo apt-get install composer
|
||||
composer global require "squizlabs/php_codesniffer=*"
|
||||
composer global require "phpunit/phpunit=8.*"
|
||||
```
|
||||
|
||||
The binaries are found in `.config/composer/vendor/bin`. You need to add this
|
||||
to your PATH as well:
|
||||
|
||||
```
|
||||
echo 'export PATH=~/.config/composer/vendor/bin:$PATH' > ~/.profile
|
||||
```
|
||||
|
||||
|
||||
## Executing Tests
|
||||
|
||||
All tests are located in the `\test` directory.
|
||||
|
||||
### Preparing the test database
|
||||
|
||||
Some of the behavioural test expect a test database to be present. You need at
|
||||
least 2GB RAM and 10GB disk space to create the database.
|
||||
|
||||
First create a separate directory for the test DB and Fetch the test planet
|
||||
data and the Tiger data for South Dakota:
|
||||
|
||||
```
|
||||
mkdir testdb
|
||||
cd testdb
|
||||
wget https://www.nominatim.org/data/test/nominatim-api-testdata.pbf
|
||||
wget -O - https://nominatim.org/data/tiger2018-nominatim-preprocessed.tar.gz | tar xz --wildcards --no-anchored '46*'
|
||||
```
|
||||
|
||||
Configure and build Nominatim in the usual way:
|
||||
|
||||
```
|
||||
cmake $USERNAME/Nominatim
|
||||
make
|
||||
```
|
||||
|
||||
Copy the test settings:
|
||||
|
||||
```
|
||||
cp $USERNAME/Nominatim/test/testdb/local.php settings/
|
||||
```
|
||||
|
||||
Inspect the file to check that all settings are correct for your local setup.
|
||||
|
||||
Now you can import the test database:
|
||||
|
||||
```
|
||||
dropdb --if-exists test_api_nominatim
|
||||
./utils/setup.php --all --osm-file nominatim-api-testdb.pbf 2>&1 | tee import.log
|
||||
./utils/specialphrases.php --wiki-import | psql -d test_api_nominatim 2>&1 | tee -a import.log
|
||||
./utils/setup.php --import-tiger-data 2>&1 | tee -a import.log
|
||||
```
|
||||
|
||||
### Running the tests
|
||||
|
||||
To run all tests just go to the test directory and run make:
|
||||
|
||||
```sh
|
||||
cd test
|
||||
make
|
||||
```
|
||||
|
||||
To skip tests that require the test database, run `make no-test-db` instead.
|
||||
|
||||
For more information about the structure of the tests and how to change and
|
||||
extend the test suite, see the [Testing chapter](Testing.md).
|
||||
|
||||
## Documentation Pages
|
||||
|
||||
The [Nominatim documentation](https://nominatim.org/release-docs/develop/) is
|
||||
built using the [MkDocs](https://www.mkdocs.org/) static site generation
|
||||
framework. The master branch is automatically deployed every night on
|
||||
[https://nominatim.org/release-docs/develop/](https://nominatim.org/release-docs/develop/)
|
||||
|
||||
To build the documentation, go to the build directory and run
|
||||
|
||||
```
|
||||
make doc
|
||||
INFO - Cleaning site directory
|
||||
INFO - Building documentation to directory: /home/vagrant/build/site-html
|
||||
```
|
||||
|
||||
This runs `mkdocs build` plus extra transformation of some files and adds
|
||||
symlinks (see `CMakeLists.txt` for the exact steps).
|
||||
|
||||
Now you can start webserver for local testing
|
||||
|
||||
```
|
||||
build> mkdocs serve
|
||||
[server:296] Serving on http://127.0.0.1:8000
|
||||
[handlers:62] Start watching changes
|
||||
```
|
||||
|
||||
If you develop inside a Vagrant virtual machine, use a port that is forwarded
|
||||
to your host:
|
||||
|
||||
```
|
||||
build> mkdocs serve --dev-addr 0.0.0.0:8088
|
||||
[server:296] Serving on http://0.0.0.0:8088
|
||||
[handlers:62] Start watching changes
|
||||
```
|
||||
@ -1,39 +0,0 @@
|
||||
# Documentation Pages
|
||||
|
||||
The [Nominatim documentation](https://nominatim.org/release-docs/develop/) is built using the [MkDocs](https://www.mkdocs.org/) static site generation framework. The master branch is automatically deployed every night on under [https://nominatim.org/release-docs/develop/](https://nominatim.org/release-docs/develop/)
|
||||
|
||||
To preview local changes, first install MkDocs
|
||||
|
||||
```
|
||||
pip3 install --user mkdocs
|
||||
```
|
||||
|
||||
If `mkdocs` can't be found after the installation, the $PATH might have not
|
||||
been set correctly yet. Try opening a new terminal session.
|
||||
|
||||
|
||||
Then go to the build directory and run
|
||||
|
||||
```
|
||||
make doc
|
||||
INFO - Cleaning site directory
|
||||
INFO - Building documentation to directory: /home/vagrant/build/site-html
|
||||
```
|
||||
|
||||
This runs `mkdocs build` plus extra transformation of some files and adds
|
||||
symlinks (see `CMakeLists.txt` for the exact steps).
|
||||
|
||||
Now you can start webserver for local testing
|
||||
|
||||
```
|
||||
build> mkdocs serve
|
||||
[server:296] Serving on http://127.0.0.1:8000
|
||||
[handlers:62] Start watching changes
|
||||
```
|
||||
|
||||
If you develop inside a Vagrant virtual machine:
|
||||
|
||||
* add port forwarding to your Vagrantfile,
|
||||
e.g. `config.vm.network "forwarded_port", guest: 8000, host: 8000`
|
||||
* use `mkdocs serve --dev-addr 0.0.0.0:8000` because the default localhost
|
||||
IP does not get forwarded.
|
||||
@ -1,52 +0,0 @@
|
||||
# Setup Test Environment
|
||||
|
||||
To test changes and contribute to Nominatim you should be able to run
|
||||
the test suite(s). For many usecases it's enough to create a Vagrant
|
||||
virtual machine (see `VAGRANT.md`), import one small country into the
|
||||
database.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Nominatim supports a range of PHP versions and PHPUnit versions also
|
||||
move fast. We try to test against the newest stable PHP release and
|
||||
PHPUnit version even though we expect many Nominatim users will install
|
||||
older version on their production servers.
|
||||
|
||||
* Python 3 (https://www.python.org/)
|
||||
* behave test framework >= 1.2.5 (https://github.com/behave/behave)
|
||||
* nose (https://nose.readthedocs.org)
|
||||
* pytidylib (http://countergram.com/open-source/pytidylib)
|
||||
* psycopg2 (http://initd.org/psycopg/)
|
||||
|
||||
#### Ubuntu 20
|
||||
|
||||
sudo apt-get install -y phpunit php-codesniffer php-cgi
|
||||
pip3 install --user behave nose
|
||||
|
||||
#### Ubuntu 18
|
||||
|
||||
pip3 install --user behave nose
|
||||
|
||||
sudo apt-get install -y composer php-cgi php-cli php-mbstring php-xml zip unzip
|
||||
|
||||
composer global require "squizlabs/php_codesniffer=*"
|
||||
sudo ln -s ~/.config/composer/vendor/bin/phpcs /usr/bin/
|
||||
|
||||
composer global require "phpunit/phpunit=8.*"
|
||||
sudo ln -s ~/.config/composer/vendor/bin/phpunit /usr/bin/
|
||||
|
||||
|
||||
#### CentOS 7 or 8
|
||||
|
||||
sudo dnf install -y php-dom php-mbstring
|
||||
pip3 install --user behave nose
|
||||
|
||||
composer global require "squizlabs/php_codesniffer=*"
|
||||
sudo ln -s ~/.config/composer/vendor/bin/phpcs /usr/bin/
|
||||
|
||||
composer global require "phpunit/phpunit=^7"
|
||||
sudo ln -s ~/.config/composer/vendor/bin/phpunit /usr/bin/
|
||||
|
||||
## Run tests, code linter, code coverage
|
||||
|
||||
See `README.md` in `test` subdirectory.
|
||||
@ -1,12 +1,10 @@
|
||||
This directory contains functional and unit tests for the Nominatim API.
|
||||
# Nominatim Test Suite
|
||||
|
||||
Prerequisites
|
||||
=============
|
||||
This chapter describes the tests in the `/test` directory, how they are
|
||||
structured and how to extend them. For a quick introduction on how to run
|
||||
the tests, see the [Development setup chapter](Development-Environment.md).
|
||||
|
||||
See `docs/develop/Setup.md`
|
||||
|
||||
Overall structure
|
||||
=================
|
||||
## Overall structure
|
||||
|
||||
There are two kind of tests in this test suite. There are functional tests
|
||||
which test the API interface using a BDD test framework and there are unit
|
||||
@ -27,8 +25,7 @@ This test directory is sturctured as follows:
|
||||
+- testdb Base data for generating API test database
|
||||
```
|
||||
|
||||
PHP Unit Tests
|
||||
==============
|
||||
## PHP Unit Tests (`test/php`)
|
||||
|
||||
Unit tests can be found in the php/ directory and tests selected php functions.
|
||||
Very low coverage.
|
||||
@ -44,14 +41,19 @@ strip and set other parameters.
|
||||
It will use (and destroy) a local database 'nominatim_unit_tests'. You can set
|
||||
a different connection string with e.g. UNIT_TEST_DSN='pgsql:dbname=foo_unit_tests'.
|
||||
|
||||
BDD Functional Tests
|
||||
====================
|
||||
## BDD Functional Tests (`test/bdd`)
|
||||
|
||||
Functional tests are written as BDD instructions. For more information on
|
||||
the philosophy of BDD testing, see http://pythonhosted.org/behave/philosophy.html
|
||||
the philosophy of BDD testing, see the
|
||||
[Behave manual](http://pythonhosted.org/behave/philosophy.html).
|
||||
|
||||
Usage
|
||||
-----
|
||||
The following explanation assume that the reader is familiar with the BDD
|
||||
notations of features, scenarios and steps.
|
||||
|
||||
All possible steps can be found in the `steps` directory and should ideally
|
||||
be documented.
|
||||
|
||||
### General Usage
|
||||
|
||||
To run the functional tests, do
|
||||
|
||||
@ -81,62 +83,29 @@ The tests can be configured with a set of environment variables (`behave -D key=
|
||||
run, otherwise the result is undefined.
|
||||
|
||||
Logging can be defined through command line parameters of behave itself. Check
|
||||
out `behave --help` for details. Also keep an eye out for the 'work-in-progress'
|
||||
out `behave --help` for details. Also have a look at the 'work-in-progress'
|
||||
feature of behave which comes in handy when writing new tests.
|
||||
|
||||
Writing Tests
|
||||
-------------
|
||||
|
||||
The following explanation assume that the reader is familiar with the BDD
|
||||
notations of features, scenarios and steps.
|
||||
|
||||
All possible steps can be found in the `steps` directory and should ideally
|
||||
be documented.
|
||||
|
||||
### API Tests (`test/bdd/api`)
|
||||
|
||||
These tests are meant to test the different API endpoints and their parameters.
|
||||
They require a to import several datasets into a test database. You need at
|
||||
least 2GB RAM and 10GB discspace.
|
||||
They require a to import several datasets into a test database.
|
||||
See the [Development Setup chapter](Development-Environment.md#preparing-the-test-database)
|
||||
for instructions on how to set up this database.
|
||||
|
||||
The official test dataset was derived from the 180924 planet (note: such
|
||||
file no longer exists at https://planet.openstreetmap.org/planet/2018/).
|
||||
Newer planets are likely to work as well but you may see isolated test
|
||||
failures where the data has changed.
|
||||
|
||||
1. Fetch the OSM planet extract (200MB). See end of document how it got created.
|
||||
The official test dataset can always be downloaded from
|
||||
[nominatim.org](https://www.nominatim.org/data/test/nominatim-api-testdata.pbf)
|
||||
To recreate the input data for the test database run:
|
||||
|
||||
```
|
||||
cd Nominatim/data
|
||||
mkdir testdb
|
||||
cd testdb
|
||||
wget https://www.nominatim.org/data/test/nominatim-api-testdata.pbf
|
||||
```
|
||||
|
||||
2. Fetch `46*` (South Dakota) Tiger data
|
||||
|
||||
```
|
||||
cd Nominatim/data/testdb
|
||||
wget https://nominatim.org/data/tiger2018-nominatim-preprocessed.tar.gz
|
||||
tar xvf tiger2018-nominatim-preprocessed.tar.gz --wildcards --no-anchored '46*'
|
||||
rm tiger2018-nominatim-preprocessed.tar.gz
|
||||
```
|
||||
|
||||
3. Adapt build/settings/local.php settings:
|
||||
|
||||
```
|
||||
@define('CONST_Database_DSN', 'pgsql:dbname=test_api_nominatim');
|
||||
@define('CONST_Use_US_Tiger_Data', true);
|
||||
@define('CONST_Tiger_Data_Path', CONST_ExtraDataPath.'/testdb');
|
||||
@define('CONST_Wikipedia_Data_Path', CONST_BasePath.'/test/testdb');
|
||||
```
|
||||
|
||||
4. Import
|
||||
|
||||
```
|
||||
LOGFILE=/tmp/nominatim-testdb.$$.log
|
||||
dropdb --if-exists test_api_nominatim
|
||||
./utils/setup.php --all --osm-file ../Nominatim/data/testdb/nominatim-api-testdb.pbf 2>&1 | tee $LOGFILE
|
||||
./utils/specialphrases.php --wiki-import > specialphrases.sql
|
||||
psql -d test_api_nominatim -f specialphrases.sql 2>&1 | tee -a $LOGFILE
|
||||
./utils/setup.php --import-tiger-data 2>&1 | tee -a $LOGFILE
|
||||
```
|
||||
```
|
||||
wget https://ftp5.gwdg.de/pub/misc/openstreetmap/planet.openstreetmap.org/pbf/planet-180924.osm.pbf
|
||||
osmconvert planet-180924.osm.pbf -B=test/testdb/testdb.polys -o=testdb.pbf
|
||||
```
|
||||
|
||||
#### Code Coverage
|
||||
|
||||
@ -146,7 +115,7 @@ On Debian/Ubuntu run:
|
||||
|
||||
apt-get install php-codecoverage php-xdebug
|
||||
|
||||
The run the API tests as follows:
|
||||
Then run the API tests as follows:
|
||||
|
||||
behave api -DPHPCOV=<coverage output dir>
|
||||
|
||||
@ -155,7 +124,7 @@ the [phpcov](https://github.com/sebastianbergmann/phpcov) tool:
|
||||
|
||||
phpcov merge --html=<report output dir> <coverage output dir>
|
||||
|
||||
### Indexing Tests (`test/bdd/db`)
|
||||
### DB Creation Tests (`test/bdd/db`)
|
||||
|
||||
These tests check the import and update of the Nominatim database. They do not
|
||||
test the correctness of osm2pgsql. Each test will write some data into the `place`
|
||||
@ -171,16 +140,3 @@ needs superuser rights for postgres.
|
||||
|
||||
These tests check that data is imported correctly into the place table. They
|
||||
use the same template database as the Indexing tests, so the same remarks apply.
|
||||
|
||||
|
||||
|
||||
How the test database extract was generated
|
||||
-------------------------------------------
|
||||
The official test dataset was derived from the 180924 planet (note: such
|
||||
file no longer exists at https://planet.openstreetmap.org/planet/2018/).
|
||||
Newer planets are likely to work as well but you may see isolated test
|
||||
failures where the data has changed. To recreate the input data
|
||||
for the test database run:
|
||||
|
||||
wget https://ftp5.gwdg.de/pub/misc/openstreetmap/planet.openstreetmap.org/pbf/planet-180924.osm.pbf
|
||||
osmconvert planet-180924.osm.pbf -B=test/testdb/testdb.polys -o=testdb.pbf
|
||||
@ -16,20 +16,20 @@ pages:
|
||||
- 'FAQ': 'api/Faq.md'
|
||||
- 'Administration Guide':
|
||||
- 'Basic Installation': 'admin/Installation.md'
|
||||
- 'Importing' : 'admin/Import.md'
|
||||
- 'Updating' : 'admin/Update.md'
|
||||
- 'Deploying' : 'admin/Deployment.md'
|
||||
- 'Import' : 'admin/Import.md'
|
||||
- 'Update' : 'admin/Update.md'
|
||||
- 'Deploy' : 'admin/Deployment.md'
|
||||
- 'Nominatim UI' : 'admin/Setup-Nominatim-UI.md'
|
||||
- 'Advanced Installations' : 'admin/Advanced-Installations.md'
|
||||
- 'Migration from older Versions' : 'admin/Migration.md'
|
||||
- 'Troubleshooting' : 'admin/Faq.md'
|
||||
- 'Developers Guide':
|
||||
- 'Overview' : 'develop/overview.md'
|
||||
- 'Setup for Development' : 'develop/Development-Environment.md'
|
||||
- 'Architecture Overview' : 'develop/overview.md'
|
||||
- 'OSM Data Import' : 'develop/Import.md'
|
||||
- 'Place Ranking' : 'develop/Ranking.md'
|
||||
- 'Postcodes' : 'develop/Postcodes.md'
|
||||
- 'Setup Test Environment' : 'develop/Setup.md'
|
||||
- 'Building Documentation' : 'develop/Documentation.md'
|
||||
- 'Testing' : 'develop/Testing.md'
|
||||
- 'External Data Sources': 'develop/data-sources.md'
|
||||
- 'Appendix':
|
||||
- 'Installation on CentOS 7' : 'appendix/Install-on-Centos-7.md'
|
||||
|
||||
@ -1,9 +1,14 @@
|
||||
all: bdd php
|
||||
no-test-db: bdd-no-test-db php
|
||||
|
||||
bdd:
|
||||
cd bdd && behave -DREMOVE_TEMPLATE=1
|
||||
|
||||
bdd-no-test-db:
|
||||
cd bdd && behave -DREMOVE_TEMPLATE=1 db osm2pgsql
|
||||
|
||||
php:
|
||||
cd php && phpunit ./
|
||||
|
||||
.PHONY: bdd php
|
||||
|
||||
.PHONY: bdd php no-test-db
|
||||
|
||||
Loading…
Reference in New Issue
Block a user