restructure developer's manual

Add a section on setting up the development environment which now
also includes the former chapter on recreating the documentation.
Move the README from test/ into the manual as the new Testing
chapter.

docs/admin/Installation.md

@@ -48,11 +48,8 @@ For running continuous updates:
 
   * [pyosmium](https://osmcode.org/pyosmium/) (with Python 3)
 
-For running tests:
+For dependencies for running tests and building documentation, see
-
+the [Development section](../develop/Development-Environment.md).
-  * [behave](http://pythonhosted.org/behave/)
-  * [nose](https://nose.readthedocs.io)
-  * [phpunit](https://phpunit.de) >= 7.3
 
 ### Hardware
 

docs/develop/Development-Environment.md

@@ -0,0 +1,170 @@
+# Setting up Nominatim for Development
+
+This chapter gives an overview how to set up Nominatim for developement
+and how to run tests.
+
+!!!Important
+This guide assumes that you develop under the latest version of Ubuntu. You
+can of course also use your favourite distribution. You just might have to
+adapt the commands below slightly, in particular the commands for installing
+additional software.
+
+## Installing Nominatim
+
+The first step is to install Nominatim itself. Please follow the installation
+instructions in the [Admin section](../admin/Installation.md). You don't need
+to set up a webserver for development, the webserver that is included with PHP
+is sufficient.
+
+If you want to run Nominatim in a VM via Vagrant, use the default `ubuntu` setup.
+Vagrant's libvirt provider runs out-of-the-box under Ubuntu. You also need to
+install an NFS daemon to enable directory sharing between host and guest. The
+following packages should get you started:
+
+    sudo apt install vagrant vagrant-libvirt libvirt-daemon nfs-kernel-server
+
+## Prerequisites for testing and documentation
+
+The Nominatim tests suite consists of behavioural tests (using behave) and
+unit tests (using PHPUnit). It has the following additional requirements:
+
+* [behave test framework](https://github.com/behave/behave) >= 1.2.5
+* [nose](https://nose.readthedocs.org)
+* [pytidylib](http://countergram.com/open-source/pytidylib)
+* [phpunit](https://phpunit.de) >= 7.3
+* [PHP CodeSniffer](https://github.com/squizlabs/PHP_CodeSniffer)
+
+The documentation is built with mkdocs:
+
+* [mkdocs](https://www.mkdocs.org/) >= 1.1.2
+
+### Installing prerequisites on Ubuntu/Debian
+
+Some of the Python packages require the newest version which is not yet
+available with the current distributions. Therefore it is recommended to
+install pip to get the newest versions.
+
+To install all necessary packages run:
+
+```sh
+sudo apt install php-cgi phpunit php-codesniffer \
+                 python3-pip python3-setuptools python3-dev
+
+pip3 install --user behave nose mkdocs
+```
+
+The `mkdocs` executable will be located in `.local/bin`. You may have to add
+this directory to your path, for example by running:
+
+```
+echo 'export PATH=~/.local/bin:$PATH' > ~/.profile
+```
+
+If your distribution does not have PHPUnit 7.3+, you can install it (as well
+as CodeSniffer) via composer:
+
+```
+sudo apt-get install composer
+composer global require "squizlabs/php_codesniffer=*"
+composer global require "phpunit/phpunit=8.*"
+```
+
+The binaries are found in `.config/composer/vendor/bin`. You need to add this
+to your PATH as well:
+
+```
+echo 'export PATH=~/.config/composer/vendor/bin:$PATH' > ~/.profile
+```
+
+
+## Executing Tests
+
+All tests are located in the `\test` directory.
+
+### Preparing the test database
+
+Some of the behavioural test expect a test database to be present. You need at
+least 2GB RAM and 10GB disk space to create the database.
+
+First create a separate directory for the test DB and Fetch the test planet
+data and the Tiger data for South Dakota:
+
+```
+mkdir testdb
+cd testdb
+wget https://www.nominatim.org/data/test/nominatim-api-testdata.pbf
+wget -O - https://nominatim.org/data/tiger2018-nominatim-preprocessed.tar.gz | tar xz --wildcards --no-anchored '46*'
+```
+
+Configure and build Nominatim in the usual way:
+
+```
+cmake $USERNAME/Nominatim
+make
+```
+
+Copy the test settings:
+
+```
+cp $USERNAME/Nominatim/test/testdb/local.php settings/
+```
+
+Inspect the file to check that all settings are correct for your local setup.
+
+Now you can import the test database:
+
+```
+dropdb --if-exists test_api_nominatim
+./utils/setup.php --all --osm-file nominatim-api-testdb.pbf 2>&1 | tee import.log
+./utils/specialphrases.php --wiki-import | psql -d test_api_nominatim 2>&1 | tee -a import.log
+./utils/setup.php --import-tiger-data 2>&1 | tee -a import.log
+```
+
+### Running the tests
+
+To run all tests just go to the test directory and run make:
+
+```sh
+cd test
+make
+```
+
+To skip tests that require the test database, run `make no-test-db` instead.
+
+For more information about the structure of the tests and how to change and
+extend the test suite, see the [Testing chapter](Testing.md).
+
+## Documentation Pages
+
+The [Nominatim documentation](https://nominatim.org/release-docs/develop/) is
+built using the [MkDocs](https://www.mkdocs.org/) static site generation
+framework. The master branch is automatically deployed every night on
+[https://nominatim.org/release-docs/develop/](https://nominatim.org/release-docs/develop/)
+
+To build the documentation, go to the build directory and run
+
+```
+make doc
+INFO - Cleaning site directory
+INFO - Building documentation to directory: /home/vagrant/build/site-html
+```
+
+This runs `mkdocs build` plus extra transformation of some files and adds
+symlinks (see `CMakeLists.txt` for the exact steps).
+
+Now you can start webserver for local testing
+
+```
+build> mkdocs serve
+[server:296] Serving on http://127.0.0.1:8000
+[handlers:62] Start watching changes
+```
+
+If you develop inside a Vagrant virtual machine, use a port that is forwarded
+to your host:
+
+```
+build> mkdocs serve --dev-addr 0.0.0.0:8088
+[server:296] Serving on http://0.0.0.0:8088
+[handlers:62] Start watching changes
+```

docs/develop/Documentation.md

@@ -1,39 +0,0 @@
-# Documentation Pages
-
-The [Nominatim documentation](https://nominatim.org/release-docs/develop/) is built using the [MkDocs](https://www.mkdocs.org/) static site generation framework. The master branch is automatically deployed every night on under [https://nominatim.org/release-docs/develop/](https://nominatim.org/release-docs/develop/)
-
-To preview local changes, first install MkDocs
-
-```
-pip3 install --user mkdocs
-```
-
-If `mkdocs` can't be found after the installation, the $PATH might have not
-been set correctly yet. Try opening a new terminal session.
-
-
-Then go to the build directory and run
-
-```
-make doc
-INFO - Cleaning site directory
-INFO - Building documentation to directory: /home/vagrant/build/site-html
-```
-
-This runs `mkdocs build` plus extra transformation of some files and adds
-symlinks (see `CMakeLists.txt` for the exact steps).
-
-Now you can start webserver for local testing
-
-```
-build> mkdocs serve
-[server:296] Serving on http://127.0.0.1:8000
-[handlers:62] Start watching changes
-```
-
-If you develop inside a Vagrant virtual machine:
-
- * add port forwarding to your Vagrantfile,
-   e.g. `config.vm.network "forwarded_port", guest: 8000, host: 8000`
- * use `mkdocs serve --dev-addr 0.0.0.0:8000` because the default localhost
-   IP does not get forwarded.

docs/develop/Setup.md

@@ -1,52 +0,0 @@
-# Setup Test Environment
-
-To test changes and contribute to Nominatim you should be able to run
-the test suite(s). For many usecases it's enough to create a Vagrant
-virtual machine (see `VAGRANT.md`), import one small country into the
-database.
-
-## Prerequisites
-
-Nominatim supports a range of PHP versions and PHPUnit versions also
-move fast. We try to test against the newest stable PHP release and
-PHPUnit version even though we expect many Nominatim users will install
-older version on their production servers.
-
- * Python 3 (https://www.python.org/)
- * behave test framework >= 1.2.5 (https://github.com/behave/behave)
- * nose (https://nose.readthedocs.org)
- * pytidylib (http://countergram.com/open-source/pytidylib)
- * psycopg2 (http://initd.org/psycopg/)
-
-#### Ubuntu 20
-
-    sudo apt-get install -y phpunit php-codesniffer php-cgi
-    pip3 install --user behave nose
-
-#### Ubuntu 18
-
-    pip3 install --user behave nose
-
-    sudo apt-get install -y composer php-cgi php-cli php-mbstring php-xml zip unzip
-
-    composer global require "squizlabs/php_codesniffer=*"
-    sudo ln -s ~/.config/composer/vendor/bin/phpcs /usr/bin/
-
-    composer global require "phpunit/phpunit=8.*"
-    sudo ln -s ~/.config/composer/vendor/bin/phpunit /usr/bin/
-
-
-#### CentOS 7 or 8
-
-    sudo dnf install -y php-dom php-mbstring
-    pip3 install --user behave nose
-
-    composer global require "squizlabs/php_codesniffer=*"
-    sudo ln -s ~/.config/composer/vendor/bin/phpcs /usr/bin/
-
-    composer global require "phpunit/phpunit=^7"
-    sudo ln -s ~/.config/composer/vendor/bin/phpunit /usr/bin/
-
-## Run tests, code linter, code coverage
-
-See `README.md` in `test` subdirectory.

docs/develop/Testing.md

@@ -1,12 +1,10 @@
-This directory contains functional and unit tests for the Nominatim API.
+# Nominatim Test Suite
 
-Prerequisites
+This chapter describes the tests in the `/test` directory, how they are
-=============
+structured and how to extend them. For a quick introduction on how to run
+the tests, see the [Development setup chapter](Development-Environment.md).
 
-See `docs/develop/Setup.md`
+## Overall structure
-
-Overall structure
-=================
 
 There are two kind of tests in this test suite. There are functional tests
 which test the API interface using a BDD test framework and there are unit
@@ -27,8 +25,7 @@ This test directory is sturctured as follows:
   +-   testdb      Base data for generating API test database
 ```
 
-PHP Unit Tests
+## PHP Unit Tests (`test/php`)
-==============
 
 Unit tests can be found in the php/ directory and tests selected php functions.
 Very low coverage.
@@ -44,14 +41,19 @@ strip and set other parameters.
 It will use (and destroy) a local database 'nominatim_unit_tests'. You can set
 a different connection string with e.g. UNIT_TEST_DSN='pgsql:dbname=foo_unit_tests'.
 
-BDD Functional Tests
+## BDD Functional Tests (`test/bdd`)
-====================
 
 Functional tests are written as BDD instructions. For more information on
-the philosophy of BDD testing, see http://pythonhosted.org/behave/philosophy.html
+the philosophy of BDD testing, see the
+[Behave manual](http://pythonhosted.org/behave/philosophy.html).
 
-Usage
+The following explanation assume that the reader is familiar with the BDD
------
+notations of features, scenarios and steps.
+
+All possible steps can be found in the `steps` directory and should ideally
+be documented.
+
+### General Usage
 
 To run the functional tests, do
 
@@ -81,62 +83,29 @@ The tests can be configured with a set of environment variables (`behave -D key=
                     run, otherwise the result is undefined.
 
 Logging can be defined through command line parameters of behave itself. Check
-out `behave --help` for details. Also keep an eye out for the 'work-in-progress'
+out `behave --help` for details. Also have a look at the 'work-in-progress'
 feature of behave which comes in handy when writing new tests.
 
-Writing Tests
--------------
-
-The following explanation assume that the reader is familiar with the BDD
-notations of features, scenarios and steps.
-
-All possible steps can be found in the `steps` directory and should ideally
-be documented.
-
 ### API Tests (`test/bdd/api`)
 
 These tests are meant to test the different API endpoints and their parameters.
-They require a to import several datasets into a test database. You need at
+They require a to import several datasets into a test database.
-least 2GB RAM and 10GB discspace.
+See the [Development Setup chapter](Development-Environment.md#preparing-the-test-database)
+for instructions on how to set up this database.
 
+The official test dataset was derived from the 180924 planet (note: such
+file no longer exists at https://planet.openstreetmap.org/planet/2018/).
+Newer planets are likely to work as well but you may see isolated test
+failures where the data has changed.
 
-1. Fetch the OSM planet extract (200MB). See end of document how it got created.
+The official test dataset can always be downloaded from
+[nominatim.org](https://www.nominatim.org/data/test/nominatim-api-testdata.pbf)
+To recreate the input data for the test database run:
 
-    ```
+```
-    cd Nominatim/data
+wget https://ftp5.gwdg.de/pub/misc/openstreetmap/planet.openstreetmap.org/pbf/planet-180924.osm.pbf
-    mkdir testdb
+osmconvert planet-180924.osm.pbf -B=test/testdb/testdb.polys -o=testdb.pbf
-    cd testdb
+```
-    wget https://www.nominatim.org/data/test/nominatim-api-testdata.pbf
-    ```
-
-2. Fetch `46*` (South Dakota) Tiger data
-
-    ```
-    cd Nominatim/data/testdb
-    wget https://nominatim.org/data/tiger2018-nominatim-preprocessed.tar.gz
-    tar xvf tiger2018-nominatim-preprocessed.tar.gz --wildcards --no-anchored '46*'
-    rm tiger2018-nominatim-preprocessed.tar.gz
-    ```
-
-3. Adapt build/settings/local.php settings:
-
-	```
-    @define('CONST_Database_DSN', 'pgsql:dbname=test_api_nominatim');
-    @define('CONST_Use_US_Tiger_Data', true);
-    @define('CONST_Tiger_Data_Path', CONST_ExtraDataPath.'/testdb');
-    @define('CONST_Wikipedia_Data_Path', CONST_BasePath.'/test/testdb');
-    ```
-
-4. Import
-
-    ```
-	LOGFILE=/tmp/nominatim-testdb.$$.log
-    dropdb --if-exists test_api_nominatim
-    ./utils/setup.php --all --osm-file ../Nominatim/data/testdb/nominatim-api-testdb.pbf 2>&1 | tee $LOGFILE
-    ./utils/specialphrases.php --wiki-import > specialphrases.sql
-    psql -d test_api_nominatim -f specialphrases.sql 2>&1 | tee -a $LOGFILE
-    ./utils/setup.php --import-tiger-data 2>&1 | tee -a $LOGFILE
-    ```
 
 #### Code Coverage
 
@@ -146,7 +115,7 @@ On Debian/Ubuntu run:
 
     apt-get install php-codecoverage php-xdebug
 
-The run the API tests as follows:
+Then run the API tests as follows:
 
     behave api -DPHPCOV=<coverage output dir>
 
@@ -155,7 +124,7 @@ the [phpcov](https://github.com/sebastianbergmann/phpcov) tool:
 
     phpcov merge --html=<report output dir> <coverage output dir>
 
-### Indexing Tests (`test/bdd/db`)
+### DB Creation Tests (`test/bdd/db`)
 
 These tests check the import and update of the Nominatim database. They do not
 test the correctness of osm2pgsql. Each test will write some data into the `place`
@@ -171,16 +140,3 @@ needs superuser rights for postgres.
 
 These tests check that data is imported correctly into the place table. They
 use the same template database as the Indexing tests, so the same remarks apply.
-
-
-
-How the test database extract was generated
--------------------------------------------
-The official test dataset was derived from the 180924 planet (note: such
-file no longer exists at https://planet.openstreetmap.org/planet/2018/).
-Newer planets are likely to work as well but you may see isolated test
-failures where the data has changed. To recreate the input data
-for the test database run:
-
-    wget https://ftp5.gwdg.de/pub/misc/openstreetmap/planet.openstreetmap.org/pbf/planet-180924.osm.pbf
-    osmconvert planet-180924.osm.pbf -B=test/testdb/testdb.polys -o=testdb.pbf

docs/mkdocs.yml

@@ -16,20 +16,20 @@ pages:
         - 'FAQ': 'api/Faq.md'
     - 'Administration Guide':
         - 'Basic Installation': 'admin/Installation.md'
-        - 'Importing' : 'admin/Import.md'
+        - 'Import' : 'admin/Import.md'
-        - 'Updating' : 'admin/Update.md'
+        - 'Update' : 'admin/Update.md'
-        - 'Deploying' : 'admin/Deployment.md'
+        - 'Deploy' : 'admin/Deployment.md'
         - 'Nominatim UI'  : 'admin/Setup-Nominatim-UI.md'
         - 'Advanced Installations' : 'admin/Advanced-Installations.md'
         - 'Migration from older Versions' : 'admin/Migration.md'
         - 'Troubleshooting' : 'admin/Faq.md'
     - 'Developers Guide':
-        - 'Overview' : 'develop/overview.md'
+        - 'Setup for Development' : 'develop/Development-Environment.md'
+        - 'Architecture Overview' : 'develop/overview.md'
         - 'OSM Data Import' : 'develop/Import.md'
         - 'Place Ranking' : 'develop/Ranking.md'
         - 'Postcodes' : 'develop/Postcodes.md'
-        - 'Setup Test Environment' : 'develop/Setup.md'
+        - 'Testing' : 'develop/Testing.md'
-        - 'Building Documentation' : 'develop/Documentation.md'
         - 'External Data Sources': 'develop/data-sources.md'
     - 'Appendix':
         - 'Installation on CentOS 7' : 'appendix/Install-on-Centos-7.md'

test/Makefile

@@ -1,9 +1,14 @@
 all: bdd php
+no-test-db: bdd-no-test-db php
 
 bdd:
 	cd bdd && behave -DREMOVE_TEMPLATE=1
 
+bdd-no-test-db:
+	cd bdd && behave -DREMOVE_TEMPLATE=1 db osm2pgsql
+
 php:
 	cd php && phpunit ./
 
-.PHONY: bdd php
+
+.PHONY: bdd php no-test-db