diff --git a/docs/admin/Advanced-Installations.md b/docs/admin/Advanced-Installations.md new file mode 100644 index 00000000..b22d9a61 --- /dev/null +++ b/docs/admin/Advanced-Installations.md @@ -0,0 +1,109 @@ +# Advanced installations + +This page contains instructions for setting up multiple countries in +your Nominatim database. It is assumed that you have already successfully +installed the Nominatim software itself, if not return to the +[installation page](Installation.md). + +## Importing multiple regions + +To import multiple regions in your database, you need to configure and run `utils/import_multiple_regions.sh` file. This script will set up the update directory which has the following structure: + +```bash +update +    ├── europe +    │   ├── andorra +    │   │   └── sequence.state +    │   └── monaco +    │   └── sequence.state +    └── tmp + ├── combined.osm.pbf + └── europe + ├── andorra-latest.osm.pbf + └── monaco-latest.osm.pbf + + +``` + +The `sequence.state` files will contain the sequence ID, which will be used by pyosmium to get updates. The tmp folder is used for import dump. + +### Configuring multiple regions + +The file `import_multiple_regions.sh` needs to be edited as per your requirement: + +1. List of countries. eg: + + COUNTRIES="europe/monaco europe/andorra" + +2. Path to Build directory. eg: + + NOMINATIMBUILD="/srv/nominatim/build" + +3. Path to Update directory. eg: + + UPDATEDIR="/srv/nominatim/update" + +4. Replication URL. eg: + + BASEURL="https://download.geofabrik.de" + DOWNCOUNTRYPOSTFIX="-latest.osm.pbf" + +!!! tip + If your database already exists and you want to add more countries, replace the setting up part + `${SETUPFILE} --osm-file ${UPDATEDIR}/tmp/combined.osm.pbf --all 2>&1` + with `${UPDATEFILE} --import-file ${UPDATEDIR}/tmp/combined.osm.pbf 2>&1`. + +### Setting up multiple regions + +Run the following command from your Nominatim directory after configuring the file. + + bash ./utils/import_multiple_regions.sh + +!!! danger "Important" + This file uses osmium-tool. It must be installed before executing the import script. + Installation instructions can be found [here](https://osmcode.org/osmium-tool/manual.html#installation). + +## Updating multiple regions + +To import multiple regions in your database, you need to configure and run ```utils/update_database.sh```. +This uses the update directory set up while setting up the DB. + +### Configuring multiple regions + +The file `update_database.sh` needs to be edited as per your requirement: + +1. List of countries. eg: + + COUNTRIES="europe/monaco europe/andorra" + +2. Path to Build directory. eg: + + NOMINATIMBUILD="/srv/nominatim/build" + +3. Path to Update directory. eg: + + UPDATEDIR="/srv/nominatim/update" + +4. Replication URL. eg: + + BASEURL="https://download.geofabrik.de" + DOWNCOUNTRYPOSTFIX="-updates" + +5. Followup can be set according to your installation. eg: For Photon, + + FOLLOWUP="curl http://localhost:2322/nominatim-update" + + will handle the indexing. + +### Updating the database + +Run the following command from your Nominatim directory after configuring the file. + + bash ./utils/update_database.sh + +This will get diffs from the replication server, import diffs and index the database. The default replication server in the script([Geofabric](https://download.geofabrik.de)) provides daily updates. + +## Verification and further setup + +Instructions for import verification and other details like importing Wikidata can be found in [import and update page](Import-and-Update.md) + diff --git a/docs/admin/Import-and-Update.md b/docs/admin/Import-and-Update.md index 554633ae..0d1bb027 100644 --- a/docs/admin/Import-and-Update.md +++ b/docs/admin/Import-and-Update.md @@ -318,6 +318,5 @@ compatibility reasons, Osmosis is not required to run this - it uses pyosmium behind the scenes.) If you have imported multiple country extracts and want to keep them -up-to-date, have a look at the script in -[issue #60](https://github.com/openstreetmap/Nominatim/issues/60). - +up-to-date, [Advanced installations section](Advanced-Installations.md) contains instructions +to set up and update multiple country extracts. \ No newline at end of file diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index 0c89dbc3..20909ff4 100644 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -16,6 +16,7 @@ pages: - 'Administration Guide': - 'Basic Installation': 'admin/Installation.md' - 'Importing and Updating' : 'admin/Import-and-Update.md' + - 'Advanced Installations' : 'admin/Advanced-Installations.md' - 'Migration from older Versions' : 'admin/Migration.md' - 'Troubleshooting' : 'admin/Faq.md' - 'Developers Guide': diff --git a/utils/import_multiple_regions.sh b/utils/import_multiple_regions.sh new file mode 100644 index 00000000..83323c2e --- /dev/null +++ b/utils/import_multiple_regions.sh @@ -0,0 +1,91 @@ +#!/bin/bash -xv + +# Script to set up Nominatim database for multiple countries + +# Steps to follow: + +# *) Get the pbf files from server + +# *) Set up sequence.state for updates + +# *) Merge the pbf files into a single file. + +# *) Setup nominatim db using 'setup.php --osm-file' + +# Hint: +# +# Use "bashdb ./update_database.sh" and bashdb's "next" command for step-by-step +# execution. + +# ****************************************************************************** + +touch2() { mkdir -p "$(dirname "$1")" && touch "$1" ; } + +# ****************************************************************************** +# Configuration section: Variables in this section should be set according to your requirements + +# REPLACE WITH LIST OF YOUR "COUNTRIES": + +COUNTRIES="europe/monaco europe/andorra" + +# SET TO YOUR NOMINATIM build FOLDER PATH: + +NOMINATIMBUILD="/srv/nominatim/build" +SETUPFILE="$NOMINATIMBUILD/utils/setup.php" +UPDATEFILE="$NOMINATIMBUILD/utils/update.php" + +# SET TO YOUR update FOLDER PATH: + +UPDATEDIR="/srv/nominatim/update" + +# SET TO YOUR replication server URL: + +BASEURL="https://download.geofabrik.de" +DOWNCOUNTRYPOSTFIX="-latest.osm.pbf" + +# End of configuration section +# ****************************************************************************** + +COMBINEFILES="osmium merge" + +mkdir -p ${UPDATEDIR} +cd ${UPDATEDIR} +rm -rf tmp +mkdir -p tmp +cd tmp + +for COUNTRY in $COUNTRIES; +do + + echo "====================================================================" + echo "$COUNTRY" + echo "====================================================================" + DIR="$UPDATEDIR/$COUNTRY" + FILE="$DIR/configuration.txt" + DOWNURL="$BASEURL/$COUNTRY$DOWNCOUNTRYPOSTFIX" + IMPORTFILE=$COUNTRY$DOWNCOUNTRYPOSTFIX + IMPORTFILEPATH=${UPDATEDIR}/tmp/${IMPORTFILE} + FILENAME=${COUNTRY//[\/]/_} + + + touch2 $IMPORTFILEPATH + wget ${DOWNURL} -O $IMPORTFILEPATH + + touch2 ${DIR}/sequence.state + pyosmium-get-changes -O $IMPORTFILEPATH -f ${DIR}/sequence.state -v + + COMBINEFILES="${COMBINEFILES} ${IMPORTFILEPATH}" + echo $IMPORTFILE + echo "====================================================================" +done + + +echo "${COMBINEFILES} -o combined.osm.pbf" +${COMBINEFILES} -o combined.osm.pbf + +echo "====================================================================" +echo "Setting up nominatim db" +${SETUPFILE} --osm-file ${UPDATEDIR}/tmp/combined.osm.pbf --all 2>&1 + +# ${UPDATEFILE} --import-file ${UPDATEDIR}/tmp/combined.osm.pbf 2>&1 +echo "====================================================================" \ No newline at end of file diff --git a/utils/update_database.sh b/utils/update_database.sh new file mode 100644 index 00000000..75d0de5d --- /dev/null +++ b/utils/update_database.sh @@ -0,0 +1,80 @@ +#!/bin/bash -xv + +# Derived from https://gist.github.com/RhinoDevel/8a35ebd2a08166f328eca01ab005c6de and edited to work with Pyosmium +# Related to https://github.com/osm-search/Nominatim/issues/1683 + +# Steps being followed: + +# *) Get the diff file from server +# 1) pyosmium-get-changes (with -f sequence.state for getting sequenceNumber) + +# *) Import diff +# 1) utils/update.php --import-diff + +# *) Index for all the countries at the end + +# Hint: +# +# Use "bashdb ./update_database.sh" and bashdb's "next" command for step-by-step +# execution. + +# ****************************************************************************** + +# REPLACE WITH LIST OF YOUR "COUNTRIES": +# + + +COUNTRIES="europe/monaco europe/andorra" + +# SET TO YOUR NOMINATIM build FOLDER PATH: +# +NOMINATIMBUILD="/srv/nominatim/build" +UPDATEFILE="$NOMINATIMBUILD/utils/update.php" + +# SET TO YOUR update data FOLDER PATH: +# +UPDATEDIR="/srv/nominatim/update" + +UPDATEBASEURL="https://download.geofabrik.de" +UPDATECOUNTRYPOSTFIX="-updates" + +# If you do not use Photon, let Nominatim handle (re-)indexing: +# +FOLLOWUP="$UPDATEFILE --index" +# +# If you use Photon, update Photon and let it handle the index +# (Photon server must be running and must have been started with "-database", +# "-user" and "-password" parameters): +# +#FOLLOWUP="curl http://localhost:2322/nominatim-update" + +# ****************************************************************************** + + +for COUNTRY in $COUNTRIES; +do + + echo "====================================================================" + echo "$COUNTRY" + echo "====================================================================" + DIR="$UPDATEDIR/$COUNTRY" + FILE="$DIR/sequence.state" + BASEURL="$UPDATEBASEURL/$COUNTRY$UPDATECOUNTRYPOSTFIX" + FILENAME=${COUNTRY//[\/]/_} + + # mkdir -p ${DIR} + cd ${DIR} + + echo "Attempting to get changes" + pyosmium-get-changes -o ${DIR}/${FILENAME}.osc.gz -f ${FILE} --server $BASEURL -v + + echo "Attempting to import diffs" + ${NOMINATIMBUILD}/utils/update.php --import-diff ${DIR}/${FILENAME}.osc.gz + rm ${DIR}/${FILENAME}.osc.gz + +done + +echo "====================================================================" +echo "Reindexing" +${FOLLOWUP} +echo "====================================================================" \ No newline at end of file