From 56d24085f9b5b2e1d0d0fd5ff8af0c905765ff04 Mon Sep 17 00:00:00 2001 From: Sarah Hoffmann Date: Sat, 14 Aug 2021 23:48:06 +0200 Subject: [PATCH] port multi-region update scripts to nominatim tool Also updates the documentation. For the simple case of just importing multiple regions, provide simplified instructions that use the new multi-file import feature. Fixes #2365. --- docs/admin/Advanced-Installations.md | 116 +++++++++++++-------------- utils/import_multiple_regions.sh | 35 ++------ utils/update_database.sh | 26 ++---- 3 files changed, 68 insertions(+), 109 deletions(-) diff --git a/docs/admin/Advanced-Installations.md b/docs/admin/Advanced-Installations.md index d5e6e889..ee38f3e4 100644 --- a/docs/admin/Advanced-Installations.md +++ b/docs/admin/Advanced-Installations.md @@ -5,9 +5,34 @@ your Nominatim database. It is assumed that you have already successfully installed the Nominatim software itself, if not return to the [installation page](Installation.md). -## Importing multiple regions +## Importing multiple regions (without updates) -To import multiple regions in your database, you need to configure and run `utils/import_multiple_regions.sh` file. This script will set up the update directory which has the following structure: +To import multiple regions in your database you can simply give multiple +OSM files to the import command: + +``` +nominatim import --osm-file file1.pbf --osm-file file2.pbf +``` + +If you already have imported a file and want to add another one, you can +use the add-data function to import the additional data as follows: + +``` +nominatim add-data --file +nominatim refresh --postcodes +nominatim index -j +``` + +Please note that adding additional data is always significantly slower than +the original import. + +## Importing multiple regions (with updates) + +If you want to import multiple regions _and_ be able to keep them up-to-date +with updates, then you can use the scripts provided in the `utils` directory. + +These scripts will set up an `update` directory in your project directory, +which has the following structure: ```bash update @@ -17,7 +42,6 @@ update    │   └── monaco    │   └── sequence.state    └── tmp - ├── combined.osm.pbf └── europe ├── andorra-latest.osm.pbf └── monaco-latest.osm.pbf @@ -25,85 +49,57 @@ update ``` -The `sequence.state` files will contain the sequence ID, which will be used by pyosmium to get updates. The tmp folder is used for import dump. +The `sequence.state` files contain the sequence ID for each region. They will +be used by pyosmium to get updates. The `tmp` folder is used for import dump and +can be deleted once the import is complete. -### Configuring multiple regions - -The file `import_multiple_regions.sh` needs to be edited as per your requirement: - -1. List of countries. eg: - - COUNTRIES="europe/monaco europe/andorra" - -2. Path to Build directory. eg: - - NOMINATIMBUILD="/srv/nominatim/build" - -3. Path to Update directory. eg: - - UPDATEDIR="/srv/nominatim/update" - -4. Replication URL. eg: - - BASEURL="https://download.geofabrik.de" - DOWNCOUNTRYPOSTFIX="-latest.osm.pbf" ### Setting up multiple regions -!!! tip - If your database already exists and you want to add more countries, - replace the setting up part - `${SETUPFILE} --osm-file ${UPDATEDIR}/tmp/combined.osm.pbf --all 2>&1` - with `${UPDATEFILE} --import-file ${UPDATEDIR}/tmp/combined.osm.pbf --index --index-instances N 2>&1` - where N is the numbers of CPUs in your system. +Create a project directory as described for the +[simple import](Import.md#creating-the-project-directory). If necessary, +you can also add an `.env` configuration with customized options. In particular, +you need to make sure that `NOMINATIM_REPLICATION_UPDATE_INTERVAL` and +`NOMINATIM_REPLICATION_RECHECK_INTERVAL` are set according to the update +interval of the extract server you use. -Run the following command from your Nominatim directory after configuring the file. +Copy the scripts `utils/import_multiple_regions.sh` and `utils/update_database.sh` +into the project directory. - bash ./utils/import_multiple_regions.sh +Now customize both files as per your requirements -!!! danger "Important" - This file uses osmium-tool. It must be installed before executing the import script. - Installation instructions can be found [here](https://osmcode.org/osmium-tool/manual.html#installation). - -### Updating multiple regions - -To import multiple regions in your database, you need to configure and run ```utils/update_database.sh```. -This uses the update directory set up while setting up the DB. - -### Configuring multiple regions - -The file `update_database.sh` needs to be edited as per your requirement: - -1. List of countries. eg: +1. List of countries. e.g. COUNTRIES="europe/monaco europe/andorra" -2. Path to Build directory. eg: +2. URL to the service providing the extracts and updates. eg: - NOMINATIMBUILD="/srv/nominatim/build" - -3. Path to Update directory. eg: - - UPDATEDIR="/srv/nominatim/update" - -4. Replication URL. eg: - BASEURL="https://download.geofabrik.de" - DOWNCOUNTRYPOSTFIX="-updates" + DOWNCOUNTRYPOSTFIX="-latest.osm.pbf" -5. Followup can be set according to your installation. eg: For Photon, +5. Followup in the update script can be set according to your installation. + E.g. for Photon, FOLLOWUP="curl http://localhost:2322/nominatim-update" will handle the indexing. + +To start the initial import, change into the project directory and run + +``` + bash import_multiple_regions.sh +``` + ### Updating the database -Run the following command from your Nominatim directory after configuring the file. +Change into the project directory and run the following command: - bash ./utils/update_database.sh + bash update_database.sh -This will get diffs from the replication server, import diffs and index the database. The default replication server in the script([Geofabrik](https://download.geofabrik.de)) provides daily updates. +This will get diffs from the replication server, import diffs and index +the database. The default replication server in the +script([Geofabrik](https://download.geofabrik.de)) provides daily updates. ## Importing Nominatim to an external PostgreSQL database diff --git a/utils/import_multiple_regions.sh b/utils/import_multiple_regions.sh index 83323c2e..d15b2f55 100644 --- a/utils/import_multiple_regions.sh +++ b/utils/import_multiple_regions.sh @@ -8,8 +8,6 @@ # *) Set up sequence.state for updates -# *) Merge the pbf files into a single file. - # *) Setup nominatim db using 'setup.php --osm-file' # Hint: @@ -28,16 +26,6 @@ touch2() { mkdir -p "$(dirname "$1")" && touch "$1" ; } COUNTRIES="europe/monaco europe/andorra" -# SET TO YOUR NOMINATIM build FOLDER PATH: - -NOMINATIMBUILD="/srv/nominatim/build" -SETUPFILE="$NOMINATIMBUILD/utils/setup.php" -UPDATEFILE="$NOMINATIMBUILD/utils/update.php" - -# SET TO YOUR update FOLDER PATH: - -UPDATEDIR="/srv/nominatim/update" - # SET TO YOUR replication server URL: BASEURL="https://download.geofabrik.de" @@ -46,27 +34,24 @@ DOWNCOUNTRYPOSTFIX="-latest.osm.pbf" # End of configuration section # ****************************************************************************** -COMBINEFILES="osmium merge" +UPDATEDIR=update +IMPORT_CMD="nominatim import" mkdir -p ${UPDATEDIR} -cd ${UPDATEDIR} +pushd ${UPDATEDIR} rm -rf tmp mkdir -p tmp -cd tmp +popd for COUNTRY in $COUNTRIES; do - echo "====================================================================" echo "$COUNTRY" echo "====================================================================" DIR="$UPDATEDIR/$COUNTRY" - FILE="$DIR/configuration.txt" DOWNURL="$BASEURL/$COUNTRY$DOWNCOUNTRYPOSTFIX" IMPORTFILE=$COUNTRY$DOWNCOUNTRYPOSTFIX IMPORTFILEPATH=${UPDATEDIR}/tmp/${IMPORTFILE} - FILENAME=${COUNTRY//[\/]/_} - touch2 $IMPORTFILEPATH wget ${DOWNURL} -O $IMPORTFILEPATH @@ -74,18 +59,12 @@ do touch2 ${DIR}/sequence.state pyosmium-get-changes -O $IMPORTFILEPATH -f ${DIR}/sequence.state -v - COMBINEFILES="${COMBINEFILES} ${IMPORTFILEPATH}" + IMPORT_CMD="${IMPORT_CMD} --osm-file ${IMPORTFILEPATH}" echo $IMPORTFILE echo "====================================================================" done - -echo "${COMBINEFILES} -o combined.osm.pbf" -${COMBINEFILES} -o combined.osm.pbf - echo "====================================================================" echo "Setting up nominatim db" -${SETUPFILE} --osm-file ${UPDATEDIR}/tmp/combined.osm.pbf --all 2>&1 - -# ${UPDATEFILE} --import-file ${UPDATEDIR}/tmp/combined.osm.pbf 2>&1 -echo "====================================================================" \ No newline at end of file +${IMPORT_CMD} 2>&1 +echo "====================================================================" diff --git a/utils/update_database.sh b/utils/update_database.sh index 75d0de5d..58f7690a 100644 --- a/utils/update_database.sh +++ b/utils/update_database.sh @@ -22,25 +22,14 @@ # REPLACE WITH LIST OF YOUR "COUNTRIES": # - - COUNTRIES="europe/monaco europe/andorra" -# SET TO YOUR NOMINATIM build FOLDER PATH: -# -NOMINATIMBUILD="/srv/nominatim/build" -UPDATEFILE="$NOMINATIMBUILD/utils/update.php" - -# SET TO YOUR update data FOLDER PATH: -# -UPDATEDIR="/srv/nominatim/update" - UPDATEBASEURL="https://download.geofabrik.de" UPDATECOUNTRYPOSTFIX="-updates" # If you do not use Photon, let Nominatim handle (re-)indexing: # -FOLLOWUP="$UPDATEFILE --index" +FOLLOWUP="nominatim index" # # If you use Photon, update Photon and let it handle the index # (Photon server must be running and must have been started with "-database", @@ -49,11 +38,10 @@ FOLLOWUP="$UPDATEFILE --index" #FOLLOWUP="curl http://localhost:2322/nominatim-update" # ****************************************************************************** - +UPDATEDIR="update" for COUNTRY in $COUNTRIES; do - echo "====================================================================" echo "$COUNTRY" echo "====================================================================" @@ -61,20 +49,16 @@ do FILE="$DIR/sequence.state" BASEURL="$UPDATEBASEURL/$COUNTRY$UPDATECOUNTRYPOSTFIX" FILENAME=${COUNTRY//[\/]/_} - - # mkdir -p ${DIR} - cd ${DIR} echo "Attempting to get changes" + rm -f ${DIR}/${FILENAME}.osc.gz pyosmium-get-changes -o ${DIR}/${FILENAME}.osc.gz -f ${FILE} --server $BASEURL -v echo "Attempting to import diffs" - ${NOMINATIMBUILD}/utils/update.php --import-diff ${DIR}/${FILENAME}.osc.gz - rm ${DIR}/${FILENAME}.osc.gz - + nominatim add-data --diff ${DIR}/${FILENAME}.osc.gz done echo "====================================================================" echo "Reindexing" ${FOLLOWUP} -echo "====================================================================" \ No newline at end of file +echo "===================================================================="