Merge pull request #1758 from krahulreddy/advanced-installations

Advanced installations

docs/admin/Advanced-Installations.md

@@ -0,0 +1,109 @@
+# Advanced installations
+
+This page contains instructions for setting up multiple countries in 
+your Nominatim database. It is assumed that you have already successfully
+installed the Nominatim software itself, if not return to the 
+[installation page](Installation.md).
+
+## Importing multiple regions
+
+To import multiple regions in your database, you need to configure and run `utils/import_multiple_regions.sh` file. This script will set up the update directory which has the following structure:
+
+```bash
+update
+    ├── europe
+    │   ├── andorra
+    │   │   └── sequence.state
+    │   └── monaco
+    │       └── sequence.state
+    └── tmp
+        ├── combined.osm.pbf
+        └── europe
+                ├── andorra-latest.osm.pbf
+                └── monaco-latest.osm.pbf
+
+
+```
+
+The `sequence.state` files will contain the sequence ID, which will be used by pyosmium to get updates. The tmp folder is used for import dump.
+
+### Configuring multiple regions
+
+The file `import_multiple_regions.sh` needs to be edited as per your requirement:
+
+1. List of countries. eg:
+
+        COUNTRIES="europe/monaco europe/andorra"
+
+2. Path to Build directory. eg:
+
+        NOMINATIMBUILD="/srv/nominatim/build"
+
+3. Path to Update directory. eg:
+        
+        UPDATEDIR="/srv/nominatim/update"
+
+4. Replication URL. eg:
+    
+        BASEURL="https://download.geofabrik.de"
+        DOWNCOUNTRYPOSTFIX="-latest.osm.pbf"
+ 
+!!! tip
+    If your database already exists and you want to add more countries, replace the setting up part
+    `${SETUPFILE} --osm-file ${UPDATEDIR}/tmp/combined.osm.pbf --all 2>&1`
+    with `${UPDATEFILE} --import-file ${UPDATEDIR}/tmp/combined.osm.pbf 2>&1`.
+
+### Setting up multiple regions
+
+Run the following command from your Nominatim directory after configuring the file.
+
+    bash ./utils/import_multiple_regions.sh
+
+!!! danger "Important"
+        This file uses osmium-tool. It must be installed before executing the import script.
+        Installation instructions can be found [here](https://osmcode.org/osmium-tool/manual.html#installation).
+
+## Updating multiple regions
+
+To import multiple regions in your database, you need to configure and run ```utils/update_database.sh```.
+This uses the update directory set up while setting up the DB.   
+
+### Configuring multiple regions
+
+The file `update_database.sh` needs to be edited as per your requirement:
+
+1. List of countries. eg:
+
+        COUNTRIES="europe/monaco europe/andorra"
+
+2. Path to Build directory. eg:
+
+        NOMINATIMBUILD="/srv/nominatim/build"
+
+3. Path to Update directory. eg:
+        
+        UPDATEDIR="/srv/nominatim/update"
+
+4. Replication URL. eg:
+    
+        BASEURL="https://download.geofabrik.de"
+        DOWNCOUNTRYPOSTFIX="-updates"
+
+5. Followup can be set according to your installation. eg: For Photon,
+
+        FOLLOWUP="curl http://localhost:2322/nominatim-update"
+
+    will handle the indexing.
+
+### Updating the database
+
+Run the following command from your Nominatim directory after configuring the file.
+
+    bash ./utils/update_database.sh
+
+This will get diffs from the replication server, import diffs and index the database. The default replication server in the script([Geofabric](https://download.geofabrik.de)) provides daily updates.
+
+## Verification and further setup
+
+Instructions for import verification and other details like importing Wikidata can be found in [import and update page](Import-and-Update.md)
+

docs/admin/Import-and-Update.md

@@ -318,6 +318,5 @@ compatibility reasons, Osmosis is not required to run this - it uses pyosmium
 behind the scenes.)
 
 If you have imported multiple country extracts and want to keep them
-up-to-date, have a look at the script in
-[issue #60](https://github.com/openstreetmap/Nominatim/issues/60).
-
+up-to-date, [Advanced installations section](Advanced-Installations.md) contains instructions 
+to set up and update multiple country extracts.

docs/mkdocs.yml

@@ -16,6 +16,7 @@ pages:
     - 'Administration Guide':
         - 'Basic Installation': 'admin/Installation.md'
         - 'Importing and Updating' : 'admin/Import-and-Update.md'
+        - 'Advanced Installations' : 'admin/Advanced-Installations.md'
         - 'Migration from older Versions' : 'admin/Migration.md'
         - 'Troubleshooting' : 'admin/Faq.md'
     - 'Developers Guide':

utils/import_multiple_regions.sh

@@ -0,0 +1,91 @@
+#!/bin/bash -xv
+
+# Script to set up Nominatim database for multiple countries
+
+# Steps to follow:
+
+#     *) Get the pbf files from server
+
+#     *) Set up sequence.state for updates
+
+#     *) Merge the pbf files into a single file.
+
+#     *) Setup nominatim db using 'setup.php --osm-file'
+
+# Hint:
+#
+# Use "bashdb ./update_database.sh" and bashdb's "next" command for step-by-step
+# execution.
+
+# ******************************************************************************
+
+touch2() { mkdir -p "$(dirname "$1")" && touch "$1" ; }
+
+# ******************************************************************************
+# Configuration section: Variables in this section should be set according to your requirements
+
+# REPLACE WITH LIST OF YOUR "COUNTRIES":
+
+COUNTRIES="europe/monaco europe/andorra"
+
+# SET TO YOUR NOMINATIM build FOLDER PATH:
+
+NOMINATIMBUILD="/srv/nominatim/build"
+SETUPFILE="$NOMINATIMBUILD/utils/setup.php"
+UPDATEFILE="$NOMINATIMBUILD/utils/update.php"
+
+# SET TO YOUR update FOLDER PATH:
+
+UPDATEDIR="/srv/nominatim/update"
+
+# SET TO YOUR replication server URL:
+
+BASEURL="https://download.geofabrik.de"
+DOWNCOUNTRYPOSTFIX="-latest.osm.pbf"
+
+# End of configuration section
+# ******************************************************************************
+
+COMBINEFILES="osmium merge"
+
+mkdir -p ${UPDATEDIR}
+cd ${UPDATEDIR}
+rm -rf tmp
+mkdir -p tmp
+cd tmp
+
+for COUNTRY in $COUNTRIES;
+do
+    
+    echo "===================================================================="
+    echo "$COUNTRY"
+    echo "===================================================================="
+    DIR="$UPDATEDIR/$COUNTRY"
+    FILE="$DIR/configuration.txt"
+    DOWNURL="$BASEURL/$COUNTRY$DOWNCOUNTRYPOSTFIX"
+    IMPORTFILE=$COUNTRY$DOWNCOUNTRYPOSTFIX
+    IMPORTFILEPATH=${UPDATEDIR}/tmp/${IMPORTFILE}
+    FILENAME=${COUNTRY//[\/]/_}
+    
+
+    touch2 $IMPORTFILEPATH
+    wget ${DOWNURL} -O $IMPORTFILEPATH
+
+    touch2 ${DIR}/sequence.state
+    pyosmium-get-changes -O $IMPORTFILEPATH -f ${DIR}/sequence.state -v
+
+    COMBINEFILES="${COMBINEFILES} ${IMPORTFILEPATH}"
+    echo $IMPORTFILE
+    echo "===================================================================="
+done
+
+
+echo "${COMBINEFILES} -o combined.osm.pbf"
+${COMBINEFILES} -o combined.osm.pbf
+
+echo "===================================================================="
+echo "Setting up nominatim db"
+${SETUPFILE} --osm-file ${UPDATEDIR}/tmp/combined.osm.pbf --all 2>&1
+
+# ${UPDATEFILE} --import-file ${UPDATEDIR}/tmp/combined.osm.pbf 2>&1
+echo "===================================================================="

utils/update_database.sh

@@ -0,0 +1,80 @@
+#!/bin/bash -xv
+
+# Derived from https://gist.github.com/RhinoDevel/8a35ebd2a08166f328eca01ab005c6de and edited to work with Pyosmium
+# Related to https://github.com/osm-search/Nominatim/issues/1683
+
+# Steps being followed:
+
+#     *) Get the diff file from server
+#         1) pyosmium-get-changes (with -f sequence.state for getting sequenceNumber)
+
+#     *) Import diff
+#         1) utils/update.php --import-diff
+
+#     *) Index for all the countries at the end
+
+# Hint:
+#
+# Use "bashdb ./update_database.sh" and bashdb's "next" command for step-by-step
+# execution.
+
+# ******************************************************************************
+
+# REPLACE WITH LIST OF YOUR "COUNTRIES":
+#
+
+
+COUNTRIES="europe/monaco europe/andorra"
+
+# SET TO YOUR NOMINATIM build FOLDER PATH:
+#
+NOMINATIMBUILD="/srv/nominatim/build"
+UPDATEFILE="$NOMINATIMBUILD/utils/update.php"
+
+# SET TO YOUR update data FOLDER PATH:
+#
+UPDATEDIR="/srv/nominatim/update"
+
+UPDATEBASEURL="https://download.geofabrik.de"
+UPDATECOUNTRYPOSTFIX="-updates"
+
+# If you do not use Photon, let Nominatim handle (re-)indexing:
+#
+FOLLOWUP="$UPDATEFILE --index"
+#
+# If you use Photon, update Photon and let it handle the index
+# (Photon server must be running and must have been started with "-database",
+# "-user" and "-password" parameters):
+#
+#FOLLOWUP="curl http://localhost:2322/nominatim-update"
+
+# ******************************************************************************
+
+
+for COUNTRY in $COUNTRIES;
+do
+    
+    echo "===================================================================="
+    echo "$COUNTRY"
+    echo "===================================================================="
+    DIR="$UPDATEDIR/$COUNTRY"
+    FILE="$DIR/sequence.state"
+    BASEURL="$UPDATEBASEURL/$COUNTRY$UPDATECOUNTRYPOSTFIX"
+    FILENAME=${COUNTRY//[\/]/_}
+    
+    # mkdir -p ${DIR}
+    cd ${DIR}
+
+    echo "Attempting to get changes"
+    pyosmium-get-changes -o ${DIR}/${FILENAME}.osc.gz -f ${FILE} --server $BASEURL -v
+
+    echo "Attempting to import diffs"
+    ${NOMINATIMBUILD}/utils/update.php --import-diff ${DIR}/${FILENAME}.osc.gz
+    rm ${DIR}/${FILENAME}.osc.gz
+
+done
+
+echo "===================================================================="
+echo "Reindexing" 
+${FOLLOWUP}
+echo "===================================================================="