mirror of
https://github.com/osm-search/Nominatim.git
synced 2024-07-07 08:06:23 +03:00
port multi-region update scripts to nominatim tool
Also updates the documentation. For the simple case of just importing multiple regions, provide simplified instructions that use the new multi-file import feature. Fixes #2365.
This commit is contained in:
Sarah Hoffmann
parent
95b82af42a
commit
56d24085f9
|
|
@ -5,9 +5,34 @@ your Nominatim database. It is assumed that you have already successfully
|
|||
installed the Nominatim software itself, if not return to the
|
||||
[installation page](Installation.md).
|
||||
|
||||
## Importing multiple regions
|
||||
## Importing multiple regions (without updates)
|
||||
|
||||
To import multiple regions in your database, you need to configure and run `utils/import_multiple_regions.sh` file. This script will set up the update directory which has the following structure:
|
||||
To import multiple regions in your database you can simply give multiple
|
||||
OSM files to the import command:
|
||||
|
||||
```
|
||||
nominatim import --osm-file file1.pbf --osm-file file2.pbf
|
||||
```
|
||||
|
||||
If you already have imported a file and want to add another one, you can
|
||||
use the add-data function to import the additional data as follows:
|
||||
|
||||
```
|
||||
nominatim add-data --file <FILE>
|
||||
nominatim refresh --postcodes
|
||||
nominatim index -j <NUMBER OF THREADS>
|
||||
```
|
||||
|
||||
Please note that adding additional data is always significantly slower than
|
||||
the original import.
|
||||
|
||||
## Importing multiple regions (with updates)
|
||||
|
||||
If you want to import multiple regions _and_ be able to keep them up-to-date
|
||||
with updates, then you can use the scripts provided in the `utils` directory.
|
||||
|
||||
These scripts will set up an `update` directory in your project directory,
|
||||
which has the following structure:
|
||||
|
||||
```bash
|
||||
update
|
||||
|
|
@ -17,7 +42,6 @@ update
|
|||
│ └── monaco
|
||||
│ └── sequence.state
|
||||
└── tmp
|
||||
├── combined.osm.pbf
|
||||
└── europe
|
||||
├── andorra-latest.osm.pbf
|
||||
└── monaco-latest.osm.pbf
|
||||
|
|
@ -25,85 +49,57 @@ update
|
|||
|
||||
```
|
||||
|
||||
The `sequence.state` files will contain the sequence ID, which will be used by pyosmium to get updates. The tmp folder is used for import dump.
|
||||
The `sequence.state` files contain the sequence ID for each region. They will
|
||||
be used by pyosmium to get updates. The `tmp` folder is used for import dump and
|
||||
can be deleted once the import is complete.
|
||||
|
||||
### Configuring multiple regions
|
||||
|
||||
The file `import_multiple_regions.sh` needs to be edited as per your requirement:
|
||||
|
||||
1. List of countries. eg:
|
||||
|
||||
COUNTRIES="europe/monaco europe/andorra"
|
||||
|
||||
2. Path to Build directory. eg:
|
||||
|
||||
NOMINATIMBUILD="/srv/nominatim/build"
|
||||
|
||||
3. Path to Update directory. eg:
|
||||
|
||||
UPDATEDIR="/srv/nominatim/update"
|
||||
|
||||
4. Replication URL. eg:
|
||||
|
||||
BASEURL="https://download.geofabrik.de"
|
||||
DOWNCOUNTRYPOSTFIX="-latest.osm.pbf"
|
||||
|
||||
### Setting up multiple regions
|
||||
|
||||
!!! tip
|
||||
If your database already exists and you want to add more countries,
|
||||
replace the setting up part
|
||||
`${SETUPFILE} --osm-file ${UPDATEDIR}/tmp/combined.osm.pbf --all 2>&1`
|
||||
with `${UPDATEFILE} --import-file ${UPDATEDIR}/tmp/combined.osm.pbf --index --index-instances N 2>&1`
|
||||
where N is the numbers of CPUs in your system.
|
||||
Create a project directory as described for the
|
||||
[simple import](Import.md#creating-the-project-directory). If necessary,
|
||||
you can also add an `.env` configuration with customized options. In particular,
|
||||
you need to make sure that `NOMINATIM_REPLICATION_UPDATE_INTERVAL` and
|
||||
`NOMINATIM_REPLICATION_RECHECK_INTERVAL` are set according to the update
|
||||
interval of the extract server you use.
|
||||
|
||||
Run the following command from your Nominatim directory after configuring the file.
|
||||
Copy the scripts `utils/import_multiple_regions.sh` and `utils/update_database.sh`
|
||||
into the project directory.
|
||||
|
||||
bash ./utils/import_multiple_regions.sh
|
||||
Now customize both files as per your requirements
|
||||
|
||||
!!! danger "Important"
|
||||
This file uses osmium-tool. It must be installed before executing the import script.
|
||||
Installation instructions can be found [here](https://osmcode.org/osmium-tool/manual.html#installation).
|
||||
|
||||
### Updating multiple regions
|
||||
|
||||
To import multiple regions in your database, you need to configure and run ```utils/update_database.sh```.
|
||||
This uses the update directory set up while setting up the DB.
|
||||
|
||||
### Configuring multiple regions
|
||||
|
||||
The file `update_database.sh` needs to be edited as per your requirement:
|
||||
|
||||
1. List of countries. eg:
|
||||
1. List of countries. e.g.
|
||||
|
||||
COUNTRIES="europe/monaco europe/andorra"
|
||||
|
||||
2. Path to Build directory. eg:
|
||||
2. URL to the service providing the extracts and updates. eg:
|
||||
|
||||
NOMINATIMBUILD="/srv/nominatim/build"
|
||||
|
||||
3. Path to Update directory. eg:
|
||||
|
||||
UPDATEDIR="/srv/nominatim/update"
|
||||
|
||||
4. Replication URL. eg:
|
||||
|
||||
BASEURL="https://download.geofabrik.de"
|
||||
DOWNCOUNTRYPOSTFIX="-updates"
|
||||
DOWNCOUNTRYPOSTFIX="-latest.osm.pbf"
|
||||
|
||||
5. Followup can be set according to your installation. eg: For Photon,
|
||||
5. Followup in the update script can be set according to your installation.
|
||||
E.g. for Photon,
|
||||
|
||||
FOLLOWUP="curl http://localhost:2322/nominatim-update"
|
||||
|
||||
will handle the indexing.
|
||||
|
||||
|
||||
To start the initial import, change into the project directory and run
|
||||
|
||||
```
|
||||
bash import_multiple_regions.sh
|
||||
```
|
||||
|
||||
### Updating the database
|
||||
|
||||
Run the following command from your Nominatim directory after configuring the file.
|
||||
Change into the project directory and run the following command:
|
||||
|
||||
bash ./utils/update_database.sh
|
||||
bash update_database.sh
|
||||
|
||||
This will get diffs from the replication server, import diffs and index the database. The default replication server in the script([Geofabrik](https://download.geofabrik.de)) provides daily updates.
|
||||
This will get diffs from the replication server, import diffs and index
|
||||
the database. The default replication server in the
|
||||
script([Geofabrik](https://download.geofabrik.de)) provides daily updates.
|
||||
|
||||
## Importing Nominatim to an external PostgreSQL database
|
||||
|
||||
|
|
|
|||
|
|
@ -8,8 +8,6 @@
|
|||
|
||||
# *) Set up sequence.state for updates
|
||||
|
||||
# *) Merge the pbf files into a single file.
|
||||
|
||||
# *) Setup nominatim db using 'setup.php --osm-file'
|
||||
|
||||
# Hint:
|
||||
|
|
@ -28,16 +26,6 @@ touch2() { mkdir -p "$(dirname "$1")" && touch "$1" ; }
|
|||
|
||||
COUNTRIES="europe/monaco europe/andorra"
|
||||
|
||||
# SET TO YOUR NOMINATIM build FOLDER PATH:
|
||||
|
||||
NOMINATIMBUILD="/srv/nominatim/build"
|
||||
SETUPFILE="$NOMINATIMBUILD/utils/setup.php"
|
||||
UPDATEFILE="$NOMINATIMBUILD/utils/update.php"
|
||||
|
||||
# SET TO YOUR update FOLDER PATH:
|
||||
|
||||
UPDATEDIR="/srv/nominatim/update"
|
||||
|
||||
# SET TO YOUR replication server URL:
|
||||
|
||||
BASEURL="https://download.geofabrik.de"
|
||||
|
|
@ -46,27 +34,24 @@ DOWNCOUNTRYPOSTFIX="-latest.osm.pbf"
|
|||
# End of configuration section
|
||||
# ******************************************************************************
|
||||
|
||||
COMBINEFILES="osmium merge"
|
||||
UPDATEDIR=update
|
||||
IMPORT_CMD="nominatim import"
|
||||
|
||||
mkdir -p ${UPDATEDIR}
|
||||
cd ${UPDATEDIR}
|
||||
pushd ${UPDATEDIR}
|
||||
rm -rf tmp
|
||||
mkdir -p tmp
|
||||
cd tmp
|
||||
popd
|
||||
|
||||
for COUNTRY in $COUNTRIES;
|
||||
do
|
||||
|
||||
echo "===================================================================="
|
||||
echo "$COUNTRY"
|
||||
echo "===================================================================="
|
||||
DIR="$UPDATEDIR/$COUNTRY"
|
||||
FILE="$DIR/configuration.txt"
|
||||
DOWNURL="$BASEURL/$COUNTRY$DOWNCOUNTRYPOSTFIX"
|
||||
IMPORTFILE=$COUNTRY$DOWNCOUNTRYPOSTFIX
|
||||
IMPORTFILEPATH=${UPDATEDIR}/tmp/${IMPORTFILE}
|
||||
FILENAME=${COUNTRY//[\/]/_}
|
||||
|
||||
|
||||
touch2 $IMPORTFILEPATH
|
||||
wget ${DOWNURL} -O $IMPORTFILEPATH
|
||||
|
|
@ -74,18 +59,12 @@ do
|
|||
touch2 ${DIR}/sequence.state
|
||||
pyosmium-get-changes -O $IMPORTFILEPATH -f ${DIR}/sequence.state -v
|
||||
|
||||
COMBINEFILES="${COMBINEFILES} ${IMPORTFILEPATH}"
|
||||
IMPORT_CMD="${IMPORT_CMD} --osm-file ${IMPORTFILEPATH}"
|
||||
echo $IMPORTFILE
|
||||
echo "===================================================================="
|
||||
done
|
||||
|
||||
|
||||
echo "${COMBINEFILES} -o combined.osm.pbf"
|
||||
${COMBINEFILES} -o combined.osm.pbf
|
||||
|
||||
echo "===================================================================="
|
||||
echo "Setting up nominatim db"
|
||||
${SETUPFILE} --osm-file ${UPDATEDIR}/tmp/combined.osm.pbf --all 2>&1
|
||||
|
||||
# ${UPDATEFILE} --import-file ${UPDATEDIR}/tmp/combined.osm.pbf 2>&1
|
||||
echo "===================================================================="
|
||||
${IMPORT_CMD} 2>&1
|
||||
echo "===================================================================="
|
||||
|
|
|
|||
|
|
@ -22,25 +22,14 @@
|
|||
|
||||
# REPLACE WITH LIST OF YOUR "COUNTRIES":
|
||||
#
|
||||
|
||||
|
||||
COUNTRIES="europe/monaco europe/andorra"
|
||||
|
||||
# SET TO YOUR NOMINATIM build FOLDER PATH:
|
||||
#
|
||||
NOMINATIMBUILD="/srv/nominatim/build"
|
||||
UPDATEFILE="$NOMINATIMBUILD/utils/update.php"
|
||||
|
||||
# SET TO YOUR update data FOLDER PATH:
|
||||
#
|
||||
UPDATEDIR="/srv/nominatim/update"
|
||||
|
||||
UPDATEBASEURL="https://download.geofabrik.de"
|
||||
UPDATECOUNTRYPOSTFIX="-updates"
|
||||
|
||||
# If you do not use Photon, let Nominatim handle (re-)indexing:
|
||||
#
|
||||
FOLLOWUP="$UPDATEFILE --index"
|
||||
FOLLOWUP="nominatim index"
|
||||
#
|
||||
# If you use Photon, update Photon and let it handle the index
|
||||
# (Photon server must be running and must have been started with "-database",
|
||||
|
|
@ -49,11 +38,10 @@ FOLLOWUP="$UPDATEFILE --index"
|
|||
#FOLLOWUP="curl http://localhost:2322/nominatim-update"
|
||||
|
||||
# ******************************************************************************
|
||||
|
||||
UPDATEDIR="update"
|
||||
|
||||
for COUNTRY in $COUNTRIES;
|
||||
do
|
||||
|
||||
echo "===================================================================="
|
||||
echo "$COUNTRY"
|
||||
echo "===================================================================="
|
||||
|
|
@ -61,20 +49,16 @@ do
|
|||
FILE="$DIR/sequence.state"
|
||||
BASEURL="$UPDATEBASEURL/$COUNTRY$UPDATECOUNTRYPOSTFIX"
|
||||
FILENAME=${COUNTRY//[\/]/_}
|
||||
|
||||
# mkdir -p ${DIR}
|
||||
cd ${DIR}
|
||||
|
||||
echo "Attempting to get changes"
|
||||
rm -f ${DIR}/${FILENAME}.osc.gz
|
||||
pyosmium-get-changes -o ${DIR}/${FILENAME}.osc.gz -f ${FILE} --server $BASEURL -v
|
||||
|
||||
echo "Attempting to import diffs"
|
||||
${NOMINATIMBUILD}/utils/update.php --import-diff ${DIR}/${FILENAME}.osc.gz
|
||||
rm ${DIR}/${FILENAME}.osc.gz
|
||||
|
||||
nominatim add-data --diff ${DIR}/${FILENAME}.osc.gz
|
||||
done
|
||||
|
||||
echo "===================================================================="
|
||||
echo "Reindexing"
|
||||
${FOLLOWUP}
|
||||
echo "===================================================================="
|
||||
echo "===================================================================="
|
||||
|
|
|
|||
Loading…
Reference in New Issue
Block a user