mirror of
https://github.com/osm-search/Nominatim.git
synced 2024-10-05 15:08:32 +03:00
port multi-region update scripts to nominatim tool
Also updates the documentation. For the simple case of just importing multiple regions, provide simplified instructions that use the new multi-file import feature. Fixes #2365.
This commit is contained in:
parent
95b82af42a
commit
56d24085f9
@ -5,9 +5,34 @@ your Nominatim database. It is assumed that you have already successfully
|
|||||||
installed the Nominatim software itself, if not return to the
|
installed the Nominatim software itself, if not return to the
|
||||||
[installation page](Installation.md).
|
[installation page](Installation.md).
|
||||||
|
|
||||||
## Importing multiple regions
|
## Importing multiple regions (without updates)
|
||||||
|
|
||||||
To import multiple regions in your database, you need to configure and run `utils/import_multiple_regions.sh` file. This script will set up the update directory which has the following structure:
|
To import multiple regions in your database you can simply give multiple
|
||||||
|
OSM files to the import command:
|
||||||
|
|
||||||
|
```
|
||||||
|
nominatim import --osm-file file1.pbf --osm-file file2.pbf
|
||||||
|
```
|
||||||
|
|
||||||
|
If you already have imported a file and want to add another one, you can
|
||||||
|
use the add-data function to import the additional data as follows:
|
||||||
|
|
||||||
|
```
|
||||||
|
nominatim add-data --file <FILE>
|
||||||
|
nominatim refresh --postcodes
|
||||||
|
nominatim index -j <NUMBER OF THREADS>
|
||||||
|
```
|
||||||
|
|
||||||
|
Please note that adding additional data is always significantly slower than
|
||||||
|
the original import.
|
||||||
|
|
||||||
|
## Importing multiple regions (with updates)
|
||||||
|
|
||||||
|
If you want to import multiple regions _and_ be able to keep them up-to-date
|
||||||
|
with updates, then you can use the scripts provided in the `utils` directory.
|
||||||
|
|
||||||
|
These scripts will set up an `update` directory in your project directory,
|
||||||
|
which has the following structure:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
update
|
update
|
||||||
@ -17,7 +42,6 @@ update
|
|||||||
│ └── monaco
|
│ └── monaco
|
||||||
│ └── sequence.state
|
│ └── sequence.state
|
||||||
└── tmp
|
└── tmp
|
||||||
├── combined.osm.pbf
|
|
||||||
└── europe
|
└── europe
|
||||||
├── andorra-latest.osm.pbf
|
├── andorra-latest.osm.pbf
|
||||||
└── monaco-latest.osm.pbf
|
└── monaco-latest.osm.pbf
|
||||||
@ -25,85 +49,57 @@ update
|
|||||||
|
|
||||||
```
|
```
|
||||||
|
|
||||||
The `sequence.state` files will contain the sequence ID, which will be used by pyosmium to get updates. The tmp folder is used for import dump.
|
The `sequence.state` files contain the sequence ID for each region. They will
|
||||||
|
be used by pyosmium to get updates. The `tmp` folder is used for import dump and
|
||||||
|
can be deleted once the import is complete.
|
||||||
|
|
||||||
### Configuring multiple regions
|
|
||||||
|
|
||||||
The file `import_multiple_regions.sh` needs to be edited as per your requirement:
|
|
||||||
|
|
||||||
1. List of countries. eg:
|
|
||||||
|
|
||||||
COUNTRIES="europe/monaco europe/andorra"
|
|
||||||
|
|
||||||
2. Path to Build directory. eg:
|
|
||||||
|
|
||||||
NOMINATIMBUILD="/srv/nominatim/build"
|
|
||||||
|
|
||||||
3. Path to Update directory. eg:
|
|
||||||
|
|
||||||
UPDATEDIR="/srv/nominatim/update"
|
|
||||||
|
|
||||||
4. Replication URL. eg:
|
|
||||||
|
|
||||||
BASEURL="https://download.geofabrik.de"
|
|
||||||
DOWNCOUNTRYPOSTFIX="-latest.osm.pbf"
|
|
||||||
|
|
||||||
### Setting up multiple regions
|
### Setting up multiple regions
|
||||||
|
|
||||||
!!! tip
|
Create a project directory as described for the
|
||||||
If your database already exists and you want to add more countries,
|
[simple import](Import.md#creating-the-project-directory). If necessary,
|
||||||
replace the setting up part
|
you can also add an `.env` configuration with customized options. In particular,
|
||||||
`${SETUPFILE} --osm-file ${UPDATEDIR}/tmp/combined.osm.pbf --all 2>&1`
|
you need to make sure that `NOMINATIM_REPLICATION_UPDATE_INTERVAL` and
|
||||||
with `${UPDATEFILE} --import-file ${UPDATEDIR}/tmp/combined.osm.pbf --index --index-instances N 2>&1`
|
`NOMINATIM_REPLICATION_RECHECK_INTERVAL` are set according to the update
|
||||||
where N is the numbers of CPUs in your system.
|
interval of the extract server you use.
|
||||||
|
|
||||||
Run the following command from your Nominatim directory after configuring the file.
|
Copy the scripts `utils/import_multiple_regions.sh` and `utils/update_database.sh`
|
||||||
|
into the project directory.
|
||||||
|
|
||||||
bash ./utils/import_multiple_regions.sh
|
Now customize both files as per your requirements
|
||||||
|
|
||||||
!!! danger "Important"
|
1. List of countries. e.g.
|
||||||
This file uses osmium-tool. It must be installed before executing the import script.
|
|
||||||
Installation instructions can be found [here](https://osmcode.org/osmium-tool/manual.html#installation).
|
|
||||||
|
|
||||||
### Updating multiple regions
|
|
||||||
|
|
||||||
To import multiple regions in your database, you need to configure and run ```utils/update_database.sh```.
|
|
||||||
This uses the update directory set up while setting up the DB.
|
|
||||||
|
|
||||||
### Configuring multiple regions
|
|
||||||
|
|
||||||
The file `update_database.sh` needs to be edited as per your requirement:
|
|
||||||
|
|
||||||
1. List of countries. eg:
|
|
||||||
|
|
||||||
COUNTRIES="europe/monaco europe/andorra"
|
COUNTRIES="europe/monaco europe/andorra"
|
||||||
|
|
||||||
2. Path to Build directory. eg:
|
2. URL to the service providing the extracts and updates. eg:
|
||||||
|
|
||||||
NOMINATIMBUILD="/srv/nominatim/build"
|
|
||||||
|
|
||||||
3. Path to Update directory. eg:
|
|
||||||
|
|
||||||
UPDATEDIR="/srv/nominatim/update"
|
|
||||||
|
|
||||||
4. Replication URL. eg:
|
|
||||||
|
|
||||||
BASEURL="https://download.geofabrik.de"
|
BASEURL="https://download.geofabrik.de"
|
||||||
DOWNCOUNTRYPOSTFIX="-updates"
|
DOWNCOUNTRYPOSTFIX="-latest.osm.pbf"
|
||||||
|
|
||||||
5. Followup can be set according to your installation. eg: For Photon,
|
5. Followup in the update script can be set according to your installation.
|
||||||
|
E.g. for Photon,
|
||||||
|
|
||||||
FOLLOWUP="curl http://localhost:2322/nominatim-update"
|
FOLLOWUP="curl http://localhost:2322/nominatim-update"
|
||||||
|
|
||||||
will handle the indexing.
|
will handle the indexing.
|
||||||
|
|
||||||
|
|
||||||
|
To start the initial import, change into the project directory and run
|
||||||
|
|
||||||
|
```
|
||||||
|
bash import_multiple_regions.sh
|
||||||
|
```
|
||||||
|
|
||||||
### Updating the database
|
### Updating the database
|
||||||
|
|
||||||
Run the following command from your Nominatim directory after configuring the file.
|
Change into the project directory and run the following command:
|
||||||
|
|
||||||
bash ./utils/update_database.sh
|
bash update_database.sh
|
||||||
|
|
||||||
This will get diffs from the replication server, import diffs and index the database. The default replication server in the script([Geofabrik](https://download.geofabrik.de)) provides daily updates.
|
This will get diffs from the replication server, import diffs and index
|
||||||
|
the database. The default replication server in the
|
||||||
|
script([Geofabrik](https://download.geofabrik.de)) provides daily updates.
|
||||||
|
|
||||||
## Importing Nominatim to an external PostgreSQL database
|
## Importing Nominatim to an external PostgreSQL database
|
||||||
|
|
||||||
|
@ -8,8 +8,6 @@
|
|||||||
|
|
||||||
# *) Set up sequence.state for updates
|
# *) Set up sequence.state for updates
|
||||||
|
|
||||||
# *) Merge the pbf files into a single file.
|
|
||||||
|
|
||||||
# *) Setup nominatim db using 'setup.php --osm-file'
|
# *) Setup nominatim db using 'setup.php --osm-file'
|
||||||
|
|
||||||
# Hint:
|
# Hint:
|
||||||
@ -28,16 +26,6 @@ touch2() { mkdir -p "$(dirname "$1")" && touch "$1" ; }
|
|||||||
|
|
||||||
COUNTRIES="europe/monaco europe/andorra"
|
COUNTRIES="europe/monaco europe/andorra"
|
||||||
|
|
||||||
# SET TO YOUR NOMINATIM build FOLDER PATH:
|
|
||||||
|
|
||||||
NOMINATIMBUILD="/srv/nominatim/build"
|
|
||||||
SETUPFILE="$NOMINATIMBUILD/utils/setup.php"
|
|
||||||
UPDATEFILE="$NOMINATIMBUILD/utils/update.php"
|
|
||||||
|
|
||||||
# SET TO YOUR update FOLDER PATH:
|
|
||||||
|
|
||||||
UPDATEDIR="/srv/nominatim/update"
|
|
||||||
|
|
||||||
# SET TO YOUR replication server URL:
|
# SET TO YOUR replication server URL:
|
||||||
|
|
||||||
BASEURL="https://download.geofabrik.de"
|
BASEURL="https://download.geofabrik.de"
|
||||||
@ -46,27 +34,24 @@ DOWNCOUNTRYPOSTFIX="-latest.osm.pbf"
|
|||||||
# End of configuration section
|
# End of configuration section
|
||||||
# ******************************************************************************
|
# ******************************************************************************
|
||||||
|
|
||||||
COMBINEFILES="osmium merge"
|
UPDATEDIR=update
|
||||||
|
IMPORT_CMD="nominatim import"
|
||||||
|
|
||||||
mkdir -p ${UPDATEDIR}
|
mkdir -p ${UPDATEDIR}
|
||||||
cd ${UPDATEDIR}
|
pushd ${UPDATEDIR}
|
||||||
rm -rf tmp
|
rm -rf tmp
|
||||||
mkdir -p tmp
|
mkdir -p tmp
|
||||||
cd tmp
|
popd
|
||||||
|
|
||||||
for COUNTRY in $COUNTRIES;
|
for COUNTRY in $COUNTRIES;
|
||||||
do
|
do
|
||||||
|
|
||||||
echo "===================================================================="
|
echo "===================================================================="
|
||||||
echo "$COUNTRY"
|
echo "$COUNTRY"
|
||||||
echo "===================================================================="
|
echo "===================================================================="
|
||||||
DIR="$UPDATEDIR/$COUNTRY"
|
DIR="$UPDATEDIR/$COUNTRY"
|
||||||
FILE="$DIR/configuration.txt"
|
|
||||||
DOWNURL="$BASEURL/$COUNTRY$DOWNCOUNTRYPOSTFIX"
|
DOWNURL="$BASEURL/$COUNTRY$DOWNCOUNTRYPOSTFIX"
|
||||||
IMPORTFILE=$COUNTRY$DOWNCOUNTRYPOSTFIX
|
IMPORTFILE=$COUNTRY$DOWNCOUNTRYPOSTFIX
|
||||||
IMPORTFILEPATH=${UPDATEDIR}/tmp/${IMPORTFILE}
|
IMPORTFILEPATH=${UPDATEDIR}/tmp/${IMPORTFILE}
|
||||||
FILENAME=${COUNTRY//[\/]/_}
|
|
||||||
|
|
||||||
|
|
||||||
touch2 $IMPORTFILEPATH
|
touch2 $IMPORTFILEPATH
|
||||||
wget ${DOWNURL} -O $IMPORTFILEPATH
|
wget ${DOWNURL} -O $IMPORTFILEPATH
|
||||||
@ -74,18 +59,12 @@ do
|
|||||||
touch2 ${DIR}/sequence.state
|
touch2 ${DIR}/sequence.state
|
||||||
pyosmium-get-changes -O $IMPORTFILEPATH -f ${DIR}/sequence.state -v
|
pyosmium-get-changes -O $IMPORTFILEPATH -f ${DIR}/sequence.state -v
|
||||||
|
|
||||||
COMBINEFILES="${COMBINEFILES} ${IMPORTFILEPATH}"
|
IMPORT_CMD="${IMPORT_CMD} --osm-file ${IMPORTFILEPATH}"
|
||||||
echo $IMPORTFILE
|
echo $IMPORTFILE
|
||||||
echo "===================================================================="
|
echo "===================================================================="
|
||||||
done
|
done
|
||||||
|
|
||||||
|
|
||||||
echo "${COMBINEFILES} -o combined.osm.pbf"
|
|
||||||
${COMBINEFILES} -o combined.osm.pbf
|
|
||||||
|
|
||||||
echo "===================================================================="
|
echo "===================================================================="
|
||||||
echo "Setting up nominatim db"
|
echo "Setting up nominatim db"
|
||||||
${SETUPFILE} --osm-file ${UPDATEDIR}/tmp/combined.osm.pbf --all 2>&1
|
${IMPORT_CMD} 2>&1
|
||||||
|
echo "===================================================================="
|
||||||
# ${UPDATEFILE} --import-file ${UPDATEDIR}/tmp/combined.osm.pbf 2>&1
|
|
||||||
echo "===================================================================="
|
|
||||||
|
@ -22,25 +22,14 @@
|
|||||||
|
|
||||||
# REPLACE WITH LIST OF YOUR "COUNTRIES":
|
# REPLACE WITH LIST OF YOUR "COUNTRIES":
|
||||||
#
|
#
|
||||||
|
|
||||||
|
|
||||||
COUNTRIES="europe/monaco europe/andorra"
|
COUNTRIES="europe/monaco europe/andorra"
|
||||||
|
|
||||||
# SET TO YOUR NOMINATIM build FOLDER PATH:
|
|
||||||
#
|
|
||||||
NOMINATIMBUILD="/srv/nominatim/build"
|
|
||||||
UPDATEFILE="$NOMINATIMBUILD/utils/update.php"
|
|
||||||
|
|
||||||
# SET TO YOUR update data FOLDER PATH:
|
|
||||||
#
|
|
||||||
UPDATEDIR="/srv/nominatim/update"
|
|
||||||
|
|
||||||
UPDATEBASEURL="https://download.geofabrik.de"
|
UPDATEBASEURL="https://download.geofabrik.de"
|
||||||
UPDATECOUNTRYPOSTFIX="-updates"
|
UPDATECOUNTRYPOSTFIX="-updates"
|
||||||
|
|
||||||
# If you do not use Photon, let Nominatim handle (re-)indexing:
|
# If you do not use Photon, let Nominatim handle (re-)indexing:
|
||||||
#
|
#
|
||||||
FOLLOWUP="$UPDATEFILE --index"
|
FOLLOWUP="nominatim index"
|
||||||
#
|
#
|
||||||
# If you use Photon, update Photon and let it handle the index
|
# If you use Photon, update Photon and let it handle the index
|
||||||
# (Photon server must be running and must have been started with "-database",
|
# (Photon server must be running and must have been started with "-database",
|
||||||
@ -49,11 +38,10 @@ FOLLOWUP="$UPDATEFILE --index"
|
|||||||
#FOLLOWUP="curl http://localhost:2322/nominatim-update"
|
#FOLLOWUP="curl http://localhost:2322/nominatim-update"
|
||||||
|
|
||||||
# ******************************************************************************
|
# ******************************************************************************
|
||||||
|
UPDATEDIR="update"
|
||||||
|
|
||||||
for COUNTRY in $COUNTRIES;
|
for COUNTRY in $COUNTRIES;
|
||||||
do
|
do
|
||||||
|
|
||||||
echo "===================================================================="
|
echo "===================================================================="
|
||||||
echo "$COUNTRY"
|
echo "$COUNTRY"
|
||||||
echo "===================================================================="
|
echo "===================================================================="
|
||||||
@ -61,20 +49,16 @@ do
|
|||||||
FILE="$DIR/sequence.state"
|
FILE="$DIR/sequence.state"
|
||||||
BASEURL="$UPDATEBASEURL/$COUNTRY$UPDATECOUNTRYPOSTFIX"
|
BASEURL="$UPDATEBASEURL/$COUNTRY$UPDATECOUNTRYPOSTFIX"
|
||||||
FILENAME=${COUNTRY//[\/]/_}
|
FILENAME=${COUNTRY//[\/]/_}
|
||||||
|
|
||||||
# mkdir -p ${DIR}
|
|
||||||
cd ${DIR}
|
|
||||||
|
|
||||||
echo "Attempting to get changes"
|
echo "Attempting to get changes"
|
||||||
|
rm -f ${DIR}/${FILENAME}.osc.gz
|
||||||
pyosmium-get-changes -o ${DIR}/${FILENAME}.osc.gz -f ${FILE} --server $BASEURL -v
|
pyosmium-get-changes -o ${DIR}/${FILENAME}.osc.gz -f ${FILE} --server $BASEURL -v
|
||||||
|
|
||||||
echo "Attempting to import diffs"
|
echo "Attempting to import diffs"
|
||||||
${NOMINATIMBUILD}/utils/update.php --import-diff ${DIR}/${FILENAME}.osc.gz
|
nominatim add-data --diff ${DIR}/${FILENAME}.osc.gz
|
||||||
rm ${DIR}/${FILENAME}.osc.gz
|
|
||||||
|
|
||||||
done
|
done
|
||||||
|
|
||||||
echo "===================================================================="
|
echo "===================================================================="
|
||||||
echo "Reindexing"
|
echo "Reindexing"
|
||||||
${FOLLOWUP}
|
${FOLLOWUP}
|
||||||
echo "===================================================================="
|
echo "===================================================================="
|
||||||
|
Loading…
Reference in New Issue
Block a user