osm-search/Nominatim
1
1
Fork 0
mirror of https://github.com/osm-search/Nominatim.git synced 2024-11-23 13:44:36 +03:00
Code Issues Packages Projects Releases Wiki Activity
4,170 Commits 28 Branches 41 Tags 164 MiB
36d068871d
Commit Graph

649 Commits

Author SHA1 Message Date
Sarah Hoffmann
d7bc846c3c fix liniting issues and add type annotations 2023-01-03 10:03:00 +01:00
Sarah Hoffmann
1adb0a9886 add support for starlette framework 2023-01-03 10:03:00 +01:00
Sarah Hoffmann
8aa01c9c8e add support for falcon as server framework 2023-01-03 10:03:00 +01:00
Sarah Hoffmann
23dabad0b0 add sanic development server implementation 2023-01-03 10:02:53 +01:00
Sarah Hoffmann
45c675bd78 implement command line status call in Python 2023-01-03 10:02:35 +01:00
Sarah Hoffmann
200eae3bc0 add tests for examples in lua style documentation 
And fix all the errors the tests have found.
 2022-12-23 17:35:28 +01:00
Sarah Hoffmann
c32f686c62 switch to osm2pgsql flex style by default 2022-12-18 10:10:58 +01:00
Sarah Hoffmann
6f51c1ba33 remove code that disables processing of forward dependencies 2022-12-11 19:35:58 +01:00
Sarah Hoffmann
2abe9e6fd9 use data paths from new nominatim.paths 2022-11-27 12:15:41 +01:00
Sarah Hoffmann
20f56dfc77 introduce paths module which contains the data paths 
This means that nominatim can find its data even when not called
from the command line tool.
 2022-11-27 12:02:37 +01:00
Sarah Hoffmann
1f3edf6eba prepare release 4.2.0 2022-11-24 10:43:29 +01:00
Sarah Hoffmann
13dbeb75c7
Merge pull request #2903 from lonvia/migration-for-index-reorganization 
Add migration for reorganization of pending indexes
 2022-11-24 10:13:38 +01:00
Sarah Hoffmann
6aded60045 add migration for reorganization of pending indexes 
Fixes #2900.
 2022-11-24 08:48:05 +01:00
Sarah Hoffmann
fd3dec8efe add sanitizer for TIGER tags 
Currently only takes over cleaning the tiger:county data. This was
done by the import until now.
 2022-11-23 10:37:27 +01:00
Sarah Hoffmann
b6ff697ff0 add experimental option for enabling forward dependencies 2022-11-21 14:48:00 +01:00
Sarah Hoffmann
74405e9684 add migration for place_to_be_deleted table 2022-11-10 09:38:25 +01:00
Sarah Hoffmann
2fac507453 change updates to handle delete/insert workflow 
This makes Nominatim compatible with osm2pgsql's default update
modus operandi of deleting and reinserting data. Deletes are diverted
into a TODO table instead of executing them. When data is reinserted,
the corresponding entry in the TODO table is deleted. After updates are
finished, the remaining entries in the TODO table are executed, doing
the same work as the delete trigger did before.

The new behaviour also works against the gazetteer output with its
insert-only mechanism.
 2022-11-10 09:38:23 +01:00
Sarah Hoffmann
51ed55cc32 initial flex import scripts 
Only implements the extratags style for the moment. Tests pass
for the same behaviour as the gazetteer output. Updates still need
to be done.
 2022-11-10 09:37:38 +01:00
Sarah Hoffmann
64c591da7f fix type issues with calls to pyosmium 2022-11-09 20:46:33 +01:00
Sarah Hoffmann
6ddb39fda3 respect socket timeout also in other replication functions 2022-11-09 09:12:37 +01:00
Sarah Hoffmann
1fdcec985a fix timeout use for replication timeout 
The timeout parameter is no longer taken into account since
pyosmium switched to the requests library. This adds the parameter
back.
 2022-11-09 09:12:37 +01:00
Sarah Hoffmann
abf349fb0d simplify use of secondary importance 
The values in the raster are already normalized between 0 and 2**16,
so a simple conversion to [0, 1] will do.

Check for existance of secondary_importance table statically when
creating the SQL function. For that to work importance tables need
to be created before the functions.
 2022-10-01 11:01:49 +02:00
Sarah Hoffmann
3185fad918 load views as a SQL file and rename to 'secondary importance' 
The only requirement for secondary importance is that a raster table
comes out of it. The generic name leaves open where the data comes
from.
 2022-10-01 11:01:49 +02:00
Tareq Al-Ahdal
0ab0f0ea44 Integrated OSM views into importance computation 2022-10-01 11:01:49 +02:00
Tareq Al-Ahdal
ac467c7a2d Enhanced the implementation of OSM views GeoTIFF import functionality 2022-10-01 11:01:49 +02:00
Tareq Al-Ahdal
c85b74497b Initial implementation of GeoTIFF import functionality 2022-10-01 11:01:49 +02:00
Sarah Hoffmann
a2ee58d8a1 only run analyze on indexing when work was done 
This speeds up processing when continuing indexing after it was
interrupted.
 2022-09-28 10:22:54 +02:00
Sarah Hoffmann
f4d3ae6f70 consolidate indexes over geometry_sectors 
The index over geometry_sectors are mainly used for ordering
the places which need indexing. That means they function effectively
as a TODO list. Consolodate them so that they always only contain
the places which are still to do. Also add the appropriate index
for the boundary indexing phase.
 2022-09-21 10:38:58 +02:00
Sarah Hoffmann
ed3dd81d04 run final index creation in parallel 2022-09-19 11:55:25 +02:00
Sarah Hoffmann
bef1aebf1c add function for parallel execution of SQL scripts 2022-09-19 11:52:17 +02:00
Sarah Hoffmann
bc94318d83 mypy: fix new warnings due to external type updates 2022-09-05 17:39:35 +02:00
Tareq Al-Ahdal
465d82a92f Integrated 'collect_os_info.py' into Nominatim's CLI tool 2022-08-13 06:18:10 +08:00
Tareq Al-Ahdal
49f889bf09 Enhanced and refactored 'collect_os_info.py' 
Changed the script to functional programming paradigm to remove the big number of local attributes to decrease memory usage when running it. Additional OS info are now included.
 2022-08-13 06:13:05 +08:00
Sarah Hoffmann
8d082c13e0 adapt to new type annotations from typeshed 
Some more functions frrom psycopg are now properly annotated.
No ignoring necessary anymore.
 2022-08-09 11:06:54 +02:00
Sarah Hoffmann
4fe797d704 remove mypy ignore for psutil.virtual_memory() 
Now available in typeshed.
 2022-08-08 09:44:45 +02:00
Sarah Hoffmann
7d68aa8f04 prepare release 4.1.0 2022-08-05 14:33:11 +02:00
Sarah Hoffmann
b19c90b9a6 export centroid to tokenizer 
May come in handy when developping sanitizers for an area smaller
than country size.
 2022-07-31 22:10:58 +02:00
Sarah Hoffmann
9864b191b1 fix various typos 2022-07-31 17:10:35 +02:00
Sarah Hoffmann
51b6d16dc6 overhaul the token analysis interface 
The functional split betweenthe two functions is now that the
first one creates the ID that is used in the word table and
the second one creates the variants. There no longer is a
requirement that the ID is the normalized version. We might
later reintroduce the requirement that a normalized version be available
but it doesn't necessarily need to be through the ID.

The function that creates the ID now gets the full PlaceName. That way
it might take into account attributes that were set by the sanitizers.

Finally rename both functions to something more sane.
 2022-07-29 15:14:11 +02:00
Sarah Hoffmann
34d27ed45c move PlaceName into the generic data module 2022-07-29 11:42:20 +02:00
Sarah Hoffmann
094100bbf6 harmonize spelling 
Stick with the American spelling of Analyze.
 2022-07-29 10:52:01 +02:00
Sarah Hoffmann
c8873d34af harmonize interface of token analysis module 
The configure() function now receives a Transliterator object instead
of the ICU rules. This harmonizes the parameters with the create
function.
 2022-07-29 10:43:07 +02:00
Sarah Hoffmann
f0d640961a add documentation for custom token analysis 2022-07-29 09:41:28 +02:00
Sarah Hoffmann
3746befd88 add documentation for sanitizer interface 
Also switches mkdocstrings to 0.18 with the rather unfortunate
consequence that now mkdocstrings-python-legacy is needed as well.
 2022-07-28 22:00:29 +02:00
Sarah Hoffmann
d819036daa add support for external token analysis modules 2022-07-25 16:27:22 +02:00
Sarah Hoffmann
6d41046b15 add support for external sanitizer modules 2022-07-25 16:10:19 +02:00
Sarah Hoffmann
7b7203c149 add function for loading plugin modules 
Loads modules for configurable code like tokenizers, sanitizers, etc.
Supports internal modules, external libraries and code from the
project directory.
 2022-07-25 16:10:10 +02:00
Kian-Meng Ang
f5e52e748f docs: fix typos 2022-07-20 22:05:31 +08:00
Sarah Hoffmann
5aad105c73 add explicit cast for fetchone 2022-07-18 10:18:51 +02:00
Sarah Hoffmann
83054af46f remove typing_extensions requirement 
The typing_extensions package is only necessary now when running mypy.
It won't be used at runtime anymore.
 2022-07-18 09:55:58 +02:00
Sarah Hoffmann
a849f3c9ec add type annotations for command line functions 2022-07-18 09:55:54 +02:00
Sarah Hoffmann
25d854dc5c add type annotations for Tiger import function 2022-07-18 09:54:29 +02:00
Sarah Hoffmann
9963261d8d add type annotations to special phrase importer 2022-07-18 09:54:29 +02:00
Sarah Hoffmann
459ab3bbdc add type annotations to database check functions 2022-07-18 09:54:29 +02:00
Sarah Hoffmann
a21d4d3ac4 add type annotations for database import functions 2022-07-18 09:54:29 +02:00
Sarah Hoffmann
4da1f0da6f add type annotations for migrations 2022-07-18 09:54:29 +02:00
Sarah Hoffmann
17bbe2637a add type annotations to tool functions 2022-07-18 09:54:27 +02:00
Sarah Hoffmann
6c6bbe5747 add type annotations for ICU tokenizer 2022-07-18 09:47:57 +02:00
Sarah Hoffmann
18b16e06ca add type annotations for legacy tokenizer 2022-07-18 09:47:57 +02:00
Sarah Hoffmann
e37cfc64d2 add type annotations to ICU tokenizer helper modules 2022-07-18 09:47:57 +02:00
Sarah Hoffmann
d35e3c25b6 add type annotations for token analysis 
No annotations for ICU types yet.
 2022-07-18 09:47:57 +02:00
Sarah Hoffmann
62eedbb8f6 add type hints for sanitizers 2022-07-18 09:47:57 +02:00
Sarah Hoffmann
5617bffe2f add type annotations for indexer 2022-07-18 09:47:57 +02:00
Sarah Hoffmann
8adab2c6ca add typing information for postcode formatter 2022-07-18 09:47:57 +02:00
Sarah Hoffmann
d0c44431d0 add typing information for place_info and country_info 2022-07-18 09:47:57 +02:00
Sarah Hoffmann
282a61ce51 add typing information for utils submodule 2022-07-18 09:47:57 +02:00
Sarah Hoffmann
7a1d22ff15 type annotations for non-blocking DB connection 2022-07-18 09:47:57 +02:00
Sarah Hoffmann
0dff71a410 add type annotations for SQL preprocessor 2022-07-18 09:47:57 +02:00
Sarah Hoffmann
26f30bff28 add type annotation to DB utils 
As a cursor is needed as type, make this a public type.
 2022-07-18 09:47:57 +02:00
Sarah Hoffmann
e6775e713c add typing information to DB properties 2022-07-18 09:47:57 +02:00
Sarah Hoffmann
69f9122bef add typing annotations for DB status module 
Requires TypedDict which is only available from Python 3.8. Require
therefore typing_extensions to make the functions available for
earlier Python versions.
 2022-07-18 09:47:57 +02:00
Sarah Hoffmann
845c43137a add type annotations to freeze functions 2022-07-18 09:47:57 +02:00
Sarah Hoffmann
aaf2b6032e fix uses of config.get_path() to expect None 2022-07-18 09:47:57 +02:00
Sarah Hoffmann
c4928c646d define type for enivronment dictionaries 2022-07-18 09:47:57 +02:00
Sarah Hoffmann
f12fe54d2b restrict return type more 2022-07-18 09:47:57 +02:00
Sarah Hoffmann
fc03c0266a add type annotations to exec_utils 2022-07-18 09:47:57 +02:00
Sarah Hoffmann
681aad7e0d avoid issues with Python < 3.9 and linting 2022-07-18 09:47:57 +02:00
Sarah Hoffmann
f22fa992f7 move complex typing annotations to extra file 2022-07-18 09:47:57 +02:00
Sarah Hoffmann
992e6f72cf type annotations for DB utils 2022-07-18 09:47:57 +02:00
Sarah Hoffmann
e6ee3c772c type annotations for DB connection 2022-07-18 09:47:57 +02:00
Sarah Hoffmann
95ed95c616 add type annotations to config module 2022-07-18 09:47:57 +02:00
Sarah Hoffmann
bf36f33e79 add type annotations for version.py 2022-07-18 09:47:57 +02:00
Sarah Hoffmann
9b636fdc10 mypy: minimal annotations to enable a clean run 2022-07-18 09:47:57 +02:00
Sarah Hoffmann
4b12d52ef5 convert admin --analyse-indexing to new indexing method 
A proper run of indexing requires the place information from the
analyzer. Add the pre-processing of place data, so the right
information is handed into the update function.
 2022-07-07 16:20:08 +02:00
Sarah Hoffmann
856925d19b remove analyze() from PlaceInfo class 
The function creates circular dependencies.
 2022-07-07 12:06:58 +02:00
Sarah Hoffmann
cbbcbb1fd7 move country_info into data submodule 2022-07-06 11:08:36 +02:00
Sarah Hoffmann
bce93d60bd move PlaceInfo into data submodule 
This data structure is shared between indexer and tokenizer.
 2022-07-06 10:54:47 +02:00
Sarah Hoffmann
612d34930b handle postcodes properly on word table updates 
update_postcodes_from_db() needs to do the full postcode treatment
in order to derive the correct word table entries.
 2022-06-23 23:42:31 +02:00
Sarah Hoffmann
5be320368c add documentation for postcode customization 2022-06-23 23:42:31 +02:00
Sarah Hoffmann
7f2ad4ac7e fix linting issue 2022-06-23 23:42:31 +02:00
Sarah Hoffmann
0f00f4968c fix up BDD tests for postcode changes 
Includes smaller code fixes found by the tests.
 2022-06-23 23:42:31 +02:00
Sarah Hoffmann
37b2c6a830 port legacy tokenizer to new postcode handling 
Also documents the changes to the SQL functions of the tokenizer.
 2022-06-23 23:42:31 +02:00
Sarah Hoffmann
e86db3001f fix postcode pattern for Mozambique 
Optional groups are not implemented yet.
 2022-06-23 23:42:31 +02:00
Sarah Hoffmann
67dfa38e60 fix liniting problems 2022-06-23 23:42:31 +02:00
Sarah Hoffmann
2eca9fc8af cache postcode normalization 2022-06-23 23:42:31 +02:00
Sarah Hoffmann
b5e5efc131 only add well-formatted postcodes to location table 2022-06-23 23:42:31 +02:00
Sarah Hoffmann
80ea13437d move postcode matcher in a separate file 2022-06-23 23:42:31 +02:00
Sarah Hoffmann
bf86b45178 move postcode centroid computation to Python 2022-06-23 23:42:31 +02:00
Sarah Hoffmann
4885fdf0f9 add class for online centroid computation 2022-06-23 23:42:31 +02:00
Sarah Hoffmann
b7704833e4 icu: switch postcodes to using the pre-formatted one 2022-06-23 23:42:31 +02:00