3014 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Sarah Hoffmann	50b6d7298c	factor out async connection handling into separate class ... Also adds a test for reconnecting regularly while indexing.	2021-04-20 14:08:37 +02:00
Sarah Hoffmann	26a81654a8	indexer: make self.conn function-local ... Also switches to our internal connect function which gives us a cursor with a sclar() function.	2021-04-20 14:08:37 +02:00
Sarah Hoffmann	6430371d7d	make index() function private	2021-04-20 14:08:37 +02:00
Sarah Hoffmann	18705b3f18	move analyse function into indexinf function	2021-04-20 14:08:37 +02:00
Sarah Hoffmann	c6bd2bb7fb	indexer: move runner into separate file	2021-04-20 14:08:37 +02:00
Sarah Hoffmann	c4fd94bd1a	Merge pull request #2284 from lonvia/cleanup-word-frequency-computation ... Rename and simplify function for word pre-computation	2021-04-19 18:28:04 +02:00
Sarah Hoffmann	b88b952f56	simplify token precomputation ... Rename function to reflect that it is only used for precomputation. The token IDs are not really needed, so don't bother to compute the array of tokens.	2021-04-19 17:24:19 +02:00
Sarah Hoffmann	d68b02d36a	remove unused word recomputation script ... Has been replaced by a script recomputing counts from search_name.	2021-04-19 16:40:57 +02:00
Sarah Hoffmann	b9b85eb208	Merge pull request #2283 from darkshredder/tiger-data-test-fix ... Fix: tiger-data tarfile test	2021-04-19 13:56:36 +02:00
Darkshredder	1f898405a6	Fix: tiger-data tarfile test	2021-04-19 16:02:52 +05:30
Sarah Hoffmann	6f6910101e	Merge pull request #2282 from lonvia/add-paths-to-config ... Include software paths in Python config object	2021-04-19 12:14:25 +02:00
Sarah Hoffmann	79d55357e8	simplify sql and website creation functions	2021-04-19 10:53:30 +02:00
Sarah Hoffmann	4fa6c0ad53	simplify constructor for SQL preprocessor ... Use sql path from config.	2021-04-19 10:26:25 +02:00
Sarah Hoffmann	8f63f9516b	simplify interface for adding tiger data ... Also simplifies tests using existing fixtures.	2021-04-19 10:26:25 +02:00
Sarah Hoffmann	995ba2c7c2	add library directories to config ... Allows to reduce the number of parameters in functions that take the config anyway.	2021-04-19 10:26:25 +02:00
Sarah Hoffmann	830e3be1e6	Merge pull request #2281 from changpingc/changping/fix-tiger-index ... fix index on location_property_tiger (parent_place_id)	2021-04-19 08:42:59 +02:00
Channgping Chen	29a314a092	fix index on location_property_tiger (parent_place_id) ... Looks like 2af82975cd accidentally renamed an index. Because of the added "if not exists" clause, the index doesn't get created. This significantly slows down reverse queries because they now require full scans on location_property_tiger. Without this fix, reverse queries can take 8s on a full planet install on an r5.8xlarge instance in EC2.	2021-04-19 00:33:15 +00:00
Sarah Hoffmann	abdba5fdc7	Merge pull request #2280 from AntoJvlt/Fix-special-phrases-import-and-tests-cleaning ... Fix regex and sanity check for the import of special phrases and tests cleaning.	2021-04-18 11:57:19 +02:00
AntoJvlt	b2ae715699	Only log a warning if a wrong input is detected on the wiki while importing special phrases	2021-04-17 20:19:39 +02:00
AntoJvlt	a95c748363	Fix occurence regex	2021-04-17 19:24:13 +02:00
AntoJvlt	ec859e41c6	Cleaned tests and add database cleaning tests on test_import_from_wiki	2021-04-17 19:23:33 +02:00
Sarah Hoffmann	7aeae9da81	Merge pull request #2279 from lonvia/add-index-for-continued-indexing ... Add index for continued indexing	2021-04-17 11:51:21 +02:00
Sarah Hoffmann	2ca11ccc6b	add tests for continuing import	2021-04-17 11:10:36 +02:00
Sarah Hoffmann	d74ae669e3	add support index when continuing import at index phase ... Indexing scans the placex table sequentially during indexing on the initial import. That is okay because we know that all rows need to be processed anywhere. When continuing the import, however, a large part might already be indexed, so that the process spends a lot of time going through rows that are no longer of interest. Create a supporting index for all unindexed rows to speed up the scan. This is the same index as used later for updates.	2021-04-17 11:07:04 +02:00
Sarah Hoffmann	9fabc5572d	Merge pull request #2278 from lonvia/remove-transistion-functions ... Remove transition functions	2021-04-17 10:13:33 +02:00
Sarah Hoffmann	da98a2102a	remove transition functions from Python	2021-04-16 18:41:14 +02:00
Sarah Hoffmann	fb3353b854	Merge pull request #2277 from lonvia/update-osm2pgsql ... Update osm2pgsql to current master	2021-04-16 17:40:43 +02:00
Sarah Hoffmann	b7e5c54593	remove PHP code for transition functions	2021-04-16 17:28:51 +02:00
Sarah Hoffmann	68beec5590	remove installation of PHP util scripts	2021-04-16 17:09:40 +02:00
Sarah Hoffmann	6ba06d1eb4	Merge pull request #2276 from lonvia/port-country-code-creation-to-python ... Port country code creation to python	2021-04-16 16:57:04 +02:00
Sarah Hoffmann	0f11e311c4	add test for new postcode import function	2021-04-16 16:11:20 +02:00
Sarah Hoffmann	886a01c796	port function to compute initial postcodes to Python	2021-04-16 16:11:20 +02:00
Sarah Hoffmann	a632b9f86a	Merge pull request #2275 from lonvia/switch-to-absolute-imports ... Use absolute imports in Python code	2021-04-16 15:04:10 +02:00
Sarah Hoffmann	76b1885595	use absolute imports in Python code ... Relative imports are no longer officially recommended.	2021-04-16 14:20:09 +02:00
Sarah Hoffmann	c55b409cf6	update osm2pgsql to current master (fixes version output)	2021-04-15 10:24:01 +02:00
Sarah Hoffmann	c64193f839	Merge pull request #2263 from AntoJvlt/special-phrases-autoupdate ... Implemented auto update of special phrases while importing them	2021-04-15 10:13:25 +02:00
Sarah Hoffmann	28a2a795ba	Merge pull request #2270 from lonvia/simplify-place-boundary-merge ... Simplify matching between place and boundary names	2021-04-15 10:12:53 +02:00
Sarah Hoffmann	e90adfc7c3	adapt database check to new index layout	2021-04-14 17:52:59 +02:00
Sarah Hoffmann	16267dc021	add migration for new placenode geometry index	2021-04-14 17:52:59 +02:00
Sarah Hoffmann	e7266b52ae	simplify name matching between boundary and place node ... Instead of normalising the names simply compare them in lower case. This removes the dependency on the tokenizer for linking boundaries and nodes. When looking up the linked places by place type also allow that one name is simply contained in the other. This catches the frequent case where one of the names has an addendum (e.g. Newport vs. City of Newport). Drops the special index for the name lookup and insted relies on a slightly extended version of the geometry index used for reverse lookup. Saves around 100MB on a planet.	2021-04-14 17:52:59 +02:00
Sarah Hoffmann	dc02610408	Merge pull request #2269 from lonvia/fix-actions ... github actions: reintroduce postgresql repo	2021-04-14 17:50:02 +02:00
Sarah Hoffmann	dc1bfe4a93	github actions: reintroduce postgresql repo	2021-04-14 17:25:44 +02:00
Sarah Hoffmann	cf69daaafb	Merge pull request #2264 from darkshredder/tiger-data-tests ... Fix: Error if last statements is wrong and improved tests in tiger data import	2021-04-14 10:56:12 +02:00
Darkshredder	49ee7505ed	Fix: Removed error if endstatement is wrong and improved tests	2021-04-13 15:44:12 +05:30
AntoJvlt	ae2b2cb9a5	Tests added for the auto update of special phrases during import	2021-04-12 14:35:29 +02:00
AntoJvlt	8c2f287ce4	Implemented auto update of special phrases while importing them	2021-04-12 14:30:48 +02:00
Sarah Hoffmann	2351f36315	Merge pull request #2260 from AntoJvlt/fix-load-languages-special-phrases ... Fix default languages loading for special phrases import	2021-04-11 23:09:45 +02:00
AntoJvlt	5ecae10713	Fix default languages loading	2021-04-11 22:26:31 +02:00
Sarah Hoffmann	2e3d657794	Merge pull request #2258 from darkshredder/code-coverage ... Disabled Code coverage status checks	2021-04-10 21:19:55 +02:00
Darkshredder	90f990b806	CodeCov comment only when codecoverage changes	2021-04-10 22:28:29 +05:30