Commit Graph

255 Commits

Author SHA1 Message Date
Dustin Carlino
272589bbbc Stop checking for osmconvert, unzip, and gunzip up-front in the importer. #602
Originally this check was useful for people to discover missing
dependencies before spending time trying to import. But the importing
process has changed considerably since then -- much of the time, it gets
called by one_step_import through the UI. It has no need for those 3
commands in that case. If somebody winds up needing any of these
external commands, they'll still get a good error eventually.
2021-04-12 17:10:18 -07:00
Dustin Carlino
361bd7b8ed Shrink boundary around Tempe to focus on the campus 2021-04-12 14:28:49 -07:00
Dustin Carlino
417850cbed Import a part of Singapore 2021-04-11 08:44:49 -07:00
Dustin Carlino
c6906f6153 Change Vec<Lane> to BTreeMap<LaneID, Lane> to prepare for
adding/deleting lanes. #597

Not regenerating all maps yet. Haven't evaluated performance impact yet.
2021-04-09 14:35:49 -07:00
Michael Kirk
61c46461e4
PERF: lower memory requirements for clip_osm (#585)
* PERF: lower memory requirements for clip_osm

- don't accumulate things unless they are at least partially within the boundary
- accumulate id's, not entire objects.

This entails doing a second pass of the file, but keeps memory usage
much lower.

e.g. a largeish area of Los Angeles, covering east hollywood to downtown, clipped thus:
- north: San Fernando Blvd
- east: LA River east of Downtown
- south: the 10
- west: Western ave,

Before: importing the above stalled out for me during clipping, which
had ballooned into swap at more than 16GB.

After: in the above scenario, clipping peaks around 55MB, and the import
process as a whole peaks during pathfinding - at around 4.0GB.

* fixup! PERF: lower memory requirements for clip_osm

Remove unnecessary casting
2021-03-24 18:03:33 -07:00
Dustin Carlino
162f09a887 Import Tripoli for WeCode Libya, also adding an Arabic font 2021-03-23 06:51:47 -07:00
Dustin Carlino
1187c1ccf0 Bikes can use bus lanes in GB. Only disallowed in Poland now.
Not regenerating yet.
2021-03-22 18:50:26 -07:00
Dustin Carlino
0572a5118b Add a map covering the north end of Cambridge. Also grab fresh OSM for the Great Kneighton map, at least fixing the crazy geometry at Portugal Place. cyipt/actdev#151 [rebuild] [release] 2021-03-21 11:24:33 -07:00
Dustin Carlino
01431827dd Fix a bug in the UI importer that happens the first time you run it. #523 2021-03-20 16:31:31 -07:00
Dustin Carlino
58ee3175ac Add a new Seattle map focused on the Arboretum, for #574, along with
prebaked data for the baseline (because it doesn't gridlock, without any
interventions needed!) and a proposal for opening up Broadmoor.
2021-03-19 17:24:11 -07:00
Dustin Carlino
d7b7c2d7cd Import a slice of London along the A5, including scenario data. It
gridlocks, of course.  #577
2021-03-19 15:51:43 -07:00
Dustin Carlino
ffc4f4222d Remove the old SRTM code, which has always been broken. Remove the elevation from importer config; the new approach will detect the right data source to use automatically using the bbox. #82 2021-03-18 10:14:43 -07:00
Dustin Carlino
565f362d73 Speed up clip_osm. From 45s for washington-latest->ballard down to 25s, just by buffering IO. Derp. #523 2021-03-18 09:13:11 -07:00
Dustin Carlino
24a60b0b29 Import Tempe, AZ for collaboration with ASU 2021-03-17 10:55:04 -07:00
Dustin Carlino
a0c82088b9 Use our native download function instead of external curl command from the importer. #523
This forces the main importer to include tokio and propagate async a
bit. That seems worth it.

Also removed the quiet param from the download helpers; we always want
progress.
2021-03-17 10:29:28 -07:00
Dustin Carlino
c1d110d506 Fix the double </osm> issue with osmio. #523 2021-03-16 12:43:30 -07:00
Dustin Carlino
77709a24d6 Refactor a method to asynchronously download a file on native, with a simple progress bar. #523
Use it in the updater and all the importer tools. Only place it's not
used yet is parking_mapper and the in-game updater.  [rebuild]

Also make the RunCommand state understand the control code for erasing
the current line. It... sort of works.
2021-03-15 19:51:45 -07:00
Dustin Carlino
359960c1db Prepare the new map importer to work in the release. [rebuild] 2021-03-15 14:16:18 -07:00
Dustin Carlino
c0e49f815f Make the UI load the newly imported map, scraping its name from the other tool. #523
For this to work, RunCommand has to be much more careful about getting
all output when the command is done.
2021-03-15 13:54:14 -07:00
Dustin Carlino
f87c1c3b1b Manage the temporary files in the importer better, and don't re-download
pbf. #523

But now we don't know which map to switch to in the UI!
2021-03-15 11:43:06 -07:00
Dustin Carlino
ef95252f4b Make clip_osm correctly retain ways that cross the boundary. #523 2021-03-15 11:07:11 -07:00
Dustin Carlino
04e0e2aa02 Cache the geofabrik index file in data/input/shared. #523 2021-03-15 10:58:11 -07:00
Dustin Carlino
944b2c337a Get the importer UI working finally! #523
Improved the target dir finding, and hacking around some of the command
output not rendering.
2021-03-13 13:41:34 -08:00
Dustin Carlino
4e665b6214 Run the one-step importer from the UI. Don't attempt a proper loading
screen yet. #523

Not working yet; one_step assumes paths to binaries, and it's now wrong.
2021-03-13 13:33:27 -08:00
Dustin Carlino
ac297f75c4 Hack around the double </osm> bug in clip_map. #523 2021-03-13 11:39:18 -08:00
Dustin Carlino
59fdc30981 Create a tool to go from just a GeoJSON boundary to the full imported map, in one step, by gluing together a bunch of other tools. #523
Breaks in a few places along the way, but the basic idea is there.
2021-03-13 11:26:33 -08:00
Dustin Carlino
27a07b02a6 Use the osmio crate to convert OSM pbf to xml, clipping it in the
process. #523

Tested really quickly, might not be working right yet. The .xml output
seems to have a duplicate '</osm>' for some reason.
2021-03-13 10:06:19 -08:00
Dustin Carlino
95b1d884cf Rename Map::new to be more clear that it should only be used in non-UI
tools. There's one place in the UI that doesn't work on the web!
2021-03-12 06:38:17 -08:00
Dustin Carlino
f0de4e4583 Don't use Bezier curves for right/left turns at tiny intersections. They
wind up looping back on themselves in a nonsensical way, causing
vehicles to visually glitch when moving through.

This was started in 081819d86b, but it
used to gridlock 2 maps. All the recent roundabout fixes seems to have
resolved those! And adjusting offstreet parking for two maps.

But wallingford does regress; plunging forward for now.
2021-03-10 15:58:16 -08:00
Dustin Carlino
6594597c3f Rebuild actdev scenarios again, remembering to include return and lunch trips. cyipt/actdev#132 2021-03-08 13:17:52 -08:00
Dustin Carlino
5970883cff Regenerating all scenario data for actdev, since cyipt/actdev#132 is
fixed
2021-03-08 08:59:30 -08:00
Dustin Carlino
0357ae0a23 Tidy up the new UK census gen code. #556 2021-03-05 16:51:02 -08:00
Dustin Carlino
8ffd249558 Get the new UK scenario generation working everywhere. #556
If a study area exists for the map, make a copy of the base/active
scenarios with the background traffic mixed in. Also remove people
living in the site, since they're redundant.

Ran it like this: for city in `ls data/system/gb/`; do ./import.sh
--scenario --city=gb/$city || break; done
2021-03-05 16:26:29 -08:00
Dustin Carlino
04be956b5e Remove no-op trips from the UK scenarios. They now work (but gridlock, of course). #556 2021-03-05 15:01:26 -08:00
Dustin Carlino
a7b7638405 Fix data sources for UK ODs. Thanks Robin! #556 2021-03-05 14:48:18 -08:00
Dustin Carlino
89f317c79d Add in normal distributions for trip departure times. #556 2021-03-05 12:24:17 -08:00
Dustin Carlino
c7042eda06 Create simple people for UK scenarios. #556
Can't quite debug yet, because for Poundbury, the matching zones (like
E02004277) have no entries in the census data...
2021-03-05 11:59:20 -08:00
Dustin Carlino
1f33916502 Match houses/workplaces to zones. #556 2021-03-05 11:27:25 -08:00
Dustin Carlino
aa59677ad2 Transform the UK OD and zone data from raw input into a clean structure. #556 2021-03-05 10:47:48 -08:00
Dustin Carlino
22dd1424db Download UK OD and zone definitions. #556 2021-03-05 09:59:08 -08:00
Dustin Carlino
92b89001ab Move road safety data to a new data/input/shared directory, so we can stop duplicating files common to many cities. About to use this for OD and zone data too. #556 2021-03-05 09:50:18 -08:00
Dustin Carlino
42bd63e79b Lift URL and viewport code from game to map_gui. Also use it for OSM viewer and the 15min tool. 2021-03-04 13:22:09 -08:00
Dustin Carlino
6d2587c0f9 Keep the 400MB zoning parcels file in memory in the importer, instead of loading it for every single Seattle map. 2021-03-04 09:27:29 -08:00
Dustin Carlino
dd42acdc28 Refactor a CityName::seattle() 2021-03-03 18:08:06 -08:00
Dustin Carlino
9e92435859 Get 2 more Seattle maps to complete without gridlock! Just had to
increase offstreet parking to better match the demand.
2021-03-03 14:53:04 -08:00
Dustin Carlino
372b8e23d7 Tool to add lunch trips to people in an existing scenario. #539 2021-03-01 13:33:59 -08:00
Dustin Carlino
99bf415ede Tool to add return trips for a scenario. #539 2021-03-01 12:42:17 -08:00
Dustin Carlino
18a0197fba Just starting the skeleton of a tool to modify scenarios for #539. 2021-03-01 11:21:45 -08:00
Dustin Carlino
bce3318605 Import halsnead for actdev. It needed a fix to not treat access=private
as a no-through-traffic zone for light rail tracks!
2021-02-27 14:35:26 -08:00
Dustin Carlino
603717a560 Remove the separate_cycleways experiment flag. Always import cyclepaths
mapped as separate OSM ways in all maps!!! #330
2021-02-26 14:58:57 -08:00