Commit Graph

33 Commits

Author SHA1 Message Date
Dustin Carlino
007cf0b28d Split the importer into individual commands with structopt. #745 2021-10-25 13:02:00 -07:00
Dustin Carlino
9f86365cf6 Upgrade to rust 1.56 and cutover to rust edition 2021 2021-10-21 09:38:06 -07:00
Dustin Carlino
c28dba4b47 Make the importer itself a library, called from the CLI tool. #745 2021-09-17 18:29:23 -07:00
Dustin Carlino
60352747bf Get importer and game crates to share dependencies again, by enabling
some features for transitive deps in each crate.
2021-06-09 10:21:25 -07:00
Vinzent Steinberg
a1899e29f8
Update dependencies (#656)
* Update dependencies

* Use github for polylabel, to make all dependencies use geo 0.18, not a mix of 0.18 and 0.17

* Downgrade lyon to avoid a crash

Co-authored-by: Dustin Carlino <dabreegster@gmail.com>
2021-05-22 16:23:06 -07:00
Dustin Carlino
1edbca6509
Refactor most of the places extracting polygons from geojson files (#644) 2021-05-15 09:05:28 -07:00
Dustin Carlino
77709a24d6 Refactor a method to asynchronously download a file on native, with a simple progress bar. #523
Use it in the updater and all the importer tools. Only place it's not
used yet is parking_mapper and the in-game updater.  [rebuild]

Also make the RunCommand state understand the control code for erasing
the current line. It... sort of works.
2021-03-15 19:51:45 -07:00
Dustin Carlino
27a07b02a6 Use the osmio crate to convert OSM pbf to xml, clipping it in the
process. #523

Tested really quickly, might not be working right yet. The .xml output
seems to have a duplicate '</osm>' for some reason.
2021-03-13 10:06:19 -08:00
Dustin Carlino
a5bfa55c8e Create a tool to find the geofabrik osm.pbf file covering a particular region. Inspired by https://github.com/ropensci/osmextract. This will make importing more maps much easier, especially for cyipt/actdev#65. 2021-02-10 12:35:59 -08:00
Michael Kirk
4b665c871d
update geo related deps. (#508)
geo-types to 0.7
geo to 0.17 which removes redundant old geo used by polylabel
geojson to 0.22
2021-02-08 16:34:56 -08:00
Dustin Carlino
73e405e382 Upgrade most dependencies 2021-02-03 18:13:06 -08:00
Dustin Carlino
c9c6f240ae Remove timer.{note, warn, error} in favor of the log crate. #262
Originally, the intention of the deleted calls was to not interrupt
Timer progress bars with warnings. But the output of things like the
importer is impossible to read anyway. Strongly considering explicitly
sending logs and timing info to separate places and using something like
multitail for live progress.

Unplumb timer from LOADS of places that just needed it for logging.
2021-01-21 19:22:02 -08:00
Dustin Carlino
d49c6c5d7d Refactor the logic of distributing N people to K buildings. This was
duplicated between the Berlin importer and the new census-based popdat
crate, and I suspect some form of it might get used for the actdev
integration. #424
2021-01-13 11:18:51 -08:00
Dustin Carlino
e99def9124 Switch all error handling to anyhow. Nothing consumes our code as a
library yet, so this is a step forward from string errors. Manually
tested how errors show up (including the nice with_context bit).
2021-01-05 11:35:03 -08:00
Dustin Carlino
d529ae2e5c Upgrade usvg and rand 2021-01-04 09:19:54 -08:00
Dustin Carlino
965d999199 Upgrade misc other dependencies 2021-01-03 20:01:27 -08:00
Dustin Carlino
2945913fb3 Upgrade rand 2021-01-03 19:48:03 -08:00
Dustin Carlino
5c04923dff Split abstio crate out from abstutil. #253
widgetry, geom, and abstutil may wind up on crates.io in some form to
let other projects use widgetry. abstio has A/B Street-specific tricks
for reading data on native/web. Note widgetry still depends on abstio,
will figure out how to clean that up next.
2021-01-02 10:28:00 -08:00
Michael Kirk
8b9b3ce2b1
census areas import (#425)
#424
2020-12-14 17:48:47 -08:00
Dustin Carlino
aa4d848a0e Figure out the number of single family and multi-family units per OSM building, from parcel data. Just store it in an extra system file for now. 2020-11-30 10:58:47 -08:00
Dustin Carlino
aa966d7533 Upgrade a bunch of dependencies 2020-11-11 12:02:53 -08:00
Dustin Carlino
ad7a25f7f2 Prototype a simple data format for collision data. Import data for Leeds and London. #87 2020-11-05 16:16:40 -08:00
Dustin Carlino
597ef473f2 Upgrade most dependencies 2020-09-24 09:36:26 -07:00
Michael Kirk
6482f8b0a4
util to convert geojson to osmosis polygon (#309) 2020-09-02 15:29:25 -07:00
Dustin Carlino
651b4f5ec9 Some code style cleanups after #296 2020-08-27 09:10:39 -07:00
Kyle Huston
90cf38a522
Fail importer immediately if dependencies are missing and make gunzip etc. configurable (#296)
* Read ImporterConfiguration from .toml file if present. Also check for presence of dependency programs.

* Actually use config.osmconvert instead of hard-coded 'osmconvert'

* Add curl to checked dependencies

* Don't require all fields to be present in importer.toml configuration. Fill in defaults.

* Remove unneeded slice

* Use json for importer config and use serde(default) attribute to simplify application of defaults

* Extract re_dependencies_callable from main.rs and remove dependency on Job

* Revert change to abstutil, use struct-level default attribute

* Add importer.json to .gitignore

* Make gunzip_args string instead of Option<string>
2020-08-27 09:06:14 -07:00
Dustin Carlino
bbe06c6075 randomly distribute residents to buildings in the planning area for #119 2020-07-21 14:00:59 -07:00
Dustin Carlino
ddf6de5f59 stop dependending directly on serde_derive 2020-05-19 15:06:32 -07:00
Dustin Carlino
e082082c32 try again to bundle importer in release for oneshot. disable gdal.
[rebuild]
2020-05-17 09:25:44 -07:00
Dustin Carlino
0fa4b30521 turn soundcast parcels into extrashapes / kml to understand skipped
parcels
2020-05-16 10:26:59 -07:00
Dustin Carlino
8be319300b fold the popdat crate into importer; nothing else should have any dependencies on it 2020-04-25 13:14:48 -07:00
Dustin Carlino
bf8c35788b moving popdat+precompute into the new unified importer 2020-03-26 10:26:01 -07:00
Dustin Carlino
0133685eb7 converted the import.sh shell script into Rust. towards #27
as part of this, wound up grabbing fresh OSM and GTFS. but actually cant
reimport OSM right now, because fixes reference old OSM IDs and things
break. so sticking with old OSM, but new GTFS. urgh.
2020-03-25 19:54:33 -07:00