Commit Graph

3 Commits

Author SHA1 Message Date
Dustin Carlino
3de821f1b8 Clear day, cloudy imports. #326
- fix self-destruct command
- ship a GDAL-enabled importer and rebuild everything for Seattle, like
  the normal local process

I'm pretty sure the full process should succeed now. Next step is
figuring out a process for finalizing the changed output files in S3.
2021-05-18 14:07:40 -07:00
Dustin Carlino
a81d33628f Working on the GCP importer workflow... #326
- Amp up number of workers (about 100 cities, so 10/worker now)
- Use an SSD, since especially the setup and upload steps are extremely
  IO bound
- Split the script into pieces that can be easily disabled to iterate
  faster
- Use the bulk API to create instances
- Make the overall start_batch_import.sh a bit quieter
- Make successful VMs self-destruct so it's easier to track which're
  done
- Setup Docker on the VMs, so elevation data works
2021-05-18 12:28:41 -07:00
Dustin Carlino
5fca901e4c Give up on Docker and AWS Batch to bulk import cities. Switch to static
sharding with GCE instead. #326
2021-05-18 09:50:28 -07:00