abstreet/docs/design/testing.md

Testing-related design notes

Simulation unit tests

To encourage testing, it should be easy to: - describe a setup - assert what the outcome should be - sometimes just that no runtime invariants are broken - pop up a UI to interactively step through the test

Some first tests to write: = car starting with no path on road with no parking spots, ensure they wind up parking at the first spot on some side street
- car stops for departing car (winds up following it) - departing car waits for other car (winds up following it) - a line of cars moving through a stop sign looks jittery right now. correct or not? - following distances for cars of different lengths

Unclear how to nicely let the test inspect stuff every tick.

Rejected ideas:

• make every pub(crate) so a unit test can reach into state anywhere. It ruins viz for testing.
• JSONify stuff and look at that. too slow, and walking the JSON structure is annoying and not type-safe.
• make one-off accessors for interesting stuff. pollutes code and is tedious.

The idea that's sticking:

• every tick, accumulate a list of events that occurred. publish these from various places.
  • most of the events are state transitions -- car leaves lane, intersection accepts ticket, car parks, bus departs
  • beyond unit testing, this will be useful for building up a compressed schedule for the time traveler
  • and already am kind of using this pattern to communicate between sim managers, spawners, etc
  • will help compute trip statistics later
  • would also be nice to log some of these

Watch tests easily

• need to organize savestate captures

  • dedicated place: data/savestates/MAP/scenario/time
    • plumb map name, scenario name
    • should be able to just point to one of these saves, not refer to the map or RNG seed again
  • also kinda needed for time traveling later

• when a problem happens, we want to back up a little bit

  • probably just need automatic occasional savestating, and to print a nice command to rerun from it

Diffing for A/B tests

Basic problem: how do we show map edits/diffs? - could be useful for debugging as new data sources come in - and is vital part of the game - UI - highlight edited things - hold a button to show the original versions of things in a transparentish overlay

How to show diffs for agents?

Test setup

Tried having some helper methods, but they're not aging well as the new trip leg stuff continues. Removing for now, will re-evaluate later.

Synthetic editor is awesome. How could it help test setup?

• could specify things like parking spots directly
  • but that may be lots of UI work
  • and then a synthetic map is only useful for one test, maybe
• could just label things in the synthetic editor
  • and easily look em up in the test setup code

Test runner

• want to tag tests and run groups of them easily
  • have to use prefixes to test names for now
• want nicer runner output
  • capture output for failed tests in files
  • print summary of each test at the end
  • for sim tests, print a command to run it in the UI

Let's experiment and make a new binary crate for all tests!