mirror of
https://github.com/a-b-street/abstreet.git
synced 2024-12-29 17:34:58 +03:00
81 lines
3.1 KiB
Markdown
81 lines
3.1 KiB
Markdown
# Testing-related design notes
|
|
|
|
## Simulation unit tests
|
|
|
|
To encourage testing, it should be easy to:
|
|
- describe a setup
|
|
- assert what the outcome should be
|
|
- sometimes just that no runtime invariants are broken
|
|
- pop up a UI to interactively step through the test
|
|
|
|
Some first tests to write:
|
|
= car starting with no path on road with no parking spots, ensure they wind up parking at the first spot on some
|
|
side street
|
|
- car stops for departing car (winds up following it)
|
|
- departing car waits for other car (winds up following it)
|
|
- a line of cars moving through a stop sign looks jittery right now. correct or not?
|
|
- following distances for cars of different lengths
|
|
|
|
Unclear how to nicely let the test inspect stuff every tick.
|
|
|
|
Rejected ideas:
|
|
- make every pub(crate) so a unit test can reach into state anywhere. It ruins viz for testing.
|
|
- JSONify stuff and look at that. too slow, and walking the JSON structure is annoying and not type-safe.
|
|
- make one-off accessors for interesting stuff. pollutes code and is tedious.
|
|
|
|
The idea that's sticking:
|
|
- every tick, accumulate a list of events that occurred. publish these from various places.
|
|
- most of the events are state transitions -- car leaves lane, intersection accepts ticket, car parks, bus departs
|
|
- beyond unit testing, this will be useful for building up a compressed schedule for the time traveler
|
|
- and already am kind of using this pattern to communicate between sim managers, spawners, etc
|
|
- will help compute trip statistics later
|
|
- would also be nice to log some of these
|
|
|
|
## Watch tests easily
|
|
|
|
- need to organize savestate captures
|
|
- dedicated place: data/savestates/MAP/scenario/time
|
|
- plumb map name, scenario name
|
|
- should be able to just point to one of these saves, not refer to the map or RNG seed again
|
|
- also kinda needed for time traveling later
|
|
|
|
- when a problem happens, we want to back up a little bit
|
|
- probably just need automatic occasional savestating, and to print a nice command to rerun from it
|
|
|
|
## Diffing for A/B tests
|
|
|
|
Basic problem: how do we show map edits/diffs?
|
|
- could be useful for debugging as new data sources come in
|
|
- and is vital part of the game
|
|
- UI
|
|
- highlight edited things
|
|
- hold a button to show the original versions of things in a transparentish overlay
|
|
|
|
How to show diffs for agents?
|
|
|
|
## Test setup
|
|
|
|
Tried having some helper methods, but they're not aging well as the new trip
|
|
leg stuff continues. Removing for now, will re-evaluate later.
|
|
|
|
Synthetic editor is awesome. How could it help test setup?
|
|
|
|
- could specify things like parking spots directly
|
|
- but that may be lots of UI work
|
|
- and then a synthetic map is only useful for one test, maybe
|
|
- could just label things in the synthetic editor
|
|
- and easily look em up in the test setup code
|
|
|
|
## Test runner
|
|
|
|
- want to tag tests and run groups of them easily
|
|
- have to use prefixes to test names for now
|
|
- want nicer runner output
|
|
- capture output for failed tests in files
|
|
- print summary of each test at the end
|
|
- for sim tests, print a command to run it in the UI
|
|
|
|
Let's experiment and make a new binary crate for all tests!
|
|
|
|
- it'd be neat to later integrate with travis.
|