graphql-engine/server/tests-py/test_tests
Brandon Simmons 91aee7fdeb Test result ordering, add --accept test mode to automatically accept changed test cases
We add a new pytest flag `--accept` that will automatically write back
yaml files with updated responses. This makes it much easier and less
error-prone to update test cases when we expect output to change, or
when authoring new tests.

Second we make sure to test that we actually preserve the order of the
selection set when returning results. This is a "SHOULD" part of the
spec but seems pretty important and something that users will rely on.

To support both of the above we use ruamel.yaml which preserves a
certain amount of formatting and comments (so that --accept can work in
a failry ergonomic way), as well as ordering (so that when we write yaml
the order of keys has meaning that's preserved during parsing).

Use ruamel.yaml everywhere for consistency (since both libraries have
different quirks).

Quirks of ruamel.yaml:
- trailing whitespace in multiline strings in yaml files isn't written
  back out as we'd like: https://bitbucket.org/ruamel/yaml/issues/47/multiline-strings-being-changed-if-they
- formatting is only sort of preserved; ruamel e.g. normalizes
  indentation. Normally the diff is pretty clean though, and you can
  always just check in portions of your test file after --accept

fixup
2019-11-05 15:15:25 -06:00
..
select_query_author_by_pkey_bad_ordering.yaml Test result ordering, add --accept test mode to automatically accept changed test cases 2019-11-05 15:15:25 -06:00
user_can_query_jsonb_values_filter_bad_order.yaml Test result ordering, add --accept test mode to automatically accept changed test cases 2019-11-05 15:15:25 -06:00
user_can_query_jsonb_values_filter_okay_orders.yaml Test result ordering, add --accept test mode to automatically accept changed test cases 2019-11-05 15:15:25 -06:00