Right now on errors, only the expected and the actual responses are
shown. The actual response sometimes may not have all the information,
and you may have to look at the logs. In this case, request id would be
of great help to get the extra information from the logs.
These aren't suitable e.g. for running in CI since some take far too
long (and an impossibly long-time when running under criterion's normal
bootstrapping sampling regime.
We might try to improve this ourselves:
https://github.com/bos/criterion/issues/218
An initial summary analysis will be in #3530.
* export metadata without nulls, empty arrays
* property tests for 'ReplaceMetadata' using QuickCheck
-> Derive Arbitrary class for 'ReplaceMetadata' dependant types
* reduce property test cases number to 30
QuickCheck generates the `ReplaceMetadata` value really large
for higher number test cases. Encoded JSON for such values is large and
consumes more memory. Thus, CI is giving up while running property
tests.
* circle-ci: Add property tests as saperate job
* add no command mode to tests
* add yaml.v2 to go mod
* remove indirect comment for yaml.v2 dependency
The connection handler in websocket transport was not using the
'UserAuthentication' interface to resolve user info. Fix resolving
user info in websocket transport to use the common
'UserAuthentication' interface
Instead of
'WITH some_alias (SELECT * from some_func()) SELECT <rows> FROM some_alias'
for SQL function queries, Use
'SELECT <rows> FROM some_func() AS some_alias'
The intention was to make this two cases, using a top-level YAML list.
The result was one test with duplicate keys (effectively only running
the second test). This is an error that's now flagged by newer ruamel.
Both tests needed to be "corrected" to pass and need review.
Tested on python 3.5 and 3.7
We make light use of pyenv to set an appropriate python version if
installed. We could easily install a correct version too if we wanted
but that seemed invasive.
The newer ruamel was an annoying upgrade but also offers some
improvements that exposed some test suite issues (fixed later).
We add a new pytest flag `--accept` that will automatically write back
yaml files with updated responses. This makes it much easier and less
error-prone to update test cases when we expect output to change, or
when authoring new tests.
Second we make sure to test that we actually preserve the order of the
selection set when returning results. This is a "SHOULD" part of the
spec but seems pretty important and something that users will rely on.
To support both of the above we use ruamel.yaml which preserves a
certain amount of formatting and comments (so that --accept can work in
a failry ergonomic way), as well as ordering (so that when we write yaml
the order of keys has meaning that's preserved during parsing).
Use ruamel.yaml everywhere for consistency (since both libraries have
different quirks).
Quirks of ruamel.yaml:
- trailing whitespace in multiline strings in yaml files isn't written
back out as we'd like: https://bitbucket.org/ruamel/yaml/issues/47/multiline-strings-being-changed-if-they
- formatting is only sort of preserved; ruamel e.g. normalizes
indentation. Normally the diff is pretty clean though, and you can
always just check in portions of your test file after --accept
fixup
* save permissions, relationships and collections in catalog with 'is_system_defined'
* Use common stanzas in the .cabal file
* Refactor migration code into lib instead of exe
* Add new server test suite that exercises migrations
* Make graphql-engine clean succeed even if the schema does not exist
This fix is a little ugly, but it’s the only simple solution without a
significant refactoring that restructures the relationship between
GraphQL/Validate and GraphQL/Resolve. The ugliness should go away if we
implement something like #2801.
* Separate DB and metadata migrations
* Refactor Migrate.hs to generate list of migrations at compile-time
* Replace ginger with shakespeare to improve performance
* Improve migration log messages
The changes in 0c74839934 adjusted the
format of error logs slightly to omit fields instead of including them
with null values. However, this was rarely triggered by this test
because it only looks at the first log message, but log messages can
sometimes be written out of order. This makes the test order-agnostic.
