b761add3c4
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/8037 GitOrigin-RevId: 4839410ba836d4b69fd1ee30875bca9b46decc66 |
||
---|---|---|
.. | ||
app-produce-feature-matrix | ||
app-render-feature-matrix | ||
ghci | ||
src | ||
src-feature-matrix/Hasura | ||
test-runner | ||
.ghci | ||
api-tests.cabal | ||
docker-compose.yaml | ||
README.md |
Golden API Tests
A set of hspec tests against graphql-engine
that
send a request, and check the actual response against some expected model.
For motivation, rationale, and more, see the test suite
RFC.
Required setup
Most of the required setup concerns (and is documented in the README for) [../test-harness/README.md](the test harness), so please follow that link for more information.
The tests need to know the location of the graphql-engine
executable:
$ export GRAPHQL_ENGINE=$(cabal list-bin exe:graphql-engine)
To be able to run tests against the BigQuery backend, in short, set the following environment variables (assuming you have a BigQuery account set up, see [../test-harness/README.md](the test harness README) for instructions):
$ export HASURA_BIGQUERY_PROJECT_ID=??? # The project ID
$ export HASURA_BIGQUERY_SERVICE_KEY=??? # The service account key
After that, BigQuery will be ready to test.
For everything else, run the following in this directory:
$ docker-compose up
Note to Hasura team: a service account is already setup for internal use, please check the wiki for further details.
Running the test suite
To run all the tests, execute the following command:
$ cabal run api-tests:exe:api-tests
To run only tests whose name contains a certain string, use the -m
or
--match=
flag:
$ cabal run api-tests:exe:api-tests -- -m "SQLServer" # SQLServer tests only
$ cabal run api-tests:exe:api-tests -- --match="Views" # All tests concerning views
The opposite flag is -s
or --skip=
, which will ignore tests containing the
given string:
$ cabal run api-tests:exe:api-tests -- -s "BigQuery" # Skip BigQuery tests
$ cabal run api-tests:exe:api-tests -- --skip="Mutations" # Skip tests around mutations
For additional information, consult the help section:
cabal run api-tests:exe:api-tests -- --help
The local databases persist even after shutting down the containers. If this is undesirable, delete the databases using the following command:
docker compose down --volumes
Enabling logging
See the logging section of [../test-harness/README.md#Logging](the test harness README) for more information.
Test Structure
The feature matrix
defines the shape of this test suite. If we are writing a test for aggregation
queries, that test should live in Test/Queries/AggregationSpec.hs
. If that
module becomes unwieldy, it should live in a module under the
Test/Queries/Aggregation
directory.
Sometimes, tests are backend-specific. Particularly in the case of Postgres,
there are features we support that aren't available on other backends. In other
cases (such as BigQuery's handling of stringified numbers), there are
backend-specific behaviours we wish to verify. In these cases, these tests
should live under backend directories such as Test/Databases/Postgres
or
Test/Databases/BigQuery
. Note that a feature matrix test currently only running on one
backend should still be in the feature matrix structure.
When tests are written to verify that a particular bug has been fixed, these
tests should be placed in the Test/Regression
directory. They should contain
both a descriptive name and the graphql-engine
repo issue number that they address.
Lastly, tests that don't seem to fit under any of the feature matrix, Regression
, or Databases
directories should be organized to mirror the Hasura Docs left-hand navigation.
Adding a new test
Tests are written using hspec
and
hspec-discover
:
- Modules are declared under the
Test
namespace. - Module names must end with
Spec
(e.g.HelloWorldSpec
). - Module names must contain some value
spec :: SpecWith GlobalTestEnvironment
, which serves as the entry point for the module.
See the documentation for hspec
and hspec-discover
, as well as other
modules in the Test
namespace, for more guidance. As well as this, the module
Test.HelloWorldSpec contains a skeleton for writing
new tests.
Test should be written (or reachable from) tests :: SpecWith TestEnvironment
,
or tests :: SpecWith (TestEnvironment, Foo)
for tests that use an additional
local state.
A typical test will look similar to this:
it "Where id=1" \testEnvironment -> do
let actual :: IO Value
actual =
postGraphql
testEnvironment
[graphql|
query {
hasura_author(where: {id: {_eq: 1}}) {
name
id
}
}
|]
expected :: Value
expected =
[yaml|
data:
hasura_author:
- name: Author 1
id: 1
|]
actual `shouldReturnYaml` expected
Note: these quasi-quoter can also perform string interpolation. See the [../test-harness/README.md](the test harness README) for more information.
Debugging
There are times when you would want to debug a test failure by playing around with the Hasura's Graphql engine or by inspecting the database. The default behavior of the test suite is to drop all the data and the tables onces the test suite finishes. To prevent that, you can modify your test module to prevent teardown. Example:
spec :: SpecWith GlobalTestEnvironment
spec =
Fixture.run
[ Fixture.fixture (Fixture.Backend Sqlserver.backendTypeMetadata)
{ Fixture.mkLocalTestEnvironment = Fixture.noLocalTestEnvironment,
setupTeardown = \testEnv ->
[ Fixture.SetupAction
{ Fixture.setupAction = SqlServer.setup schema testEnv,
- Fixture.teardownAction = \_ -> SqlServer.teardown schema testEnv
+ Fixture.teardownAction = \_ -> pure ()
}
]
}]
Now re-run the particular test case again so that the local database is setup. You will still have access to that data once the test suite finishes running. Now based on what you want to, you can either run the Hasura's Graphql engine to debug this further or directly inspect the database using any of its clients.
Logging
By default logs are written to tests-hspec.log
. To view the logs as the tests
run, use
HASURA_TEST_LOGTYPE=stdout
or HASURA_TYPE_LOGTYPE=stderr
.
Using GHCI
Alternatively it is also possible to manually start up the test environment in the GHCI repl.
An example session:
$ cabal repl api-tests
GHCi, version 9.2.4: https://www.haskell.org/ghc/ :? for help
[ 1 of 59] Compiling Harness.Constants ( lib/api-tests/Harness/Constants.hs, interpreted )
...
[59 of 59] Compiling Main ( lib/api-tests/Spec.hs, interpreted )
Ok, 59 modules loaded.
> :module *Main *SpecHook *Test.SomeSpecImDeveloping
> te <- SpecHook.setupTestEnvironment
> te
<TestEnvironment: http://127.0.0.1:35975 >
> -- Setup the instance according to the Fixture
> cleanupPG <- Fixture.fixtureRepl Test.SomeSpecImDeveloping.postgresFixture te
>
> -- run tests or parts of tests manually here
> Test.SomeSpecImDeveloping.someExample te
>
> -- run the test with the hspec runner
> hspec (aroundAllWith (\a () ->a te) Test.SomeSpecImDeveloping>.spec)
Postgres
... [✔]
Citus
... [✔]
>
> -- Or perform other manual inspections, e.g. via the console or ghci.
>
> -- Cleanup before reloading
> cleanupPG
> SpecHook.teardownTestEnvironment te
> -- Reload changes made to the test module or HGE.
> :reload
Points to note:
SpecHook.setupTestEnvironment
starts the HGE server, and its url is revealed byinstance Show TestEnvironment
.SpecHook.teardownTestEnvironment
stops it again.- This is a good idea to do before issuing the
:reload
command, because reloading loses thete
reference but leaves the thread running!
- This is a good idea to do before issuing the
Fixture.fixtureRepl
runs the setup action of a givenFixture
and returns a corresponding teardown action.- After running this you can interact with the HGE console in the same state as when the tests are run.
- Or you can run individual test
Example
s orSpec
s.
- To successfully debug/develop a test in the GHCI repl, the test module
should:
- define its
Fixture
s as toplevel values, - define its
Example
s as toplevel values, - ... such that they can be used directly in the repl.
- define its
Style guide
Stick to Simple Haskell
This test suite should remain accessible to contributors who are new to Haskell and/or the GraphQL engine codebase. Consider the power-to-weight ratio of features, language extensions or abstractions before you introduce them. For example, try to fully leverage Haskell '98 or 2010 features before making use of more advanced ones.
Write small, atomic, autonomous specs
Small: Keep specs short and succinct wherever possible. Consider reorganising modules that grow much longer than ~200-300 lines of code.
For example: The TestGraphQLQueryBasic
pytest
class was ported to the hspec suite
as separate BasicFields
, LimitOffset
, Where
, Ordering
, Directives
and
Views
specs.
Atomic: Each spec should test only one feature against the backends (or contexts) that support it. Each module should contain only the context setup and teardown, and the tests themselves. The database schema, test data, and feature expectations should be easy to reason about without navigating to different module.
For example: BasicFieldsSpec.hs
Autonomous: Each test should run independently of other tests, and not be dependent on the results of a previous test. Shared test state, where unavoidable, should be made explicit.
For example: Remote relationship tests explicitly require shared state.
Use the Harness.*
hierarchy for common functions
Avoid functions or types in tests, other than calls to the Harness.*
API.
Any supporting code should be in the Harness.*
hierarchy and apply broadly to
the test suites overall.
Troubleshooting
Database 'hasura' already exists. Choose a different database name.
or schema "hasura" does not exist
This typically indicates persistent DB state between test runs. Try docker compose down --volumes
to delete the DBs and restart the containers.
General DataConnector
failures
The DataConnector agent might be out of date. If you are getting a lot of test
failures, first try rebuilding the containers with docker compose build
to
make sure you are using the latest version of the agent.
make test-sqlserver
fails with Inconsistent object: mssql connection error
Try updating the mssql-tools
symlink:
brew install microsoft/mssql-release/mssql-tools@18
brew unlink mssql-tools18 && brew link mssql-tools18
Microsoft SQL Server failures on Apple aarch64 chips
This applies to all Apple hardware that uses aarch64 chips, e.g. the MacBook M1 or M2.
We have a few problems with Microsoft SQL Server on Apple aarch64:
-
Microsoft has not yet released SQL Server for aarch64. We need to use Azure SQL Edge instead.
You don't need to do anything if you're using the
make
commands; they will provide the correct image automatically.If you run
docker compose
directly, make sure to set the environment variable yourself:export MSSQL_IMAGE='mcr.microsoft.com/azure-sql-edge'
You can add this to your .envrc.local file if you like.
-
Azure SQL Edge for aarch64 does not ship with the
sqlcmd
utility with which we use to setup the SQL Server schema.If you need it, you can instead use the
mssql-tools
Docker image, for example:docker run --rm -it --platform=linux/amd64 --net=host mcr.microsoft.com/mssql-tools \ /opt/mssql-tools/bin/sqlcmd -S localhost,65003 -U SA -P <password>
To make this easier, you might want to define an alias:
alias sqlcmd='docker run --rm -it --platform=linux/amd64 --net=host mcr.microsoft.com/mssql-tools /opt/mssql-tools/bin/sqlcmd'
You can also install them directly with
brew install microsoft/mssql-release/mssql-tools
.