graphql-engine/server/tests-hspec

tests-hspec

Graphql-engine integration tests written in Haskell using the hspec testing framework.

For motivation, rationale, and more, see the test suite rfc.

Table of Contents

• tests-hspec
  • Required setup for BigQuery tests
  • Running the test suite
  • Enabling logging
  • Test suite structure
    • Harness
    • Test
  • Adding a new test
    • Specifying contexts
      • Make local testEnvironment action
      • Setup action
      • Teardown action
    • Writing tests
  • Debugging
    • Using GHCI
  • Style guide
    • Stick to Simple Haskell
    • Write small, atomic, autonomous specs
    • Use the Harness.* hierarchy for common functions
  • Troubleshooting
    • Database 'hasura' already exists. Choose a different database name.
    • General DataConnector failures
    • SQLServer failures on Apple M1 chips

Required setup for BigQuery tests

Running integration tests against a BigQuery data source is a more involved due to the necessary service account requirements:

HASURA_BIGQUERY_PROJECT_ID=# the project ID of the service account
HASURA_BIGQUERY_SERVICE_KEY=# the service account key
# optional variable used to verify the account setup in step 4 below
HASURA_BIGQUERY_SERVICE_ACCOUNT_EMAIL=# eg. "<<SERVICE_ACCOUNT_NAME>>@<<PROJECT_NAME>>.iam.gserviceaccount.com"

Before running the test suite:

1. Ensure you have access to a Google Cloud Console service account. Store the project ID and account email in HASURA_BIGQUERY_PROJECT_ID variable.

2. Create and download a new service account key. Store the contents of file in a HASURA_BIGQUERY_SERVICE_KEY variable.

   export HASURA_BIGQUERY_SERVICE_KEY=$(cat /path/to/service/account)
   

3. Login and activate the service account, if it is not already activated.

4. Verify the service account is accessible via the BigQuery API, by running the following command:

   ./scripts/verify-bigquery-creds.sh
   

   If the query succeeds, the service account is setup correctly to run tests against BigQuery locally.

5. If necessary, create a dataset called "hasura" in the BigQuery workspace.

6. Finally, run the BigQuery tests once the HASURA_BIGQUERY_SERVICE_KEY and HASURA_BIGQUERY_PROJECT_ID environment variables set. For example:

   cabal run tests-hspec -- -m "BigQuery"
   

Note to Hasura team: a service account is already setup for internal use, please check the wiki for further details.

Running the test suite

1. To run the Haskell integration test suite, we'll first need to start the backends:

   docker compose up
   

   This will start up Postgres, SQL Server, Citus, MariaDB and the Hasura Data Connectors' reference agent.

   Note

   : on ARM64 architecture we'll need additional steps in order to test mssql properly. See SQLServer failures on Apple M1 chips for more details.

2. Once the containers are up, you can run the test suite via

   cabal run tests-hspec
   

   You can also further refine which tests to run using the -m flag:

   cabal run tests-hspec -- -m "SQLServer"
   

   For additional information, consult the help section:

   cabal run tests-hspec -- --help
   

3. The local databases persist even after shutting down the containers. If this is undesirable, delete the databases using the following command:

   docker compose down --volumes
   

Enabling logging

In order to enable logging, you have to manually edit the engineLogLevel term in Harness/Constants.hs.

This pairs well with running a single test via the -m flag (see the section above).

Test suite structure

Harness

Modules under the Harness/ namespace provide the infrastructure and supporting code for writing and running the tests. It includes quasiquoters, interacting with backends, interfacing with HTTP, constants, and so on.

Supporting code should be added under the Harness.* namespace instead of added ad-hoc in test specs, to improve readability and reuse.

Test

Modules under the Test/ namespace define integration test specifications for various features and backends.

The feature matrix defines the shape of this test suite. If we are writing a test for aggregation queries, that test should live in Test/Queries/AggregationSpec.hs. If that module becomes unwieldy, it should live in a module under the Test/Queries/Aggregation directory.

Sometimes, tests are backend-specific. Particularly in the case of Postgres, there are features we support that aren't available on other backends. In other cases (such as BigQuery's handling of stringified numbers), there are backend-specific behaviours we wish to verify. In these cases, these tests should live under backend directories such as Test/Postgres or Test/BigQuery. Note that a feature matrix test currently only running on one backend should still be in the feature matrix structure.

When tests are written to verify that a particular bug has been fixed, these tests should be placed in the Test/Regression directory. They should contain both a descriptive name and the issue number that they address.

Adding a new test

Tests are written using hspec and hspec-discover:

• Modules are declared under the Test namespace.
• Module names must end with Spec (e.g. HelloWorldSpec).
• Module names must contain some value spec :: SpecWith TestEnvironment, which serves as the entry point for the module.

See the documentation for hspec and hspec-discover, as well as other modules in the Test namespace, for more guidance. As well as this, the module Test.HelloWorldSpec contains a skeleton for writing new tests.

Specifying fixtures

We often want to run the same tests several times with slightly different configuration. Most commonly, we want to assert that a given behaviour works consistently across different backends.

Harness.Test.Fixture defines two functions for running test trees in terms of a list of Fixture as.

Each Fixture a requires:

• a unique name, of type FixtureName
• a mkLocalTestEnvironment action, of type TestEnvironment -> IO a
• a setupTeardown action, of type (TestEnvironment, a) -> [SetupAction]
• an customOptions parameter, which will be threaded through the tests themselves to modify behavior for a particular Fixture

Of these two functions, whether one wishes to use Harness.Test.Fixture.run or Harness.Test.Fixture.runWithLocalTestEnvironment will depend on if their test can be written in terms of information provided by the global TestEnvironment type or if it depends on some additional "local" state.

More often than not, test authors should use Harness.Test.Fixture.run, which is written in terms of Fixture (). This uses () for the local test which does not carry any "useful" state information, and is therefore omitted from the body of the tests themselves.

In the rare cases where some local state is necessary, test authors should use Harness.Test.Fixture.runWithLocalTestEnvironment. This function takes a type parameter for its local testEnvironment, which will be provided to the body of tests themselves.

Make local testEnvironment action

This refers to the function mkLocalTestEnvironment defined for Fixture:

mkLocalTestEnvironment :: TestEnvironment -> IO a

Its return value, IO a, matches the a of Fixture a: it is the additional local state that is required throughout the tests, in addition to the global TestEnvironment. Some tests, such as tests which check remote relationships, need to keep some state which is local to the context, but most tests do not need additional state, and define mkLocalTestEnvironment to be Harness.Test.Fixture.noLocalTestEnvironment.

Setup / teardown actions

To ensure things are cleaned up properly in the event of errors in setting up and tearing down tests, test setup is defined in terms of SetupActions.

These look like this:

data SetupAction = forall a.
  SetupAction
    { setupAction :: IO a,
      teardownAction :: Maybe a -> IO ()
    }

A SetupAction encodes how to setup and tear down a single piece of test system state.

The value produced by a setupAction is to be input into the corresponding teardownAction, if the setupAction completed without throwing an exception.

Therefore one SetupAction could create the DB tables, and the matching teardown removes them. Pairing setup / teardown in this way makes it easier to remove everything in the right order.

Setup

The setup actions are responsible for creating the environment for the test. They need to:

1. Clear and reconfigure the metadata
2. Setup tables and insert values
3. Track tables, add relationships, permissions

etc.

These actions can be created by running POST requests against graphql-engine using Harness.GraphqlEngine.post_, or by running SQL requests against the backend using Backend.<backend>.run_.

Teardown

These actions are responsible for freeing acquired resources, and reverting all local modifications: dropping newly created tables, deleting custom functions, removing the changes made to the metadata, and so on.

These actions can be created by running POST requests against graphql-engine using Harness.GraphqlEngine.post_, or by running SQL requests against the backend using Backend.<backend>.run_.

Writing tests

Test should be written (or reachable from) tests :: SpecWith TestEnvironment, or tests :: SpecWith (TestEnvironment, Foo) for tests that use an additional local state.

A typical test will look similar to this:

  it "Where id=1" \testEnvironment ->
    shouldReturnYaml
      ( GraphqlEngine.postGraphql
          testEnvironment
          [graphql|
            query {
              hasura_author(where: {id: {_eq: 1}}) {
                name
                id
              }
            }
          |]
      )
      [yaml|
        data:
          hasura_author:
          - name: Author 1
            id: 1
      |]

• it specifies the name of the test
• shouldReturnYaml creates an Expectation which does the following:
  • Runs a POST request against graphql-engine which can be specified using the graphql quasi-quoter.
  • Compares the response to an expected result which can be specified using the yaml quasi-quoter.

Note: these quasi-quoter can also perform string interpolation. See the relevant modules under the Harness.Quoter namespace.

Debugging

There are times when you would want to debug a test failure by playing around with the Hasura's Graphql engine or by inspecting the database. The default behavior of the test suite is to drop all the data and the tables onces the test suite finishes. To prevent that, you can modify your test module to prevent teardown. Example:

spec :: SpecWith TestEnvironment
spec =
  Fixture.run
    [ Fixture.fixture (Fixture.Backend Fixture.SQLServer)
        { Fixture.mkLocalTestEnvironment = Fixture.noLocalTestEnvironment,
          setupTeardown = \testEnv ->
            [ Fixture.SetupAction
               { Fixture.setupAction = SqlServer.setup schema testEnv,
-                Fixture.teardownAction = \_ -> SqlServer.teardown schema testEnv
+                Fixture.teardownAction = \_ -> pure ()
               }
            ]
        }]

Now re-run the particular test case again so that the local database is setup. You will still have access to that data once the test suite finishes running. Now based on what you want to, you can either run the Hasura's Graphql engine to debug this further or directly inspect the database using any of it's clients.

Using GHCI

Alternatively it is also possible to manually start up the test environment in the GHCI repl.

An example session:

$ cabal repl graphql-engine:tests-hspec
GHCi, version 8.10.7: https://www.haskell.org/ghc/  :? for help
[ 1 of 59] Compiling Harness.Constants ( tests-hspec/Harness/Constants.hs, interpreted )
...
[59 of 59] Compiling Main             ( tests-hspec/Spec.hs, interpreted )
Ok, 59 modules loaded.
*Main> :module *Main *SpecHook *Test.SomeSpecImDeveloping
*Main *SpecHook *Test.SomeSpecImDeveloping> te <- SpecHook.setupTestEnvironment
*Main *SpecHook *Test.SomeSpecImDeveloping> te
<TestEnvironment: http://127.0.0.1:35975 >
*Main *SpecHook *Test.SomeSpecImDeveloping> -- Setup the instance according to the Fixture
*Main *SpecHook *Test.SomeSpecImDeveloping> cleanupPG <- Fixture.fixtureRepl Test.SomeSpecImDeveloping.postgresFixture te
*Main *SpecHook *Test.SomeSpecImDeveloping>
*Main *SpecHook *Test.SomeSpecImDeveloping> -- run tests or parts of tests manually here
*Main *SpecHook *Test.SomeSpecImDeveloping> Test.SomeSpecImDeveloping.someExample te
*Main *SpecHook *Test.SomeSpecImDeveloping>
*Main *SpecHook *Test.SomeSpecImDeveloping> -- run the test with the hspec runner
*Main *SpecHook *Test.SomeSpecImDeveloping> hspec (aroundAllWith (\a () ->a te) Test.SomeSpecImDeveloping>.spec)
Postgres
  ... [✔]
Citus
  ... [✔]
*Main *SpecHook *Test.SomeSpecImDeveloping>
*Main *SpecHook *Test.SomeSpecImDeveloping> -- Or perform other manual inspections, e.g. via the console or ghci.
*Main *SpecHook *Test.SomeSpecImDeveloping>
*Main *SpecHook *Test.SomeSpecImDeveloping> -- Cleanup before reloading
*Main *SpecHook *Test.SomeSpecImDeveloping> cleanupPG
*Main *SpecHook *Test.SomeSpecImDeveloping> SpecHook.teardownTestEnvironment te

*Main *SpecHook *Test.SomeSpecImDeveloping> -- Reload changes made to the test module or HGE.
*Main *SpecHook *Test.SomeSpecImDeveloping> :reload

Points to note:

• SpecHook.setupTestEnvironment starts the HGE server, and its url is revealed by instance Show TestEnvironment.
• SpecHook.teardownTestEnvironment stops it again.
  • This is a good idea to do before issuing the :reload command, because reloading loses the te reference but leaves the thread running!
• Fixture.fixtureRepl runs the setup action of a given Fixture and returns a corresponding teardown action.
  • After running this you can interact with the HGE console in the same state as when the tests are run.
  • Or you can run individual test Examples or Specs.
• To successfully debug/develop a test in the GHCI repl, the test module should:
  • define its Fixtures as toplevel values,
  • define its Examples as toplevel values,
  • ... such that they can be used directly in the repl.

Style guide

Stick to Simple Haskell

This test suite should remain accessible to contributors who are new to Haskell and/or the GraphQL engine codebase. Consider the power-to-weight ratio of features, language extensions or abstractions before you introduce them. For example, try to fully leverage Haskell '98 or 2010 features before making use of more advanced ones.

Write small, atomic, autonomous specs

Small: Keep specs short and succinct wherever possible. Consider reorganising modules that grow much longer than ~200-300 lines of code.

For example: The TestGraphQLQueryBasic pytest class was ported to the hspec suite as separate BasicFields, LimitOffset, Where, Ordering, Directivesand Views specs.

Atomic: Each spec should test only one feature against the backends (or contexts) that support it. Each module should contain only the context setup and teardown, and the tests themselves. The database schema, test data, and feature expectations should be easy to reason about without navigating to different module.

For example: BasicFieldsSpec.hs

Autonomous: Each test should run independently of other tests, and not be dependent on the results of a previous test. Shared test state, where unavoidable, should be made explicit.

For example: Remote relationship tests explicitly require shared state.

Use the Harness.* hierarchy for common functions

Avoid functions or types in tests, other than calls to the Harness.* API.

Any supporting code should be in the Harness.* hierarchy and apply broadly to the test suites overall.

Troubleshooting

Database 'hasura' already exists. Choose a different database name. or schema "hasura" does not exist

This typically indicates persistent DB state between test runs. Try docker compose down --volumes to delete the DBs and restart the containers.

General DataConnector failures

The DataConnector agent might be out of date. If you are getting a lot of test failures, first try rebuilding the containers with docker compose build to make sure you are using the latest version of the agent.

Microsoft SQL Server failures on Apple aarch64 chips

This applies to all Apple hardware that uses aarch64 chips, e.g. the MacBook M1 or M2.

We have a few problems with Microsoft SQL Server on Apple aarch64:

1. Due to compiler bugs in GHC 8.10.7, we need to use patched Haskell ODBC bindings as a workaround for aarch64 systems.

   Add the following to cabal.project.local:

   source-repository-package
     type: git
     location: https://github.com/soupi/odbc.git
     tag: a6acf6b4eca31022babbf8045f31a0f7f26c5923
   

2. Microsoft has not yet released SQL Server for aarch64. We need to use Azure SQL Edge instead.

   You don't need to do anything if you're using the make commands; they will provide the correct image automatically.

   If you run docker compose directly, make sure to set the environment variable yourself:

   export MSSQL_IMAGE='mcr.microsoft.com/azure-sql-edge'
   

   You can add this to your .envrc.local file if you like.

3. Azure SQL Edge for aarch64 does not ship with the sqlcmd utility with which we use to setup the SQL Server schema.

   If you need it, you can instead use the mssql-tools Docker image, for example:

   docker run --rm -it --platform=linux/amd64 --net=host mcr.microsoft.com/mssql-tools \
     /opt/mssql-tools/bin/sqlcmd -S localhost,65003 -U SA -P <password>
   

   To make this easier, you might want to define an alias:

   alias sqlcmd='docker run --rm -it --platform=linux/amd64 --net=host mcr.microsoft.com/mssql-tools /opt/mssql-tools/bin/sqlcmd'
   

   You can also install them directly with brew install microsoft/mssql-release/mssql-tools.