hasura/graphql-engine
1
0
Fork 0
mirror of https://github.com/hasura/graphql-engine.git synced 2024-12-21 14:31:55 +03:00
Code Issues Projects Releases Wiki Activity
530e01d458
graphql-engine/server/tests-py/pytest.ini

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

20 lines
1.0 KiB
INI
Raw Normal View History

Tests for server with access control, and some more tests (#710) * 1) Tests for creating permissions 2) Test for constraint_on with GraphQL insert on_conflict * Run tests with access key and webhook * Tests for GraphQL query with quoted columns * Rewrite test-server.sh so that it can be run locally * JWT based tests * Tests with various postgres types * For tests on select queries, run setup only once per class * Tests for v1 count queries * Skip teardown for tests that does not modify data * Workaround for hpc 'parse error when reading .tix file' * Move GeoJson tests to the new structure * Basic tests for v1 queries * Tests for column, table or operator not found error cases on GraphQL queries * Skip test teardown for mutation tests which does not change database state, even when it returns 200.
2018-10-28 21:27:49 +03:00
[pytest]
run default tests in test_server_upgrade (#3718) * run basic tests after upgrade * terminate before specifying file in pytest cmd * Move fixture definitions out of test classes Previously we had abstract classes with the fixtures defined in them. The test classes then inherits these super classes. This is creating inheritence problems, especially when you want to just inherit the tests in class, but not the fixtures. We have now moved all those fixture definitions outside of the class (in conftest.py). These fixtures are now used by the test classes when and where they are required. * Run pytests on server upgrade Server upgrade tests are run by 1) Run pytest with schema/metadata setup but do not do schema/metadata teardown 2) Upgrade the server 3) Run pytest using the above schema and teardown at the end of the tests 4) Cleanup hasura metadata and start again with next set of tests We have added options --skip-schema-setup and --skip-schema-teardown to help running server upgrade tests. While running the tests, we noticed that error codes and messages for some of the tests have changed. So we have added another option to pytest `--avoid-error-message-checks`. If this flag is set, and if comparing expected and response message fails, and if the expected response has an error message, Pytest will throw warnings instead of an error. * Use marks to specify server-upgrade tests Not all tests can be run as serve upgrade tests, particularly those which themselves change the schema. We introduce two pytest markers. Marker allow_server_upgrade_test will add the test into the list of server upgrade tests that can be run. skip_server_upgrade_test removes it from the list. With this we have added tests for queries, mutations, and selected event trigger and remote schema tests to the list of server upgrade tests. * Remove components not needed anymore * Install curl * Fix error in query validation * Fix error in test_v1_queries.py * install procps for server upgrade tests * Use postgres image which has postgis installed * set pager off with psql * quote the bash variable WORKTREE_DIR Co-authored-by: nizar-m <19857260+nizar-m@users.noreply.github.com> Co-authored-by: Vamshi Surabhi <0x777@users.noreply.github.com>
2020-02-13 12:14:02 +03:00
norecursedirs = queries webhook test_upgrade
Test result ordering, add `--accept` test mode to automatically accept changed test cases We add a new pytest flag `--accept` that will automatically write back yaml files with updated responses. This makes it much easier and less error-prone to update test cases when we expect output to change, or when authoring new tests. Second we make sure to test that we actually preserve the order of the selection set when returning results. This is a "SHOULD" part of the spec but seems pretty important and something that users will rely on. To support both of the above we use ruamel.yaml which preserves a certain amount of formatting and comments (so that --accept can work in a failry ergonomic way), as well as ordering (so that when we write yaml the order of keys has meaning that's preserved during parsing). Use ruamel.yaml everywhere for consistency (since both libraries have different quirks). Quirks of ruamel.yaml: - trailing whitespace in multiline strings in yaml files isn't written back out as we'd like: https://bitbucket.org/ruamel/yaml/issues/47/multiline-strings-being-changed-if-they - formatting is only sort of preserved; ruamel e.g. normalizes indentation. Normally the diff is pretty clean though, and you can always just check in portions of your test file after --accept fixup
2019-09-04 18:02:35 +03:00
; Turn any expected failures that pass ("xpassed") into hard failures. This
; lets us use 'xfail' to create test cases that validate other tests, and also
; means we're sure to notice if e.g. a known bug is fixed.
xfail_strict = true
run default tests in test_server_upgrade (#3718) * run basic tests after upgrade * terminate before specifying file in pytest cmd * Move fixture definitions out of test classes Previously we had abstract classes with the fixtures defined in them. The test classes then inherits these super classes. This is creating inheritence problems, especially when you want to just inherit the tests in class, but not the fixtures. We have now moved all those fixture definitions outside of the class (in conftest.py). These fixtures are now used by the test classes when and where they are required. * Run pytests on server upgrade Server upgrade tests are run by 1) Run pytest with schema/metadata setup but do not do schema/metadata teardown 2) Upgrade the server 3) Run pytest using the above schema and teardown at the end of the tests 4) Cleanup hasura metadata and start again with next set of tests We have added options --skip-schema-setup and --skip-schema-teardown to help running server upgrade tests. While running the tests, we noticed that error codes and messages for some of the tests have changed. So we have added another option to pytest `--avoid-error-message-checks`. If this flag is set, and if comparing expected and response message fails, and if the expected response has an error message, Pytest will throw warnings instead of an error. * Use marks to specify server-upgrade tests Not all tests can be run as serve upgrade tests, particularly those which themselves change the schema. We introduce two pytest markers. Marker allow_server_upgrade_test will add the test into the list of server upgrade tests that can be run. skip_server_upgrade_test removes it from the list. With this we have added tests for queries, mutations, and selected event trigger and remote schema tests to the list of server upgrade tests. * Remove components not needed anymore * Install curl * Fix error in query validation * Fix error in test_v1_queries.py * install procps for server upgrade tests * Use postgres image which has postgis installed * set pager off with psql * quote the bash variable WORKTREE_DIR Co-authored-by: nizar-m <19857260+nizar-m@users.noreply.github.com> Co-authored-by: Vamshi Surabhi <0x777@users.noreply.github.com>
2020-02-13 12:14:02 +03:00
markers =
server/tests-py: Use markers for backends instead of parameterized tests. I'm trying to shore up the Python integration tests to make them more reliable. In doing so, I noticed this. --- It feels a lot more sensible as we never run on more than one backend at a time. This also removes the `check_file_exists` parameter from the setup functions; it never worked. It was always set to the result of a comparison between a backend name and a function, which was always `False`. Enabling it breaks things. PR-URL: https://github.com/hasura/graphql-engine-mono/pull/5254 GitOrigin-RevId: 8718ab21527c2ba0a7205d1c01ebaac1a10be844
2022-08-02 22:32:46 +03:00
backend: The backends supported by the test case
server/tests-py: Provide the admin secret to the HGE server. When we run the HGE server inside the test harness, it needs to run with an admin secret for some tests to make sense. This tags each test that requires an admin secret with `pytest.mark.admin_secret`, which then generates a UUID and injects that into both the server and the test case (if required). It also simplifies the way the test harness picks up an existing admin secret, allowing it to use the environment variable instead of requiring it via a parameter. PR-URL: https://github.com/hasura/graphql-engine-mono/pull/6120 GitOrigin-RevId: 55c5b9e8c99bdad9c8304098444ddb9516749a2c
2022-09-29 20:18:49 +03:00
admin_secret: Generate and use an admin secret
server/tests-py: Start webhook.py inside the test harness. We use a helper service to start a webhook-based authentication service for some tests. This moves the initialization of the service out of _test-server.sh_ and into the Python test harness, as a fixture. In order to do this, I had to make a few changes. The main deviation is that we no longer run _all_ tests against an HGE with this authentication service, just a few (those in _test_webhook.py_). Because this reduced coverage, I have added some more tests there, which actually cover some areas not exacerbated elsewhere (mainly trying to use webhook credentials to talk to an admin-only endpoint). The webhook service can run both with and without TLS, and decide whether it's necessary to skip one of these based on the arguments passed and how HGE is started, according to the following logic: * If a TLS CA certificate is passed in, it will run with TLS, otherwise it will skip it. * If HGE was started externally and a TLS certificate is provided, it will skip running without TLS, as it will assume that HGE was configured to talk to a webhook over HTTPS. * Some tests should only be run with TLS; this is marked with a `tls_webhook_server` marker. * Some tests should only be run _without_ TLS; this is marked with a `no_tls_webhook_server` marker. The actual parameterization of the webhook service configuration is done through test subclasses, because normal pytest parameterization doesn't work with the `hge_fixture_env` hack that we use. Because `hge_fixture_env` is not a sanctioned way of conveying data between fixtures (and, unfortunately, there isn't a sanctioned way of doing this when the fixtures in question may not know about each other directly), parameterizing the `webhook_server` fixture doesn't actually parameterize `hge_server` properly. Subclassing forces this to work correctly. The certificate generation is moved to a Python fixture, so that we don't have to revoke the CA certificate for _test_webhook_insecure.py_; we can just generate a bogus certificate instead. The CA certificate is still generated in the _test-server.sh_ script, as it needs to be installed into the OS certificate store. Interestingly, the CA certificate installation wasn't actually working, because the certificates were written to the wrong location. This didn't cause any failures, as we weren't actually testing this behavior. This is now fixed with the other changes. PR-URL: https://github.com/hasura/graphql-engine-mono/pull/6363 GitOrigin-RevId: 0f277d374daa64f657257ed2a4c2057c74b911db
2022-10-20 21:58:36 +03:00
no_admin_secret: Skip if an admin secret is provided (legacy)
server/tests-py: Parallelize JWT tests. This rewrites the JWT tests to generate and specify the secrets per test class, and to provide the server configuration to the HGE fixture. It covers the tests in: - *test_jwt.py* - *test_jwt_claims_map.py* - *test_config_api.py* - *test_graphql_queries.py* (just a couple here) This does reduce the number of code paths exercised with JWT, as we were previously running *all* tests with JWT tokens. However, this seems excessive; we don't need to tread every code path, just enough to ensure we handle the tokens appropriately. I believe that the test coverage in *test_jwt.py* does this well enough (though I'd prefer if we moved the coverage lower down in the stack as unit tests). These tests were configured in multiple different ways by *test-server.sh*; this configuration is now moved to test subclasses within the various files. This results in a bit of duplication. Unfortunately, the tests would ideally use parameterization rather than subclassing, but that doesn't work because of `hge_fixture_env`, which creates a "soft" dependency between the environment variables and `hge_server`. Parameterizing the former *should* force the latter to be recreated for each new set of environment variables, but `hge_server` isn't actually aware there's a dependency. It currently looks like this adds lines of code; we'll more than make up for it when we delete the relevant lines from *test-server.sh*. I am not doing that here because I plan on deleting the whole file in a subsequent changeset. [NDAT-538]: https://hasurahq.atlassian.net/browse/NDAT-538?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ PR-URL: https://github.com/hasura/graphql-engine-mono/pull/8803 GitOrigin-RevId: f7f2caa62de0b0a45e42964b69a8ae73d1575fe8
2023-04-19 13:28:49 +03:00
requires_an_admin_secret: Skip if no admin secret is provided
server/tests-py: Declaratively state the HGE environment variables. This has two purposes: * When running the Python integration tests against a running HGE instance, with `--hge-url`, it will check the environment variables available and actively skip the test if they aren't set. This replaces the previous ad-hoc skip behavior. * More interestingly, when running against a binary with `--hge-bin`, the environment variables are passed through, which means different tests can run with different environment variables. On top of this, the various services we use for testing now also provide their own environment variables, rather than expecting a test script to do it. In order to make this work, I also had to invert the dependency between various services and `hge_ctx`. I extracted a `pg_version` fixture to provide the PostgreSQL version, and now pass the `hge_url` and `hge_key` explicitly to `ActionsWebhookServer`. PR-URL: https://github.com/hasura/graphql-engine-mono/pull/6028 GitOrigin-RevId: 16d866741dba5887da1adf4e1ade8182ccc9d344
2022-09-28 12:19:47 +03:00
hge_env: Pass additional environment variables to the GraphQL Engine
server/tests-py: Start helper services in the test harness. This makes it possible for the test harness to start the test JWK server and the test remote schema server. In order to do this, we still generate the TLS certificates in the test script (because we need to install the generated CA certificate in the OS certificate store), and then pass the certificate and key paths into the test runner. Because we are still using _test-server.sh_ for now, we don't use the JWK server fixture in that case, as HGE needs the JWK server to be up and running when it starts. Instead, we keep running it outside (for now). This is also the case for the GraphQL server fixture when we are running the server upgrade/downgrade tests. I have also refactored _graphql_server.py_ so there isn't a global `HGE_URLS` value, but instead the value is passed through. PR-URL: https://github.com/hasura/graphql-engine-mono/pull/6303 GitOrigin-RevId: 06f05ff674372dc5d632e55d68e661f5c7a17c10
2022-10-13 18:43:59 +03:00
jwk_path: When running a JWK server, the URL path that HGE should use
server/tests-py: Parallelize JWT tests. This rewrites the JWT tests to generate and specify the secrets per test class, and to provide the server configuration to the HGE fixture. It covers the tests in: - *test_jwt.py* - *test_jwt_claims_map.py* - *test_config_api.py* - *test_graphql_queries.py* (just a couple here) This does reduce the number of code paths exercised with JWT, as we were previously running *all* tests with JWT tokens. However, this seems excessive; we don't need to tread every code path, just enough to ensure we handle the tokens appropriately. I believe that the test coverage in *test_jwt.py* does this well enough (though I'd prefer if we moved the coverage lower down in the stack as unit tests). These tests were configured in multiple different ways by *test-server.sh*; this configuration is now moved to test subclasses within the various files. This results in a bit of duplication. Unfortunately, the tests would ideally use parameterization rather than subclassing, but that doesn't work because of `hge_fixture_env`, which creates a "soft" dependency between the environment variables and `hge_server`. Parameterizing the former *should* force the latter to be recreated for each new set of environment variables, but `hge_server` isn't actually aware there's a dependency. It currently looks like this adds lines of code; we'll more than make up for it when we delete the relevant lines from *test-server.sh*. I am not doing that here because I plan on deleting the whole file in a subsequent changeset. [NDAT-538]: https://hasurahq.atlassian.net/browse/NDAT-538?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ PR-URL: https://github.com/hasura/graphql-engine-mono/pull/8803 GitOrigin-RevId: f7f2caa62de0b0a45e42964b69a8ae73d1575fe8
2023-04-19 13:28:49 +03:00
jwt: JWT configuration for the test
server/tests-py: Start webhook.py inside the test harness. We use a helper service to start a webhook-based authentication service for some tests. This moves the initialization of the service out of _test-server.sh_ and into the Python test harness, as a fixture. In order to do this, I had to make a few changes. The main deviation is that we no longer run _all_ tests against an HGE with this authentication service, just a few (those in _test_webhook.py_). Because this reduced coverage, I have added some more tests there, which actually cover some areas not exacerbated elsewhere (mainly trying to use webhook credentials to talk to an admin-only endpoint). The webhook service can run both with and without TLS, and decide whether it's necessary to skip one of these based on the arguments passed and how HGE is started, according to the following logic: * If a TLS CA certificate is passed in, it will run with TLS, otherwise it will skip it. * If HGE was started externally and a TLS certificate is provided, it will skip running without TLS, as it will assume that HGE was configured to talk to a webhook over HTTPS. * Some tests should only be run with TLS; this is marked with a `tls_webhook_server` marker. * Some tests should only be run _without_ TLS; this is marked with a `no_tls_webhook_server` marker. The actual parameterization of the webhook service configuration is done through test subclasses, because normal pytest parameterization doesn't work with the `hge_fixture_env` hack that we use. Because `hge_fixture_env` is not a sanctioned way of conveying data between fixtures (and, unfortunately, there isn't a sanctioned way of doing this when the fixtures in question may not know about each other directly), parameterizing the `webhook_server` fixture doesn't actually parameterize `hge_server` properly. Subclassing forces this to work correctly. The certificate generation is moved to a Python fixture, so that we don't have to revoke the CA certificate for _test_webhook_insecure.py_; we can just generate a bogus certificate instead. The CA certificate is still generated in the _test-server.sh_ script, as it needs to be installed into the OS certificate store. Interestingly, the CA certificate installation wasn't actually working, because the certificates were written to the wrong location. This didn't cause any failures, as we weren't actually testing this behavior. This is now fixed with the other changes. PR-URL: https://github.com/hasura/graphql-engine-mono/pull/6363 GitOrigin-RevId: 0f277d374daa64f657257ed2a4c2057c74b911db
2022-10-20 21:58:36 +03:00
tls_webhook_server: Only run the webhook server with TLS enabled
no_tls_webhook_server: Only run the webhook server with TLS disabled
tls_insecure_certificate: Create an insecure (self-signed) certificate for the webhook server
server/tests-py: Parallelize test_logging.py. This adds the ability to capture logs to the HGE fixture, and uses this in test_logging.py to analyze the logs, instead of relying on a shell script redirecting the logs to a file. We then inject the logs into the tests and parse the JSON. Because we're no longer reading a file, we need to do this in a separate thread, as we'll block on reading rather than the stream ending. (Once HGE stops, the stream will be closed.) Some of the tests require a JWK server, so this has been extracted from test_jwk.py. [NDAT-540]: https://hasurahq.atlassian.net/browse/NDAT-540?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ PR-URL: https://github.com/hasura/graphql-engine-mono/pull/8772 GitOrigin-RevId: 9413e714f1c42b8a0991d0d30c4358209fd30c0c
2023-04-17 16:07:30 +03:00
capture_hge_logs: Enable to capture HGE logs for later inspection in a test
server/tests-py: Start webhook.py inside the test harness. We use a helper service to start a webhook-based authentication service for some tests. This moves the initialization of the service out of _test-server.sh_ and into the Python test harness, as a fixture. In order to do this, I had to make a few changes. The main deviation is that we no longer run _all_ tests against an HGE with this authentication service, just a few (those in _test_webhook.py_). Because this reduced coverage, I have added some more tests there, which actually cover some areas not exacerbated elsewhere (mainly trying to use webhook credentials to talk to an admin-only endpoint). The webhook service can run both with and without TLS, and decide whether it's necessary to skip one of these based on the arguments passed and how HGE is started, according to the following logic: * If a TLS CA certificate is passed in, it will run with TLS, otherwise it will skip it. * If HGE was started externally and a TLS certificate is provided, it will skip running without TLS, as it will assume that HGE was configured to talk to a webhook over HTTPS. * Some tests should only be run with TLS; this is marked with a `tls_webhook_server` marker. * Some tests should only be run _without_ TLS; this is marked with a `no_tls_webhook_server` marker. The actual parameterization of the webhook service configuration is done through test subclasses, because normal pytest parameterization doesn't work with the `hge_fixture_env` hack that we use. Because `hge_fixture_env` is not a sanctioned way of conveying data between fixtures (and, unfortunately, there isn't a sanctioned way of doing this when the fixtures in question may not know about each other directly), parameterizing the `webhook_server` fixture doesn't actually parameterize `hge_server` properly. Subclassing forces this to work correctly. The certificate generation is moved to a Python fixture, so that we don't have to revoke the CA certificate for _test_webhook_insecure.py_; we can just generate a bogus certificate instead. The CA certificate is still generated in the _test-server.sh_ script, as it needs to be installed into the OS certificate store. Interestingly, the CA certificate installation wasn't actually working, because the certificates were written to the wrong location. This didn't cause any failures, as we weren't actually testing this behavior. This is now fixed with the other changes. PR-URL: https://github.com/hasura/graphql-engine-mono/pull/6363 GitOrigin-RevId: 0f277d374daa64f657257ed2a4c2057c74b911db
2022-10-20 21:58:36 +03:00
value: A value used by self-tests
Reference in New Issue Copy Permalink