I'm trying to shore up the Python integration tests to make them more reliable. In doing so, I noticed this.
---
Rather than hard-coding hostnames and ports, we can (and already do) inject these into the HGE process using environment variables.
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/5255
GitOrigin-RevId: 6bb593999ece42cedef6619f31f9d9b2e39f30ef
Docker Compose is now a plugin for Docker, bundled by default in Docker Desktop and many Linux distribution packages. The standalone `docker-compose` binary has been deprecated since Docker Compose v2.
Using the new version directly allows us to write development scripts that do not require `docker-compose` to be installed.
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/5185
GitOrigin-RevId: c8542b8b2405d1aa32288991688c6fde4af96383
We currently use `hpack` to generate the Cabal files from _package.yaml_
files for the two small libraries in _server/lib_. While this is more
convenient, we also check the Cabal files into the repository to avoid
needing an extra step upon pulling changes.
In order to ensure that the Cabal files do not get out of sync with the
hpack files, this introduces a few improvements:
1. Makefile targets to automatically generate the Cabal files without
needing to know the correct incantation. These targets are a
dependency of all build targets, so you can simply run
`make build-all` and it will work.
2. An extra comment at the top of all generated Cabal files that
explains how to regenerate it.
3. A `lint-hpack` Makefile target that verifies that the Cabal files
are up-to-date.
4. A CI job that runs `make lint-hpack`, to stop inconsistencies
getting merged into trunk.
Most of these changes are ported from #4794.
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/5217
GitOrigin-RevId: d3dfbe19ec00528368d357b6d0215a7ba4062f68
* Fix `make start-sqlserver` so it can find `sqlcmd`.
* Start the DC reference agent when testing all backends.
* Start backends in parallel so it doesn't take so long to spin them up. This is achieved by splitting all `start-*` tasks into `spawn-*` and `wait-for-*`.
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/5176
GitOrigin-RevId: ea483a755daa3fa6a8701f9d3dd0f393bc56510f
There's no reason to pin it to v3.7. This wasn't causing the Docker
Compose issues; it's a dependency version issue.
We actually don't need to install `docker-compose` at all. It's now a
Docker plugin (and is bundled in Docker Desktop) since Docker Compose
v2. You can run it with `docker compose`.
This also fixes a bug in _dev.sh_ due to a moved `make` target.
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/5197
GitOrigin-RevId: 638b2eb39c99e7a990e7a6c66388831438aed45f
Rather than a homebrewed approach, we can use `make` to figure out when
it's necessary to regenerate our venv.
This Makefile will regenerate _requirements.txt_ from
_requirements-top-level.txt_ when the latter is changed.
It will also regenerate the venv when _requirements.txt_ is changed
(i.e. changes are pulled, or it's regenerated as described above).
`make` uses file/directory timestamps to figure out what to rebuild.
This is probably more reliable than expecting people to update a version
number whenever they change a file.
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/5152
GitOrigin-RevId: 24b27d49bf6c4ba1d57ac38ea38ae278216c6d66
This helps us use the same versions locally as in CI. If you're using the Nix setup, it guarantees it.
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/5111
GitOrigin-RevId: 6e00cd7a78593df1e60fac37cc1195aba60e488f
I intend to use the Make targets around starting and stopping backend databases in order to wrap the Python integration tests in targets.
To that end, I have:
1. Factored out test infrastructure targets into _test-infrastructure.mk_.
2. Actually run the functions from _util.sh_; previously we were just running a bash script with no explicit commands. _util.sh_ must be `source`d and then you invoke the relevant function.
3. Run `docker-compose` through Nix if possible.
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/5136
GitOrigin-RevId: d7a01958045a82199c969ec3f88387d7d76d254b
I tried re-freezing _server/tests-py/requirements-top-level.txt_
recently, and discovered that it caused the tests to fail.
This pins a couple of dependencies so that we can safely re-freeze.
Specifically:
- `cryptography` is pinned at v3.*
- `graphene` is pinned at v2.*
- `PyJWT` is pinned at v2.3.*
- `websocket-client` is pinned at v0.56.0 (this was done in
_requirements.txt_ already, but that file is supposed to be
regenerated)
Upgrading `SQLAlchemy` required changing PostgreSQL URLs to use
"postgresql://" as the URL scheme, not "postgres://".
Updating `ruamel.yaml` caused a few tests to fail as we are passing
`ruamel.yaml.scalarstring.LiteralScalarString` values as header values.
This is fixed by explicitly converting header values to strings.
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/5120
GitOrigin-RevId: 9c12a3013c3d1f23dddbe781037663838b23f6f5
When pytest rewrites assertions to make them more useful, it also
truncates long assertion messages in the middle, often obscuring the
actual error.
Disabling this allows us to see the full message, which should hopefully
allow us to see the actual error.
In order to support this, we need to make sure that custom assertion
messages include the actual assertion information, as this will no
longer be rendered by pytest.
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/4835
GitOrigin-RevId: de6839a3b40d0edc7cb96e46625eebca0aaf0c95
This improves the experience of testing against BigQuery in a few ways:
1. There is an explicit instruction to create a dataset, which was not
documented.
2. The verification script uses the environment variables, just like the
tests.
3. The verification script passes the correct content type headers.
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/4706
GitOrigin-RevId: 15b5bd28b1762c130c0ec09d6babe8c08e05ab15
(Work here originally done by awjchen, rebased and fixed up for merge by
jberryman)
This is part of a merge train towards GHC 9.2 compatibility. The main
issue is the use of the new abstract `KeyMap` in 2.0. See:
https://hackage.haskell.org/package/aeson-2.0.3.0/changelog
Alex's original work is here:
#4305
BEHAVIOR CHANGE NOTE: This change causes a different arbitrary ordering
of serialized Json, for example during metadata export. CLI users care
about this in particular, and so we need to call it out as a _behavior
change_ as we did in v2.5.0. The good news though is that after this
change ordering should be more stable (alphabetical key order).
See: https://hasurahq.slack.com/archives/C01M20G1YRW/p1654012632634389
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/4611
Co-authored-by: awjchen <13142944+awjchen@users.noreply.github.com>
GitOrigin-RevId: 700265162c782739b2bb88300ee3cda3819b2e87
…p CI (#3975)
Trying out a new convention here for jobs that generate multiple
artifacts at different points in time, and where downstream jobs depend
on just a particular subset of artifacts and don't need to wait for the
entire job. We accomplish this with separate `Input` steps, which we
unblock manually via calls to the buildkite API once particular sets of
artifacts are complete.
We could accomplish the same thing by having multiple build server jobs
focused on particular targets ( this is in fact how it used to work)
but that is quite wasteful of CI resources; in order to build any of
the binaries quickly, we need significant number of cores; but if we
only build particular targets, we don't take advantage I've all those
cores relative to the parallelism a full cabal build gives us. That
would likely be more difficult to maintain as well.
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/4256
Co-authored-by: Vishnu Bharathi <4211715+scriptnull@users.noreply.github.com>
GitOrigin-RevId: 5f5603da8a85f9a491877d51a7c08764c7199602
UPDATE: After testing in CI it turns out that the compile time Improvement is better than expected: even though we always have to recompile the OSS lib (due to Version.hs), downstream packages like Pro and multi-tenant can still benefit from some caching and avoid full recompilation. In the best case this takes us from 22 minutes to 13 minutes total.
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/4104
GitOrigin-RevId: 76cbfc157064b33856e30f4c2b2ab2366f9c6089
## Description
This PR updates the notes-extracting script, to allow for a stable sort across runs:
- files are now treated in alphabetical order of the full file name:
- within a file, notes are treated in order
- that order is reflected in the generated index file
- within a note, references are sorted by file, then by line number
Additionally it fixes(?) a note whose format was unexpected.
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/3352
GitOrigin-RevId: 2b2b57ec0aa2657d75a88e4951e6b19bb2d665e3
_(This PR is on top of #3352.)_
## Description
This PR overhauls our documentation CI steps to push all generated server documentation to the `gh-pages` branch of the OSS repo. The goal of this PR is to arrive in the situation where `https://hasura.github.io/graphql-engine/server/` is automatically populated to contain the following:
- all the markdown files from `server/documentation`, copied verbatim, no transformation applied
- all the notes, collected from the code by the `extract-notes.sh` script, in `server/notes`
- the generated haddock documentation for each major release or branch in `server/haddock`.
To do so, this PR does the following:
- it includes the script to extract notes from #3352,
- it rewrites the documentation checking CI step, to generate the notes and publish the resulting "server/documentation" folder,
- it includes a new CI step to deploy the documentation to the `gh-pages` branch
Of note:
- we will generate a different haddock folder for each main branch and release; in practice, that means the _main_, _stable_, _alpha_, _beta_ branches, and every build tagged with a version number
- the step that builds the haddock documentation checks that ALL projects in the repo build, including pro, but the deploy only deploys the graphql-engine documentation, as it pushes it to a publicly-accessible place
## Required work
**DO NOT MERGE THIS PR IT IS NOT READY**. Some work needs to go into this PR before it is ready.
First of all: the `gh-pages` branch of the OSS repo does NOT yet contain the documentation scaffolding that this new process assumes. At the bare minimum, it should be a orphan branch that contains a top-level README.md file, and a _server_ folder. An example of the bare minimum required can be previewed [on my fork](https://nicuveo.github.io/graphql-engine/server/).
The content of the `server/documentation` folder needs to be adjusted to reflect this; at the very least, a `README.md` file needs to be added to do the indexing (again, see the placeholder [on my fork](https://nicuveo.github.io/graphql-engine/server/) for an example).
This way of publishing documentation must be validated against [proposed changes to the documentation](https://github.com/hasura/graphql-engine-mono/pull/3294). @marionschleifer what do you think?
~~The buildkite code in this branch is currently untested, and I am not sure how to test it.~~
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/3380
GitOrigin-RevId: b24f6759c64ae29886c1f1b481b172febc512032
There's two major parts to this change:
1. cut down on environment variables needed to run the tests:
- lux parameters that don't change are now in code
- most remaining parameters have reasonable defaults
- the only variable that is still required to be set is HASURA_MULTITENANT_SPEC_DB_URL
- env.sh is no longer needed
2. find a better work-around for the problems running graphql-engine-multitenant
via cabal run (https://github.com/haskell/cabal/issues/7914), by adding a shell
script that implements a more correct version of cabal run.
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/3461
GitOrigin-RevId: 0939a79cc45cd3c1c103719552b12099678850dd
So far we've only used `hlint` to lint the OSS code. This moves some things around to lint the Pro code as well.
Note that the CI action only runs `hlint` on Haskell files that are _changed_ by a PR, relative to its merge-base (usually `main`).
As a reminder, you can use the `ignore-server-hlint-checks` label to prevent this from blocking a merge.
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/3383
GitOrigin-RevId: d5779c63d780f526a1d58ae4107f0d5262a23ec1
This PR upgrades some of the pinned dependencies do not build with python 3.10 - cffi, ruamel, py. Further, it upgrades other packages where the effort is minimal.
For the reviewers: Please review it commit by commit.
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/3367
GitOrigin-RevId: c5401fe289d3185a79c4d382297f86fbde139825
