…p CI (#3975)
Trying out a new convention here for jobs that generate multiple
artifacts at different points in time, and where downstream jobs depend
on just a particular subset of artifacts and don't need to wait for the
entire job. We accomplish this with separate `Input` steps, which we
unblock manually via calls to the buildkite API once particular sets of
artifacts are complete.
We could accomplish the same thing by having multiple build server jobs
focused on particular targets ( this is in fact how it used to work)
but that is quite wasteful of CI resources; in order to build any of
the binaries quickly, we need significant number of cores; but if we
only build particular targets, we don't take advantage I've all those
cores relative to the parallelism a full cabal build gives us. That
would likely be more difficult to maintain as well.
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/4256
Co-authored-by: Vishnu Bharathi <4211715+scriptnull@users.noreply.github.com>
GitOrigin-RevId: 5f5603da8a85f9a491877d51a7c08764c7199602
UPDATE: After testing in CI it turns out that the compile time Improvement is better than expected: even though we always have to recompile the OSS lib (due to Version.hs), downstream packages like Pro and multi-tenant can still benefit from some caching and avoid full recompilation. In the best case this takes us from 22 minutes to 13 minutes total.
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/4104
GitOrigin-RevId: 76cbfc157064b33856e30f4c2b2ab2366f9c6089
## Description
This PR updates the notes-extracting script, to allow for a stable sort across runs:
- files are now treated in alphabetical order of the full file name:
- within a file, notes are treated in order
- that order is reflected in the generated index file
- within a note, references are sorted by file, then by line number
Additionally it fixes(?) a note whose format was unexpected.
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/3352
GitOrigin-RevId: 2b2b57ec0aa2657d75a88e4951e6b19bb2d665e3
_(This PR is on top of #3352.)_
## Description
This PR overhauls our documentation CI steps to push all generated server documentation to the `gh-pages` branch of the OSS repo. The goal of this PR is to arrive in the situation where `https://hasura.github.io/graphql-engine/server/` is automatically populated to contain the following:
- all the markdown files from `server/documentation`, copied verbatim, no transformation applied
- all the notes, collected from the code by the `extract-notes.sh` script, in `server/notes`
- the generated haddock documentation for each major release or branch in `server/haddock`.
To do so, this PR does the following:
- it includes the script to extract notes from #3352,
- it rewrites the documentation checking CI step, to generate the notes and publish the resulting "server/documentation" folder,
- it includes a new CI step to deploy the documentation to the `gh-pages` branch
Of note:
- we will generate a different haddock folder for each main branch and release; in practice, that means the _main_, _stable_, _alpha_, _beta_ branches, and every build tagged with a version number
- the step that builds the haddock documentation checks that ALL projects in the repo build, including pro, but the deploy only deploys the graphql-engine documentation, as it pushes it to a publicly-accessible place
## Required work
**DO NOT MERGE THIS PR IT IS NOT READY**. Some work needs to go into this PR before it is ready.
First of all: the `gh-pages` branch of the OSS repo does NOT yet contain the documentation scaffolding that this new process assumes. At the bare minimum, it should be a orphan branch that contains a top-level README.md file, and a _server_ folder. An example of the bare minimum required can be previewed [on my fork](https://nicuveo.github.io/graphql-engine/server/).
The content of the `server/documentation` folder needs to be adjusted to reflect this; at the very least, a `README.md` file needs to be added to do the indexing (again, see the placeholder [on my fork](https://nicuveo.github.io/graphql-engine/server/) for an example).
This way of publishing documentation must be validated against [proposed changes to the documentation](https://github.com/hasura/graphql-engine-mono/pull/3294). @marionschleifer what do you think?
~~The buildkite code in this branch is currently untested, and I am not sure how to test it.~~
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/3380
GitOrigin-RevId: b24f6759c64ae29886c1f1b481b172febc512032
There's two major parts to this change:
1. cut down on environment variables needed to run the tests:
- lux parameters that don't change are now in code
- most remaining parameters have reasonable defaults
- the only variable that is still required to be set is HASURA_MULTITENANT_SPEC_DB_URL
- env.sh is no longer needed
2. find a better work-around for the problems running graphql-engine-multitenant
via cabal run (https://github.com/haskell/cabal/issues/7914), by adding a shell
script that implements a more correct version of cabal run.
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/3461
GitOrigin-RevId: 0939a79cc45cd3c1c103719552b12099678850dd
So far we've only used `hlint` to lint the OSS code. This moves some things around to lint the Pro code as well.
Note that the CI action only runs `hlint` on Haskell files that are _changed_ by a PR, relative to its merge-base (usually `main`).
As a reminder, you can use the `ignore-server-hlint-checks` label to prevent this from blocking a merge.
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/3383
GitOrigin-RevId: d5779c63d780f526a1d58ae4107f0d5262a23ec1
This PR upgrades some of the pinned dependencies do not build with python 3.10 - cffi, ruamel, py. Further, it upgrades other packages where the effort is minimal.
For the reviewers: Please review it commit by commit.
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/3367
GitOrigin-RevId: c5401fe289d3185a79c4d382297f86fbde139825
The motivation here is:
- save some CI resources
- make it cleaner/more sensible to do library caching
Since OSS is a dependency of pro, and at the same time the various
executables can be built in parallel, doing this all in the same job
probably is more sensible.
Future improvements: there are separate pro and oss pipelines and one or
the other of those will need to wait unnecessarily while everything
finishes. In practice, currently this isn't a big deal, but we could use
the BuildKite blocking job API to get more fine-grained downstream
triggering.
GitOrigin-RevId: aa917996cbfc678becaaafca490d6edc7de89b1f
### Description
This PR fixes two small issues with the `dev.sh` script:
- when running the script without a specific test flag (`./scripts/dev.sh test`), we would start some of the containers more than once; this PR fixes this by checking whether the container is already running before starting it
- we were still doing some bigquery setup in the main code, despite that setup being already done in `parse-pytest-backend`, which would result in biquery setup errors even when testing non-bigquery backends; this PR just deletes the redundant line.
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/3017
GitOrigin-RevId: 86a53b1fe974aae733a02bdf045389a0d7436cb7
### Description
The inherited roles integration tests were behind a flag, and its corresponding fixture, presumably to avoid enabling the option globally. However, #2288 introduced a new test using inherited roles that was not gated behind the flag, which fails when run with `dev.sh`. However, that test works on CI... because inherited roles are globally enabled there.
Consequently, this PR:
- globally enables inherited roles in dev.sh
- removes the flag and the associated fixture
https://github.com/hasura/graphql-engine-mono/pull/2358
Co-authored-by: Vishnu Bharathi <4211715+scriptnull@users.noreply.github.com>
GitOrigin-RevId: ebfa6754873324bed15b2cc5e37ec2d8008e8f8d
### Description
When running the tests with `dev.sh`, we always build the engine, before attempting to run the tests. This builds the engine twice, due to the flag change. This PR changes this to only do this first build if we're planning to actually run the integration tests.
https://github.com/hasura/graphql-engine-mono/pull/1964
GitOrigin-RevId: 32e8a6568f3665ad99adaf42421a711241974638
This essentially restores the original code from c425b554b8
(https://github.com/hasura/graphql-engine/pull/4013). Prior to this
commit we would slurp messages as fast as possible from the database
(one thing c425b55 fixed).
Another thing broken as a consequence of the same logic was the
removeEventFromLockedEvents logic which unlocks in-flight events
(breaking at-least-once delivery)
Some archeology, post-c425b55:
- cc8e2ccc erroneously attempted to refactor using `bracket`, resulting
in the same slurp-all-events behavior (since we don't ever wait for
processEvent to complete)
- at some point event processing within a batch is made serial, this
reported as a bug. See: https://github.com/hasura/graphql-engine/issues/5189
- in 0ef52292b5 (which I approved...) an `async` is added, again
causing the same issue...
GitOrigin-RevId: d8cbaab385267a4c3f1f173e268a385265980fb1
This claws back ~7min from integration tests (run serially, as with `dev.sh test --integration`
Further improvements would do well to focus on optimizing metadata operations, as `setup` dominates
GitOrigin-RevId: 76637d6fa953c2404627c4391447a05bf09355fa
This also seems to squash a stubborn space leak we see with
subscriptions (linking to canonical #3388 for reference).
This may also fix some of the "Unexpected exception" websockets
exceptions we are now surfacing (see e.g. #4344)
Also: dev.sh: fix hpc reporting
Initial work on this done by Vamshi.
- Created new job test_and_build_cli_migrations which runs after test_and_build_cli
- Build the cli-migrations and cli-migrations-v2 and save the images as tar image.
- Run the test defined in each workflow v1 and v2.
- Load the image that was built earlier in deploy step
This is the result of a general audit of how we fork threads, with a
detour into how we're using mutable state especially in websocket
codepaths, making more robust to async exceptions and exceptions
resulting from bugs.
Some highlights:
- use a wrapper around 'immortal' so threads that die due to bugs are
restarted, and log the error
- use 'withAsync' some places
- use bracket a few places where we might break invariants
- log some codepaths that represent bugs
- export UnstructuredLog for ad hoc logging (the alternative is we
continue not logging useful stuff)
I had to timebox this. There are a few TODOs I didn't want to address.
And we'll wait until this is merged to attempt #3705 for
Control.Concurrent.Extended
* export metadata without nulls, empty arrays
* property tests for 'ReplaceMetadata' using QuickCheck
-> Derive Arbitrary class for 'ReplaceMetadata' dependant types
* reduce property test cases number to 30
QuickCheck generates the `ReplaceMetadata` value really large
for higher number test cases. Encoded JSON for such values is large and
consumes more memory. Thus, CI is giving up while running property
tests.
* circle-ci: Add property tests as saperate job
* add no command mode to tests
* add yaml.v2 to go mod
* remove indirect comment for yaml.v2 dependency
