- Updating our config to have `--check-coverage` enforces that the coverage meets a certain level.
- The default is 95 I believe, but our coverage is lower.
- I've set the levels to our current levels, so any drop below these numbers will cause the build to fail.
- I've also set the reporters to be text, html and cobertura so we always have a mini report, the full HTML files to navigate and cobertura for CI
- Cleaned up CI so we don't use the cov:unit command as we're now using codecov
- This also means we can remove the cov:unit command which was weird to use because it uses the last test run, which can be confusing
refs https://github.com/TryGhost/Toolbox/issues/208
- we currently run all test commands separately in CI and this can make
it harder to replicate any issues we see in CI because we have to
manually go and copy the order of the tests
- it's also nicer if there's just one command that runs so we can make
changes globally to CI
- this commit adds a test:ci yarn command which will run the tests in
order of speed/importance, with the -b AKA fail-fast flag so we
don't have to wait for all tests to run before finding out about
issues
- this also cleans up running unit tests in the DB matrix
refs https://github.com/TryGhost/Toolbox/issues/208
- unit tests (shouldn't) be database dependent so they don't need to be
included in the DB matrix
- this prevents an extra 6 executions of unit tests, which isn't a big
deal given they only take 10s to run, but semantically they belong
outside so this commit does that
- also updates the canary step to rely on this extra test run
no issue
- we might fail the tests but not produce any logs in Ghost-CLI
- concatenating the logs would then fail because of the lack of existence
of a directory
- this commit adds a check that files exist before trying to get the
contents of the files
refs https://github.com/TryGhost/Toolbox/issues/181
- we officially support MySQL 8 but we don't run tests for it -- oops
- this commit adds MySQL 8 to the DB matrix in CI tests
- I had to switch to a fork of the `mysql-action` repo so I could get
the native authentication plugin working, but I expect to revert this
once we've merged support for mysql2
refs
https://docs.github.com/en/actions/learn-github-actions/workflow-syntax-for-github-actions#example-using-a-fallback-value
- CI sometimes becomes clogged up with concurrent runs for a PR if the
PR is updated
- this sometimes happens when we merge a bunch of Renovate PRs and other
PRs rebase on top of `main` multiple times
- we shouldn't continue to run tests for PRs if the branches have been pushed
to since they started
- from the referenced link above, this is the built-in solution that
cancels previous in-progress runs if a new job starts
no issue
Includes a timeout if TailScale is completely inaccessible, and a
continue-on-error statement which should handle all forms of
failure. The following step also includes a continue-on-error step, as
reporting metrics depends on TailScale succeeding, and should also
never fail the build.
refs https://github.com/TryGhost/Ghost/pull/13716
refs https://github.com/actions/setup-node/issues/317#issuecomment-929694556
- the `setup-node` GitHub Action seems to use a shell command to get the
cache path, but these are colorised when `FORCE_COLOR` is enabled
- this causes the Action to fail to read the path correctly
- the comment referenced above suggests to remove `FORCE_COLOR` but it's
nice to have colored output for our tests
- instead, I'm disabling the environment variable on the `setup-node`
action so it still works
- I've tested with the referenced PR and this unblocks dependency caching 🎉
Refs CORE-120
When test runs execute, use the new @tryghost/metrics package to send metrics to configurable backends.
At the moment, we're just sending the test run duration to our metric store in preparation of changes to the test suite.
refs https://github.com/TryGhost/Toolbox/issues/71
- Node 16 is now LTS so we're adding support in Ghost
- we're also bumping the minimum Node 14 version to 14.17.0 so we can merge
dependencies who have higher Node 14 requirements than current
- this commit adds Node 16 to the `package.json` engines and to CI
- also bumps all dependencies that needed new versions to add Node 16
support
refs linear.app/tryghost/issue/CORE-74/improve-the-test-situation
- this commit adds the codecov GitHub Action into CI so we can upload
coverage reports
- the coverage files need to be in XML for them to work with
codecov, so this commit also adds cobertura (XML) as a reporter
no issue
- coverage allows us to see how much of our code we're hitting in tests
and it's time we started to get visibility on this in Ghost
- we can then make informed decisions on which well-tested internal libraries
can be pulled out into their own packages
- this commit:
- adds the c8 dev dependency to Ghost
- prepends the `test:unit` command with c8 with the `text-summary` reporter
- adds a `coverage:unit` command to get the `text` report
- you can do `yarn coverage:unit --reporter html` to get a HTML report etc
- uses this new test coverage reporter command in CI tests
no issue
- as we're making more and more use of branches for releases, we want
tests to run on them
- this commit adds the v4 wildcard to the GitHub workflow so it'll
trigger when we push to them
This commit achieves a few things:
- ☑️ No longer having to remember whether a command is yarn something or grunt something
- ☑️ Simplification of tools hopefully making them easier to remember and use
- ☑️ Complete removal of the need for grunt from our test tooling
Several of the tools still use grunt under the hood, but the **entrypoint** should aways be `yarn xxx`.
- `grunt main` -> `yarn main`
- `grunt dev` -> `yarn dev`
- `grunt build` -> `yarn build`
- `grunt test:file-or-folder` -> `yarn test file-or-folder`
- `grunt test-unit` -> `yarn test:unit`
- `grunt test-acceptance` -> `yarn test:acceptance`
- `grunt test-regression` -> `yarn test:regression`
- `grunt validate` -> removed due to lack of use
There is now also `yarn test:all` to run all 3 classes of tests
This PR also reorders & restructures the Gruntfile extensively so that:
- The remaining useful commands are all at the top of the file
- Config and other blah happens after all the useful commands
- All release-only config happens in the release task at the very end of the file
---
DONE:
* Removed all references to npm/bower
* Removed all references to lint / deprecated command
* Moved debug to yarn dev:debug
* Removed all references to travis
* Removed broken help task + useless comment
* Removed unused knex-migrator and clean:test setup tasks
* Added new test commands, removed grunt validate
* Moved stubClientFiles to test utility and use in the few tests that need it
* Used mocha in yarn directly except grunt test:x
* Swapped grunt test for yarn test
* extensive cleanup and reshuffling
no issue
- if Ghost-CLI tests fail, the CLI will put some extra debug logs in
`~/.ghost/logs`
- these are useful to help see what the issue is, so we should print
them out if we're experiencing a failure
- this commit cats the contents of the folder so we can see it in CI
no issue
- by extracting it to a different workflow that uses the
`workflow_dispatch` trigger, we can manually run a build, but also
trigger it from Admin so we can always stay up-to-date
refs 57ff38da8a
- we're dropping support for Node 10 which involved bumping the Node
version we run our Ghost-CLI tests on but they fail because Ghost v1
and v2 don't support Node 12
- this commit disables these tests until we come up with a good
workaround
refs https://github.com/TryGhost/Team/issues/658
- Node 10 become EOL as of April 30th so it's time to drop support
- this commit:
- removes the Node 10 range from the `node` `engines` block
- removes Node 10 from CI tests
- switches Node 10 in the CLI test to Node 12 so we can ensure it
installs on our oldest supported Node version
no issue
- now we've officially released v4, we need to check updates from v3
- this commit adds a test copied from the other tests above
- also renames the steps from Upgrade to Update to match internal terminology
no issue
- when building the canary zip, we currently use `main` within Casper,
but the 4.0 changes are on the 4.0 branch
- this commit updates the workflow to checkout that branch as part of
the pipeline
no issue
- some new GScan changes mean that CI now throws a warning when using v3
Casper, which causes tests to fail
- we need to switch to v4 to get the tests to pass
- we can't put v4 Casper in `main` because it'll autodeploy and be
unusable for anyone downloading until Ghost 4.0 is released
no issue
- Ghost-CLI uses gscan under the hood to check the theme, but GScan has
not yet been updated to support v4
- disabling these tests for now until it does
no issue
- in 4.0, we don't expect to rollback to 3.0 because there are
irreversible migrations
- the existing test attempts to do this because it was written for v3
- this commit updates the test to apply to v4, although this will need
to change soon when we switch branches
no issue
- we want tests to run on the 4.0 branch, along with Slack notifications
- we also want a canary build of the 4.0 branch, but this requires some
alterations to bump to a premajor + use a different artifact name
- if we only run 'knex-migrator init', the settings don't get populated like
they would when running Ghost
- switching to running Ghost for initialization is more realistic of
what would happen IRL
