There are a few issues this covers:
- Some fixtures like `page` and `context` are not allowed in `beforeAll`
hooks, so using them in automatic fixture makes it throw.
- Running automatic fixture solely for `afterAll` is unexpected.
This currently happens when `afterAll` is run for cleanup after
fixture timeout/throw.
For built-in playwright fixture, we keep `'all-hooks-included'` auto mode.
Added a doc explaining the execution order.
Reland: worker.stop() before worker.run() was hanging because
`_runFinished` promise was not initially resolved.
---
This moves `beforeAll`, `afterAll` and some modifiers from running
as a separate entity into running inside a test.
Pros:
- All errors are reported as test errors.
- All artifacts are collected as test artifacts.
- Reporters support this out of the box.
Details:
- Each test computes the necessary hooks to run and runs them.
- Teardown is usually performed during the test (on test failure or worker stop).
- `skipRemaining` is added to `DonePayload` to preserve the behavior
where `beforeAll` hook failure skips subsequent tests.
This behavior can now be improved to only target tests affected by this hook.
* Revert "fix(hooks): separate test timeout from beforeAll/afterAll timeouts (#12413)"
This reverts commit 73dee69558.
* Revert "fix(test-runner): rely on test title paths instead of ordinal (#12414)"
This reverts commit d744a87aee.
* Revert "chore(test runner): run hooks/modifiers as a part of the test (#12329)"
This reverts commit 47045ba48d.
chore(test runner): run hooks/modifiers as a part of the test
This moves `beforeAll`, `afterAll` and some modifiers from running
as a separate entity into running inside a test.
Pros:
- All errors are reported as test errors.
- All artifacts are collected as test artifacts.
- Reporters support this out of the box.
Details:
- Each test computes the necessary hooks to run and runs them.
- Teardown is usually performed during the test (on test failure or worker stop).
- `skipRemaining` is added to `DonePayload` to preserve the behavior
where `beforeAll` hook failure skips subsequent tests.
This behavior can now be improved to only target tests affected by this hook.
1. Fixtures defined in test.extend() can now have `{ option: true }` configuration that makes them overridable in the config. Options support all other properties of fixtures - value/function, scope, auto.
```
const test = base.extend<MyOptions>({
foo: ['default', { option: true }],
});
```
2. test.declare() and project.define are removed.
3. project.use applies overrides to default option values and nothing else. Any test.extend() and test.use() calls take priority over config options.
Required user changes: if someone used to define fixture options with test.extend(), overriding them in config will stop working. The solution is to add `{ option: true }`.
```
// Old code
export const test = base.extend<{ myOption: number, myFixture: number }>({
myOption: 123,
myFixture: ({ myOption }, use) => use(2 * myOption),
});
// New code
export const test = base.extend<{ myOption: number, myFixture: number }>({
myOption: [123, { option: true }],
myFixture: ({ myOption }, use) => use(2 * myOption),
});
```
When the test fails (usually with timeout), we wait until all hooks are run
and worker scope is teared down before reporting test end result.
This allows us to collect any error details populated by teardown
in addition to the "timed out" message.
Each hook gets its own test scope. This is not too useful for
object fixtures like `page` (although one can use a page in
`beforeAll` to save storage state), but much more useful for option
fixtures like `viewport`.
- Source now lives at `src/test`.
- Former folio tests live at `tests/playwright-test`.
- We use `src/test/internal.ts` that exposes base test without
Playwright fixtures for most tests (to avoid modifications for now).
- Test types live in `types/testFoo.d.ts`.
- Stable test runner is installed to `tests/config/test-runner` during `npm install`.
- All deps including test-only are now listed in `package.json`.
Non-test deps must also be listed in `build_package.js` to get included.
