Commit Graph

31 Commits

Author SHA1 Message Date
Pavel Feldman
f5df0940c9
chore: refactor timeout manager to use scopes (1) (#24315) 2023-07-20 17:21:21 -07:00
Dmitry Gozman
dee0f2791e
test: unflake "should give enough time for fixture teardown" (#22763) 2023-05-02 17:40:10 -07:00
Dmitry Gozman
6d27d2c029
fix(test runner): improved fixture teardown timeout error message (#21572)
The new message looks like this:

```
Test finished within timeout of 500ms, but tearing down "playwright configuration" ran out of time.
Please allow more time for the test, since teardown is attributed towards the test timeout budget.·
```

References #21259.
2023-03-10 13:45:47 -08:00
Dmitry Gozman
91da67fab1
test: remove magic headers in ttest (#20867)
Instead, explicitly import from '@playwright/test'.
2023-02-14 19:20:56 -08:00
Pavel Feldman
1b941bcf2e
chore: simplify ttests (#20733) 2023-02-07 15:11:44 -08:00
Pavel Feldman
233664bd30
chore: report more fatal errors via reporter (#19640) 2022-12-22 17:31:02 -08:00
Dmitry Gozman
62e4e80599
feat(test runner): show the number of fatal errors at the end (#15975) 2022-07-28 14:46:21 -07:00
Dmitry Gozman
c0d78c5642
fix(runner): friendly error message instead of "digests do not match" (#15939) 2022-07-26 08:53:32 -07:00
Dmitry Gozman
a56d801352
fix(test runner): show fixture name when worker times out (#15724) 2022-07-15 13:05:48 -07:00
Dmitry Gozman
894ead5198
fix(test runner): mention fixture in worker teardown timeout error message (#15499) 2022-07-08 11:44:37 -07:00
Dmitry Gozman
ee82837fb5
fix(fixtures): do not run user function when required fixture has failed (#15385)
Currently, it is possible to run a function, e.g. a second `beforeEach` hook,
that will receive `null` for the fixture that has already failed before.

This PR skips running any user function that uses a fixture that has already
failed, just like if the fixture would be initialized again and failing for
the second time.
2022-07-05 17:15:28 -07:00
Dmitry Gozman
b86926e691
fix(test runner): update timeout error messages (#15117)
Some of the current timeout error messages are confusing, because they do not suggest that the issue is most likely a slow test. This PR updates timeout messages as follows:
- Test timeout of 30000ms exceeded.
- Test timeout of 30000ms exceeded while setting up "browser".
- Test timeout of 30000ms exceeded while tearing down "context".
- Test timeout of 30000ms exceeded while setting up "playwright configuration".
- Test timeout of 30000ms exceeded while running "beforeEach" hook.
- Test timeout of 30000ms exceeded while running "afterEach" hook.
- "beforeAll" hook timeout of 30000ms exceeded.
- "afterAll" hook timeout of 30000ms exceeded.
- Worker teardown timeout of 30000ms exceeded.
- "skip" modifier timeout of 30000ms exceeded.
- Fixture "myCustomFixture" timeout of 5000ms exceeded.
2022-06-30 17:05:08 -07:00
Dmitry Gozman
ef32069299
chore: small test runner changes in preparation of global fixtures (#13899)
The main change is splitting up options from the config from other
fixtures to ensure unique location for them.
2022-05-03 15:19:27 +01:00
Dmitry Gozman
166675b9c1
feat(test): more information about timing out fixtures (#13546)
- Always show a fixture that was running during timeout.
- Give custom titles to built-in fixtures.
- Specify setup/teardown fixture phase in the message.
- Split connect vs launch browser fixtures for better naming.

Example timeout message:

```log
Timeout of 2000ms exceeded while running fixture "built-in playwright configuration" teardown.
```
2022-04-13 15:13:31 -07:00
Dmitry Gozman
de0af27837
feat(test runner): show last tests in the worker on teardown error (#13139) 2022-03-28 19:58:24 -07:00
Dmitry Gozman
009185bb89
fix(test runner): do not print fixture location without a separate timeout (#12894)
Otherwise, user randomly sees some "extend" calls that have no real meaning.
2022-03-18 16:09:41 -07:00
Dmitry Gozman
25483452c0
feat(fixtures): per-fixture timeout (#12751)
By default, fixtures share timeout with the test they are instantiated for.
However, for more heavy fixtures, especially worker-scoped ones, it makes
sense to have a separate timeout.

This introduces `{ timeout: number }` option to the list of fixture options
that opts the fixture into a dedicated timeout rather than sharing it
with the test.
2022-03-17 09:36:03 -07:00
Dmitry Gozman
9cd704220c
reland(test runner): run hooks/modifiers as a part of the test (#12518)
Reland: worker.stop() before worker.run() was hanging because
`_runFinished` promise was not initially resolved.
---
This moves `beforeAll`, `afterAll` and some modifiers from running
as a separate entity into running inside a test.

Pros:
- All errors are reported as test errors.
- All artifacts are collected as test artifacts.
- Reporters support this out of the box.

Details:
- Each test computes the necessary hooks to run and runs them.
- Teardown is usually performed during the test (on test failure or worker stop).
- `skipRemaining` is added to `DonePayload` to preserve the behavior
  where `beforeAll` hook failure skips subsequent tests.
  This behavior can now be improved to only target tests affected by this hook.
2022-03-08 16:35:14 -08:00
Pavel Feldman
6a663ef54f
chore(test-runner): revert recent changes to fix tests (#12439)
* Revert "fix(hooks): separate test timeout from beforeAll/afterAll timeouts (#12413)"

This reverts commit 73dee69558.

* Revert "fix(test-runner): rely on test title paths instead of ordinal (#12414)"

This reverts commit d744a87aee.

* Revert "chore(test runner): run hooks/modifiers as a part of the test  (#12329)"

This reverts commit 47045ba48d.
2022-03-01 09:11:17 -08:00
Dmitry Gozman
47045ba48d
chore(test runner): run hooks/modifiers as a part of the test (#12329)
chore(test runner): run hooks/modifiers as a part of the test

This moves `beforeAll`, `afterAll` and some modifiers from running
as a separate entity into running inside a test.

Pros:
- All errors are reported as test errors.
- All artifacts are collected as test artifacts.
- Reporters support this out of the box.

Details:
- Each test computes the necessary hooks to run and runs them.
- Teardown is usually performed during the test (on test failure or worker stop).
- `skipRemaining` is added to `DonePayload` to preserve the behavior
  where `beforeAll` hook failure skips subsequent tests.
  This behavior can now be improved to only target tests affected by this hook.
2022-02-28 11:42:47 -08:00
Dmitry Gozman
85cb3c9713
test: add a test that fixture error after timeout is not a fatal error (#12141) 2022-02-15 18:05:20 -08:00
Andrey Lushnikov
2336692e8a
feat: support clarification message for expect (#11735)
The clarification message is displayed in the HTML report as the name of the step:

![image](https://user-images.githubusercontent.com/746130/151852652-48194140-5ea4-439d-afee-12583a8caf71.png)

It is also shown in terminal output:

![image](https://user-images.githubusercontent.com/746130/151852666-5c956ef1-6e94-4bc2-8e55-b58688dfc7e0.png)

Fixes #7816
2022-01-31 17:14:59 -08:00
Dmitry Gozman
9d5bf0e90d
fix(test runner): fixture teardown double error, testInfo.attach() (#11365)
- Use file path, not content to calculate the attachment hash.
- Always cleanup fixture from the list on teardown, to avoid reporting
  teardown error multiple times: from the test, and from the cleanup.
2022-01-13 10:38:47 -08:00
Dmitry Gozman
f5304e3bda
fix(fixtures): await fixture teardown when shutting down the worker (#11033) 2021-12-20 16:19:21 -08:00
Dmitry Gozman
ed9b42a92d
feat(test runner): collect test error from worker teardown (#9190)
When the test fails (usually with timeout), we wait until all hooks are run
and worker scope is teared down before reporting test end result.

This allows us to collect any error details populated by teardown
in addition to the "timed out" message.
2021-09-28 10:56:50 -07:00
Dmitry Gozman
87548f94c1
feat(test runner): support test fixtures in beforeAll/afterAll (#8082)
Each hook gets its own test scope. This is not too useful for
object fixtures like `page` (although one can use a page in
`beforeAll` to save storage state), but much more useful for option
fixtures like `viewport`.
2021-08-09 13:26:33 -07:00
Dmitry Gozman
41949e559e
Revert "feat(test runner): file scope fixtures (#7969)" (#8081)
This reverts commit 1bbf86d060,
leaving small improvements around.
2021-08-09 12:33:16 -07:00
Dmitry Gozman
1bbf86d060
feat(test runner): file scope fixtures (#7969)
These are reset after running tests from a single file.
2021-08-04 21:11:02 -07:00
Dmitry Gozman
4c6fa42810
fix(test runner): expose real stack traces and speed up locations (#7265)
Stop wrapping/prepending error messages so that we do not loose the stack trace. For this, update a few manually thrown errors with better messages (usually including a file path).

Speed up locations by doing manual `sourceMapSupport.wrapCallSite()` for a single call site. Performance gain in the runner process with 100 files x 100 tests each:
- 25% on the fresh run without babel cache;
- 80% on the cached run where babel is almost instant.

Also some obvious cleanups around stack traces (removing unused code).
2021-06-23 10:30:54 -07:00
Dmitry Gozman
46a0213769
chore: remove internal uses of "folio" (#6931)
Replaced by "pwt" or "playwright test".
2021-06-06 22:07:07 -07:00
Dmitry Gozman
f745bf1fbc
chore: bring in folio source (#6923)
- Source now lives at `src/test`.
- Former folio tests live at `tests/playwright-test`.
- We use `src/test/internal.ts` that exposes base test without
  Playwright fixtures for most tests (to avoid modifications for now).
- Test types live in `types/testFoo.d.ts`.
- Stable test runner is installed to `tests/config/test-runner` during `npm install`.
- All deps including test-only are now listed in `package.json`.
  Non-test deps must also be listed in `build_package.js` to get included.
2021-06-06 17:09:53 -07:00