Commit Graph

370 Commits

Author SHA1 Message Date
Andrey Lushnikov
75eef09c0d
feat: show expectation name as part of toHaveScreenshot title (#12612)
This patch adds snapshot file name as part of `toHaveScreenshot`
and `toMatchSnapshot` step title.
2022-03-09 20:09:45 -08:00
Dmitry Gozman
92430cc56c
chore(fixtures): do not force-close contexts in beforeAll (#12416)
This now happens the same way as for the test, in the worker cleanup
when browser closes, which is reported as a part of the test.
2022-03-09 07:26:46 -08:00
Dmitry Gozman
e8ce5d0258
feat(test runner): on beforeAll failure, precisely skip the tests (#12417)
Previously, we used to skip all the tests from the same file when
any `beforeAll` fails in the file.

Now, we only skip the rest of the tests affected by this particular
`beforeAll` and continue with other tests in the new worker.
2022-03-08 20:29:31 -08:00
Dmitry Gozman
e895bc2751
reland(test-runner): rely on test title paths instead of ordinal (#12611)
Fixes #11904

Co-authored-by: Andrey Lushnikov <lushnikov@chromium.org>
2022-03-08 19:06:25 -08:00
Dmitry Gozman
682eeb4ae0
reland(hooks): separate test timeout from beforeAll/afterAll timeouts (#12609)
This makes it possible to have longer `beforeAll`/`afterAll` and not
affect first/last test timeout.
2022-03-08 19:05:23 -08:00
Dmitry Gozman
9cd704220c
reland(test runner): run hooks/modifiers as a part of the test (#12518)
Reland: worker.stop() before worker.run() was hanging because
`_runFinished` promise was not initially resolved.
---
This moves `beforeAll`, `afterAll` and some modifiers from running
as a separate entity into running inside a test.

Pros:
- All errors are reported as test errors.
- All artifacts are collected as test artifacts.
- Reporters support this out of the box.

Details:
- Each test computes the necessary hooks to run and runs them.
- Teardown is usually performed during the test (on test failure or worker stop).
- `skipRemaining` is added to `DonePayload` to preserve the behavior
  where `beforeAll` hook failure skips subsequent tests.
  This behavior can now be improved to only target tests affected by this hook.
2022-03-08 16:35:14 -08:00
Andrey Lushnikov
b8af8458d6
fix: explicitly ignore maxDiffPixels in toMatchSnapshot (#12570)
Fixes #12564
2022-03-07 16:55:35 -08:00
Andrey Lushnikov
d7648d390c
fix(toHaveScreenshot): round ratio up (#12540)
This way the ratio value could be used right away in configuation.
2022-03-04 18:55:48 -08:00
Andrey Lushnikov
a3dff45974
chore: remove maxDiffPixels from toMatchSnapshot (#12539)
References #12441
2022-03-04 18:27:04 -08:00
Andrey Lushnikov
4953fc4845
chore: introduce toHaveScreenshot configuration options (#12507)
References https://github.com/microsoft/playwright/issues/12441
2022-03-04 17:30:43 -08:00
Andrey Lushnikov
b45c788a33
fix(disable-animations): rename boolean to enum (#12503)
Enums are more flexible in the long run since they let us easily
extend API.

References #12441
2022-03-04 11:02:59 -08:00
Andrey Lushnikov
1f5cfcaeec
fix(to-have-screenshot): show number of bad pixels in mismatched error (#12502)
Drive-by: do not show empty call log.

References #12441
2022-03-04 11:01:05 -08:00
Andrey Lushnikov
279e579097
chore: address pixelCount and pixelRatio review comments (#12501)
- documentation fixes
- rename pixelCount to maxDiffPixels
- rename pixelRatio to maxDiffPixelRatio
2022-03-03 23:17:31 -08:00
Pavel Feldman
34bf0d4b87
feat(test-runner): allow grepping in project (#12469) 2022-03-02 15:02:35 -08:00
Max Schmitt
cfcdc3e19c
chore: roll stable test-runner to ToT (#12472) 2022-03-02 23:47:55 +01:00
Pavel Feldman
4b19d59ec5
feat(test): introduce fully parallel mode (#12446) 2022-03-01 18:12:21 -08:00
Pavel Feldman
6a663ef54f
chore(test-runner): revert recent changes to fix tests (#12439)
* Revert "fix(hooks): separate test timeout from beforeAll/afterAll timeouts (#12413)"

This reverts commit 73dee69558.

* Revert "fix(test-runner): rely on test title paths instead of ordinal (#12414)"

This reverts commit d744a87aee.

* Revert "chore(test runner): run hooks/modifiers as a part of the test  (#12329)"

This reverts commit 47045ba48d.
2022-03-01 09:11:17 -08:00
Pavel Feldman
d14ecec612
chore: remove flaky-by-design test (#12434) 2022-03-01 08:52:52 -08:00
Andrey Lushnikov
d744a87aee
fix(test-runner): rely on test title paths instead of ordinal (#12414)
Fixes #11904
2022-02-28 15:40:23 -08:00
Dmitry Gozman
73dee69558
fix(hooks): separate test timeout from beforeAll/afterAll timeouts (#12413)
This makes it possible to have longer `beforeAll`/`afterAll` and not
affect first/last test timeout.
2022-02-28 15:09:04 -08:00
Andrey Lushnikov
f47423d315
devops: upload test-runner test results to flakiness dashboard (#12412)
Fixes #12152
2022-02-28 13:41:30 -08:00
Andrey Lushnikov
396d920145
feat(test-runner): implement expect(pageOrLocator).toHaveScreenshot (#12242)
Fixes #9938
2022-02-28 12:25:59 -08:00
Dmitry Gozman
47045ba48d
chore(test runner): run hooks/modifiers as a part of the test (#12329)
chore(test runner): run hooks/modifiers as a part of the test

This moves `beforeAll`, `afterAll` and some modifiers from running
as a separate entity into running inside a test.

Pros:
- All errors are reported as test errors.
- All artifacts are collected as test artifacts.
- Reporters support this out of the box.

Details:
- Each test computes the necessary hooks to run and runs them.
- Teardown is usually performed during the test (on test failure or worker stop).
- `skipRemaining` is added to `DonePayload` to preserve the behavior
  where `beforeAll` hook failure skips subsequent tests.
  This behavior can now be improved to only target tests affected by this hook.
2022-02-28 11:42:47 -08:00
Dmitry Gozman
3c2bca2768
fix(test loader): implement tsconfig paths through resolveFilename (#12357)
This uses `Module._resolveFilename` to intercept module resolution and
check `tsconfig.paths` similarly to pirates usage ot `Module._compile`.

Previously, we resolved during compilation that required reproducible
resolution due to caching. Now we can resolve as we go and support
all `tsconfig.paths`.
2022-02-25 15:43:58 -08:00
Dmitry Gozman
91672595f2
fix(reporters): normalize usage of isTTY, env.CI and debug env variables (#12295)
- `stdout.isTTY` controls whether list reporter updates lines or just adds them;
- `env.CI` is used in a few places to affect the defaults:
  - whether to open interactive html;
  - default reporter dot/line;
  - default terminal reporter added to non-terminal reporters;
- `env.PWTEST_SKIP_TEST_OUTPUT` is removed;
- `env.PW_TEST_DEBUG_REPORTERS` is introduced specifically for tests.
2022-02-24 12:39:28 -08:00
Andrey Lushnikov
5879c7f362
chore: refactor toMatchSnapshot once again (#12313)
Keep massaging code in preparation for `toHaveScreenshot`.

References #9938
2022-02-23 13:17:37 -08:00
Dmitry Gozman
4399623f9f
chore(test runner): make timeout a separate error in TestInfo.errors (#12315)
This way we control the timeout error message from the runner,
so that later on we can differentiate between test timeout, fixture
timeout and hook timeout.
2022-02-23 12:32:12 -08:00
Dmitry Gozman
114edecd3a
fix(list mode): keep outputDir intact (#12291) 2022-02-22 12:50:26 -08:00
Dmitry Gozman
d3c4323021
fix(test runner): improve error message for unexpected calls (#12240) 2022-02-18 18:25:18 -08:00
Dmitry Gozman
ee0dd6ec71
fix(test runner): trim full output path (#12239)
Previously, we only trimmed the "full title" component, but
we should also trim the "relative path" component.
2022-02-18 15:40:36 -08:00
Andrey Lushnikov
0682672242
chore: move comparator logic to playwright-core (#12232)
This will enable implementation of `toHaveScreenshot` on the
server-side.

Drive-by: drop blink-diff

References #9938
2022-02-18 14:39:17 -08:00
Andrey Lushnikov
b9e86b79ee
chore: refactor toMatchSnapshot implementation (#12206)
This patch prepares for the `toHaveScreenshot` implementation
by splitting common parts from `toMatchSnapshot`.

Drive-by: fix default extension generation from `.bin` to `.dat`
for unknown buffers.
2022-02-18 11:21:58 -08:00
Dmitry Gozman
34eec7eae7
fix(webServer): route output through reporters (#12198) 2022-02-18 07:54:01 -08:00
Andrey Lushnikov
a98babec69
feat(test-runner): introduce pixelCount and pixelRatio options (#12169)
This patch adds additional options to `toMatchSnapshot` method:
- `pixelCount` - acceptable number of pixels that differ to still
  consider images equal. Unset by default.
- `pixelRatio` - acceptable ratio of all image pixels (from 0 to 1) that differ to still
  consider images equal. Unset by default.

Fixes #12167, #10219
2022-02-17 15:44:03 -08:00
Andrey Lushnikov
086333cd60
feat(test-runner): support unnamed snapshots (#12161)
Fixes #9007
2022-02-16 14:22:01 -08:00
Yury Semikhatsky
7ee35ae30d
fix(html-report): open all test traces in one viewer (#12142) 2022-02-16 09:09:42 -08:00
Dmitry Gozman
85cb3c9713
test: add a test that fixture error after timeout is not a fatal error (#12141) 2022-02-15 18:05:20 -08:00
pierscowburn
5db7ce5964
fix: propagate exit code in experimental mode (#12070)
In experimental ESM mode a child process is forked in order to run the tests. Currently the exit code of this child process is not propagated to the exit code of the parent process, which means that the process exits with a status code of `0` even if some of the tests failed.

This makes it difficult to use Playwright in CI in experimental mode, as the CI pipeline as a whole will pass despite the test failures.

This change addresses this by propagating the exit code in the case where it is non-zero.
2022-02-15 13:10:35 -08:00
Andrey Lushnikov
ef21ce3f56
feat(test-runner): filter out syntax error stack traces (#12095)
Filter out long stack traces from babel when it fails compilation
due to syntax error in test.
2022-02-14 14:33:14 -08:00
Dmitry Gozman
9814c592d2
fix(test runner): shutdown redundant workers (#12062) 2022-02-14 10:57:15 -08:00
Pavel Feldman
96b5831a49
chore: add support private methods in TS (#12051) 2022-02-11 14:46:49 -08:00
Dmitry Gozman
d8db785c0a
fix(reporters): correctly handle missing stdout.columns (#12016)
When columns are not available, do not trim the output.
2022-02-11 08:33:56 -08:00
Ross Wollman
0d42c16a17
fix(test-runner): undefined body crash with manual attachments (#11995)
The new (as of 1.18) `async testInfo.attach(…)` API handles this
gracefully (and is part of the reason for the new API's existence).
However, for the foreseeable future, it's still possible to manually
push onto the attachments array where we can't validate the contents
until it's too late, so this change ensures more graceful handling in
that case.

Fixes #11565
2022-02-10 12:33:38 -08:00
Pavel Feldman
fbc05b74a7
chore: revert "fix(test-runner): escape backslashes in win cli (#11763)" (#12004) 2022-02-10 07:02:37 -08:00
Andrey Lushnikov
6904b3294e
fix(test-runner): fix browser initialization in test modifiers (#11984)
Fixes #11985
2022-02-09 15:30:14 -08:00
Dmitry Gozman
ae7c52154f
fix(fixtures): make sure connected browser respects context options (#11990)
Connected browser was wired up to the wrong browserType object.
2022-02-09 15:25:15 -08:00
Andrey Lushnikov
40b4218962
chore: roll stable test runner to Feb 9, 2022 (#11971) 2022-02-09 09:34:25 -08:00
Yury Semikhatsky
706c897031
feat(fixtures): respect tracing config for APIRequestContext (#11954) 2022-02-09 08:54:09 -08:00
Pavel Feldman
e9e5de2d57
feat(debug): allow preprocessing JS scripts as well (#11953) 2022-02-09 07:14:11 -08:00
Dmitry Gozman
19368e93af
feat(test runner): support connectOptions (#11919)
This allows to specify `connectOptions` in the config that
switch built-in `browser` to be remotely connected.
2022-02-08 20:45:42 -08:00