Commit Graph

1099 Commits

Author SHA1 Message Date
Dmitry Gozman
3a009531b1
feat(hooks): add a step per hook (#12867)
It is now possible to see which hooks were run in the html report.
2022-03-17 19:33:01 -07:00
Pavel Feldman
c7d6f96328
chore: disambiguate internal flags (#12866) 2022-03-17 18:27:33 -07:00
Dmitry Gozman
25483452c0
feat(fixtures): per-fixture timeout (#12751)
By default, fixtures share timeout with the test they are instantiated for.
However, for more heavy fixtures, especially worker-scoped ones, it makes
sense to have a separate timeout.

This introduces `{ timeout: number }` option to the list of fixture options
that opts the fixture into a dedicated timeout rather than sharing it
with the test.
2022-03-17 09:36:03 -07:00
Max Schmitt
e231db1810
chore: restore expect.any()/expect.anything() (#12820) 2022-03-16 21:34:41 +01:00
Dmitry Gozman
f8c4cb3d24
fix(screenshot): do not stall on hideHighlight (#12764) 2022-03-15 14:13:45 -07:00
Andrey Lushnikov
88610c8b4c
fix: properly define apiName for web-first assertions (#12706)
Turns out relying on PWTRAP in stack is not reliable: depending on the
call structure, the stack might be cut unpredictably by Node.js.

This patch removes PWTRAP and instead plumbs explicit stack and
pre-set `apiName` all the way down to `wrapApiCall`.
2022-03-14 18:01:13 -07:00
Dmitry Gozman
237954212c
Revert "feat(line reporter): show stats, handle tty" (#12735)
This reverts commit be817d1a53, PR #12695.
Reason: found issues with stdout messing with stats.
2022-03-14 10:37:43 -07:00
Max Schmitt
adcd32fc6e
fix(expect): toHaveCSS with custom CSS properties (#12709) 2022-03-12 09:32:40 -08:00
Andrey Lushnikov
acae63c409
test: skip fonts test on Windows (#12708)
This test requires windows-specific test expectations. We'll use
it as a playground for the rebaseline workflow.

References #12707
2022-03-12 00:14:56 -08:00
Andrey Lushnikov
ef4b34fb7a
feat(toHaveScreenshot): enhance messaging in case of failures (#12677)
This patch:
- adds call logs to track screenshot timeouts, e.g. due to
  waiting for web fonts
- makes sure all snapshot expectations have `.png` extension
- throws a polite error when given a buffer or a string instead of a
  page or a locator
- removes stray NL between error description and call log
- makes sure `apiName` is always correct (and adds a test for it)
2022-03-11 22:40:28 -08:00
Dmitry Gozman
be817d1a53
feat(line reporter): show stats, handle tty (#12695)
- Line reporter now shows stats in addition to the test name:
  ```log
  [chromium] › page/page-click-react.spec.ts:108:1 › should not retarget when element changes on hover
  23% [21/93] Passed: 17 Flaky: 0 Failed: 0 Skipped: 4 (7s)
  ```

- When connected to a TTY or with `env.PLAYWRIGHT_LIVE_TERMINAL`
set to anything but `'0'` or `'false'`, line reporter updates in place.

- When not connected to a TTY, line reporter prints an update
after each ~1% of tests done, so it never prints more than 100 lines.

- Updated tests to the golden style.
2022-03-11 18:19:33 -08:00
Ross Wollman
72b95b9ffa
chore: dogfood dotenv (#12673)
Use a top-level .env file to control the internal testing setup.

This allows for easy manipulation of environment variables regardless
of your setup (VSCode Extension, CLI, etc.).
2022-03-11 15:12:25 -08:00
Andrey Lushnikov
95c774cf84
feat: support multiple screenshots in HTML report (#12577)
This patch adds support to multiple diffs. These are possible
due to soft assertions.

Drive-by: rename second screenshot in `toHaveScreenshot` failure when
re-generating expectations from "expected" to "previous".
2022-03-11 08:46:13 -08:00
Andrey Lushnikov
e3ab6388a9
feat(toHaveScreenshot): enable sensible defaults (#12675)
This patch:
- Enables configuration of certain defaults for some options of `expect.toHaveScreenshot` method via `TestProject.expect.toHaveScreenshot` property
- Sets sensible defaults for these options:
  * `fonts: "ready"`
  * `animations: "disabled"`
  * `size: "css"`
2022-03-11 08:45:36 -08:00
Andrey Lushnikov
1dc581c1a0
test: fix test that gave 2 pixel screenshot diff on Linux (#12671) 2022-03-10 20:26:31 -08:00
Andrey Lushnikov
b51157bdea
feat: bring back maxDiffPixels in toMatchSnapshot (#12627)
This patch reverts 2 commits that removed the feature from the method:
- "fix: explicitly ignore maxDiffPixels in toMatchSnapshot (#12570)"
  commit b8af8458d6.
- "chore: remove `maxDiffPixels` from toMatchSnapshot (#12539)"
  commit a3dff45974.
2022-03-10 18:41:16 -08:00
Andrey Lushnikov
42765804bc
chore: add "allow" as a default value for animations. (#12663) 2022-03-10 17:15:36 -08:00
Andrey Lushnikov
49e66c7f08
feat(screenshot): introduce new "fonts" option for screenshots (#12661)
This option will wait for webfonts to load before taking screenshots.
2022-03-10 16:54:36 -08:00
Andrey Lushnikov
12d8a262be
feat: support screenshotsDir option (#12642)
The `screenshotsDir` option controls the expectation storage
for `toHaveScreenshot()` function.

The new expectation management for screenshots has the following
key properties:
- All screenshots are stored in a single folder called `screenshotsDir`.
- Screenshot names **do not** respect `snapshotDir` and `snapshotSuffix`
  configurations.
- `screenshotsDir` is configurable per project. This way a "smoke tests"
  project can re-use screenshots from "all tests" project.
- Host platform is a top-level folder.

For example, given the following config:

```js
// playwright.config.ts
module.exports = {
  projects: [
    { name: 'Mobile Safari' },
    { name: 'Desktop Chrome' },
  ],
};
```

And the following test structure:

```
smoke-tests/
└── basic.spec.ts
```

Will result in the following screenshots folder structure by default:

```
__screenshots__/
└── darwin/
    ├── Mobile Safari/
    │   └── smoke-tests/
    │       └── basic.spec.ts/
    │           └── screenshot-expectation.png
    └── Desktop Chrome/
        └── smoke-tests/
            └── basic.spec.ts/
                └── screenshot-expectation.png
```
2022-03-10 16:50:26 -08:00
Pavel Feldman
4d3d0e2bda
chore: enable ts-esm support by default (#12658) 2022-03-10 14:48:33 -08:00
Dmitry Gozman
a388bb2302
feat(screenshot): size:'css'|'device' option (#12634)
With this experimental option, screenshot dimensions are in CSS pixels,
not physical device pixels, effectively ignoring the device scale factor.
2022-03-10 13:07:10 -08:00
Max Schmitt
90e76f9adb
test: rename #smoke to @smoke as test tags (#12652) 2022-03-10 19:42:52 +01:00
Max Schmitt
3f468abedd
fix(test-runner): ESM compatibly on Windows (#12615) 2022-03-10 19:27:25 +01:00
Max Schmitt
d805cd9097
chore: skip failing android tests 2 (#12647) 2022-03-10 18:46:50 +01:00
Andrey Lushnikov
75eef09c0d
feat: show expectation name as part of toHaveScreenshot title (#12612)
This patch adds snapshot file name as part of `toHaveScreenshot`
and `toMatchSnapshot` step title.
2022-03-09 20:09:45 -08:00
Dmitry Gozman
92430cc56c
chore(fixtures): do not force-close contexts in beforeAll (#12416)
This now happens the same way as for the test, in the worker cleanup
when browser closes, which is reported as a part of the test.
2022-03-09 07:26:46 -08:00
Dmitry Gozman
e8ce5d0258
feat(test runner): on beforeAll failure, precisely skip the tests (#12417)
Previously, we used to skip all the tests from the same file when
any `beforeAll` fails in the file.

Now, we only skip the rest of the tests affected by this particular
`beforeAll` and continue with other tests in the new worker.
2022-03-08 20:29:31 -08:00
Dmitry Gozman
e895bc2751
reland(test-runner): rely on test title paths instead of ordinal (#12611)
Fixes #11904

Co-authored-by: Andrey Lushnikov <lushnikov@chromium.org>
2022-03-08 19:06:25 -08:00
Dmitry Gozman
682eeb4ae0
reland(hooks): separate test timeout from beforeAll/afterAll timeouts (#12609)
This makes it possible to have longer `beforeAll`/`afterAll` and not
affect first/last test timeout.
2022-03-08 19:05:23 -08:00
Dmitry Gozman
62a032baa0
chore: remove env.PLAYWRIGHT_NO_LAYOUT_SHIFT_CHECK (#12605) 2022-03-08 16:42:39 -08:00
Dmitry Gozman
9cd704220c
reland(test runner): run hooks/modifiers as a part of the test (#12518)
Reland: worker.stop() before worker.run() was hanging because
`_runFinished` promise was not initially resolved.
---
This moves `beforeAll`, `afterAll` and some modifiers from running
as a separate entity into running inside a test.

Pros:
- All errors are reported as test errors.
- All artifacts are collected as test artifacts.
- Reporters support this out of the box.

Details:
- Each test computes the necessary hooks to run and runs them.
- Teardown is usually performed during the test (on test failure or worker stop).
- `skipRemaining` is added to `DonePayload` to preserve the behavior
  where `beforeAll` hook failure skips subsequent tests.
  This behavior can now be improved to only target tests affected by this hook.
2022-03-08 16:35:14 -08:00
Andrey Lushnikov
b8af8458d6
fix: explicitly ignore maxDiffPixels in toMatchSnapshot (#12570)
Fixes #12564
2022-03-07 16:55:35 -08:00
Andrey Lushnikov
e8ad7eb3f2
browser(firefox): roll Firefox to M97 (#12574)
Firefox 98 doesn't work on Ubuntu aarch64 yet.

https://github.com/microsoft/playwright/issues/12557
2022-03-07 16:16:42 -08:00
Dmitry Gozman
d836ed41d3
feat(remote): send 'x-playwright-browser' header (#12556) 2022-03-07 10:30:53 -08:00
github-actions[bot]
702536d962
feat(webkit): roll to r1616 (#12541)
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Yury Semikhatsky <yurys@chromium.org>
2022-03-06 12:22:47 +01:00
Andrey Lushnikov
d7648d390c
fix(toHaveScreenshot): round ratio up (#12540)
This way the ratio value could be used right away in configuation.
2022-03-04 18:55:48 -08:00
Andrey Lushnikov
a3dff45974
chore: remove maxDiffPixels from toMatchSnapshot (#12539)
References #12441
2022-03-04 18:27:04 -08:00
Andrey Lushnikov
4953fc4845
chore: introduce toHaveScreenshot configuration options (#12507)
References https://github.com/microsoft/playwright/issues/12441
2022-03-04 17:30:43 -08:00
Ross Wollman
281945c8b6
test: typo fix (#12531) 2022-03-04 17:09:18 -08:00
Karan Shah
fd1a1a2b1c
feat(android): Adding custom port parameter to connect to different adb server port (#12220) 2022-03-04 20:27:25 +01:00
Andrey Lushnikov
b45c788a33
fix(disable-animations): rename boolean to enum (#12503)
Enums are more flexible in the long run since they let us easily
extend API.

References #12441
2022-03-04 11:02:59 -08:00
Andrey Lushnikov
1f5cfcaeec
fix(to-have-screenshot): show number of bad pixels in mismatched error (#12502)
Drive-by: do not show empty call log.

References #12441
2022-03-04 11:01:05 -08:00
Andrey Lushnikov
279e579097
chore: address pixelCount and pixelRatio review comments (#12501)
- documentation fixes
- rename pixelCount to maxDiffPixels
- rename pixelRatio to maxDiffPixelRatio
2022-03-03 23:17:31 -08:00
Pavel Feldman
34bf0d4b87
feat(test-runner): allow grepping in project (#12469) 2022-03-02 15:02:35 -08:00
Max Schmitt
cfcdc3e19c
chore: roll stable test-runner to ToT (#12472) 2022-03-02 23:47:55 +01:00
Max Schmitt
cb5b4cf06b
test: skip highlight tests on != default mode (#12471) 2022-03-02 23:46:33 +01:00
Max Schmitt
b2e3357613
chore: throw error if setCookies expires value is not valid (#12470) 2022-03-02 23:10:40 +01:00
Yury Semikhatsky
608873e945
test: inputValue works for label selector (#12447) 2022-03-02 09:33:44 -08:00
Yury Semikhatsky
2d7ec26dc2
fix(fetch): send secure cookies for http://localhost requests (#12450) 2022-03-02 09:33:30 -08:00
Pavel Feldman
4b19d59ec5
feat(test): introduce fully parallel mode (#12446) 2022-03-01 18:12:21 -08:00