This patch:
- starts using directory of `package.json` to resolve default
output directory path
- starts using either `package.json` directory or configuration
directory to resolve all relative paths
References #12970
Resolves#11318.
* Adds `TestConfig.attachments` public API. (We opted to not implement an analog to the async `TestInfo.attach(…)` API.)
* Adds `TestConfig.attachments` to common reporters.
* Dogfoods some git and CI-info inference to generate useful atttachments
* Updates HTML Reporter to include a side bar to present a pre-defined set of attachments (a.k.a git/commit context sidebar)
Here's what it looks like:
<img width="1738" alt="Screen Shot 2022-03-21 at 3 23 28 PM" src="https://user-images.githubusercontent.com/11915034/159373291-8b937d30-fba3-472a-853a-766018f6b3e2.png">
See `tests/playwright-test/reporter-html.spec.ts` for an example of usage (for dogfood-ing only). In the future, if this becomes user-facing, there the Global Setup bit would likely become unnecessary (as would interaction with attachments array); there would likely just be a nice top-level config and/or CLI flag to enable collecting of info.
This patch aligns the strategies that are used to generate new
screnshot expectations and to compare screenshot expectations against
baseline.
With this patch, `toHaveScreenshot` will:
- when generating a new expectation: will wait for 2 consecutive
screenshots to match and accept the last one as expectation.
- when given an expectation:
* will compare first screenshot against expectation. If matches,
resolve successfully
* if first screenshot doesn't match, then wait for 2 consecutive
screenshots to match and then compare last screenshot with the
expectation.
An example of a new detailed call log:
```
1) a.spec.ts:3:1 › should work ===================================================================
Error: Screenshot comparison failed:
20000 pixels (ratio 0.03 of all image pixels) are different
Call log:
- expect.toHaveScreenshot with timeout 5000ms
- verifying given screenshot expectation
- fast-path: checking first screenshot to match expectation
- taking page screenshot
- disabled all CSS animations
- waiting for fonts to load...
- fonts in all frames are loaded
- fast-path failed: first screenshot did not match expectation - 20000 pixels (ratio 0.03 of all image pixels) are different
- waiting for 2 consecutive screenshots to match
- waiting 100ms before taking screenshot
- taking page screenshot
- disabled all CSS animations
- waiting for fonts to load...
- fonts in all frames are loaded
- 2 consecutive screenshots matched
- final screenshot did not match expectation - 20000 pixels (ratio 0.03 of all image pixels) are different
- 20000 pixels (ratio 0.03 of all image pixels) are different
Expected: /Users/andreylushnikov/tmp/test-results/a-should-work/should-work-1-expected.png
Received: /Users/andreylushnikov/tmp/test-results/a-should-work/should-work-1-actual.png
Diff: /Users/andreylushnikov/tmp/test-results/a-should-work/should-work-1-diff.png
3 | test('should work', async ({ page }) => {
4 | await page.goto('file:///Users/andreylushnikov/prog/playwright/tests/assets/rotate-z.html');
> 5 | await expect(page).toHaveScreenshot();
| ^
6 | });
7 |
```
By default, fixtures share timeout with the test they are instantiated for.
However, for more heavy fixtures, especially worker-scoped ones, it makes
sense to have a separate timeout.
This introduces `{ timeout: number }` option to the list of fixture options
that opts the fixture into a dedicated timeout rather than sharing it
with the test.
Turns out relying on PWTRAP in stack is not reliable: depending on the
call structure, the stack might be cut unpredictably by Node.js.
This patch removes PWTRAP and instead plumbs explicit stack and
pre-set `apiName` all the way down to `wrapApiCall`.
This patch:
- adds call logs to track screenshot timeouts, e.g. due to
waiting for web fonts
- makes sure all snapshot expectations have `.png` extension
- throws a polite error when given a buffer or a string instead of a
page or a locator
- removes stray NL between error description and call log
- makes sure `apiName` is always correct (and adds a test for it)
- Line reporter now shows stats in addition to the test name:
```log
[chromium] › page/page-click-react.spec.ts:108:1 › should not retarget when element changes on hover
23% [21/93] Passed: 17 Flaky: 0 Failed: 0 Skipped: 4 (7s)
```
- When connected to a TTY or with `env.PLAYWRIGHT_LIVE_TERMINAL`
set to anything but `'0'` or `'false'`, line reporter updates in place.
- When not connected to a TTY, line reporter prints an update
after each ~1% of tests done, so it never prints more than 100 lines.
- Updated tests to the golden style.
Use a top-level .env file to control the internal testing setup.
This allows for easy manipulation of environment variables regardless
of your setup (VSCode Extension, CLI, etc.).
This patch adds support to multiple diffs. These are possible
due to soft assertions.
Drive-by: rename second screenshot in `toHaveScreenshot` failure when
re-generating expectations from "expected" to "previous".
This patch:
- Enables configuration of certain defaults for some options of `expect.toHaveScreenshot` method via `TestProject.expect.toHaveScreenshot` property
- Sets sensible defaults for these options:
* `fonts: "ready"`
* `animations: "disabled"`
* `size: "css"`
This patch reverts 2 commits that removed the feature from the method:
- "fix: explicitly ignore maxDiffPixels in toMatchSnapshot (#12570)"
commit b8af8458d6.
- "chore: remove `maxDiffPixels` from toMatchSnapshot (#12539)"
commit a3dff45974.
The `screenshotsDir` option controls the expectation storage
for `toHaveScreenshot()` function.
The new expectation management for screenshots has the following
key properties:
- All screenshots are stored in a single folder called `screenshotsDir`.
- Screenshot names **do not** respect `snapshotDir` and `snapshotSuffix`
configurations.
- `screenshotsDir` is configurable per project. This way a "smoke tests"
project can re-use screenshots from "all tests" project.
- Host platform is a top-level folder.
For example, given the following config:
```js
// playwright.config.ts
module.exports = {
projects: [
{ name: 'Mobile Safari' },
{ name: 'Desktop Chrome' },
],
};
```
And the following test structure:
```
smoke-tests/
└── basic.spec.ts
```
Will result in the following screenshots folder structure by default:
```
__screenshots__/
└── darwin/
├── Mobile Safari/
│ └── smoke-tests/
│ └── basic.spec.ts/
│ └── screenshot-expectation.png
└── Desktop Chrome/
└── smoke-tests/
└── basic.spec.ts/
└── screenshot-expectation.png
```
Previously, we used to skip all the tests from the same file when
any `beforeAll` fails in the file.
Now, we only skip the rest of the tests affected by this particular
`beforeAll` and continue with other tests in the new worker.
Reland: worker.stop() before worker.run() was hanging because
`_runFinished` promise was not initially resolved.
---
This moves `beforeAll`, `afterAll` and some modifiers from running
as a separate entity into running inside a test.
Pros:
- All errors are reported as test errors.
- All artifacts are collected as test artifacts.
- Reporters support this out of the box.
Details:
- Each test computes the necessary hooks to run and runs them.
- Teardown is usually performed during the test (on test failure or worker stop).
- `skipRemaining` is added to `DonePayload` to preserve the behavior
where `beforeAll` hook failure skips subsequent tests.
This behavior can now be improved to only target tests affected by this hook.
* Revert "fix(hooks): separate test timeout from beforeAll/afterAll timeouts (#12413)"
This reverts commit 73dee69558.
* Revert "fix(test-runner): rely on test title paths instead of ordinal (#12414)"
This reverts commit d744a87aee.
* Revert "chore(test runner): run hooks/modifiers as a part of the test (#12329)"
This reverts commit 47045ba48d.
chore(test runner): run hooks/modifiers as a part of the test
This moves `beforeAll`, `afterAll` and some modifiers from running
as a separate entity into running inside a test.
Pros:
- All errors are reported as test errors.
- All artifacts are collected as test artifacts.
- Reporters support this out of the box.
Details:
- Each test computes the necessary hooks to run and runs them.
- Teardown is usually performed during the test (on test failure or worker stop).
- `skipRemaining` is added to `DonePayload` to preserve the behavior
where `beforeAll` hook failure skips subsequent tests.
This behavior can now be improved to only target tests affected by this hook.
