This patch implements a new image comparison function, codenamed
"ssim-cie94". The goal of the new comparison function is to cancel out
browser non-determenistic rendering.
To use the new comparison function:
```ts
await expect(page).toHaveScreenshot({
comparator: 'ssim-cie94',
});
```
As of Nov 30, 2022, we identified the following sources of
non-determenistic rendering for Chromium:
- Anti-aliasing for certain shapes might be different due to the
way skia rasterizes certain shapes.
- Color blending might be different on `x86` and `aarch64`
architectures.
The new function employs a few heuristics to fight these
differences.
Consider two non-equal image pixels `(r1, g1, b1)` and `(r2, g2, b2)`:
1. If the [CIE94] metric is less then 1.0, then we consider these pixels
**EQUAL**. (The value `1.0` is the [just-noticeable difference] for
[CIE94].). Otherwise, proceed to next step.
1. If all the 8 neighbors of the first pixel match its color, or
if the 8 neighbors of the second pixel match its color, then these
pixels are **DIFFERENT**. (In case of anti-aliasing, some of the
direct neighbors have to be blended up or down.) Otherwise, proceed
to next step.
1. If SSIM in some locality around the different pixels is more than
0.99, then consider this pixels to be **EQUAL**. Otherwise, mark them
as **DIFFERENT**. (Local SSIM for anti-aliased pixels turns out to be
very close to 1.0).
[CIE94]: https://en.wikipedia.org/wiki/Color_difference#CIE94
[just-noticeable difference]:
https://en.wikipedia.org/wiki/Just-noticeable_difference
The behavior regarding filters (both in config, command line and .only)
is the following:
- if some of tests match and none of setup match then we'll run all
setup files and all matching tests
- otherwise the filters apply to setup files the same way as to regular
tests
Running `npx playwright test file:123` will have the following behavior
- if only test files match then only matching subset of tests will run
but all setup files will run as well
- if only setup files match the filter then only those setup tests will
run
- if both setup and test files match an error will be thrown
This patch:
- updates documentation to lead users from `TestConfig.snapshotDir` and
`testInfo.snapshotSuffix` to `TestConfig.snapshotPathTemplate` as a
better and more flexible alternative.
- drops `{snapshotSuffix}` from documentation
- stops using `snapshotSuffix = ''` in our own tests and switches us
to the `snapshotPathTemplate`.
- adds `{testName}` token.
- Use `snapshotPathTemplate` for docker screenshots in html-reporter
- Mark the snapshot path template test as slow since it re-spawns
worker for each project.
- Fix docker smoke tests
This configuration option allows to set a string with template
values for precise control over snapshot path location.
An example of `snapshotPathTemplate` usage:
```ts
// playwright.config.ts
// Notice the `testDir` configuration!
export default {
testDir: './tests',
snapshotPathTemplate: './__screenshots__/{platform}/{projectName}/{testFilePath}/{arg}{ext}',
}
```
Currently supported "magic tokens" inside the `snapshotPathTemplate`
are:
- `{testDir}` - project's `testDir`
- `{snapshotDir}` - project's `snapshotDir`
- `{platform}` - `process.platform`
- `{projectName}` - Project's sanitized name
- `{testFileDir}` - Directories in relative path from `testDir` to test
file path (e.g. `page/` in the example below)
- `{testFileName}` - Test file name (with extension) (e.g.
`page-click.spec.ts` in the example below)
- `{testFilePath}` - Relative path from `testDir` to test file path
(e.g. `page/page-click.spec.ts` in the example below)
- `{ext}` - snapshot extension (with dots)
- `{arg}` - joined snapshot name parts, without extension (e.g.
`foo/bar/baz` in the example below)
- `{snapshotSuffix}` - `testInfo.snapshotSuffix` value.
Consider the following file structure:
```
playwright.config.ts
tests/
└── page/
└── page-click.spec.ts
```
The following `page-click.spec.ts`:
```ts
// page-click.spec.ts
import { test, expect } from '@playwright/test';
test('should work', async ({ page }) => {
await expect(page).toHaveScreenshot(['foo', 'bar', 'baz.png']);
});
```
Fixes#7792
Per discussion in #12950, adding sanitized name to the output filename
prefix. This can make debugging easier, and the filename structure more
human friendly.
We have to reliably know whether test timed out or not, and soft expect
error could have marked it with `status=failed` but it would still time
out. Now we have a separate `_didTimeout` flag for this.
Fixes#18023.
