<img width="1676" alt="Screenshot 2022-12-14 at 2 05 29 PM"
src="https://user-images.githubusercontent.com/11915034/207726274-167d9ee7-c01b-4379-aff1-37dad979d23b.png">
Flakiness dashboard reveals on Window's variants of Chromium, the
decimal for actual vs. expected was slightly off (e.g. 1705568002.677066
vs. 1705568002.674173).
This test fix assumes it's acceptable to suppress the roundtrip floating
point difference (which appears to be constrained to Windows).
Signed-off-by: Ross Wollman <rwoll@users.noreply.github.com>
- `<loopback>` for local interfaces;
- `123.123.123.123` for IPv4;
- `[1:2:3:4:5:6:7:8]` for IPv6;
- `*.example.com` and `.example.com` for subdomains;
- `example.com` for domains;
- `anything:3000` for port matching.
This patch adds a reverse proxy in front of novnc and playwright
server inside the container.
As a result:
- container exposes a single HTTP port to the host
- all Playwright clients can connect using this exposed port, e.g.
`http://127.0.0.1:5400`
- navigating to the exposed port in the browser lands on a nice HTML
page
with a link to view container screen
- users can also manually navigate to `http://127.0.0.1:5400/screen` to
view screen
This patch adds a grid border around the image so that the SSIM
resolution doesn't drop for the border pixels.
We also add a test with WebKit rendering artifacts to make sure
new approach helps to fight this.
While migrating Playwright tests from pixelmatch comparator to the
ssim-cie94 comparator, we noticed a rendering artifact that happens
on playwright screenshot tests on WebKit Linux.
This PR adds a set of cropped-down examples and tunes the ssim-cie94
comparator to accept these rendering artifacts.
This patch implements a new image comparison function, codenamed
"ssim-cie94". The goal of the new comparison function is to cancel out
browser non-determenistic rendering.
To use the new comparison function:
```ts
await expect(page).toHaveScreenshot({
comparator: 'ssim-cie94',
});
```
As of Nov 30, 2022, we identified the following sources of
non-determenistic rendering for Chromium:
- Anti-aliasing for certain shapes might be different due to the
way skia rasterizes certain shapes.
- Color blending might be different on `x86` and `aarch64`
architectures.
The new function employs a few heuristics to fight these
differences.
Consider two non-equal image pixels `(r1, g1, b1)` and `(r2, g2, b2)`:
1. If the [CIE94] metric is less then 1.0, then we consider these pixels
**EQUAL**. (The value `1.0` is the [just-noticeable difference] for
[CIE94].). Otherwise, proceed to next step.
1. If all the 8 neighbors of the first pixel match its color, or
if the 8 neighbors of the second pixel match its color, then these
pixels are **DIFFERENT**. (In case of anti-aliasing, some of the
direct neighbors have to be blended up or down.) Otherwise, proceed
to next step.
1. If SSIM in some locality around the different pixels is more than
0.99, then consider this pixels to be **EQUAL**. Otherwise, mark them
as **DIFFERENT**. (Local SSIM for anti-aliased pixels turns out to be
very close to 1.0).
[CIE94]: https://en.wikipedia.org/wiki/Color_difference#CIE94
[just-noticeable difference]:
https://en.wikipedia.org/wiki/Just-noticeable_difference