Andrey Lushnikov 6d82460a02 feat: implement a new image comparison function (#19166) ... This patch implements a new image comparison function, codenamed "ssim-cie94". The goal of the new comparison function is to cancel out browser non-determenistic rendering. To use the new comparison function: ```ts await expect(page).toHaveScreenshot({ comparator: 'ssim-cie94', }); ``` As of Nov 30, 2022, we identified the following sources of non-determenistic rendering for Chromium: - Anti-aliasing for certain shapes might be different due to the way skia rasterizes certain shapes. - Color blending might be different on `x86` and `aarch64` architectures. The new function employs a few heuristics to fight these differences. Consider two non-equal image pixels `(r1, g1, b1)` and `(r2, g2, b2)`: 1. If the [CIE94] metric is less then 1.0, then we consider these pixels **EQUAL**. (The value `1.0` is the [just-noticeable difference] for [CIE94].). Otherwise, proceed to next step. 1. If all the 8 neighbors of the first pixel match its color, or if the 8 neighbors of the second pixel match its color, then these pixels are **DIFFERENT**. (In case of anti-aliasing, some of the direct neighbors have to be blended up or down.) Otherwise, proceed to next step. 1. If SSIM in some locality around the different pixels is more than 0.99, then consider this pixels to be **EQUAL**. Otherwise, mark them as **DIFFERENT**. (Local SSIM for anti-aliased pixels turns out to be very close to 1.0). [CIE94]: https://en.wikipedia.org/wiki/Color_difference#CIE94 [just-noticeable difference]: https://en.wikipedia.org/wiki/Just-noticeable_difference		2022-12-02 15:22:05 -08:00
..
assets	Revert "feat: send SIGTERM to webserver before SIGKILL'ing it. (#18220)" (#18661)	2022-11-09 09:18:33 -08:00
stable-test-runner	chore: roll test runner to tot (#17351)	2022-09-14 19:51:22 -07:00
access-data.spec.ts	chore: align more ESLint rules with VSCode formatting (#16647)	2022-08-18 20:12:33 +02:00
babel.spec.ts	feat(ct): support cra w/ .js (#14326)	2022-05-24 19:43:28 -07:00
basic.spec.ts	feat(test runner): separate interrupted status from skipped (#16124)	2022-08-02 12:55:43 -07:00
command-line-filter.spec.ts	fix(pwtest): fix focused line with dirname specified (#18189)	2022-10-19 15:05:59 -07:00
config.spec.ts	Revert "chore: hide TestProject.setup and TestInfo.storage (#18800)" (#18836)	2022-11-16 11:23:10 -08:00
esm.spec.ts	fix(esm+tsconfig): allow mapped ts files in esm mode (#17862)	2022-10-05 18:00:49 -07:00
exit-code.spec.ts	feat: --pass-with-no-tests option (#16902)	2022-08-29 15:46:34 -07:00
expect-poll.spec.ts	feat(poll): expose custom poll interval (#13776)	2022-04-26 21:32:38 -07:00
expect-soft.spec.ts	chore(test-runner): rewrite poll, soft, wrap (#13335)	2022-04-05 17:47:35 -07:00
expect.spec.ts	test: unflake some timeout-dependent tests (#19139)	2022-11-29 12:54:53 -08:00
fixture-errors.spec.ts	feat(test runner): show the number of fatal errors at the end (#15975)	2022-07-28 14:46:21 -07:00
fixtures.spec.ts	fix(test runner): update timeout error messages (#15117)	2022-06-30 17:05:08 -07:00
gitignore.spec.ts	fix(runner): ignore .gitignore if testDir is explicitly configured (#14430)	2022-05-26 14:39:51 -07:00
global-setup.spec.ts	chore: remove TestProject.projectSetup (#16321)	2022-08-05 15:24:30 -07:00
golden.spec.ts	feat: implement a new image comparison function (#19166)	2022-12-02 15:22:05 -08:00
hooks.spec.ts	test: unflake some timeout-dependent tests (#19139)	2022-11-29 12:54:53 -08:00
list-mode.spec.ts	fix(list mode): print errors to stderr (#13016)	2022-03-24 07:33:33 -07:00
loader.spec.ts	fix(ts): resolve .js to .ts in non-ESM mode too (#18219)	2022-10-20 12:21:22 -07:00
match-grep.spec.ts	test(test-runner): add tests to show that grep is case insensitive (#8091)	2021-08-09 14:05:51 -04:00
max-failures.spec.ts	fix(test runner): show tests as interrupted when maxFailures stops them (#16178)	2022-08-03 15:25:25 -07:00
override-timeout.spec.ts	test(test-runner): unflake override-timeout.spec.ts (#7869)	2021-07-29 13:34:46 -07:00
playwright-test-fixtures.ts	feat(runner): allow filtering setup and tests (#18900)	2022-11-18 11:35:29 -08:00
playwright.artifacts.spec.ts	test: improve a few test runner tests that time out (#15287)	2022-06-30 17:03:01 -07:00
playwright.config.spec.ts	feat(api): introduce getByTestId (#17645)	2022-09-27 21:06:07 -07:00
playwright.config.ts	feat: implement a new image comparison function (#19166)	2022-12-02 15:22:05 -08:00
playwright.connect.spec.ts	chore: optionally connect instead of launching (#17174)	2022-09-07 15:57:20 -07:00
playwright.ct-build.spec.ts	core(ct): import paths (#18131)	2022-10-19 19:41:21 -07:00
playwright.ct-react.spec.ts	core(ct): import paths (#18131)	2022-10-19 19:41:21 -07:00
playwright.fetch.spec.ts	fix: stop har recording when APIRequestContext is disposed (#17007)	2022-08-31 21:51:38 -07:00
playwright.reuse.spec.ts	fix(reuse): stop pending operations upon reuse/disconnect (#18997)	2022-11-22 15:21:20 -08:00
playwright.spec.ts	chore: use consistent asLocator() in all logs (#18586)	2022-11-04 15:19:16 -07:00
playwright.trace.spec.ts	feat(test runner): add --trace cli option (#16277)	2022-08-05 09:20:39 -07:00
playwright.unhandled.spec.ts	feat(test runner): separate interrupted status from skipped (#16124)	2022-08-02 12:55:43 -07:00
project-setup.spec.ts	chore(runner): add all setup files in separate pass (#18978)	2022-11-22 16:22:48 -08:00
repeat-each.spec.ts	chore: enable object-curly-spacing in ESLint (#9168)	2021-09-27 18:58:08 +02:00
reporter-attachment.spec.ts	feat(testinfo): add name to attachment output name (#18440)	2022-11-03 13:54:51 -07:00
reporter-base.spec.ts	chore: do not point to node_modules in code frames (#18358)	2022-10-26 15:18:31 -07:00
reporter-dot.spec.ts	feat: support clarification message for expect (#11735)	2022-01-31 17:14:59 -08:00
reporter-github.spec.ts	feat(test-runner): support expect.soft (#11800)	2022-02-02 18:33:51 -08:00
reporter-html.spec.ts	feat(playwright-test): introduce snapshotPathTemplate configuration (#18568)	2022-11-09 15:29:07 -08:00
reporter-json.spec.ts	chore: add snippet to the json report (#17567)	2022-09-27 13:23:08 -07:00
reporter-junit.spec.ts	fix(junit): escape null control character (#18325)	2022-10-25 14:39:16 -07:00
reporter-line.spec.ts	Revert "fix(test runner): handle istty in line reporter (#15455)" (#15633)	2022-07-13 15:11:56 -07:00
reporter-list.spec.ts	chore: render steps in list reporter (#18269)	2022-10-24 15:54:53 -07:00
reporter-raw.spec.ts	feat(testinfo): add name to attachment output name (#18440)	2022-11-03 13:54:51 -07:00
reporter.spec.ts	chore: use consistent asLocator() in all logs (#18586)	2022-11-04 15:19:16 -07:00
resolver.spec.ts	feat(test runner): support jsconfig.json's baseUrl+paths (#19219)	2022-12-01 16:42:25 -08:00
retry.spec.ts	feat(test runner): test.describe.configure({ retries, timeout }) (#18388)	2022-10-27 15:53:27 -07:00
runner.spec.ts	chore(runner): add all setup files in separate pass (#18978)	2022-11-22 16:22:48 -08:00
shard.spec.ts	fix(shards): zero-sized shard should not include all tests (#19148)	2022-11-29 16:02:11 -08:00
snapshot-path-template.spec.ts	chore: rewrite error message for missing snapshot (#19104)	2022-11-29 12:51:15 -08:00
stdio.spec.ts	fix: support hasColors and getColorDepth in worker (#15590)	2022-07-12 14:47:14 -07:00
storage.spec.ts	Revert "chore: hide TestProject.setup and TestInfo.storage (#18800)" (#18836)	2022-11-16 11:23:10 -08:00
test-extend.spec.ts	Revert #18561 "feat(test runner): test.reset() to reset options to default/config value" (#18704)	2022-11-10 14:45:05 -08:00
test-grep.spec.ts	fix: --grep and --grep-invert should intersect with config (#17716)	2022-09-29 16:39:21 -07:00
test-ignore.spec.ts	chore: revert "fix(test-runner): escape backslashes in win cli (#11763)" (#12004)	2022-02-10 07:02:37 -08:00
test-info.spec.ts	feat: support clarification message for expect (#11735)	2022-01-31 17:14:59 -08:00
test-modifiers.spec.ts	chore: align more ESLint rules with VSCode formatting (#16647)	2022-08-18 20:12:33 +02:00
test-output-dir.spec.ts	feat(playwright-test): introduce snapshotPathTemplate configuration (#18568)	2022-11-09 15:29:07 -08:00
test-parallel.spec.ts	fix(parallel): minimize the number of beforeAll/afterAll hooks in parallel mode (#13138)	2022-03-28 16:10:32 -07:00
test-serial.spec.ts	fix(test runner): serial suites inside parallel suite should run in parallel (#15769)	2022-07-26 13:38:25 -07:00
test-step.spec.ts	feat: return value from step (#16060)	2022-07-29 15:16:07 -07:00
test-use.spec.ts	Revert #18561 "feat(test runner): test.reset() to reset options to default/config value" (#18704)	2022-11-10 14:45:05 -08:00
timeout.spec.ts	feat(test runner): test.describe.configure({ retries, timeout }) (#18388)	2022-10-27 15:53:27 -07:00
to-have-screenshot.spec.ts	feat: implement a new image comparison function (#19166)	2022-12-02 15:22:05 -08:00
types-2.spec.ts	feat(test runner): test.describe.configure({ retries, timeout }) (#18388)	2022-10-27 15:53:27 -07:00
types.spec.ts	Revert "chore: hide TestProject.setup and TestInfo.storage (#18800)" (#18836)	2022-11-16 11:23:10 -08:00
web-server.spec.ts	Revert "feat: send SIGTERM to webserver before SIGKILL'ing it. (#18220)" (#18661)	2022-11-09 09:18:33 -08:00
worker-index.spec.ts	feat(test runner): introduce TestInfo.parallelIndex (#9762)	2021-11-01 10:37:34 -07:00