fix(test runner): more firendly test duration
Reported test duration now does not include time spent in
`beforeAll`, `afterAll` and fixtures that have a separate timeout.
This is to avoid different reported test execution time,
depending on the test execution order.
Also reverted screenshot test changes from 4de14e7 as the breaking change was reverted upstream in Revert [251706@main] getComputedStyle(img).height returns string of a rounded int not a float WebKit/WebKit#1902
Fixes#8206.
Since #8206 is a long-awaited (~ 1 year old), popular (~ 45 reactions, frequently requested in community channels, etc.), this PR aims to unblock folks.
Notably, we do not innovate on the `webServer` API, despite knowing we're not in love with it. We'll save the innovation for either Plugins or a new `LaunchConfigs` option. (We haven't yet arrived at a Plugin API we like, and instead of launching a new option guessing what the "better" launchConfig API would be, let's wait and see how folks use this new Array-variant of `webServer` which—despite its name—can be used for non-Web Server launches!
When running without tty, line reporter outputs a line for each
percent of the tests, thus limiting the output to ~100 lines.
In addition, reporters now support PLAYWRIGHT_LIVE_TERMINAL
env variable to force tty mode.
Useful to set some options without coming up with an artificial name.
```js
test.describe(() => {
test.use({ colorScheme: 'dark' });
test('my test', () => {});
})
```
Currently, it is possible to run a function, e.g. a second `beforeEach` hook,
that will receive `null` for the fixture that has already failed before.
This PR skips running any user function that uses a fixture that has already
failed, just like if the fixture would be initialized again and failing for
the second time.
The following options now work across languages:
- `recordHar`
- `serviceWorkers`
In addition, object properties are now sorted alphabetically.
Drive-by: fixed `--target` help message to include all available targets.
Currently, if `text.fixme()` or `test.skip()` is used within a test, we
add a `fixme` or `skip` annotation. However, if the wrapper style is
used:
```
test.fixme('should work', () => {…})
```
the annotations were missing. This change adds annotations for the
above.
These annotations are important for reporting purposes and knowing
exactly what flavor of "skipped" was used.
Fixes#15239.
- Never use open shadow root for highlight. This messes up
our selectors that accidentally match internal preview elements.
- Remove failing electron test that we do not care about.
- Skip `channels.spec.ts` in non-default mode.
Previously, we only validated/converted on the way to the server,
but not from the server.
Validating both ways catches issues earlier, and allows us to
perform automatic conversions, for example only converting
buffers to base64 when sending over wire.
Some of the current timeout error messages are confusing, because they do not suggest that the issue is most likely a slow test. This PR updates timeout messages as follows:
- Test timeout of 30000ms exceeded.
- Test timeout of 30000ms exceeded while setting up "browser".
- Test timeout of 30000ms exceeded while tearing down "context".
- Test timeout of 30000ms exceeded while setting up "playwright configuration".
- Test timeout of 30000ms exceeded while running "beforeEach" hook.
- Test timeout of 30000ms exceeded while running "afterEach" hook.
- "beforeAll" hook timeout of 30000ms exceeded.
- "afterAll" hook timeout of 30000ms exceeded.
- Worker teardown timeout of 30000ms exceeded.
- "skip" modifier timeout of 30000ms exceeded.
- Fixture "myCustomFixture" timeout of 5000ms exceeded.
React takes time to render on overbooked machines. Let's use web-first assertions to test that they work as expected.
Error: expect(received).toBe(expected) // Object.is equality
Expected: 1
Received: 0
at /home/runner/work/playwright/playwright/tests/page/selectors-react.spec.ts:34:71
<img width="887" alt="image" src="https://user-images.githubusercontent.com/17984549/176618419-7653e5d8-4853-4618-af3b-2698827879d2.png">
Previously, screenshot was taken after hooks and fixtures teardown.
However, hooks can easily modify the state of the page, and
screenshot would not reflect the moment of failure.
Instead, we take screenshots immediately after the test function
finishes with an error.
These mean "I don't want to specify this fixture/option"
instead of "I want the value of undefined", aligned with how TypeScript works.
We already do similar things in the config.
fix(network): make allHeaders wait until all header are available
Before, calling `allHeaders()` from `page.on('request')` would yield
provisional headers instead.
With these changes:
- In Firefox, all headers are available immediately.
- In Chromium, all headers are available upon requestWillBeSentExtraInfo.
- In WebKit, all headers are available upon responseReceived.
- In all browsers, intercepted requests use "provisional" headers
as all headers, since there is no network stack to change the headers.
Drive-by: migrated Chromium to `hasExtraInfo` flags that simplifies
the logic quite a bit.
When target element is inside a non-main frame, there could be an
overlay in some of the parent frames that intercepts pointer events.
However, we never detected this case.
Fixes#14904.
This is done to make looking at the raw contents of the report
friendlier when using a file browser. However, it should be noted, the
public API of the HTML Reporter makes no guarantees of its contents
structure/layout/naming-conventions.
Currently, `loadstate` and `load` are two separate events in the protocol,
and are fired in this order. As a result, `waitForLoadState()` sometimes
resolves before the `'load'` event is fired, which is unexpected.
Also fixes a flaky test that assumed `load` event comes after `domcontentloaded`
for the empty page, which is not always a case in Chromium.
- `har` option is now an object `{ path, fallback }`.
- Allows falling back to `abort()`, `continue()` or throwing.
- Matches based on url + method.
- Follows redirects in the HAR file.
- Nice error/stack when throwing.
- Tests.
Adds cross-browser support for easily allowing/blocking Service Workers via a Context option.
Includes plumbing for Playwright Test's `use`.
Resolves#14522.
Relates #1090.
Supercedes #14321.
`npx playwright open --save-har=file.har` produces a HAR file that
can be later inspected or used for mocks.
`--save-har-glob="**/*.{png,woff2}"` option can be used to
produce a smaller har that only contains entries matching the glob.
Language override behavior changed upstream in WebKit/WebKit@039ebd9
New logic is closer to the actual behavior of WebKit on macOS, meaning that when the user changes system language the actual locale changes according to some weird OS rules:
ru-RU => navigator.language === 'ru'
fr-CH => navigator.language === 'fr-FR'
es-MX => navigator.language === 'es-MX'
Our locale emulation is aligned with that, so setting locale to fr-CH will result in fr-FR etc.
This restores the old hit target check, in addition to the new
hit target interceptor.
This way, we got some coverage for iframes and other quirky cases,
but keep the bullet-proof hit target check in place.
If the tests are in an explicitly configured testDir (either at the global config level or per project) .gitignore filters are not applied.
Fixes#14381
Fixes focus and blur management when `page.locator(…).focus()` and `page.locator(…).type(…)` are used which was regressed by 7a5b070 (#13510).
#13510 relied on an implicit assumption that this (conditional) [`blur`](7a5b070e95/packages/playwright-core/src/server/injected/injectedScript.ts (L672)) call would always be followed by a call that resulted in a newly focused element via this [`focus`](7a5b070e95/packages/playwright-core/src/server/injected/injectedScript.ts (L674)) call.
However, some elements are [not focusable](https://html.spec.whatwg.org/multipage/interaction.html#focusable-area), so we were blurring incorrectly, and losing focus that we should have maintained.
Two regression tests were added that pass on the commit prior to 7a5b070e95 (and match manual testing/expectations):
* `page.locator(…).focus()`: _keeps focus on element when attempting to focus a non-focusable element_
* `page.locator(…).type(…)`: _should type repeatedly in input in shadow dom_
Additionally, a third test (_should type repeatedly in input in shadow dom_) was added to check the invariant from #13510 that states:
> This affects [contenteditable] elements, but not input elements.
and allows us to introduce the targeted fix (contenteditble check before blur) without breaking FF again.
And _should type repeatedly in contenteditable in shadow dom with nested elements_ was added to ensure the above fix works with nest contenteditble detection.
Fixes#14254.
There are a few issues this covers:
- Some fixtures like `page` and `context` are not allowed in `beforeAll`
hooks, so using them in automatic fixture makes it throw.
- Running automatic fixture solely for `afterAll` is unexpected.
This currently happens when `afterAll` is run for cleanup after
fixture timeout/throw.
For built-in playwright fixture, we keep `'all-hooks-included'` auto mode.
Added a doc explaining the execution order.
Note: all toHaveScreenshot tests still use `__screenshots__` directory
for their expectations. One more test was added to make sure that
by default, `toHaveScreenshot` uses old snapshots.
This fixes a few issues:
- strict mode was producing false negatives if multiple query paths
lead to the same element being picked;
- in some cases the number of intermediate items in the list was
exponential and crashed quickly.
What changed:
- `visible` engine is a real engine now;
- `capture` selectors are transformed to `has=` selectors for
easier implementation;
- chained querying switched from a list to a set to avoid
exponential size.
We skip waiting for "visible" state that enforces non-zero size.
Other invisible conditions like "display:none" fail during the
actual "scrolling" step and will retry.
feat: support experimental doc entries
- Params/options/members are marked as experimental in the docs.
- `experimental.d.ts` is generated that contains all types and
includes experimental features.
- `experimental.d.ts` is references in our tests so that we
can test experimental features.
- `fonts` option is restored as experimental.
- Always show a fixture that was running during timeout.
- Give custom titles to built-in fixtures.
- Specify setup/teardown fixture phase in the message.
- Split connect vs launch browser fixtures for better naming.
Example timeout message:
```log
Timeout of 2000ms exceeded while running fixture "built-in playwright configuration" teardown.
```
Firefox has a bug: calling `node.focus()` does make the node focused,
but some internal "current contenteditable node" is not changed.
Blurring the previous one and focusing the new one helps.
This is a speculative fix to leftover tmp directories.
When users issues SIGINT twice, we enter `gracefullyClose()`
twice, and shortcut the second time. It turns out, we do
not wait for directories removal.
Note: it is unknown how often we reach this codepath in practice.
A few changes:
- `Matchers<R, T>` now carries both return and argument type.
- Based on the argument type, we apply playwright-specific Page/Locator matchers.
- Return type is usually void, unless wrapped with `expect.resolves`,
`expect.rejects` or `expect.poll()`.
- To preserve compatibility with any extended types in the wild,
argument type is optional.
If you typo'd an `expect` property, you got a cryptic error message:
```
Uncaught TypeError: Cannot create proxy with a non-object as target or handler
```
Now we get this nice friendly message:
```
1) a.spec.ts:6:9 › explodes ======================================================================
Error: expect: Property 'toBeLessThen' not found.
Did you mean 'toBeLessThan'?
See https://playwright.dev/docs/test-assertions for available options and documentation.
5 | const { test } = pwt;
6 | test('explodes', () => {
> 7 | expect.soft(1).toBeLessThen();
| ^
8 | });
9 |
```
Fixes#13218
- This supports `role=button[name=Hello]` similarly to CSS selectors.
- Does not change `_react` or `_vue` behavior that insist on quoting the string.
- Uses CSS notion of "identifier" characters.
It turns out that "non stalling evaluate" can stall in Chromium
in some weird conditions, like `document.open` after some weird
`iframe.src` value.
We now only hide highlight in those frames where we did install
highlight in the first place.
- Added docs to `selectors.md`.
- `[pressed]` and `[checked]` do not match `"mixed"` states.
- Disallow `[name]` shorthand without a value.
- Renamed `includeHidden` to `include-hidden`.
Previously, any unpaired quote in the text selector "escaped"
everything till the end of the selector string, and so any
subsequent chained selectors, including ">>" separator were ignored.
An example of misbehaving selector: `text=19" >> nth=1`.
Now, when text selector contains a non-leading quote, selector parser
does not assume it should escape ">>" separator and correctly
tokenizes all selectors from the chain.
Note that this behavior is a workaround for the fact that our
text selectors is somewhat poorly defined in this area. That said,
this workaround seems to be safe enough. It still does not work for
unpaired leading quotes like this: `text="19 >> nth=1`.
This has two values:
- `"hide"` to hide input caret for taking screenshot
- `"initial"` to keep caret behavior unchanged
Defaults to `"hide"`.
Fixes#12643
Previously, we always formed groups consisting of a single test.
Now, we group tests that share `beforeAll`/`afterAll` hooks into
`config.workers` equally-sized groups.
Closes#13131.
Per the visibility spec on https://playwright.dev/docs/next/actionability#visible:
> Element is considered visible when it has non-empty bounding box and does not have visibility:hidden computed style. Note that elements of zero size or with display:none are not considered visible.
✅ non-empty bounding box
✅ does not have visibility:hidden
Given the above conditions are satisfied, the locator is considered visible.
https://github.com/microsoft/playwright/issues/8740 proposes something like `isInViewport()` that would be better suited for checking if an element is offscreen.
This introduces `role=button[name="Click me"][pressed]` attribute-style
role selector. It is only available under `env.PLAYWRIGHT_EXPERIMENTAL_FEATURES`.
Supported attributes:
- `role` is required, for example `role=button`;
- `name` is accessible name, supports matching operators and regular expressions:
`role=button[name=/Click(me)?/]`;
- `checked` boolean/mixed, for example `role=checkbox[checked=false]`;
- `selected` boolean, for example `role=option[selected]`;
- `expanded` boolean, for example `role=button[expanded=true]`;
- `disabled` boolean, for example `role=button[disabled]`;
- `level` number, for example `role=heading[level=3]`;
- `pressed` boolean/mixed, for example `role=button[pressed="mixed"]`;
- `includeHidden` - by default, only non-hidden elements are considered.
Passing `role=button[includeHidden]` matches hidden elements as well.
This patch:
- starts using directory of `package.json` to resolve default
output directory path
- starts using either `package.json` directory or configuration
directory to resolve all relative paths
References #12970
Supports inline regex in addition to string: `_react=BookItem[author = /Ann?a/i]`.
This is similar to `text=` selector, but applies to `_react` and `_vue`
selectors. In the future, will also apply to `role=` selector.
Resolves#11318.
* Adds `TestConfig.attachments` public API. (We opted to not implement an analog to the async `TestInfo.attach(…)` API.)
* Adds `TestConfig.attachments` to common reporters.
* Dogfoods some git and CI-info inference to generate useful atttachments
* Updates HTML Reporter to include a side bar to present a pre-defined set of attachments (a.k.a git/commit context sidebar)
Here's what it looks like:
<img width="1738" alt="Screen Shot 2022-03-21 at 3 23 28 PM" src="https://user-images.githubusercontent.com/11915034/159373291-8b937d30-fba3-472a-853a-766018f6b3e2.png">
See `tests/playwright-test/reporter-html.spec.ts` for an example of usage (for dogfood-ing only). In the future, if this becomes user-facing, there the Global Setup bit would likely become unnecessary (as would interaction with attachments array); there would likely just be a nice top-level config and/or CLI flag to enable collecting of info.
Previously, we preserved input/textarea values by providing
`value` attribute or text child. This produces DOM that does not
actually match the original page.
This change starts using special attributes to modify values
directly when rendering.
Same treatment is also applied to options in `select` and
`checked` property of checkboxes and radio buttons.
This patch aligns the strategies that are used to generate new
screnshot expectations and to compare screenshot expectations against
baseline.
With this patch, `toHaveScreenshot` will:
- when generating a new expectation: will wait for 2 consecutive
screenshots to match and accept the last one as expectation.
- when given an expectation:
* will compare first screenshot against expectation. If matches,
resolve successfully
* if first screenshot doesn't match, then wait for 2 consecutive
screenshots to match and then compare last screenshot with the
expectation.
An example of a new detailed call log:
```
1) a.spec.ts:3:1 › should work ===================================================================
Error: Screenshot comparison failed:
20000 pixels (ratio 0.03 of all image pixels) are different
Call log:
- expect.toHaveScreenshot with timeout 5000ms
- verifying given screenshot expectation
- fast-path: checking first screenshot to match expectation
- taking page screenshot
- disabled all CSS animations
- waiting for fonts to load...
- fonts in all frames are loaded
- fast-path failed: first screenshot did not match expectation - 20000 pixels (ratio 0.03 of all image pixels) are different
- waiting for 2 consecutive screenshots to match
- waiting 100ms before taking screenshot
- taking page screenshot
- disabled all CSS animations
- waiting for fonts to load...
- fonts in all frames are loaded
- 2 consecutive screenshots matched
- final screenshot did not match expectation - 20000 pixels (ratio 0.03 of all image pixels) are different
- 20000 pixels (ratio 0.03 of all image pixels) are different
Expected: /Users/andreylushnikov/tmp/test-results/a-should-work/should-work-1-expected.png
Received: /Users/andreylushnikov/tmp/test-results/a-should-work/should-work-1-actual.png
Diff: /Users/andreylushnikov/tmp/test-results/a-should-work/should-work-1-diff.png
3 | test('should work', async ({ page }) => {
4 | await page.goto('file:///Users/andreylushnikov/prog/playwright/tests/assets/rotate-z.html');
> 5 | await expect(page).toHaveScreenshot();
| ^
6 | });
7 |
```
By default, fixtures share timeout with the test they are instantiated for.
However, for more heavy fixtures, especially worker-scoped ones, it makes
sense to have a separate timeout.
This introduces `{ timeout: number }` option to the list of fixture options
that opts the fixture into a dedicated timeout rather than sharing it
with the test.
Turns out relying on PWTRAP in stack is not reliable: depending on the
call structure, the stack might be cut unpredictably by Node.js.
This patch removes PWTRAP and instead plumbs explicit stack and
pre-set `apiName` all the way down to `wrapApiCall`.
This patch:
- adds call logs to track screenshot timeouts, e.g. due to
waiting for web fonts
- makes sure all snapshot expectations have `.png` extension
- throws a polite error when given a buffer or a string instead of a
page or a locator
- removes stray NL between error description and call log
- makes sure `apiName` is always correct (and adds a test for it)
- Line reporter now shows stats in addition to the test name:
```log
[chromium] › page/page-click-react.spec.ts:108:1 › should not retarget when element changes on hover
23% [21/93] Passed: 17 Flaky: 0 Failed: 0 Skipped: 4 (7s)
```
- When connected to a TTY or with `env.PLAYWRIGHT_LIVE_TERMINAL`
set to anything but `'0'` or `'false'`, line reporter updates in place.
- When not connected to a TTY, line reporter prints an update
after each ~1% of tests done, so it never prints more than 100 lines.
- Updated tests to the golden style.
Use a top-level .env file to control the internal testing setup.
This allows for easy manipulation of environment variables regardless
of your setup (VSCode Extension, CLI, etc.).
This patch adds support to multiple diffs. These are possible
due to soft assertions.
Drive-by: rename second screenshot in `toHaveScreenshot` failure when
re-generating expectations from "expected" to "previous".
This patch:
- Enables configuration of certain defaults for some options of `expect.toHaveScreenshot` method via `TestProject.expect.toHaveScreenshot` property
- Sets sensible defaults for these options:
* `fonts: "ready"`
* `animations: "disabled"`
* `size: "css"`
This patch reverts 2 commits that removed the feature from the method:
- "fix: explicitly ignore maxDiffPixels in toMatchSnapshot (#12570)"
commit b8af8458d6.
- "chore: remove `maxDiffPixels` from toMatchSnapshot (#12539)"
commit a3dff45974.
The `screenshotsDir` option controls the expectation storage
for `toHaveScreenshot()` function.
The new expectation management for screenshots has the following
key properties:
- All screenshots are stored in a single folder called `screenshotsDir`.
- Screenshot names **do not** respect `snapshotDir` and `snapshotSuffix`
configurations.
- `screenshotsDir` is configurable per project. This way a "smoke tests"
project can re-use screenshots from "all tests" project.
- Host platform is a top-level folder.
For example, given the following config:
```js
// playwright.config.ts
module.exports = {
projects: [
{ name: 'Mobile Safari' },
{ name: 'Desktop Chrome' },
],
};
```
And the following test structure:
```
smoke-tests/
└── basic.spec.ts
```
Will result in the following screenshots folder structure by default:
```
__screenshots__/
└── darwin/
├── Mobile Safari/
│ └── smoke-tests/
│ └── basic.spec.ts/
│ └── screenshot-expectation.png
└── Desktop Chrome/
└── smoke-tests/
└── basic.spec.ts/
└── screenshot-expectation.png
```
Previously, we used to skip all the tests from the same file when
any `beforeAll` fails in the file.
Now, we only skip the rest of the tests affected by this particular
`beforeAll` and continue with other tests in the new worker.
Reland: worker.stop() before worker.run() was hanging because
`_runFinished` promise was not initially resolved.
---
This moves `beforeAll`, `afterAll` and some modifiers from running
as a separate entity into running inside a test.
Pros:
- All errors are reported as test errors.
- All artifacts are collected as test artifacts.
- Reporters support this out of the box.
Details:
- Each test computes the necessary hooks to run and runs them.
- Teardown is usually performed during the test (on test failure or worker stop).
- `skipRemaining` is added to `DonePayload` to preserve the behavior
where `beforeAll` hook failure skips subsequent tests.
This behavior can now be improved to only target tests affected by this hook.
