1
1
mirror of https://github.com/microsoft/playwright.git synced 2024-12-15 14:11:50 +03:00
Commit Graph

440 Commits

Author SHA1 Message Date
Dmitry Gozman
209bde5000
feat(test runner): descrbe.skip ()
`describe.skip` declares a test group that is skipped.
2022-03-18 16:07:11 -07:00
Dmitry Gozman
3a009531b1
feat(hooks): add a step per hook ()
It is now possible to see which hooks were run in the html report.
2022-03-17 19:33:01 -07:00
Dmitry Gozman
25483452c0
feat(fixtures): per-fixture timeout ()
By default, fixtures share timeout with the test they are instantiated for.
However, for more heavy fixtures, especially worker-scoped ones, it makes
sense to have a separate timeout.

This introduces `{ timeout: number }` option to the list of fixture options
that opts the fixture into a dedicated timeout rather than sharing it
with the test.
2022-03-17 09:36:03 -07:00
Henrik Storck
05387b7835
fix: fix folder traversal for isModule check () () 2022-03-16 15:52:39 -08:00
Max Schmitt
e231db1810
chore: restore expect.any()/expect.anything() () 2022-03-16 21:34:41 +01:00
Andrey Lushnikov
88610c8b4c
fix: properly define apiName for web-first assertions ()
Turns out relying on PWTRAP in stack is not reliable: depending on the
call structure, the stack might be cut unpredictably by Node.js.

This patch removes PWTRAP and instead plumbs explicit stack and
pre-set `apiName` all the way down to `wrapApiCall`.
2022-03-14 18:01:13 -07:00
Andrey Lushnikov
e3bd7ce119
fix(test-runner): resolve outputDir wrt rootDir ()
Currently, running `npx playwright test` in a subfolder of your project
will result in a `test-results` folder created in your `cwd`.

This is unexpected; instead, we should always resolve all paths
against `rootDir` - directory that contains config.
2022-03-14 17:20:17 -07:00
Dmitry Gozman
237954212c
Revert "feat(line reporter): show stats, handle tty" ()
This reverts commit be817d1a53, PR .
Reason: found issues with stdout messing with stats.
2022-03-14 10:37:43 -07:00
Andrey Lushnikov
ef4b34fb7a
feat(toHaveScreenshot): enhance messaging in case of failures ()
This patch:
- adds call logs to track screenshot timeouts, e.g. due to
  waiting for web fonts
- makes sure all snapshot expectations have `.png` extension
- throws a polite error when given a buffer or a string instead of a
  page or a locator
- removes stray NL between error description and call log
- makes sure `apiName` is always correct (and adds a test for it)
2022-03-11 22:40:28 -08:00
Dmitry Gozman
be817d1a53
feat(line reporter): show stats, handle tty ()
- Line reporter now shows stats in addition to the test name:
  ```log
  [chromium] › page/page-click-react.spec.ts:108:1 › should not retarget when element changes on hover
  23% [21/93] Passed: 17 Flaky: 0 Failed: 0 Skipped: 4 (7s)
  ```

- When connected to a TTY or with `env.PLAYWRIGHT_LIVE_TERMINAL`
set to anything but `'0'` or `'false'`, line reporter updates in place.

- When not connected to a TTY, line reporter prints an update
after each ~1% of tests done, so it never prints more than 100 lines.

- Updated tests to the golden style.
2022-03-11 18:19:33 -08:00
Pavel Feldman
e9fa098bf3
chore: add ct examples () 2022-03-11 15:46:11 -08:00
Andrey Lushnikov
95c774cf84
feat: support multiple screenshots in HTML report ()
This patch adds support to multiple diffs. These are possible
due to soft assertions.

Drive-by: rename second screenshot in `toHaveScreenshot` failure when
re-generating expectations from "expected" to "previous".
2022-03-11 08:46:13 -08:00
Andrey Lushnikov
e3ab6388a9
feat(toHaveScreenshot): enable sensible defaults ()
This patch:
- Enables configuration of certain defaults for some options of `expect.toHaveScreenshot` method via `TestProject.expect.toHaveScreenshot` property
- Sets sensible defaults for these options:
  * `fonts: "ready"`
  * `animations: "disabled"`
  * `size: "css"`
2022-03-11 08:45:36 -08:00
Pavel Feldman
af55f48754
chore: add experimental mount () 2022-03-11 08:00:46 -08:00
Ross Wollman
ebae5e76df
feat(test-runner): more default workers for M1 ()
New default is based on the very (unscientific) output of repeated a few
times:

```
$ for i in `seq 5 13`; do time npm run ctest -- --reporter=line --workers=$i 'page/*' 'selector*' 'channels*' 'resource-timing*'; done
```
2022-03-11 02:12:07 -08:00
Andrey Lushnikov
b51157bdea
feat: bring back maxDiffPixels in toMatchSnapshot ()
This patch reverts 2 commits that removed the feature from the method:
- "fix: explicitly ignore maxDiffPixels in toMatchSnapshot ()"
  commit b8af8458d6.
- "chore: remove `maxDiffPixels` from toMatchSnapshot ()"
  commit a3dff45974.
2022-03-10 18:41:16 -08:00
Andrey Lushnikov
12d8a262be
feat: support screenshotsDir option ()
The `screenshotsDir` option controls the expectation storage
for `toHaveScreenshot()` function.

The new expectation management for screenshots has the following
key properties:
- All screenshots are stored in a single folder called `screenshotsDir`.
- Screenshot names **do not** respect `snapshotDir` and `snapshotSuffix`
  configurations.
- `screenshotsDir` is configurable per project. This way a "smoke tests"
  project can re-use screenshots from "all tests" project.
- Host platform is a top-level folder.

For example, given the following config:

```js
// playwright.config.ts
module.exports = {
  projects: [
    { name: 'Mobile Safari' },
    { name: 'Desktop Chrome' },
  ],
};
```

And the following test structure:

```
smoke-tests/
└── basic.spec.ts
```

Will result in the following screenshots folder structure by default:

```
__screenshots__/
└── darwin/
    ├── Mobile Safari/
    │   └── smoke-tests/
    │       └── basic.spec.ts/
    │           └── screenshot-expectation.png
    └── Desktop Chrome/
        └── smoke-tests/
            └── basic.spec.ts/
                └── screenshot-expectation.png
```
2022-03-10 16:50:26 -08:00
Pavel Feldman
4d3d0e2bda
chore: enable ts-esm support by default () 2022-03-10 14:48:33 -08:00
Max Schmitt
3f468abedd
fix(test-runner): ESM compatibly on Windows () 2022-03-10 19:27:25 +01:00
Andrey Lushnikov
75eef09c0d
feat: show expectation name as part of toHaveScreenshot title ()
This patch adds snapshot file name as part of `toHaveScreenshot`
and `toMatchSnapshot` step title.
2022-03-09 20:09:45 -08:00
Dmitry Gozman
92430cc56c
chore(fixtures): do not force-close contexts in beforeAll ()
This now happens the same way as for the test, in the worker cleanup
when browser closes, which is reported as a part of the test.
2022-03-09 07:26:46 -08:00
Dmitry Gozman
e8ce5d0258
feat(test runner): on beforeAll failure, precisely skip the tests ()
Previously, we used to skip all the tests from the same file when
any `beforeAll` fails in the file.

Now, we only skip the rest of the tests affected by this particular
`beforeAll` and continue with other tests in the new worker.
2022-03-08 20:29:31 -08:00
Dmitry Gozman
4a768294b4
chore: cleanup hooks notion from raw/html reporters () 2022-03-08 19:08:31 -08:00
Dmitry Gozman
e895bc2751
reland(test-runner): rely on test title paths instead of ordinal ()
Fixes 

Co-authored-by: Andrey Lushnikov <lushnikov@chromium.org>
2022-03-08 19:06:25 -08:00
Dmitry Gozman
682eeb4ae0
reland(hooks): separate test timeout from beforeAll/afterAll timeouts ()
This makes it possible to have longer `beforeAll`/`afterAll` and not
affect first/last test timeout.
2022-03-08 19:05:23 -08:00
Yury Semikhatsky
176846672f
docs: clarify multiple before/after hooks execution order () 2022-03-08 17:42:50 -08:00
Dmitry Gozman
9cd704220c
reland(test runner): run hooks/modifiers as a part of the test ()
Reland: worker.stop() before worker.run() was hanging because
`_runFinished` promise was not initially resolved.
---
This moves `beforeAll`, `afterAll` and some modifiers from running
as a separate entity into running inside a test.

Pros:
- All errors are reported as test errors.
- All artifacts are collected as test artifacts.
- Reporters support this out of the box.

Details:
- Each test computes the necessary hooks to run and runs them.
- Teardown is usually performed during the test (on test failure or worker stop).
- `skipRemaining` is added to `DonePayload` to preserve the behavior
  where `beforeAll` hook failure skips subsequent tests.
  This behavior can now be improved to only target tests affected by this hook.
2022-03-08 16:35:14 -08:00
Andrey Lushnikov
4bccf2f6aa
chore: cut v1.20 branch () 2022-03-08 14:01:20 -08:00
Andrey Lushnikov
b8af8458d6
fix: explicitly ignore maxDiffPixels in toMatchSnapshot ()
Fixes 
2022-03-07 16:55:35 -08:00
Andrey Lushnikov
a3dff45974
chore: remove maxDiffPixels from toMatchSnapshot ()
References 
2022-03-04 18:27:04 -08:00
Andrey Lushnikov
4953fc4845
chore: introduce toHaveScreenshot configuration options ()
References https://github.com/microsoft/playwright/issues/12441
2022-03-04 17:30:43 -08:00
Andrey Lushnikov
1f5cfcaeec
fix(to-have-screenshot): show number of bad pixels in mismatched error ()
Drive-by: do not show empty call log.

References 
2022-03-04 11:01:05 -08:00
Basarat Ali Syed
dd83ece3cb
chore(test-runner): correct warning instruction for port reuse () 2022-03-04 10:14:49 +01:00
Andrey Lushnikov
279e579097
chore: address pixelCount and pixelRatio review comments ()
- documentation fixes
- rename pixelCount to maxDiffPixels
- rename pixelRatio to maxDiffPixelRatio
2022-03-03 23:17:31 -08:00
Pavel Feldman
34bf0d4b87
feat(test-runner): allow grepping in project () 2022-03-02 15:02:35 -08:00
Pavel Feldman
4b19d59ec5
feat(test): introduce fully parallel mode () 2022-03-01 18:12:21 -08:00
Pavel Feldman
b79bb32c82
chore: do not stall on actions when test timeout is 0 () 2022-03-01 13:43:38 -08:00
Pavel Feldman
a612a3526a
chore: auto-detect ts esm mode () 2022-03-01 12:56:26 -08:00
Max Schmitt
5b7db90cb6
chore: hide unsupported expect methods () 2022-03-01 20:04:34 +01:00
Pavel Feldman
96945ab0a2
chore: do not print empty attachments in terminal () 2022-03-01 09:11:53 -08:00
Pavel Feldman
6a663ef54f
chore(test-runner): revert recent changes to fix tests ()
* Revert "fix(hooks): separate test timeout from beforeAll/afterAll timeouts ()"

This reverts commit 73dee69558.

* Revert "fix(test-runner): rely on test title paths instead of ordinal ()"

This reverts commit d744a87aee.

* Revert "chore(test runner): run hooks/modifiers as a part of the test  ()"

This reverts commit 47045ba48d.
2022-03-01 09:11:17 -08:00
Andrey Lushnikov
d744a87aee
fix(test-runner): rely on test title paths instead of ordinal ()
Fixes 
2022-02-28 15:40:23 -08:00
Dmitry Gozman
73dee69558
fix(hooks): separate test timeout from beforeAll/afterAll timeouts ()
This makes it possible to have longer `beforeAll`/`afterAll` and not
affect first/last test timeout.
2022-02-28 15:09:04 -08:00
Andrey Lushnikov
396d920145
feat(test-runner): implement expect(pageOrLocator).toHaveScreenshot ()
Fixes 
2022-02-28 12:25:59 -08:00
Dmitry Gozman
47045ba48d
chore(test runner): run hooks/modifiers as a part of the test ()
chore(test runner): run hooks/modifiers as a part of the test

This moves `beforeAll`, `afterAll` and some modifiers from running
as a separate entity into running inside a test.

Pros:
- All errors are reported as test errors.
- All artifacts are collected as test artifacts.
- Reporters support this out of the box.

Details:
- Each test computes the necessary hooks to run and runs them.
- Teardown is usually performed during the test (on test failure or worker stop).
- `skipRemaining` is added to `DonePayload` to preserve the behavior
  where `beforeAll` hook failure skips subsequent tests.
  This behavior can now be improved to only target tests affected by this hook.
2022-02-28 11:42:47 -08:00
Dmitry Gozman
3c2bca2768
fix(test loader): implement tsconfig paths through resolveFilename ()
This uses `Module._resolveFilename` to intercept module resolution and
check `tsconfig.paths` similarly to pirates usage ot `Module._compile`.

Previously, we resolved during compilation that required reproducible
resolution due to caching. Now we can resolve as we go and support
all `tsconfig.paths`.
2022-02-25 15:43:58 -08:00
Dmitry Gozman
91672595f2
fix(reporters): normalize usage of isTTY, env.CI and debug env variables ()
- `stdout.isTTY` controls whether list reporter updates lines or just adds them;
- `env.CI` is used in a few places to affect the defaults:
  - whether to open interactive html;
  - default reporter dot/line;
  - default terminal reporter added to non-terminal reporters;
- `env.PWTEST_SKIP_TEST_OUTPUT` is removed;
- `env.PW_TEST_DEBUG_REPORTERS` is introduced specifically for tests.
2022-02-24 12:39:28 -08:00
Andrey Lushnikov
71edad327b
fix: tolerate EBUSY error when removing output directories ()
- Previous attempt: https://github.com/microsoft/playwright/pull/12300
- Revert with reasoning: ebc52d10e4

Fixes 
2022-02-23 14:23:57 -08:00
Andrey Lushnikov
df7fbe7857
Revert "chore: best-effort cleanup for output folders that are mounted ()" ()
This reverts commit e5c9d1e39f.

Reason for revert: turns out this fix results in a 5-second delay
when starting tests in docker, with `test-results` folder being
a non-removable mount.

The reason for the delay is the `maxBusyTries` option that we
supply by default to rimraf when trying to remove the folder.

While this option might come handy when removing temporary
browser profile folder, it doesn't serve us well in this particular
usecase.

References 
2022-02-23 14:10:11 -08:00
Andrey Lushnikov
5879c7f362
chore: refactor toMatchSnapshot once again ()
Keep massaging code in preparation for `toHaveScreenshot`.

References 
2022-02-23 13:17:37 -08:00
Andrey Lushnikov
e5c9d1e39f
chore: best-effort cleanup for output folders that are mounted ()
Fixes 
2022-02-23 12:55:41 -08:00
Dmitry Gozman
4399623f9f
chore(test runner): make timeout a separate error in TestInfo.errors ()
This way we control the timeout error message from the runner,
so that later on we can differentiate between test timeout, fixture
timeout and hook timeout.
2022-02-23 12:32:12 -08:00
Dmitry Gozman
114edecd3a
fix(list mode): keep outputDir intact () 2022-02-22 12:50:26 -08:00
Dmitry Gozman
d3c4323021
fix(test runner): improve error message for unexpected calls () 2022-02-18 18:25:18 -08:00
Dmitry Gozman
ee0dd6ec71
fix(test runner): trim full output path ()
Previously, we only trimmed the "full title" component, but
we should also trim the "relative path" component.
2022-02-18 15:40:36 -08:00
Andrey Lushnikov
0682672242
chore: move comparator logic to playwright-core ()
This will enable implementation of `toHaveScreenshot` on the
server-side.

Drive-by: drop blink-diff

References 
2022-02-18 14:39:17 -08:00
Andrey Lushnikov
b9e86b79ee
chore: refactor toMatchSnapshot implementation ()
This patch prepares for the `toHaveScreenshot` implementation
by splitting common parts from `toMatchSnapshot`.

Drive-by: fix default extension generation from `.bin` to `.dat`
for unknown buffers.
2022-02-18 11:21:58 -08:00
Dmitry Gozman
edac84d072
feat(remote): send browser/channel/headless in a header () 2022-02-18 07:54:33 -08:00
Dmitry Gozman
34eec7eae7
fix(webServer): route output through reporters () 2022-02-18 07:54:01 -08:00
Andrey Lushnikov
a98babec69
feat(test-runner): introduce pixelCount and pixelRatio options ()
This patch adds additional options to `toMatchSnapshot` method:
- `pixelCount` - acceptable number of pixels that differ to still
  consider images equal. Unset by default.
- `pixelRatio` - acceptable ratio of all image pixels (from 0 to 1) that differ to still
  consider images equal. Unset by default.

Fixes , 
2022-02-17 15:44:03 -08:00
Pavel Feldman
51344c6924
chore: parse-skip .tsx () 2022-02-16 15:45:35 -08:00
Andrey Lushnikov
086333cd60
feat(test-runner): support unnamed snapshots ()
Fixes 
2022-02-16 14:22:01 -08:00
Andrey Lushnikov
ef21ce3f56
feat(test-runner): filter out syntax error stack traces ()
Filter out long stack traces from babel when it fails compilation
due to syntax error in test.
2022-02-14 14:33:14 -08:00
Dmitry Gozman
9814c592d2
fix(test runner): shutdown redundant workers () 2022-02-14 10:57:15 -08:00
Pavel Feldman
96b5831a49
chore: add support private methods in TS () 2022-02-11 14:46:49 -08:00
Dmitry Gozman
d8db785c0a
fix(reporters): correctly handle missing stdout.columns ()
When columns are not available, do not trim the output.
2022-02-11 08:33:56 -08:00
Pavel Feldman
ea590c6f67
fix(teardown): await teardown in failed test runs () 2022-02-10 12:44:42 -08:00
Ross Wollman
0d42c16a17
fix(test-runner): undefined body crash with manual attachments ()
The new (as of 1.18) `async testInfo.attach(…)` API handles this
gracefully (and is part of the reason for the new API's existence).
However, for the foreseeable future, it's still possible to manually
push onto the attachments array where we can't validate the contents
until it's too late, so this change ensures more graceful handling in
that case.

Fixes 
2022-02-10 12:33:38 -08:00
Pavel Feldman
fbc05b74a7
chore: revert "fix(test-runner): escape backslashes in win cli ()" () 2022-02-10 07:02:37 -08:00
Andrey Lushnikov
6904b3294e
fix(test-runner): fix browser initialization in test modifiers ()
Fixes 
2022-02-09 15:30:14 -08:00
Yury Semikhatsky
706c897031
feat(fixtures): respect tracing config for APIRequestContext () 2022-02-09 08:54:09 -08:00
Pavel Feldman
e9e5de2d57
feat(debug): allow preprocessing JS scripts as well () 2022-02-09 07:14:11 -08:00
Dmitry Gozman
19368e93af
feat(test runner): support connectOptions ()
This allows to specify `connectOptions` in the config that
switch built-in `browser` to be remotely connected.
2022-02-08 20:45:42 -08:00
Dmitry Gozman
5881a46ecf
fix(test runner): skip beforeAll/afterAll when all tests are skipped ()
There is a corner case where tests were skipped like this:

```js
test.skip('title', () => {});
```
2022-02-08 16:36:30 -08:00
Andrey Lushnikov
3eba252f2e
chore: cut v1.19.0 () 2022-02-08 16:01:44 -08:00
Dmitry Gozman
e92caf01b3
fix(webServer): do not set baseURL equal to webServer.url () 2022-02-08 15:57:36 -08:00
Pavel Feldman
8dff2e35c8
chore: don't throw on no testDir () 2022-02-08 15:27:05 -08:00
Pavel Feldman
d5158e8d24
chore: revert source-map-support to 0.4.18 () 2022-02-08 14:50:20 -08:00
Pavel Feldman
4bcca2c87e
chore: add a parallel mode note () 2022-02-08 14:36:14 -08:00
Dmitry Gozman
8a5c93436d
fix(reporter): do not report parallel tests as slow () 2022-02-07 20:10:13 -08:00
Dmitry Gozman
7912c515a3
fix(fixtures): account for default options being undefined () 2022-02-07 17:11:36 -08:00
Pavel Feldman
9116adc684
chore: allow opt-into the legacy global setup mode () 2022-02-07 10:41:56 -08:00
Yury Semikhatsky
46dfa45b4e
docs: fix parallel test docs build () 2022-02-04 16:44:04 -08:00
Yury Semikhatsky
afc4b505aa
docs: fix duplicate sections () 2022-02-04 13:50:46 -08:00
Dmitry Gozman
d9a8bb057d
fix(test-fail): allow unhandled expects in test.fail ()
Previously, we would consider this a worker error, but
we make an exception for "expect()" calls.
2022-02-03 17:14:12 -08:00
Pavel Feldman
72424dc904
chore: allow setting reporter via env () 2022-02-03 16:10:39 -08:00
Pavel Feldman
fdda759a9d
feat(parallel): allow setting enclosing scope parallel mode () 2022-02-02 20:44:11 -08:00
Andrey Lushnikov
ba0c7e679b
feat(test-runner): support expect.soft ()
Soft expects will still fail the test, but will not abort it's execution. As a consequence of this:
-  `TestResult` now might have multiple errors, which is reflected with a new `testResult.erros: TestError[]` field.
- `TestInfo` now might have multiple errors as well, which is reflected with a new `testInfo.errors: TestError[]` field.

Fixes 
2022-02-02 18:33:51 -08:00
Max Schmitt
c752b28516
chore: pin production dependencies () 2022-02-02 11:14:41 +01:00
Andrey Lushnikov
b0daa7754f
feat: filter stack traces to exclude test runner frames ()
Before:

```bash
Running 1 test using 1 worker
  1) [chromium] › tests/example.spec.ts:3:1 › should work ==========================================

    Error: expect(received).toBe(expected) // Object.is equality

    Expected: 2
    Received: 1

      2 |
      3 | test('should work', async({page}) => {
    > 4 |   expect(1).toBe(2);
        |             ^
      5 | });
      6 |

        at Proxy.<anonymous> (/Users/andreylushnikov/prog/playwright/packages/playwright-test/src/expect.ts:151:30)
        at /Users/andreylushnikov/tmp/tests/example.spec.ts:4:13
        at /Users/andreylushnikov/prog/playwright/packages/playwright-test/src/workerRunner.ts:335:13
        at runNextTicks (node:internal/process/task_queues:61:5)
        at processImmediate (node:internal/timers:437:9)
        at TestInfoImpl._runFn (/Users/andreylushnikov/prog/playwright/packages/playwright-test/src/testInfo.ts:164:7)
        at WorkerRunner._runTestWithBeforeHooks (/Users/andreylushnikov/prog/playwright/packages/playwright-test/src/workerRunner.ts:317:24)
        at TimeoutRunner.run (/Users/andreylushnikov/prog/playwright/packages/playwright-core/src/utils/async.ts:48:14)
        at TestInfoImpl._runWithTimeout (/Users/andreylushnikov/prog/playwright/packages/playwright-test/src/testInfo.ts:151:7)
        at WorkerRunner._runTestOrAllHook (/Users/andreylushnikov/prog/playwright/packages/playwright-test/src/workerRunner.ts:276:5)
        at WorkerRunner._runSuite (/Users/andreylushnikov/prog/playwright/packages/playwright-test/src/workerRunner.ts:190:11)
        at WorkerRunner.run (/Users/andreylushnikov/prog/playwright/packages/playwright-test/src/workerRunner.ts:137:9)
        at process.<anonymous> (/Users/andreylushnikov/prog/playwright/packages/playwright-test/src/worker.ts:87:5)
```

after:

```
Running 1 test using 1 worker
  1) [chromium] › tests/example.spec.ts:3:1 › should work ==========================================

    Error: expect(received).toBe(expected) // Object.is equality

    Expected: 2
    Received: 1

      2 |
      3 | test('should work', async({page}) => {
    > 4 |   expect(1).toBe(2);
        |             ^
      5 | });
      6 |

        at /Users/andreylushnikov/tmp/tests/example.spec.ts:4:13
```
2022-02-01 18:40:44 -08:00
Pavel Feldman
ceb3027bdf
chore(test-runner): force colors in worker processes () 2022-02-01 17:29:15 -08:00
Pavel Feldman
6e2fcc4700
chore: do not expose suite load error () 2022-02-01 15:34:16 -08:00
Pavel Feldman
f875ebe730
fix(test-runner): respect source maps when reporting test files () 2022-02-01 13:04:54 -08:00
Max Schmitt
028afb167b
chore: bump production dependencies () 2022-02-01 21:27:34 +01:00
Pavel Feldman
a3bc911a3d
fix(debug): don't apply infinite timeouts when using debugger () 2022-02-01 11:51:37 -08:00
Caio Agiani
b6b60decdf
fix: typos () 2022-02-01 11:09:41 -08:00
Pavel Feldman
3a5e8184b5
fix(html-reporter): open tests from required file () 2022-02-01 11:01:52 -08:00
Pavel Feldman
c82f2641d7
fet(list-files): report per-project test dir and filters () 2022-02-01 08:08:56 -08:00
Pavel Feldman
c5d852f1bb
fix(test-runner): escape backslashes in win cli () 2022-01-31 17:48:05 -08:00
Andrey Lushnikov
2336692e8a
feat: support clarification message for expect ()
The clarification message is displayed in the HTML report as the name of the step:

![image](https://user-images.githubusercontent.com/746130/151852652-48194140-5ea4-439d-afee-12583a8caf71.png)

It is also shown in terminal output:

![image](https://user-images.githubusercontent.com/746130/151852666-5c956ef1-6e94-4bc2-8e55-b58688dfc7e0.png)

Fixes 
2022-01-31 17:14:59 -08:00
Pavel Feldman
2b55adaafa
feat(breaking): always report onBegin/onEnd, report file errors () 2022-01-31 17:09:04 -08:00
Andrey Lushnikov
3a4e506479
chore: extract SigIntWatcher ()
This is to reduce the size of the long `_run` method in the `runner.ts`.
It also might come handy around the codebase.
2022-01-31 08:51:22 -08:00
Andrey Lushnikov
b58b004f0f
fix: run reporter.onEnd after webserver teardown ()
Fixes 
2022-01-31 06:19:33 -08:00
Dmitry Gozman
eeebcd53ae
chore(test runner): extract TestInfoImpl () 2022-01-28 17:39:42 -08:00
Dmitry Gozman
30f3cbd6da
fix(fixtures): make sure defaultLaunchOptions are always there () 2022-01-28 10:51:55 -08:00
Pavel Feldman
29a84b2df8
chore: allow pre-processing scripts () 2022-01-27 14:32:23 -08:00
Feng Yu
0698254253
docs: fix broken link ()
fix 
2022-01-27 13:00:09 -08:00
Dmitry Gozman
b1fbc4fdbe
fix(test runner): resolve tsconfig for each file ()
This allows us to properly handle path mappings
that are not too ambiguous.
2022-01-26 18:28:42 -08:00
divdavem
512a245f13
feat(test-runner): wait for a url before starting tests ()
The webServer configuration in @playwright/test now accepts a url as an
alternative to a port number to wait for a url to return a 2xx status code.
2022-01-26 16:32:58 -08:00
Dmitry Gozman
19820de7a9
fix(test runner): correctly save videos when running remotely () 2022-01-26 07:43:07 -08:00
Dmitry Gozman
4f42484c64
chore(test runner): simplify runTestWithBeforeHooks () 2022-01-25 16:46:50 -08:00
Dmitry Gozman
800b813d4b
chore(test runner): prepare to per-fixture timeout ()
This reworks DeadlineRunner to use exception to signal timeout. This way,
we'll be able to run fixtures against a shared deadline vs their own
deadline and still get an easy control-flow timeout handling.
2022-01-25 11:22:28 -08:00
Andrey Lushnikov
42876a0528
fix: support PWDEBUG=0 to disable debug ()
Fixes 
2022-01-25 08:40:24 -08:00
Ross Wollman
64e7557fb9
fix: falsey behavior in route.continue, page.post, testInfo.attach ()
In several of the Playwright APIs, falsey values were not handled correctly. This changeset adds tests (and some fixes):

- route.continue: If options.postData was the empty string, the continue failed to override the post data.
- page.post (application/json with options.data: false|''|0|null): Raw falsey values were getting dropped (i.e. you can't do the equivalent of curl --header application/json … -d 'false'). This has been fixed with most values across all browsers, but an additional fix is needed for 'null' which the channel serializer treats extra specially.
- testInfo.attach: This didn't get reported as an error when options.path was the empty string, but should have been.
 (and its fix ) inspired this search as they are the same
class of bug.
2022-01-24 15:06:36 -08:00
Pavel Feldman
9542b007cf
fix(esm): don't emit module shorthand warning () 2022-01-24 11:22:56 -08:00
Pavel Feldman
0b99d83911
chore: include config dir in quick list report () 2022-01-22 17:02:46 -08:00
Pavel Feldman
baf0e5b47a
chore: implement lightweight test listing () 2022-01-21 19:11:22 -08:00
Pavel Feldman
6b3e596fd8
fix(baseurl): support path-less baseurl () 2022-01-20 18:11:56 -08:00
Pavel Feldman
50fa839e64
chore: include transpilation mode in the compute cache hash () 2022-01-20 13:33:40 -08:00
Ross Wollman
33f32368e5
fix(test runner): dispatcher transformation of empty attachments () 2022-01-17 18:47:24 -08:00
Pavel Feldman
8ecf581b45
chore: focus by line should override only () 2022-01-16 08:47:09 -08:00
Pavel Feldman
8b0ef4ad6b
chore: introduce oop testing mode for debugging () 2022-01-15 19:25:32 -08:00
Max Schmitt
badb5b4d13
chore(pw-test): forward web-server stdout to debug target () 2022-01-13 23:55:46 +01:00
Dmitry Gozman
9d5bf0e90d
fix(test runner): fixture teardown double error, testInfo.attach() ()
- Use file path, not content to calculate the attachment hash.
- Always cleanup fixture from the list on teardown, to avoid reporting
  teardown error multiple times: from the test, and from the cleanup.
2022-01-13 10:38:47 -08:00
Pavel Feldman
feb7148b3f
chore: don't write into stdout from the test runner, use reporters instead () 2022-01-12 19:52:40 -08:00
Andrey Lushnikov
9285596806
chore: cut v1.18.0 ()
Drive-by: fix the `//utils/bump_package_versions.js` script.
2022-01-12 11:50:44 -08:00
Andrey Lushnikov
6f932fcb4a
fix: default list reporter should dump stderr to stderr ()
This is consistent with all other reporters we have in place.
2022-01-12 07:43:25 -08:00
Dmitry Gozman
4efb30999f
feat(request): show request context methods as steps () 2022-01-11 17:33:41 -08:00
Andrey Lushnikov
2a0930c7a4
docs: clarification about trailing slash in docs ()
Fixes 
2022-01-11 07:50:58 -08:00
Dmitry Gozman
da1f39fb29
chore(test runner): refactor worker runner parts ()
This will make it easier to change in the future.
2022-01-10 20:25:56 -08:00
Dmitry Gozman
d36ff8a96c
fix(test runner): remove test output for failures () 2022-01-10 13:22:09 -08:00
Dmitry Gozman
14fd837e94
fix(test runner): hide beforeAll/afterAll hooks from the reporter api ()
This api is not ready yet.
2022-01-10 12:09:51 -08:00
Andrey Lushnikov
7b1fecc009
fix: default to en-US locale in all browsers in Playwright Test ()
It makes sense to default to en-US by default across all our browsers in Playwright Test.

Fixes 
2022-01-10 08:37:55 -08:00
Michaël De Boey
ff5f75dc10
fix: pin colors to 1.4.0 () 2022-01-09 16:18:21 -08:00
Dmitry Gozman
058f98d3dd
fix(test runner): revert error location from top frame ()
This does not play nicely with some internal Playwright errors, so
it needs more tweaks.
2022-01-07 11:06:47 -08:00
Dmitry Gozman
19460d50e4
chore(test runner): process-wide cache of required test files ()
This is a preparation to TestRunner api, where we should be
able to run tests multiple times in the same runner process.
2022-01-06 15:38:10 -08:00
Pavel Feldman
373042ed89
chore: don't allow importing @playwright/test twice () 2022-01-06 09:29:05 -08:00
Dmitry Gozman
3ecac56cc0
fix(test runner): testInfo.attach api review changes ()
Remove overload, require name, merge options.
2022-01-05 16:39:33 -08:00
Dmitry Gozman
f77c874e8a
feat(test runner): make _extendTest experimental ()
Hidden from types and docs.
2022-01-05 15:54:00 -08:00
Dmitry Gozman
ccc61e31ea
chore(test runner): preparation for TestRunner api () 2022-01-05 15:49:01 -08:00
Dmitry Gozman
3839917eb6
chore(test runner): move config loading to runner ()
In preparation for the TestRunner api.
2022-01-05 13:44:29 -08:00
Dmitry Gozman
16a779a5ff
fix(test runner): show codeframe and location from the error top stack frame ()
Previously, reporter would look for a stack frame directly in the test file.
Often times, that is not a top stack frame, especially when the test uses
some helper functions.

This changes error snippets and locations to use the top frame. When top
frame does not match the test file, we additionally show the location
to avoid confusion:

```
  1) a.spec.ts:7:7 › foobar ========================================================================

    Error: oh my

       at helper.ts:5

      3 |
      4 |       export function ohMy() {
    > 5 |         throw new Error('oh my');
        |               ^
      6 |       }
      7 |

        at ohMy (.../reporter-base-should-print-codeframe-from-a-helper/helper.ts:5:15)
        at .../reporter-base-should-print-codeframe-from-a-helper/a.spec.ts:8:9
        at FixtureRunner.resolveParametersAndRunHookOrTest (.../src/fixtures.ts:281:12)
```
2022-01-04 16:00:55 -08:00
Pavel Feldman
7bfa6f9b5f
feat(html): group similar items in the report () 2022-01-03 21:17:17 -08:00
Pavel Feldman
a0aeaeb929
test: expose repeatEachIndex () 2022-01-03 17:29:54 -08:00
Ross Wollman
dadb5cbc30
fix(html reporter): fix too much strikethrough in diffs ()
Textual snapshot diffs were previously broken in the HTML Report. The strikethrough'd text extended beyond the intended region.

HTML Report Before: 
<img width="693" alt="Screen Shot 2021-12-27 at 4 43 35 PM" src="https://user-images.githubusercontent.com/11915034/147518750-a60f9002-6eed-48a1-a412-20fabd076fa6.png">

HTML Report After:
<img width="206" alt="Screen Shot 2021-12-27 at 4 48 37 PM" src="https://user-images.githubusercontent.com/11915034/147518762-19a4c8f9-ccc3-4a3c-a962-5a42edc6fc5d.png">

This now matches what's expected and shown in the terminal (which has always been correct):

<img width="1384" alt="Screen Shot 2021-12-27 at 4 36 29 PM" src="https://user-images.githubusercontent.com/11915034/147518799-f538259e-5a45-4d6f-916c-a12ccb620c5b.png">

NB: This MR is a workaround, but not a root cause fix. It works, but I never fully got to the root cause so a bug upstream may be required. It's unclear whether it's (1) in [`colors`](https://www.npmjs.com/package/colors), (2) in [`ansi-to-html`](https://www.npmjs.com/package/ansi-to-html), or (3) Playwright's use of the two. Since the terminal output is correct, I suspect it is in `ansi-to-html`. For example:

```js
const colors = require("colors");
const Convert = require('ansi-to-html');
const convert = new Convert();

// original (strike incorrectly wraps everything in the HTML)
console.log(convert.toHtml(colors.strikethrough("crossed out") + ' ' + colors.red("red")))
// prints: <strike>crossed out <span style="color:#A00">red<span style="color:#FFF"></span></span></strike>

// workaround
console.log(convert.toHtml(colors.reset(colors.strikethrough("crossed out")) + ' ' + colors.red("red")))
// prints: <strike>crossed out</strike> <span style="color:#A00">red<span style="color:#FFF"></span></span>
```

Fixes 
2021-12-28 09:56:34 -08:00
Dmitry Gozman
f933759ad1
chore(test runner): minor improvements ()
- Types for fixture options and more.
- Refined type for deadline runner.
2021-12-22 09:59:58 -08:00
Dmitry Gozman
f5304e3bda
fix(fixtures): await fixture teardown when shutting down the worker () 2021-12-20 16:19:21 -08:00
Dmitry Gozman
2d00836f0e
fix(test runner): show the location of afterAll timeout () 2021-12-18 09:32:41 -08:00
Dmitry Gozman
c9ba49936f
feat(reporters): show retry #x when running a test () 2021-12-17 21:07:04 -08:00
Pavel Feldman
8f98074fc8
chore: add blink-diff third party library () 2021-12-17 15:53:37 -08:00
Dmitry Gozman
b6aad54b9f
fix(test runner): test.setTimeout whould not break debugging ()
We ignore test.setTimeout() when timeout is already zero for debugging.
2021-12-17 15:17:48 -08:00
Dmitry Gozman
f5780be41b
fix(list reporter): make sure that duration suffix survives truncation () 2021-12-17 13:08:02 -08:00
Dmitry Gozman
93ad12978c
fix(test runner): disable expect, action and navigation timeouts on debug ()
We disable these timeouts when test timeout is set to zero.
This covers PWDEBUG=1, --debug and manual `test.setTimeout(0)` scenarios.
2021-12-16 18:32:46 -08:00
Dmitry Gozman
192071d5bc
feat(test runner): save traces for beforeAll/afterAll hooks () 2021-12-15 16:06:10 -08:00
Dmitry Gozman
19b08332ce
feat(test runner): disable test timeout on page.pause() call () 2021-12-15 11:12:52 -08:00
Dmitry Gozman
0d54afab9c
feat(test runner): show beforeAll/afterAll hooks similar to tests ()
Reporters now get notified about hooks start/end/steps.
2021-12-15 10:39:49 -08:00
Pavel Feldman
f579f9c806
chore: parse tsx tests () 2021-12-14 19:25:07 -08:00
Dmitry Gozman
34b84841b0
chore(test runner): create TestResult instances lazily in dispatcher ()
This prepares for beforeAll/afterAll hooks to be handled in the same way.
Since we do not know in advance whether a hook will run, we must create
TestResults lazily.
2021-12-14 14:10:56 -08:00
Dmitry Gozman
5c4ebdce54
fix(line reporter): clarify about retries when going over total counter () 2021-12-13 19:06:13 -08:00
Joel Einbinder
c27491cd4d
feat(test-runner): shorten long output paths () 2021-12-13 10:56:03 -08:00
Joel Einbinder
7a02c52144
feat(test-runner): specific playwright types for expect () 2021-12-13 13:42:36 -05:00
Dmitry Gozman
308c7b4e32
docs: update test advanced guides ()
Linking to the API reference, using better examples and newer docs.
2021-12-10 11:15:01 -08:00
Dmitry Gozman
897e41c6c1
docs: document TestOptions.video.size () 2021-12-09 07:38:58 -08:00
Pavel Feldman
b5933db279
feat(tsconfig): respect baseUrl and paths from tsconfig () 2021-12-08 22:43:00 -08:00
Pavel Feldman
4d683cef7f
fix(html): render text attachments as text () 2021-12-08 08:51:44 -08:00
Pavel Feldman
a08a41f6c9
chore: render annotations in html report () 2021-12-07 18:35:06 -08:00
Pavel Feldman
feb4c62da1
fix(html): html reporter fixes () 2021-12-07 16:47:47 -08:00
Pavel Feldman
7765131a14
feat(acceptDownload): revert acceptDownload () 2021-12-06 09:25:24 -08:00
Pavel Feldman
518d67add5
feat(test.info): expose information on the currently running test () 2021-12-06 09:25:11 -08:00
Dmitry Gozman
98e2f40bb0
docs: replace TestCase.suite with TestCase.parent ()
It is there by mistake.
2021-12-02 14:24:43 -08:00
Pavel Feldman
31e0a63fcd
feat(toBeChecked): allow passing checked: false () 2021-12-02 10:31:26 -08:00
Pavel Feldman
b9731a904e
chore: add validations into check_deps () 2021-12-01 18:14:13 -08:00
Saransh Miglani
f05252874a
chore: minor code rearrangement () 2021-12-01 09:18:16 -08:00
Yury Semikhatsky
d66b7aab3b
feat(expext): toBeOK for APIResponse () 2021-11-30 18:12:19 -08:00
Dmitry Gozman
729da65eba
fix(test runner): allow multiple missing snapshots per test ()
Instead of failing right away, continue test execution but mark
the test as failed.
2021-11-30 17:50:19 -08:00
Max Schmitt
3337920c76
docs: clarify webServer ipv4/ipv6 and baseURL protocol () 2021-11-30 20:01:28 +01:00
Max Schmitt
682d1f41c7
fix(test-runner): wrong test-runner options got determined ()
Follow-up for 
2021-11-30 19:59:35 +01:00
Max Schmitt
9337aa3cdf
fix(test-runner): webServer port detection on Node.js 17 () 2021-11-29 19:36:35 +01:00
Joel Einbinder
15053d9b3b
fix(test-runner): apply default options to all browserTypes () 2021-11-29 12:21:15 -05:00
Max Schmitt
293c233a49
docs(test-runner): add webServer () 2021-11-29 18:01:14 +01:00
Joel Einbinder
20c0facfb9
fix(test runner): dont mangle test names with multiple dashes () 2021-11-24 19:36:38 -05:00
Pavel Feldman
5d19f16601
feat(esm): introduce experimental PW_EXPERIMENTAL_TS_ESM option () 2021-11-24 14:17:01 -08:00
Pavel Feldman
7eb3f76f49
feat(esm): allow running tests in type module projects () 2021-11-24 12:42:48 -08:00
Max Schmitt
85197e68c9
chore: support range requests in trace viewer http server () 2021-11-23 21:37:55 +01:00
Ross Wollman
854f321532
feat(api): add explicit async testInfo.attach ()
feat(api): add explicit async testInfo.attach

We add an explicit async API for attaching file paths (and Buffers) to
tests that can be awaited to help users ensure they are attaching files
that actually exist at both the time of the invocation and later when
reporters (like the HTML Reporter) run and package up test artifacts.

This is intended to help surface attachment issues as soon as possible
so you aren't silently left with a missing attachment
minutes/days/months later when you go to debug a suddenly breaking test
expecting an attachment to be there.

NB: The current implemntation incurs an extra file copy compared to
manipulating the raw attachments array. If users encounter performance
issues because of this, we can consider an option parameter that uses
rename under the hood instead of copy. However, that would need to be
used with care if the file were to be accessed later in the test.
2021-11-23 09:30:53 -08:00
Dmitry Gozman
e647f0420c
docs: add more references to TestInfo.retry () 2021-11-22 10:06:20 -08:00
Dmitry Gozman
fde2f6a77f
docs: separate doc for test timeouts () 2021-11-19 17:06:46 -08:00
Dmitry Gozman
4eaeb3b59c
docs: explain that beforeAll/afterAll run again in the new worker process () 2021-11-19 13:47:30 -08:00
Dmitry Gozman
0302e759df
feat(test runner): allow top-level test.fixme similar to test.skip ()
```js
test.fixme('my test name', () => {});
```
2021-11-19 11:40:40 -08:00
Dmitry Gozman
d9f849fb14
feat(test runner): replace declare/define with "options" ()
1. Fixtures defined in test.extend() can now have `{ option: true }` configuration that makes them overridable in the config. Options support all other properties of fixtures - value/function, scope, auto.
```
const test = base.extend<MyOptions>({
  foo: ['default', { option: true }],
});
```

2. test.declare() and project.define are removed.

3. project.use applies overrides to default option values and nothing else. Any test.extend() and test.use() calls take priority over config options.

Required user changes: if someone used to define fixture options with test.extend(), overriding them in config will stop working. The solution is to add `{ option: true }`.

```
// Old code
export const test = base.extend<{ myOption: number, myFixture: number }>({
  myOption: 123,
  myFixture: ({ myOption }, use) => use(2 * myOption),
});

// New code
export const test = base.extend<{ myOption: number, myFixture: number }>({
  myOption: [123, { option: true }],
  myFixture: ({ myOption }, use) => use(2 * myOption),
});
```
2021-11-18 15:45:52 -08:00
Dmitry Gozman
8f43f4c98f
feat(serial): better errors from beforeAll ()
When beforeAll hook times out or fails with an exception, we now
close the context and show a nice error.
2021-11-18 14:36:55 -08:00
Dmitry Gozman
bd93fc499f
fix(html reporter): show missing attachments as warnings () 2021-11-17 18:03:13 -08:00
Dmitry Gozman
ce2c0c59a7
feat(expect): show expect timeout in the error message ()
Makes it easier to understand that expect does indeed have a separate timeout.

```
    Error: expect(received).toHaveCount(expected) // deep equality

    Expected: 0
    Received: 1

    Call log:
      - expect.toHaveCount with timeout 500ms
      - waiting for selector "span"
      -   selector resolved to 1 element
      -   unexpected value "1"
      -   selector resolved to 1 element
      -   unexpected value "1"
      -   selector resolved to 1 element
      -   unexpected value "1"
```
2021-11-17 17:28:30 -08:00
Max Schmitt
0781d0303b
docs(python): enable web-first assertions () 2021-11-18 00:46:30 +01:00
Dmitry Gozman
638ebd6dd6
fix(test runner): do not validate fixtures in tests/hooks that are never run () 2021-11-15 13:17:26 -08:00
Dmitry Gozman
4bb123d4b7
fix(test runner): route more errors to reporter ()
For example, top-level errors in files or global setup issues.
2021-11-12 06:47:41 -08:00
Andrey Lushnikov
bd1ce399e3
fix(html-report): fix command to open HTML report () 2021-11-12 00:12:23 -08:00
Pavel Feldman
8dcd77270f
fix(sigint): remove sigint handler early () 2021-11-11 17:14:36 -08:00
Dmitry Gozman
7eec66d0f9
chore(test runner): route runner errors through Reporter.onError () 2021-11-11 16:48:08 -08:00
Dmitry Gozman
f38f611478
chore: add npm run lint-tests () 2021-11-11 13:27:50 -08:00