Commit Graph

6659 Commits

Author SHA1 Message Date
Dmitry Gozman
2d00836f0e
fix(test runner): show the location of afterAll timeout (#11007) 2021-12-18 09:32:41 -08:00
Dmitry Gozman
c9ba49936f
feat(reporters): show retry #x when running a test (#11003) 2021-12-17 21:07:04 -08:00
Pavel Feldman
8f98074fc8
chore: add blink-diff third party library (#10984) 2021-12-17 15:53:37 -08:00
Dmitry Gozman
b6aad54b9f
fix(test runner): test.setTimeout whould not break debugging (#11004)
We ignore test.setTimeout() when timeout is already zero for debugging.
2021-12-17 15:17:48 -08:00
Dmitry Gozman
f5780be41b
fix(list reporter): make sure that duration suffix survives truncation (#11002) 2021-12-17 13:08:02 -08:00
Pavel Feldman
037baf0945
fix(trace-viewer): scroll action into view (#10978) 2021-12-17 12:20:01 -08:00
Yury Semikhatsky
eb9c8a8f06
fix(tracing): check if source exists before adding to zip (#10986) 2021-12-17 11:52:28 -08:00
Dmitry Gozman
0d277fa589
fix(selenium): allow setting additional capabilities (#11000)
Also clarify docs about Selenium v4.
2021-12-17 11:33:24 -08:00
Aarjith Nandakumar
64c3b189c4
docs(intro): fix optionally install browsers url (#11001) 2021-12-17 11:14:05 -08:00
Yury Semikhatsky
1dd8325c9a
docs(java): add API for providing source locations (#10998) 2021-12-17 10:50:47 -08:00
musou1500
7d39d345c2
fix(tracing): trace frame (#10846) 2021-12-17 10:43:19 -08:00
github-actions[bot]
551c8d1309
browser(chromium): roll to r952597 (#10991)
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
2021-12-17 06:20:46 -08:00
Dmitry Gozman
93ad12978c
fix(test runner): disable expect, action and navigation timeouts on debug (#10958)
We disable these timeouts when test timeout is set to zero.
This covers PWDEBUG=1, --debug and manual `test.setTimeout(0)` scenarios.
2021-12-16 18:32:46 -08:00
Dmitry Gozman
dd57843404
fix(inspector): do not show internal calls (#10982)
Also mark setDefault{Navigation,}Timeout as always internal.
2021-12-16 17:17:24 -08:00
Andrey Lushnikov
773ebe04c6
chore: follow-up to handling docker corner cases (#10981) 2021-12-16 15:29:20 -08:00
Mario Nebl
70ed020c87
fix: handle docker edge cases (#10501)
## Before

* When docker wasn't running: crashed with a TypeError 
* When any image without `RepoTags` was present: crashed with TypeError
* Typo in recommended command ("playwight")
* No indication of `PW_TEST_IMAGE` env var in error output

<details>
<summary>Docker not running</summary>

```
Using config at ~redacted/playwright.config.ts
ERROR get /images/json 500 dial unix docker.raw.sock: connect: connection refused

TypeError: Cannot read property 'find' of null
    at launchDockerGridAgent (~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.js:67:26)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at Object.launch (~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.js:43:9)
    at GridServer.createAgent (~redacted/node_modules/playwright-core/lib/grid/gridServer.js:356:5)
    at launchDockerContainer (~redacted/node_modules/@playwright/test/lib/cli.js:259:3)
    at Runner._run (~redacted/node_modules/@playwright/test/lib/runner.js:236:98)

Running 1 test using 1 worker

     test.ts:9:1 › some redacted description
ERROR get /images/json 500 dial unix docker.raw.sock: connect: connection refused

TypeError: Cannot read property 'find' of null
    at launchDockerGridAgent (~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.js:67:26)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at Object.launch (~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.js
```

</details>

<details>
<summary>Image without repo tags</summary>

```
Using config at ~redacted/playwright.config.ts
TypeError: Cannot read property 'includes' of undefined
    at ~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.js:67:55
    at Array.find (<anonymous>)
    at launchDockerGridAgent (~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.js:67:26)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at Object.launch (~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.js:43:9)
    at GridServer.createAgent (~redacted/node_modules/playwright-core/lib/grid/gridServer.js:356:5)
    at launchDockerContainer (~redacted/node_modules/@playwright/test/lib/cli.js:259:3)
    at Runner._run (~redacted/node_modules/@playwright/test/lib/runner.js:236:98)

Running 1 test using 1 worker

     some-test.ts › some description
TypeError: Cannot read property 'includes' of undefined
    at ~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.js:67:55
    at Array.find (<anonymous>)
    at launchDockerGridAgent (~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.js:67:26)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at Object.launch (~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.j  ✘  editor-editor-wysiwyg--editor-wysiwyg.vr.ts:9:1 › actual element matches expected element (26ms)


  1) some-test.ts › some description =====

    'Grid agent creation failed'


  1 failed
  some-test.ts › some description ======
```

</details>

## After

* Helpful error message if docker isn't running
* Doesn't crash when local-only images in list
* No typo in `playwright install docker-image` command
* When other playwright images are found they are listed and  `PW_TEST_IMAGE` is mentioned

### After: Docker not running

```
Using config at ~redacted/playwright.config.ts
Error fetching json: Error: connect ECONNREFUSED /var/run/docker.sock
Error:
╔═════════════════════════════════════════╗
║ Failed to list docker images            ║
║ Please ensure docker daemon is running. ║
║                                         ║
║ <3 Playwright Team                      ║
╚═════════════════════════════════════════╝
    at launchDockerGridAgent (~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.js:61:11)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at Object.launch (~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.js:43:9)
    at GridServer.createAgent (~redacted/node_modules/playwright-core/lib/grid/gridServer.js:356:5)
    at launchDockerContainer (~redacted/node_modules/@playwright/test/lib/cli.js:259:3)
    at Runner._run (~redacted/node_modules/@playwright/test/lib/runner.js:236:98)

Running 1 test using 1 worker

     test.ts:9:1 › description
Error fetching json: Error: connect ECONNREFUSED /var/run/docker.sock
Error:
╔═════════════════════════════════════════╗
║ Failed to list docker images            ║
║ Please ensure docker daemon is running. ║
║                                         ║
║ <3 Playwright Team                      ║
╚═════════════════════════════════════════╝
    at launchDockerGridAgent (~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.js:61:11)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at Object.launch (~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.j  ✘  test.ts:9:1 › description


  1) test.ts:9:1 › description =====

    'Grid agent creation failed'


  1 failed
    test.ts:9:1 › description ======
```

## After: No exact match found 

```
Using config at ~redacted/playwright.config.ts
Error:
╔════════════════════════════════════════════════════════════════════════════╗
║ Failed to find mcr.microsoft.com/playwright:v1.16.3-focal docker image.    ║
║                                                                            ║
║ Available images:                                                          ║
║ - mcr.microsoft.com/playwright:v1.16.3                                     ║
║ - mcr.microsoft.com/playwright:v1.16.2-focal                               ║
║                                                                            ║
║ Use available images via PWTEST_IMAGE_NAME environment variable:           ║
║     PWTEST_IMAGE_NAME=mcr.microsoft.com/playwright:v1.16.3 playwright test ║
║                                                                            ║
║ Alternatively, please pull docker image with the following command:        ║
║                                                                            ║
║     npx playwright install docker-image                                    ║
║                                                                            ║
║ <3 Playwright Team                                                         ║
╚════════════════════════════════════════════════════════════════════════════╝
    at launchDockerGridAgent (~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.js:92:11)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at Object.launch (~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.js:43:9)
    at GridServer.createAgent (~redacted/node_modules/playwright-core/lib/grid/gridServer.js:356:5)
    at launchDockerContainer (~redacted/node_modules/@playwright/test/lib/cli.js:259:3)
    at Runner._run (~redacted/node_modules/@playwright/test/lib/runner.js:236:98)

Running 1 test using 1 worker

     editor-editor-wysiwyg--editor-wysiwyg.vr.ts:9:1 › actual element matches expected element
Error:
╔════════════════════════════════════════════════════════════════════════════╗
║ Failed to find mcr.microsoft.com/playwright:v1.16.3-focal docker image.    ║
║                                                                            ║
║ Available images:                                                          ║
║ - mcr.microsoft.com/playwright:v1.16.3                                     ║
║ - mcr.microsoft.com/playwright:v1.16.2-focal                               ║
║                                                                            ║
║ Use available images via PWTEST_IMAGE_NAME environment variable:           ║
║     PWTEST_IMAGE_NAME=mcr.microsoft.com/playwright:v1.16.3 playwright test ║
║                                                                            ║
║ Alternatively, please pull docker image with the following command:        ║
║                                                                            ║
║     npx playwright install docker-image                                    ║
║                                                                            ║
║ <3 Playwright Team                                                         ║
╚════════════════════════════════════════════════════════════════════════════╝
    at launchDockerGridAgent (~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.js:92:11)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at Object.launch (~redacted/node_modules/playwright-core/lib/grid/dockerGridFactory.j  ✘  test.ts:9:1 › description


  1) test.ts:9:1 › description =====

    'Grid agent creation failed'


  1 failed
    test.ts:9:1 › description ======
```
2021-12-16 15:11:51 -08:00
Yury Semikhatsky
707befd6f7
fix(fetch): basic authentication without realm (#10979) 2021-12-16 13:40:52 -08:00
Yury Semikhatsky
921aa02ce4
feat(webkit): roll to r1588 (#10976) 2021-12-16 12:43:57 -08:00
Yury Semikhatsky
7c3629b72a
feat: export tagged pdf by default (#10915) 2021-12-16 11:46:02 -08:00
Yury Semikhatsky
0eaa19d5e7
docs(expect): APIResponse assertions (#10957) 2021-12-16 11:27:30 -08:00
github-actions[bot]
b606b5b8a0
feat(firefox): roll to r1311 (#10963)
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
2021-12-15 21:47:58 -08:00
Pavel Feldman
24ee6a8a1e
test: add screenshots to html tests (#10925) 2021-12-15 19:19:43 -08:00
Andrey Lushnikov
d379097107
feat(firefox-beta): roll Firefox-beta to 1311 (#10961) 2021-12-15 18:26:35 -08:00
Andrey Lushnikov
bff93f830e
devops: fix firefox packaging for Ubuntu 20.04 arm64 (#10962)
This was originally landed in
https://github.com/microsoft/playwright/pull/10091 but got accidentally
reverted by https://github.com/microsoft/playwright/pull/10760

References #10759
2021-12-15 18:10:26 -08:00
Yury Semikhatsky
fee98947d6
test(tracing): skip test in ff and wk (#10955) 2021-12-15 17:40:34 -08:00
Andrey Lushnikov
2957b7b013
feat(firefox-beta): roll to r1310 (#10954) 2021-12-15 17:33:09 -08:00
Yury Semikhatsky
fcccac0c08
fix(tracing): return early if no trace was produced (#10953)
This fixes [page/page-set-input-files.spec.ts](https://devops.aslushnikov.com/flakiness2.html#filter_spec=page%2Fpage-set-input-files.spec.ts&commits=50&test_parameter_filters=%5B%5B%22browserName%22%2C%5B%5B%22electron%22%2C%22exclude%22%5D%5D%5D%2C%5B%22platform%22%2C%5B%5B%22Android%22%2C%22exclude%22%5D%5D%5D%2C%5B%22video%22%2C%5B%5Btrue%2C%22exclude%22%5D%5D%5D%2C%5B%22trace%22%2C%5B%5Btrue%2C%22include%22%5D%5D%5D%5D&timestamp=1639590204582) failures.
2021-12-15 17:19:00 -08:00
Andrey Lushnikov
9b488f5a65
devops(firefox): do not bootstrap firefox-release ever on MacOS (#10960)
References #10759
2021-12-15 17:14:08 -08:00
Dmitry Gozman
192071d5bc
feat(test runner): save traces for beforeAll/afterAll hooks (#10950) 2021-12-15 16:06:10 -08:00
Yury Semikhatsky
230e0b7049
fix(wheel): throw on wheel access in mobile WebKit (#10946) 2021-12-15 11:57:28 -08:00
github-actions[bot]
0b86fbde90
feat(chromium): roll to r950899 (#10930)
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Max Schmitt <max@schmitt.mx>
2021-12-15 11:23:01 -08:00
Max Schmitt
c81c927e23
chore: make PR template more accessible (#10952) 2021-12-15 11:20:30 -08:00
Dmitry Gozman
19b08332ce
feat(test runner): disable test timeout on page.pause() call (#10944) 2021-12-15 11:12:52 -08:00
Andrey Lushnikov
acb75460bc
devops(firefox): fix packaging of mozilla-release (#10949)
`MOZ_AUTOMATION` env variable requires us to set a bunch of other
variables to package the build.
2021-12-15 11:12:44 -08:00
Max Schmitt
bfc6cc2701
chore: do not make HEAD request when opening remote traces (#10948) 2021-12-15 11:12:31 -08:00
Max Schmitt
4fd8a40129
chore: fix wrong aligned protocol YAML (#10947) 2021-12-15 11:12:13 -08:00
Yury Semikhatsky
ce225e50f4
test: lower screenshot count expectation (#10945) 2021-12-15 10:40:18 -08:00
Dmitry Gozman
0d54afab9c
feat(test runner): show beforeAll/afterAll hooks similar to tests (#10923)
Reporters now get notified about hooks start/end/steps.
2021-12-15 10:39:49 -08:00
Andrey Lushnikov
cb79cd88a9
browser(firefox): use pre-downloaded toolchains for release builds (#10929) 2021-12-15 08:31:33 -08:00
Andrey Lushnikov
3791065b8e
browser(firefox-beta): remove use of mach create-mach-environment (#10931)
This command has been removed and is no longer needed: abeedf3bbd (diff-080f2a5f770785d46fa3a2404de9430136a5a6c5a4739dab3c47cfbd7421ce79)
2021-12-15 00:28:20 -08:00
github-actions[bot]
8499d59bdc
feat(firefox-beta): roll to r1309 (#10926)
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
2021-12-14 23:27:23 -08:00
Andrey Lushnikov
66292a5895
browser(firefox-beta): roll Firefox beta to 96.0b5 (#10924)
Review URL: 4cedc704b1

Firefox 96 changes default sameSite cookie value, so this patch
updates tests to accomodate for this.
2021-12-14 23:24:25 -08:00
Pavel Feldman
f579f9c806
chore: parse tsx tests (#10917) 2021-12-14 19:25:07 -08:00
Pavel Feldman
04e82ce71c
feat(api): make withText an option (#10922) 2021-12-14 15:37:31 -08:00
Dmitry Gozman
34b84841b0
chore(test runner): create TestResult instances lazily in dispatcher (#10921)
This prepares for beforeAll/afterAll hooks to be handled in the same way.
Since we do not know in advance whether a hook will run, we must create
TestResults lazily.
2021-12-14 14:10:56 -08:00
Max Schmitt
b310c6bd6c
chore(dotnet): make JSHandle inherit from IAsyncDisposable (#10905) 2021-12-14 12:40:56 -08:00
Andrey Lushnikov
6552556608
devops: attempt to fix Firefox M95 release compilation (#10908)
Builders fail to download release toolchains; fallback to
LLVM from Homebrew on Mac.

References #10759
2021-12-14 00:49:35 -08:00
Andrey Lushnikov
2a8801be1e
browser(firefox): fix Firefox crashes (#10904)
Review URL: aff16fc8e4

This patch fixes 2 firefox crashers:
- color scheme override code was not used, but was called
  from multiple threads, which caused a weakptr use violation (cannot
  be used from multiple threads)
- snapshot listener callback was reset asynchronously, so when
  `HeadlessWindowCapturer` was destroyed, it was still occasionally
  called (see `HeadlessWindowCapturer::~HeadlessWindowCapturer`)

With this patch, I no londer experience tracing crashes in firefox.

References #10259
2021-12-13 21:23:25 -08:00
github-actions[bot]
93f7246f4e
browser(chromium): roll to r950899 (#10881)
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
2021-12-13 19:12:24 -08:00
Dmitry Gozman
5c4ebdce54
fix(line reporter): clarify about retries when going over total counter (#10901) 2021-12-13 19:06:13 -08:00