Commit Graph

93 Commits

Author SHA1 Message Date
Pavel Feldman
06dcf967ba
feat(testrunner): allow external reporters (#3603) 2020-08-24 19:16:20 -07:00
Joel Einbinder
012f9425bf
chore(test-runner): move into its own folder and typescript project (#3548) 2020-08-20 16:04:27 -07:00
Pavel Feldman
30f4c0c9fe
test runner: remove dependencies on playwright (#3539) 2020-08-19 17:47:24 -07:00
Max Schmitt
e679b82339
fix(devops): auto roll tests (#3536) 2020-08-19 15:04:52 -07:00
Max Schmitt
8ea280090d
devops: first implementation of browser auto-roll bot (#3455)
Relates #3258

After a chat with @aslushnikov we add this in two iterations. The first one (this PR) is about scheduled runs for testing the latest tip-of-tree version of the browsers daily and the next PR is about opening an automated PRs once its passing (not sure if its worth to add for failing too) or sending Telegram/Slack notifications.

Current status is that Firefox (around 2 hours) and WebKit (around 3-4 hours) works.

Known issues which I fix before we merge:
- ~~Changes in `test/base.fixture.ts` will get extracted in #3453~~

Feel free to review, current blockers are before we can merge:
- ~~#3453~~
- potential git clone optimisation by aslushnikov
2020-08-17 11:23:46 -07:00
Pavel Feldman
262e886940
test: organize golden files under __snapshots__ folder (#3494) 2020-08-16 19:19:52 -07:00
Dmitry Gozman
9b52ca8676
chore: remove unused non-rpc code, test options, infra, bots (#3444) 2020-08-13 16:00:23 -07:00
Dmitry Gozman
84ca0120f8
test: upload output from headful bots (#3439) 2020-08-13 09:33:43 -07:00
Joel Einbinder
d3677357b8
feat(testrunner): take the first argument as the test root dir (#3423) 2020-08-12 20:01:13 -07:00
Yury Semikhatsky
f0fcdc8f51
test(firefox): make headful screencast tests work under xvfb (#3421) 2020-08-12 16:07:01 -07:00
Pavel Feldman
a4a07c46c3
test: support --grep, --forbid-only (#3417) 2020-08-12 14:57:37 -07:00
Pavel Feldman
962ddc09c4
test: consolidate runner files (#3415) 2020-08-12 13:47:44 -07:00
Pavel Feldman
7e07634cc6
test: use mocha in ci/cd (#3406) 2020-08-12 11:48:30 -07:00
Dmitry Gozman
8bb6d73b44
feat(rpc): keep non-rpc linux bots for now (#3381) 2020-08-10 20:49:53 -07:00
Ross Wollman
cbd33f9601
devops: avoid running publish on external contrib (#3257)
Everytime I sync my fork to upstream, I get failure notifications:

```
Run failed for master (3edfb2a)
Repository: rwoll/playwright
Workflow: devrelease
Duration: 10 minutes and 34.0 seconds
Finished: 2020-07-31 18:30:13 UTC
```

Since forks should never have the necessary secrets to publish
the npm and Docker packages, we limit the running of these jobs to avoid
noisy failures for contributors.
2020-07-31 14:09:36 -07:00
Andrey Lushnikov
421f6f48c7
devops: use playwright-github-action@v1 (#3221)
The playwright-github-action@v1 now points to v1.4.3, so we can
switch to it once again.
2020-07-30 17:47:39 -07:00
Andrey Lushnikov
e7ddf868bf
devops: rename docker image tag for tip-of-tree images (#3222)
We currently tag tip-of-tree docker images with `dev` tag. It'll
be much nicer to have consistent taggin with our `npm` which tags
with `next` tag.

This patch removes the `dev` tag and starts using the `next` tag instead
for docker images. Since we haven't announced `dev` tag support
anywhere, I think it's fine to remove it rather than have both `next`
and `dev`.
2020-07-30 11:45:54 -07:00
Andrey Lushnikov
77b1c4b883
devops: enable Ubuntu 20.04 tests (#3178) 2020-07-29 11:36:38 -07:00
Yury Semikhatsky
79ab07bd0d
devops: install Media Pack on Windows bot (#3137) 2020-07-23 18:06:16 -07:00
Dmitry Gozman
d234dac752
chore: support esm imports (#3125) 2020-07-23 15:14:36 -07:00
Joel Einbinder
773ee08e9d
chore(test): restore api coverage checks (#3068)
* chore(test): restore api coverage checks

* try to catch the correct error on the bots

* try to trick github into taking the new code

* uninstall coverage

* dgozman comments
2020-07-22 09:08:10 -07:00
Joel Einbinder
3c151d8f1d
fix(test): don't output babel's debug info on the bots (#3073) 2020-07-21 16:23:46 -07:00
Pavel Feldman
b5f9985d3a
devops: make headful a matrix, collect test results (#3027) 2020-07-18 10:50:58 -07:00
Andrey Lushnikov
7d2078ef6f
devops: bake browsers into Docker image (#2990)
This patch bakes browsers into docker image.

Important observations:
- We now re-build docker image everytime we roll browsers.
- Docker image size almost doubles: from `225MB` to `496MB`.

References #2926
2020-07-17 16:51:39 -07:00
Pavel Feldman
9d9801192e
test: wire test commands to jest (#3014) 2020-07-17 16:09:27 -07:00
Pavel Feldman
424f11d165
test: convert some tests to the jest+fixtures (#2983) 2020-07-16 16:59:45 -07:00
Pavel Feldman
ecc130c644
test: convert evaluation.spec to jest+fixtures (#2968) 2020-07-16 11:18:38 -07:00
Pavel Feldman
19cd96c46a
test: add the jest-circus experimental runner (#2962) 2020-07-15 16:52:41 -07:00
Andrey Lushnikov
d58a57c4b2
devops: fix docker publishing (#2939)
- Repository name has to start with `public/` to be exported to dockerhub.
- Remove `DOCKER_USERNAME` to avoid unnecessary masking in our logs.
- Publish dev versions of Docker image only when changes to docker file
happen. (this is why NPM publishing and Docker publishing for dev
releases are now separate).
- Release publishing in two separate jobs to make them independent.

References #2926
2020-07-13 17:15:10 -07:00
Andrey Lushnikov
b7d68d1cdb
devops: automate Docker image publishing (#2937)
This starts publishing our Docker images using Github Actions:
- on every commit, a new image will be pushed, tagged with a SHA of the
  commit, e.g.
  `mcr.microsoft.com/playwright:sha-fe997eca88d6de48378b05133dadfe032bd8b0ea`
- on every release, a new image will be pushed, tagged with the release
  version, e.g. `mcr.microsoft.com/playwright:v1.2.1`

**Note:** today we don't version our images and instead push them all as
`bionic`. This patch retains the tag - along with `latest`. Both
`bionic` and `latest` tags will point to the latest release.
2020-07-13 16:18:51 -07:00
Dmitry Gozman
2151757621
feat(rpc): run rpc tests in-process and out-of-process (#2929) 2020-07-13 15:00:20 -07:00
Dmitry Gozman
6d94c92053
feat(rpc): support no-serialization mode, run hook tests (#2925)
Added rpc_json_linux bots to excercise serialization - these
do not run hook tests.
2020-07-13 08:31:20 -07:00
Pavel Feldman
982e5e3550
devops: collect artifacts from browser locations (#2913) 2020-07-10 16:15:00 -07:00
Pavel Feldman
d561465214
devops: use matrix in GHA, add non-linux rpc (#2907) 2020-07-10 13:15:57 -07:00
Dmitry Gozman
5c4751d5dd
chore: generate protocol during browser roll (#2719)
We now commit protocol.ts files during the roll.
New utils/roll_browser.js helps with that.

This makes our installation very shallow:
- build installer;
- download browsers.
2020-07-01 15:22:29 -07:00
Andrey Lushnikov
6afb38d32f
devops: remove folder creation in github actions (#2779)
This should be no longer required.
2020-06-30 23:34:36 -07:00
Pavel Feldman
10a9eef832
chore(rpc): add a channel bot (#2773) 2020-06-30 18:51:36 -07:00
Yury Semikhatsky
22b4a6bd27
devops: enable coredumps for headful tests (#2678) 2020-06-23 17:18:29 -07:00
Yury Semikhatsky
49cb96c433
devops: enable core dumps in the subshell (#2555) 2020-06-11 23:58:24 -07:00
Yury Semikhatsky
48088222ed
devops: enable core dumps on Linux CI (#2552) 2020-06-11 16:41:58 -07:00
Andrey Lushnikov
37ec3a6ae6
fix(types): properly export typescript types from packages (#2364)
Drive-by: move package tests from `//test/installation-tests` to
`//packages/installation-tests`

Fix #2349
2020-05-26 17:19:05 -07:00
Dmitry Gozman
963dc72dd2
devops: add headful linux bot (#2060) 2020-05-04 15:15:51 -07:00
Ross Wollman
995c3eb4b5
devops: run install tests on common node versions (#2069)
Fixes #2059
2020-05-01 00:10:52 -07:00
Arjun Attam
c921cc19ec
chore: use xvfb from newer github action (#2035) 2020-04-29 15:07:21 -07:00
Andrey Lushnikov
0b7f789de4
devops: try using another Github Actions event to trigger releases (#1931)
The [official
documentation](https://help.github.com/en/actions/language-and-framework-guides/publishing-nodejs-packages#publishing-packages-to-the-npm-registry)
suggests using `created` event, but it didn't work for us.

As someone mentioned on [github
community](https://github.community/t5/GitHub-Actions/Workflow-set-for-on-release-not-triggering-not-showing-up/m-p/53236#M8758),
it looks like the `created` event is actually issued when release draft
is first created.

Try using
[`published`](https://help.github.com/en/actions/reference/events-that-trigger-workflows#release-event-release)
event instead.
2020-04-22 22:52:32 -07:00
Andrey Lushnikov
c474c540a3
devops: run CI against release branches too (#1910) 2020-04-21 19:20:19 -07:00
Andrey Lushnikov
0f338eced7
devops: introduce release publishing workflow (#1894)
This workflow is based on the Github Actions docs: [publishing node.js
packages](https://help.github.com/en/actions/language-and-framework-guides/publishing-nodejs-packages)

It should be triggered once we create a github release.
It's unclear how to test this workflow, so I tried to re-use as much
code from `publish_canary.yml` as possible.
2020-04-21 15:10:00 -07:00
Andrey Lushnikov
ac8a30c526
devops: start releasing from Github Actions (#1890)
This patch:
- removes releasing from Travis CI
- sets up a new GH Action that releases @next version from tip-of-tree

Once this GH Action proves to be working, we'll setup a `publish_release.yml`
workflow that will be triggered only by **release** github events
and that will publish released version with `LATEST` tag.

NOTE: this workflow does not actually run publishing - we're doing
`--dry-run` for now to see how it works in
`//utils/publish_all_packages.sh`.
2020-04-20 17:45:29 -07:00
Andrey Lushnikov
193582445d
devops: add package-lock.json (#1859) (#1889)
We need to commit `package-lock.json` to repository to ensure
reproducible builds.

This also starts using [`npm ci`](https://docs.npmjs.com/cli/ci.html) to
install dependencies.

Drive-by: remove non-functioning `rebase` workflow.
2020-04-20 16:57:01 -07:00
Pavel Feldman
92b6bc0eb9 Revert "devops: add package-lock.json (#1859)"
This reverts commit 75f35e48a8.

It is unclear how to add/remove dependencies now.
2020-04-18 22:27:25 -07:00