Commit Graph

69 Commits

Author SHA1 Message Date
Max Schmitt
e72e49ecc4 devops: #8253 follow-up fixes 2021-08-17 14:32:53 +02:00
Max Schmitt
e92924221a
devops: add workflow to roll new browser into PW (#8253) 2021-08-17 14:23:40 +02:00
Max Schmitt
e3060080cc
devops: include GH Action run URL in TG msg when browser build is failing (#8219) 2021-08-16 20:19:10 +02:00
Andrey Lushnikov
b1b4d7b819
chore: quote all bash variables when used (#8066)
This way bash won't expand and post-process variable values in any way.
The changes are driven with
[`shellcheck`](https://github.com/koalaman/shellcheck)
2021-08-07 05:32:18 -07:00
Andrey Lushnikov
04de4ad93e
devops: cleanup browser build configs (#7180)
- certain builds weren't listed in EXPECTED_BUILDS
- wekbit had one unused build configuration
2021-06-16 12:22:33 -07:00
Andrey Lushnikov
53ac9066da
devops: fix missing host in build scripts (#7178) 2021-06-16 10:38:00 -07:00
Andrey Lushnikov
312fd3ed4e
devops: fix reporting for chromium-with-symbols (#7159)
Chromium-with-symbols is the first time we produce a second set of
builds for the same browser.

This patch adds support for this usecase to report build progress
appropriately.
2021-06-15 16:01:11 -07:00
Andrey Lushnikov
bb5da5061c
devops: support building chromium with symbols (#7131)
References #7130
2021-06-15 01:18:00 -07:00
Andrey Lushnikov
12d8c54e99
chore: swap firefox-stable and firefox (#6950)
With this patch:
- `//browser_patches/firefox` becomes firefox-stable
- `//browser_patches/firefox-stable` becomes firefox-beta

References #6817
2021-06-07 17:05:51 -07:00
Andrey Lushnikov
17c6406e6c
devops: add firefox-stable channel browser (#6173)
This adds a firefox-stable application to build on our bots.
This is basically a rebaselined version of 66541552d0

The firefox base revision is bb9bf7e886
Which is taken from `about://buildconfig` of a stable Firefox version
on Mac as of Apr 9, 2021.

References #5993
2021-04-09 22:13:19 -07:00
Andrey Lushnikov
46949cd2df
devops: start doing separate builds for Firefox @ Ubuntu 20.04 (#6058)
This will fix webgl issues with headful firefox on Ubuntu 20.04

See
https://github.com/microsoft/playwright/pull/5951#issuecomment-8125735670
2021-04-02 14:16:25 -07:00
Andrey Lushnikov
8dc740570a
devops: refactor check_cdn.sh script (#5835)
Introduce `EXPECTED_BUILDS` list in each application folder and use
these lists when determining if all builds are ready.
2021-03-16 01:14:45 -07:00
Andrey Lushnikov
e64f66685a
devops: fork webkit into a separate browser (#5834)
Official WebKit no longer supports Mac 10.14. However, since
this system is still very much in use, we want to be able to
keep it running for a while.

This patch adds a new browser that we would compile and maintain
specifically for Mac 10.14: `deprecated-webkit-mac-10.14`. This
browser is a clone of Webkit r1443 that is the last known revision
to compile on Mac 10.14.

As we move on, we're free to modify this browser however we want,
backporting important patches.

References #5833
2021-03-16 01:08:21 -07:00
Andrey Lushnikov
a891becfba
chore: remove //browser_patches/buildbots folder (#5535)
Since we migrated all our builders/testers to the GitHub self-hosted
runners, we no longer need scripts to operate our own bots.
2021-02-22 09:09:19 -08:00
Andrey Lushnikov
e2a935b3d6
devops: fix nits in browser compilation infrastructure (#5526)
This patch:
- makes sure that failed runs actually show as failed in GHA
- attempts to use MINGW's `du` and `awk` to get upload size
2021-02-19 14:43:27 -08:00
Andrey Lushnikov
b2d9af5e15
browser(firefox): properly initialize debugging pipe on windows (#5514)
browser(firefox): properly initialize debugging pipe on windows

Firefox on Windows has 2 launch modes:
- default: a special "launcher process" is used to start browser as a
  sub-process
- non-default: browser process starts right away

Firefox has a logic to detect how successful was the use of the
launcher process to do self-recovery when things go wrong. Namely:
- when attempting to use launcher process, firefox records a timestamp
  of the attempt beginning
- once the launcher process successfully launches browser sub-process,
  firefox records another timestamp of the completion

On a new launch, firefox checks what timestamps are present. If there's
a timestamp that signifies start of launcher process, but no successful
timestamp, it decides that last "launcher process" use was not
successful and falls back to launching browser right away.

When launching 2 firefox processes right away, the first process
uses attempts to use launcher process and records the first timestamp.

At the same time, the second instance sees the first timestamp and
doesn't see the second timestamp, and falls back to launching browser
right away. Our debugging pipe code, however, does not support
non-launcher-process code path.

This patch adds support for remote debugging pipe in case of
non-launcher-process startup.

Drive-by:
- disable crashreporter altogether
- remove stray dcheck that breaks firefox debug compilation
- disable compilation of firefox update agent
- do not use WIN32_DISTRIB flag unless doing full builds since
  it kills incremental compilation


References #4660
2021-02-19 10:32:47 -08:00
Andrey Lushnikov
8c18b90038
devops: refactor chromium automation scripts (#5486)
Split preparing checkout, archiving and compiling into separate
scripts similarly how we do it with other browsers.
2021-02-17 14:43:19 -08:00
Andrey Lushnikov
21c24c2357
devops: do not check for logs existance when building browsers (#5367)
In our *old world*, browser buildbots were polling repo and were
trying to rebuild browsers every 5 minutes. They relied on the presence
of either build or build log on the CDN to avoid continiously
re-building.

In the new world, we trigger builds only when we change the
`BUILD_NUMBER` files, so there's no polling any more.

This patch removes the check for log presence so that those builds that
were failing due to misconfiguration (e.g. Visual Studio license
required updated) could be restarted just from the GitHub UI.
2021-02-08 16:19:37 -08:00
Andrey Lushnikov
f3a5bba2c4
devops: infra to automate chromium builds (#5347) 2021-02-07 23:54:10 -08:00
Andrey Lushnikov
b392c57a53
devops: attempt to install Firefox build deps on buildbots (#5271) 2021-02-02 21:35:12 -08:00
Andrey Lushnikov
51d90c593b
devops: support Chromium mac compilation (#5202) 2021-01-28 01:14:44 -08:00
Andrey Lushnikov
90bc837e55
devops: start compiling Chromium on Linux (#5166) 2021-01-27 08:42:35 -08:00
Andrey Lushnikov
ff75073cc8
devops: automation to compile chromium for mac arm64 (#5101)
References #5067
2021-01-22 11:27:40 +03:00
Andrey Lushnikov
7665a6ec7f
devops: support apple silicon builds of Firefox (#4979) 2021-01-12 03:57:59 +03:00
Andrey Lushnikov
b486e840ad
devops: revert ability to skip architecture enforcement (#4667) 2020-12-10 08:43:42 -08:00
Andrey Lushnikov
93c362de1b
devops: fix architecture enforcement (#4645) 2020-12-08 18:45:45 -08:00
Andrey Lushnikov
6d3278f1eb
devops: add ability to skip architecture enforcement (#4644)
Github self-hosted runners currently run under rosetta:
https://github.com/actions/runner/issues/805

This patch is an attempt to build arm webkit from-inside rosetta
shell on arm hardware.
2020-12-08 18:31:58 -08:00
Andrey Lushnikov
dd9c312b77
devops: start producing WebKit builds for Apple Silicon (#4643)
\m/
2020-12-08 17:15:44 -08:00
Andrey Lushnikov
4d8ef423ba
devops: add instructions to build mac on BigSur (#4350) 2020-11-05 00:49:05 -08:00
Andrey Lushnikov
b08d3dc9ff
devops: always ensure linux deps when building webkit (#4249) 2020-10-26 21:53:27 -07:00
Andrey Lushnikov
fda31dfca2
devops: build winldd on buildbots (#3917)
This patch:
- moves PrintDepsWindows folder to `//browser_patches/winldd`
- adds `build.sh`, `archive.sh`, `clean.sh` and `BUILD_NUMBER` to
  power builds on buildbots
- starts building `winldd-win64` on windows buildbot
2020-09-18 09:43:43 -07:00
Andrey Lushnikov
2a66f8a066
devops: build ffmpeg for linux (#3880) 2020-09-14 15:12:35 -07:00
Andrey Lushnikov
245d1001b1
devops: produce ffmpeg builds on bots (#3820)
This patch moves FFMPEG building to buildbots:
- `ffmpeg-mac.zip` is built on Mac 10.14 machine
- `ffmpeg-win32.zip` and `ffmpeg-win64.zip` are cross-compiled on
  Ubuntu 20.04 machine

All builds across the platforms share the same config:
- the same versions of `ffmpeg` and `libvpx`
- the same build configuration for both `ffmpeg` and `libvpx`

The config could be found in the `//browser_patches/ffmpeg/CONFIG.sh`.

The builds will be then copied manually and committed to the git
repository.
2020-09-09 17:05:08 -07:00
Andrey Lushnikov
2a0cbda8bc
devops: mirror chromium builds to our CDN (#3357)
Start mirroring chromium builds to our CDN.

References #3259
2020-08-07 16:22:05 -07:00
Andrey Lushnikov
678d16454a
devops: normalize blob names on the CDN (#3136)
This establishes a single naming for all our blobs with browser
builds that we upload to CDN: `<browser-name>-<os-version>`

- `<browser-name>` is either `firefox` or `webkit`.
- `os-version` is the OS that was used to produce the build.

References #2745
2020-07-24 00:31:18 -07:00
Andrey Lushnikov
c1032ae483
devops: simplify building webkit on linux bots (#3127)
Originally there was no way to build both webkit-gtk and webkit-wpe
in the same checkout. As a result, we were:
- building webkit-gtk & uploading it to CDN
- building webkit-wpe & uploading it to CDN
- downloading webkit-gtk & webkit-wpe from CDN and compiling a single
webkit-gtk-wpe folder that we uploaded to CDN

As of today, however, we can build WPE and GTK ports together. This
patch starts using this to drastically simplify build process:
- build webkit-gtk and webkit-wpe
- use `archive.sh` script that compiles a single folder with both ports
- upload archive to the cloud

This should also fix currently failing webkit build that fails to
assemble the `webkit-gtk-wpe` folder (I missed these codepaths while preparing
for Ubuntu-20.04 bot).

References #2745
2020-07-23 15:57:53 -07:00
Andrey Lushnikov
cb77d33a42
devops: add script for ubuntu 20.04 buildbot (#3123)
This will start producing Ubuntu 20.04 webkit builds

References #2745
2020-07-23 14:42:14 -07:00
Andrey Lushnikov
63689e36a6
devops: prepare buildbots to the introduction of Ubuntu 20.04 builder (#3116)
This patch:
- specializes "linux" scripts into "Ubuntu 18.04" scripts
- renames all future linux blobs on CDN:
    * `firefox-linux.zip => firefox-ubuntu-18.04.zip`
    * `minibrowser-gtk.zip => minibrowser-gtk-ubuntu-18.04.zip`
    * `minibrowser-wpe.zip => minibrowser-wpe-ubuntu-18.04.zip`
    * `minibrowser-gtk-wpe.zip => minibrowser-gtk-wpe-ubuntu-18.04.zip`
- updates downloader to deal with the new names

References #2745
2020-07-23 11:59:23 -07:00
Andrey Lushnikov
b1b6d3f531
devops: add signature to BUILD_NUMBER to force rebaseline (#2810)
Currently, it might happen that two different patches clash for the
same build number for the browsers. In this case, authors might
not even know that they need to rebaseline.

This patch starts adding a second line to `BUILD_NUMBER` files - the
signature and date of the `BUILD_NUMBER` change. These are guaranteed
to clash, so it should not be possible to land patches without
re-baselining them.
2020-07-02 12:09:27 -07:00
Andrey Lushnikov
de32d39988
Revert "devops: teach buildbots to run sanity check script (#2064)" (#2075)
This reverts commit 4c2c4855f7.
We'd rather add a mac 10.14 test bot.
2020-05-01 12:09:36 -07:00
Andrey Lushnikov
4c2c4855f7
devops: teach buildbots to run sanity check script (#2064)
This should make sure that build actually runs.
2020-04-30 17:42:52 -07:00
Andrey Lushnikov
b8259837a4 devops: disable previews on telegram bot messages 2020-04-20 04:24:18 -07:00
Andrey Lushnikov
e0d3e4811d devops: use node.js to gzip logs
This way we can keep streaming logs to STDOUT.
2020-04-20 02:52:26 -07:00
Andrey Lushnikov
ea95a9122d
devops: start uploading build logs to bots (#1870)
Logs will be saved next to the build archive, with the same name
and double extension `.log.zip`.
2020-04-20 01:25:25 -07:00
Yohix
bfd32fe6d2
doc: fix typos (#1284) 2020-03-07 08:37:47 -08:00
Andrey Lushnikov
b188f397cf devops: do not assume that checkout exists
We will prepare it later on, so the check itself is not needed.
2020-02-12 18:19:00 -08:00
Andrey Lushnikov
3acc65d285 devops: teach //browser_patches/webkit/build.sh to build both GTK & WPE
- use `//browser_patches/webkit/build.sh --gtk` to build GTK only
- use `//browser_patches/webkit/build.sh --wpe` to build WPE only
2020-02-07 13:57:08 -08:00
Andrey Lushnikov
871235939e devops: prettify telegram messages 2020-02-07 12:42:20 -08:00
Andrey Lushnikov
d62080ff1e devops: report upload build size via TG notification 2020-01-17 16:56:04 -08:00
Andrey Lushnikov
c517542f39 devops: send tg notifications for webkit-gtk-wpe 2020-01-17 16:37:49 -08:00