playwright/browser_patches/buildbots

Setting Up Build Bots

We currently have 4 build bots that produce 6 builds

• [buildbot-linux] Ubuntu 18.04 machine
  • builds: Webkit-Linux, Firefox-Linux
• [buildbot-mac-10.14] Mac 10.14 machine
  • builds: WebKit-mac-10.14, Firefox-Mac
• [buildbot-mac-10.15] machine
  • builds: WebKit-mac-10.15
• [buildbot-windows] Windows 10 machine
  • builds: Firefox-win32, Firefox-win64

This document describes setting up bots infrastructure to produce browser builds.

Each bot configuration has 3 parts:

1. Setup toolchains to build browsers
2. Setup bot-specific environment required for bot operations

• azure-cli
• setting AZ_ACCOUNT_KEY, AZ_ACCOUNT_NAME, TELEGRAM_BOT_KEY env variables

3. Running relevant build script //browser_patches/buildbots/buildbot-*.sh using host scheduling system (cron on Linux, launchctl on Mac, polling on Win).

• Windows
  • Setting Up Browser Toolchains
  • Setting Bot Environment
  • Running Build Loop
• Mac
  • Setting Up Browser Toolchains
  • Setting Bot Environment
  • Running Build Loop
• Linux
  • Setting Up Browser Toolchains
  • Setting Bot Environment
  • Running Build Loop

Windows

Setting Up Browser Toolchains

We currently only build firefox on Windows. Follow instructions on Building Firefox for Windows. Get the checkout with mercurial and run "./mach bootstrap" from mercurial root.

After this step, you should have c:\mozilla-build folder and c:\mozilla-source folder with firefox checkout.

Setting Bot Environment

1. Install azure-cli

Install azure-cli for windows using MS Installer

2. Export "az" to the mingw world

Run cmd as administrator and run the following line:

> echo cmd.exe /c "\"C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\wbin\az.cmd\" $1 $2 $3 $4 $5 $6 $7 $8 $9 ${10} ${11} ${12} ${13} ${14} ${15} ${16}" > "%SYSTEMROOT%\az"

This command will create a c:\Windows\az file that will call azure-cli with passed parameters (Not the most beautiful solution, but it works!)

3. Set custom env variables to mingw env

Edit c:\mozilla-build\start-shell.bat and add the following lines in the beginning:

SET AZ_ACCOUNT_NAME=<account-name>
SET AZ_ACCOUNT_KEY=<account-key>
SET TELEGRAM_BOT_KEY=<bot_key>

change <account-name> and <account-key> with relevant keys/names.

NOTE: No spaces or quotes are allowed here!

4. Disable git autocrlf

Run c:\mozilla-build\start-shell.bat and run git config --global core.autocrlf false.

5. Checkout PlayWright to /c/

Run c:\mozilla-build\start-shell.bat and checkout PlayWright repo to /c/playwright.

Running Build Loop

1. Launch c:\mozilla-build/start-shell.bat
2. Run /c/playwright/browser_patches/buildbots/buildbot-windows.sh
3. Disable "QuickEdit" terminal mode to avoid terminal freezing and postponing builds

Mac

Setting Up Browser Toolchains

1. Install XCode from AppStore
2. Run XCode once and install components, if it requires any.
3. Install XCode command-line tools: xcode-select --install
4. Install homebrew: https://brew.sh/

Mac 10.14 builds both firefox and webkit, whereas we only build webkit on mac 10.15. Browser Toolchains:

• Building Firefox On Mac
• Building WebKit On Mac (though as of Dec, 2019 it does not require any additional steps)

Setting Bot Environment

1. Install azure-cli
2. Clone https://github.com/microsoft/playwright
3. Run //browser_patches/prepare_checkout.sh for every browser you care about
4. Make sure //browser_patches/{webkit,firefox}/build.sh works and compiles browsers

Running Build Loop

We use launchctl on Mac instead of cron since launchctl lets us run daemons even for non-logged-in users.

Create a /Library/LaunchDaemons/dev.playwright.plist with the contents below (will require sudo access). Make sure to change the following fields:

1. Set values for all keys in the EnvironmentVariables dict.
2. Put a proper path to the Program
3. Make sure to put correct UserName

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
  <dict>
    <key>Label</key>
    <string>dev.playwright</string>

    <key>Program</key>
    <string>/Users/aslushnikov/prog/cron/playwright/browser_patches/buildbots/buildbot-mac-10.14.sh</string>

    <key>UserName</key>
    <string>aslushnikov</string>

    <key>EnvironmentVariables</key>
    <dict>
      <key>PATH</key>
      <string>/usr/bin:/usr/sbin</string>

      <key>TELEGRAM_BOT_KEY</key>
      <string></string>

      <key>AZ_ACCOUNT_NAME</key>
      <string></string>

      <key>AZ_ACCOUNT_KEY</key>
      <string></string>

      <key>MOZ_NOSPAM</key>
      <string>1</string>
    </dict>


    <key>StandardOutPath</key>
    <string>/tmp/launchctl-playwright-buildbot.log</string>

    <key>StandardErrorPath</key>
    <string>/tmp/launchctl-playwright-buildbot.errorlog</string>

    <key>StartInterval</key>
    <integer>300</integer>
  </dict>
</plist>

Next, you can either use launchctl load command to load the daemon, or reboot bot to make sure it auto-starts.

Note

: mozbuild uses terminal-notifier which hangs in launchctl environment. The MOZ_NOSPAM env variable disables terminal notifications.

Finally, MacBooks tend to go to sleep no matter what their "energy settings" are. To disable sleep permanently on Macs (source):

sudo pmset -a sleep 0; sudo pmset -a hibernatemode 0; sudo pmset -a disablesleep 1;

Linux

Setting Up Browser Toolchains

1. Note: firefox binaries will crash randomly if compiled with clang 6. They do work when compiled with clang 9. To install clang 9 on ubuntu and make it default:

$ sudo apt-get install clang-9
$ sudo update-alternatives --install /usr/bin/clang++ clang++ /usr/bin/clang++-9 100
$ sudo update-alternatives --install /usr/bin/clang clang /usr/bin/clang-9 100

Note

: Firefox build config can be checked official Firefox builds, navigating to about:buildconfig URL.

Setting Bot Environment

TODO: instructions to set up linux bot

Running Build Loop

TODO: instructions to set up cron jobs