Broken since https://github.com/microsoft/playwright/pull/8049
Previously when doing `build.sh --full` it was installing deps + building. Currently when passing `--full` it does not install the deps before, since it tries to compile before.
Instead of building firefox using `//browser_patches/buid.sh ff-beta`,
one can use `//browser_patches/repack-juggler.mjs ff-beta`.
The script will download the last Playwright build, and repack
Juggler and preferences there.
These are just proxy scripts to the browsers, so you can do:
```sh
$ ./browser_patches/prepare_checkout.sh ff-beta
$ ./browser_patches/clean.sh ff-beta
$ ./browser_patches/build.sh ff-beta
$ ./browser_patches/export.sh ff-beta
```
This way bash won't expand and post-process variable values in any way.
The changes are driven with
[`shellcheck`](https://github.com/koalaman/shellcheck)
Both `libyuv::I420Copy` and `libyuv::I420Scale` support image cropping
by offsetting coordinates inside planes, but offsets must be even
numbers.
References #7998
Notable changes:
- javaScriptEnabled moved to the browser side.
- enabling new UI since it no longer helps to fix screencast
Review branch: https://github.com/aslushnikov/juggler/tree/roll-to-91-0-b10
Note: this will have 2 screencast tests failing. Investigation is in a follow-up.
Browsers require particular xcode versions to be installed on the host
machine to produce successful build.
This patch starts using explicit xcode versions for all mac browser
builds.
When Firefox decides to perform an http->https redirect based on HSTS
information, it issues an "internal" redirect and cancels the http request.
Since there is no response for the http request, we forge 307 redirect
in this case, following Chromium lead.
The relevant code is in nsHttpChannel::StartRedirectChannelToHttps.
With this change, the script is now 2-stage:
- first, prepare browser build with `repack-juggler.js --prepare`
- afterwards, run the script to repack tot juggler version with the
bubild
Chromium-with-symbols is the first time we produce a second set of
builds for the same browser.
This patch adds support for this usecase to report build progress
appropriately.
This patch adds a new script -
`//browser_patches/firefox/repack-juggler.mjs`
This script lets you test Firefox changes if they are Juggler-specific
without firefox checkout / compilation.
The `repack-juggler.mjs` script does the following:
- downloads the latest build of Firefox-with-juggler
- finds an `omni.ja` file with Juggler files inside
- replaces juggler in omni.ja with the current juggler files in the index
The new message is much more prominent in the logs:
```
aslushnikov:~/prog/playwright(make-deprecation-error-more-prominent)$ node a.js
node:internal/process/promises:246
triggerUncaughtException(err, true /* fromPromise */);
^
browserType.launch: Protocol error (Playwright.enable): Browser closed.
==================== Browser output: ====================
<launching> /Users/aslushnikov/prog/playwright/browser_patches/deprecated-webkit-mac-10.14/pw_run.sh --inspector-pipe --headless --no-startup-window
<launched> pid=5120
[pid=5120][out] ****************************************************************
[pid=5120][out] ****************************************************************
[pid=5120][out]
[pid=5120][out] ERROR: MacOS version is too old!
[pid=5120][out]
[pid=5120][out] This version of Playwright does not support running
[pid=5120][out] WebKit on MacOS 10.14. Please either:
[pid=5120][out] - update your operating system to version 10.15 or higher
[pid=5120][out] - use Playwright v1.11 or older
[pid=5120][out]
[pid=5120][out] ****************************************************************
[pid=5120][out] ****************************************************************
[pid=5120] <process did exit: exitCode=1, signal=null>
[pid=5120] starting temporary directories cleanup
```
Before that it actually didn't install all needed dependencies and failed after that when starting the compiler suite. WebKit also installs dependencies with apt, so it should be fine to install them on OS level.
- to read post data of requests, we have to read stream
- to restore the stream later on, we have to rewind it back
- however, if the stream is large enough, it cannot be rewound back
This patch starts cloning post data streams if possible to avoid
back-rewinding them later on.
References #4704
- unset `IFS` variable so that it doesn't affect future script execution
- use `\n\r` separator on windows
- fix script to be Python2/Python3 compliant
Every patch to Firefox should also go to firefox-stable. This patch
starts re-using Firefox-beta checkout for firefox-stable, making
possible to easily rebaseline work atop of firefox-stable.
With this patch, working on a patch in Firefox is a 2-step
process:
1. work on a patch against Firefox-Beta
2. rebaseline your work atop of Firefox-Stable
Working on Firefox-Beta is as usual:
- setup an up-to-date firefox checkout:
```sh
$ ./browser_patches/prepare_checkout.sh firefox
```
- create a new branch for your firefox work off `playwright-build`:
```sh
$ cd ./browser_patches/firefox/checkout
$ git checkout -b my-feature
```
- once work is done, export your branch:
```sh
$ ./browser_patches/export.sh firefox
```
Rebaselining your work for Firefox-Stable takes advantage of a single
checkout:
- prepare a firefox-stable checkout:
```sh
$ ./browser_patches/prepare_checkout.sh ff-stable
```
- rebaseline your feature branch atop of stable:
```sh
$ cd ./browser_patches/firefox/checkout
$ git checkout my-feature
$ git rebase -i playwright-build
```
- make sure firefox-stable compiles:
```sh
$ ./browser_patches/firefox-stable/build.sh
```
- export firefox-stable:
```sh
$ ./browser_patches/export.sh ff-stable
```
This adds a firefox-stable application to build on our bots.
This is basically a rebaselined version of 66541552d0
The firefox base revision is bb9bf7e886
Which is taken from `about://buildconfig` of a stable Firefox version
on Mac as of Apr 9, 2021.
References #5993
Merging conflicts:
038f367c30
After this roll, we no longer require our custom `onFrameLocationChanged` instrumentation
since gecko now fixed the `NOTIFY_LOCATION` behavior so that it works
for same-document navigations inside iframes.
Official WebKit no longer supports Mac 10.14. However, since
this system is still very much in use, we want to be able to
keep it running for a while.
This patch adds a new browser that we would compile and maintain
specifically for Mac 10.14: `deprecated-webkit-mac-10.14`. This
browser is a clone of Webkit r1443 that is the last known revision
to compile on Mac 10.14.
As we move on, we're free to modify this browser however we want,
backporting important patches.
References #5833
This fixes a bug where the script was creating a folder with a new line
symbol in its name. As a result, Windows was failing to remove this
folder, and Chromium Windows BuildBot was failing to even initialize
checkout.
browser(firefox): properly initialize debugging pipe on windows
Firefox on Windows has 2 launch modes:
- default: a special "launcher process" is used to start browser as a
sub-process
- non-default: browser process starts right away
Firefox has a logic to detect how successful was the use of the
launcher process to do self-recovery when things go wrong. Namely:
- when attempting to use launcher process, firefox records a timestamp
of the attempt beginning
- once the launcher process successfully launches browser sub-process,
firefox records another timestamp of the completion
On a new launch, firefox checks what timestamps are present. If there's
a timestamp that signifies start of launcher process, but no successful
timestamp, it decides that last "launcher process" use was not
successful and falls back to launching browser right away.
When launching 2 firefox processes right away, the first process
uses attempts to use launcher process and records the first timestamp.
At the same time, the second instance sees the first timestamp and
doesn't see the second timestamp, and falls back to launching browser
right away. Our debugging pipe code, however, does not support
non-launcher-process code path.
This patch adds support for remote debugging pipe in case of
non-launcher-process startup.
Drive-by:
- disable crashreporter altogether
- remove stray dcheck that breaks firefox debug compilation
- disable compilation of firefox update agent
- do not use WIN32_DISTRIB flag unless doing full builds since
it kills incremental compilation
References #4660
goma searches for all `GOMA_*` env variables and fails if it doesn't
understand any.
To avoid this, rename our `GOMA_LOGIN_COOKIE` env variable into a
`PLAYWRIGHT_GOMA_LOGIN_COOKIE`.
In our *old world*, browser buildbots were polling repo and were
trying to rebuild browsers every 5 minutes. They relied on the presence
of either build or build log on the CDN to avoid continiously
re-building.
In the new world, we trigger builds only when we change the
`BUILD_NUMBER` files, so there's no polling any more.
This patch removes the check for log presence so that those builds that
were failing due to misconfiguration (e.g. Visual Studio license
required updated) could be restarted just from the GitHub UI.
When we create the first page in the default context in headless mode on mac,
it gets NSWindow that is "not visible". Although we call [window setIsVisible:YES],
later on window.isVisible still returns NO.
We create our offscreen "headless" NSWindow directly from applicationDidFinishLaunching:.
Experiments show that delaying this by 100ms makes everything work. As a symptom,
we get applicationDidUnhide: notification that does not happen when we create the window
immediately.
Perhaps, we create the window too early, and there is some essential initialization
that happens after applicationDidFinishLaunching:. However, if we call
[NSApp activateIgnoringOtherApps:YES] like we do in headful mode, everything works.
The only solution that worked so far is creating the first page after a timeout.
Even though we're undergoing migration to GitHub self-hosted runners,
they don't currently support running natively under Arm. The resulting build ends up to be
x86_64.
See https://github.com/actions/runner/issues/805
Github self-hosted runners currently run under rosetta:
https://github.com/actions/runner/issues/805
This patch is an attempt to build arm webkit from-inside rosetta
shell on arm hardware.
`WK_CHECKOUT_PATH` defines location of webkit checkout on the
file system. All browser-related scripts, like `prepare_checkout.sh` and
`export.sh` respect this environment variable on all platforms.
**Preamble**
1. We're trying to setup a windows-based github self-hosted runner in the
playwright-internal repo.
1. Commands on Windows are mandated to have total arguments length
less then 32767 characters.
1. On windows, github self-hosted runner framework puts repository
checkout at `c:\w\playwright-internal\playwright-internal`
1. Our scripts create a checkout at
`c:\w\playwright-internal\playwright-internal\browser_patches\firefox\checkout`
1. One of the scripts in Firefox buildsystem tries to execute a command,
passing lots of absolute paths to various webidl's
1. The command fails due to restriction in (2)
**Problem**
Firefox build fails since checkout is deeply nested and hits max arg
size on windows.
**Solution**
This patch introduces a new variable `FF_CHECKOUT_PATH` that is
respected by all browser-related scripts. This way we'll be able
to checkout firefox to `c:\firefox` and avoid hitting long arguments
limit.
The `$PROGRAMFILES` variable is defined per the bitness of the
application: 32-bit shells get `C:\Program Files (x86)`, whereas
64-bit shells get `C:\Program Files`.
Visual Studio, however, has only 32-bit build and is always located in
the `C:\Program Files (x86)` folder.
This patch fixes `//browser_patches/firefox/build.sh` on 64-bit windows
shell.
browser(firefox): ensure detachedFromTarget is always sent
LinkedBrowser can throw when removing listeners in PageTarget.dispose,
and that prevents BrowserHandler from sending Browser.detachedFromTarget.
Using a try-catch seems good enough.
Currently, browser.version() returns `83.0`, whereas launching firefox
with `--version` flag returns `83.0b3`. This patch alings protocol's
`Browser.version()` with flag output.
Somehow, we get WebProgress state changes when worker is loaded
with a blob url. This messes up frame navigation detection.
Luckily, it's easy to filter out non-document state changes.
reverts #4357 and provides the real fix. Apple accidentally landed some code that will crash the web process if it is not signed by them. I'm sure they will figure it out once they get test bots for macos 11. But until then, we can just revert.
https://bugs.webkit.org/show_bug.cgi?id=218342
Instead of checking out the whole repository, we now do a shallow
clone.
We then gradually "unshallow" the clone, looking for the `BASE_REVISION`.
This should fix experimental mac-11 builder.
As Joel noticed recently, MessageManager in firefox doesn't guarantee
message delivery if the opposite end hasn't been initialized yet. In
this case, message will be silently dropped on the ground.
To fix this, we establish a handshake in SimpleChannel to make sure that
both ends are initialized, end buffer outgoing messages until this
happens.
Drive-by: serialize dialog events to only deliver *after* the
`Page.ready` protocol event. Otherwise, we deliver dialog events to the
unreported page.
This should fix current build failure on Mac.
5e1333ea77 is necessary to fix the following compilation error:
CMake Error at Source/cmake/WebKitFeatures.cmake:21 (message):
ENABLE_LEGACY_CSS_VENDOR_PREFIXES is not a valid WebKit option
Call Stack (most recent call first):
Source/cmake/WebKitFeatures.cmake:40 (_ENSURE_IS_WEBKIT_OPTION)
Source/cmake/OptionsWPE.cmake:89 (WEBKIT_OPTION_DEFAULT_PORT_VALUE)
Source/cmake/WebKitCommon.cmake:58 (include)
CMakeLists.txt:173 (include)
We try to avoid closing firefox too early, fearing that this
might terminate clean shutdown sequence.
Usually we assume that `Browser.enable` is called before `Browser.close`
- however, this is not the case in certain tests. So we have to
ensure browser initialization in `Browser.close` as well.
In my local tests, this fixes the Firefox Pipe problem: it looks like
we were closing browser too quickly.
It looks like terminating browser when search service or addon manager is
not fully initialized results in a broken shutdown sequence. As of
today, this results in multiple errors in the browser STDERR. In future,
this might also result in browser stalling instead of terminating.
This starts awaiting search and addon manager termination.
References #3995
Using WebProgressListener events works in all cases. Currently
used `pageshow` event will stop being emitted in future when loading
was stopped with `window.stop()` api.
References #3995
This patch:
- moves `SimpleChannel` to synchronously dispatch buffered commands
instead of a `await Promise.resolve()` hack
- moves dialog & screencast handling from `PageHandler` to
`TargetManager`. This leaves `PageHandler` to be concerned solely about
protocol.
- removes `attach` and `detach` methods for worker channels: since
channels are buffering messages until the namespace registers, there's
no chance to loose any events.
- slightly simplifies `PageNetwork` class: it's lifetime is now
identical to the lifetime of the associated `PageTarget`, so a lot can
be simplified later on.
References #3995
In the current tip-of-tree Firefox, document channel is enabled by
default, so we have to enable it in order to roll further.
This patch:
1. Removes content disposition sniffing from content process since it
crashes renderer with document channel.
2. Merges all page-related handlers in a single `PageHandler` and
serializes network events wrt the `Page.frameAttached` event.
The serialization mentioned in (2) is necessary: frame attachment is
reported from the content process, and network events are reported from
the browsers process. This is an inherent race, that becomes exposed by
the document channel.
On a side note, (2) makes it possible to synchronously report all
buffered events in `SimpleChannel` (cc offline discussion with @dgozman
that highlighted an unsighty approach that we currently employ there: reporting
events in a subsequent microtask.)
References #3995
This patch:
1. Changes `SimpleChannel` to buffer messages to the namespace that
hasn't been registered yet. This allows us to create `SimpleChannel`
per target on the browser side right away.
2. Removes multisession support. Now there's only one `PageAgent` in the
content process, which talks to a single `PageHandler` on the browser
side. Both ends can be created as-soon-as-needed; thanks to
`SimpleChannel` bufferring, no messages will be lost and all messages
will be delivered in proper order. (This is currently the reason why
build 1178 flakes on windows).
3. Straightens up the target reporting. Targets are reported as soon
as they appear on the browser side.
**NOTE:** this doesn't yet remove sessions from protocol.
References #3995
We currently might double-attach to the target in `BrowserHandler` since we iterate over all targets, and then subscribe to the additional event when target is getting initialized.
This patch fixes this race condition and should unblock the roll to r1177.
References #3995
As of today, we create `PageTarget` instances whenever we get a
sync IPC from the content process. This, however, breaks an invariant
that `browserContext.pages` always has all pages (and *browsing contexts* - not to be confused with *browser contexts*), associated with browser context. This invariant will be especially important when we move
user agent emulation to browser-side.
This patch makes `PageTarget` lifecycle symmetrical:
- `PageTarget` instance is created when tab is opened
- `PageTarget` is destroyed when tab is crashed or closed
This should also fix a bunch of race conditions with persistent mode, since sometimes we arrive to the window after its
initialization.
Drive-by: straighten viewport management and put a nice descriptive comment.
Juggler code had a bug where we subscribed to window and tab
events, but did not iterate collections of current windows and tabs.
As a result, we were sometimes failing to set viewport size for the
initial window, and implemented an artificial promise to workaround
the problem.
This patch:
- starts calling `onOpenWindow` and `onOpenTabListener` callbacks
for *all* windows and tabs - current and future, eliminating the
race condition.
This worked too well and we started overriding window sizes that
were set by users with `window.open(url, 'width=300;height=400')` (we
have a test for this). To fix this, we now plumb `CHROME_WITH_SIZE`
flag from appWindow and override viewport iff this flag is not set.
After this patch, we will use the `onTabOpened` event to move user
agent emulation to the browser-side.
References #3995
BrowsingContextIDs are consistent across the processes, so we can use
them to target frames in both browser and content processes. This will
aid browser-side navigation.
As a nice side-effect, we can drop a round-trip to the content process
for every `requestWillBeSent` event since we *almost* always can
attribute all network events to the proper parent frames.
I say "almost", because we in fact **fail** to correctly attribute requests
from workers that are instantiated by subframes. This, however, is
not working in Chromium ATM, so I consider this to be a minor regression
that is worth the simplification.
This patch:
- moves PrintDepsWindows folder to `//browser_patches/winldd`
- adds `build.sh`, `archive.sh`, `clean.sh` and `BUILD_NUMBER` to
power builds on buildbots
- starts building `winldd-win64` on windows buildbot
This patch:
- removes `--enable-gpl` and `--enable-version3` flags. This defaults
builds to LGPL2 license
- includes `--disable-autodetect` to ensure determenism
- includes extra version suffix to link from binary files back to these
build scripts
This patch moves FFMPEG building to buildbots:
- `ffmpeg-mac.zip` is built on Mac 10.14 machine
- `ffmpeg-win32.zip` and `ffmpeg-win64.zip` are cross-compiled on
Ubuntu 20.04 machine
All builds across the platforms share the same config:
- the same versions of `ffmpeg` and `libvpx`
- the same build configuration for both `ffmpeg` and `libvpx`
The config could be found in the `//browser_patches/ffmpeg/CONFIG.sh`.
The builds will be then copied manually and committed to the git
repository.
browser(firefox): fix automatic http->https redirect
Sometimes, Firefox does an automatic http->https redirect without hitting
the network (e.g. for http://wikipedia.org). In this case, the http request
is very strange:
- it does not actually hit the network;
- it is never intercepted;
- we cannot access its response because there was no actual response.
So, we had a bug where:
- redirects inherited the original request's listener;
- that listener was throwing an error.
This lead to the error in the listeners onDataAvailable call chain,
and original listener that renders the response was never called,
resulting in an empty page.
This change:
- ignores the original request that did not hit the network;
- does not inherit the listener;
- adds try/catch around problematic calls.
If there's no platform specified for the chromium build, we should
detect the host platform.
This will make it pleasant to verify Chromium rolls locally.
Assuming there's a `CR` env variable pointing to the local chromium,
rolling would look like this:
- bump a revision in `//browser_patches/chromium/BUILD_NUMBER`
- run `//browser_patches/chromium/build.sh`
- run tests with pulled chromium: `CRPATH=$CR npm run ctest`
This is an alternative approach to #3698 that was setting up a custom
mapping between chromium revisions and our mirrored builds. For example, we were
taking chromium `792639` and re-packaging it to our CDN as Chromium 1000.
One big downside of this opaque mapping was inability to quickly
understand which Chromium is mirrored to CDN.
To solve this, this patch starts treating browser revision as a fractional number,
with and integer part being a chromium revision, and fractional
part being our build number. For example, we can generate builds `792639`, `792639.1`,
`792639.2` etc, all of which will pick Chromium `792639` and re-package it to our CDN.
In the Playwright code itself, there are a handful of places that treat
browser revision as integer, exclusively to compare revision with some particular
revision numbers. This code would still work as-is, but I changed these places
to use `parseFloat` instead of `parseInt` for correctness.
Currently, we mirror Chromium revisions from gbuckets to our cdn and
name them same way as Chromium revision.
However, with the upcoming bundling of FFMPEG, we'd like to revision
Chromium packages separately, since our Chromium package will depend from
a number of factors:
- chromium upstream revision
- ffmpeg version
- extra files to add to the package or remove from the package
We should be able to produce a new Chromium build once any of these changes.
With this patch, to roll Chromium browser:
- update chromium revision number in the
`//browser_patches/chromium/UPSTREAM_CONFIG.sh`
- bump the build number in the `//browser_patches/chromium/BUILD_NUMBER`
Reference #3680
Firefox will sometimes send multiple requests with the same http channel id. When a frame is loaded, the favicon is requested in the parent frame, but with the same channel id. This can cause the document request to report the wrong frame, causing the test 'should capture iframe navigation request' to fail. It fails consistently on my computer.
This patch adds the content policy type into the http channelId to better distinguish requests. Maybe there is something better we can do? It looks like we use channelId has request ids, so there might be more bugs with these favicon requests in playwright?
* This changes the archive.sh script to use the webkit generate-bundle
tool to generate the MiniBrowser bundles instead of using our custom
logic.
* pw_run.sh is simplified because the new bundles include a wrapper
script that sets the needed LD_LIBRARY_PATH environment variables.
pw_run.sh now calls this wrapper script (for GTK or WPE) instead of
calling the MiniBrowser binary directly when running from a bundle.
Co-authored-by: Andrey Lushnikov <aslushnikov@gmail.com>
