Resolves#11318.
* Adds `TestConfig.attachments` public API. (We opted to not implement an analog to the async `TestInfo.attach(…)` API.)
* Adds `TestConfig.attachments` to common reporters.
* Dogfoods some git and CI-info inference to generate useful atttachments
* Updates HTML Reporter to include a side bar to present a pre-defined set of attachments (a.k.a git/commit context sidebar)
Here's what it looks like:
<img width="1738" alt="Screen Shot 2022-03-21 at 3 23 28 PM" src="https://user-images.githubusercontent.com/11915034/159373291-8b937d30-fba3-472a-853a-766018f6b3e2.png">
See `tests/playwright-test/reporter-html.spec.ts` for an example of usage (for dogfood-ing only). In the future, if this becomes user-facing, there the Global Setup bit would likely become unnecessary (as would interaction with attachments array); there would likely just be a nice top-level config and/or CLI flag to enable collecting of info.
In several of the Playwright APIs, falsey values were not handled correctly. This changeset adds tests (and some fixes):
- route.continue: If options.postData was the empty string, the continue failed to override the post data.
- page.post (application/json with options.data: false|''|0|null): Raw falsey values were getting dropped (i.e. you can't do the equivalent of curl --header application/json … -d 'false'). This has been fixed with most values across all browsers, but an additional fix is needed for 'null' which the channel serializer treats extra specially.
- testInfo.attach: This didn't get reported as an error when options.path was the empty string, but should have been.
#11413 (and its fix#11414) inspired this search as they are the same
class of bug.
- Use file path, not content to calculate the attachment hash.
- Always cleanup fixture from the list on teardown, to avoid reporting
teardown error multiple times: from the test, and from the cleanup.
feat(api): add explicit async testInfo.attach
We add an explicit async API for attaching file paths (and Buffers) to
tests that can be awaited to help users ensure they are attaching files
that actually exist at both the time of the invocation and later when
reporters (like the HTML Reporter) run and package up test artifacts.
This is intended to help surface attachment issues as soon as possible
so you aren't silently left with a missing attachment
minutes/days/months later when you go to debug a suddenly breaking test
expecting an attachment to be there.
NB: The current implemntation incurs an extra file copy compared to
manipulating the raw attachments array. If users encounter performance
issues because of this, we can consider an option parameter that uses
rename under the hood instead of copy. However, that would need to be
used with care if the file were to be accessed later in the test.
