feat(api): add explicit async testInfo.attach
We add an explicit async API for attaching file paths (and Buffers) to
tests that can be awaited to help users ensure they are attaching files
that actually exist at both the time of the invocation and later when
reporters (like the HTML Reporter) run and package up test artifacts.
This is intended to help surface attachment issues as soon as possible
so you aren't silently left with a missing attachment
minutes/days/months later when you go to debug a suddenly breaking test
expecting an attachment to be there.
NB: The current implemntation incurs an extra file copy compared to
manipulating the raw attachments array. If users encounter performance
issues because of this, we can consider an option parameter that uses
rename under the hood instead of copy. However, that would need to be
used with care if the file were to be accessed later in the test.
1. Fixtures defined in test.extend() can now have `{ option: true }` configuration that makes them overridable in the config. Options support all other properties of fixtures - value/function, scope, auto.
```
const test = base.extend<MyOptions>({
foo: ['default', { option: true }],
});
```
2. test.declare() and project.define are removed.
3. project.use applies overrides to default option values and nothing else. Any test.extend() and test.use() calls take priority over config options.
Required user changes: if someone used to define fixture options with test.extend(), overriding them in config will stop working. The solution is to add `{ option: true }`.
```
// Old code
export const test = base.extend<{ myOption: number, myFixture: number }>({
myOption: 123,
myFixture: ({ myOption }, use) => use(2 * myOption),
});
// New code
export const test = base.extend<{ myOption: number, myFixture: number }>({
myOption: [123, { option: true }],
myFixture: ({ myOption }, use) => use(2 * myOption),
});
```
Makes it easier to understand that expect does indeed have a separate timeout.
```
Error: expect(received).toHaveCount(expected) // deep equality
Expected: 0
Received: 1
Call log:
- expect.toHaveCount with timeout 500ms
- waiting for selector "span"
- selector resolved to 1 element
- unexpected value "1"
- selector resolved to 1 element
- unexpected value "1"
- selector resolved to 1 element
- unexpected value "1"
```
This patch:
- rolls stable-test-runner to Nov 2, 2021 tip-of-tree
- introduces a new npm script, `npm run vtest`, to run Visual Regression
Tests for our HTML reporter
Two bug fixes:
- Do not use the worker that is being shutdown for a new job.
- Report unhandled errors during "expected to fail" tests as
fatal errors.
