It's not possible to calculate the total test duration of the tests in
the UI, since our `msToString` function is lossy. This patch unflakes
the
[test](https://github.com/microsoft/playwright/actions/runs/6183832112/job/16787806162?pr=27074#step:7:254)
when it took longer than a second to start, by writing the test-ids to
the DOM, getting them into the Node.js process, and calculating their
test duration sum based on the JSON report.
Drive-by: I found a bug that we used the JSON report from the main
test-runner process rather than from the merge process, so the test IDs
were still old. (Without that change the `useIntermediateMergeReport`
tests were not passing.
A summary about the attempts we tried in
https://github.com/microsoft/playwright/pull/26931 for
https://github.com/microsoft/playwright/issues/26859 and
https://github.com/microsoft/playwright/issues/24591.
1. attempt: collect all the stdio from the worker host.
- this made it hard to capture the live console logs for the live trace
2. attempt: collect all the stdio from the worker host, but keep
proxying `process.{stdout,stderr}.write`.
- this solved the live console logs problem, since we could still listen
on them
- we found out about a race that the stdout/stderr streams can still
buffer data while the test already had finished
- we created a best effort flush implementation but didn't like the
complexity it introduced to the project
3. attempt: keep things like they are but also listen to stdio from the
worker host
- this solves the race of console.log in the test (things stay like they
are)
- this keeps live trace working (things stay like they are)
- a known pitfall is that it could result in logs which don't end up in
the reporter onStdOut/onStdErr, but it is already a significant
improvement over how it was before.
- before: they were not getting displayed
- after: they are getting displayed and there is a low likelihood that
they don't end up in the reporter API if the write happens slightly
before a test finished.
Closes#26931Fixes#26859Fixes#24591
- remove error details from the reports
- collapse flaky tests by default
- limit comment to 65365 character
GitHub API has comment length limit 65536 chars:
```
Unhandled error: HttpError: Validation Failed: {"resource":"IssueComment","code":"unprocessable","field":"data","message":"Body is too long (maximum is 65536 characters)"}
```
This way we can reuse it for:
- tracking `maxFailures` across phases;
- tracking failures for runner;
- tracking failures for `runJob` helper class later on.
Fixes#26344.
