Abstract issue: I have parameterised tests via the titlePath (describe)
and they are not shown on the flakiness dashboard. Before #24486 only
title() was shown, after #24486 titlePath() will be displayed, but we
still did have the testId based on the test.title() and not
test.titlePath(). This ends up that they will still be treated as a
single spec.
After this change they are not treated as a single spec anymore and
treated as different ones:
<img width="891" alt="image"
src="https://github.com/microsoft/playwright/assets/17984549/f24284cd-5d94-4f7e-a45d-8c8e5eb537ef">
Note: This is tested.
Follow-up on https://github.com/microsoft/playwright/pull/24486.
JSON report will change again in https://github.com/microsoft/playwright/pull/6656
This patch modifies azure function to accomodate to the new format.
Note: with the new notion of projects instead of per-test parameters,
we can actually compress dashboard per commit much more efficient. If
the format sticks, we'll follow-up with a `dashboard_compressed_v2.js`.
New folio changed the JSON report, so we have to keep with the changes.
The most notable changes:
- there are no parameters any more. We recreate these as `testInfo.data`
arguments that are saved under test result's data.
- `test.runs` are renamed into `test.results`
I didn't find other changes so far - let's see if this works in the
cloud!
Compressed dashboard is 10 times smaller yet has all the data to
render flakiness.
Drive-by: remove old dashboard implementations since they are no
longer used.
Computing V1 flakiness dashboard was very expensive (>1 minute for AZ
function to run).
Disable it now in favor of V2 flakiness dashboard that proves to be very
reliable.
This patch:
- starts processing dashboards serially to avoid hitting node.js azure
function heap limit
- fixes typo in the new `dashboards_raw` processor
This refactors azure function and adds a new dashboard format implementation that is 15x smaller while tracking specs with non-empty annotations.
Old dashboard is still updated.
