Yury Semikhatsky
bafa426231
feat(runner): support multiple names in project filter ( #8600 )
2021-09-02 09:29:55 -07:00
Dmitry Gozman
b1260602ac
docs: update global setup docs ( #8637 )
...
Changing example to "authenticate once", and also updating the auth doc.
Adding a test with the same setup.
2021-09-01 15:35:46 -07:00
Dmitry Gozman
4f4cf448c2
fix(test runner): generate unique outputDir for beforeAll/afterAll ( #8633 )
2021-09-01 13:41:35 -07:00
Dmitry Gozman
1a9215a6ec
fix(html reporter): do not throw when attachment is actually missing ( #8630 )
2021-09-01 12:20:28 -07:00
Dmitry Gozman
b8f8ca7493
feat(tracing): tracing.{start,stop}Chunk instead of tracing._export ( #8521 )
2021-08-31 17:03:31 -07:00
Dmitry Gozman
bee8ed117b
feat(html reporter): preview source code, steps and step errors ( #8598 )
2021-08-31 16:34:52 -07:00
Dmitry Gozman
900362ec0b
fix(test runner): report unhandled rejections during worker teardown ( #8592 )
2021-08-31 10:50:30 -07:00
Dmitry Gozman
37a897c9b4
fix(test runner): report beforeAll timeout instead of hanging ( #8529 )
...
We used to not report fatal error and hang forever because worker
did not run any tests but also did not report any errors.
Also properly show stack-less errors.
2021-08-28 07:19:45 -07:00
Michael Rienstra
ed0c47a06f
fix(test-runner): minor error message typo ( #8489 )
2021-08-27 10:15:16 +02:00
Dmitry Gozman
a479cb6f52
fix(test runner): add an overload for test.skip(title, fn) ( #8454 )
...
We shipped this feature, but forgot to add the right overload to d.ts.
2021-08-25 14:36:36 -07:00
Dmitry Gozman
de85d8bb83
fix(test runner): do not special case test.fail ( #8447 )
...
This makes `test.fail` tests considered as passing when they actually fail:
- Stop restarting the worker.
- Retry when it passes instead of a fail.
- Behaves similar to regular tests in a `describe.serial` suite.
2021-08-25 12:19:50 -07:00
Dmitry Gozman
75fb77355a
test(test runner): check retries with beforeAll failure and multiple tests ( #8413 )
2021-08-24 12:22:16 -07:00
Joel Einbinder
a8a3799e9d
fix(test-runner): use describe name in output dir ( #8282 )
2021-08-24 10:33:40 -04:00
Dmitry Gozman
f9b87268a7
fix(test runner): afterAll error should not mask beforeAll error ( #8358 )
2021-08-23 09:21:40 -07:00
Pavel Feldman
0997c13151
fix(test-runner): do not attach non-existent diff ( #8297 )
2021-08-20 13:40:27 -07:00
Dmitry Gozman
e5be2c9205
feat(test runner): show failure details for flaky test runs ( #8332 )
...
Currently, we just say "foo.spec.ts > my test" is flaky, but do not
show how exactly the failed run went.
2021-08-19 18:20:53 -07:00
Dmitry Gozman
70eca0401d
fix(test runner): avoid internal error for step end without begin ( #8327 )
...
fix(test runner): avoid internal error for step end without begin
Consider the following scenario:
- Test finishes and starts tearing down fixtures.
- Fixture teardown starts a step S and then times out.
- We declare the test finished (with timeout).
- Dispatcher shuts down the worker and spins a new one for a retry.
Additionally, it clears steps information for the test to be
ready for the new retry. Step S information is lost.
- Meanwhile, during worker teardown, the step S does
actually finish (usually with an error), and we send stepEnd for S.
- Dispatcher does not know what to do with step S end and
prints an internal error.
The fix is to ignore certain messages from the shutting down worker that failed.
2021-08-19 17:04:09 -07:00
Pavel Feldman
710cec80a0
feat(test-runner): render step titles ( #8270 )
2021-08-17 16:41:36 -07:00
Pavel Feldman
97ba4f22f3
feat(test-runner): report nested steps ( #8266 )
2021-08-17 13:57:26 -07:00
Dmitry Gozman
2aff06ec73
fix(test runner): make sure tracing is not running on non-retries ( #8232 )
...
When sharing a context between tests and using `'on-first-retry'` we
could end up with tracing still running in non-retried tests. That's
extra overhead without a reason.
2021-08-16 16:46:35 -07:00
Dmitry Gozman
d32d50a906
fix(test runner): make sure we always teardown all fixtures ( #8158 )
...
Even if one of the fixtures throws, we should teardown all of them
so that we can run afterAll hooks.
2021-08-12 09:08:56 -07:00
Pavel Feldman
d2d71c4cdb
fix(reporter): group fixture initialization under before hooks ( #8072 )
2021-08-12 07:58:00 -07:00
Dmitry Gozman
44cdda43fe
feat(test runner): show stdio for failures in terminal reporters ( #8150 )
2021-08-11 16:44:19 -07:00
Dmitry Gozman
052e0e197c
fix(types): allow specifying scope when overriding fixtures ( #8139 )
...
Otherwise it show a confusing error.
2021-08-11 10:44:15 -07:00
Dmitry Gozman
a5e0965087
feat(test runner): implement test.describe.serial ( #8132 )
2021-08-10 21:26:45 -07:00
Joel Einbinder
64da74fba8
feat(test-runner): allow non-ascii characters in the output dir path ( #8093 )
2021-08-10 21:24:35 -07:00
Dmitry Gozman
c9c1ea6546
fix(test runner): disallow use(workerFixture) in describes ( #8119 )
...
Using a worker fixture forces a new worker. This might be unexpected
when part of the test file runs in one worker, and another runs
in another worker. Top-level use of worker fixtures is still fine.
2021-08-10 16:32:32 -07:00
Dmitry Gozman
9d6c7cdf20
fix(test runner): ensure we run after hooks after failures ( #8102 )
2021-08-10 10:54:05 -07:00
Dmitry Gozman
2744cd6c9a
feat(test runner): remove createContext fixture ( #8109 )
...
We can now use `browser.newContext()` or `browser.newPage()` instead.
2021-08-10 09:26:36 -07:00
Dmitry Gozman
3bf3318350
fix(test runner): make sure options, trace and screenshot apply to all contexts ( #8083 )
...
- Uses some auto fixtures to set default options and instrumentation on BrowserType.
- Moves screenshot, trace and video to worker-scoped fixtures.
- Throws in page/context when used from beforeAll/afterAll.
- Plumbs around BrowserType to be accessible from Browser and BrowserContext.
2021-08-09 18:09:11 -07:00
Dmitry Gozman
e638c4597f
fix(test runner): do not swallow afterAll failure ( #8099 )
2021-08-09 14:21:53 -07:00
Dmitry Gozman
87548f94c1
feat(test runner): support test fixtures in beforeAll/afterAll ( #8082 )
...
Each hook gets its own test scope. This is not too useful for
object fixtures like `page` (although one can use a page in
`beforeAll` to save storage state), but much more useful for option
fixtures like `viewport`.
2021-08-09 13:26:33 -07:00
Dmitry Gozman
41949e559e
Revert "feat(test runner): file scope fixtures ( #7969 )" ( #8081 )
...
This reverts commit 1bbf86d060
2021-08-09 12:33:16 -07:00
Joel Einbinder
91394b257c
test(test-runner): add tests to show that grep is case insensitive ( #8091 )
2021-08-09 14:05:51 -04:00
Pavel Feldman
290f601dae
feat(expect): introduce explicit default async expect timeout ( #8071 )
2021-08-07 22:08:56 -07:00
Pavel Feldman
3424f59e67
chore(expect): polish matcher names, remote arguable ones ( #8060 )
2021-08-06 16:58:42 -07:00
Dmitry Gozman
5f297b6894
feat(junit reporter): add attachments to stdout ( #8059 )
...
`JUnitReporter` follows the common format for attachments in JUnit reports,
recognized by GitLab and Jenkins among others.
2021-08-06 15:47:54 -07:00
Dmitry Gozman
d846c05619
fix(test runner): make obtainWorker() resolve with null when stopping ( #8018 )
...
This ensures that we properly exit from `Dispatcher.run()`, print
epilogue and set the right exit code.
2021-08-05 15:00:00 -07:00
Dmitry Gozman
1bbf86d060
feat(test runner): file scope fixtures ( #7969 )
...
These are reset after running tests from a single file.
2021-08-04 21:11:02 -07:00
Dmitry Gozman
ea4f42b7ed
feat(report): add video to attachments ( #7976 )
2021-08-03 17:07:57 -07:00
Max Schmitt
385d489b35
feat(test-runner): re-enable web server ( #7906 )
...
Co-authored-by: Joel Einbinder <joel.einbinder@gmail.com>
2021-08-03 23:24:14 +02:00
Pavel Feldman
3b34e57ee4
feat(test-runner): introduce test.step ( #7956 )
2021-08-02 22:11:37 -07:00
Pavel Feldman
5803035c1b
feat(test-runner): introduce steps ( #7952 )
2021-08-02 17:17:20 -07:00
Pavel Feldman
af30d267b6
feat(test-runner): experiemental expect plumbing ( #7926 )
2021-07-30 16:07:02 -07:00
Dmitry Gozman
081b8683a3
feat(test runner): expect(locator) matchers to show a nice error on timeout ( #7935 )
2021-07-30 13:12:49 -07:00
Dmitry Gozman
5a3ebfc9f9
fix(test runner): dot reporter incorrectly splits by 80 ( #7925 )
2021-07-30 01:34:28 -07:00
Pavel Feldman
4163cec93b
feat(test-runner): introduce actionTimeout and navigationTimeout ( #7919 )
2021-07-29 21:03:50 -07:00
Dmitry Gozman
34c0c342fa
fix(test runner): make test order stable when fixtures are changing ( #7923 )
...
We used to sort based on workerHash, and that changes depending on
the exact worker fixtures list. Now we replace workerHash with
an ordinal when constructing the TestGroup list to preserve the
natural order.
2021-07-29 18:27:47 -07:00
Dmitry Gozman
40901e8b9a
feat(test runner): test.skip(title, testFunction) syntax ( #7922 )
2021-07-29 14:33:37 -07:00
Dmitry Gozman
dd0b089d13
feat(test runner): createContext fixture for multi-context scenarios ( #7779 )
2021-07-29 14:03:58 -07:00
Joel Einbinder
626dd23ce1
test(test-runner): unflake override-timeout.spec.ts ( #7869 )
2021-07-29 13:34:46 -07:00
Dmitry Gozman
82d2ec0b2d
test: unflake "max-failures should stop workers" ( #7918 )
2021-07-29 13:32:44 -07:00
Pavel Feldman
1807142eb7
feat(expect): even more matchers ( #7902 )
2021-07-29 07:33:19 -07:00
Pavel Feldman
3187ffdebf
feat(expect): add more matchers ( #7891 )
2021-07-28 15:44:44 -07:00
Dmitry Gozman
3f0485486d
feat(test runner): show errors from interrupted tests when available ( #7874 )
...
This shows the exact operation that is timing out (like click) when
user hits Ctrl+C.
2021-07-28 15:43:37 -07:00
Pavel Feldman
49e9f8c15e
feat(expect): add text and true matchers ( #7873 )
2021-07-28 12:07:11 -07:00
Pavel Feldman
b8dc0b9156
feat(expect): implement toMatchText ( #7871 )
2021-07-27 20:26:12 -07:00
Dmitry Gozman
da9b488d0d
fix(test-runner): sharding tests does not show a lot of skips ( #7708 )
2021-07-27 11:04:38 -07:00
Dmitry Gozman
cb978848d9
test(test runner): check custom reporter with .only tests ( #7862 )
2021-07-27 10:37:18 -07:00
Dmitry Gozman
cf886b3829
fix(test runner): align shard info to be one-based everywhere ( #7859 )
...
We used to treat shard from cli and shard from config differently.
2021-07-27 09:13:04 -07:00
Dmitry Gozman
bf6482a58d
fix(test runner): do not override browserName when using without --browser ( #7806 )
2021-07-23 09:04:20 -07:00
Dmitry Gozman
03ebe21323
faet(test runner): help when describe() is misused ( #7753 )
2021-07-22 12:34:37 -07:00
Joel Einbinder
483b7d6a0d
feat(test-runner): mark launch as experimental ( #7757 )
2021-07-21 09:51:27 -05:00
Joel Einbinder
20ba7c3a96
feat(test-runner): use require.resolve for globalSetup and globalTeardown ( #7752 )
2021-07-20 15:13:40 -05:00
Joel Einbinder
051dc332a6
feat(test-runner): use require to resolve reporters ( #7749 )
2021-07-20 15:03:01 -05:00
Max Schmitt
c84c5c8c9b
fix(test-runner): do not list tests to stdout when JSON reporter is used ( #7730 )
2021-07-20 01:10:43 +02:00
Joel Einbinder
0cf9cf0829
fix(test-runner): set expand: false for expect. ( #7722 )
2021-07-19 11:59:53 -05:00
Dmitry Gozman
66ea613c4d
feat(test-runner): small changes to Reporter api ( #7709 )
...
- `TestResult.startTime`
- `Suite.location` is optional now
- `Test.status()` renamed to `Test.outcome()` to differentiate against a
`Test.expectedStatus` and `TestResult.status` of the different type.
2021-07-18 17:40:59 -07:00
Dmitry Gozman
602d815981
fix(test-runner): list mode should print tests ( #7665 )
...
It was not doing anything before.
2021-07-16 22:34:55 -07:00
Dmitry Gozman
18be5f5319
feat(test-runner): suite per project ( #7688 )
...
This makes our suite structure the following:
```
Root(title='') > Project(title=projectName) > File(title=relativeFilePath) > ...suites > test
```
Removed `fullTitle()` because it is not used directly by anyone.
Default reporters now report each test as
```
[project-name] › relative/file/path.spec.ts:42:42 › suite subsuite test title
```
2021-07-16 15:23:50 -07:00
Pavel Feldman
bde764085c
feat(test-runner): introduce attachments ( #7685 )
2021-07-16 13:48:37 -07:00
Dmitry Gozman
31572fc372
chore(test-runner): misc changes to reporter api ( #7664 )
...
- `Location` with `file`, `line` and `column`.
- `fullTitle` does not include project name.
- `titlePath` method.
- All methods of `Reporter` are optional.
- Removed `Test.skipped` property that is superseeded by `Test.status()`.
- Replaced `Suite.findTest()` with `Suite.allTests()`.
- Removed `Test.suite` property.
2021-07-16 12:40:33 -07:00
Dmitry Gozman
8b2dd2e3d1
chore(test-runner): remove the notion of Spec ( #7661 )
...
We now have Suites and Tests. When running multiple projects the whole
suite is cloned for each project. Same happens for repeatEach.
This simplifies the reporters API, but there is still room for improvement.
JSON reporter continues to produce old json output.
2021-07-15 22:02:10 -07:00
Dmitry Gozman
bb34d7a953
fix(test-runner): sanitize snapshot name before constructing a path ( #7620 )
...
This avoids problems with `toMatchSnapshot('../../dir/file.png')`
where we append this path to `snapshotDir` and end up in some random
place.
Also added a note to documentation.
2021-07-14 16:31:19 -07:00
Max Schmitt
6cc2fe178e
feat(test-runner): migrate to launch config/server ( #7603 )
2021-07-15 01:19:45 +02:00
Max Schmitt
a26fe65db3
fix(test-runner): start webServer before globalSetup/teardown ( #7604 )
2021-07-14 19:01:46 +02:00
Dmitry Gozman
0742cb9076
chore: move working with browser channels to Registry Executables ( #7581 )
2021-07-13 19:03:49 -07:00
Dmitry Gozman
57c5e4d8cf
chore: replace Registry api with Executable list ( #7544 )
2021-07-13 15:57:40 -07:00
Yury Semikhatsky
5d62d01450
fix(runner): mark tests skipped due to sharding in junit report ( #7578 )
2021-07-13 05:06:08 -07:00
Joel Einbinder
eb31b9e4a9
feat(test-runner): support esm modules in more places ( #7542 )
2021-07-12 11:59:58 -05:00
Dmitry Gozman
65606c093a
chore: simplify Registry api ( #7451 )
2021-07-09 16:10:23 -07:00
Max Schmitt
bc2f161a55
chore: roll test-runner to 1.13.0-next-1625774143000 ( #7519 )
2021-07-08 23:36:11 -07:00
Yury Semikhatsky
f5b552fde7
test(runner): new worker after test failure ( #7518 )
2021-07-08 23:30:15 -07:00
Dmitry Gozman
77deca1d6b
feat(test runner): export testInfo.data ( #7525 )
...
This is a key-value storage for any information that goes into the report.
Also export JSONReport types.
2021-07-08 17:16:36 -07:00
Max Schmitt
e604f185ca
fix(test-runner): with automatic fixtures workerInfo was undefined after conditional skip ( #7521 )
2021-07-08 21:55:43 +02:00
Yury Semikhatsky
b2742976a8
fix(runner): do not restart worker after skipping tests ( #7511 )
2021-07-08 09:36:28 -07:00
Max Schmitt
ae489b1c43
fix(test-runner): do not override error with unhandled error ( #7507 )
2021-07-08 18:24:07 +02:00
Dmitry Zakharov
5464ad849e
fix(test-runner): handle negated toMatchSnapshot result ( #7345 )
2021-07-07 18:51:38 -07:00
Dmitry Gozman
2073193c36
fix(test runner): keep track of remaining tests on the runner side ( #7486 )
...
This fixes two issues:
- Sudden worker process exit is properly accounted for.
- We can stop() workers willy-nilly, e.g. after reaching maxFailures.
Details:
- DonePayload does not send `reamining` anymore, and worker does not track it.
- Instead, `Dispatcher._runJob` track remaining tests and acts accordingly.
- Upon worker exit, we emulate a fatal error for all remaining tests.
Drive-by:
- Do not report onTestBegin after reaching maxFailures to avoid confusion.
Before, we did report onTestBegin, but not onTestEnd.
- List reporter aligned between "running" and "finished" state - it was
one character misplaced.
- Added a SIGINT test.
2021-07-07 12:04:43 -07:00
Max Schmitt
98bcf26656
feat(test-runner): add webServer ( #7368 )
2021-07-07 20:19:42 +02:00
Dmitry Gozman
444d1eb51a
feat(test runner): run modifier functions once if they do not depend on test fixtures ( #7436 )
2021-07-02 15:49:05 -07:00
Joel Einbinder
7629587914
fix(test-runner): work with .mjs files ( #7373 )
2021-06-29 15:28:41 -07:00
Dmitry Gozman
6b3614fd4c
feat(test runner): support test.setTimeout for the whole block ( #7387 )
...
This enables `test.setTimeout()` outside of the test body, that
affects all tests in the block (either file or describe).
2021-06-29 13:33:13 -07:00
Joel Einbinder
368880962f
fix(test-runner): better test match glob ( #7382 )
2021-06-29 11:49:50 -07:00
Dmitry Gozman
6aefa02e91
feat(test runner): improve reporters api ( #7370 )
...
- onEnd may return a Promise
- onEnd now takes a result for the full run
- onTimeout is replaced with onEnd(result)
2021-06-29 10:55:46 -07:00
Max Schmitt
0776cf76a2
feat(test-runner): do only allow unique spec titles per suite ( #7300 )
2021-06-28 22:13:35 +02:00
Max Schmitt
bd86e70465
feat(test-runner): allow to focus a test in a location ( #7208 )
2021-06-24 10:02:34 +02:00
Dmitry Gozman
4c6fa42810
fix(test runner): expose real stack traces and speed up locations ( #7265 )
...
Stop wrapping/prepending error messages so that we do not loose the stack trace. For this, update a few manually thrown errors with better messages (usually including a file path).
Speed up locations by doing manual `sourceMapSupport.wrapCallSite()` for a single call site. Performance gain in the runner process with 100 files x 100 tests each:
- 25% on the fresh run without babel cache;
- 80% on the cached run where babel is almost instant.
Also some obvious cleanups around stack traces (removing unused code).
2021-06-23 10:30:54 -07:00
Max Schmitt
103f8ddd06
Revert "chore: update eslint config ( #6840 )" partly ( #7268 )
...
This reverts commit ec7d37d92f
2021-06-23 11:08:35 +02:00
Max Schmitt
99bbc51760
fix(test-runner): support ANSII terminals with list reporter ( #7258 )
2021-06-22 19:04:24 +02:00
Dmitry Gozman
6118d16edd
fix(test runner): properly keep track of requireFile to support helpers/wrappers ( #7243 )
...
This fixes an issue where we incorrectly labeled and assigned ids for tests
that declared tests in require'd files or used test wrappers.
See new tests for examples.
2021-06-21 11:25:15 -07:00
Pavel Feldman
ea4eebeb2d
feat(test-runner): document tagging, implement grep-invert ( #7227 )
2021-06-18 17:56:59 -07:00
Dmitry Gozman
388c6f5196
feature(test runner): capture error and pending calls when timing out ( #7193 )
2021-06-17 15:09:38 -07:00
Pavel Feldman
82a50b0e1d
fix(test-runner): property handle artifacts in context of preserveOutput ( #7181 )
2021-06-16 16:05:30 -07:00
Pavel Feldman
184f2c2e93
feat(test-runner): allow specifying video size ( #7158 )
2021-06-16 07:51:54 -07:00
Joel Einbinder
38e27c9c6c
fix(test-runner): case insensitive test filtering
2021-06-15 17:27:52 -07:00
Joel Einbinder
4c2a3fb443
fix(test-runner): accept relative paths for outputDir ( #7149 )
...
fixes #7124
2021-06-15 13:39:07 -07:00
Dmitry Gozman
742cce3a1d
feat(test runner): configurable reportSlowTests ( #7120 )
...
Also splits tests by projects and reports them with nice relative paths.
2021-06-14 22:45:58 -07:00
Joel Einbinder
aa72b2b9bb
fix(pwt): max-failures should work with retries ( #7127 )
...
fixes #7112
2021-06-14 22:16:16 -07:00
Pavel Feldman
970bb6a70d
feat(test-runner): allow setting pixel match threshold for project ( #7123 )
2021-06-14 21:52:10 -07:00
Max Schmitt
a3a9ccad6f
fix(test-runner): do not require esModuleInterop=true ( #7100 )
2021-06-14 21:58:10 +02:00
Andrey Lushnikov
8a8b3932f8
Revert "fix(test-runner): support passing slowMo option ( #6991 )" ( #7077 )
...
This reverts commit 178489d091
2021-06-11 16:19:50 -07:00
Pavel Feldman
49a8f67c0f
fix(test-runner): resolve global hooks relative to the config dir ( #7061 )
2021-06-10 22:31:27 -07:00
Max Schmitt
05382c997b
fix(test-runner): do only match JS/TS files when collecting ( #7014 )
2021-06-10 16:41:57 +02:00
Max Schmitt
178489d091
fix(test-runner): support passing slowMo option ( #6991 )
...
Fixes #6984
Reverts #6967
2021-06-09 14:02:05 -07:00
Dmitry Gozman
1d6ca5284c
feat(test runner): add tests for playwright-specific fixtures ( #6952 )
2021-06-08 15:52:08 -07:00
Andrey Lushnikov
37af9ce3cf
test(playwright-test): fix fixtures test ( #6967 )
...
🤷♂️
2021-06-08 13:10:15 -07:00
Dmitry Gozman
8c13f679b7
fix(test runner): remove folio/jest namespaces in expect matchers ( #6930 )
2021-06-07 08:02:01 -07:00
Dmitry Gozman
46a0213769
chore: remove internal uses of "folio" ( #6931 )
...
Replaced by "pwt" or "playwright test".
2021-06-06 22:07:07 -07:00
Dmitry Gozman
b556ee6f5b
chore: brush up playwright-test types ( #6928 )
2021-06-06 20:18:47 -07:00
Dmitry Gozman
f745bf1fbc
chore: bring in folio source ( #6923 )
...
- Source now lives at `src/test`.
- Former folio tests live at `tests/playwright-test`.
- We use `src/test/internal.ts` that exposes base test without
Playwright fixtures for most tests (to avoid modifications for now).
- Test types live in `types/testFoo.d.ts`.
- Stable test runner is installed to `tests/config/test-runner` during `npm install`.
- All deps including test-only are now listed in `package.json`.
Non-test deps must also be listed in `build_package.js` to get included.
2021-06-06 17:09:53 -07:00
