microsoft/playwright
1
1
Fork 0
mirror of https://github.com/microsoft/playwright.git synced 2025-01-05 19:04:43 +03:00
Code Issues Projects Releases Wiki Activity

chore: remove --tag and tagFilter (#29468)

Browse Source 
This makes new tags filtered by the `grep`. New `TestCase.tags` API now
includes both old and new tags.
This commit is contained in:
Dmitry Gozman 2024-02-15 11:37:16 -08:00 committed by GitHub
parent fb48bfcbe6
commit bd5403dcad
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
21 changed files with 224 additions and 380 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

46
docs/src/test-annotations-js.md
View File

@ -13,7 +13,7 @@ You can add your own tags and annotations at any moment, but Playwright comes wi
- [`method: Test.fixme`] marks the test as failing. Playwright will not run this test, as opposed to the `fail` annotation. Use `fixme` when running the test is slow or crashes.
- [`method: Test.slow`] marks the test as slow and triples the test timeout.
Annotations can be used on a single test or a group of tests.
Annotations can be added to a single test or a group of tests.
Built-in annotations can be conditional, in which case they apply when the condition is truthy, and may depend on test fixtures. There could be multiple annotations on the same test, possibly in different configurations.
@ -67,7 +67,9 @@ test.describe('two tests', () => {
## Tag tests
Sometimes you want to tag your tests as `@fast` or `@slow`, and then filter by tag in the test report. Or you might want to only run tests that have a certain tag. To do this, provide additional details when declaring a test.
Sometimes you want to tag your tests as `@fast` or `@slow`, and then filter by tag in the test report. Or you might want to only run tests that have a certain tag.
To tag a test, either provide an additional details object when declaring a test, or add `@`-token to the test title. Note that tags must start with `@` symbol.
```js
import { test, expect } from '@playwright/test';
@ -78,9 +80,7 @@ test('test login page', {
// ...
});
test('test full report', {
tag: '@slow',
}, async ({ page }) => {
test('test full report @slow', async ({ page }) => {
// ...
});
```
@ -105,49 +105,56 @@ test.describe('group', {
});
```
You can now run tests that have a particular tag.
You can now run tests that have a particular tag with [`--grep`](./test-cli.md#reference) command line option.
```bash tab=bash-bash
npx playwright test --tag @fast
npx playwright test --grep @fast
```
```powershell tab=bash-powershell
npx playwright test --tag "@fast"
npx playwright test --grep "@fast"
```
```batch tab=bash-batch
npx playwright test --tag @fast
npx playwright test --grep @fast
```
Or if you want the opposite, you can skip the tests with a certain tag:
```bash tab=bash-bash
npx playwright test --tag "not @fast"
npx playwright test --grep-invert @fast
```
```powershell tab=bash-powershell
npx playwright test --tag "not @fast"
npx playwright test --grep-invert "@fast"
```
```batch tab=bash-batch
npx playwright test --tag "not @fast"
npx playwright test --grep-invert @fast
```
The `--tag` option supports logical tag expressions. You can use `and`, `or` and `not` operators, as well as group with parenthesis. For example, to run `@smoke` tests that are either `@slow` or `@very-slow`:
To run tests containing either tag (logical `OR` operator):
```bash tab=bash-bash
npx playwright test --tag "@smoke and (@slow or @very-slow)"
npx playwright test --grep "@fast|@slow"
```
```powershell tab=bash-powershell
npx playwright test --tag "@smoke and (@slow or @very-slow)"
npx playwright test --grep --% "@fast^|@slow"
```
```batch tab=bash-batch
npx playwright test --tag "@smoke and (@slow or @very-slow)"
npx playwright test --grep "@fast^|@slow"
```
You can also filter tests in the configuration file via [`property: TestConfig.tagFilter`] and [`property: TestProject.tagFilter`].
Or run tests containing both tags (logical `AND` operator) using regex lookaheads:
```bash
npx playwright test --grep "(?=.*@fast)(?=.*@slow)"
```
You can also filter tests in the configuration file via [`property: TestConfig.grep`] and [`property: TestProject.grep`].
## Annotate tests
@ -233,9 +240,9 @@ test('user profile', async ({ page }) => {
});
```
## Dynamic annotations
## Runtime annotations
While the test is running, you can add dynamic annotations to [`test.info().annotations`](./api/class-testinfo#test-info-annotations).
While the test is already running, you can add annotations to [`test.info().annotations`](./api/class-testinfo#test-info-annotations).
```js title="example.spec.ts"
@ -245,6 +252,7 @@ test('example test', async ({ page, browser }) => {
type: 'browser version',
description: browser.version(),
});
// ...
});
```

26
docs/src/test-api/class-test.md
View File

@ -35,7 +35,7 @@ test('basic test', async ({ page }) => {
**Tags**
You can tag tests by providing additional test details. Note that each tag must start with `@` symbol.
You can tag tests by providing additional test details. Alternatively, you can include tags in the test title. Note that each tag must start with `@` symbol.
```js
import { test, expect } from '@playwright/test';
@ -46,13 +46,18 @@ test('basic test', {
await page.goto('https://playwright.dev/');
// ...
});
test('another test @smoke', async ({ page }) => {
await page.goto('https://playwright.dev/');
// ...
});
```
Test tags are displayed in the test report, and are available to a custom reporter via `TestCase.tags` property.
You can also filter tests by their tags during test execution:
* in the [command line](../test-cli.md#reference);
* in the config with [`property: TestConfig.tagFilter`] and [`property: TestProject.tagFilter`];
* in the config with [`property: TestConfig.grep`] and [`property: TestProject.grep`];
Learn more about [tagging](../test-annotations.md#tag-tests).
@ -76,7 +81,7 @@ test('basic test', {
Test annotations are displayed in the test report, and are available to a custom reporter via `TestCase.annotations` property.
You can also add dynamic annotations by manipulating [`property: TestInfo.annotations`].
You can also add annotations during runtime by manipulating [`property: TestInfo.annotations`].
Learn more about [test annotations](../test-annotations.md).
@ -1033,11 +1038,14 @@ An object containing fixtures and/or options. Learn more about [fixtures format]
Marks a test as "should fail". Playwright runs this test and ensures that it is actually failing. This is useful for documentation purposes to acknowledge that some functionality is broken until it is fixed.
To declare a "failing" test:
* `test.fail(title, body)`
* `test.fail(title, details, body)`
* `test.fail()`
To annotate test as "failing" at runtime:
* `test.fail(condition, description)`
* `test.fail(callback, description)`
* `test.fail()`
**Usage**
@ -1135,11 +1143,14 @@ Optional description that will be reflected in a test report.
Mark a test as "fixme", with the intention to fix it. Playwright will not run the test past the `test.fixme()` call.
To declare a "fixme" test:
* `test.fixme(title, body)`
* `test.fixme(title, details, body)`
* `test.fixme()`
To annotate test as "fixme" at runtime:
* `test.fixme(condition, description)`
* `test.fixme(callback, description)`
* `test.fixme()`
**Usage**
@ -1354,11 +1365,14 @@ Skip a test. Playwright will not run the test past the `test.skip()` call.
Skipped tests are not supposed to be ever run. If you intent to fix the test, use [`method: Test.fixme`] instead.
To declare a skipped test:
* `test.skip(title, body)`
* `test.skip(title, details, body)`
* `test.skip()`
To skip a test at runtime:
* `test.skip(condition, description)`
* `test.skip(callback, description)`
* `test.skip()`
**Usage**

24
docs/src/test-api/class-testconfig.md
View File

@ -475,30 +475,6 @@ export default defineConfig({
```
## property: TestConfig.tagFilter
* since: v1.42
- type: ?<[string]>
Filter to only run tests with or without particular tag(s). This property must be a logical expression containing tags, parenthesis `(` and `)`, and operators `and`, `or` and `not`.
Learn more about [tagging](../test-annotations.md#tag-tests).
**Usage**
```js title="playwright.config.ts"
import { defineConfig } from '@playwright/test';
export default defineConfig({
tagFilter: '@smoke',
});
```
Tag expression examples:
* `tagFilter: '@smoke'` - run only "smoke" tests;
* `tagFilter: '@smoke and not @production'` - run only "smoke" tests that are not tagged as "production";
* `tagFilter: '(@smoke or @fast) and @v2'` - run "v2" tests that are tagged as "smoke", "fast" or both.
## property: TestConfig.testDir
* since: v1.10
- type: ?<[string]>

28
docs/src/test-api/class-testproject.md
View File

@ -200,34 +200,6 @@ Use [`method: Test.describe.configure`] to change the number of retries for a sp
Use [`property: TestConfig.retries`] to change this option for all projects.
## property: TestProject.tagFilter
* since: v1.42
- type: ?<[string]>
Filter to only run tests with or without particular tag(s). This property must be a logical expression containing tags, parenthesis `(` and `)`, and operators `and`, `or` and `not`.
Learn more about [tagging](../test-annotations.md#tag-tests).
**Usage**
```js title="playwright.config.ts"
import { defineConfig } from '@playwright/test';
export default defineConfig({
projects: [
{
name: 'tests',
tagFilter: '@smoke',
},
],
});
```
Tag expression examples:
* `tagFilter: '@smoke'` - run only "smoke" tests;
* `tagFilter: '@smoke and not @production'` - run only "smoke" tests that are not tagged as "production";
* `tagFilter: '(@smoke or @fast) and @v2'` - run "v2" tests that are tagged as "smoke", "fast" or both.
## property: TestProject.teardown
* since: v1.34
- type: ?<[string]>

2
docs/src/test-cli-js.md
View File

@ -84,7 +84,7 @@ Complete set of Playwright Test options is available in the [configuration file]
| `--debug`| Run tests with Playwright Inspector. Shortcut for `PWDEBUG=1` environment variable and `--timeout=0 --max-failures=1 --headed --workers=1` options.|
| `-c <file>` or `--config <file>`| Configuration file. If not passed, defaults to `playwright.config.ts` or `playwright.config.js` in the current directory. |
| `--forbid-only` | Whether to disallow `test.only`. Useful on CI.|
| `-g <grep>` or `--grep <grep>` | Only run tests matching this regular expression. For example, this will run `'should add to cart'` when passed `-g "add to cart"`. The regular expression will be tested against the string that consists of the test file name, `test.describe` name (if any) and the test name divided by spaces, e.g. `my-test.spec.ts my-suite my-test`. The filter does not apply to the tests from dependcy projects, i.e. Playwright will still run all tests from [project dependencies](./test-projects.md#dependencies). |
| `-g <grep>` or `--grep <grep>` | Only run tests matching this regular expression. For example, this will run `'should add to cart'` when passed `-g "add to cart"`. The regular expression will be tested against the string that consists of the test file name, `test.describe` titles if any, test title and all test tags, separated by spaces, e.g. `my-test.spec.ts my-suite my-test @smoke`. The filter does not apply to the tests from dependcy projects, i.e. Playwright will still run all tests from [project dependencies](./test-projects.md#dependencies). |
| `--grep-invert <grep>` | Only run tests **not** matching this regular expression. The opposite of `--grep`. The filter does not apply to the tests from dependcy projects, i.e. Playwright will still run all tests from [project dependencies](./test-projects.md#dependencies).|
| `--global-timeout <number>` | Total timeout for the whole test run in milliseconds. By default, there is no global timeout. Learn more about [various timeouts](./test-timeouts.md).|
| `--list` | list all the tests, but do not run them.|

2
docs/src/test-reporter-api/class-testcase.md
View File

@ -86,7 +86,7 @@ Learn more about [test retries](../test-retries.md#retries).
* since: v1.42
- type: <[Array]<[string]>>
The list of tags defined on the test or suite via [`method: Test.(call)`] or [`method: Test.describe`].
The list of tags defined on the test or suite via [`method: Test.(call)`] or [`method: Test.describe`], as well as `@`-tokens extracted from test and suite titles.
Learn more about [test tags](../test-annotations.md#tag-tests).

6
packages/html-reporter/src/labelUtils.tsx
View File

@ -25,17 +25,11 @@ export function testCaseLabels(test: TestCaseSummary): string[] {
if (test.botName)
labels.push('@' + test.botName);
labels.push(...test.tags);
labels.push(...matchTags(test.path.join(' ') + ' ' + test.title).sort((a, b) => a.localeCompare(b)));
(test as any)[labelsSymbol] = labels;
}
return (test as any)[labelsSymbol];
}
// match all tags in test title
function matchTags(title: string): string[] {
return title.match(/@([\S]+)/g) || [];
}
// hash string to integer in range [0, 6] for color index, to get same color for same tag
export function hashStringToInt(str: string) {
let hash = 0;

3
packages/playwright/src/common/config.ts
View File

@ -46,7 +46,6 @@ export class FullConfigInternal {
cliArgs: string[] = [];
cliGrep: string | undefined;
cliGrepInvert: string | undefined;
cliTagFilter: string | undefined;
cliProjectFilter?: string[];
cliProjectGrep?: string;
cliListOnly = false;
@ -166,7 +165,6 @@ export class FullProjectInternal {
id = '';
deps: FullProjectInternal[] = [];
teardown: FullProjectInternal | undefined;
tagFilter: string | undefined;
constructor(configDir: string, config: Config, fullConfig: FullConfigInternal, projectConfig: Project, configCLIOverrides: ConfigCLIOverrides, throwawayArtifactsPath: string) {
this.fullConfig = fullConfig;
@ -195,7 +193,6 @@ export class FullProjectInternal {
};
this.fullyParallel = takeFirst(configCLIOverrides.fullyParallel, projectConfig.fullyParallel, config.fullyParallel, undefined);
this.expect = takeFirst(projectConfig.expect, config.expect, {});
this.tagFilter = takeFirst(projectConfig.tagFilter, config.tagFilter, undefined);
if (this.expect.toHaveScreenshot?.stylePath) {
const stylePaths = Array.isArray(this.expect.toHaveScreenshot.stylePath) ? this.expect.toHaveScreenshot.stylePath : [this.expect.toHaveScreenshot.stylePath];
this.expect.toHaveScreenshot.stylePath = stylePaths.map(stylePath => path.resolve(configDir, stylePath));

9
packages/playwright/src/common/suiteUtils.ts
View File

@ -61,11 +61,10 @@ export function bindFileSuiteToProject(project: FullProjectInternal, suite: Suit
// Inherit properties from parent suites.
let inheritedRetries: number | undefined;
let inheritedTimeout: number | undefined;
test.annotations = [];
for (let parentSuite: Suite | undefined = suite; parentSuite; parentSuite = parentSuite.parent) {
if (parentSuite._staticAnnotations.length)
test._staticAnnotations = [...parentSuite._staticAnnotations, ...test._staticAnnotations];
if (parentSuite._tags.length)
test.tags = [...parentSuite._tags, ...test.tags];
test.annotations = [...parentSuite._staticAnnotations, ...test.annotations];
if (inheritedRetries === undefined && parentSuite._retries !== undefined)
inheritedRetries = parentSuite._retries;
if (inheritedTimeout === undefined && parentSuite._timeout !== undefined)
@ -73,10 +72,10 @@ export function bindFileSuiteToProject(project: FullProjectInternal, suite: Suit
}
test.retries = inheritedRetries ?? project.project.retries;
test.timeout = inheritedTimeout ?? project.project.timeout;
test.annotations = [...test._staticAnnotations];
test.annotations.push(...test._staticAnnotations);
// Skip annotations imply skipped expectedStatus.
if (test._staticAnnotations.some(a => a.type === 'skip' || a.type === 'fixme'))
if (test.annotations.some(a => a.type === 'skip' || a.type === 'fixme'))
test.expectedStatus = 'skipped';
// We only compute / set digest in the runner.

31
packages/playwright/src/common/test.ts
View File

@ -48,7 +48,9 @@ export class Suite extends Base implements SuitePrivate {
_hooks: { type: 'beforeEach' | 'afterEach' | 'beforeAll' | 'afterAll', fn: Function, title: string, location: Location }[] = [];
_timeout: number | undefined;
_retries: number | undefined;
// Annotations known statically before running the test, e.g. `test.describe.skip()` or `test.describe({ annotation }, body)`.
_staticAnnotations: Annotation[] = [];
// Explicitly declared tags that are not a part of the title.
_tags: string[] = [];
_modifiers: Modifier[] = [];
_parallelMode: 'none' | 'default' | 'serial' | 'parallel' = 'none';
@ -124,6 +126,14 @@ export class Suite extends Base implements SuitePrivate {
return titlePath;
}
_collectGrepTitlePath(path: string[]) {
if (this.parent)
this.parent._collectGrepTitlePath(path);
if (this.title || this._type !== 'describe')
path.push(this.title);
path.push(...this._tags);
}
_getOnlyItems(): (TestCase | Suite)[] {
const items: (TestCase | Suite)[] = [];
if (this._only)
@ -237,7 +247,6 @@ export class TestCase extends Base implements reporterTypes.TestCase {
annotations: Annotation[] = [];
retries = 0;
repeatEachIndex = 0;
tags: string[] = [];
_testType: TestTypeImpl;
id = '';
@ -245,8 +254,10 @@ export class TestCase extends Base implements reporterTypes.TestCase {
_poolDigest = '';
_workerHash = '';
_projectId = '';
// Annotations known statically before running the test, e.g. `test.skip()` or `test.describe.skip()`.
// Annotations known statically before running the test, e.g. `test.skip()` or `test(title, { annotation }, body)`.
_staticAnnotations: Annotation[] = [];
// Explicitly declared tags that are not a part of the title.
_tags: string[] = [];
constructor(title: string, fn: Function, testType: TestTypeImpl, location: Location) {
super(title);
@ -284,6 +295,10 @@ export class TestCase extends Base implements reporterTypes.TestCase {
return status === 'expected' || status === 'flaky' || status === 'skipped';
}
get tags(): string[] {
return this._grepTitle().match(/@[\S]+/g) || [];
}
_serialize(): any {
return {
kind: 'test',
@ -299,7 +314,7 @@ export class TestCase extends Base implements reporterTypes.TestCase {
workerHash: this._workerHash,
staticAnnotations: this._staticAnnotations.slice(),
annotations: this.annotations.slice(),
tags: this.tags.slice(),
tags: this._tags.slice(),
projectId: this._projectId,
};
}
@ -316,7 +331,7 @@ export class TestCase extends Base implements reporterTypes.TestCase {
test._workerHash = data.workerHash;
test._staticAnnotations = data.staticAnnotations;
test.annotations = data.annotations;
test.tags = data.tags;
test._tags = data.tags;
test._projectId = data.projectId;
return test;
}
@ -346,4 +361,12 @@ export class TestCase extends Base implements reporterTypes.TestCase {
this.results.push(result);
return result;
}
_grepTitle() {
const path: string[] = [];
this.parent._collectGrepTitlePath(path);
path.push(this.title);
path.push(...this._tags);
return path.join(' ');
}
}

2
packages/playwright/src/common/testType.ts
View File

@ -107,7 +107,7 @@ export class TestTypeImpl {
const test = new TestCase(title, body, this, location);
test._requireFile = suite._requireFile;
test._staticAnnotations.push(...validatedDetails.annotations);
test.tags.push(...validatedDetails.tags);
test._tags.push(...validatedDetails.tags);
suite._addTest(test);
if (type === 'only')

2
packages/playwright/src/program.ts
View File

@ -165,7 +165,6 @@ async function runTests(args: string[], opts: { [key: string]: any }) {
config.cliArgs = args;
config.cliGrep = opts.grep as string | undefined;
config.cliGrepInvert = opts.grepInvert as string | undefined;
config.cliTagFilter = opts.tag;
config.cliListOnly = !!opts.list;
config.cliProjectFilter = opts.project || undefined;
config.cliProjectGrep = opts.projectGrep || undefined;
@ -332,7 +331,6 @@ const testOptions: [string, string][] = [
['--reporter <reporter>', `Reporter to use, comma-separated, can be ${builtInReporters.map(name => `"${name}"`).join(', ')} (default: "${defaultReporter}")`],
['--retries <retries>', `Maximum retry count for flaky tests, zero for no retries (default: no retries)`],
['--shard <shard>', `Shard tests and execute only the selected shard, specify in the form "current/all", 1-based, for example "3/5"`],
['--tag <tag expression>', `Only run tests with a tag(s) matching the specified expression (default: no filtering)`],
['--timeout <timeout>', `Specify test timeout threshold in milliseconds, zero for unlimited (default: ${defaultTimeout})`],
['--trace <mode>', `Force tracing mode, can be ${kTraceModes.map(mode => `"${mode}"`).join(', ')}`],
['--ui', `Run tests in interactive UI mode`],

9
packages/playwright/src/reporters/json.ts
View File

@ -169,7 +169,7 @@ class JSONReporter extends EmptyReporter {
return {
title: test.title,
ok: test.ok(),
tags: extractTags(test),
tags: test.tags.map(tag => tag.substring(1)), // Strip '@'.
tests: [this._serializeTest(projectId, projectName, test)],
id: test.id,
...this._relativeLocation(test.location),
@ -256,13 +256,6 @@ function reportOutputNameFromEnv(): string | undefined {
return undefined;
}
function extractTags(test: TestCase) {
return [
...test.tags,
...(test.title.match(/@[\S]+/g) || []),
];
}
export function serializePatterns(patterns: string | RegExp | (string | RegExp)[]): string[] {
if (!Array.isArray(patterns))
patterns = [patterns];

20
packages/playwright/src/runner/loadUtils.ts
View File

@ -21,8 +21,8 @@ import { Suite } from '../common/test';
import type { TestCase } from '../common/test';
import type { FullProjectInternal } from '../common/config';
import type { FullConfigInternal } from '../common/config';
import { createFileMatcherFromArguments, createFileFiltersFromArguments, createTitleMatcher, errorWithFile, forceRegExp, createTagMatcher } from '../util';
import type { Matcher, TagMatcher, TestFileFilter } from '../util';
import { createFileMatcherFromArguments, createFileFiltersFromArguments, createTitleMatcher, errorWithFile, forceRegExp } from '../util';
import type { Matcher, TestFileFilter } from '../util';
import { buildProjectsClosure, collectFilesForProject, filterProjects } from './projectUtils';
import type { TestRun } from './tasks';
import { requireOrImport } from '../transform/transform';
@ -135,13 +135,12 @@ export async function createRootSuite(testRun: TestRun, errors: TestError[], sho
const grepMatcher = config.cliGrep ? createTitleMatcher(forceRegExp(config.cliGrep)) : () => true;
const grepInvertMatcher = config.cliGrepInvert ? createTitleMatcher(forceRegExp(config.cliGrepInvert)) : () => false;
const cliTitleMatcher = (title: string) => !grepInvertMatcher(title) && grepMatcher(title);
const cliTagMatcher = config.cliTagFilter ? createTagMatcher(config.cliTagFilter) : undefined;
// Filter file suites for all projects.
for (const [project, fileSuites] of testRun.projectSuites) {
const projectSuite = createProjectSuite(project, fileSuites);
projectSuites.set(project, projectSuite);
const filteredProjectSuite = filterProjectSuite(projectSuite, { cliFileFilters, cliTitleMatcher, cliTagMatcher, testIdMatcher: config.testIdMatcher });
const filteredProjectSuite = filterProjectSuite(projectSuite, { cliFileFilters, cliTitleMatcher, testIdMatcher: config.testIdMatcher });
filteredProjectSuites.set(project, filteredProjectSuite);
}
}
@ -218,11 +217,8 @@ function createProjectSuite(project: FullProjectInternal, fileSuites: Suite[]):
const grepMatcher = createTitleMatcher(project.project.grep);
const grepInvertMatcher = project.project.grepInvert ? createTitleMatcher(project.project.grepInvert) : null;
const tagMatcher = project.tagFilter ? createTagMatcher(project.tagFilter) : undefined;
filterTestsRemoveEmptySuites(projectSuite, (test: TestCase) => {
if (tagMatcher && !tagMatcher(test.tags))
return false;
const grepTitle = test.titlePath().join(' ');
const grepTitle = test._grepTitle();
if (grepInvertMatcher?.(grepTitle))
return false;
return grepMatcher(grepTitle);
@ -230,9 +226,9 @@ function createProjectSuite(project: FullProjectInternal, fileSuites: Suite[]):
return projectSuite;
}
function filterProjectSuite(projectSuite: Suite, options: { cliFileFilters: TestFileFilter[], cliTitleMatcher?: Matcher, cliTagMatcher?: TagMatcher, testIdMatcher?: Matcher }): Suite {
function filterProjectSuite(projectSuite: Suite, options: { cliFileFilters: TestFileFilter[], cliTitleMatcher?: Matcher, testIdMatcher?: Matcher }): Suite {
// Fast path.
if (!options.cliFileFilters.length && !options.cliTitleMatcher && !options.testIdMatcher && !options.cliTagMatcher)
if (!options.cliFileFilters.length && !options.cliTitleMatcher && !options.testIdMatcher)
return projectSuite;
const result = projectSuite._deepClone();
@ -241,9 +237,7 @@ function filterProjectSuite(projectSuite: Suite, options: { cliFileFilters: Test
if (options.testIdMatcher)
filterByTestIds(result, options.testIdMatcher);
filterTestsRemoveEmptySuites(result, (test: TestCase) => {
if (options.cliTagMatcher && !options.cliTagMatcher(test.tags))
return false;
if (options.cliTitleMatcher && !options.cliTitleMatcher(test.titlePath().join(' ')))
if (options.cliTitleMatcher && !options.cliTitleMatcher(test._grepTitle()))
return false;
return true;
});

50
packages/playwright/src/util.ts
View File

@ -71,7 +71,6 @@ export function serializeError(error: Error | any): TestInfoError {
}
export type Matcher = (value: string) => boolean;
export type TagMatcher = (tags: string[]) => boolean;
export type TestFileFilter = {
re?: RegExp;
@ -145,55 +144,6 @@ export function createTitleMatcher(patterns: RegExp | RegExp[]): Matcher {
};
}
export function createTagMatcher(tagFilter: string): TagMatcher {
const tokens: (string | null)[] = tagFilter.split(/([()]|\s+)/).filter(s => !!s.trim());
tokens.push(null); // eof
let pos = 0;
function parseSingle(): TagMatcher {
if (tokens[pos] === '(') {
pos++;
const result = parseOr();
if (tokens[pos] !== ')')
throw new Error(`Expected matching ")" when parsing tag expression: ${tagFilter}`);
pos++;
return result;
}
if (tokens[pos] === 'not') {
pos++;
const inner = parseSingle();
return (tags: string[]) => !inner(tags);
}
const tag = tokens[pos++];
if (tag === null)
throw new Error(`Unexpected end of tag expression: ${tagFilter}`);
return (tags: string[]) => tags.includes(tag);
}
function parseAnd(): TagMatcher {
const singles = [parseSingle()];
while (tokens[pos] === 'and') {
pos++;
singles.push(parseSingle());
}
return (tags: string[]) => singles.every(s => s(tags));
}
function parseOr(): TagMatcher {
const ands = [parseAnd()];
while (tokens[pos] === 'or') {
pos++;
ands.push(parseAnd());
}
return (tags: string[]) => ands.some(a => a(tags));
}
const result = parseOr();
if (tokens[pos] !== null)
throw new Error(`Unexpected extra tokens in the tag expression: ${tagFilter}`);
return result;
}
export function mergeObjects<A extends object, B extends object, C extends object>(a: A | undefined | void, b: B | undefined | void, c: B | undefined | void): A & B & C {
const result = { ...a } as any;
for (const x of [b, c].filter(Boolean)) {

2
packages/playwright/src/worker/workerMain.ts
View File

@ -296,7 +296,7 @@ export class WorkerMain extends ProcessRunner {
const nextSuites = new Set(getSuites(nextTest));
testInfo._timeoutManager.setTimeout(test.timeout);
for (const annotation of test._staticAnnotations)
for (const annotation of test.annotations)
processAnnotation(annotation);
// Process existing annotations dynamically set for parent suites.

173
packages/playwright/types/test.d.ts vendored
View File

@ -1244,30 +1244,6 @@ interface TestConfig {
*/
snapshotPathTemplate?: string;
/**
* Filter to only run tests with or without particular tag(s). This property must be a logical expression containing
* tags, parenthesis `(` and `)`, and operators `and`, `or` and `not`.
*
* Learn more about [tagging](https://playwright.dev/docs/test-annotations#tag-tests).
*
* **Usage**
*
* ```js
* // playwright.config.ts
* import { defineConfig } from '@playwright/test';
*
* export default defineConfig({
* tagFilter: '@smoke',
* });
* ```
*
* Tag expression examples:
* - `tagFilter: '@smoke'` - run only "smoke" tests;
* - `tagFilter: '@smoke and not @production'` - run only "smoke" tests that are not tagged as "production";
* - `tagFilter: '(@smoke or @fast) and @v2'` - run "v2" tests that are tagged as "smoke", "fast" or both.
*/
tagFilter?: string;
/**
* Directory that will be recursively scanned for test files. Defaults to the directory of the configuration file.
*
@ -2696,7 +2672,8 @@ interface TestFunction<TestArgs> {
*
* **Tags**
*
* You can tag tests by providing additional test details. Note that each tag must start with `@` symbol.
* You can tag tests by providing additional test details. Alternatively, you can include tags in the test title. Note
* that each tag must start with `@` symbol.
*
* ```js
* import { test, expect } from '@playwright/test';
@ -2707,15 +2684,19 @@ interface TestFunction<TestArgs> {
* await page.goto('https://playwright.dev/');
* // ...
* });
*
* test('another test @smoke', async ({ page }) => {
* await page.goto('https://playwright.dev/');
* // ...
* });
* ```
*
* Test tags are displayed in the test report, and are available to a custom reporter via `TestCase.tags` property.
*
* You can also filter tests by their tags during test execution:
* - in the [command line](https://playwright.dev/docs/test-cli#reference);
* - in the config with
* [testConfig.tagFilter](https://playwright.dev/docs/api/class-testconfig#test-config-tag-filter) and
* [testProject.tagFilter](https://playwright.dev/docs/api/class-testproject#test-project-tag-filter);
* - in the config with [testConfig.grep](https://playwright.dev/docs/api/class-testconfig#test-config-grep) and
* [testProject.grep](https://playwright.dev/docs/api/class-testproject#test-project-grep);
*
* Learn more about [tagging](https://playwright.dev/docs/test-annotations#tag-tests).
*
@ -2740,7 +2721,7 @@ interface TestFunction<TestArgs> {
* Test annotations are displayed in the test report, and are available to a custom reporter via
* `TestCase.annotations` property.
*
* You can also add dynamic annotations by manipulating
* You can also add annotations during runtime by manipulating
* [testInfo.annotations](https://playwright.dev/docs/api/class-testinfo#test-info-annotations).
*
* Learn more about [test annotations](https://playwright.dev/docs/test-annotations).
@ -2767,7 +2748,8 @@ interface TestFunction<TestArgs> {
*
* **Tags**
*
* You can tag tests by providing additional test details. Note that each tag must start with `@` symbol.
* You can tag tests by providing additional test details. Alternatively, you can include tags in the test title. Note
* that each tag must start with `@` symbol.
*
* ```js
* import { test, expect } from '@playwright/test';
@ -2778,15 +2760,19 @@ interface TestFunction<TestArgs> {
* await page.goto('https://playwright.dev/');
* // ...
* });
*
* test('another test @smoke', async ({ page }) => {
* await page.goto('https://playwright.dev/');
* // ...
* });
* ```
*
* Test tags are displayed in the test report, and are available to a custom reporter via `TestCase.tags` property.
*
* You can also filter tests by their tags during test execution:
* - in the [command line](https://playwright.dev/docs/test-cli#reference);
* - in the config with
* [testConfig.tagFilter](https://playwright.dev/docs/api/class-testconfig#test-config-tag-filter) and
* [testProject.tagFilter](https://playwright.dev/docs/api/class-testproject#test-project-tag-filter);
* - in the config with [testConfig.grep](https://playwright.dev/docs/api/class-testconfig#test-config-grep) and
* [testProject.grep](https://playwright.dev/docs/api/class-testproject#test-project-grep);
*
* Learn more about [tagging](https://playwright.dev/docs/test-annotations#tag-tests).
*
@ -2811,7 +2797,7 @@ interface TestFunction<TestArgs> {
* Test annotations are displayed in the test report, and are available to a custom reporter via
* `TestCase.annotations` property.
*
* You can also add dynamic annotations by manipulating
* You can also add annotations during runtime by manipulating
* [testInfo.annotations](https://playwright.dev/docs/api/class-testinfo#test-info-annotations).
*
* Learn more about [test annotations](https://playwright.dev/docs/test-annotations).
@ -3277,11 +3263,15 @@ export interface TestType<TestArgs extends KeyValue, WorkerArgs extends KeyValue
* Skipped tests are not supposed to be ever run. If you intent to fix the test, use
* [test.fixme([title, details, body, condition, callback, description])](https://playwright.dev/docs/api/class-test#test-fixme)
* instead.
*
* To declare a skipped test:
* - `test.skip(title, body)`
* - `test.skip(title, details, body)`
* - `test.skip()`
*
* To skip a test at runtime:
* - `test.skip(condition, description)`
* - `test.skip(callback, description)`
* - `test.skip()`
*
* **Usage**
*
@ -3353,11 +3343,15 @@ export interface TestType<TestArgs extends KeyValue, WorkerArgs extends KeyValue
* Skipped tests are not supposed to be ever run. If you intent to fix the test, use
* [test.fixme([title, details, body, condition, callback, description])](https://playwright.dev/docs/api/class-test#test-fixme)
* instead.
*
* To declare a skipped test:
* - `test.skip(title, body)`
* - `test.skip(title, details, body)`
* - `test.skip()`
*
* To skip a test at runtime:
* - `test.skip(condition, description)`
* - `test.skip(callback, description)`
* - `test.skip()`
*
* **Usage**
*
@ -3429,11 +3423,15 @@ export interface TestType<TestArgs extends KeyValue, WorkerArgs extends KeyValue
* Skipped tests are not supposed to be ever run. If you intent to fix the test, use
* [test.fixme([title, details, body, condition, callback, description])](https://playwright.dev/docs/api/class-test#test-fixme)
* instead.
*
* To declare a skipped test:
* - `test.skip(title, body)`
* - `test.skip(title, details, body)`
* - `test.skip()`
*
* To skip a test at runtime:
* - `test.skip(condition, description)`
* - `test.skip(callback, description)`
* - `test.skip()`
*
* **Usage**
*
@ -3505,11 +3503,15 @@ export interface TestType<TestArgs extends KeyValue, WorkerArgs extends KeyValue
* Skipped tests are not supposed to be ever run. If you intent to fix the test, use
* [test.fixme([title, details, body, condition, callback, description])](https://playwright.dev/docs/api/class-test#test-fixme)
* instead.
*
* To declare a skipped test:
* - `test.skip(title, body)`
* - `test.skip(title, details, body)`
* - `test.skip()`
*
* To skip a test at runtime:
* - `test.skip(condition, description)`
* - `test.skip(callback, description)`
* - `test.skip()`
*
* **Usage**
*
@ -3581,11 +3583,15 @@ export interface TestType<TestArgs extends KeyValue, WorkerArgs extends KeyValue
* Skipped tests are not supposed to be ever run. If you intent to fix the test, use
* [test.fixme([title, details, body, condition, callback, description])](https://playwright.dev/docs/api/class-test#test-fixme)
* instead.
*
* To declare a skipped test:
* - `test.skip(title, body)`
* - `test.skip(title, details, body)`
* - `test.skip()`
*
* To skip a test at runtime:
* - `test.skip(condition, description)`
* - `test.skip(callback, description)`
* - `test.skip()`
*
* **Usage**
*
@ -3654,11 +3660,15 @@ export interface TestType<TestArgs extends KeyValue, WorkerArgs extends KeyValue
/**
* Mark a test as "fixme", with the intention to fix it. Playwright will not run the test past the `test.fixme()`
* call.
*
* To declare a "fixme" test:
* - `test.fixme(title, body)`
* - `test.fixme(title, details, body)`
* - `test.fixme()`
*
* To annotate test as "fixme" at runtime:
* - `test.fixme(condition, description)`
* - `test.fixme(callback, description)`
* - `test.fixme()`
*
* **Usage**
*
@ -3727,11 +3737,15 @@ export interface TestType<TestArgs extends KeyValue, WorkerArgs extends KeyValue
/**
* Mark a test as "fixme", with the intention to fix it. Playwright will not run the test past the `test.fixme()`
* call.
*
* To declare a "fixme" test:
* - `test.fixme(title, body)`
* - `test.fixme(title, details, body)`
* - `test.fixme()`
*
* To annotate test as "fixme" at runtime:
* - `test.fixme(condition, description)`
* - `test.fixme(callback, description)`
* - `test.fixme()`
*
* **Usage**
*
@ -3800,11 +3814,15 @@ export interface TestType<TestArgs extends KeyValue, WorkerArgs extends KeyValue
/**
* Mark a test as "fixme", with the intention to fix it. Playwright will not run the test past the `test.fixme()`
* call.
*
* To declare a "fixme" test:
* - `test.fixme(title, body)`
* - `test.fixme(title, details, body)`
* - `test.fixme()`
*
* To annotate test as "fixme" at runtime:
* - `test.fixme(condition, description)`
* - `test.fixme(callback, description)`
* - `test.fixme()`
*
* **Usage**
*
@ -3873,11 +3891,15 @@ export interface TestType<TestArgs extends KeyValue, WorkerArgs extends KeyValue
/**
* Mark a test as "fixme", with the intention to fix it. Playwright will not run the test past the `test.fixme()`
* call.
*
* To declare a "fixme" test:
* - `test.fixme(title, body)`
* - `test.fixme(title, details, body)`
* - `test.fixme()`
*
* To annotate test as "fixme" at runtime:
* - `test.fixme(condition, description)`
* - `test.fixme(callback, description)`
* - `test.fixme()`
*
* **Usage**
*
@ -3946,11 +3968,15 @@ export interface TestType<TestArgs extends KeyValue, WorkerArgs extends KeyValue
/**
* Mark a test as "fixme", with the intention to fix it. Playwright will not run the test past the `test.fixme()`
* call.
*
* To declare a "fixme" test:
* - `test.fixme(title, body)`
* - `test.fixme(title, details, body)`
* - `test.fixme()`
*
* To annotate test as "fixme" at runtime:
* - `test.fixme(condition, description)`
* - `test.fixme(callback, description)`
* - `test.fixme()`
*
* **Usage**
*
@ -4019,11 +4045,15 @@ export interface TestType<TestArgs extends KeyValue, WorkerArgs extends KeyValue
/**
* Marks a test as "should fail". Playwright runs this test and ensures that it is actually failing. This is useful
* for documentation purposes to acknowledge that some functionality is broken until it is fixed.
*
* To declare a "failing" test:
* - `test.fail(title, body)`
* - `test.fail(title, details, body)`
* - `test.fail()`
*
* To annotate test as "failing" at runtime:
* - `test.fail(condition, description)`
* - `test.fail(callback, description)`
* - `test.fail()`
*
* **Usage**
*
@ -4091,11 +4121,15 @@ export interface TestType<TestArgs extends KeyValue, WorkerArgs extends KeyValue
/**
* Marks a test as "should fail". Playwright runs this test and ensures that it is actually failing. This is useful
* for documentation purposes to acknowledge that some functionality is broken until it is fixed.
*
* To declare a "failing" test:
* - `test.fail(title, body)`
* - `test.fail(title, details, body)`
* - `test.fail()`
*
* To annotate test as "failing" at runtime:
* - `test.fail(condition, description)`
* - `test.fail(callback, description)`
* - `test.fail()`
*
* **Usage**
*
@ -4163,11 +4197,15 @@ export interface TestType<TestArgs extends KeyValue, WorkerArgs extends KeyValue
/**
* Marks a test as "should fail". Playwright runs this test and ensures that it is actually failing. This is useful
* for documentation purposes to acknowledge that some functionality is broken until it is fixed.
*
* To declare a "failing" test:
* - `test.fail(title, body)`
* - `test.fail(title, details, body)`
* - `test.fail()`
*
* To annotate test as "failing" at runtime:
* - `test.fail(condition, description)`
* - `test.fail(callback, description)`
* - `test.fail()`
*
* **Usage**
*
@ -4235,11 +4273,15 @@ export interface TestType<TestArgs extends KeyValue, WorkerArgs extends KeyValue
/**
* Marks a test as "should fail". Playwright runs this test and ensures that it is actually failing. This is useful
* for documentation purposes to acknowledge that some functionality is broken until it is fixed.
*
* To declare a "failing" test:
* - `test.fail(title, body)`
* - `test.fail(title, details, body)`
* - `test.fail()`
*
* To annotate test as "failing" at runtime:
* - `test.fail(condition, description)`
* - `test.fail(callback, description)`
* - `test.fail()`
*
* **Usage**
*
@ -4307,11 +4349,15 @@ export interface TestType<TestArgs extends KeyValue, WorkerArgs extends KeyValue
/**
* Marks a test as "should fail". Playwright runs this test and ensures that it is actually failing. This is useful
* for documentation purposes to acknowledge that some functionality is broken until it is fixed.
*
* To declare a "failing" test:
* - `test.fail(title, body)`
* - `test.fail(title, details, body)`
* - `test.fail()`
*
* To annotate test as "failing" at runtime:
* - `test.fail(condition, description)`
* - `test.fail(callback, description)`
* - `test.fail()`
*
* **Usage**
*
@ -8725,35 +8771,6 @@ interface TestProject {
*/
snapshotPathTemplate?: string;
/**
* Filter to only run tests with or without particular tag(s). This property must be a logical expression containing
* tags, parenthesis `(` and `)`, and operators `and`, `or` and `not`.
*
* Learn more about [tagging](https://playwright.dev/docs/test-annotations#tag-tests).
*
* **Usage**
*
* ```js
* // playwright.config.ts
* import { defineConfig } from '@playwright/test';
*
* export default defineConfig({
* projects: [
* {
* name: 'tests',
* tagFilter: '@smoke',
* },
* ],
* });
* ```
*
* Tag expression examples:
* - `tagFilter: '@smoke'` - run only "smoke" tests;
* - `tagFilter: '@smoke and not @production'` - run only "smoke" tests that are not tagged as "production";
* - `tagFilter: '(@smoke or @fast) and @v2'` - run "v2" tests that are tagged as "smoke", "fast" or both.
*/
tagFilter?: string;
/**
* Name of a project that needs to run after this and all dependent projects have finished. Teardown is useful to
* cleanup any resources acquired by this project.

3
packages/playwright/types/testReporter.d.ts vendored
View File

@ -200,7 +200,8 @@ export interface TestCase {
/**
* The list of tags defined on the test or suite via
* [test.(call)(title[, details, body])](https://playwright.dev/docs/api/class-test#test-call) or
* [test.describe([title, details, callback])](https://playwright.dev/docs/api/class-test#test-describe).
* [test.describe([title, details, callback])](https://playwright.dev/docs/api/class-test#test-describe), as well as
* `@`-tokens extracted from test and suite titles.
*
* Learn more about [test tags](https://playwright.dev/docs/test-annotations#tag-tests).
*/

44
tests/playwright-test/reporter-html.spec.ts
View File

@ -1150,64 +1150,64 @@ for (const useIntermediateMergeReport of [false, true] as const) {
await expect(page.locator('.test-file-test', { has: page.getByText('@regression @failed failed', { exact: true }) }).locator('.label')).toHaveText([
'chromium',
'foo',
'failed',
'regression',
'failed',
'foo',
'firefox',
'foo',
'failed',
'regression',
'webkit',
'foo',
'failed',
'regression'
'foo',
'webkit',
'regression',
'failed',
'foo',
]);
await expect(page.locator('.test-file-test', { has: page.getByText('@regression @flaky flaky', { exact: true }) }).locator('.label')).toHaveText([
'chromium',
'flaky',
'regression',
'flaky',
'firefox',
'flaky',
'regression',
'flaky',
'webkit',
'flaky',
'regression',
'flaky',
]);
await expect(page.locator('.test-file-test', { has: page.getByText('@regression skipped', { exact: true }) }).locator('.label')).toHaveText([
'chromium',
'regression',
'foo',
'bar',
'regression',
'firefox',
'regression',
'foo',
'bar',
'regression',
'webkit',
'regression',
'foo',
'bar',
'regression',
]);
await expect(page.locator('.test-file-test', { has: page.getByText('@smoke @passed passed', { exact: true }) }).locator('.label')).toHaveText([
'chromium',
'passed',
'smoke',
'passed',
'firefox',
'passed',
'smoke',
'webkit',
'passed',
'smoke'
'webkit',
'smoke',
'passed',
]);
await expect(page.locator('.test-file-test', { has: page.getByText('@smoke @failed failed', { exact: true }) }).locator('.label')).toHaveText([
'chromium',
'failed',
'smoke',
'failed',
'firefox',
'failed',
'smoke',
'failed',
'webkit',
'failed',
'smoke',
'failed',
]);
});
@ -1961,7 +1961,7 @@ for (const useIntermediateMergeReport of [false, true] as const) {
await expect(page).toHaveURL(/testId/);
await expect(page.locator('.test-case-path')).toHaveText('Root describe › @Notifications');
await expect(page.locator('.test-case-title')).toHaveText('Test failed -- @call @call-details @e2e @regression #VQ458');
await expect(page.locator('.label')).toHaveText(['chromium', 'call', 'call-details', 'e2e', 'Notifications', 'regression']);
await expect(page.locator('.label')).toHaveText(['chromium', 'Notifications', 'call', 'call-details', 'e2e', 'regression']);
await page.goBack();
await expect(page).not.toHaveURL(/testId/);
@ -1973,7 +1973,7 @@ for (const useIntermediateMergeReport of [false, true] as const) {
await expect(page).toHaveURL(/testId/);
await expect(page.locator('.test-case-path')).toHaveText('Root describe › @Monitoring');
await expect(page.locator('.test-case-title')).toHaveText('Test passed -- @call @call-details @e2e @regression #VQ457');
await expect(page.locator('.label')).toHaveText(['firefox', 'call', 'call-details', 'e2e', 'Monitoring', 'regression']);
await expect(page.locator('.label')).toHaveText(['firefox', 'Monitoring', 'call', 'call-details', 'e2e', 'regression']);
});
});

2
tests/playwright-test/reporter-json.spec.ts
View File

@ -168,7 +168,7 @@ test('should display tags separately from title', async ({ runInlineTest }) => {
});
expect(result.exitCode).toBe(0);
expect(result.report.suites[0].specs[0].tags).toEqual(['@foo', '@USR-MATH-001', '@USR-MATH-002']);
expect(result.report.suites[0].specs[0].tags).toEqual(['USR-MATH-001', 'USR-MATH-002', 'foo']);
});
test('should have relative always-posix paths', async ({ runInlineTest }) => {

120
tests/playwright-test/test-tag.spec.ts
View File

@ -44,8 +44,8 @@ test('should have correct tags', async ({ runInlineTest }) => {
test('no-tags', () => {
expect(test.info()._test.tags).toEqual([]);
});
test('foo-tag', { tag: '@foo' }, () => {
expect(test.info()._test.tags).toEqual(['@foo']);
test('foo-tag @inline', { tag: '@foo' }, () => {
expect(test.info()._test.tags).toEqual(['@inline', '@foo']);
});
test('foo-bar-tags', { tag: ['@foo', '@bar'] }, () => {
expect(test.info()._test.tags).toEqual(['@foo', '@bar']);
@ -57,13 +57,13 @@ test('should have correct tags', async ({ runInlineTest }) => {
test.fail('fail-foo-bar-tags', { tag: ['@foo', '@bar'] }, () => {
expect(1).toBe(2);
});
test.describe('suite', { tag: '@foo' }, () => {
test.describe('suite @inline', { tag: '@foo' }, () => {
test('foo-suite', () => {
expect(test.info()._test.tags).toEqual(['@foo']);
expect(test.info()._test.tags).toEqual(['@inline', '@foo']);
});
test.describe('inner', { tag: '@bar' }, () => {
test('foo-bar-suite', () => {
expect(test.info()._test.tags).toEqual(['@foo', '@bar']);
expect(test.info()._test.tags).toEqual(['@inline', '@foo', '@bar']);
});
});
});
@ -80,22 +80,22 @@ test('should have correct tags', async ({ runInlineTest }) => {
expect(result.exitCode).toBe(0);
expect(result.outputLines).toEqual([
`title=no-tags, tags=`,
`title=foo-tag, tags=@foo`,
`title=foo-tag @inline, tags=@inline,@foo`,
`title=foo-bar-tags, tags=@foo,@bar`,
`title=skip-foo-tag, tags=@foo`,
`title=fixme-bar-tag, tags=@bar`,
`title=fail-foo-bar-tags, tags=@foo,@bar`,
`title=foo-suite, tags=@foo`,
`title=foo-bar-suite, tags=@foo,@bar`,
`title=foo-suite, tags=@inline,@foo`,
`title=foo-bar-suite, tags=@inline,@foo,@bar`,
`title=skip-foo-suite, tags=@foo`,
`title=fixme-bar-suite, tags=@bar`,
]);
});
test('config.tagFilter should work', async ({ runInlineTest }) => {
test('config.grep should work', async ({ runInlineTest }) => {
const result = await runInlineTest({
'playwright.config.ts': `
module.exports = { tagFilter: '@tag1' };
module.exports = { grep: /@tag1/ };
`,
'a.test.ts': `
import { test, expect } from '@playwright/test';
@ -108,12 +108,12 @@ test('config.tagFilter should work', async ({ runInlineTest }) => {
expect(result.outputLines).toEqual(['test1']);
});
test('config.project.tag should work', async ({ runInlineTest }) => {
test('config.project.grep should work', async ({ runInlineTest }) => {
const result = await runInlineTest({
'playwright.config.ts': `
module.exports = { projects: [
{ name: 'p1' },
{ name: 'p2', tagFilter: '@tag1' }
{ name: 'p2', grep: /@tag1/ }
] };
`,
'a.test.ts': `
@ -127,63 +127,19 @@ test('config.project.tag should work', async ({ runInlineTest }) => {
expect(result.outputLines).toEqual(['test1-p1', 'test2-p1', 'test1-p2']);
});
test('--tag should work', async ({ runInlineTest }) => {
test('--grep should work', async ({ runInlineTest }) => {
const result = await runInlineTest({
'a.test.ts': `
import { test, expect } from '@playwright/test';
test('test1', { tag: '@tag1' }, async () => { console.log('\\n%% test1'); });
test('test2', async () => { console.log('\\n%% test2'); });
`,
}, { tag: '@tag1' });
}, { grep: '@tag1' });
expect(result.exitCode).toBe(0);
expect(result.passed).toBe(1);
expect(result.outputLines).toEqual(['test1']);
});
test('should parse tag expressions', async ({ runInlineTest }) => {
const result = await runInlineTest({
'playwright.config.ts': `
module.exports = {
projects: [
{ name: 'p1', tagFilter: '@foo' },
{ name: 'p2', tagFilter: 'not @foo' },
{ name: 'p3', tagFilter: ' @foo and @bar' },
{ name: 'p4', tagFilter: '@bar or not @foo' },
{ name: 'p5', tagFilter: '@bar and (@foo or not @foo)' },
{ name: 'p6', tagFilter: '@qux or @foo and @bar' },
{ name: 'p7', tagFilter: '@qux and (@foo or @bar)' },
{ name: 'p8', tagFilter: 'not not not @foo' },
]
};
`,
'stdio.spec.js': `
import { test, expect } from '@playwright/test';
test('test1', { tag: '@foo' }, () => {
console.log('\\n%% foo-' + test.info().project.name);
});
test('test2', { tag: '@bar' }, () => {
console.log('\\n%% bar-' + test.info().project.name);
});
test('test3', { tag: ['@foo', '@bar'] }, () => {
console.log('\\n%% foobar-' + test.info().project.name);
});
`
}, { workers: 1 });
expect(result.exitCode).toBe(0);
expect(result.outputLines).toEqual([
`foo-p1`,
`foobar-p1`,
`bar-p2`,
`foobar-p3`,
`bar-p4`,
`foobar-p4`,
`bar-p5`,
`foobar-p5`,
`foobar-p6`,
`bar-p8`,
]);
});
test('should enforce @ symbol', async ({ runInlineTest }) => {
const result = await runInlineTest({
'stdio.spec.js': `
@ -195,51 +151,3 @@ test('should enforce @ symbol', async ({ runInlineTest }) => {
expect(result.exitCode).toBe(1);
expect(result.output).toContain(`Error: Tag must start with "@" symbol, got "foo" instead.`);
});
test('should report tag expression error 1', async ({ runInlineTest }) => {
const result = await runInlineTest({
'stdio.spec.js': `
import { test, expect } from '@playwright/test';
test('test1', { tag: '@foo' }, () => {
});
`
}, { tag: '(@foo' });
expect(result.exitCode).toBe(1);
expect(result.output).toContain(`Error: Expected matching ")" when parsing tag expression: (@foo`);
});
test('should report tag expression error 2', async ({ runInlineTest }) => {
const result = await runInlineTest({
'stdio.spec.js': `
import { test, expect } from '@playwright/test';
test('test1', { tag: '@foo' }, () => {
});
`
}, { tag: '(@foo)@bar' });
expect(result.exitCode).toBe(1);
expect(result.output).toContain(`Error: Unexpected extra tokens in the tag expression: (@foo)@bar`);
});
test('should report tag expression error 3', async ({ runInlineTest }) => {
const result = await runInlineTest({
'stdio.spec.js': `
import { test, expect } from '@playwright/test';
test('test1', { tag: '@foo' }, () => {
});
`
}, { tag: '@foo and' });
expect(result.exitCode).toBe(1);
expect(result.output).toContain(`Error: Unexpected end of tag expression: @foo and`);
});
test('should report tag expression error 4', async ({ runInlineTest }) => {
const result = await runInlineTest({
'stdio.spec.js': `
import { test, expect } from '@playwright/test';
test('test1', { tag: '@foo' }, () => {
});
`
}, { tag: '@foo @bar' });
expect(result.exitCode).toBe(1);
expect(result.output).toContain(`Error: Unexpected extra tokens in the tag expression: @foo @bar`);
});