ci

199 notes in this subject, read out of this brain and free to use. This is page 3 of 4.

Filter tests by tag with --grep command line option

Run tests that have a particular tag using the --grep command line option: npx playwright test --grep @fast.

Invert tag filter with --grep-invert

Skip tests with a certain tag using the --grep-invert command line option: npx playwright test --grep-invert @fast.

Run tests with both tags using regex lookaheads

To run tests containing both tags (logical AND operator), use --grep with regex lookaheads: npx playwright test --grep "(?=.*@fast)(?=.*@slow)".

npx playwright test basic syntax

Run Playwright tests using the command `npx playwright test [options] [test-filter...]`. Test filters are regular expressions matched against the full test file path.

Run tests at a specific line

To run tests at a specific line number, use the syntax `npx playwright test my-spec.ts:42` where 42 is the line number.

Run tests by title

To run tests matching a title pattern, use `npx playwright test -g "add a todo item"` with the -g or --grep option followed by a regular expression.

Run tests in headed browsers

Use `npx playwright test --headed` to run tests in headed browsers instead of the default headless mode.

Run tests for a specific project

Use `npx playwright test --project=chromium` to run tests only for the specified project.

Disable parallelization in test runs

Use `npx playwright test --workers=1` to run tests in a single worker, disabling parallelization.

Run tests in interactive UI mode

Use `npx playwright test --ui` to run tests in interactive UI mode.

npx playwright test common options table

Common options for `npx playwright test`: | Option | Description | | --debug | Run tests with Playwright Inspector. Shortcut for PWDEBUG=1 and --timeout=0 --max-failures=1 --headed --workers=1. | | --headed | Run tests in headed browsers (default: headless). | | -g <grep>, --grep <grep> | Only run tests matching this regular expression (default: ".*"). | | --project <project-name...> | Only run tests from the specified list of projects, supports '*' wildcard (default: run all projects). | | --ui | Run tests in interactive UI mode. | | -j <workers>, --workers <workers> | Number of concurrent workers or percentage of logical CPU cores, use 1 to run in a single worker (default: 50%). |

npx playwright test all CLI options table

All options for `npx playwright test`: | Option | Description | | Non-option arguments | Each argument is treated as a regular expression matched against the full test file path. Only tests from files matching the pattern will be executed. Special symbols like $ or * should be escaped with \\. | | --add-reporter <reporter> | Reporter to add on top of the reporters configured in the config file, comma-separated. Can be a built-in reporter name or a path to a custom reporter file. Unlike --reporter, this keeps the configured reporters instead of replacing them. | | -c <file>, --config <file> | Configuration file, or a test directory with optional "playwright.config.{m,c}?{js,ts}". Defaults to playwright.config.ts or playwright.config.js in the current directory. | | --debug | Run tests with Playwright Inspector. Shortcut for PWDEBUG=1 and --timeout=0 --max-failures=1 --headed --workers=1. | | --fail-on-flaky-tests | Fail if any test is flagged as flaky (default: false). | | --forbid-only | Fail if test.only is called (default: false). Useful on CI. | | --fully-parallel | Run all tests in parallel (default: false). | | --global-timeout <timeout> | Maximum time this test suite can run in milliseconds (default: unlimited). | | -g <grep>, --grep <grep> | Only run tests matching this regular expression (default: ".*"). | | -G <grep>, --grep-invert <grep> | Only run tests that do not match this regular expression. | | --headed | Run tests in headed browsers (default: headless). | | --ignore-snapshots | Ignore screenshot and snapshot expectations. | | -j <workers>, --workers <workers> | Number of concurrent workers or percentage of logical CPU cores, use 1 to run in a single worker (default: 50%). | | --last-failed | Only re-run the failures. | | --last-failed-file <file> | Override the default last-run JSON path for --last-failed (default: <outputDir>/.last-run.json). Same as PLAYWRIGHT_LAST_RUN_OUTPUT_FILE environment variable. | | --list | Collect all the tests and report them, but do not run. | | --max-failures <N>, -x | Stop after the first N failures. Passing -x stops after the first failure. | | --no-deps | Do not run project dependencies. | | --output <dir> | Folder for output artifacts (default: "test-results"). | | --only-changed [ref] | Only run test files that have been changed between 'HEAD' and 'ref'. Defaults to running all uncommitted changes. Only supports Git. | | --pass-with-no-tests | Makes test run succeed even if no tests were found. | | --project <project-name...> | Only run tests from the specified list of projects, supports '*' wildcard (default: run all projects). | | --quiet | Suppress stdio. | | --repeat-each <N> | Run each test N times (default: 1). | | --reporter <reporter> | Reporter to use, comma-separated, can be "dot", "line", "list", or others (default: "list"). You can also pass a path to a custom reporter file. | | --retries <retries> | Maximum retry count for flaky tests, zero for no retries (default: no retries). | | --shard <shard> | Shard tests and execute only the selected shard, specified in the form "current/all", 1-based, e.g., "3/5". | | --test-list <file> | Path to a file containing a list of tests to run. | | --test-list-invert <file> | Path to a file containing a list of tests to skip. | | --timeout <timeout> | Specify test timeout threshold in milliseconds, zero for unlimited (default: 30 seconds). | | --trace <mode> | Force tracing mode, can be on, off, on-first-retry, on-all-retries, retain-on-failure, retain-on-first-failure, retain-on-failure-and-retries. | | --tsconfig <path> | Path to a single tsconfig applicable to all imported files (default: look up tsconfig for each imported file separately). | | --ui | Run tests in interactive UI mode. | | --ui-host <host> | Host to serve UI on; specifying this option opens UI in a browser tab. | | --ui-port <port> | Port to serve UI on, 0 for any free port; specifying this option opens UI in a browser tab. | | -u, --update-snapshots [mode] | Update snapshots with actual results. Possible values are "all", "changed", "missing", and "none". Running tests without the flag defaults to "missing"; running tests with the flag but without a value defaults to "changed". | | --update-source-method [mode] | Update snapshots with actual results. Possible values are "patch" (default), "3way" and "overwrite". "Patch" creates a unified diff file. "3way" generates merge conflict markers. "Overwrite" overwrites source code with new snapshot values. | | -x | Stop after the first failure. |

Test list file format

The --test-list and --test-list-invert options accept a path to a test list file. This file lists tests in a format similar to the output from --list mode. Supported formats include: full file paths (path/to/example.spec.ts), project-specific paths ([chromium] › path/to/example.spec.ts), suite paths (path/to/example.spec.ts › suite name), nested suite paths, fully qualified tests with projects ([chromium] › path/to/example.spec.ts:3:9 › suite › nested suite › example test), and tests for all projects (path/to/example.spec.ts:3:9 › example test). Both "›" and ">" are accepted as separators. Line and column numbers are ignored; omitting them still refers to the same test.

Update source method modes

The --update-source-method option accepts modes: "patch" (default), "3way", and "overwrite". "Patch" creates a unified diff file that can be used to update the source code later. "3way" generates merge conflict markers in source code. "Overwrite" overwrites the source code with the new snapshot values.

Show HTML report from test run

Use `npx playwright show-report [report] [options]` to display the HTML report from a previous test run. If no report path is specified, it shows the latest report.

Show report options table

Update snapshot modes for --update-snapshots

The --update-snapshots option accepts modes: "all", "changed", "missing", and "none". Running tests without the flag defaults to "missing"; running tests with the flag but without a value defaults to "changed".

Install Playwright browsers

Use `npx playwright install [options] [browser...]`, `npx playwright install-deps [options] [browser...]`, or `npx playwright uninstall` to manage Playwright browser installations.

Browser installation deps options table

Options for `npx playwright install-deps`: | Option | Description | | --dry-run | Don't modify the system. On Linux, simulates the install via apt-get and exits with a non-zero code if any required packages are missing — useful for non-interactive verification scripts. On Windows, prints the install command. |

Browser installation options table

Merge test reports

Use `npx playwright merge-reports [options] <blob dir>` to read blob reports and combine them.

Merge reports options table

Options for `npx playwright merge-reports`: | Option | Description | | -c, --config <file> | Configuration file. Can be used to specify additional configuration for the output report | | --reporter <reporter> | Reporter to use, comma-separated, can be "list", "line", "dot", "json", "junit", "null", "github", "html", "blob" (default: "list") |

Clear Playwright caches

Use `npx playwright clear-cache` to clear all Playwright caches.

Use blob reporter for merging shard reports

When running on CI with multiple shards, add blob reporter to the configuration to generate blob reports that can be merged: reporter: process.env.CI ? 'blob' : 'html'. Blob reports contain information about all tests, their results, and attachments such as traces and screenshot diffs. By default, blob reports are generated into the blob-report directory.

Shard tests across machines with --shard=x/y

To shard the test suite across multiple machines, pass --shard=x/y to the command line. For example, to split the suite into four shards, each running one fourth of the tests: npx playwright test --shard=1/4, npx playwright test --shard=2/4, npx playwright test --shard=3/4, and npx playwright test --shard=4/4. Running these shards in parallel on different jobs makes the test suite complete four times faster.

Playwright can only shard tests that run in parallel

By default, Playwright shards test files. Only tests that can be run in parallel can be sharded.

Sharding with fullyParallel: true uses test-level granularity

When fullyParallel: true is enabled in TestProject, Playwright Test runs individual tests in parallel across multiple shards, ensuring each shard receives an even distribution of tests. This allows for test-level granularity, with each shard attempting to balance the number of individual tests it runs. This is the preferred mode for ensuring even load distribution when sharding.

Sharding without fullyParallel uses file-level granularity

Without the fullyParallel setting, Playwright Test defaults to file-level granularity, meaning entire test files are assigned to shards. If test files are not evenly sized (some containing many more tests than others), certain shards may end up running significantly more tests while others run fewer or none. To balance shards without fullyParallel, keep test files small and evenly sized.

Merge multiple shard blob reports with merge-reports command

To merge blob reports from multiple shards, place all blob report files into a single directory (for example all-blob-reports), then run: npx playwright merge-reports --reporter html ./all-blob-reports. This produces a standard HTML report into the playwright-report directory. Blob report names contain shard numbers, so they will not clash.

GitHub Actions example: configure matrix sharding

Example GitHub Actions workflow to run tests on four machines in parallel: add a matrix option with shardIndex: [1, 2, 3, 4] and shardTotal: [4]. Run tests with: npx playwright test --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }}. Upload blob reports with actions/upload-artifact@v4, specifying name: blob-report-${{ matrix.shardIndex }} and path: blob-report with retention-days: 1.

GitHub Actions: merge reports job depends on sharded tests job

To ensure execution order, make the merge-reports job depend on the playwright-tests job by adding needs: [playwright-tests]. Set if: ${{ !cancelled() }} to merge reports even if some shards have failed. Download blob reports with actions/download-artifact@v5 using path: all-blob-reports and pattern: blob-report-* with merge-multiple: true.

Tag tests for multiple environments with TestConfig.tag

To run the same tests in multiple environments (not sharded across machines), use the TestConfig.tag property to tag all tests with the environment name. This tag is automatically picked up by the blob report and merge tool. Example: tag: process.env.CI_ENVIRONMENT_NAME.

merge-reports CLI basic usage

The command npx playwright merge-reports path/to/blob-reports-dir reads all blob reports from the passed directory and merges them into a single report.

merge-reports --reporter option

The --reporter option specifies which report format to produce when merging. It can be a single reporter or multiple reporters separated by comma. Example: npx playwright merge-reports --reporter=html,github ./blob-reports

merge-reports --config option

The --config option specifies the Playwright configuration file with output reporters. Use this to pass additional configuration to the output reporter. The configuration file can differ from the one used during creation of blob reports. This is required when merging reports from different operating systems. Example: npx playwright merge-reports --config=merge.config.ts ./blob-reports

HTML reporter output directory

By default, the HTML report is written to the playwright-report folder in the current working directory. This location can be overridden using the PLAYWRIGHT_HTML_OUTPUT_DIR environment variable or the outputFolder configuration option.

Opening HTML reports with show-report command

Use npx playwright show-report to open the last test run report. Pass a custom folder name as argument for a specific folder, for example npx playwright show-report my-report. You can also pass a .zip archive such as one downloaded from CI artifacts; the archive must contain index.html at its top level and Playwright will extract and serve it.

JUnit reporter configuration options

JUnit reporter configuration options: PLAYWRIGHT_JUNIT_OUTPUT_DIR (no config option, default: cwd or config directory) directory to save output file, ignored if output file is specified; PLAYWRIGHT_JUNIT_OUTPUT_NAME or outputFile (both refer to same thing, default: prints to stdout) base file name relative to output dir; PLAYWRIGHT_JUNIT_OUTPUT_FILE (use outputFile) full path to output file, if defined previous options are ignored; stripANSIControlSequences (env: PLAYWRIGHT_JUNIT_STRIP_ANSI, default: false) removes ANSI control sequences from text; includeProjectInTestName (env: PLAYWRIGHT_JUNIT_INCLUDE_PROJECT_IN_TEST_NAME, default: false) includes Playwright project name as prefix in test case names; omitTags (env: PLAYWRIGHT_JUNIT_OMIT_TAGS, default: false) omits test tags from failure details; PLAYWRIGHT_JUNIT_SUITE_ID (no config option, default: empty string) sets id attribute on root <testsuites/> entry; PLAYWRIGHT_JUNIT_SUITE_NAME (no config option, default: empty string) sets name attribute on root <testsuites/> entry.

GitHub Actions reporter annotations

The built-in github reporter provides automatic failure annotations when running in GitHub Actions. All other reporters work on GitHub Actions but do not provide annotations. It is not recommended to use this annotation type if running tests with a matrix strategy as stack trace failures will multiply and obscure the GitHub file view.

GitHub Actions reporter omitTags option

The github reporter accepts omitTags configuration option (or PLAYWRIGHT_GITHUB_OMIT_TAGS environment variable) to suppress test tags in its annotations, for example reporter: [['github', { omitTags: true }]].

Multiple reporters simultaneously

You can use multiple reporters at the same time. For example, use list reporter for nice terminal output and json reporter to get a comprehensive JSON file with test results. Configure as an array in the configuration file: reporter: [['list'], ['json', { outputFile: 'test-results.json' }]].

Conditional reporters for CI and local

You can use different reporters for local and CI environments. For example, use concise dot reporter on CI to avoid excessive output and list reporter locally. Configure as: reporter: process.env.CI ? 'dot' : 'list'.

Custom reporter implementation

Create a custom reporter by implementing a class with reporter methods. Import types FullConfig, FullResult, Reporter, Suite, TestCase, TestResult from @playwright/test/reporter. Implement methods like onBegin(config, suite), onTestBegin(test, result), onTestEnd(test, result), and onEnd(result). Use the reporter in configuration file with reporter: './my-awesome-reporter.ts' or pass as command line option --reporter="./myreporter/my-awesome-reporter.ts".

List reporter default location

The list reporter is the default reporter except on CI where the dot reporter is default.

List reporter configuration options

The list reporter supports these configuration options: printSteps (boolean, default false) controls whether each step prints on its own line; printFailuresInline (boolean, default false) controls whether failure details print immediately after a failed test instead of at the end; omitTags (boolean, default false) controls whether to omit test tags automatically appended to test titles. Environment variables PLAYWRIGHT_LIST_PRINT_STEPS, PLAYWRIGHT_LIST_PRINT_FAILURES_INLINE, and PLAYWRIGHT_LIST_OMIT_TAGS can also be used.

List reporter TTY and color environment variables

PLAYWRIGHT_FORCE_TTY environment variable controls whether output is suitable for a live terminal, supporting values: true, 1, false, 0, [WIDTH], or [WIDTH]x[HEIGHT]. Default is true when terminal is in TTY mode, false otherwise. FORCE_COLOR environment variable controls colored output, defaulting to true in TTY mode and false otherwise. NO_COLOR environment variable (any non-empty value) disables colored output per no-color.org standard.

Line reporter description

The line reporter is more concise than the list reporter. It uses a single line to report the last finished test and prints failures when they occur. Line reporter is useful for large test suites where it shows progress without spamming output by listing all tests.

Line reporter configuration options

The line reporter supports these configuration options: omitTags (boolean, default false) controls whether to omit test tags automatically appended to test titles. Environment variable PLAYWRIGHT_LINE_OMIT_TAGS can also be used. Additionally PLAYWRIGHT_FORCE_TTY, FORCE_COLOR, and NO_COLOR environment variables apply as with other reporters.

Dot reporter description

The dot reporter is very concise, producing a single character per successful test run. It is the default on CI and useful where minimal output is desired.

Dot reporter character meanings

In dot reporter output, characters represent test status: · (middle dot) for passed, F for failed, × (multiplication sign) for failed or timed out and will be retried, ± (plus-minus) for passed on retry (flaky), T for timed out, ° (degree symbol) for skipped.

Dot reporter configuration options

The dot reporter supports these configuration options: omitTags (boolean, default false) controls whether to omit test tags automatically appended to test titles. Environment variable PLAYWRIGHT_DOT_OMIT_TAGS can also be used. Additionally PLAYWRIGHT_FORCE_TTY, FORCE_COLOR, and NO_COLOR environment variables apply as with other reporters.

HTML reporter automatic opening behavior

By default, the HTML report opens automatically if some tests failed. This behavior is controlled via the open property in the Playwright config or PLAYWRIGHT_HTML_OPEN environment variable. Possible values are always, never, and on-failure (default).

HTML reporter attachments base URL

The attachmentsBaseURL option in HTML reporter configuration allows specifying a separate location where attachments from the data subdirectory are uploaded. This is only needed when uploading the report and data folder to different locations.

Reporter specified in configuration file

Reporters can be specified programmatically in the configuration file using the reporter property of defineConfig. This allows for more control than command line specification.

Reporter specified via command line

Reporters can be specified via command line using the --reporter option, for example npx playwright test --reporter=line.

HTML reporter configuration options

HTML reporter configuration options: title (env: PLAYWRIGHT_HTML_TITLE, default: no title) displays in the generated report; outputFolder (env: PLAYWRIGHT_HTML_OUTPUT_DIR, default: playwright-report) directory to save report to; open (env: PLAYWRIGHT_HTML_OPEN, default: on-failure) when to open in browser (always/never/on-failure); host (env: PLAYWRIGHT_HTML_HOST, default: localhost) hostname for serving report; port (env: PLAYWRIGHT_HTML_PORT, default: 9323 or available port) port for serving report; attachmentsBaseURL (env: PLAYWRIGHT_HTML_ATTACHMENTS_BASE_URL, default: data/) separate location for attachments; noCopyPrompt (env: PLAYWRIGHT_HTML_NO_COPY_PROMPT, default: false) disables Copy prompt for errors; noSnippets (env: PLAYWRIGHT_HTML_NO_SNIPPETS, default: false) disables code snippets in action log; doNotInlineAssets (env: PLAYWRIGHT_HTML_DO_NOT_INLINE_ASSETS, default: false) writes JavaScript, CSS and data as separate files instead of inline; mergeFiles (env: PLAYWRIGHT_HTML_MERGE_FILES, default: false) groups tests by top-level test.describe() title instead of file.

Blob reporter purpose

Blob reports contain all details about a test run and can be used later to produce any other report. Their primary function is to facilitate merging reports from sharded tests.

Blob reporter default output directory

By default, blob report is written to the blob-report directory in the package.json directory or current working directory if no package.json is found.

Blob reporter file naming

Blob report file names follow the pattern report-<hash>.zip or report-<hash>-<shard_number>.zip when sharding is used. The hash is computed from --grep, --grepInverted, --project, tag property, and file filters passed as command line arguments, guaranteeing different but stable report names between runs with different options.

Blob reporter configuration options

Blob reporter configuration options: outputDir (env: PLAYWRIGHT_BLOB_OUTPUT_DIR, default: blob-report) directory to save output, existing content is deleted before writing new report; fileName (env: PLAYWRIGHT_BLOB_OUTPUT_NAME, default: report-<project>-<hash>-<shard_number>.zip) report file name; outputFile (env: PLAYWRIGHT_BLOB_OUTPUT_FILE, default: undefined) full path to output file, if defined outputDir and fileName are ignored.

Give your agent this brain