Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/src/best-practices-js.md
Original file line number Diff line number Diff line change
Expand Up @@ -480,7 +480,7 @@ pnpm exec playwright --version

Setup CI/CD and run your tests frequently. The more often you run your tests the better. Ideally you should run your tests on each commit and pull request. Playwright comes with a [GitHub actions workflow](/ci-intro.md) so that tests will run on CI for you with no setup required. Playwright can also be setup on the [CI environment](/ci.md) of your choice.

Use Linux when running your tests on CI as it is cheaper. Developers can use whatever environment when running locally but use linux on CI. Consider setting up [Sharding](./test-sharding.md) to make CI faster.
Use Linux when running your tests on CI as it is cheaper. Developers can use whatever environment when running locally but use linux on CI. Consider using a larger CI runner or setting up [Sharding](./test-sharding.md) to make CI faster.


#### Optimize browser downloads on CI
Expand Down
7 changes: 6 additions & 1 deletion docs/src/ci.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,12 @@ configurations for common CI providers.
## Workers
* langs: js

We recommend setting [workers](./api/class-testconfig.md#test-config-workers) to "1" in CI environments to prioritize stability and reproducibility. Running tests sequentially ensures each test gets the full system resources, avoiding potential conflicts. However, if you have a powerful self-hosted CI system, you may enable [parallel](./test-parallel.md) tests. For wider parallelization, consider [sharding](./test-parallel.md#shard-tests-between-multiple-machines) - distributing tests across multiple CI jobs.
If you have powerful CI machines available, enable [parallel](./test-parallel.md) tests. By default, Playwright uses a number of workers equal to 50% of available CPU cores. If you need finer control, configure the worker count explicitly via [`workers`](./api/class-testconfig.md#test-config-workers).

In some test suites, parallel execution can lead to instability and flakiness due to hidden race conditions and shared state.
If you can't resolve this, set [workers](./api/class-testconfig.md#test-config-workers) to `1` in CI environments to ensure each test gets the full system resources, avoiding potential conflicts.
For wider parallelization, use [sharding](./test-parallel.md#shard-tests-between-multiple-machines) to distribute tests across multiple CI jobs.
This comes at the cost of per-shard overhead (checkout, installing dependencies, downloading browsers, etc).

```js title="playwright.config.ts"
import { defineConfig, devices } from '@playwright/test';
Expand Down
2 changes: 1 addition & 1 deletion docs/src/test-parallel-js.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ title: "Parallelism"
Playwright Test runs tests in parallel. In order to achieve that, it runs several worker processes that run at the same time. By default, **test files** are run in parallel. Tests in a single file are run in order, in the same worker process.

- You can configure tests using [`test.describe.configure`](#parallelize-tests-in-a-single-file) to run **tests in a single file** in parallel.
- You can configure **entire project** to have all tests in all files to run in parallel using [`property: TestProject.fullyParallel`] or [`property: TestConfig.fullyParallel`].
- You can configure the **entire project** to have all tests in all files to run in parallel using [`property: TestProject.fullyParallel`] or [`property: TestConfig.fullyParallel`].
- To **disable** parallelism limit the number of [workers to one](#disable-parallelism).

You can control the number of [parallel worker processes](#limit-workers) and [limit the number of failures](#limit-failures-and-fail-fast) in the whole test suite for efficiency.
Expand Down
10 changes: 8 additions & 2 deletions docs/src/test-sharding-js.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,12 @@ title: "Sharding"

## Introduction

By default, Playwright runs test files in [parallel](./test-parallel.md) and strives for optimal utilization of CPU cores on your machine. In order to achieve even greater parallelisation, you can further scale Playwright test execution by running tests on multiple machines simultaneously. We call this mode of operation "sharding". Sharding in Playwright means splitting your tests into smaller parts called "shards". Each shard is like a separate job that can run independently. The whole purpose is to divide your tests to speed up test runtime.
By default, Playwright runs test files in [parallel](./test-parallel.md) and strives for optimal utilization of CPU cores on your machine.
To speed up your test runs, often increasing the number of available CPU cores in your CI environment and [not limiting workers](./ci.md#workers) is the best way.

For some test suites, this can lead to instability and flakiness due to hidden race conditions and shared state.
If you face this issue and can't resolve it, you can instead run tests on multiple machines simultaneously.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd not recommend shards as a solution to flakiness. This creates an impression that sharding is a tool for reducing flakiness rather than improving run time.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The point I want to get across is that sharding is a tool for improving runtime that, compared to parallelism, is less susceptible to introducing flakiness. I think it's an important distinction, but I agree it's hard to express this without creating the wrong impression 🤔

We call this mode of operation "sharding". Sharding in Playwright means splitting your tests into smaller parts called "shards". Each shard is like a separate job that can run independently. The whole purpose is to divide your tests to speed up test runtime.

When you shard your tests, each shard can run on its own, utilizing the available CPU cores. This helps speed up the testing process by doing tasks simultaneously.

Expand All @@ -22,7 +27,8 @@ npx playwright test --shard=3/4
npx playwright test --shard=4/4
```

Now, if you run these shards in parallel on different jobs, your test suite completes four times faster.
Now, if you run these shards in parallel on different jobs, your test suite completes roughly four times faster.
Keep in mind that each shard incurs some overhead (checkout, installing dependencies, downloading browsers, etc), so keep the number of shards reasonable to avoid diminishing returns.

Note that Playwright can only shard tests that can be run in parallel. By default, this means Playwright will shard test files. Learn about other options in the [parallelism guide](./test-parallel.md).

Expand Down