Sharding Playwright runs

Outside GitHub Actions, each playwright test creates its own run by default, so four shards make four runs. In GitHub Actions the shards of one workflow run share one run automatically (parallel jobs). Elsewhere, to get one run for the whole pipeline, pick one of two patterns. Every job that reports needs PROBARA_API_TOKEN, set up as in your CI provider’s section.

Pattern	Choose it when
1. Merge the blob reports	Simplest, and Playwright’s own way. One job reports; no open run if a shard crashes.
2. Create the run, report from each shard, close it	Results should appear while the shards run.

Parallel workers of one playwright test need nothing: they all report through the one reporter of the main process, into one run.

Pattern 1: merge the blob reports

Each shard writes a Playwright blob report and keeps it as an artifact; a last job merges them with the reporter. The merged run holds every attempt of every shard, with the same keys as one run of the whole suite, and the files the blob reports carry (screenshots, traces, testInfo.attach() files).

1. Register blob on CI and the reporter elsewhere:

   playwright.config.ts

   import { defineConfig } from '@playwright/test';
   
   export default defineConfig({
     reporter: process.env.CI ? 'blob' : [['list'], ['@probara/playwright-reporter']],
   });

2. Run each shard: npx playwright test --shard=1/4 writes blob-report/. The shards need no token.

3. Collect every blob-report/ into one folder, and merge it with the token in the environment:

   npx playwright merge-reports --reporter @probara/playwright-reporter ./all-blob-reports

To give the reporter options while merging, put them in a config for the merge (npx playwright merge-reports --config merge.config.ts ./all-blob-reports):

merge.config.ts

export default {
  testDir: './tests',
  reporter: [['@probara/playwright-reporter', { projectId: 'SHOP', run: { tags: ['sharded'] } }]],
};

Keep testDir the same as in playwright.config: the keys are relative to it. Complete workflows are in the GitHub Actions, GitLab CI and Bitbucket Pipelines sections.

Pattern 2: share one run

Create the run, report from each shard, close it:

1. A first job runs probara run create, which creates an empty automated run and prints its ULID on stdout.
2. Every shard runs with that ULID in PROBARA_RUN_ULID. A shard never closes a run it did not create, so they all report into the open run. Probara reconciles shards that report at the same moment: no case is added to the run twice.
3. A last job runs probara run close, after every shard, even when shards failed.

In a shell, the whole flow is:

PROBARA_RUN_ULID=$(npx @probara/cli run create)
export PROBARA_RUN_ULID
npx playwright test --shard=1/2
npx playwright test --shard=2/2
npx @probara/cli run close

Assign the ULID on its own line: export PROBARA_RUN_ULID=$(...) would hide a failed creation. In CI, the jobs below pin the CLI (npx --yes @probara/cli@0.2), which is not a dependency of the reporter; with @probara/cli in your devDependencies, npx @probara/cli after npm ci runs the locked version instead. probara is the Probara CLI; it takes the same token and project. Options of a new run go to run create (--run-name, --environment, --milestone, --plan, --tag…), since the shards reuse the run and ignore their own run options.

GitHub Actions

jobs:
  probara-run:
    runs-on: ubuntu-latest
    outputs:
      ulid: ${{ steps.create.outputs.ulid }}
    env:
      PROBARA_API_TOKEN: ${{ secrets.PROBARA_API_TOKEN }}
      PROBARA_PROJECT: SHOP
      # A fork's build gets no secret: reporting off, so run create prints nothing and exits 0.
      PROBARA_ENABLED: ${{ secrets.PROBARA_API_TOKEN != '' }}
    steps:
      - uses: actions/setup-node@v5
        with:
          node-version: 22
      - id: create
        run: |
          ulid=$(npx --yes @probara/cli@0.2 run create)
          echo "ulid=$ulid" >> "$GITHUB_OUTPUT"

  test:
    needs: probara-run
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        shardIndex: [1, 2, 3, 4]
        shardTotal: [4]
    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-node@v5
        with:
          node-version: 22
      - run: npm ci
      - run: npx playwright install --with-deps
      - run: npx playwright test --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }}
        env:
          PROBARA_API_TOKEN: ${{ secrets.PROBARA_API_TOKEN }}
          PROBARA_PROJECT: SHOP
          PROBARA_RUN_ULID: ${{ needs.probara-run.outputs.ulid }}

  close-probara-run:
    needs: [probara-run, test]
    if: always() && needs.probara-run.outputs.ulid != ''
    runs-on: ubuntu-latest
    steps:
      - uses: actions/setup-node@v5
        with:
          node-version: 22
      - run: npx --yes @probara/cli@0.2 run close
        env:
          PROBARA_API_TOKEN: ${{ secrets.PROBARA_API_TOKEN }}
          PROBARA_PROJECT: SHOP
          PROBARA_RUN_ULID: ${{ needs.probara-run.outputs.ulid }}

Without a token the CLI exits 2, so the first job sets PROBARA_ENABLED from whether the secret is there: in a fork’s build, run create prints nothing and exits 0, the shards report nothing, and the close job is skipped.

GitLab CI

A dotenv artifact hands the ULID to every later job:

stages:
  - test

create-probara-run:
  stage: .pre
  image: node:22
  variables:
    PROBARA_PROJECT: SHOP
  rules:
    - if: $PROBARA_API_TOKEN
  script:
    - PROBARA_RUN_ULID=$(npx --yes @probara/cli@0.2 run create)
    - echo "PROBARA_RUN_ULID=$PROBARA_RUN_ULID" > probara.env
  artifacts:
    reports:
      dotenv: probara.env

playwright:
  stage: test
  image: mcr.microsoft.com/playwright:v1.56.1-noble
  parallel: 4
  variables:
    PROBARA_PROJECT: SHOP
  script:
    - npm ci
    - npx playwright test --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL

close-probara-run:
  stage: .post
  image: node:22
  variables:
    PROBARA_PROJECT: SHOP
  rules:
    - if: $PROBARA_API_TOKEN
      when: always
  script:
    - npx --yes @probara/cli@0.2 run close

.pre and .post are GitLab’s built-in first and last stages; when: always closes the run even after failed shards. CircleCI, Azure Pipelines and Buildkite use the same pattern: see their CI provider sections.

If the close never ran

A run stays open until it is closed. Close it by hand, with the ULID from the log of the create job:

npx @probara/cli run close --run-ulid 01J9Z3K4M5N6P7Q8R9S0T1V2W3

Which pattern loses less

Something goes wrong	Pattern 1 (merge)	Pattern 2 (shared run)
A shard crashes before it finishes	Its blob report is missing; the rest is reported	Its results so far are in the run
Probara is unreachable	The merge job fails to report; re-run it	Each shard’s results file keeps it (a sibling file on a shared disk)
The close job does not run	Nothing to close	The run stays open: close it by hand

Several Probara projects

One Playwright suite can hold the tests of several Probara projects: a shared end-to-end suite whose cases live in SHOP, WEB and API. List the other projects in projects (PROBARA_PROJECTS=WEB,API), and each result goes to the project of its case, into a run of that project.

reporter: [['@probara/playwright-reporter', { projectId: 'SHOP', projects: ['WEB'] }]],

tests/shared.spec.ts

import { test } from '@playwright/test';

test('SHOP-12 pays', async () => {});

test('WEB-3 shows the landing page', async () => {});

test(
  'answers the health check',
  { annotation: { type: 'probara_case', description: 'API-7' } },
  async () => {},
);

test('opens the help', async () => {});

$ npx playwright test
[probara] Did not send the results linked to cases of API: API is neither the project (SHOP) nor one of projects (PROBARA_PROJECTS) (first seen in "answers the health check"; repeats are logged at debug)
[probara] Sending 3 results of 3 tests (3 passed, 0 failed, 0 skipped, 0 blocked); 1 linked only to cases of unlisted projects
[probara] Recorded 2 results (1 new case, 0 unmatched) in R-1 of SHOP (closed): https://app.probara.net/projects/SHOP/runs/R-1
[probara] Recorded 1 result (0 new cases, 0 unmatched) in R-2 of WEB (closed): https://app.probara.net/projects/WEB/runs/R-2

What goes where

• By the case’s project. A result linked to WEB-3 goes into a run of WEB. The ids of every listed project link from titles too, and leave the key, like the configured project’s.
• A test linked to cases of several projects is sent once per case, each to its project, with its files uploaded to each result.
• A test without a case goes to the configured project (projectId), where it is matched by its key or created: automation keys belong to one project, so only the configured project creates cases, under suiteUlid.
• A case of a project that is not listed is not sent: it would land in the wrong project, where Probara refuses its id. One warning per project says so, and the count in the log leaves those attempts out. List the project to report it.

One run per project

Each listed project gets its own run, created with its first result (a project without results gets no run), with the same name, description, environment (by name), tags and CI source as the configured project’s, and closed at the end like any run the reporter creates.

The milestone, plan and configurations are planning entities of one project: a name that exists in SHOP may not exist in WEB, where Probara would refuse the whole report. So run.milestone, run.plan and run.configurations, like the ULID forms (run.milestoneId, run.configurationUlids, run.environmentId), only go with the configured project’s run, and one warning names them. To give another project’s run its own, create it first and reuse it:

WEB_RUN=$(npx @probara/cli run create --project WEB --milestone "Web sprint 4")
PROBARA_RUN_ULIDS="WEB=$WEB_RUN" PROBARA_PROJECTS=WEB npx playwright test

run.ulids (PROBARA_RUN_ULIDS=SHOP=<ulid>,WEB=<ulid>) names the run to reuse in each project; the configured project’s entry counts as run.ulid. A reused run stays open unless closeRun is true: close it with probara run close --project WEB --run-ulid <ulid>.

Sharded jobs with several projects

Create one run per project before the shards, pass them all in PROBARA_RUN_ULIDS, and close them after the last shard:

SHOP_RUN=$(npx @probara/cli run create)
WEB_RUN=$(npx @probara/cli run create --project WEB)
export PROBARA_PROJECTS=WEB PROBARA_RUN_ULIDS="SHOP=$SHOP_RUN,WEB=$WEB_RUN"
npx playwright test --shard=1/2
npx @probara/cli run close --run-ulid "$SHOP_RUN"
npx @probara/cli run close --project WEB --run-ulid "$WEB_RUN"

When the configured project’s run is reused but a listed project has no run in run.ulids, each shard creates its own run in that project, and a warning says so.

Failures

A failed report stops only its project: the others go on. The log names the project of every error (WEB: ...), and a results file keeps what each project could not send, with the run it goes back into: one file for the whole run, which probara import results 'probara-results*.json' sends to each project.

Next: CI providers, or the probara run create and run close reference.