Skip to content
Docs · DevelopersProbara Docs

Sharding Vitest and Vitest projects

Outside GitHub Actions, each vitest run --shard makes its own run unless the shards are tied together. In GitHub Actions the shards of one workflow run share one run automatically (parallel jobs). Elsewhere, this page shows the two ways to get one run for a whole pipeline, how Vitest’s own projects behave, and how one test suite reports into several Probara projects.

The workers of one vitest run need nothing: they all report through the one reporter of Vitest’s main process, into one run.

One run for every shard

Section titled “One run for every shard”
PatternChoose it when
1. Merge the blob reportsThe usual way, and Vitest’s own: one job reports every shard, and the shards need no token
2. Create the run, report from each shard, close itEach shard’s results should reach Probara as soon as it ends, runCasesOnly should apply, or no job can merge at the shards’ path

The jobs that report need PROBARA_API_TOKEN, an app token from the Vitest card in Workspace › Integrations, set up as in your CI provider’s section.

Pattern 1: merge the blob reports

Section titled “Pattern 1: merge the blob reports”

Each shard writes a Vitest blob report, with the reporter beside the blob reporter, and one job merges the blobs: vitest --merge-reports reports every attempt of every shard, with its steps and files, into one run, and closes it.

  1. Each shard runs with both reporters on its command line. --reporter replaces the reporters of the config, so name this one beside blob: npx vitest run --shard=1/4 --reporter=blob --reporter=@probara/vitest-reporter. The shard writes its blob, keeps the files its tests attached, sends nothing, and says so. It needs no token, and none of the reporter’s options: the merge reports with those of the config.
  2. Copy the blobs and the attachments of every shard into the job that merges, at the same path as on the shards (below).
  3. That job runs npx vitest --merge-reports with the token: the reporter of the config sends every shard’s results into one run.

In one shell, the whole flow is:

$ npx vitest run --shard=1/2 --reporter=blob --reporter=@probara/vitest-reporter
[probara] Sharded run with the blob reporter: the results are reported by vitest --merge-reports
$ npx vitest run --shard=2/2 --reporter=blob --reporter=@probara/vitest-reporter
[probara] Sharded run with the blob reporter: the results are reported by vitest --merge-reports
$ npx vitest --merge-reports
[probara] Sending 3 results of 3 tests (3 passed, 0 failed, 0 skipped, 0 blocked)
[probara] Recorded 3 results (2 new cases, 0 unmatched) in R-1 (closed): https://app.probara.net/projects/SHOP/runs/R-1

The complete workflows are in the GitHub Actions and GitLab CI sections of CI providers.

The reporter goes beside blob

Section titled “The reporter goes beside blob”
The shards run withWhat the merge sends
--reporter=blob --reporter=@probara/vitest-reporterEvery attempt, the probara.* calls and their files
--reporter=blob aloneOnly the final attempt of a retried test (one warning says how many), and no probara.* call: the reporter never ran in the shards (when retries run)

Keep blob out of the config vitest --merge-reports reads: Vitest refuses to merge with it (Cannot merge reports when `--reporter=blob` is used).

Carry the blobs and the attachments

Section titled “Carry the blobs and the attachments”
WhatVitest 3 and 4Vitest 5
The blobs, which vitest --merge-reports reads.vitest-reports/.vitest/blob/
attachmentsDir: the files the tests attached.vitest-attachments/.vitest/attachments/

Keep both folders of every shard as artifacts, and download them all into the same folders of the merging job: each blob has a name of its own (blob-1-4.json), and so has each copy of an attached file. A file the merge cannot find is skipped, with the warning Skipped an attachment: its file is missing or unreadable, and its result is sent all the same (where the copies are kept).

Merge at the same path

Section titled “Merge at the same path”

A blob names each test file, and each attached file, by its full path on the shard, and an automation key starts with the test file’s path from the root of the merging run. Merged at another path, every key changes, and every test gets a new case. Run the merge in a checkout at the same path as the shards. The logs show it: the root the merge’s RUN line names must be the folder that holds the .vitest folder (.vitest-reports on Vitest 3 and 4) of each shard’s blob report written to line.

The same Vitest everywhere

Section titled “The same Vitest everywhere”

vitest --merge-reports reads only the blobs of its own Vitest version, and fails on any other. Install from the lockfile in every job (npm ci), and upgrade the shards and the merging job together (any upgrade of a sharded pipeline).

Run selection

Section titled “Run selection”

runCasesOnly does not apply to shards that write a blob: each shard warns (runCasesOnly does not apply to a sharded run with the blob reporter: every test runs, and vitest --merge-reports reports them all), runs every test, and the merge reports them all. To run only the tests of a Probara run on shards, use pattern 2 (run selection).

Pattern 2: create the run, report from each shard, close it

Section titled “Pattern 2: create the run, report from each shard, close it”
  1. A first job runs probara run create, which creates an empty automated run and prints its ULID on stdout.
  2. Every shard runs npx vitest run --shard=<i>/<n> with that ULID in PROBARA_RUN_ULID, and sends its results when it ends. A shard never closes a run it did not create, so they all report into the open run. Probara reconciles shards that report at the same moment: no case is added to the run twice.
  3. A last job runs probara run close, after every shard, even when shards failed.

In a shell, the whole flow is:

Terminal window
PROBARA_RUN_ULID=$(npx @probara/cli run create)
export PROBARA_RUN_ULID
npx vitest run --shard=1/2
npx vitest run --shard=2/2
npx @probara/cli run close

Assign the ULID on its own line: export PROBARA_RUN_ULID=$(...) would hide a failed creation. probara is the Probara CLI; it takes the same token and project. Options of a new run go to run create (--run-name, --environment, --milestone, --plan, --tag…): the shards reuse the run, and ignore their own run options with one warning (an existing run). With runCasesOnly, each shard reads the cases of the run, and runs only those of its tests that belong to the run (run selection).

GitHub Actions

Section titled “GitHub Actions”
jobs:
  probara-run:
    runs-on: ubuntu-latest
    outputs:
      ulid: ${{ steps.create.outputs.ulid }}
    env:
      PROBARA_API_TOKEN: ${{ secrets.PROBARA_API_TOKEN }}
      PROBARA_PROJECT: SHOP
      # Fork pull requests get no secrets: report nothing there.
      PROBARA_ENABLED: ${{ secrets.PROBARA_API_TOKEN != '' }}
    steps:
      - uses: actions/setup-node@v5
        with:
          node-version: 22
      - id: create
        run: |
          ulid=$(npx @probara/cli run create)
          echo "ulid=$ulid" >> "$GITHUB_OUTPUT"


  test:
    needs: probara-run
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        shardIndex: [1, 2, 3, 4]
        shardTotal: [4]
    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-node@v5
        with:
          node-version: 22
      - run: npm ci
      - run: npx vitest run --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }}
        env:
          PROBARA_API_TOKEN: ${{ secrets.PROBARA_API_TOKEN }}
          PROBARA_PROJECT: SHOP
          PROBARA_ENABLED: ${{ secrets.PROBARA_API_TOKEN != '' }}
          PROBARA_RUN_ULID: ${{ needs.probara-run.outputs.ulid }}


  close-probara-run:
    needs: [probara-run, test]
    if: always() && needs.probara-run.outputs.ulid != ''
    runs-on: ubuntu-latest
    steps:
      - uses: actions/setup-node@v5
        with:
          node-version: 22
      - run: npx @probara/cli run close
        env:
          PROBARA_API_TOKEN: ${{ secrets.PROBARA_API_TOKEN }}
          PROBARA_PROJECT: SHOP
          PROBARA_RUN_ULID: ${{ needs.probara-run.outputs.ulid }}

Without the token (a fork), PROBARA_ENABLED is false: run create prints nothing and exits 0, the shards report nothing, and the close job is skipped.

GitLab CI

Section titled “GitLab CI”

A dotenv artifact hands the ULID to every later job:

stages:
  - test


create-probara-run:
  stage: .pre
  image: node:22
  variables:
    PROBARA_PROJECT: SHOP
  rules:
    - if: $PROBARA_API_TOKEN
  script:
    - PROBARA_RUN_ULID=$(npx @probara/cli run create)
    - echo "PROBARA_RUN_ULID=$PROBARA_RUN_ULID" > probara.env
  artifacts:
    reports:
      dotenv: probara.env


vitest:
  stage: test
  image: node:22
  parallel: 4
  variables:
    PROBARA_PROJECT: SHOP
  script:
    - npm ci
    - npx vitest run --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL


close-probara-run:
  stage: .post
  image: node:22
  variables:
    PROBARA_PROJECT: SHOP
  rules:
    - if: $PROBARA_API_TOKEN
      when: always
  script:
    - npx @probara/cli run close

.pre and .post are GitLab’s built-in first and last stages; when: always closes the run even after failed shards. A pipeline without the variable (a fork’s) has neither job, and its shards report nothing.

Other CI providers

Section titled “Other CI providers”

CI providers shows the same three steps for CircleCI, Jenkins, Azure Pipelines, Bitbucket Pipelines and Buildkite. Pattern 1 works on them too when the merging job gets every shard’s folders at the shards’ path, and runs after failed shards: Bitbucket skips the steps after a failed one.

If the close never ran

Section titled “If the close never ran”

A run stays open until it is closed. Close it by hand, with the ULID from the log of the create job:

Terminal window
npx @probara/cli run close --run-ulid 01J9Z3K4M5N6P7Q8R9S0T1V2W3

Which pattern loses less

Section titled “Which pattern loses less”
Something goes wrongPattern 1 (merge)Pattern 2 (shared run)
A shard is killed before Vitest endsIts blob is missing: the other shards are reportedIt sends nothing: the other shards are reported
Probara is unreachableThe merging job’s results file keeps what it could not send (if set)Each shard’s results file keeps what it could not send (if set)
The close job does not runNothing to close: the merge closes its own runThe run stays open: close it by hand

In both, the exit code of every job stays the tests’ own: a reporting problem never fails a shard or the merge.

Vitest projects

Section titled “Vitest projects”

Two different things are called projects here. Vitest’s projects split one vitest command into several configurations (a unit and a smoke one, say); the reporter’s projects option lets one test suite report to several Probara projects. Each works on its own.

Register the reporter once, in the root config, next to projects: Vitest runs one set of reporters for all its projects, and the reporter sends every project’s tests into one Probara run. It adds its setup file and runner to each project itself.

// vitest.config.mts
import { defineConfig } from 'vitest/config';


export default defineConfig({
  test: {
    reporters: ['default', ['@probara/vitest-reporter', { projectId: 'SHOP' }]],
    projects: [
      { test: { name: 'unit', include: ['tests/**/*.test.ts'] } },
      { test: { name: 'smoke', include: ['tests/cart.test.ts'] } },
    ],
  },
});
tests/cart.test.ts
import { probara } from '@probara/vitest-reporter';
import { test } from 'vitest';


test('adds an item', () => {
  probara.parameters({ currency: 'EUR' });
});
tests/orders.test.ts
import { test } from 'vitest';


test('lists the orders', () => {});
[
  {
    "automationKey": "tests/cart.test.ts > adds an item",
    "parameters": { "project": "unit", "currency": "EUR" }
  },
  {
    "automationKey": "tests/orders.test.ts > lists the orders",
    "parameters": { "project": "unit" }
  },
  {
    "automationKey": "tests/cart.test.ts > adds an item",
    "parameters": { "project": "smoke", "currency": "EUR" }
  }
]
  • Keys have no project. As in Vitest’s JUnit output, a Vitest project adds nothing to the automation key: the file (relative to the Vitest root), then the names. Moving tests between Vitest projects keeps their cases.
  • A file that two projects run (their include overlap, as tests/cart.test.ts above) is reported once per project, under the same key: two results of one case, like two retries.
  • The project parameter names the Vitest project of each result when the config has more than one. A config with one project, or none, sends no project parameter. A project you set with probara.parameters() wins over Vitest’s; attempt and repeat win over both.
  • Each project’s run of a file is a test of its own: its attempts, probara.* calls and files stay with that project’s result, as currency above shows.
  • Nothing to add per project. The reporter puts its setup file first in each project’s setupFiles (your own setup files keep their order) and wraps each project’s runner, so retries, run selection and the helpers work in every project. A browser project gets the browser setup file instead (browser mode).

Several Probara projects

Section titled “Several Probara projects”

One Vitest suite can hold the tests of several Probara projects: a shared suite whose cases live in SHOP, WEB and API. List the other projects in projects, and each result goes to the project of its case, into a run of that project.

reporters: ['default', ['@probara/vitest-reporter', { projectId: 'SHOP', projects: ['WEB'] }]],
tests/shared.test.ts
import { probara } from '@probara/vitest-reporter';
import { test } from 'vitest';


test('SHOP-12 pays', () => {});


test('WEB-3 shows the landing page', () => {});


test('API-7 answers the health check', () => {});


test('opens the help', () => {
  probara.id('API-9');
});
$ npx vitest run
[probara] Did not send the results linked to cases of API: API is neither the project (SHOP) nor one of projects (PROBARA_PROJECTS) (first seen in "opens the help"; repeats are logged at debug)
[probara] Sending 3 results of 3 tests (3 passed, 0 failed, 0 skipped, 0 blocked); 1 linked only to cases of unlisted projects
[probara] Recorded 2 results (1 new case, 0 unmatched) in R-1 of SHOP (closed): https://app.probara.net/projects/SHOP/runs/R-1
[probara] Recorded 1 result (0 new cases, 0 unmatched) in R-2 of WEB (closed): https://app.probara.net/projects/WEB/runs/R-2

PROBARA_PROJECTS=WEB,API lists them from the environment.

What goes where

Section titled “What goes where”
  • By the case’s project. A result linked to WEB-3 goes into a run of WEB. The ids of every listed project link from titles too, and leave the key, like the configured project’s.
  • A test linked to cases of several projects is sent once per case, each to its project, with its files uploaded to each result.
  • A test without a case goes to the configured project (projectId), where it is matched by its key or created: automation keys belong to one project, so only the configured project creates cases, under suiteUlid.
  • An id of a project that is not listed in a title (API-7 above) is no id: it stays in the title and the key, like any text, and the test goes to the configured project. Given to probara.id() (API-9), it is a case the reporter cannot send anywhere: the result would land in the wrong project, where Probara refuses its id. It is left out, with one warning per project, and the count of the Sending line says so. List the project to report it.

One run per project

Section titled “One run per project”

Each listed project gets its own run, created with its first result (a project without results gets no run), with the same name, description, environment (by name), tags and CI source as the configured project’s, and closed at the end like any run the reporter creates.

The milestone, plan and configurations are planning entities of one project: a name that exists in SHOP may not exist in WEB, where Probara would refuse the whole report. So run.milestone, run.plan and run.configurations, like their ULID forms (run.milestoneId, run.configurationUlids, run.environmentId), only go with the configured project’s run, and one warning names them. To give another project’s run its own, create it first and reuse it:

Terminal window
WEB_RUN=$(npx @probara/cli run create --project WEB --milestone "Web sprint 4")
PROBARA_RUN_ULIDS="WEB=$WEB_RUN" PROBARA_PROJECTS=WEB npx vitest run

run.ulids (PROBARA_RUN_ULIDS=SHOP=<ulid>,WEB=<ulid>) names the run to reuse in each project; the configured project’s entry counts as run.ulid. A reused run stays open unless closeRun is true: close it with probara run close --project WEB --run-ulid <ulid>.

Sharded jobs with several projects

Section titled “Sharded jobs with several projects”

Shards that each send their own results (no blob reporter) share runs the same way: create one run per project before the shards, pass them all in PROBARA_RUN_ULIDS, and close them after the last shard (pattern 2):

Terminal window
SHOP_RUN=$(npx @probara/cli run create)
WEB_RUN=$(npx @probara/cli run create --project WEB)
export PROBARA_PROJECTS=WEB PROBARA_RUN_ULIDS="SHOP=$SHOP_RUN,WEB=$WEB_RUN"
npx vitest run --shard=1/2
npx vitest run --shard=2/2
npx @probara/cli run close --run-ulid "$SHOP_RUN"
npx @probara/cli run close --project WEB --run-ulid "$WEB_RUN"

When the configured project’s run is reused but a listed project has no run in run.ulids, each shard creates its own run in that project, and a warning says so. Shards that write a blob send nothing: vitest --merge-reports reports every project’s results once, as one run does.

Failures

Section titled “Failures”

A failed report stops only its project: the others go on. The log names the project of every error (WEB: ...), and a results file keeps what each project could not send, with the run it goes back into: one file for the whole run, which probara import results 'probara-results*.json' sends to each project.

Related: CI providers · Results and retries · CLI · Source: ci/sharding.md, projects.md