Skip to content
Docs · DevelopersProbara Docs

Sharding Jest and Jest projects

Outside GitHub Actions, each jest command creates its own run by default, so four shards (--shard=1/4 … --shard=4/4) make four runs. In GitHub Actions the shards of one workflow run share one run automatically (parallel jobs). Elsewhere, this page shows how to get one run for a whole pipeline, how Jest’s own projects behave, and how one test suite reports into several Probara projects.

Parallel workers of one jest (--maxWorkers) need nothing: they all report through the one reporter of Jest’s main process, into one run.

One run for every shard

Section titled “One run for every shard”
PatternChoose it when
Create the run, report from each shard, close itThe usual way: results appear while the shards run
Results files, imported by one jobThe shards cannot hold the token, or one job should do all the sending

Every job that reports needs PROBARA_API_TOKEN, an app token from the Jest card in Workspace › Integrations, set up as in your CI provider’s section.

Create the run, report from each shard, close it

Section titled “Create the run, report from each shard, close it”
  1. A first job runs probara run create, which creates an empty automated run and prints its ULID on stdout.
  2. Every shard runs npx jest --shard=<i>/<n> with that ULID in PROBARA_RUN_ULID. A shard never closes a run it did not create, so they all report into the open run. Probara reconciles shards that report at the same moment: no case is added to the run twice.
  3. A last job runs probara run close, after every shard, even when shards failed.

In a shell, the whole flow is:

Terminal window
PROBARA_RUN_ULID=$(npx @probara/cli run create)
export PROBARA_RUN_ULID
npx jest --shard=1/2
npx jest --shard=2/2
npx @probara/cli run close

Assign the ULID on its own line: export PROBARA_RUN_ULID=$(...) would hide a failed creation. In CI, the jobs below pin the CLI (npx --yes @probara/cli@0.2), which is not a dependency of the reporter; with @probara/cli in your devDependencies, npx @probara/cli after npm ci runs the locked version instead. probara is the Probara CLI; it takes the same token and project. Options of a new run go to run create (--run-name, --environment, --milestone, --plan, --tag…), since the shards reuse the run and ignore their own run options.

GitHub Actions

Section titled “GitHub Actions”
jobs:
  probara-run:
    runs-on: ubuntu-latest
    outputs:
      ulid: ${{ steps.create.outputs.ulid }}
    env:
      PROBARA_API_TOKEN: ${{ secrets.PROBARA_API_TOKEN }}
      PROBARA_PROJECT: SHOP
      # A fork's build gets no secret: reporting off, so run create prints nothing and exits 0.
      PROBARA_ENABLED: ${{ secrets.PROBARA_API_TOKEN != '' }}
    steps:
      - uses: actions/setup-node@v5
        with:
          node-version: 22
      - id: create
        run: |
          ulid=$(npx --yes @probara/cli@0.2 run create)
          echo "ulid=$ulid" >> "$GITHUB_OUTPUT"


  test:
    needs: probara-run
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        shardIndex: [1, 2, 3, 4]
        shardTotal: [4]
    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-node@v5
        with:
          node-version: 22
      - run: npm ci
      - run: npx jest --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }}
        env:
          PROBARA_API_TOKEN: ${{ secrets.PROBARA_API_TOKEN }}
          PROBARA_PROJECT: SHOP
          PROBARA_RUN_ULID: ${{ needs.probara-run.outputs.ulid }}


  close-probara-run:
    needs: [probara-run, test]
    if: always() && needs.probara-run.outputs.ulid != ''
    runs-on: ubuntu-latest
    steps:
      - uses: actions/setup-node@v5
        with:
          node-version: 22
      - run: npx --yes @probara/cli@0.2 run close
        env:
          PROBARA_API_TOKEN: ${{ secrets.PROBARA_API_TOKEN }}
          PROBARA_PROJECT: SHOP
          PROBARA_RUN_ULID: ${{ needs.probara-run.outputs.ulid }}

Without a token the CLI exits 2, so the first job sets PROBARA_ENABLED from whether the secret is there: in a fork’s build, run create prints nothing and exits 0, the shards report nothing, and the close job is skipped.

GitLab CI

Section titled “GitLab CI”

A dotenv artifact hands the ULID to every later job:

stages:
  - test


create-probara-run:
  stage: .pre
  image: node:22
  variables:
    PROBARA_PROJECT: SHOP
  rules:
    - if: $PROBARA_API_TOKEN
  script:
    - PROBARA_RUN_ULID=$(npx --yes @probara/cli@0.2 run create)
    - echo "PROBARA_RUN_ULID=$PROBARA_RUN_ULID" > probara.env
  artifacts:
    reports:
      dotenv: probara.env


jest:
  stage: test
  image: node:22
  parallel: 4
  variables:
    PROBARA_PROJECT: SHOP
  script:
    - npm ci
    - npx jest --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL


close-probara-run:
  stage: .post
  image: node:22
  variables:
    PROBARA_PROJECT: SHOP
  rules:
    - if: $PROBARA_API_TOKEN
      when: always
  script:
    - npx --yes @probara/cli@0.2 run close

.pre and .post are GitLab’s built-in first and last stages; when: always closes the run even after failed shards.

Other CI providers

Section titled “Other CI providers”

Jenkins, CircleCI, Azure Pipelines, Bitbucket Pipelines and Buildkite follow the same three steps; their complete sharded pipelines are in CI providers.

Alternative: results files, imported by one job

Section titled “Alternative: results files, imported by one job”

Each shard runs with reporting off and writes every result to a results file, kept as an artifact with its attachments folder. One last job, which holds the token, downloads them all and sends them. The attachments are referenced relative to each results file, so the files and their folders can be downloaded anywhere.

A results file written with reporting off names no run: imported on its own, each file creates a run of its own. To get one run, create it first, import every file into it, and close it:

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        shardIndex: [1, 2, 3, 4]
        shardTotal: [4]
    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-node@v5
        with:
          node-version: 22
      - run: npm ci
      - run: npx jest --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }}
        env:
          PROBARA_ENABLED: 'false'
          PROBARA_PROJECT: SHOP
          PROBARA_RESULTS_FILE: probara-results-${{ matrix.shardIndex }}.json
      - uses: actions/upload-artifact@v4
        if: ${{ !cancelled() }}
        with:
          name: probara-results-${{ matrix.shardIndex }}
          path: probara-results-*
          retention-days: 1


  report:
    needs: test
    # Forks get no token: there is nothing to send with.
    if: ${{ !cancelled() && github.event.pull_request.head.repo.fork != true }}
    runs-on: ubuntu-latest
    env:
      PROBARA_API_TOKEN: ${{ secrets.PROBARA_API_TOKEN }}
      PROBARA_PROJECT: SHOP
    steps:
      - uses: actions/setup-node@v5
        with:
          node-version: 22
      - uses: actions/download-artifact@v4
        with:
          pattern: probara-results-*
          merge-multiple: true
      - run: |
          PROBARA_RUN_ULID=$(npx --yes @probara/cli@0.2 run create)
          export PROBARA_RUN_ULID
          npx --yes @probara/cli@0.2 import results 'probara-results*.json'
          npx --yes @probara/cli@0.2 run close
  • Give each shard its own file name (probara-results-<shard>.json), so the downloads never overwrite each other.
  • probara import results sends every file into PROBARA_RUN_ULID, leaves that run open (it reused it), and deletes each file once sent. run close then closes it.
  • Nothing reaches Probara until the last job: a crashed shard leaves no results file, and its results are missing.
  • Re-running the report job runs run create again: the shards’ files, downloaded again as they were, go into a second run with the same results. Close or delete the one you do not keep.

If the close never ran

Section titled “If the close never ran”

A run stays open until it is closed. Close it by hand, with the ULID from the log of the create job:

Terminal window
npx @probara/cli run close --run-ulid 01J9Z3K4M5N6P7Q8R9S0T1V2W3

Which approach loses less

Section titled “Which approach loses less”
Something goes wrongShared run (create, shards, close)Results files, one import
A shard crashes before it finishesIts results so far are in the runIts file is missing: none of its results are sent
Probara is unreachableEach shard’s results file keeps what it could not send (if one is set)The import exits 1 and keeps in each file what it could not send
The close job does not runThe run stays open: close it by handSame

Jest projects

Section titled “Jest projects”

Two different things are called projects here. Jest’s projects split one jest command into several configs (a node and a jsdom one, say); the reporter’s projects option lets one test suite report to several Probara projects. Each works on its own.

Register the reporter once, at the top level of the config, next to projects: Jest runs one set of reporters for all its projects, and the reporter sends every project’s tests into one Probara run.

jest.config.js
module.exports = {
  reporters: ['default', ['@probara/jest-reporter', { projectId: 'SHOP' }]],
  projects: [
    {
      displayName: 'api',
      testEnvironment: 'node',
      testMatch: ['<rootDir>/tests/api/**/*.test.js'],
    },
    { displayName: 'ui', testEnvironment: 'jsdom', testMatch: ['<rootDir>/tests/ui/**/*.test.js'] },
  ],
};
[
  { "automationKey": "tests/ui/cart.test.js > shows the cart" },
  { "automationKey": "tests/api/orders.test.js > lists the orders" }
]
  • Keys have no project. As in jest-junit’s report, a Jest project adds nothing to the automation key: the file (relative to the directory jest runs in, not to the project’s rootDir), then the names. Moving tests between Jest projects keeps their cases.
  • A file that two projects run (their testMatch overlap, such as a shared folder) is reported once per project, under the same key: two results of one case, like two retries.
  • Helpers in a file two projects run at once. Jest may run both at the same moment, in two workers, and a probara.* call cannot tell which project’s run it belongs to. What the helpers said about that file’s tests is then left out, with one warning that names the file; the results are sent all the same. Give each project its own files, or run with --runInBand, to keep them.
  • Setup files per project. setupFilesAfterEnv: ['@probara/jest-reporter/setup'] (for captureOutput and run selection) is a project option: give it to every project whose tests need it.
  • reporters inside a project is ignored by Jest: register the reporter at the top level.

{displayName} in a title

Section titled “{displayName} in a title”

jest-junit fills {displayName} in a test’s name with the name of the Jest project that runs it, and so does the reporter, so a test that runs in two projects can have a key, and a case, per project:

jest.config.js
module.exports = {
  reporters: ['default', ['@probara/jest-reporter', { projectId: 'SHOP' }]],
  projects: [
    { displayName: 'node', testEnvironment: 'node' },
    { displayName: 'browser', testEnvironment: 'jsdom' },
  ],
};
tests/format.test.js
test('formats prices in {displayName}', () => {});
[
  { "automationKey": "tests/format.test.js > formats prices in browser" },
  { "automationKey": "tests/format.test.js > formats prices in node" }
]

A project without a displayName fills in undefined, as jest-junit does. When the two projects run such a file at the same moment, each sends only the last attempt of such a test, with one warning: Jest does not say which project a single attempt belongs to until the file ends.

Several Probara projects

Section titled “Several Probara projects”

One Jest suite can hold the tests of several Probara projects: a shared suite whose cases live in SHOP, WEB and API. List the other projects in projects, and each result goes to the project of its case, into a run of that project.

reporters: ['default', ['@probara/jest-reporter', { projectId: 'SHOP', projects: ['WEB'] }]],
tests/shared.test.js
const { probara } = require('@probara/jest-reporter');


test('SHOP-12 pays', () => {});


test('WEB-3 shows the landing page', () => {});


test('API-7 answers the health check', () => {});


test('opens the help', () => {
  probara.id('API-9');
});
$ npx jest
[probara] Did not send the results linked to cases of API: API is neither the project (SHOP) nor one of projects (PROBARA_PROJECTS) (first seen in "opens the help"; repeats are logged at debug)
[probara] Sending 3 results of 3 tests (3 passed, 0 failed, 0 skipped, 0 blocked); 1 linked only to cases of unlisted projects
[probara] Recorded 2 results (1 new case, 0 unmatched) in R-1 of SHOP (closed): https://app.probara.net/projects/SHOP/runs/R-1
[probara] Recorded 1 result (0 new cases, 0 unmatched) in R-2 of WEB (closed): https://app.probara.net/projects/WEB/runs/R-2

PROBARA_PROJECTS=WEB,API lists them from the environment.

What goes where

Section titled “What goes where”
  • By the case’s project. A result linked to WEB-3 goes into a run of WEB. The ids of every listed project link from titles too, and leave the key, like the configured project’s.
  • A test linked to cases of several projects is sent once per case, each to its project, with its files uploaded to each result.
  • A test without a case goes to the configured project (projectId), where it is matched by its key or created: automation keys belong to one project, so only the configured project creates cases, under suiteUlid.
  • An id of a project that is not listed in a title (API-7 above) is no id: it stays in the title and the key, like any text, and the test goes to the configured project. Given to probara.id() (API-9), it is a case the reporter cannot send anywhere: it is left out, with one warning per project, and the count in the log says so. List the project to report it.

One run per project

Section titled “One run per project”

Each listed project gets its own run, created with its first result (a project without results gets no run), with the same name, description, environment (by name), tags and CI source as the configured project’s, and closed at the end like any run the reporter creates.

The milestone, plan and configurations are planning entities of one project: a name that exists in SHOP may not exist in WEB, where Probara would refuse the whole report. So run.milestone, run.plan and run.configurations, like their ULID forms (run.milestoneId, run.configurationUlids, run.environmentId), only go with the configured project’s run, and one warning names them. To give another project’s run its own, create it first and reuse it:

Terminal window
WEB_RUN=$(npx @probara/cli run create --project WEB --milestone "Web sprint 4")
PROBARA_RUN_ULIDS="WEB=$WEB_RUN" PROBARA_PROJECTS=WEB npx jest

run.ulids (PROBARA_RUN_ULIDS=SHOP=<ulid>,WEB=<ulid>) names the run to reuse in each project; the configured project’s entry counts as run.ulid. A reused run stays open unless closeRun is true: close it with probara run close --project WEB --run-ulid <ulid>.

Sharded jobs with several projects

Section titled “Sharded jobs with several projects”

Create one run per project before the shards, pass them all in PROBARA_RUN_ULIDS, and close them after the last shard:

Terminal window
SHOP_RUN=$(npx @probara/cli run create)
WEB_RUN=$(npx @probara/cli run create --project WEB)
export PROBARA_PROJECTS=WEB PROBARA_RUN_ULIDS="SHOP=$SHOP_RUN,WEB=$WEB_RUN"
npx jest --shard=1/2
npx jest --shard=2/2
npx @probara/cli run close --run-ulid "$SHOP_RUN"
npx @probara/cli run close --project WEB --run-ulid "$WEB_RUN"

When the configured project’s run is reused but a listed project has no run in run.ulids, each shard creates its own run in that project, and a warning says so.

Failures

Section titled “Failures”

A failed report stops only its project: the others go on. The log names the project of every error (WEB: ...), and a results file keeps what each project could not send, with the run it goes back into: one file for the whole run, which probara import results 'probara-results*.json' sends to each project.

Related: CI providers · Results and retries · CLI · Source: ci/sharding.md, projects.md