Jest reporter in CI

The reporter runs inside npx jest: give the step that runs your tests the token, and every result is sent when the command ends. There is no separate report step. Every example assumes the reporter is registered in jest.config.js as in Get started.

In every CI

The token is a secret, the Project ID is not. Store the app token (from the Jest card in Workspace › Integrations) as a secret named PROBARA_API_TOKEN. Put the Project ID in jest.config.js (projectId) or in the pipeline (PROBARA_PROJECT).
Node.js 22.12 or later on the agent, like the Jest run the reporter is part of.
Failed tests fail the step, reporting problems do not: Jest’s exit code stays the tests’ own.
npm test works too when your test script runs jest: the reporter comes from the config, not from the command.
Pin the version in package.json (or the lockfile), so a new release never changes a pipeline by surprise. The CLI is not a dependency of the reporter: the steps below that run it pin it in the command (npx --yes @probara/cli@0.2). With @probara/cli in your devDependencies, npx @probara/cli after npm ci runs the locked version instead.
Builds of forks report nothing: without the secret, the reporter sends nothing (with the project set, it logs that the token is missing) and the tests run as usual. The CLI exits 2 without a token, so the recipes turn reporting off (PROBARA_ENABLED=false) where a fork’s build would run it: it then exits 0. Each section below says how to keep the secret out of fork builds on that CI.
Sharded jobs make one run per shard unless they share one. In GitHub Actions the jobs of one workflow run share one run automatically (parallel jobs). Elsewhere, create the run first, pass its ULID to every shard, and close it at the end (sharding).

What CI detection fills in

Unless an option or a PROBARA_* variable sets them, the run gets its name and source from the CI:

CI	Detected by	Run name
GitHub Actions	`GITHUB_ACTIONS=true`	`GITHUB_WORKFLOW #GITHUB_RUN_NUMBER`
GitLab CI	`GITLAB_CI=true`	`CI_PROJECT_NAME #CI_PIPELINE_IID`
Jenkins	`JENKINS_URL`	`JOB_NAME #BUILD_NUMBER`
CircleCI	`CIRCLECI=true`	`CIRCLE_PROJECT_REPONAME #CIRCLE_BUILD_NUM`
Azure Pipelines	`TF_BUILD=True`	`BUILD_DEFINITIONNAME #BUILD_BUILDNUMBER`
Bitbucket Pipelines	`BITBUCKET_BUILD_NUMBER`	`BITBUCKET_REPO_SLUG #BITBUCKET_BUILD_NUMBER`
Buildkite	`BUILDKITE=true`	`BUILDKITE_PIPELINE_SLUG #BUILDKITE_BUILD_NUMBER`

The branch, commit and build URL of each are listed in its section, and in @probara/core’s CI detection. Override them with PROBARA_BRANCH, PROBARA_COMMIT and PROBARA_BUILD_URL.

GitHub Actions

1. Add the secret

In the repository (or organization): Settings → Secrets and variables → Actions → New repository secret, named PROBARA_API_TOKEN, holding the app token.

2. Give the test step the token

name: Jest

on:
  push:
    branches: [main]
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-node@v5
        with:
          node-version: 22
      - run: npm ci
      - name: Run Jest tests
        run: npx jest --ci
        env:
          PROBARA_API_TOKEN: ${{ secrets.PROBARA_API_TOKEN }}
          PROBARA_PROJECT: SHOP

Parallel jobs: one run, closed by the final job

In GitHub Actions the jobs of one workflow run report into one run: the run’s source names the workflow run by its repository, run id and attempt, and the first job to report creates the run. A job closes its run at its end by default, so the parallel jobs set PROBARA_CLOSE_RUN: false. A final job closes the run with probara run close, which needs no run ULID there: needs makes it wait for every test job, and if: always() runs it when they fail.

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        shard: [1, 2]
    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-node@v5
        with:
          node-version: 22
      - run: npm ci
      - run: npx jest --ci --shard=${{ matrix.shard }}/2
        env:
          PROBARA_API_TOKEN: ${{ secrets.PROBARA_API_TOKEN }}
          PROBARA_PROJECT: SHOP
          # A shard leaves the run open for the others.
          PROBARA_CLOSE_RUN: false

  probara:
    needs: test
    if: always()
    runs-on: ubuntu-latest
    steps:
      - uses: actions/setup-node@v5
        with:
          node-version: 22
      - run: npx --yes @probara/cli@0.6 run close
        env:
          PROBARA_API_TOKEN: ${{ secrets.PROBARA_API_TOKEN }}
          PROBARA_PROJECT: SHOP

A job after the close is refused with 409 ci_run_closed: the reporter logs why, sends nothing more, and the tests keep their own exit code (troubleshooting). Keep every reporting job in needs.
A re-run of the workflow is another attempt: it reports into a run of its own.
One job only needs neither: its run is closed at the end, as anywhere else.

CI detection

Field	From
Run name	`GITHUB_WORKFLOW #GITHUB_RUN_NUMBER`, such as `Jest #42`
Branch	`GITHUB_HEAD_REF` (the pull request’s branch), else `GITHUB_REF_NAME`; none for a tag
Commit	`GITHUB_SHA`
Build URL	`GITHUB_SERVER_URL/GITHUB_REPOSITORY/actions/runs/GITHUB_RUN_ID`
Repository	`GITHUB_REPOSITORY`
Commit author	`GITHUB_ACTOR`
Pull request	`<n>` of `GITHUB_REF` `refs/pull/<n>/merge`
Workflow	`GITHUB_WORKFLOW`
CI run	`GITHUB_RUN_ID`, attempt `GITHUB_RUN_ATTEMPT`
CI job	`GITHUB_JOB`

On pull_request events GITHUB_SHA is the merge commit GitHub made, not the branch’s last commit. To record the latter, add PROBARA_COMMIT: ${{ github.event.pull_request.head.sha }}. In GitHub Actions the commit must be a 7 to 40 character hex SHA: another value is dropped with a warning.

Fork pull requests

Workflows triggered by a pull request from a fork get empty secrets. Without a token the reporter stays off, so fork builds run their tests and report nothing, with no extra setting. Do not use pull_request_target to get the secret into a fork build: it runs the fork’s code with your secrets.

GitLab CI

1. Add the variable

Settings → CI/CD → Variables → Add variable: key PROBARA_API_TOKEN, the token as the value, with Mask variable on. Protect variable limits it to protected branches and tags: turn it on only if you report from those alone.

2. The test job

stages:
  - test

jest:
  stage: test
  image: node:22
  variables:
    PROBARA_PROJECT: SHOP
  script:
    - npm ci
    - npx jest --ci

A job without the variable (a fork’s pipeline, an unprotected branch with a protected variable) runs its tests and reports nothing.
To keep what could not be sent, set PROBARA_RESULTS_FILE and keep the file as an artifact (results file).
parallel: shards sharing one run, with a .pre job that creates it and a .post job that closes it, are in sharding.

CI detection

Field	From
Run name	`CI_PROJECT_NAME #CI_PIPELINE_IID`, such as `shop #42`
Branch	`CI_MERGE_REQUEST_SOURCE_BRANCH_NAME` in a merge request pipeline, else `CI_COMMIT_REF_NAME`; none for a tag pipeline
Commit	`CI_COMMIT_SHA`
Build URL	`CI_PIPELINE_URL`

Fork merge requests

A merge request from a fork runs its pipeline in the fork’s project, which does not have your CI/CD variables: the reporter stays off there. If maintainers run fork pipelines in the parent project, those pipelines do get the variable, so review the change before you run it.

Jenkins

1. Add the credential

Manage Jenkins → Credentials → (a domain) → Add Credentials: kind Secret text, ID probara-api-token, the token as the secret.

2. The pipeline

A declarative Jenkinsfile with Node.js 22 on the agent binds the token into the test stage with withCredentials:

pipeline {
  agent any
  environment {
    PROBARA_PROJECT = 'SHOP'
  }
  stages {
    stage('Test') {
      steps {
        sh 'npm ci'
        withCredentials([string(credentialsId: 'probara-api-token', variable: 'PROBARA_API_TOKEN')]) {
          sh 'npx jest --ci'
        }
      }
    }
  }
}

Jenkins masks the bound secret in the build log.

Parallel stages: one run

Parallel stages that run --shards make one run each unless they share one: create it in a first stage, pass it to each, and close it in a post { always { ... } } block:

pipeline {
  agent any
  environment {
    PROBARA_PROJECT = 'SHOP'
  }
  stages {
    stage('Create the Probara run') {
      steps {
        sh 'npm ci'
        withCredentials([string(credentialsId: 'probara-api-token', variable: 'PROBARA_API_TOKEN')]) {
          sh 'npx --yes @probara/cli@0.2 run create > probara-run-ulid'
        }
        script {
          env.PROBARA_RUN_ULID = readFile('probara-run-ulid').trim()
        }
      }
    }
    stage('Test') {
      parallel {
        stage('Shard 1') {
          steps {
            withCredentials([string(credentialsId: 'probara-api-token', variable: 'PROBARA_API_TOKEN')]) {
              sh 'npx jest --ci --shard=1/2'
            }
          }
        }
        stage('Shard 2') {
          steps {
            withCredentials([string(credentialsId: 'probara-api-token', variable: 'PROBARA_API_TOKEN')]) {
              sh 'npx jest --ci --shard=2/2'
            }
          }
        }
      }
    }
  }
  post {
    always {
      withCredentials([string(credentialsId: 'probara-api-token', variable: 'PROBARA_API_TOKEN')]) {
        sh 'npx --yes @probara/cli@0.2 run close'
      }
    }
  }
}

Both shards share one workspace here, installed once before they start (two npm ci at the same time in one folder would break each other); on separate agents, each checks out and installs on its own.

Fork pull requests

Jenkins does not hide credentials from builds of pull requests by itself: a multibranch job that builds forks can hand the token to their code. Set Discover pull requests from forks → Trust so that a pull request from anyone else builds with the Jenkinsfile of its target branch: in the GitHub branch source, From users with Admin or Write permission (or Nobody); in the Bitbucket branch source, Nobody, or Forks in the same account from plugin version 871 on (the fix for CVE-2024-28152). Everyone builds a fork with its own Jenkinsfile, which can bind the credential itself, and makes the env.CHANGE_FORK guard below useless.

In the target branch’s Jenkinsfile, bind the credential only when env.CHANGE_FORK is not set (Jenkins sets it for a pull request from a fork), and turn reporting off for a fork build:

pipeline {
  agent any
  environment {
    PROBARA_PROJECT = 'SHOP'
  }
  stages {
    stage('Test') {
      steps {
        sh 'npm ci'
        script {
          if (env.CHANGE_FORK) {
            withEnv(['PROBARA_ENABLED=false']) {
              sh 'npx jest --ci'
            }
          } else {
            withCredentials([string(credentialsId: 'probara-api-token', variable: 'PROBARA_API_TOKEN')]) {
              sh 'npx jest --ci'
            }
          }
        }
      }
    }
  }
}

In the parallel pipeline, wrap each withCredentials the same way: with reporting off, run create writes no ULID and run close exits 0 without a token.

CI detection

With JENKINS_URL set, the run is named JOB_NAME #BUILD_NUMBER, and its branch, commit and build URL come from BRANCH_NAME (or GIT_BRANCH), GIT_COMMIT and BUILD_URL.

CircleCI

1. Add the token

Organization Settings → Contexts → Create Context (such as probara-reporting), then add the environment variable PROBARA_API_TOKEN. A project environment variable (Project Settings → Environment Variables) works too; then drop context: below.

2. The job

A .circleci/config.yml in a Node.js image, sharded with parallelism into one run created up front. Without parallelism, keep only the test job and run npx jest --ci:

version: 2.1

jobs:
  create-run:
    docker:
      - image: cimg/node:22.12
    environment:
      PROBARA_PROJECT: SHOP
    steps:
      - run: |
          # Without the token (a fork's build), turn reporting off: run create prints nothing.
          [ -n "$PROBARA_API_TOKEN" ] || export PROBARA_ENABLED=false
          PROBARA_RUN_ULID=$(npx --yes @probara/cli@0.2 run create)
          echo "PROBARA_RUN_ULID=$PROBARA_RUN_ULID" > probara.env
      - persist_to_workspace:
          root: .
          paths: [probara.env]
  test:
    docker:
      - image: cimg/node:22.12
    parallelism: 4
    environment:
      PROBARA_PROJECT: SHOP
    steps:
      - checkout
      - attach_workspace:
          at: .
      - run: npm ci
      - run: |
          export $(cat probara.env)
          npx jest --ci --shard=$((CIRCLE_NODE_INDEX + 1))/$CIRCLE_NODE_TOTAL
  close-run:
    docker:
      - image: cimg/node:22.12
    environment:
      PROBARA_PROJECT: SHOP
    steps:
      - attach_workspace:
          at: .
      - run:
          when: always
          command: |
            export $(cat probara.env)
            [ -n "$PROBARA_API_TOKEN" ] || export PROBARA_ENABLED=false
            npx --yes @probara/cli@0.2 run close

workflows:
  jest:
    jobs:
      - create-run:
          context: probara-reporting
      - test:
          context: probara-reporting
          requires: [create-run]
      - close-run:
          context: probara-reporting
          requires:
            - test: [success, failed]

CIRCLE_NODE_INDEX counts from 0, Jest’s shards from 1.

Forks and CI detection

CircleCI does not pass contexts or project variables to builds of forked pull requests unless Pass secrets to builds from forked pull requests is on: leave it off. A fork’s jobs that use a context then do not run at all (CircleCI shows an authorization error). With project variables they run without the token: the reporter sends nothing, and the PROBARA_ENABLED=false line of create-run and close-run makes run create print nothing and both exit 0.

With CIRCLECI=true, the run is named CIRCLE_PROJECT_REPONAME #CIRCLE_BUILD_NUM, with the branch CIRCLE_BRANCH, the commit CIRCLE_SHA1 and the build URL CIRCLE_BUILD_URL.

Azure Pipelines

1. Add the variable

Pipelines → your pipeline → Edit → Variables → New variable: name PROBARA_API_TOKEN, the token as the value, Keep this value secret on. A variable group of the library works too. Secret variables never reach scripts on their own: map the variable into the step’s env, as below.

2. The job

variables:
  # Off in a fork's build, which gets no secrets (see forks below).
  PROBARA_ENABLED: $[ ne(variables['System.PullRequest.IsFork'], 'True') ]

pool:
  vmImage: ubuntu-latest

steps:
  - task: NodeTool@0
    inputs:
      versionSpec: '22.x'
  - script: npm ci
  - script: npx jest --ci
    env:
      PROBARA_API_TOKEN: $(PROBARA_API_TOKEN)
      PROBARA_PROJECT: SHOP

Sharded jobs: one run

Sharded with parallel into one run created by a first job, and closed by a last one that runs even after failed shards:

variables:
  PROBARA_ENABLED: $[ ne(variables['System.PullRequest.IsFork'], 'True') ]

jobs:
  - job: create_run
    pool:
      vmImage: ubuntu-latest
    steps:
      - task: NodeTool@0
        inputs:
          versionSpec: '22.x'
      - script: |
          set -e
          ulid=$(npx --yes @probara/cli@0.2 run create)
          echo "##vso[task.setvariable variable=ulid;isOutput=true]$ulid"
        name: probara
        env:
          PROBARA_API_TOKEN: $(PROBARA_API_TOKEN)
          PROBARA_PROJECT: SHOP

  - job: test
    dependsOn: create_run
    strategy:
      parallel: 4
    variables:
      PROBARA_RUN_ULID: $[ dependencies.create_run.outputs['probara.ulid'] ]
    pool:
      vmImage: ubuntu-latest
    steps:
      - task: NodeTool@0
        inputs:
          versionSpec: '22.x'
      - script: npm ci
      - script: npx jest --ci --shard=$(System.JobPositionInPhase)/$(System.TotalJobsInPhase)
        env:
          PROBARA_API_TOKEN: $(PROBARA_API_TOKEN)
          PROBARA_PROJECT: SHOP
          PROBARA_RUN_ULID: $(PROBARA_RUN_ULID)

  - job: close_run
    dependsOn: [create_run, test]
    condition: always()
    variables:
      PROBARA_RUN_ULID: $[ dependencies.create_run.outputs['probara.ulid'] ]
    pool:
      vmImage: ubuntu-latest
    steps:
      - task: NodeTool@0
        inputs:
          versionSpec: '22.x'
      - script: npx --yes @probara/cli@0.2 run close
        env:
          PROBARA_API_TOKEN: $(PROBARA_API_TOKEN)
          PROBARA_PROJECT: SHOP
          PROBARA_RUN_ULID: $(PROBARA_RUN_ULID)

System.JobPositionInPhase counts from 1, like Jest’s shards.

Forks and CI detection

Builds of pull requests from forks get no secret variables unless Make secrets available to builds of forks is on: leave it off. Azure leaves a macro it cannot resolve as it is, so in a fork’s build $(PROBARA_API_TOKEN) reaches the step as that literal text, a token Probara refuses with 401: the reporter would log the 401 on every run (Jest’s exit code stays the tests’ own), and run create would fail the first job. The pipeline’s PROBARA_ENABLED variable is False when System.PullRequest.IsFork is True, so a fork’s build runs its tests with reporting off, and run create and run close exit 0.

With TF_BUILD=True, the run is named BUILD_DEFINITIONNAME #BUILD_BUILDNUMBER, and its branch, commit and build URL come from the pipeline’s variables.

Bitbucket Pipelines

1. Add the variable

Repository settings → Pipelines → Repository variables: name PROBARA_API_TOKEN, the token as the value, Secured on. Bitbucket passes repository variables to every step, so the reporter finds it in npx jest.

2. The pipeline

image: node:22

pipelines:
  default:
    - step:
        name: Jest
        script:
          - npm ci
          - export PROBARA_PROJECT=SHOP
          - npx jest --ci

Parallel steps: one run

Parallel steps that each run a --shard report into one run created by a first step and closed by a last one:

image: node:22

pipelines:
  default:
    - step:
        name: Create the Probara run
        script:
          - export PROBARA_PROJECT=SHOP
          # Without the token (a fork's build), turn reporting off: run create prints nothing.
          - test -n "$PROBARA_API_TOKEN" || export PROBARA_ENABLED=false
          - PROBARA_RUN_ULID=$(npx --yes @probara/cli@0.2 run create)
          - echo "PROBARA_RUN_ULID=$PROBARA_RUN_ULID" > probara.env
        artifacts:
          - probara.env
    - parallel:
        - step:
            name: Shard 1
            script:
              - npm ci
              - export PROBARA_PROJECT=SHOP $(cat probara.env)
              - npx jest --ci --shard=1/2 || touch shard-1.failed
            artifacts:
              - shard-*.failed
        - step:
            name: Shard 2
            script:
              - npm ci
              - export PROBARA_PROJECT=SHOP $(cat probara.env)
              - npx jest --ci --shard=2/2 || touch shard-2.failed
            artifacts:
              - shard-*.failed
    - step:
        name: Close the Probara run
        script:
          - export PROBARA_PROJECT=SHOP $(cat probara.env)
          - test -n "$PROBARA_API_TOKEN" || export PROBARA_ENABLED=false
          - npx --yes @probara/cli@0.2 run close
          - if ls shard-*.failed >/dev/null 2>&1; then echo "Tests failed in a shard" >&2; exit 1; fi

Bitbucket runs no step after a failed one, so a shard that failed its step would leave the run open. Each shard records failed tests in a shard-<n>.failed artifact instead, and the last step closes the run, then fails the pipeline when a shard left one: the shards show as passed, and the last step carries the failure. If the close still never ran, close the run in Probara, or by hand.

Forks and CI detection

Pull requests from forks get no secured variables: the reporter sends nothing there. With BITBUCKET_BUILD_NUMBER set, the run is named BITBUCKET_REPO_SLUG #BITBUCKET_BUILD_NUMBER, with the branch BITBUCKET_BRANCH, the commit BITBUCKET_COMMIT and a link to the pipeline.

Buildkite

1. Add the token

Expose PROBARA_API_TOKEN to the steps from a secret store: an agent environment hook that reads it from your secrets manager, or Buildkite secrets.

2. The pipeline

A pipeline.yml whose parallel jobs report into one run created by a first step and closed after them, even when they failed. Without parallelism, one step running npx jest --ci is enough:

env:
  PROBARA_PROJECT: SHOP

steps:
  - label: Create the Probara run
    key: create-run
    command: |
      PROBARA_RUN_ULID="$(npx --yes @probara/cli@0.2 run create)"
      if [ -n "$$PROBARA_RUN_ULID" ]; then buildkite-agent meta-data set probara-run "$$PROBARA_RUN_ULID"; fi

  - label: Jest
    depends_on: create-run
    parallelism: 4
    command: |
      npm ci
      export PROBARA_RUN_ULID="$(buildkite-agent meta-data get probara-run --default '')"
      npx jest --ci --shard=$((BUILDKITE_PARALLEL_JOB + 1))/$$BUILDKITE_PARALLEL_JOB_COUNT

  - wait: ~
    continue_on_failure: true

  - label: Close the Probara run
    command: PROBARA_RUN_ULID="$(buildkite-agent meta-data get probara-run --default '')" npx --yes @probara/cli@0.2 run close

buildkite-agent pipeline upload fills in $NAME and ${NAME} when it uploads the pipeline, and a variable it does not know then, such as BUILDKITE_PARALLEL_JOB_COUNT that only the parallel jobs get, becomes blank; a $$ is left as one $ for the job’s shell, while $(...) and $((...)) are left as they are.
The run is created on a line of its own, so a failed run create fails the first step rather than storing a blank ULID. With reporting off (a fork build on an agent that sets PROBARA_ENABLED=false, as below), run create prints no ULID and exits 0: nothing is stored, meta-data get --default '' gives the other steps an empty ULID instead of an error, and they run with reporting off too.
BUILDKITE_PARALLEL_JOB counts from 0, Jest’s shards from 1.

With BUILDKITE=true, the run is named BUILDKITE_PIPELINE_SLUG #BUILDKITE_BUILD_NUMBER, with the branch BUILDKITE_BRANCH, the commit BUILDKITE_COMMIT and the build URL BUILDKITE_BUILD_URL.

Fork pull requests

Buildkite builds pull requests from forks only if the pipeline allows it (off by default). Those builds run on your agents, and pipeline upload reads the fork’s own pipeline.yml, which also picks the agents’ queue. The fork’s code runs in the job’s shell as the agent’s user, so on an agent that can read the token from your secret store, the fork’s code can read it too: no hook or step keeps it out.

Keep fork builds off, or keep them off those agents: their environment hook refuses a build of a fork (BUILDKITE_PULL_REQUEST_REPO names it), and forks build only on a separate queue or cluster whose agents cannot read the secret store:

# The environment hook of the agents that can read the token: no build of a fork runs there.
if [ -n "$BUILDKITE_PULL_REQUEST_REPO" ]; then
  case "${BUILDKITE_PULL_REQUEST_REPO#*://}" in
    github.com/acme/shop.git) ;;
    *)
      echo "Refusing a build of a fork: $BUILDKITE_PULL_REQUEST_REPO" >&2
      exit 1
      ;;
  esac
fi
export PROBARA_API_TOKEN="$(your-secrets-manager get probara-api-token)"

BUILDKITE_PULL_REQUEST_REPO is empty outside pull requests, and has been an https:// URL since 2022 (a git:// one before), so the hook compares it without the scheme: put your repository in place of github.com/acme/shop.git. When it does not match, your own pull requests fail on these agents too, which the first one shows. On the agents that build forks, set PROBARA_ENABLED=false in their environment hook: the fork’s tests run and npx jest reports nothing, and run create and run close exit 0 without a token.

Related: Sharding and projects · Results file in CI · Troubleshooting · Source: docs/ci