Vitest reporter in CI

The reporter runs inside npx vitest run: give the step that runs your tests the token, and every result is sent when Vitest ends. There is no separate report step, and the token is never printed: a problem names the variable, never its value. Every example assumes the reporter is registered in vitest.config.mts as in Get started.

In every CI

The token is a secret, the Project ID is not. Store the app token (from the Vitest card in Workspace › Integrations) as a secret named PROBARA_API_TOKEN. Put the Project ID in vitest.config.mts (projectId) or in the pipeline (PROBARA_PROJECT).
Node.js 22.12 or later on the agent, like the Vitest run the reporter is part of.
Failed tests fail the step, reporting problems do not: Vitest’s exit code stays the tests’ own (what the reporter never does).
npm test works too when your test script runs vitest run: the reporter comes from the config, not from the command. A --reporter on the command line replaces the config’s reporters, this one included.
Pin the version in package.json (or the lockfile), so a new release never changes a pipeline by surprise (upgrading).
Builds of forks report nothing: without the secret, the reporter sends nothing and the tests run as usual. Each section below says how to keep the secret out of fork builds on that CI, and how its recipes turn reporting off (PROBARA_ENABLED=false) there.
Sharded jobs make one run per shard unless they are tied together. In GitHub Actions the jobs of one workflow run share one run automatically (parallel jobs). Elsewhere, merge the shards’ blob reports in one job, or create the run first, pass its ULID to every shard, and close it at the end (sharding).

What CI detection fills in

Unless an option or a PROBARA_* variable sets them, the run gets its name and source from the CI:

CI	Detected by	Run name
GitHub Actions	`GITHUB_ACTIONS=true`	`GITHUB_WORKFLOW #GITHUB_RUN_NUMBER`
GitLab CI	`GITLAB_CI=true`	`CI_PROJECT_NAME #CI_PIPELINE_IID`
Jenkins	`JENKINS_URL`	`JOB_NAME #BUILD_NUMBER`
CircleCI	`CIRCLECI=true`	`CIRCLE_PROJECT_REPONAME #CIRCLE_BUILD_NUM`
Azure Pipelines	`TF_BUILD=True`	`BUILD_DEFINITIONNAME #BUILD_BUILDNUMBER`
Bitbucket Pipelines	`BITBUCKET_BUILD_NUMBER`	`BITBUCKET_REPO_SLUG #BITBUCKET_BUILD_NUMBER`
Buildkite	`BUILDKITE=true`	`BUILDKITE_PIPELINE_SLUG #BUILDKITE_BUILD_NUMBER`

The branch, commit and build URL of each are listed in its section, and in @probara/core’s CI detection. Override them with PROBARA_BRANCH, PROBARA_COMMIT and PROBARA_BUILD_URL.

GitHub Actions

1. Add the secret

In the repository (or organization): Settings → Secrets and variables → Actions → New repository secret, named PROBARA_API_TOKEN, holding the app token.

2. Give the test step the token

name: Vitest

on:
  push:
    branches: [main]
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-node@v5
        with:
          node-version: 22
      - run: npm ci
      - name: Run Vitest tests
        run: npx vitest run
        env:
          PROBARA_API_TOKEN: ${{ secrets.PROBARA_API_TOKEN }}
          PROBARA_PROJECT: SHOP
          # Fork pull requests get no secrets: report nothing there.
          PROBARA_ENABLED: ${{ secrets.PROBARA_API_TOKEN != '' }}

Sharded jobs: one run from the blobs

The shards write Vitest blob reports with the reporter beside blob, and one job merges them: the whole pipeline is one run, every attempt and file included (pattern 1). Only the merging job needs the token:

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        shardIndex: [1, 2, 3, 4]
        shardTotal: [4]
    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-node@v5
        with:
          node-version: 22
      - run: npm ci
      - run: npx vitest run --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }} --reporter=blob --reporter=@probara/vitest-reporter
      - uses: actions/upload-artifact@v4
        if: ${{ !cancelled() }}
        with:
          name: vitest-blob-${{ matrix.shardIndex }}
          path: .vitest
          include-hidden-files: true
          retention-days: 1

  report:
    needs: test
    if: ${{ !cancelled() }}
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-node@v5
        with:
          node-version: 22
      - run: npm ci
      - uses: actions/download-artifact@v4
        with:
          pattern: vitest-blob-*
          path: .vitest
          merge-multiple: true
      - run: npx vitest --merge-reports
        env:
          PROBARA_API_TOKEN: ${{ secrets.PROBARA_API_TOKEN }}
          PROBARA_PROJECT: SHOP
          PROBARA_ENABLED: ${{ secrets.PROBARA_API_TOKEN != '' }}

.vitest holds Vitest 5’s blobs and attachmentsDir; it is hidden, hence include-hidden-files: true. On Vitest 3 and 4, upload .vitest-reports and .vitest-attachments (one per line of path:), and download them to ..
GitHub-hosted runners check out every job at the same path, so the merge keeps every key (merge at the same path).
The report job runs after failed shards too, and its exit code is that of the merged tests.

Parallel jobs: one run, closed by the final job

In GitHub Actions the jobs of one workflow run report into one run: the run’s source names the workflow run by its repository, run id and attempt, and the first job to report creates the run. A job closes its run at its end by default, so the parallel jobs set PROBARA_CLOSE_RUN: false. A final job closes the run with probara run close, which needs no run ULID there: needs makes it wait for every test job, and if: always() runs it when they fail. Each shard reports as it ends.

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        shard: [1, 2]
    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-node@v5
        with:
          node-version: 22
      - run: npm ci
      - run: npx vitest run --shard=${{ matrix.shard }}/2
        env:
          PROBARA_API_TOKEN: ${{ secrets.PROBARA_API_TOKEN }}
          PROBARA_PROJECT: SHOP
          # A shard leaves the run open for the others.
          PROBARA_CLOSE_RUN: false

  probara:
    needs: test
    if: always()
    runs-on: ubuntu-latest
    steps:
      - uses: actions/setup-node@v5
        with:
          node-version: 22
      - run: npx --yes @probara/cli@0.6 run close
        env:
          PROBARA_API_TOKEN: ${{ secrets.PROBARA_API_TOKEN }}
          PROBARA_PROJECT: SHOP

A job after the close is refused with 409 ci_run_closed: the reporter logs why, sends nothing more, and the tests keep their own exit code (troubleshooting). Keep every reporting job in needs.
A re-run of the workflow is another attempt: it reports into a run of its own.
One job only needs neither: its run is closed at the end, as anywhere else.

CI detection

Field	From
Run name	`GITHUB_WORKFLOW #GITHUB_RUN_NUMBER`, such as `Vitest #42`
Branch	`GITHUB_HEAD_REF` (the pull request’s branch), else `GITHUB_REF_NAME`; none for a tag
Commit	`GITHUB_SHA`
Build URL	`GITHUB_SERVER_URL/GITHUB_REPOSITORY/actions/runs/GITHUB_RUN_ID`
Repository	`GITHUB_REPOSITORY`
Commit author	`GITHUB_ACTOR`
Pull request	`<n>` of `GITHUB_REF` `refs/pull/<n>/merge`
Workflow	`GITHUB_WORKFLOW`
CI run	`GITHUB_RUN_ID`, attempt `GITHUB_RUN_ATTEMPT`
CI job	`GITHUB_JOB`

On pull_request events GITHUB_SHA is the merge commit GitHub made, not the branch’s last commit. To record the latter, add PROBARA_COMMIT: ${{ github.event.pull_request.head.sha }}. In GitHub Actions the commit must be a 7 to 40 character hex SHA: another value is dropped with a warning.

Fork pull requests

Workflows triggered by a pull request from a fork get empty secrets. Without the token, the reporter says so, sends nothing, and the tests run as usual:

$ PROBARA_PROJECT=SHOP npx vitest run
[probara] Probara reporting is off: The API token is not set: pass apiToken or set PROBARA_API_TOKEN

The workflows above set PROBARA_ENABLED from whether the secret is there, so fork builds report nothing and log nothing. Do not use pull_request_target to get the secret into a fork build: it runs the fork’s code with your secrets.

GitLab CI

1. Add the variable

Settings → CI/CD → Variables → Add variable: key PROBARA_API_TOKEN, the token as the value, with Mask variable on. Protect variable limits it to protected branches and tags: turn it on only if you report from those alone.

2. The test job

stages:
  - test

vitest:
  stage: test
  image: node:22
  variables:
    PROBARA_PROJECT: SHOP
  script:
    - npm ci
    - npx vitest run

A job without the variable (a fork’s pipeline, an unprotected branch with a protected variable) runs its tests and sends nothing; the reporter logs that the token is not set.
To keep what could not be sent, set PROBARA_RESULTS_FILE and keep the file as an artifact (results file).

`parallel:` shards: one run from the blobs

parallel: shards write Vitest blob reports with the reporter beside blob, and one job merges them into one run (pattern 1). Only the merging job needs the token:

stages:
  - test
  - report

vitest:
  stage: test
  image: node:22
  parallel: 4
  script:
    - npm ci
    - npx vitest run --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL --reporter=blob --reporter=@probara/vitest-reporter
  artifacts:
    when: always
    paths:
      - .vitest/

report-to-probara:
  stage: report
  image: node:22
  needs:
    - job: vitest
      artifacts: true
  variables:
    PROBARA_PROJECT: SHOP
  rules:
    # After failed shards too, and only where the token is set.
    - if: $PROBARA_API_TOKEN
      when: always
  script:
    - npm ci
    - npx vitest --merge-reports

A needs: on a parallel: job downloads the artifacts of every instance into the same folder; each shard’s blob has its own file name (blob-1-4.json…), so they sit side by side.
.vitest/ holds Vitest 5’s blobs and attachmentsDir. On Vitest 3 and 4, keep .vitest-reports/ and .vitest-attachments/ instead.
The merging job must check out at the shards’ CI_PROJECT_DIR, or every key changes (merge at the same path).

To have each shard report as it ends instead, create the run in a .pre job: see pattern 2.

CI detection

Field	From
Run name	`CI_PROJECT_NAME #CI_PIPELINE_IID`, such as `shop #42`
Branch	`CI_MERGE_REQUEST_SOURCE_BRANCH_NAME` in a merge request pipeline, else `CI_COMMIT_REF_NAME`; none for a tag pipeline
Commit	`CI_COMMIT_SHA`
Build URL	`CI_PIPELINE_URL`

Fork merge requests

A merge request from a fork runs its pipeline in the fork’s project, which does not have your CI/CD variables: the reporter sends nothing there, and the rule if: $PROBARA_API_TOKEN leaves the merging job out. If maintainers run fork pipelines in the parent project, those pipelines do get the variable, so review the change before you run it.

Jenkins

1. Add the credential

Manage Jenkins → Credentials → (a domain) → Add Credentials: kind Secret text, ID probara-api-token, the token as the secret.

2. The pipeline

A declarative Jenkinsfile with Node.js 22 on the agent binds the token into the test stage with withCredentials:

pipeline {
  agent any
  environment {
    PROBARA_PROJECT = 'SHOP'
  }
  stages {
    stage('Test') {
      steps {
        sh 'npm ci'
        withCredentials([string(credentialsId: 'probara-api-token', variable: 'PROBARA_API_TOKEN')]) {
          sh 'npx vitest run'
        }
      }
    }
  }
}

Jenkins masks the bound secret in the build log, and the reporter never prints it.

Parallel stages: one run

Parallel stages that run --shards make one run each unless they share one: create it in a first stage, pass it to each, and close it in a post { always { ... } } block (pattern 2):

pipeline {
  agent any
  environment {
    PROBARA_PROJECT = 'SHOP'
  }
  stages {
    stage('Create the Probara run') {
      steps {
        sh 'npm ci'
        withCredentials([string(credentialsId: 'probara-api-token', variable: 'PROBARA_API_TOKEN')]) {
          sh 'npx @probara/cli run create > probara-run-ulid'
        }
        script {
          env.PROBARA_RUN_ULID = readFile('probara-run-ulid').trim()
        }
      }
    }
    stage('Test') {
      parallel {
        stage('Shard 1') {
          steps {
            withCredentials([string(credentialsId: 'probara-api-token', variable: 'PROBARA_API_TOKEN')]) {
              sh 'npx vitest run --shard=1/2'
            }
          }
        }
        stage('Shard 2') {
          steps {
            withCredentials([string(credentialsId: 'probara-api-token', variable: 'PROBARA_API_TOKEN')]) {
              sh 'npx vitest run --shard=2/2'
            }
          }
        }
      }
    }
  }
  post {
    always {
      withCredentials([string(credentialsId: 'probara-api-token', variable: 'PROBARA_API_TOKEN')]) {
        sh 'npx @probara/cli run close'
      }
    }
  }
}

Both shards share one workspace here, installed once before they start (two npm ci at the same time in one folder would break each other); on separate agents, each checks out and installs on its own. Each shard sends its results when it ends, into the run of the first stage.

Fork pull requests

Jenkins does not hide credentials from builds of pull requests by itself: a multibranch job that builds forks can hand the token to their code. Set Discover pull requests from forks → Trust so that a pull request from anyone else builds with the Jenkinsfile of its target branch: in the GitHub branch source, From users with Admin or Write permission (or Nobody); in the Bitbucket branch source, Nobody, or Forks in the same account from plugin version 871 on (the fix for CVE-2024-28152). Everyone builds a fork with its own Jenkinsfile, which can bind the credential itself, and makes the env.CHANGE_FORK guard below useless.

In the target branch’s Jenkinsfile, bind the credential only when env.CHANGE_FORK is not set (Jenkins sets it for a pull request from a fork), and turn reporting off for a fork build:

pipeline {
  agent any
  environment {
    PROBARA_PROJECT = 'SHOP'
  }
  stages {
    stage('Test') {
      steps {
        sh 'npm ci'
        script {
          if (env.CHANGE_FORK) {
            withEnv(['PROBARA_ENABLED=false']) {
              sh 'npx vitest run'
            }
          } else {
            withCredentials([string(credentialsId: 'probara-api-token', variable: 'PROBARA_API_TOKEN')]) {
              sh 'npx vitest run'
            }
          }
        }
      }
    }
  }
}

In the parallel pipeline, wrap each withCredentials the same way: with reporting off, run create writes no ULID and run close exits 0 without a token.

CI detection

With JENKINS_URL set, the run is named JOB_NAME #BUILD_NUMBER, and its branch, commit and build URL come from BRANCH_NAME (or GIT_BRANCH), GIT_COMMIT and BUILD_URL.

CircleCI

1. Add the token

Organization Settings → Contexts → Create Context (such as probara-reporting), then add the environment variable PROBARA_API_TOKEN. A project environment variable (Project Settings → Environment Variables) works too; then drop context: below.

2. The job

A .circleci/config.yml in a Node.js image, sharded with parallelism into one run created up front (pattern 2). Without parallelism, keep only the test job and run npx vitest run:

version: 2.1

jobs:
  create-run:
    docker:
      - image: cimg/node:22.12
    environment:
      PROBARA_PROJECT: SHOP
    steps:
      - run: |
          # Forked pull requests get no secrets: report nothing there.
          if [ -z "$PROBARA_API_TOKEN" ]; then export PROBARA_ENABLED=false; fi
          PROBARA_RUN_ULID=$(npx @probara/cli run create)
          echo "PROBARA_RUN_ULID=$PROBARA_RUN_ULID" > probara.env
      - persist_to_workspace:
          root: .
          paths: [probara.env]
  test:
    docker:
      - image: cimg/node:22.12
    parallelism: 4
    environment:
      PROBARA_PROJECT: SHOP
    steps:
      - checkout
      - attach_workspace:
          at: .
      - run: npm ci
      - run: |
          if [ -z "$PROBARA_API_TOKEN" ]; then export PROBARA_ENABLED=false; fi
          export $(cat probara.env)
          npx vitest run --shard=$((CIRCLE_NODE_INDEX + 1))/$CIRCLE_NODE_TOTAL
  close-run:
    docker:
      - image: cimg/node:22.12
    environment:
      PROBARA_PROJECT: SHOP
    steps:
      - attach_workspace:
          at: .
      - run:
          when: always
          command: |
            if [ -z "$PROBARA_API_TOKEN" ]; then export PROBARA_ENABLED=false; fi
            export $(cat probara.env)
            npx @probara/cli run close

workflows:
  vitest:
    jobs:
      - create-run:
          context: probara-reporting
      - test:
          context: probara-reporting
          requires: [create-run]
      - close-run:
          context: probara-reporting
          requires:
            - test: [success, failed]

CIRCLE_NODE_INDEX counts from 0, Vitest’s shards from 1. Each shard sends its results when it ends, into the run of run create; close-run closes it after failed shards too.

Forks and CI detection

CircleCI does not pass contexts or project variables to builds of forked pull requests (unless Pass secrets to builds from forked pull requests is on: leave it off). The first line of each step turns reporting off there: run create prints no ULID and exits 0, the shards report nothing, and run close exits 0. Without that line, run create exits 2 for the missing token.

With CIRCLECI=true, the run is named CIRCLE_PROJECT_REPONAME #CIRCLE_BUILD_NUM, with the branch CIRCLE_BRANCH, the commit CIRCLE_SHA1 and the build URL CIRCLE_BUILD_URL.

Azure Pipelines

1. Add the variable

Pipelines → your pipeline → Edit → Variables → New variable: name PROBARA_API_TOKEN, the token as the value, Keep this value secret on. A variable group of the library works too. Secret variables never reach scripts on their own: map the variable into the step’s env, as below.

2. The job

pool:
  vmImage: ubuntu-latest

steps:
  - task: NodeTool@0
    inputs:
      versionSpec: '22.x'
  - script: npm ci
  - script: npx vitest run
    env:
      PROBARA_API_TOKEN: $(PROBARA_API_TOKEN)
      PROBARA_PROJECT: SHOP

Sharded jobs: one run

Sharded with parallel into one run created by a first job, and closed by a last one that runs even after failed shards (pattern 2):

jobs:
  - job: create_run
    pool:
      vmImage: ubuntu-latest
    steps:
      - task: NodeTool@0
        inputs:
          versionSpec: '22.x'
      - script: |
          set -e
          ulid=$(npx @probara/cli run create)
          echo "##vso[task.setvariable variable=ulid;isOutput=true]$ulid"
        name: probara
        env:
          PROBARA_API_TOKEN: $(PROBARA_API_TOKEN)
          PROBARA_PROJECT: SHOP

  - job: test
    dependsOn: create_run
    strategy:
      parallel: 4
    variables:
      PROBARA_RUN_ULID: $[ dependencies.create_run.outputs['probara.ulid'] ]
    pool:
      vmImage: ubuntu-latest
    steps:
      - task: NodeTool@0
        inputs:
          versionSpec: '22.x'
      - script: npm ci
      - script: npx vitest run --shard=$(System.JobPositionInPhase)/$(System.TotalJobsInPhase)
        env:
          PROBARA_API_TOKEN: $(PROBARA_API_TOKEN)
          PROBARA_PROJECT: SHOP
          PROBARA_RUN_ULID: $(PROBARA_RUN_ULID)

  - job: close_run
    dependsOn: [create_run, test]
    condition: always()
    variables:
      PROBARA_RUN_ULID: $[ dependencies.create_run.outputs['probara.ulid'] ]
    pool:
      vmImage: ubuntu-latest
    steps:
      - task: NodeTool@0
        inputs:
          versionSpec: '22.x'
      - script: npx @probara/cli run close
        env:
          PROBARA_API_TOKEN: $(PROBARA_API_TOKEN)
          PROBARA_PROJECT: SHOP
          PROBARA_RUN_ULID: $(PROBARA_RUN_ULID)

System.JobPositionInPhase counts from 1, like Vitest’s shards. Each shard sends its results when it ends, into the run of create_run.

Forks and CI detection

Builds of pull requests from forks get no secret variables unless Make secrets available to builds of forks is on (leave it off). There $(PROBARA_API_TOKEN) is not replaced and reaches the reporter as text: Probara refuses it (401), the reporter logs that the results were not sent, and the tests run as usual. In the sharded pipeline, a fork’s run create exits 1 the same way, and its shards never start. To report nothing in fork builds, start each script that runs vitest or probara with if [ "$SYSTEM_PULLREQUEST_ISFORK" = True ]; then export PROBARA_ENABLED=false; fi: run create then prints no ULID and exits 0, the shards report nothing, and run close exits 0.

With TF_BUILD=True, the run is named BUILD_DEFINITIONNAME #BUILD_BUILDNUMBER, and its branch, commit and build URL come from the pipeline’s variables.

Bitbucket Pipelines

1. Add the variable

Repository settings → Pipelines → Repository variables: name PROBARA_API_TOKEN, the token as the value, Secured on. Bitbucket passes repository variables to every step.

2. The pipeline

image: node:22

pipelines:
  default:
    - step:
        name: Vitest
        script:
          - npm ci
          - export PROBARA_PROJECT=SHOP
          - npx vitest run

Parallel steps: one run

Parallel steps that each run a --shard report into one run created by a first step and closed by a last one (pattern 2):

image: node:22

pipelines:
  default:
    - step:
        name: Create the Probara run
        script:
          # Builds without the secured variable (such as forks) report nothing.
          - if [ -z "$PROBARA_API_TOKEN" ]; then export PROBARA_ENABLED=false; fi
          - export PROBARA_PROJECT=SHOP
          - PROBARA_RUN_ULID=$(npx @probara/cli run create)
          - echo "PROBARA_RUN_ULID=$PROBARA_RUN_ULID" > probara.env
        artifacts:
          - probara.env
    - parallel:
        - step:
            name: Shard 1
            script:
              - if [ -z "$PROBARA_API_TOKEN" ]; then export PROBARA_ENABLED=false; fi
              - npm ci
              - export PROBARA_PROJECT=SHOP $(cat probara.env)
              - npx vitest run --shard=1/2
        - step:
            name: Shard 2
            script:
              - if [ -z "$PROBARA_API_TOKEN" ]; then export PROBARA_ENABLED=false; fi
              - npm ci
              - export PROBARA_PROJECT=SHOP $(cat probara.env)
              - npx vitest run --shard=2/2
    - step:
        name: Close the Probara run
        script:
          - if [ -z "$PROBARA_API_TOKEN" ]; then export PROBARA_ENABLED=false; fi
          - export PROBARA_PROJECT=SHOP $(cat probara.env)
          - npx @probara/cli run close

Each shard sends its results when it ends, failed tests included. Bitbucket skips the steps after a failed one, so a shard with failed tests leaves the run open: close it in Probara, or by hand (if the close never ran). For the same reason this pipeline does not merge blob reports: the merging step would not run after a shard with failed tests, and none of the shards would be reported.

Forks and CI detection

Pipelines of a fork run in the fork’s repository, with the fork’s variables, so they have no token. The if line of each step turns reporting off there: run create prints no ULID and exits 0, the shards report nothing, and run close exits 0. Without it, run create exits 2 for the missing token. In the single-step pipeline, the reporter only logs that the token is not set.

With BITBUCKET_BUILD_NUMBER set, the run is named BITBUCKET_REPO_SLUG #BITBUCKET_BUILD_NUMBER, with the branch BITBUCKET_BRANCH, the commit BITBUCKET_COMMIT and a link to the pipeline.

Buildkite

1. Add the token

Expose PROBARA_API_TOKEN to the steps from a secret store: an agent environment hook that reads it from your secrets manager, or Buildkite secrets.

2. The pipeline

A pipeline.yml whose parallel jobs report into one run created by a first step and closed after them, even when they failed (pattern 2). Without parallelism, one step running npx vitest run is enough:

env:
  PROBARA_PROJECT: SHOP

steps:
  - label: Create the Probara run
    key: create-run
    command: |
      PROBARA_RUN_ULID="$(npx @probara/cli run create)"
      if [ -n "$$PROBARA_RUN_ULID" ]; then buildkite-agent meta-data set probara-run "$$PROBARA_RUN_ULID"; fi

  - label: Vitest
    depends_on: create-run
    parallelism: 4
    command: |
      npm ci
      export PROBARA_RUN_ULID="$(buildkite-agent meta-data get probara-run)"
      npx vitest run --shard=$((BUILDKITE_PARALLEL_JOB + 1))/$$BUILDKITE_PARALLEL_JOB_COUNT

  - wait: ~
    continue_on_failure: true

  - label: Close the Probara run
    command: PROBARA_RUN_ULID="$(buildkite-agent meta-data get probara-run)" npx @probara/cli run close

buildkite-agent pipeline upload fills in $NAME and ${NAME} when it uploads the pipeline, and a variable it does not know then, such as BUILDKITE_PARALLEL_JOB_COUNT that only the parallel jobs get, becomes blank; a $$ is left as one $ for the job’s shell, while $(...) and $((...)) are left as they are. The run is created on a line of its own, so a failed run create fails the first step rather than storing a blank ULID. With reporting off (a fork build, below), run create prints no ULID and exits 0: there is nothing to store, and the other steps run with reporting off too. Each shard sends its results when it ends. BUILDKITE_PARALLEL_JOB counts from 0, Vitest’s shards from 1.

With BUILDKITE=true, the run is named BUILDKITE_PIPELINE_SLUG #BUILDKITE_BUILD_NUMBER, with the branch BUILDKITE_BRANCH, the commit BUILDKITE_COMMIT and the build URL BUILDKITE_BUILD_URL.

Fork pull requests

Buildkite builds pull requests from forks only if the pipeline allows it (off by default). Those builds run on your agents, and pipeline upload reads the fork’s own pipeline.yml, which also picks the agents’ queue. The fork’s code runs in the job’s shell as the agent’s user, so on an agent that can read the token from your secret store, the fork’s code can read it too: no hook or step keeps it out. Keep fork builds off, or keep them off those agents: their environment hook refuses a build of a fork (BUILDKITE_PULL_REQUEST_REPO names it), and forks build only on a separate queue or cluster whose agents cannot read the secret store:

# The environment hook of the agents that can read the token: no build of a fork runs there.
if [ -n "$BUILDKITE_PULL_REQUEST_REPO" ]; then
  case "${BUILDKITE_PULL_REQUEST_REPO#*://}" in
    github.com/acme/shop.git) ;;
    *)
      echo "Refusing a build of a fork: $BUILDKITE_PULL_REQUEST_REPO" >&2
      exit 1
      ;;
  esac
fi
export PROBARA_API_TOKEN="$(your-secrets-manager get probara-api-token)"

BUILDKITE_PULL_REQUEST_REPO is empty outside pull requests, and has been an https:// URL since 2022 (a git:// one before), so the hook compares it without the scheme: put your repository in place of github.com/acme/shop.git. When it does not match, your own pull requests fail on these agents too, which the first one shows. On the agents that build forks, set PROBARA_ENABLED=false in their environment hook: the fork’s tests run and npx vitest run reports nothing, and run create and run close exit 0 without a token.

Related: Sharding and projects · Results file in CI · Troubleshooting · Source: docs/ci

Vitest reporter in CI

In every CI

What CI detection fills in

GitHub Actions

1. Add the secret

2. Give the test step the token

Sharded jobs: one run from the blobs

Parallel jobs: one run, closed by the final job

CI detection

Fork pull requests

GitLab CI

1. Add the variable

2. The test job

parallel: shards: one run from the blobs

CI detection

Fork merge requests

Jenkins

1. Add the credential

2. The pipeline

Parallel stages: one run

Fork pull requests

CI detection

CircleCI

1. Add the token

2. The job

Forks and CI detection

Azure Pipelines

1. Add the variable

2. The job

Sharded jobs: one run

Forks and CI detection

Bitbucket Pipelines

1. Add the variable

2. The pipeline

Parallel steps: one run

Forks and CI detection

Buildkite

1. Add the token

2. The pipeline

Fork pull requests

`parallel:` shards: one run from the blobs