# Vitest reporter in CI

> Complete pipelines for @probara/vitest-reporter on GitHub Actions, GitLab CI, Jenkins, CircleCI, Azure Pipelines, Bitbucket Pipelines and Buildkite, with secrets, sharding, CI detection and fork safety.

The reporter runs inside `npx vitest run`: give the step that runs your tests the token, and every
result is sent when Vitest ends. There is no separate report step, and the token is never printed:
a problem names the variable, never its value. Every example assumes the reporter is registered in
`vitest.config.mts` as in
[Get started](https://docs.probara.net/en/developers/ci-integrations/vitest/#3-add-it-to-the-vitest-config).

## In every CI

- **The token is a secret, the Project ID is not.** Store the app token (from the **Vitest** card
  in **Workspace › Integrations**) as a secret named `PROBARA_API_TOKEN`. Put the Project ID in
  `vitest.config.mts` (`projectId`) or in the pipeline (`PROBARA_PROJECT`).
- **Node.js 22.12 or later** on the agent, like the Vitest run the reporter is part of.
- **Failed tests fail the step, reporting problems do not**: Vitest's exit code stays the tests' own
  ([what the reporter never does](https://docs.probara.net/en/developers/ci-integrations/vitest/troubleshooting/#what-the-reporter-never-does-to-the-run)).
- **`npm test` works too** when your `test` script runs `vitest run`: the reporter comes from the
  config, not from the command. A `--reporter` on the command line replaces the config's reporters,
  this one included.
- **Pin the version** in `package.json` (or the lockfile), so a new release never changes a
  pipeline by surprise
  ([upgrading](https://docs.probara.net/en/developers/ci-integrations/vitest/migrating/#upgrading-the-reporter)).
- **Builds of forks report nothing**: without the secret, the reporter sends nothing and the tests
  run as usual. Each section below says how to keep the secret out of fork builds on that CI, and
  how its recipes turn reporting off (`PROBARA_ENABLED=false`) there.
- **Sharded jobs** make one run per shard unless they are tied together. In GitHub Actions the jobs of
  one workflow run share one run automatically
  ([parallel jobs](#parallel-jobs-one-run-closed-by-the-final-job)). Elsewhere, merge the shards' blob
  reports in one job, or create the run first, pass its ULID to every shard, and close it at the end
  ([sharding](https://docs.probara.net/en/developers/ci-integrations/vitest/sharding/)).

### What CI detection fills in

Unless an option or a `PROBARA_*` variable sets them, the run gets its name and source from the CI:

| CI                  | Detected by              | Run name                                          |
| ------------------- | ------------------------ | ------------------------------------------------- |
| GitHub Actions      | `GITHUB_ACTIONS=true`    | `GITHUB_WORKFLOW #GITHUB_RUN_NUMBER`              |
| GitLab CI           | `GITLAB_CI=true`         | `CI_PROJECT_NAME #CI_PIPELINE_IID`                |
| Jenkins             | `JENKINS_URL`            | `JOB_NAME #BUILD_NUMBER`                          |
| CircleCI            | `CIRCLECI=true`          | `CIRCLE_PROJECT_REPONAME #CIRCLE_BUILD_NUM`       |
| Azure Pipelines     | `TF_BUILD=True`          | `BUILD_DEFINITIONNAME #BUILD_BUILDNUMBER`         |
| Bitbucket Pipelines | `BITBUCKET_BUILD_NUMBER` | `BITBUCKET_REPO_SLUG #BITBUCKET_BUILD_NUMBER`     |
| Buildkite           | `BUILDKITE=true`         | `BUILDKITE_PIPELINE_SLUG #BUILDKITE_BUILD_NUMBER` |

The branch, commit and build URL of each are listed in its section, and in
[`@probara/core`'s CI detection](https://github.com/cynch-me/probara-sdk/blob/main/packages/core/README.md#ci-detection).
Override them with `PROBARA_BRANCH`, `PROBARA_COMMIT` and `PROBARA_BUILD_URL`.

## GitHub Actions

### 1. Add the secret

In the repository (or organization): **Settings → Secrets and variables → Actions → New repository
secret**, named `PROBARA_API_TOKEN`, holding the app token.

### 2. Give the test step the token

```yaml
name: Vitest

on:
  push:
    branches: [main]
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-node@v5
        with:
          node-version: 22
      - run: npm ci
      - name: Run Vitest tests
        run: npx vitest run
        env:
          PROBARA_API_TOKEN: ${{ secrets.PROBARA_API_TOKEN }}
          PROBARA_PROJECT: SHOP
          # Fork pull requests get no secrets: report nothing there.
          PROBARA_ENABLED: ${{ secrets.PROBARA_API_TOKEN != '' }}
```

### Sharded jobs: one run from the blobs

The shards write Vitest blob reports with the reporter beside `blob`, and one job merges them: the
whole pipeline is one run, every attempt and file included
([pattern 1](https://docs.probara.net/en/developers/ci-integrations/vitest/sharding/#pattern-1-merge-the-blob-reports)).
Only the merging job needs the token:

```yaml
jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        shardIndex: [1, 2, 3, 4]
        shardTotal: [4]
    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-node@v5
        with:
          node-version: 22
      - run: npm ci
      - run: npx vitest run --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }} --reporter=blob --reporter=@probara/vitest-reporter
      - uses: actions/upload-artifact@v4
        if: ${{ !cancelled() }}
        with:
          name: vitest-blob-${{ matrix.shardIndex }}
          path: .vitest
          include-hidden-files: true
          retention-days: 1

  report:
    needs: test
    if: ${{ !cancelled() }}
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-node@v5
        with:
          node-version: 22
      - run: npm ci
      - uses: actions/download-artifact@v4
        with:
          pattern: vitest-blob-*
          path: .vitest
          merge-multiple: true
      - run: npx vitest --merge-reports
        env:
          PROBARA_API_TOKEN: ${{ secrets.PROBARA_API_TOKEN }}
          PROBARA_PROJECT: SHOP
          PROBARA_ENABLED: ${{ secrets.PROBARA_API_TOKEN != '' }}
```

- `.vitest` holds Vitest 5's blobs and `attachmentsDir`; it is hidden, hence
  `include-hidden-files: true`. On Vitest 3 and 4, upload `.vitest-reports` and
  `.vitest-attachments` (one per line of `path:`), and download them to `.`.
- GitHub-hosted runners check out every job at the same path, so the merge keeps every key
  ([merge at the same path](https://docs.probara.net/en/developers/ci-integrations/vitest/sharding/#merge-at-the-same-path)).
- The `report` job runs after failed shards too, and its exit code is that of the merged tests.

### Parallel jobs: one run, closed by the final job

In GitHub Actions the jobs of one workflow run report into one run: the run's source names the
workflow run by its repository, run id and attempt, and the first job to report creates the run. A
job closes its run at its end by default, so the parallel jobs set `PROBARA_CLOSE_RUN: false`. A
final job closes the run with `probara run close`, which needs no run ULID there: `needs` makes it
wait for every test job, and `if: always()` runs it when they fail. Each shard reports as it ends.

```yaml
jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        shard: [1, 2]
    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-node@v5
        with:
          node-version: 22
      - run: npm ci
      - run: npx vitest run --shard=${{ matrix.shard }}/2
        env:
          PROBARA_API_TOKEN: ${{ secrets.PROBARA_API_TOKEN }}
          PROBARA_PROJECT: SHOP
          # A shard leaves the run open for the others.
          PROBARA_CLOSE_RUN: false

  probara:
    needs: test
    if: always()
    runs-on: ubuntu-latest
    steps:
      - uses: actions/setup-node@v5
        with:
          node-version: 22
      - run: npx --yes @probara/cli@0.6 run close
        env:
          PROBARA_API_TOKEN: ${{ secrets.PROBARA_API_TOKEN }}
          PROBARA_PROJECT: SHOP
```

- **A job after the close** is refused with `409 ci_run_closed`: the reporter logs why, sends nothing
  more, and the tests keep their own exit code
  ([troubleshooting](https://docs.probara.net/en/developers/ci-integrations/vitest/troubleshooting/#a-job-reports-after-its-run-was-closed)).
  Keep every reporting job in `needs`.
- **A re-run** of the workflow is another attempt: it reports into a run of its own.
- **One job only** needs neither: its run is closed at the end, as anywhere else.

### CI detection

| Field         | From                                                                                  |
| ------------- | ------------------------------------------------------------------------------------- |
| Run name      | `GITHUB_WORKFLOW #GITHUB_RUN_NUMBER`, such as `Vitest #42`                            |
| Branch        | `GITHUB_HEAD_REF` (the pull request's branch), else `GITHUB_REF_NAME`; none for a tag |
| Commit        | `GITHUB_SHA`                                                                          |
| Build URL     | `GITHUB_SERVER_URL/GITHUB_REPOSITORY/actions/runs/GITHUB_RUN_ID`                      |
| Repository    | `GITHUB_REPOSITORY`                                                                   |
| Commit author | `GITHUB_ACTOR`                                                                        |
| Pull request  | `<n>` of `GITHUB_REF` `refs/pull/<n>/merge`                                           |
| Workflow      | `GITHUB_WORKFLOW`                                                                     |
| CI run        | `GITHUB_RUN_ID`, attempt `GITHUB_RUN_ATTEMPT`                                         |
| CI job        | `GITHUB_JOB`                                                                          |

On `pull_request` events `GITHUB_SHA` is the merge commit GitHub made, not the branch's last commit.
To record the latter, add `PROBARA_COMMIT: ${{ github.event.pull_request.head.sha }}`. In GitHub
Actions the commit must be a 7 to 40 character hex SHA: another value is dropped with a warning.

### Fork pull requests

Workflows triggered by a pull request from a fork get empty secrets. Without the token, the
reporter says so, sends nothing, and the tests run as usual:

```text
$ PROBARA_PROJECT=SHOP npx vitest run
[probara] Probara reporting is off: The API token is not set: pass apiToken or set PROBARA_API_TOKEN
```

The workflows above set `PROBARA_ENABLED` from whether the secret is there, so fork builds report
nothing and log nothing. Do not use `pull_request_target` to get the secret into a fork build: it
runs the fork's code with your secrets.

## GitLab CI

### 1. Add the variable

**Settings → CI/CD → Variables → Add variable**: key `PROBARA_API_TOKEN`, the token as the value,
with **Mask variable** on. **Protect variable** limits it to protected branches and tags: turn it on
only if you report from those alone.

### 2. The test job

```yaml
stages:
  - test

vitest:
  stage: test
  image: node:22
  variables:
    PROBARA_PROJECT: SHOP
  script:
    - npm ci
    - npx vitest run
```

- A job without the variable (a fork's pipeline, an unprotected branch with a protected variable)
  runs its tests and sends nothing; the reporter logs that the token is not set.
- To keep what could not be sent, set `PROBARA_RESULTS_FILE` and keep the file as an artifact
  ([results file](https://docs.probara.net/en/developers/ci-integrations/vitest/results-and-retries/#in-ci)).

### `parallel:` shards: one run from the blobs

`parallel:` shards write Vitest blob reports with the reporter beside `blob`, and one job merges
them into one run
([pattern 1](https://docs.probara.net/en/developers/ci-integrations/vitest/sharding/#pattern-1-merge-the-blob-reports)).
Only the merging job needs the token:

```yaml
stages:
  - test
  - report

vitest:
  stage: test
  image: node:22
  parallel: 4
  script:
    - npm ci
    - npx vitest run --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL --reporter=blob --reporter=@probara/vitest-reporter
  artifacts:
    when: always
    paths:
      - .vitest/

report-to-probara:
  stage: report
  image: node:22
  needs:
    - job: vitest
      artifacts: true
  variables:
    PROBARA_PROJECT: SHOP
  rules:
    # After failed shards too, and only where the token is set.
    - if: $PROBARA_API_TOKEN
      when: always
  script:
    - npm ci
    - npx vitest --merge-reports
```

- A `needs:` on a `parallel:` job downloads the artifacts of every instance into the same folder;
  each shard's blob has its own file name (`blob-1-4.json`...), so they sit side by side.
- `.vitest/` holds Vitest 5's blobs and `attachmentsDir`. On Vitest 3 and 4, keep
  `.vitest-reports/` and `.vitest-attachments/` instead.
- The merging job must check out at the shards' `CI_PROJECT_DIR`, or every key changes
  ([merge at the same path](https://docs.probara.net/en/developers/ci-integrations/vitest/sharding/#merge-at-the-same-path)).

To have each shard report as it ends instead, create the run in a `.pre` job: see
[pattern 2](https://docs.probara.net/en/developers/ci-integrations/vitest/sharding/#gitlab-ci).

### CI detection

| Field     | From                                                                                                                  |
| --------- | --------------------------------------------------------------------------------------------------------------------- |
| Run name  | `CI_PROJECT_NAME #CI_PIPELINE_IID`, such as `shop #42`                                                                |
| Branch    | `CI_MERGE_REQUEST_SOURCE_BRANCH_NAME` in a merge request pipeline, else `CI_COMMIT_REF_NAME`; none for a tag pipeline |
| Commit    | `CI_COMMIT_SHA`                                                                                                       |
| Build URL | `CI_PIPELINE_URL`                                                                                                     |

### Fork merge requests

A merge request from a fork runs its pipeline in the fork's project, which does not have your CI/CD
variables: the reporter sends nothing there, and the rule `if: $PROBARA_API_TOKEN` leaves the
merging job out. If maintainers run fork pipelines in the parent project, those pipelines do get
the variable, so review the change before you run it.

## Jenkins

### 1. Add the credential

**Manage Jenkins → Credentials → (a domain) → Add Credentials**: kind **Secret text**, ID
`probara-api-token`, the token as the secret.

### 2. The pipeline

A declarative `Jenkinsfile` with Node.js 22 on the agent binds the token into the test stage with
`withCredentials`:

```groovy
pipeline {
  agent any
  environment {
    PROBARA_PROJECT = 'SHOP'
  }
  stages {
    stage('Test') {
      steps {
        sh 'npm ci'
        withCredentials([string(credentialsId: 'probara-api-token', variable: 'PROBARA_API_TOKEN')]) {
          sh 'npx vitest run'
        }
      }
    }
  }
}
```

Jenkins masks the bound secret in the build log, and the reporter never prints it.

### Parallel stages: one run

Parallel stages that run `--shard`s make one run each unless they share one: create it in a first
stage, pass it to each, and close it in a `post { always { ... } }` block
([pattern 2](https://docs.probara.net/en/developers/ci-integrations/vitest/sharding/#pattern-2-create-the-run-report-from-each-shard-close-it)):

```groovy
pipeline {
  agent any
  environment {
    PROBARA_PROJECT = 'SHOP'
  }
  stages {
    stage('Create the Probara run') {
      steps {
        sh 'npm ci'
        withCredentials([string(credentialsId: 'probara-api-token', variable: 'PROBARA_API_TOKEN')]) {
          sh 'npx @probara/cli run create > probara-run-ulid'
        }
        script {
          env.PROBARA_RUN_ULID = readFile('probara-run-ulid').trim()
        }
      }
    }
    stage('Test') {
      parallel {
        stage('Shard 1') {
          steps {
            withCredentials([string(credentialsId: 'probara-api-token', variable: 'PROBARA_API_TOKEN')]) {
              sh 'npx vitest run --shard=1/2'
            }
          }
        }
        stage('Shard 2') {
          steps {
            withCredentials([string(credentialsId: 'probara-api-token', variable: 'PROBARA_API_TOKEN')]) {
              sh 'npx vitest run --shard=2/2'
            }
          }
        }
      }
    }
  }
  post {
    always {
      withCredentials([string(credentialsId: 'probara-api-token', variable: 'PROBARA_API_TOKEN')]) {
        sh 'npx @probara/cli run close'
      }
    }
  }
}
```

Both shards share one workspace here, installed once before they start (two `npm ci` at the same
time in one folder would break each other); on separate agents, each checks out and installs on its
own. Each shard sends its results when it ends, into the run of the first stage.

### Fork pull requests

Jenkins does not hide credentials from builds of pull requests by itself: a multibranch job that
builds forks can hand the token to their code. Set **Discover pull requests from forks → Trust** so
that a pull request from anyone else builds with the `Jenkinsfile` of its target branch: in the
GitHub branch source, **From users with Admin or Write permission** (or **Nobody**); in the
Bitbucket branch source, **Nobody**, or **Forks in the same account** from plugin version 871 on
(the fix for CVE-2024-28152). **Everyone** builds a fork with its own `Jenkinsfile`, which can bind
the credential itself, and makes the `env.CHANGE_FORK` guard below useless.

In the target branch's `Jenkinsfile`, bind the credential only when `env.CHANGE_FORK` is not set
(Jenkins sets it for a pull request from a fork), and turn reporting off for a fork build:

```groovy
pipeline {
  agent any
  environment {
    PROBARA_PROJECT = 'SHOP'
  }
  stages {
    stage('Test') {
      steps {
        sh 'npm ci'
        script {
          if (env.CHANGE_FORK) {
            withEnv(['PROBARA_ENABLED=false']) {
              sh 'npx vitest run'
            }
          } else {
            withCredentials([string(credentialsId: 'probara-api-token', variable: 'PROBARA_API_TOKEN')]) {
              sh 'npx vitest run'
            }
          }
        }
      }
    }
  }
}
```

In the parallel pipeline, wrap each `withCredentials` the same way: with reporting off,
`run create` writes no ULID and `run close` exits 0 without a token.

### CI detection

With `JENKINS_URL` set, the run is named `JOB_NAME #BUILD_NUMBER`, and its branch, commit and build
URL come from `BRANCH_NAME` (or `GIT_BRANCH`), `GIT_COMMIT` and `BUILD_URL`.

## CircleCI

### 1. Add the token

**Organization Settings → Contexts → Create Context** (such as `probara-reporting`), then add the
environment variable `PROBARA_API_TOKEN`. A project environment variable (**Project Settings →
Environment Variables**) works too; then drop `context:` below.

### 2. The job

A `.circleci/config.yml` in a Node.js image, sharded with `parallelism` into one run created up
front
([pattern 2](https://docs.probara.net/en/developers/ci-integrations/vitest/sharding/#pattern-2-create-the-run-report-from-each-shard-close-it)).
Without `parallelism`, keep only the `test` job and run `npx vitest run`:

```yaml
version: 2.1

jobs:
  create-run:
    docker:
      - image: cimg/node:22.12
    environment:
      PROBARA_PROJECT: SHOP
    steps:
      - run: |
          # Forked pull requests get no secrets: report nothing there.
          if [ -z "$PROBARA_API_TOKEN" ]; then export PROBARA_ENABLED=false; fi
          PROBARA_RUN_ULID=$(npx @probara/cli run create)
          echo "PROBARA_RUN_ULID=$PROBARA_RUN_ULID" > probara.env
      - persist_to_workspace:
          root: .
          paths: [probara.env]
  test:
    docker:
      - image: cimg/node:22.12
    parallelism: 4
    environment:
      PROBARA_PROJECT: SHOP
    steps:
      - checkout
      - attach_workspace:
          at: .
      - run: npm ci
      - run: |
          if [ -z "$PROBARA_API_TOKEN" ]; then export PROBARA_ENABLED=false; fi
          export $(cat probara.env)
          npx vitest run --shard=$((CIRCLE_NODE_INDEX + 1))/$CIRCLE_NODE_TOTAL
  close-run:
    docker:
      - image: cimg/node:22.12
    environment:
      PROBARA_PROJECT: SHOP
    steps:
      - attach_workspace:
          at: .
      - run:
          when: always
          command: |
            if [ -z "$PROBARA_API_TOKEN" ]; then export PROBARA_ENABLED=false; fi
            export $(cat probara.env)
            npx @probara/cli run close

workflows:
  vitest:
    jobs:
      - create-run:
          context: probara-reporting
      - test:
          context: probara-reporting
          requires: [create-run]
      - close-run:
          context: probara-reporting
          requires:
            - test: [success, failed]
```

`CIRCLE_NODE_INDEX` counts from 0, Vitest's shards from 1. Each shard sends its results when it
ends, into the run of `run create`; `close-run` closes it after failed shards too.

### Forks and CI detection

CircleCI does not pass contexts or project variables to builds of forked pull requests (unless
**Pass secrets to builds from forked pull requests** is on: leave it off). The first line of each
step turns reporting off there: `run create` prints no ULID and exits 0, the shards report nothing,
and `run close` exits 0. Without that line, `run create` exits 2 for the missing token.

With `CIRCLECI=true`, the run is named `CIRCLE_PROJECT_REPONAME #CIRCLE_BUILD_NUM`, with the branch
`CIRCLE_BRANCH`, the commit `CIRCLE_SHA1` and the build URL `CIRCLE_BUILD_URL`.

## Azure Pipelines

### 1. Add the variable

**Pipelines → your pipeline → Edit → Variables → New variable**: name `PROBARA_API_TOKEN`, the token
as the value, **Keep this value secret** on. A variable group of the library works too. Secret
variables never reach scripts on their own: map the variable into the step's `env`, as below.

### 2. The job

```yaml
pool:
  vmImage: ubuntu-latest

steps:
  - task: NodeTool@0
    inputs:
      versionSpec: '22.x'
  - script: npm ci
  - script: npx vitest run
    env:
      PROBARA_API_TOKEN: $(PROBARA_API_TOKEN)
      PROBARA_PROJECT: SHOP
```

### Sharded jobs: one run

Sharded with `parallel` into one run created by a first job, and closed by a last one that runs even
after failed shards
([pattern 2](https://docs.probara.net/en/developers/ci-integrations/vitest/sharding/#pattern-2-create-the-run-report-from-each-shard-close-it)):

```yaml
jobs:
  - job: create_run
    pool:
      vmImage: ubuntu-latest
    steps:
      - task: NodeTool@0
        inputs:
          versionSpec: '22.x'
      - script: |
          set -e
          ulid=$(npx @probara/cli run create)
          echo "##vso[task.setvariable variable=ulid;isOutput=true]$ulid"
        name: probara
        env:
          PROBARA_API_TOKEN: $(PROBARA_API_TOKEN)
          PROBARA_PROJECT: SHOP

  - job: test
    dependsOn: create_run
    strategy:
      parallel: 4
    variables:
      PROBARA_RUN_ULID: $[ dependencies.create_run.outputs['probara.ulid'] ]
    pool:
      vmImage: ubuntu-latest
    steps:
      - task: NodeTool@0
        inputs:
          versionSpec: '22.x'
      - script: npm ci
      - script: npx vitest run --shard=$(System.JobPositionInPhase)/$(System.TotalJobsInPhase)
        env:
          PROBARA_API_TOKEN: $(PROBARA_API_TOKEN)
          PROBARA_PROJECT: SHOP
          PROBARA_RUN_ULID: $(PROBARA_RUN_ULID)

  - job: close_run
    dependsOn: [create_run, test]
    condition: always()
    variables:
      PROBARA_RUN_ULID: $[ dependencies.create_run.outputs['probara.ulid'] ]
    pool:
      vmImage: ubuntu-latest
    steps:
      - task: NodeTool@0
        inputs:
          versionSpec: '22.x'
      - script: npx @probara/cli run close
        env:
          PROBARA_API_TOKEN: $(PROBARA_API_TOKEN)
          PROBARA_PROJECT: SHOP
          PROBARA_RUN_ULID: $(PROBARA_RUN_ULID)
```

`System.JobPositionInPhase` counts from 1, like Vitest's shards. Each shard sends its results when
it ends, into the run of `create_run`.

### Forks and CI detection

Builds of pull requests from forks get no secret variables unless **Make secrets available to
builds of forks** is on (leave it off). There `$(PROBARA_API_TOKEN)` is not replaced and reaches the
reporter as text: Probara refuses it (`401`), the reporter logs that the results were not sent, and
the tests run as usual. In the sharded pipeline, a fork's `run create` exits 1 the same way, and its
shards never start. To report nothing in fork builds, start each script that runs `vitest` or
`probara` with `if [ "$SYSTEM_PULLREQUEST_ISFORK" = True ]; then export PROBARA_ENABLED=false; fi`:
`run create` then prints no ULID and exits 0, the shards report nothing, and `run close` exits 0.

With `TF_BUILD=True`, the run is named `BUILD_DEFINITIONNAME #BUILD_BUILDNUMBER`, and its branch,
commit and build URL come from the pipeline's variables.

## Bitbucket Pipelines

### 1. Add the variable

**Repository settings → Pipelines → Repository variables**: name `PROBARA_API_TOKEN`, the token as
the value, **Secured** on. Bitbucket passes repository variables to every step.

### 2. The pipeline

```yaml
image: node:22

pipelines:
  default:
    - step:
        name: Vitest
        script:
          - npm ci
          - export PROBARA_PROJECT=SHOP
          - npx vitest run
```

### Parallel steps: one run

Parallel steps that each run a `--shard` report into one run created by a first step and closed by
a last one
([pattern 2](https://docs.probara.net/en/developers/ci-integrations/vitest/sharding/#pattern-2-create-the-run-report-from-each-shard-close-it)):

```yaml
image: node:22

pipelines:
  default:
    - step:
        name: Create the Probara run
        script:
          # Builds without the secured variable (such as forks) report nothing.
          - if [ -z "$PROBARA_API_TOKEN" ]; then export PROBARA_ENABLED=false; fi
          - export PROBARA_PROJECT=SHOP
          - PROBARA_RUN_ULID=$(npx @probara/cli run create)
          - echo "PROBARA_RUN_ULID=$PROBARA_RUN_ULID" > probara.env
        artifacts:
          - probara.env
    - parallel:
        - step:
            name: Shard 1
            script:
              - if [ -z "$PROBARA_API_TOKEN" ]; then export PROBARA_ENABLED=false; fi
              - npm ci
              - export PROBARA_PROJECT=SHOP $(cat probara.env)
              - npx vitest run --shard=1/2
        - step:
            name: Shard 2
            script:
              - if [ -z "$PROBARA_API_TOKEN" ]; then export PROBARA_ENABLED=false; fi
              - npm ci
              - export PROBARA_PROJECT=SHOP $(cat probara.env)
              - npx vitest run --shard=2/2
    - step:
        name: Close the Probara run
        script:
          - if [ -z "$PROBARA_API_TOKEN" ]; then export PROBARA_ENABLED=false; fi
          - export PROBARA_PROJECT=SHOP $(cat probara.env)
          - npx @probara/cli run close
```

Each shard sends its results when it ends, failed tests included. Bitbucket skips the steps after
a failed one, so a shard with failed tests leaves the run open: close it in Probara, or by hand
([if the close never ran](https://docs.probara.net/en/developers/ci-integrations/vitest/sharding/#if-the-close-never-ran)).
For the same reason this pipeline does not merge blob reports: the merging step would not run after
a shard with failed tests, and none of the shards would be reported.

### Forks and CI detection

Pipelines of a fork run in the fork's repository, with the fork's variables, so they have no token.
The `if` line of each step turns reporting off there: `run create` prints no ULID and exits 0, the
shards report nothing, and `run close` exits 0. Without it, `run create` exits 2 for the missing
token. In the single-step pipeline, the reporter only logs that the token is not set.

With `BITBUCKET_BUILD_NUMBER` set, the run is named `BITBUCKET_REPO_SLUG #BITBUCKET_BUILD_NUMBER`,
with the branch `BITBUCKET_BRANCH`, the commit `BITBUCKET_COMMIT` and a link to the pipeline.

## Buildkite

### 1. Add the token

Expose `PROBARA_API_TOKEN` to the steps from a secret store: an agent `environment` hook that reads
it from your secrets manager, or
[Buildkite secrets](https://buildkite.com/docs/pipelines/security/secrets/buildkite-secrets).

### 2. The pipeline

A `pipeline.yml` whose parallel jobs report into one run created by a first step and closed after
them, even when they failed
([pattern 2](https://docs.probara.net/en/developers/ci-integrations/vitest/sharding/#pattern-2-create-the-run-report-from-each-shard-close-it)).
Without `parallelism`, one step running `npx vitest run` is enough:

```yaml
env:
  PROBARA_PROJECT: SHOP

steps:
  - label: Create the Probara run
    key: create-run
    command: |
      PROBARA_RUN_ULID="$(npx @probara/cli run create)"
      if [ -n "$$PROBARA_RUN_ULID" ]; then buildkite-agent meta-data set probara-run "$$PROBARA_RUN_ULID"; fi

  - label: Vitest
    depends_on: create-run
    parallelism: 4
    command: |
      npm ci
      export PROBARA_RUN_ULID="$(buildkite-agent meta-data get probara-run)"
      npx vitest run --shard=$((BUILDKITE_PARALLEL_JOB + 1))/$$BUILDKITE_PARALLEL_JOB_COUNT

  - wait: ~
    continue_on_failure: true

  - label: Close the Probara run
    command: PROBARA_RUN_ULID="$(buildkite-agent meta-data get probara-run)" npx @probara/cli run close
```

`buildkite-agent pipeline upload` fills in `$NAME` and `${NAME}` when it uploads the pipeline, and a
variable it does not know then, such as `BUILDKITE_PARALLEL_JOB_COUNT` that only the parallel jobs
get, becomes blank; a `$$` is left as one `$` for the job's shell, while `$(...)` and `$((...))`
are left as they are. The run is created on a line of its own, so a failed `run create` fails the
first step rather than storing a blank ULID. With reporting off (a fork build, below), `run create`
prints no ULID and exits 0: there is nothing to store, and the other steps run with reporting off
too. Each shard sends its results when it ends.
`BUILDKITE_PARALLEL_JOB` counts from 0, Vitest's shards from 1.

With `BUILDKITE=true`, the run is named `BUILDKITE_PIPELINE_SLUG #BUILDKITE_BUILD_NUMBER`, with the
branch `BUILDKITE_BRANCH`, the commit `BUILDKITE_COMMIT` and the build URL `BUILDKITE_BUILD_URL`.

### Fork pull requests

Buildkite builds pull requests from forks only if the pipeline allows it (off by default). Those
builds run on your agents, and `pipeline upload` reads the fork's own `pipeline.yml`, which also
picks the agents' queue. The fork's code runs in the job's shell as the agent's user, so on an agent
that can read the token from your secret store, the fork's code can read it too: no hook or step
keeps it out. Keep fork builds off, or keep them off those agents: their `environment` hook refuses
a build of a fork (`BUILDKITE_PULL_REQUEST_REPO` names it), and forks build only on a separate
queue or cluster whose agents cannot read the secret store:

```bash
# The environment hook of the agents that can read the token: no build of a fork runs there.
if [ -n "$BUILDKITE_PULL_REQUEST_REPO" ]; then
  case "${BUILDKITE_PULL_REQUEST_REPO#*://}" in
    github.com/acme/shop.git) ;;
    *)
      echo "Refusing a build of a fork: $BUILDKITE_PULL_REQUEST_REPO" >&2
      exit 1
      ;;
  esac
fi
export PROBARA_API_TOKEN="$(your-secrets-manager get probara-api-token)"
```

`BUILDKITE_PULL_REQUEST_REPO` is empty outside pull requests, and has been an `https://` URL since
2022 (a `git://` one before), so the hook compares it without the scheme: put your repository in
place of `github.com/acme/shop.git`. When it does not match, your own pull requests fail on these
agents too, which the first one shows. On the agents that build forks, set `PROBARA_ENABLED=false`
in their `environment` hook: the fork's tests run and `npx vitest run` reports nothing, and
`run create` and `run close` exit 0 without a token.

Related: [Sharding and projects](https://docs.probara.net/en/developers/ci-integrations/vitest/sharding/) ·
[Results file in CI](https://docs.probara.net/en/developers/ci-integrations/vitest/results-and-retries/#in-ci) ·
[Troubleshooting](https://docs.probara.net/en/developers/ci-integrations/vitest/troubleshooting/) · Source:
[docs/ci](https://github.com/cynch-me/probara-sdk/tree/main/packages/vitest-reporter/docs/ci)