# Cypress reporter in CI

> Complete pipelines for @probara/cypress-reporter on GitHub Actions, GitLab CI, Jenkins, CircleCI, Azure Pipelines, Bitbucket Pipelines and Buildkite, with secrets, split specs, CI detection and fork safety.

The reporter runs inside `npx cypress run`: give the step that runs your tests the token, and every
result is sent when the run ends. There is no separate report step. Every example assumes the
reporter, its plugin and its support file are registered as in
[Get started](https://docs.probara.net/en/developers/ci-integrations/cypress/#3-register-the-reporter-and-its-plugin).

## In every CI

- **The token is a secret, the Project ID is not.** Store the app token (from the **Cypress** card
  in **Workspace › Integrations**) as a secret named `PROBARA_API_TOKEN`. Put the Project ID in
  `cypress.config.js` (`projectId`) or in the pipeline (`PROBARA_PROJECT`).
- **Node.js 22.12 or later** for the Node.js that starts `cypress`, which is where the plugin runs,
  and the browsers and system libraries Cypress needs on the agent.
- **Failed tests fail the step, reporting problems do not**: the exit code of `cypress run` stays the
  number of its failed tests.
- **`cypress-io/github-action` and `npm test` work too**: the reporter comes from the config, not
  from the command. Give their step the same environment.
- **Pin the version** in `package.json` (or the lockfile), so a new release never changes a
  pipeline by surprise. The CLI is not a dependency of the reporter: the steps below that run it pin
  it in the command (`npx --yes @probara/cli@0.3`). With `@probara/cli` in your `devDependencies`,
  `npx @probara/cli` after `npm ci` runs the locked version instead.
- **Builds of forks report nothing**: without the secret, the reporter sends nothing and the tests
  run as usual. The CLI exits 2 without a token, so the recipes turn reporting off
  (`PROBARA_ENABLED=false`) where a fork's build would run it: it then exits 0. Each section below
  says how to keep the secret out of fork builds on that CI.
- **Split pipelines** make one run per machine unless they share one. In GitHub Actions the jobs of one
  workflow run share one run automatically
  ([parallel jobs](#parallel-jobs-one-run-closed-by-the-final-job)). Elsewhere, create the run first,
  pass its ULID to every machine, and close it at the end
  ([sharding](https://docs.probara.net/en/developers/ci-integrations/cypress/sharding/)).

### What CI detection fills in

Unless an option or a `PROBARA_*` variable sets them, the run gets its name and source from the CI:

| CI                  | Detected by              | Run name                                          |
| ------------------- | ------------------------ | ------------------------------------------------- |
| GitHub Actions      | `GITHUB_ACTIONS=true`    | `GITHUB_WORKFLOW #GITHUB_RUN_NUMBER`              |
| GitLab CI           | `GITLAB_CI=true`         | `CI_PROJECT_NAME #CI_PIPELINE_IID`                |
| Jenkins             | `JENKINS_URL`            | `JOB_NAME #BUILD_NUMBER`                          |
| CircleCI            | `CIRCLECI=true`          | `CIRCLE_PROJECT_REPONAME #CIRCLE_BUILD_NUM`       |
| Azure Pipelines     | `TF_BUILD=True`          | `BUILD_DEFINITIONNAME #BUILD_BUILDNUMBER`         |
| Bitbucket Pipelines | `BITBUCKET_BUILD_NUMBER` | `BITBUCKET_REPO_SLUG #BITBUCKET_BUILD_NUMBER`     |
| Buildkite           | `BUILDKITE=true`         | `BUILDKITE_PIPELINE_SLUG #BUILDKITE_BUILD_NUMBER` |

The branch, commit and build URL of each are listed in its section, and in
[`@probara/core`'s CI detection](https://github.com/cynch-me/probara-sdk/blob/main/packages/core/README.md#ci-detection).
Override them with `PROBARA_BRANCH`, `PROBARA_COMMIT` and `PROBARA_BUILD_URL`.

## GitHub Actions

### 1. Add the secret

In the repository (or organization): **Settings → Secrets and variables → Actions → New repository
secret**, named `PROBARA_API_TOKEN`, holding the app token.

### 2. Give the test step the token

```yaml
name: Cypress

on:
  push:
    branches: [main]
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-node@v5
        with:
          node-version: 22
      - run: npm ci
      - name: Run Cypress tests
        run: npx cypress run
        env:
          PROBARA_API_TOKEN: ${{ secrets.PROBARA_API_TOKEN }}
          PROBARA_PROJECT: SHOP
```

### Parallel jobs: one run, closed by the final job

In GitHub Actions the jobs of one workflow run report into one run: the run's source names the
workflow run by its repository, run id and attempt, and the first job to report creates the run. A
job closes its run at its end by default, so the parallel jobs set `PROBARA_CLOSE_RUN: false`. A
final job closes the run with `probara run close`, which needs no run ULID there: `needs` makes it
wait for every test job, and `if: always()` runs it when they fail. Cypress has no `--shard` flag,
so each job runs its own specs, here with `--spec`.

```yaml
jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        spec: ['cypress/e2e/cart/**', 'cypress/e2e/login/**']
    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-node@v5
        with:
          node-version: 22
      - run: npm ci
      - run: npx cypress run --spec '${{ matrix.spec }}'
        env:
          PROBARA_API_TOKEN: ${{ secrets.PROBARA_API_TOKEN }}
          PROBARA_PROJECT: SHOP
          # A job leaves the run open for the others.
          PROBARA_CLOSE_RUN: false

  probara:
    needs: test
    if: always()
    runs-on: ubuntu-latest
    steps:
      - uses: actions/setup-node@v5
        with:
          node-version: 22
      - run: npx --yes @probara/cli@0.6 run close
        env:
          PROBARA_API_TOKEN: ${{ secrets.PROBARA_API_TOKEN }}
          PROBARA_PROJECT: SHOP
```

- **A job after the close** is refused with `409 ci_run_closed`: the reporter logs why, sends nothing
  more, and the tests keep their own exit code
  ([troubleshooting](https://docs.probara.net/en/developers/ci-integrations/cypress/troubleshooting/#a-job-reports-after-its-run-was-closed)).
  Keep every reporting job in `needs`.
- **A re-run** of the workflow is another attempt: it reports into a run of its own.
- **One job only** needs neither: its run is closed at the end, as anywhere else.

### CI detection

| Field         | From                                                                                  |
| ------------- | ------------------------------------------------------------------------------------- |
| Run name      | `GITHUB_WORKFLOW #GITHUB_RUN_NUMBER`, such as `Cypress #42`                           |
| Branch        | `GITHUB_HEAD_REF` (the pull request's branch), else `GITHUB_REF_NAME`; none for a tag |
| Commit        | `GITHUB_SHA`                                                                          |
| Build URL     | `GITHUB_SERVER_URL/GITHUB_REPOSITORY/actions/runs/GITHUB_RUN_ID`                      |
| Repository    | `GITHUB_REPOSITORY`                                                                   |
| Commit author | `GITHUB_ACTOR`                                                                        |
| Pull request  | `<n>` of `GITHUB_REF` `refs/pull/<n>/merge`                                           |
| Workflow      | `GITHUB_WORKFLOW`                                                                     |
| CI run        | `GITHUB_RUN_ID`, attempt `GITHUB_RUN_ATTEMPT`                                         |
| CI job        | `GITHUB_JOB`                                                                          |

On `pull_request` events `GITHUB_SHA` is the merge commit GitHub made, not the branch's last commit.
To record the latter, add `PROBARA_COMMIT: ${{ github.event.pull_request.head.sha }}`. In GitHub
Actions the commit must be a 7 to 40 character hex SHA: another value is dropped with a warning.

### Fork pull requests

Workflows triggered by a pull request from a fork get empty secrets. Without a token the reporter
stays off, so fork builds run their tests and report nothing, with no extra setting. Do not use
`pull_request_target` to get the secret into a fork build: it runs the fork's code with your
secrets.

## GitLab CI

### 1. Add the variable

**Settings → CI/CD → Variables → Add variable**: key `PROBARA_API_TOKEN`, the token as the value,
with **Mask variable** on. **Protect variable** limits it to protected branches and tags: turn it on
only if you report from those alone.

### 2. The test job

```yaml
stages:
  - test

cypress:
  stage: test
  image: cypress/browsers:latest
  variables:
    PROBARA_PROJECT: SHOP
  script:
    - npm ci
    - npx cypress run
```

- `cypress/browsers` holds Node.js, the browsers and the system libraries Cypress needs; pin a tag
  whose Node.js is 22.12 or later.
- A job without the variable (a fork's pipeline, an unprotected branch with a protected variable)
  runs its tests and reports nothing.
- To keep what could not be sent, set `PROBARA_RESULTS_FILE` and keep the file and its attachments
  folder as artifacts
  ([results file](https://docs.probara.net/en/developers/ci-integrations/cypress/results-and-retries/#results-file)).
- `parallel:` jobs sharing one run, with a `.pre` job that creates it and a `.post` job that closes
  it, are in [sharding](https://docs.probara.net/en/developers/ci-integrations/cypress/sharding/#gitlab-ci).

### CI detection

| Field     | From                                                                                                                  |
| --------- | --------------------------------------------------------------------------------------------------------------------- |
| Run name  | `CI_PROJECT_NAME #CI_PIPELINE_IID`, such as `shop #42`                                                                |
| Branch    | `CI_MERGE_REQUEST_SOURCE_BRANCH_NAME` in a merge request pipeline, else `CI_COMMIT_REF_NAME`; none for a tag pipeline |
| Commit    | `CI_COMMIT_SHA`                                                                                                       |
| Build URL | `CI_PIPELINE_URL`                                                                                                     |

### Fork merge requests

A merge request from a fork runs its pipeline in the fork's project, which does not have your CI/CD
variables: the reporter stays off there. If maintainers run fork pipelines in the parent project,
those pipelines do get the variable, so review the change before you run it.

## Jenkins

### 1. Add the credential

**Manage Jenkins → Credentials → (a domain) → Add Credentials**: kind **Secret text**, ID
`probara-api-token`, the token as the secret.

### 2. The pipeline

A declarative `Jenkinsfile`, with Node.js 22 and the system libraries Cypress needs on the agent,
binds the token into the test stage with `withCredentials`:

```groovy
pipeline {
  agent any
  environment {
    PROBARA_PROJECT = 'SHOP'
  }
  stages {
    stage('Test') {
      steps {
        sh 'npm ci'
        withCredentials([string(credentialsId: 'probara-api-token', variable: 'PROBARA_API_TOKEN')]) {
          sh 'npx cypress run'
        }
      }
    }
  }
}
```

Jenkins masks the bound secret in the build log.

### Parallel stages: one run

Parallel stages that each run part of the specs make one run each unless they share one: create it
in a first stage, pass it to each, and close it in a `post { always { ... } }` block:

```groovy
pipeline {
  agent any
  environment {
    PROBARA_PROJECT = 'SHOP'
  }
  stages {
    stage('Create the Probara run') {
      steps {
        sh 'npm ci'
        withCredentials([string(credentialsId: 'probara-api-token', variable: 'PROBARA_API_TOKEN')]) {
          sh 'npx --yes @probara/cli@0.3 run create > probara-run-ulid'
        }
        script {
          env.PROBARA_RUN_ULID = readFile('probara-run-ulid').trim()
        }
      }
    }
    stage('Test') {
      parallel {
        stage('Cart') {
          steps {
            withCredentials([string(credentialsId: 'probara-api-token', variable: 'PROBARA_API_TOKEN')]) {
              sh "npx cypress run --spec 'cypress/e2e/cart/**' --config screenshotsFolder=cypress/screenshots-cart"
            }
          }
        }
        stage('Login') {
          steps {
            withCredentials([string(credentialsId: 'probara-api-token', variable: 'PROBARA_API_TOKEN')]) {
              sh "npx cypress run --spec 'cypress/e2e/login/**' --config screenshotsFolder=cypress/screenshots-login"
            }
          }
        }
      }
    }
  }
  post {
    always {
      withCredentials([string(credentialsId: 'probara-api-token', variable: 'PROBARA_API_TOKEN')]) {
        sh 'npx --yes @probara/cli@0.3 run close'
      }
    }
  }
}
```

Both stages share one workspace here, installed once before they start (two `npm ci` at the same
time in one folder would break each other). Each `cypress run` empties its screenshots folder when
it starts, so on one agent each stage gets its own `screenshotsFolder`. On separate agents, each
checks out and installs on its own.

### Fork pull requests

Jenkins does not hide credentials from builds of pull requests by itself: a multibranch job that
builds forks can hand the token to their code. Set **Discover pull requests from forks → Trust** so
that a pull request from anyone else builds with the `Jenkinsfile` of its target branch: in the
GitHub branch source, **From users with Admin or Write permission** (or **Nobody**); in the
Bitbucket branch source, **Nobody**, or **Forks in the same account** from plugin version 871 on
(the fix for CVE-2024-28152). **Everyone** builds a fork with its own `Jenkinsfile`, which can bind
the credential itself, and makes the `env.CHANGE_FORK` guard below useless.

In the target branch's `Jenkinsfile`, bind the credential only when `env.CHANGE_FORK` is not set
(Jenkins sets it for a pull request from a fork), and turn reporting off for a fork build:

```groovy
pipeline {
  agent any
  environment {
    PROBARA_PROJECT = 'SHOP'
  }
  stages {
    stage('Test') {
      steps {
        sh 'npm ci'
        script {
          if (env.CHANGE_FORK) {
            withEnv(['PROBARA_ENABLED=false']) {
              sh 'npx cypress run'
            }
          } else {
            withCredentials([string(credentialsId: 'probara-api-token', variable: 'PROBARA_API_TOKEN')]) {
              sh 'npx cypress run'
            }
          }
        }
      }
    }
  }
}
```

In the parallel pipeline, wrap each `withCredentials` the same way: with reporting off,
`run create` writes no ULID and `run close` exits 0 without a token.

### CI detection

With `JENKINS_URL` set, the run is named `JOB_NAME #BUILD_NUMBER`, and its branch, commit and build
URL come from `BRANCH_NAME` (or `GIT_BRANCH`), `GIT_COMMIT` and `BUILD_URL`.

## CircleCI

### 1. Add the token

**Organization Settings → Contexts → Create Context** (such as `probara-reporting`), then add the
environment variable `PROBARA_API_TOKEN`. A project environment variable (**Project Settings →
Environment Variables**) works too; then drop `context:` below.

### 2. The job

A `.circleci/config.yml` whose specs are split across `parallelism` machines by
`circleci tests split`, into one run created up front. Without `parallelism`, keep only the `test`
job and run `npx cypress run`:

```yaml
version: 2.1

jobs:
  create-run:
    docker:
      - image: cimg/node:22.12
    environment:
      PROBARA_PROJECT: SHOP
    steps:
      - run: |
          # Without the token (a fork's build), turn reporting off: run create prints nothing.
          [ -n "$PROBARA_API_TOKEN" ] || export PROBARA_ENABLED=false
          PROBARA_RUN_ULID=$(npx --yes @probara/cli@0.3 run create)
          echo "PROBARA_RUN_ULID=$PROBARA_RUN_ULID" > probara.env
      - persist_to_workspace:
          root: .
          paths: [probara.env]
  test:
    docker:
      - image: cimg/node:22.12-browsers
    parallelism: 4
    environment:
      PROBARA_PROJECT: SHOP
    steps:
      - checkout
      - attach_workspace:
          at: .
      - run: npm ci
      - run: |
          export $(cat probara.env)
          SPECS=$(circleci tests glob 'cypress/e2e/**/*.cy.js' | circleci tests split | paste -sd, -)
          npx cypress run --spec "$SPECS"
  close-run:
    docker:
      - image: cimg/node:22.12
    environment:
      PROBARA_PROJECT: SHOP
    steps:
      - attach_workspace:
          at: .
      - run:
          when: always
          command: |
            export $(cat probara.env)
            [ -n "$PROBARA_API_TOKEN" ] || export PROBARA_ENABLED=false
            npx --yes @probara/cli@0.3 run close

workflows:
  cypress:
    jobs:
      - create-run:
          context: probara-reporting
      - test:
          context: probara-reporting
          requires: [create-run]
      - close-run:
          context: probara-reporting
          requires:
            - test: [success, failed]
```

`circleci tests split` gives each machine its share of the spec files, and `--spec` takes them as
one comma-separated list. The `-browsers` image holds the browsers and the system libraries Cypress
needs.

### Forks and CI detection

CircleCI does not pass contexts or project variables to builds of forked pull requests unless **Pass
secrets to builds from forked pull requests** is on: leave it off. A fork's jobs that use a context
then do not run at all (CircleCI shows an authorization error). With project variables they run
without the token: the reporter sends nothing, and the `PROBARA_ENABLED=false` line of `create-run`
and `close-run` makes `run create` print nothing and both exit 0.

With `CIRCLECI=true`, the run is named `CIRCLE_PROJECT_REPONAME #CIRCLE_BUILD_NUM`, with the branch
`CIRCLE_BRANCH`, the commit `CIRCLE_SHA1` and the build URL `CIRCLE_BUILD_URL`.

## Azure Pipelines

### 1. Add the variable

**Pipelines → your pipeline → Edit → Variables → New variable**: name `PROBARA_API_TOKEN`, the token
as the value, **Keep this value secret** on. A variable group of the library works too. Secret
variables never reach scripts on their own: map the variable into the step's `env`, as below.

### 2. The job

```yaml
variables:
  # Off in a fork's build, which gets no secrets (see forks below).
  PROBARA_ENABLED: $[ ne(variables['System.PullRequest.IsFork'], 'True') ]

pool:
  vmImage: ubuntu-latest

steps:
  - task: NodeTool@0
    inputs:
      versionSpec: '22.x'
  - script: npm ci
  - script: npx cypress run
    env:
      PROBARA_API_TOKEN: $(PROBARA_API_TOKEN)
      PROBARA_PROJECT: SHOP
```

`ubuntu-latest` has the browsers and the system libraries Cypress needs.

### Parallel jobs: one run

A `matrix` of jobs that each run part of the specs, into one run created by a first job and closed
by a last one that runs even after failed tests:

```yaml
variables:
  PROBARA_ENABLED: $[ ne(variables['System.PullRequest.IsFork'], 'True') ]

jobs:
  - job: create_run
    pool:
      vmImage: ubuntu-latest
    steps:
      - task: NodeTool@0
        inputs:
          versionSpec: '22.x'
      - script: |
          set -e
          ulid=$(npx --yes @probara/cli@0.3 run create)
          echo "##vso[task.setvariable variable=ulid;isOutput=true]$ulid"
        name: probara
        env:
          PROBARA_API_TOKEN: $(PROBARA_API_TOKEN)
          PROBARA_PROJECT: SHOP

  - job: test
    dependsOn: create_run
    strategy:
      matrix:
        cart:
          SPEC: 'cypress/e2e/cart/**'
        login:
          SPEC: 'cypress/e2e/login/**'
    variables:
      PROBARA_RUN_ULID: $[ dependencies.create_run.outputs['probara.ulid'] ]
    pool:
      vmImage: ubuntu-latest
    steps:
      - task: NodeTool@0
        inputs:
          versionSpec: '22.x'
      - script: npm ci
      - script: npx cypress run --spec '$(SPEC)'
        env:
          PROBARA_API_TOKEN: $(PROBARA_API_TOKEN)
          PROBARA_PROJECT: SHOP
          PROBARA_RUN_ULID: $(PROBARA_RUN_ULID)

  - job: close_run
    dependsOn: [create_run, test]
    condition: always()
    variables:
      PROBARA_RUN_ULID: $[ dependencies.create_run.outputs['probara.ulid'] ]
    pool:
      vmImage: ubuntu-latest
    steps:
      - task: NodeTool@0
        inputs:
          versionSpec: '22.x'
      - script: npx --yes @probara/cli@0.3 run close
        env:
          PROBARA_API_TOKEN: $(PROBARA_API_TOKEN)
          PROBARA_PROJECT: SHOP
          PROBARA_RUN_ULID: $(PROBARA_RUN_ULID)
```

### Forks and CI detection

Builds of pull requests from forks get no secret variables unless **Make secrets available to builds
of forks** is on: leave it off. Azure leaves a macro it cannot resolve as it is, so in a fork's
build `$(PROBARA_API_TOKEN)` reaches the step as that literal text, a token Probara refuses with 401. The pipeline's `PROBARA_ENABLED` variable is `False` when `System.PullRequest.IsFork` is
`True`, so a fork's build runs its tests with reporting off, and `run create` and `run close` exit 0.

With `TF_BUILD=True`, the run is named `BUILD_DEFINITIONNAME #BUILD_BUILDNUMBER`, and its branch,
commit and build URL come from the pipeline's variables.

## Bitbucket Pipelines

### 1. Add the variable

**Repository settings → Pipelines → Repository variables**: name `PROBARA_API_TOKEN`, the token as
the value, **Secured** on. Bitbucket passes repository variables to every step, so the reporter finds
it in `npx cypress run`.

### 2. The pipeline

```yaml
image: cypress/browsers:latest

pipelines:
  default:
    - step:
        name: Cypress
        script:
          - npm ci
          - export PROBARA_PROJECT=SHOP
          - npx cypress run
```

### Parallel steps: one run

Parallel steps that each run part of the specs report into one run created by a first step and
closed by a last one:

```yaml
image: cypress/browsers:latest

pipelines:
  default:
    - step:
        name: Create the Probara run
        script:
          - export PROBARA_PROJECT=SHOP
          # Without the token (a fork's build), turn reporting off: run create prints nothing.
          - test -n "$PROBARA_API_TOKEN" || export PROBARA_ENABLED=false
          - PROBARA_RUN_ULID=$(npx --yes @probara/cli@0.3 run create)
          - echo "PROBARA_RUN_ULID=$PROBARA_RUN_ULID" > probara.env
        artifacts:
          - probara.env
    - parallel:
        - step:
            name: Cart
            script:
              - npm ci
              - export PROBARA_PROJECT=SHOP $(cat probara.env)
              - npx cypress run --spec 'cypress/e2e/cart/**' || touch cart.failed
            artifacts:
              - '*.failed'
        - step:
            name: Login
            script:
              - npm ci
              - export PROBARA_PROJECT=SHOP $(cat probara.env)
              - npx cypress run --spec 'cypress/e2e/login/**' || touch login.failed
            artifacts:
              - '*.failed'
    - step:
        name: Close the Probara run
        script:
          - export PROBARA_PROJECT=SHOP $(cat probara.env)
          - test -n "$PROBARA_API_TOKEN" || export PROBARA_ENABLED=false
          - npx --yes @probara/cli@0.3 run close
          - if ls *.failed >/dev/null 2>&1; then echo "Tests failed in a parallel step" >&2; exit 1; fi
```

Bitbucket runs no step after a failed one, so a parallel step that failed would leave the run open.
Each step records failed tests in a `<name>.failed` artifact instead, and the last step closes the
run, then fails the pipeline when a step left one: the parallel steps show as passed, and the last
step carries the failure. If the close still never ran, close the run in Probara, or
[by hand](https://docs.probara.net/en/developers/ci-integrations/cypress/sharding/#if-the-close-never-ran).

### Forks and CI detection

Pull requests from forks get no secured variables: the reporter sends nothing there. With
`BITBUCKET_BUILD_NUMBER` set, the run is named `BITBUCKET_REPO_SLUG #BITBUCKET_BUILD_NUMBER`, with
the branch `BITBUCKET_BRANCH`, the commit `BITBUCKET_COMMIT` and a link to the pipeline.

## Buildkite

### 1. Add the token

Expose `PROBARA_API_TOKEN` to the steps from a secret store: an agent `environment` hook that reads
it from your secrets manager, or
[Buildkite secrets](https://buildkite.com/docs/pipelines/security/secrets/buildkite-secrets).

### 2. The pipeline

A `pipeline.yml` whose parallel jobs report into one run created by a first step and closed after
them, even when they failed. Without `parallelism`, one step running `npx cypress run` is enough:

```yaml
env:
  PROBARA_PROJECT: SHOP

steps:
  - label: Create the Probara run
    key: create-run
    command: |
      PROBARA_RUN_ULID="$(npx --yes @probara/cli@0.3 run create)"
      if [ -n "$$PROBARA_RUN_ULID" ]; then buildkite-agent meta-data set probara-run "$$PROBARA_RUN_ULID"; fi

  - label: Cypress
    depends_on: create-run
    parallelism: 4
    command: |
      npm ci
      export PROBARA_RUN_ULID="$(buildkite-agent meta-data get probara-run --default '')"
      SPECS=$(ls cypress/e2e/*.cy.js | awk -v i="$$BUILDKITE_PARALLEL_JOB" -v n="$$BUILDKITE_PARALLEL_JOB_COUNT" 'NR % n == i' | paste -sd, -)
      npx cypress run --spec "$$SPECS"

  - wait: ~
    continue_on_failure: true

  - label: Close the Probara run
    command: PROBARA_RUN_ULID="$(buildkite-agent meta-data get probara-run --default '')" npx --yes @probara/cli@0.3 run close
```

- `buildkite-agent pipeline upload` fills in `$NAME` and `${NAME}` when it uploads the pipeline, and
  a variable it does not know then, such as `BUILDKITE_PARALLEL_JOB_COUNT` that only the parallel
  jobs get, becomes blank; a `$$` is left as one `$` for the job's shell, while `$(...)` and
  `$((...))` are left as they are.
- The run is created on a line of its own, so a failed `run create` fails the first step rather than
  storing a blank ULID. With reporting off (a fork build on an agent that sets
  `PROBARA_ENABLED=false`, as below), `run create` prints no ULID and exits 0: nothing is stored,
  `meta-data get --default ''` gives the other steps an empty ULID instead of an error, and they run
  with reporting off too.
- Each parallel job takes every `n`-th spec file (`BUILDKITE_PARALLEL_JOB` counts from 0), and
  `--spec` takes them as one comma-separated list. The agents need the browsers and the system
  libraries Cypress needs.

With `BUILDKITE=true`, the run is named `BUILDKITE_PIPELINE_SLUG #BUILDKITE_BUILD_NUMBER`, with the
branch `BUILDKITE_BRANCH`, the commit `BUILDKITE_COMMIT` and the build URL `BUILDKITE_BUILD_URL`.

### Fork pull requests

Buildkite builds pull requests from forks only if the pipeline allows it (off by default). Those
builds run on your agents, and `pipeline upload` reads the fork's own `pipeline.yml`, which also
picks the agents' queue. The fork's code runs in the job's shell as the agent's user, so on an agent
that can read the token from your secret store, the fork's code can read it too: no hook or step
keeps it out.

Keep fork builds off, or keep them off those agents: their `environment` hook refuses a build of a
fork (`BUILDKITE_PULL_REQUEST_REPO` names it), and forks build only on a separate queue or cluster
whose agents cannot read the secret store:

```bash
# The environment hook of the agents that can read the token: no build of a fork runs there.
if [ -n "$BUILDKITE_PULL_REQUEST_REPO" ]; then
  case "${BUILDKITE_PULL_REQUEST_REPO#*://}" in
    github.com/acme/shop.git) ;;
    *)
      echo "Refusing a build of a fork: $BUILDKITE_PULL_REQUEST_REPO" >&2
      exit 1
      ;;
  esac
fi
export PROBARA_API_TOKEN="$(your-secrets-manager get probara-api-token)"
```

`BUILDKITE_PULL_REQUEST_REPO` is empty outside pull requests, and has been an `https://` URL since
2022 (a `git://` one before), so the hook compares it without the scheme: put your repository in
place of `github.com/acme/shop.git`. When it does not match, your own pull requests fail on these
agents too, which the first one shows. On the agents that build forks, set `PROBARA_ENABLED=false`
in their `environment` hook: the fork's tests run and `npx cypress run` reports nothing, and
`run create` and `run close` exit 0 without a token.

Related: [Sharding and projects](https://docs.probara.net/en/developers/ci-integrations/cypress/sharding/) ·
[Results file](https://docs.probara.net/en/developers/ci-integrations/cypress/results-and-retries/#results-file) ·
[Troubleshooting](https://docs.probara.net/en/developers/ci-integrations/cypress/troubleshooting/) · Source:
[docs/ci](https://github.com/cynch-me/probara-sdk/tree/main/packages/cypress-reporter/docs/ci)