CI integrations

Probara records the tests your pipeline runs as an automated test run: one result per test, linked to its test case, with the files the test produced. You choose how the results get there. Each way below creates the run, links every test to its case (and creates the cases that do not exist yet), sends the results and closes the run, so the pipeline needs no extra step to finish it.

Choose how to report

Way	Use it when
CLI and JUnit XML (`@probara/cli`)	Your framework writes JUnit XML: Jest, pytest, Playwright, Maven Surefire, Go with gotestsum, or anything else.
CLI and Cucumber JSON (`@probara/cli`)	Your tests run with a tool that writes Cucumber JSON, such as Cucumber-JVM or playwright-bdd, or you import a report you already have.
Playwright reporter (`@probara/playwright-reporter`)	You run Playwright and want steps, attachments and every attempt of a retried test.
Jest reporter (`@probara/jest-reporter`)	You run Jest and want steps, attachments, links, or to run only the tests of a Probara run.
Cypress reporter (`@probara/cypress-reporter`)	You run Cypress and want each attempt, the failure screenshots and to run only the tests of a Probara run.
Vitest reporter (`@probara/vitest-reporter`)	You run Vitest and want each attempt and repeat, steps, attachments, or to run only the tests of a Probara run.
Cucumber reporter (`@probara/cucumber-reporter`)	You run cucumber-js and want every scenario and step, with its attachments, on the case its synced scenario names, with no JSON file.

All six are open source, in the probara-sdk repository, and published on npm. They need Node.js 22.12 or later on the machine that reports.

What each one sends

What	CLI and JUnit XML	Playwright reporter	Jest reporter	Cypress reporter	Vitest reporter	Cucumber reporter
Framework	Any that writes JUnit XML	`@playwright/test` 1.42 or later	Jest 29.6 or later	Cypress 15.10.0 or later	Vitest 3.2 or later	`@cucumber/cucumber` 13.3 or later
Install	`npx @probara/cli`, or `npm i -D @probara/cli`	`npm i -D @probara/playwright-reporter`	`npm install --save-dev @probara/jest-reporter`	`npm install --save-dev @probara/cypress-reporter`	`npm install --save-dev @probara/vitest-reporter`	`npm i -D @probara/cucumber-reporter`
Results	One per test, with its final outcome	One per attempt	One per attempt	One per attempt	One per attempt	One per attempt, and one per row of a Scenario Outline
Retries and attempts	Earlier attempts summarized in the notes, when the report keeps them	Every attempt is its own result	Every `jest.retryTimes` attempt is its own result	Every Cypress retry is its own result	Every `retry` attempt and every repeat is its own result	Every `--retry` attempt is its own result
Steps	No: JUnit XML has none	`test.step` and hooks, nested	`probara.step()`, nested	`probara.step()`, nested	`probara.step()`, nested	Every step with its keyword, status and data; Background steps nested
Attachments	Files the report points to, and test output with `--attach-output`	Screenshots, videos, traces and every attached file	`probara.attach()` files, and console output with `captureOutput`	Failure screenshots, the spec video, `probara.attach()` files, console output with `captureOutput`	`probara.attach()` and `annotate` files, failure screenshots in browser mode, console output with `captureOutput`	Files of `this.attach`, `this.log` and `this.link`
Parameters	Only as part of the test name	The parameters of each attempt	`probara.parameters()`	`probara.parameters()`, and the browser each spec ran in	`probara.parameters()`, and the Vitest project each test ran in	The values of each Scenario Outline row
Fields of new cases	No	Title, suite, tags, system and custom fields	Title, suite, tags, system and custom fields	Title, suite, tags, system and custom fields	Title, suite, tags, system and custom fields	Title, and the suite of `suiteUlid`
Run selection	No	No	Only the tests of a Probara run	Only the tests of a Probara run	Only the tests of a Probara run	No
Sharded CI	One run for every shard	Merged blob reports, or one shared run	One shared run for every shard	One shared run for every machine	Merged blob reports, or one shared run	One shared run: automatic on GitHub Actions, else `PROBARA_RUN_ULID`

A reporter never changes the exit code of your tests: a failed test fails the command, a reporting problem does not. The CLI runs as its own step, so only a reporting problem fails it.

An app token

Every way signs in with an app token: a token that belongs to your organization, not to a person. It uses no seat and keeps working when the person who created it leaves. An owner or admin creates it in Workspace › Integrations, from the card of the tool you use (JUnit XML, Playwright, Jest, Cypress, Vitest or Cucumber); it starts with probara_app_ and Probara shows it once.

Store it as a CI secret named PROBARA_API_TOKEN. The CLI and the reporters read the token only from that environment variable, so it never goes in a configuration file or on a command line. The token can only do what the reporters and the CLI need: create automated runs, send reports, add attachments to their results, close runs, read the case keys of a run, and sync feature files. Every other call answers 403 forbidden.

A paid plan

Reporting from CI is part of the Pro and Enterprise plans, and of a Pro trial. On the Free plan, Probara keeps your app tokens but refuses them with 403 forbidden; they work again once the organization is back on a paid plan.

How a test finds its test case

Each test becomes a result on one Probara test case:

A case id such as SHOP-12 links the test to that case: in the test title (the CLI and the Playwright, Jest, Cypress and Vitest reporters), in a probara_case property of the JUnit report or a probara_case annotation in Playwright, or with probara.id() in any reporter. A Cucumber scenario links with a @probara:SHOP-12 tag instead.
Otherwise, the automation key does: a key built from the test’s file and title, which each reporter’s documentation shows in full. By default the first report creates the case, and every later report finds it again by the same key. The Cucumber reporter builds the keys of probara sync gherkin, so a result lands on the case its scenario synced to.

The Jest, Cypress and Vitest reporters build the same keys as a JUnit import of jest-junit, cypress-junit and Vitest’s junit reporter output, so moving from the CLI to a reporter keeps every case and its history.

An automated run

Each command creates one automated run, named after the CI build, and closes it when every result is in. To report several jobs into one run, the way depends on the CI provider. In GitHub Actions, the jobs of one workflow run share a run automatically: set PROBARA_CLOSE_RUN: false on the parallel jobs, and close the run in a final job with probara run close, which needs no ULID there. On other CI providers, create the run first with probara run create, pass its ULID to every job in PROBARA_RUN_ULID, and close it at the end with probara run close. Runs, results and attachments show the app’s name and the token name, not a person.

With the GitHub integration, Probara can also create the run and start your GitHub Actions workflow with its ULID: see Running tests with GitHub Actions.

BDD with Gherkin

Keep your .feature files in Git, and let the pipeline sync them and report their results:

Sync the scenarios on each push to the default branch, so Git stays the source of truth for their cases. See Syncing feature files from Git.
Report the scenarios with the Cucumber reporter, which sends each scenario’s result to its case at the end of the run. A cucumber-js run needs no JSON file.

name: BDD

on:
  push:
    branches: [main]

jobs:
  bdd:
    runs-on: ubuntu-latest
    env:
      PROBARA_API_TOKEN: ${{ secrets.PROBARA_API_TOKEN }}
      PROBARA_PROJECT: <PROJECT_CODE>
    steps:
      - uses: actions/checkout@v5
      - run: npm ci
      - name: Sync the feature files
        run: npx @probara/cli sync gherkin "features/**/*.feature"
      - name: Run the scenarios
        run: npx cucumber-js

The sync runs first, so the cases exist before the results arrive. The reporter is set in cucumber.js (see the quick start), so the failing scenarios are reported too. Other tools that write Cucumber JSON, such as Cucumber-JVM, send their report with probara import cucumber after the run instead. The Cucumber CI providers page has the same pipeline for GitLab CI, Jenkins, CircleCI, Azure Pipelines, Bitbucket Pipelines and Buildkite.

Quick start

Any framework with JUnit XML: add npx @probara/cli import junit <report> after your tests, with PROBARA_API_TOKEN and PROBARA_PROJECT in the environment. The CLI page has the step.
Playwright: install the reporter and add ['@probara/playwright-reporter', { projectId: 'SHOP' }] to the reporters of playwright.config. Start with Get started.
Jest: install the reporter and add ['@probara/jest-reporter', { projectId: 'SHOP' }] to the reporters of jest.config.js, after default. Start with Get started.
Cypress: install the reporter and set it as the e2e reporter of cypress.config.js, with its setupNodeEvents helper and its support file. Start with Get started.
Vitest: install the reporter and add ['@probara/vitest-reporter', { projectId: 'SHOP' }] to the reporters of vitest.config.ts, after default. Start with Get started.
Cucumber: install @cucumber/cucumber and @probara/cucumber-reporter, and add @probara/cucumber-reporter to the format of cucumber.js, with your Project ID under formatOptions.probara. Start with Get started.

Without a token and a project, the reporters send nothing, so local runs stay local.

Setup guides

The guides cover the Probara side: the card, its app tokens, and what Probara shows.