Cucumber reporter

@probara/cucumber-reporter is a Cucumber formatter for cucumber-js. It sends every scenario of a run to Probara at the end of the run: each attempt with its steps, their keywords and data, their errors and their attachments, keyed like the scenario’s synced case. A scenario tagged @probara:SHOP-12 lands on that case. Cases that are missing are created, and the run is closed once everything is in.

It is open source, published on npm at 0.1.0, and lives in the probara-sdk repository. Before 1.0, a minor version may bring breaking changes: pin the version and read the changelog before you upgrade.

Sync your .feature files on the default branch first, so Git holds the cases. The reporter then reports the tests. Syncing feature files from Git covers the sync.

What you get

• One result per attempt. A scenario retried with --retry sends each attempt, in order. Each row of a Scenario Outline is a result of its own, recorded on the run case with the row’s values.
• Every step, with its keyword (Given, When, Then, And, But or *), status, duration, error, and the Doc String or Data Table it had. Background steps sit under one Background step.
• Attachments. The files a step attaches with this.attach, this.log and this.link are uploaded with its result.
• Linking by the scenario’s key, built the way probara sync gherkin builds it, or by a @probara:<ID> tag.
• Runs named after the CI build, closed when the run ends.
• It never breaks your test run. A reporting problem is logged on stderr. The exit code of the run stays cucumber-js’s own.

Before you start

You need	Why
Node.js 22.12 or later	The package’s engines range.
@cucumber/cucumber 13.3 or later (13.3 is tested)	The formatter loads from the same install as cucumber-js.
A paid plan (Pro or Enterprise, or any trial)	Reporting from CI is not part of the Free plan. On Free, Probara answers 403 forbidden.
A Probara owner or admin	Only owners and admins create and revoke app tokens.
The Project ID to report into	Such as SHOP: a capital letter, then capital letters or digits.

1. Create an app token

The reporter signs in with an app token: a token that belongs to your organization, not to a person. It uses no seat and keeps working when the person who created it leaves.

1. In Probara, open Workspace › Integrations and select the Cucumber card.
2. Open the Access tokens tab and select Create an app token.
3. Enter a Token name after what uses it (the repository or the pipeline) and select Create token.
4. Select Copy token. It is shown once: store it right away as a CI secret named PROBARA_API_TOKEN.

The Cucumber integration guide covers the card, managing and revoking tokens, and what Probara shows for the runs an app token creates.

2. Install the reporter

Install it next to cucumber-js, in the same project:

npm install --save-dev @cucumber/cucumber @probara/cucumber-reporter

3. Add it to cucumber.js

Add the formatter to format, and your Project ID to formatOptions:

cucumber.js

module.exports = {
  default: {
    format: ['@probara/cucumber-reporter'],
    formatOptions: { probara: { projectId: 'SHOP' } },
  },
};

The formatter replaces cucumber-js’s own progress output. To keep the progress output, give the reporter a file path as well: format: ['progress', '@probara/cucumber-reporter:reports/probara.txt']. The configuration page explains why.

4. Run your tests with the token

Give the step that runs cucumber-js the secret. In GitHub Actions:

- name: Run Cucumber tests
  run: npx cucumber-js
  env:
    PROBARA_API_TOKEN: ${{ secrets.PROBARA_API_TOKEN }}

There is no separate upload step: the reporter sends the results at the end of the run, and closes the run then. CI providers has complete pipelines for GitHub Actions, GitLab CI, Jenkins, CircleCI, Azure Pipelines, Bitbucket Pipelines and Buildkite. They sync the feature files on the default branch before the tests.

What you see

The reporter logs on stderr and prints nothing on stdout. When the run ends, it logs how many results it sent, the run they went to, and, when it uploaded files, how many:

$ npx cucumber-js
[probara] Sending 3 results of 3 tests (3 passed, 0 failed, 0 skipped, 0 blocked)
[probara] Recorded 3 results (2 new cases, 0 unmatched) in R-1 (closed): https://app.probara.net/projects/SHOP/runs/R-1
[probara] Attached 2 files to results (0 skipped, 0 failed)

In Probara, the run appears in the project’s run list as an automated run, named after the CI build (or Automated run <date> outside CI), with its branch, commit and build URL. It shows the Cucumber app and the token’s name (such as Cucumber · e2e-ci) instead of a person.

How Cucumber maps to Probara

Cucumber	In Probara
A scenario, once per attempt	A result on the case with its automation key, or on the case its @probara:<ID> tag names
A row of a Scenario Outline	A result of its own, recorded on the run case with the row’s values
A step	A step of the result, with its keyword, status, duration, error and data
The Background steps	One Background step of the result, with them nested in it
this.attach, this.log and this.link	Files of the result, or of the step that attached them
A Before or After hook that fails	The scenario fails, with the hook’s error or The hook failed. The hook itself is not sent
A scenario that matches no case	A new case, named after the scenario, under the suite of suiteUlid (the project root by default)
The cucumber-js run	One automated run, named after the CI build, closed at the end

Hooks are not steps, so they never appear in the step list. Each hook’s duration still counts in its scenario’s duration.

Reporting never breaks your test run

The reporter never throws into cucumber-js and never changes its exit code: a failing scenario fails the command, and a reporting problem does not. Every problem (a wrong setting, Probara unreachable, a refused token) is logged on stderr, never with the token. To keep what could not be sent, set a results file.

Reference

Page	What it covers
Configuration	Every option, its variable, type and default; precedence; the output; network
Linking test cases	The automation key, outlines, the @probara:<ID> tag and cases that do not exist yet
Steps and attachments	Step fields, Background, hooks, Doc Strings and Data Tables, and the files a scenario attaches
Results and retries	Statuses, retries, status mapping and filter, durations, the results file
CI providers	GitHub Actions, GitLab CI, Jenkins, CircleCI, Azure Pipelines, Bitbucket Pipelines, Buildkite
Troubleshooting	Every message the reporter logs, and what to do about it
Migrating and upgrading	From probara import cucumber, from Qase’s cucumber-js reporter, and upgrading

Related: CI integrations overview · CLI (sync gherkin, and import cucumber for other tools’ JSON reports) · Cucumber integration guide · Syncing feature files from Git · Source on GitHub