Reporting from CI with Cucumber

The Cucumber app sends your cucumber-js results to Probara from any CI pipeline, with the Probara reporter for Cucumber. Each run becomes an automated test run with its results: every scenario and step, the test cases each scenario links to, and the files it attached. The same app syncs your .feature files, so Git keeps their scenarios as the cases in Probara.

The reporter signs in with an app token: a token that belongs to the organization, not to a person. It uses no seat, and it keeps working when the person who created it leaves. This guide covers the card in Probara and the setup. The reporter’s documentation covers every reporter option, the steps and attachments, and the rules that link a scenario to its case.

Before you start

Set it up

1. In Workspace › Integrations, select the Cucumber card. The dialog lists what the app can do and links to this guide.

2. Open the Access tokens tab and select Create an app token.

3. Enter a Token name and select Create token. Name it after what uses it: the repository or the pipeline, so you know which one to revoke later.

4. Select Copy token in Copy your new token. This is the only time Probara shows it: store it in your CI as a secret named PROBARA_API_TOKEN before you select Done.

5. Install the reporter next to cucumber-js:

   npm install --save-dev @cucumber/cucumber @probara/cucumber-reporter

6. Add the formatter to format in cucumber.js, and replace <PROJECT_CODE> with your project’s Project ID, such as SHOP:

   module.exports = {
     default: {
       format: ['@probara/cucumber-reporter'],
       formatOptions: { probara: { projectId: '<PROJECT_CODE>' } },
     },
   };

   The reporter replaces cucumber-js’s progress output. To keep it, give the reporter a file path too: format: ['progress', '@probara/cucumber-reporter:reports/probara.txt'].

7. Pass the secret to your tests and run them. For example, in GitHub Actions:

   - name: Run Cucumber tests
     run: npx cucumber-js
     env:
       PROBARA_API_TOKEN: ${{ secrets.PROBARA_API_TOKEN }}

The token never goes in cucumber.js: the reporter reads it from the environment. The card on the Integrations page shows how many active tokens the app has.

Sync the feature files

Results land on cases, so sync the .feature files first, on your default branch, with the same token:

- name: Sync the feature files to Probara
  run: npx @probara/cli sync gherkin "features/**/*.feature"
  env:
    PROBARA_API_TOKEN: ${{ secrets.PROBARA_API_TOKEN }}
    PROBARA_PROJECT: <PROJECT_CODE>

The Syncing feature files from Git guide covers the sync: its keys, moves and bindings, archiving, and which fields of a synced case stay read-only. The CI providers page has complete pipelines that sync and run the tests.

What the reporter sends

A reporting problem never fails your tests. An option under probara that the reporter does not know is ignored, with a warning in the log that names it.

Same cases as the Cucumber JSON import

If you already send the Cucumber JSON report with probara import cucumber, the reporter finds the same cases: its keys are the keys of the sync and of the import, so switching keeps each scenario on its case and its history. Two differences are worth knowing. The reporter sends the results at the end of the run, and it uploads the files its steps attach, which the import does not send. Remove the import step once the reporter runs, so that the same results are not sent twice.

Other tools that write Cucumber JSON

Cucumber-JVM, playwright-bdd and other tools write Cucumber JSON reports. Send those reports after the run with probara import cucumber, which the CLI page describes. The reporter is for cucumber-js.

Manage tokens

Under App tokens, the Access tokens tab lists each active token with who created it (or Created by a removed user), when, and when it was last used (Never until its first report).

Revoke a token, and confirm, when it may have leaked or when someone who could read it leaves your team: pipelines that use it stop reporting at once, and it cannot be undone. Create a new token for them first. Because an app token outlives the person who created it, rotating it is your call: Probara never revokes it on its own.

What an app token can do

An app token can only do what the reporter and the sync do: create automated runs, send reports (which can create the test cases and suites they name), add attachments to their results, close runs, and sync feature files (which create, update and archive synced cases). Every other call answers 403 forbidden. The exact calls are listed in the authentication reference. For scripts and other API clients, use a personal API token.

Runs, activity entries and attachments that an app token created show the app’s logo and name beside the token name, for example Cucumber · e2e-ci, in the runs list, the run and milestone summaries, Workspace activity and the attachments list.

When the plan changes

If your organization moves to the Free plan, its app tokens are kept but Probara refuses them, so pipelines stop reporting and syncing. The card’s dialog says so and offers See plans. You can still see and revoke the tokens. They work again as soon as the organization is back on a paid plan, with nothing to set up again. Manual test runs are not affected.

Next step

• Reporter documentation: the options, linking and results and retries.
• Syncing feature files from Git: the sync’s keys, moves, archiving and read-only fields.
• CI integrations: how the Cucumber reporter compares with the CLI and the Playwright, Jest, Cypress and Vitest reporters.
• Reports API: the endpoint the reporter calls, for your own tools.