# Reporting from CI with Cucumber

> Create the app token the Probara Cucumber reporter signs in with, add the reporter to your cucumber.js, sync your .feature files, and see what it sends.

The **Cucumber** app sends your cucumber-js results to Probara from any CI pipeline, with the Probara
reporter for Cucumber. Each run becomes an automated test run with its results: every scenario and
step, the test cases each scenario links to, and the files it attached. The same app syncs your
`.feature` files, so Git keeps their scenarios as the cases in Probara.

The reporter signs in with an **app token**: a token that belongs to the organization, not to a
person. It uses no seat, and it keeps working when the person who created it leaves. This guide covers
the card in Probara and the setup. The
[reporter's documentation](https://docs.probara.net/en/developers/ci-integrations/cucumber/) covers every
[reporter option](https://docs.probara.net/en/developers/ci-integrations/cucumber/configuration/), the steps and attachments, and
the rules that link a scenario to its case.

## Before you start

| You need                                                                              | Why                                                                                                                                               |
| ------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- |
| A paid plan (**Pro** or **Enterprise**, or any trial)                                 | Reporting from CI and the feature sync are not part of the Free plan. On Free, its dialog shows **See plans** instead of **Create an app token**. |
| A Probara **owner** or **admin**                                                      | Only owners and admins can create and manage app tokens. Other members see the card without its tokens.                                           |
| A cucumber-js project (`@cucumber/cucumber` 13.3 or later, on Node.js 22.12 or later) | The reporter runs with your scenarios, in the same `npx cucumber-js` command.                                                                     |

## Set it up

1. In **Workspace › Integrations**, select the **Cucumber** card. The dialog lists what the app can
   do and links to this guide.
2. Open the **Access tokens** tab and select **Create an app token**.
3. Enter a **Token name** and select **Create token**. Name it after what uses it: the repository or
   the pipeline, so you know which one to revoke later.
4. Select **Copy token** in **Copy your new token**. This is the only time Probara shows it: store it
   in your CI as a secret named `PROBARA_API_TOKEN` before you select **Done**.
5. Install the reporter next to cucumber-js:

   ```bash
   npm install --save-dev @cucumber/cucumber @probara/cucumber-reporter
   ```

6. Add the formatter to `format` in `cucumber.js`, and replace `<PROJECT_CODE>` with your project's
   Project ID, such as `SHOP`:

   ```js
   // cucumber.js
   module.exports = {
     default: {
       format: ['@probara/cucumber-reporter'],
       formatOptions: { probara: { projectId: '<PROJECT_CODE>' } },
     },
   };
   ```

   The reporter replaces cucumber-js's progress output. To keep it, give the reporter a file path too:
   `format: ['progress', '@probara/cucumber-reporter:reports/probara.txt']`.

7. Pass the secret to your tests and run them. For example, in GitHub Actions:

   ```yaml
   - name: Run Cucumber tests
     run: npx cucumber-js
     env:
       PROBARA_API_TOKEN: ${{ secrets.PROBARA_API_TOKEN }}
   ```

The token never goes in `cucumber.js`: the reporter reads it from the environment. The card on the
Integrations page shows how many active tokens the app has.

## Sync the feature files

Results land on cases, so sync the `.feature` files first, on your default branch, with the same token:

```yaml
- name: Sync the feature files to Probara
  run: npx @probara/cli sync gherkin "features/**/*.feature"
  env:
    PROBARA_API_TOKEN: ${{ secrets.PROBARA_API_TOKEN }}
    PROBARA_PROJECT: <PROJECT_CODE>
```

The [Syncing feature files from Git](https://docs.probara.net/en/guides/syncing-feature-files-from-git/) guide covers the sync:
its keys, moves and bindings, archiving, and which fields of a synced case stay read-only. The
[CI providers](https://docs.probara.net/en/developers/ci-integrations/cucumber/ci-providers/) page has complete pipelines that
sync and run the tests.

## What the reporter sends

| What           | How Probara records it                                                                                                                                                                                                              |
| -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Results        | One result per attempt: a scenario retried with `--retry` sends each attempt, in order. Each row of a Scenario Outline is a result of its own, recorded on the run case with the row's values. The run case keeps the last outcome. |
| Steps          | Each step with its keyword (`Given`, `When`, `Then`, `And`, `But` or `*`), status, duration, error, and its Doc String or Data Table. Background steps sit under one `Background` step.                                             |
| Statuses       | A step that failed, or matches more than one definition, is failed. A step that is skipped, pending or undefined is skipped. A scenario fails when a step or a `Before` or `After` hook fails.                                      |
| Attachments    | The files a scenario attaches with `this.attach`, `this.log` and `this.link`: on its result, or on the step that attached them.                                                                                                     |
| Links          | A scenario tagged `@probara:SHOP-12` reports to the case with that display ID, whatever its key. Otherwise its key links it to its synced case.                                                                                     |
| New test cases | A scenario that matches no case creates one, named after the scenario, unless `createMissingCases` is off. Its key is the one `probara sync gherkin` builds, so the next sync finds it.                                             |
| Existing cases | An archived case a scenario reports to is restored. A case found by its automation key moves to the suite of its scenario's feature file; a case your Git sync manages, or one linked by its ID, keeps its suite.                   |
| Runs           | One automated run per cucumber-js run, named after the CI build, closed when every result is in.                                                                                                                                    |

A reporting problem never fails your tests. An option under `probara` that the reporter does not know
is ignored, with a warning in the log that names it.

## Same cases as the Cucumber JSON import

If you already send the Cucumber JSON report with `probara import cucumber`, the reporter finds the same
cases: its keys are the keys of the sync and of the import, so switching keeps each scenario on its case
and its history. Two differences are worth knowing. The reporter sends the results at the end of the run, and
it uploads the files its steps attach, which the import does not send. Remove the import step once the
reporter runs, so that the same results are not sent twice.

## Other tools that write Cucumber JSON

Cucumber-JVM, playwright-bdd and other tools write Cucumber JSON reports. Send those reports after the
run with `probara import cucumber`, which the [CLI page](https://docs.probara.net/en/developers/cli/#import-cucumber-results)
describes. The reporter is for cucumber-js.

## Manage tokens

Under **App tokens**, the **Access tokens** tab lists each active token with who created it (or
**Created by a removed user**), when, and when it was last used (**Never** until its first report).

**Revoke** a token, and confirm, when it may have leaked or when someone who could read it leaves your
team: pipelines that use it stop reporting at once, and it cannot be undone. Create a new token for them
first. Because an app token outlives the person who created it, rotating it is your call: Probara never
revokes it on its own.

## What an app token can do

An app token can only do what the reporter and the sync do: create automated runs, send reports (which
can create the test cases and suites they name), add attachments to their results, close runs, and sync
feature files (which create, update and archive synced cases). Every other call answers
`403 forbidden`. The [exact calls](https://docs.probara.net/en/developers/authentication/#allowed-calls) are listed in the
authentication reference. For scripts and other API clients, use a personal
[API token](https://docs.probara.net/en/developers/authentication/).

Runs, activity entries and attachments that an app token created show the app's logo and name beside
the token name, for example **Cucumber · e2e-ci**, in the runs list, the run and milestone summaries,
[Workspace activity](https://docs.probara.net/en/guides/workspace/activity/) and the attachments list.

## When the plan changes

If your organization moves to the Free plan, its app tokens are kept but Probara refuses them, so
pipelines stop reporting and syncing. The card's dialog says so and offers **See plans**. You can still
see and revoke the tokens. They work again as soon as the organization is back on a paid plan, with
nothing to set up again. Manual test runs are not affected.

## Next step

- [Reporter documentation](https://docs.probara.net/en/developers/ci-integrations/cucumber/): the
  [options](https://docs.probara.net/en/developers/ci-integrations/cucumber/configuration/),
  [linking](https://docs.probara.net/en/developers/ci-integrations/cucumber/linking/) and
  [results and retries](https://docs.probara.net/en/developers/ci-integrations/cucumber/results-and-retries/).
- [Syncing feature files from Git](https://docs.probara.net/en/guides/syncing-feature-files-from-git/): the sync's keys,
  moves, archiving and read-only fields.
- [CI integrations](https://docs.probara.net/en/developers/ci-integrations/): how the Cucumber reporter compares with the
  [CLI](https://docs.probara.net/en/developers/cli/) and the Playwright, Jest, Cypress and Vitest reporters.
- [Reports API](https://docs.probara.net/en/developers/api-reference/#reports): the endpoint the reporter calls, for your own
  tools.