# Cucumber reporter

> Send every Cucumber scenario and step to Probara at the end of the run with @probara/cucumber-reporter — install, configure and see your runs.

`@probara/cucumber-reporter` is a [Cucumber](https://cucumber.io) formatter for cucumber-js. It sends
every scenario of a run to Probara at the end of the run: each attempt with its steps, their keywords and
data, their errors and their attachments, keyed like the scenario's synced case. A scenario tagged
`@probara:SHOP-12` lands on that case. Cases that are missing are created, and the run is closed once
everything is in.

It is open source, published on [npm](https://www.npmjs.com/package/@probara/cucumber-reporter) at
`0.1.0`, and lives in the
[probara-sdk repository](https://github.com/cynch-me/probara-sdk/tree/main/packages/cucumber-reporter).
Before 1.0, a minor version may bring breaking changes: pin the version and read the
[changelog](https://github.com/cynch-me/probara-sdk/blob/main/packages/cucumber-reporter/CHANGELOG.md)
before you upgrade.

Sync your `.feature` files on the default branch first, so Git holds the cases. The reporter then
reports the tests. [Syncing feature files from Git](https://docs.probara.net/en/guides/syncing-feature-files-from-git/)
covers the sync.

## What you get

- **One result per attempt.** A scenario retried with `--retry` sends each attempt, in order. Each
  row of a Scenario Outline is a result of its own, recorded on the run case with the row's values.
- **Every step**, with its keyword (`Given`, `When`, `Then`, `And`, `But` or `*`), status, duration,
  error, and the Doc String or Data Table it had. Background steps sit under one `Background` step.
- **Attachments.** The files a step attaches with `this.attach`, `this.log` and `this.link` are
  uploaded with its result.
- **Linking** by the scenario's key, built the way `probara sync gherkin` builds it, or by a
  `@probara:<ID>` tag.
- **Runs** named after the CI build, closed when the run ends.
- **It never breaks your test run.** A reporting problem is logged on stderr. The exit code of the
  run stays cucumber-js's own.

## Before you start

| You need                                              | Why                                                                                       |
| ----------------------------------------------------- | ----------------------------------------------------------------------------------------- |
| Node.js 22.12 or later                                | The package's `engines` range.                                                            |
| `@cucumber/cucumber` 13.3 or later (13.3 is tested)   | The formatter loads from the same install as cucumber-js.                                 |
| A paid plan (**Pro** or **Enterprise**, or any trial) | Reporting from CI is not part of the Free plan. On Free, Probara answers `403 forbidden`. |
| A Probara **owner** or **admin**                      | Only owners and admins create and revoke app tokens.                                      |
| The **Project ID** to report into                     | Such as `SHOP`: a capital letter, then capital letters or digits.                         |

## 1. Create an app token

The reporter signs in with an [app token](https://docs.probara.net/en/developers/authentication/#app-tokens): a token that
belongs to your organization, not to a person. It uses no seat and keeps working when the person who
created it leaves.

1. In Probara, open **Workspace › Integrations** and select the **Cucumber** card.
2. Open the **Access tokens** tab and select **Create an app token**.
3. Enter a **Token name** after what uses it (the repository or the pipeline) and select
   **Create token**.
4. Select **Copy token**. It is shown once: store it right away as a CI secret named
   `PROBARA_API_TOKEN`.

The [Cucumber integration guide](https://docs.probara.net/en/guides/integrations/cucumber/) covers the card, managing and
revoking tokens, and what Probara shows for the runs an app token creates.

## 2. Install the reporter

Install it next to cucumber-js, in the same project:

```bash
npm install --save-dev @cucumber/cucumber @probara/cucumber-reporter
```

## 3. Add it to cucumber.js

Add the formatter to `format`, and your Project ID to `formatOptions`:

```js
// cucumber.js
module.exports = {
  default: {
    format: ['@probara/cucumber-reporter'],
    formatOptions: { probara: { projectId: 'SHOP' } },
  },
};
```

The formatter replaces cucumber-js's own progress output. To keep the progress output, give the
reporter a file path as well:
`format: ['progress', '@probara/cucumber-reporter:reports/probara.txt']`. The
[configuration page](https://docs.probara.net/en/developers/ci-integrations/cucumber/configuration/#where-the-output-goes)
explains why.

## 4. Run your tests with the token

Give the step that runs cucumber-js the secret. In GitHub Actions:

```yaml
- name: Run Cucumber tests
  run: npx cucumber-js
  env:
    PROBARA_API_TOKEN: ${{ secrets.PROBARA_API_TOKEN }}
```

There is no separate upload step: the reporter sends the results at the end of the run, and closes
the run then. [CI providers](https://docs.probara.net/en/developers/ci-integrations/cucumber/ci-providers/) has complete
pipelines for GitHub Actions, GitLab CI, Jenkins, CircleCI, Azure Pipelines, Bitbucket Pipelines and
Buildkite. They sync the feature files on the default branch before the tests.

## What you see

The reporter logs on stderr and prints nothing on stdout. When the run ends, it logs how many results it
sent, the run they went to, and, when it uploaded files, how many:

```text
$ npx cucumber-js
[probara] Sending 3 results of 3 tests (3 passed, 0 failed, 0 skipped, 0 blocked)
[probara] Recorded 3 results (2 new cases, 0 unmatched) in R-1 (closed): https://app.probara.net/projects/SHOP/runs/R-1
[probara] Attached 2 files to results (0 skipped, 0 failed)
```

In Probara, the run appears in the project's run list as an automated run, named after the CI build
(or `Automated run <date>` outside CI), with its branch, commit and build URL. It shows the **Cucumber**
app and the token's name (such as **Cucumber · e2e-ci**) instead of a person.

## How Cucumber maps to Probara

| Cucumber                                  | In Probara                                                                                         |
| ----------------------------------------- | -------------------------------------------------------------------------------------------------- |
| A scenario, once per attempt              | A result on the case with its automation key, or on the case its `@probara:<ID>` tag names         |
| A row of a Scenario Outline               | A result of its own, recorded on the run case with the row's values                                |
| A step                                    | A step of the result, with its keyword, status, duration, error and data                           |
| The Background steps                      | One `Background` step of the result, with them nested in it                                        |
| `this.attach`, `this.log` and `this.link` | Files of the result, or of the step that attached them                                             |
| A `Before` or `After` hook that fails     | The scenario fails, with the hook's error or `The hook failed`. The hook itself is not sent        |
| A scenario that matches no case           | A new case, named after the scenario, under the suite of `suiteUlid` (the project root by default) |
| The cucumber-js run                       | One automated run, named after the CI build, closed at the end                                     |

Hooks are not steps, so they never appear in the step list. Each hook's duration still counts in its
scenario's duration.

## Reporting never breaks your test run

The reporter never throws into cucumber-js and never changes its exit code: a failing scenario fails
the command, and a reporting problem does not. Every problem (a wrong setting, Probara unreachable, a
refused token) is logged on stderr, never with the token. To keep what could not be sent, set a
[results file](https://docs.probara.net/en/developers/ci-integrations/cucumber/results-and-retries/#results-file).

## Reference

| Page                                                                                    | What it covers                                                                                 |
| --------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- |
| [Configuration](https://docs.probara.net/en/developers/ci-integrations/cucumber/configuration/)                 | Every option, its variable, type and default; precedence; the output; network                  |
| [Linking test cases](https://docs.probara.net/en/developers/ci-integrations/cucumber/linking/)                  | The automation key, outlines, the `@probara:<ID>` tag and cases that do not exist yet          |
| [Steps and attachments](https://docs.probara.net/en/developers/ci-integrations/cucumber/steps-and-attachments/) | Step fields, Background, hooks, Doc Strings and Data Tables, and the files a scenario attaches |
| [Results and retries](https://docs.probara.net/en/developers/ci-integrations/cucumber/results-and-retries/)     | Statuses, retries, status mapping and filter, durations, the results file                      |
| [CI providers](https://docs.probara.net/en/developers/ci-integrations/cucumber/ci-providers/)                   | GitHub Actions, GitLab CI, Jenkins, CircleCI, Azure Pipelines, Bitbucket Pipelines, Buildkite  |
| [Troubleshooting](https://docs.probara.net/en/developers/ci-integrations/cucumber/troubleshooting/)             | Every message the reporter logs, and what to do about it                                       |
| [Migrating and upgrading](https://docs.probara.net/en/developers/ci-integrations/cucumber/migrating/)           | From `probara import cucumber`, from Qase's cucumber-js reporter, and upgrading                |

Related: [CI integrations overview](https://docs.probara.net/en/developers/ci-integrations/) ·
[CLI](https://docs.probara.net/en/developers/cli/) (`sync gherkin`, and `import cucumber` for other tools' JSON reports) ·
[Cucumber integration guide](https://docs.probara.net/en/guides/integrations/cucumber/) ·
[Syncing feature files from Git](https://docs.probara.net/en/guides/syncing-feature-files-from-git/) ·
[Source on GitHub](https://github.com/cynch-me/probara-sdk/tree/main/packages/cucumber-reporter)
