# Cypress reporter

> Send every Cypress test result to Probara from any CI with @probara/cypress-reporter — install, register the reporter, its plugin and its support file, and see your runs.

`@probara/cypress-reporter` is a [Cypress](https://www.cypress.io) reporter that sends every test
result of a `cypress run` to Probara: each attempt with its status, duration and error, linked to its
test case, with the screenshot of each failure and the steps, attachments and links your tests add.
Cases that are missing are created, and the run is closed once everything is in.

It is open source, published on [npm](https://www.npmjs.com/package/@probara/cypress-reporter) at
`0.1.0`, and lives in the
[probara-sdk repository](https://github.com/cynch-me/probara-sdk/tree/main/packages/cypress-reporter).
Before 1.0, a minor version may bring breaking changes: pin the version
([upgrading](https://docs.probara.net/en/developers/ci-integrations/cypress/migrating/#upgrading-the-reporter)).

## What you get

- **One result per attempt**, Cypress `retries` included, with Cypress's own verdict: `it.skip` and
  `this.skip()` are skipped, and every test a failing hook kept from running is skipped too.
- **The same automation keys as `probara import junit`** on the JUnit XML of Cypress's `junit`
  reporter or of cypress-junit, so switching from the JUnit import keeps every case and its history.
- **Screenshots attached to the exact attempt** Cypress took them for, and the video of a spec on
  its failed results with `attachVideos`.
- **Linking** by a case id in a test or `describe` title (`SHOP-12 adds an item`) or with
  `probara.id()`, and the browser as a `browser` parameter of every result, never in the key.
- **Test helpers** that run where the spec runs: `probara.step()`, `probara.attach()`,
  `probara.link()`, `probara.issue()`, `probara.parameters()`, and the title, suite, tags and fields
  of a new case.
- **Console output** of each test as attachments, and **run selection**: run only the tests of a
  run created in Probara.
- **Runs** with a name, environment, milestone, test plan, configurations and tags; one run for
  every machine of a split pipeline.
- **It never breaks your test run.** A reporting problem is logged, never thrown, and `cypress run`
  still ends with the number of its own failed tests. What could not be sent can be kept in a file
  and sent later.

## Before you start

| You need                                              | Why                                                                                                                           |
| ----------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- |
| Node.js 22.12 or later                                | The package's `engines` range, for the Node.js that starts `cypress`.                                                         |
| Cypress 15.10.0 or later                              | The peer range: 15.10.0 is the first Cypress with `Cypress.expose()`, which hands the browser its settings. Tested on 16.1.1. |
| `cypress run`                                         | Cypress builds a reporter only in `cypress run`: `cypress open` reports nothing.                                              |
| A paid plan (**Pro** or **Enterprise**, or any trial) | Reporting from CI is not part of the Free plan. On Free, Probara answers `403 forbidden`.                                     |
| A Probara **owner** or **admin**                      | Only owners and admins create and revoke app tokens.                                                                          |
| The **Project ID** to report into                     | Such as `SHOP`: a capital letter, then capital letters or digits.                                                             |

## 1. Create an app token

The reporter signs in with an [app token](https://docs.probara.net/en/developers/authentication/#app-tokens): a token that
belongs to your organization, not to a person. It uses no seat and keeps working when the person
who created it leaves.

1. In Probara, open **Workspace › Integrations** and select the **Cypress** card.
2. Open the **Access tokens** tab and select **Create an app token**.
3. Enter a **Token name** after what uses it (the repository or the pipeline) and select
   **Create token**.
4. Select **Copy token** in **Copy your new token**. It is shown once: store it right away as a CI
   secret named `PROBARA_API_TOKEN`.

The [Cypress integration guide](https://docs.probara.net/en/guides/integrations/cypress/) covers the card, managing and
revoking tokens, and what Probara shows for runs an app token creates.

## 2. Install the reporter

```bash
npm install --save-dev @probara/cypress-reporter
```

## 3. Register the reporter and its plugin

Set the reporter as the `e2e` reporter, with your Project ID, and return its plugin from
`setupNodeEvents`:

```js
// cypress.config.js
const { defineConfig } = require('cypress');
const { probaraNodeEvents } = require('@probara/cypress-reporter/setup');

module.exports = defineConfig({
  e2e: {
    reporter: '@probara/cypress-reporter',
    reporterOptions: { projectId: 'SHOP' },
    setupNodeEvents(on, config) {
      return probaraNodeEvents(on, config);
    },
  },
});
```

The reporter turns Cypress's test events into results. The plugin owns the Probara run (one for the
whole `cypress run`, closed at the end), collects the screenshots and the video, and carries what
the `probara.*` helpers say. It reads the same `reporterOptions`, so it needs no settings of its
own. If `setupNodeEvents` already registers other plugins, see
[with other plugins](https://docs.probara.net/en/developers/ci-integrations/cypress/configuration/#with-other-plugins).

You can leave `projectId` out and set `PROBARA_PROJECT` in the environment instead. The token never
goes in the config: the config is committed, and the reporter reads the token from
`PROBARA_API_TOKEN`.

Cypress runs one reporter: with this one, the per-test lines of its default `spec` reporter no
longer print, while Cypress's own tables of each spec and of the run still do. To keep the `spec`
lines too, list both reporters in
[`cypress-multi-reporters`](https://docs.probara.net/en/developers/ci-integrations/cypress/configuration/#with-cypress-multi-reporters).

## 4. Load the support file

Import the support file once, in the Cypress support file:

```js
// cypress/support/e2e.js
import '@probara/cypress-reporter/support';
```

It runs in the browser, next to your specs, and turns on the `probara.*` helpers (a global: specs
call them with no import), [`captureOutput`](https://docs.probara.net/en/developers/ci-integrations/cypress/steps-and-attachments/#console-output)
and [run selection](https://docs.probara.net/en/developers/ci-integrations/cypress/run-selection/).

## 5. Run your tests with the token

Give the step that runs `cypress run` the secret. In GitHub Actions:

```yaml
- name: Run Cypress tests
  run: npx cypress run
  env:
    PROBARA_API_TOKEN: ${{ secrets.PROBARA_API_TOKEN }}
```

There is no separate upload step: the plugin sends the results when the run ends.
[CI providers](https://docs.probara.net/en/developers/ci-integrations/cypress/ci-providers/) has complete pipelines for
GitHub Actions, GitLab CI, Jenkins, CircleCI, Azure Pipelines, Bitbucket Pipelines and Buildkite.

## What you see

Without a token and a project the reporter stays off and quiet, so local runs send nothing. In CI it
logs on stdout, in `[probara]` lines among Cypress's own output, and ends with the link to the run:

```text
$ npx cypress run --spec cypress/e2e/cart.cy.js
[probara] Sending 5 results of 4 tests (3 passed, 2 failed, 0 skipped, 0 blocked)
[probara] Recorded 5 results (2 new cases, 0 unmatched) in R-1 (closed): https://app.probara.net/projects/SHOP/runs/R-1
```

In Probara, the run appears in the project's run list as an automated run, named after the CI build
(such as `Cypress #42`), with its branch, commit and build URL. It shows the **Cypress** app and the
token's name (such as **Cypress · shop-e2e**) instead of a person.

## Each registration, and what it turns on

| What is registered                   | What you get                                                                                                                           |
| ------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------- |
| Reporter, plugin and support file    | Everything: one run for the whole `cypress run`, closed at the end, the screenshots of failed attempts, the video, the helpers         |
| Reporter and plugin, no support file | Results, screenshots and the video. Every `probara.*` call does nothing, `captureOutput` captures nothing, run selection skips nothing |
| Reporter alone, no plugin            | Every result is sent, but in **one run per spec**, with no screenshot, no video and no helper. One warning says what is missing        |
| Plugin alone, no reporter            | Nothing is reported: Cypress never builds the reporter                                                                                 |

## How Cypress maps to Probara

| Cypress                                                           | In Probara                                                                |
| ----------------------------------------------------------------- | ------------------------------------------------------------------------- |
| A test                                                            | A test case, matched by its automation key or linked by a case id         |
| Each attempt (Cypress `retries` too)                              | A result in the run, with its status, duration, start time and error      |
| `it.skip`, `this.skip()`, a test a failing hook kept from running | A skipped result                                                          |
| A failing `beforeEach` or `before`                                | One failed result named as Cypress names it, plus the tests it stopped    |
| `probara.step()` calls                                            | The steps of the result, nested, with their status, duration and error    |
| `probara.attach()`, `captureOutput`, the screenshot of a failure  | Files of the result, or of the step they were attached in                 |
| `probara.link()`, `probara.issue()`                               | Links of the result                                                       |
| Other `probara.*` calls in the test                               | The title, suite, tags, fields and steps of a new case; result parameters |
| A spec Cypress cannot parse, or one that throws as it loads       | One failed result, as Cypress names it, so it never shows green           |
| The `cypress run` command                                         | One automated run, named after the CI build, closed at the end            |

## Same cases as the JUnit import

The reporter gives each test the automation key `probara import junit` gives it on the report of
Cypress's built-in `junit` reporter or of [cypress-junit](https://www.npmjs.com/package/cypress-junit)
written with their default options: the spec file relative to the project root, then `>` between
spaces, then the `describe` titles and the `it` title joined by spaces, without the case ids. For
these two specs:

```js
// cypress/e2e/cart.cy.js
describe('Cart', () => {
  it('SHOP-12 adds an item', () => {
    cy.get('.add').click();
  });

  it('removes an item', () => {
    cy.get('.remove').click();
  });
});
```

```js
// cypress/e2e/login.cy.js
describe('Login', () => {
  it('logs in with a valid password', () => {
    cy.get('#submit').click();
  });
});
```

the reporter sends:

```json
[
  {
    "automationKey": "cypress/e2e/cart.cy.js > Cart adds an item",
    "caseDisplayId": "SHOP-12",
    "status": "passed"
  },
  { "automationKey": "cypress/e2e/cart.cy.js > Cart removes an item", "status": "passed" },
  {
    "automationKey": "cypress/e2e/login.cy.js > Login logs in with a valid password",
    "status": "passed"
  }
]
```

If your JUnit reports were written without the `file` attribute of their root suite, set
`keyIncludesFile: false`: the keys then leave the spec file out, like those reports. Either way a
test keeps its case and its history when you switch
([migrating from the JUnit import](https://docs.probara.net/en/developers/ci-integrations/cypress/migrating/#from-the-junit-import)).

## Reporting never breaks your test run

The reporter never throws into Cypress and never changes its exit code: a failed test fails the
command, a reporting problem does not. Every problem (a wrong setting, Probara unreachable, a
refused token) is logged, never with the token, and the tests keep their outcome. `probara.*` calls
never fail a test either: a call Probara cannot use is left out with a warning. There is no option to
fail the command on a reporting error; to keep what could not be sent, set a
[results file](https://docs.probara.net/en/developers/ci-integrations/cypress/results-and-retries/#results-file).

## Reference

| Page                                                                                   | What it covers                                                                                   |
| -------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ |
| [Configuration](https://docs.probara.net/en/developers/ci-integrations/cypress/configuration/)                 | The three registrations, every option, its variable, type and default; network; interactive mode |
| [Linking tests to cases](https://docs.probara.net/en/developers/ci-integrations/cypress/linking/)              | Case ids in titles and describes, `probara.id()`, automation keys, `it.only`                     |
| [Metadata and links](https://docs.probara.net/en/developers/ci-integrations/cypress/metadata/)                 | Title, suite, comment, tags, fields, parameters, ignoring an attempt; links and issues           |
| [Steps and attachments](https://docs.probara.net/en/developers/ci-integrations/cypress/steps-and-attachments/) | `probara.step()`, `probara.attach()`, screenshots, the video, console output, limits             |
| [Results and retries](https://docs.probara.net/en/developers/ci-integrations/cypress/results-and-retries/)     | Run options, statuses, hooks and broken specs, retries, the results file, assigning failures     |
| [Run selection](https://docs.probara.net/en/developers/ci-integrations/cypress/run-selection/)                 | `runCasesOnly`: run only the tests of a Probara run                                              |
| [Sharding and projects](https://docs.probara.net/en/developers/ci-integrations/cypress/sharding/)              | One run for every machine; several Probara projects                                              |
| [CI providers](https://docs.probara.net/en/developers/ci-integrations/cypress/ci-providers/)                   | GitHub Actions, GitLab CI, Jenkins, CircleCI, Azure Pipelines, Bitbucket Pipelines, Buildkite    |
| [Troubleshooting](https://docs.probara.net/en/developers/ci-integrations/cypress/troubleshooting/)             | Problems and their solutions, every line the reporter logs, debugging                            |
| [Migrating](https://docs.probara.net/en/developers/ci-integrations/cypress/migrating/)                         | From the JUnit import, Qase, Allure Cypress and TestRail's `trcli`; upgrading                    |

Related: [CI integrations overview](https://docs.probara.net/en/developers/ci-integrations/) ·
[Jest reporter](https://docs.probara.net/en/developers/ci-integrations/jest/) ·
[Playwright reporter](https://docs.probara.net/en/developers/ci-integrations/playwright/) ·
[Vitest reporter](https://docs.probara.net/en/developers/ci-integrations/vitest/) · [CLI](https://docs.probara.net/en/developers/cli/) ·
[Cypress integration guide](https://docs.probara.net/en/guides/integrations/cypress/) ·
[Source on GitHub](https://github.com/cynch-me/probara-sdk/tree/main/packages/cypress-reporter)
