Cypress reporter

@probara/cypress-reporter is a Cypress reporter that sends every test result of a cypress run to Probara: each attempt with its status, duration and error, linked to its test case, with the screenshot of each failure and the steps, attachments and links your tests add. Cases that are missing are created, and the run is closed once everything is in.

It is open source, published on npm at 0.1.0, and lives in the probara-sdk repository. Before 1.0, a minor version may bring breaking changes: pin the version (upgrading).

What you get

One result per attempt, Cypress retries included, with Cypress’s own verdict: it.skip and this.skip() are skipped, and every test a failing hook kept from running is skipped too.
The same automation keys as probara import junit on the JUnit XML of Cypress’s junit reporter or of cypress-junit, so switching from the JUnit import keeps every case and its history.
Screenshots attached to the exact attempt Cypress took them for, and the video of a spec on its failed results with attachVideos.
Linking by a case id in a test or describe title (SHOP-12 adds an item) or with probara.id(), and the browser as a browser parameter of every result, never in the key.
Test helpers that run where the spec runs: probara.step(), probara.attach(), probara.link(), probara.issue(), probara.parameters(), and the title, suite, tags and fields of a new case.
Console output of each test as attachments, and run selection: run only the tests of a run created in Probara.
Runs with a name, environment, milestone, test plan, configurations and tags; one run for every machine of a split pipeline.
It never breaks your test run. A reporting problem is logged, never thrown, and cypress run still ends with the number of its own failed tests. What could not be sent can be kept in a file and sent later.

Before you start

You need	Why
Node.js 22.12 or later	The package’s `engines` range, for the Node.js that starts `cypress`.
Cypress 15.10.0 or later	The peer range: 15.10.0 is the first Cypress with `Cypress.expose()`, which hands the browser its settings. Tested on 16.1.1.
`cypress run`	Cypress builds a reporter only in `cypress run`: `cypress open` reports nothing.
A paid plan (Pro or Enterprise, or any trial)	Reporting from CI is not part of the Free plan. On Free, Probara answers `403 forbidden`.
A Probara owner or admin	Only owners and admins create and revoke app tokens.
The Project ID to report into	Such as `SHOP`: a capital letter, then capital letters or digits.

1. Create an app token

The reporter signs in with an app token: a token that belongs to your organization, not to a person. It uses no seat and keeps working when the person who created it leaves.

In Probara, open Workspace › Integrations and select the Cypress card.
Open the Access tokens tab and select Create an app token.
Enter a Token name after what uses it (the repository or the pipeline) and select Create token.
Select Copy token in Copy your new token. It is shown once: store it right away as a CI secret named PROBARA_API_TOKEN.

The Cypress integration guide covers the card, managing and revoking tokens, and what Probara shows for runs an app token creates.

2. Install the reporter

npm install --save-dev @probara/cypress-reporter

3. Register the reporter and its plugin

Set the reporter as the e2e reporter, with your Project ID, and return its plugin from setupNodeEvents:

const { defineConfig } = require('cypress');
const { probaraNodeEvents } = require('@probara/cypress-reporter/setup');

module.exports = defineConfig({
  e2e: {
    reporter: '@probara/cypress-reporter',
    reporterOptions: { projectId: 'SHOP' },
    setupNodeEvents(on, config) {
      return probaraNodeEvents(on, config);
    },
  },
});

The reporter turns Cypress’s test events into results. The plugin owns the Probara run (one for the whole cypress run, closed at the end), collects the screenshots and the video, and carries what the probara.* helpers say. It reads the same reporterOptions, so it needs no settings of its own. If setupNodeEvents already registers other plugins, see with other plugins.

You can leave projectId out and set PROBARA_PROJECT in the environment instead. The token never goes in the config: the config is committed, and the reporter reads the token from PROBARA_API_TOKEN.

Cypress runs one reporter: with this one, the per-test lines of its default spec reporter no longer print, while Cypress’s own tables of each spec and of the run still do. To keep the spec lines too, list both reporters in cypress-multi-reporters.

4. Load the support file

Import the support file once, in the Cypress support file:

import '@probara/cypress-reporter/support';

It runs in the browser, next to your specs, and turns on the probara.* helpers (a global: specs call them with no import), captureOutput and run selection.

5. Run your tests with the token

Give the step that runs cypress run the secret. In GitHub Actions:

- name: Run Cypress tests
  run: npx cypress run
  env:
    PROBARA_API_TOKEN: ${{ secrets.PROBARA_API_TOKEN }}

There is no separate upload step: the plugin sends the results when the run ends. CI providers has complete pipelines for GitHub Actions, GitLab CI, Jenkins, CircleCI, Azure Pipelines, Bitbucket Pipelines and Buildkite.

What you see

Without a token and a project the reporter stays off and quiet, so local runs send nothing. In CI it logs on stdout, in [probara] lines among Cypress’s own output, and ends with the link to the run:

$ npx cypress run --spec cypress/e2e/cart.cy.js
[probara] Sending 5 results of 4 tests (3 passed, 2 failed, 0 skipped, 0 blocked)
[probara] Recorded 5 results (2 new cases, 0 unmatched) in R-1 (closed): https://app.probara.net/projects/SHOP/runs/R-1

In Probara, the run appears in the project’s run list as an automated run, named after the CI build (such as Cypress #42), with its branch, commit and build URL. It shows the Cypress app and the token’s name (such as Cypress · shop-e2e) instead of a person.

Each registration, and what it turns on

What is registered	What you get
Reporter, plugin and support file	Everything: one run for the whole `cypress run`, closed at the end, the screenshots of failed attempts, the video, the helpers
Reporter and plugin, no support file	Results, screenshots and the video. Every `probara.*` call does nothing, `captureOutput` captures nothing, run selection skips nothing
Reporter alone, no plugin	Every result is sent, but in one run per spec, with no screenshot, no video and no helper. One warning says what is missing
Plugin alone, no reporter	Nothing is reported: Cypress never builds the reporter

How Cypress maps to Probara

Cypress	In Probara
A test	A test case, matched by its automation key or linked by a case id
Each attempt (Cypress `retries` too)	A result in the run, with its status, duration, start time and error
`it.skip`, `this.skip()`, a test a failing hook kept from running	A skipped result
A failing `beforeEach` or `before`	One failed result named as Cypress names it, plus the tests it stopped
`probara.step()` calls	The steps of the result, nested, with their status, duration and error
`probara.attach()`, `captureOutput`, the screenshot of a failure	Files of the result, or of the step they were attached in
`probara.link()`, `probara.issue()`	Links of the result
Other `probara.*` calls in the test	The title, suite, tags, fields and steps of a new case; result parameters
A spec Cypress cannot parse, or one that throws as it loads	One failed result, as Cypress names it, so it never shows green
The `cypress run` command	One automated run, named after the CI build, closed at the end

Same cases as the JUnit import

The reporter gives each test the automation key probara import junit gives it on the report of Cypress’s built-in junit reporter or of cypress-junit written with their default options: the spec file relative to the project root, then > between spaces, then the describe titles and the it title joined by spaces, without the case ids. For these two specs:

describe('Cart', () => {
  it('SHOP-12 adds an item', () => {
    cy.get('.add').click();
  });

  it('removes an item', () => {
    cy.get('.remove').click();
  });
});

describe('Login', () => {
  it('logs in with a valid password', () => {
    cy.get('#submit').click();
  });
});

the reporter sends:

[
  {
    "automationKey": "cypress/e2e/cart.cy.js > Cart adds an item",
    "caseDisplayId": "SHOP-12",
    "status": "passed"
  },
  { "automationKey": "cypress/e2e/cart.cy.js > Cart removes an item", "status": "passed" },
  {
    "automationKey": "cypress/e2e/login.cy.js > Login logs in with a valid password",
    "status": "passed"
  }
]

If your JUnit reports were written without the file attribute of their root suite, set keyIncludesFile: false: the keys then leave the spec file out, like those reports. Either way a test keeps its case and its history when you switch (migrating from the JUnit import).

Reporting never breaks your test run

The reporter never throws into Cypress and never changes its exit code: a failed test fails the command, a reporting problem does not. Every problem (a wrong setting, Probara unreachable, a refused token) is logged, never with the token, and the tests keep their outcome. probara.* calls never fail a test either: a call Probara cannot use is left out with a warning. There is no option to fail the command on a reporting error; to keep what could not be sent, set a results file.

Reference

Page	What it covers
Configuration	The three registrations, every option, its variable, type and default; network; interactive mode
Linking tests to cases	Case ids in titles and describes, `probara.id()`, automation keys, `it.only`
Metadata and links	Title, suite, comment, tags, fields, parameters, ignoring an attempt; links and issues
Steps and attachments	`probara.step()`, `probara.attach()`, screenshots, the video, console output, limits
Results and retries	Run options, statuses, hooks and broken specs, retries, the results file, assigning failures
Run selection	`runCasesOnly`: run only the tests of a Probara run
Sharding and projects	One run for every machine; several Probara projects
CI providers	GitHub Actions, GitLab CI, Jenkins, CircleCI, Azure Pipelines, Bitbucket Pipelines, Buildkite
Troubleshooting	Problems and their solutions, every line the reporter logs, debugging
Migrating	From the JUnit import, Qase, Allure Cypress and TestRail’s `trcli`; upgrading