Cypress results, statuses and retries

Each cypress run reports into one Probara run: a new automated run by default, closed at the end, or an existing one. Every attempt of every test is a result in it, with Cypress’s own verdict. This page covers the run, the statuses, retries, what happens when results cannot be sent, and who gets the failures.

The run

The plugin creates the run, sends every result of every spec into it, and closes it when Cypress ends the run. These options describe the run it creates:

reporterOptions: {
  projectId: 'SHOP',
  run: {
    name: 'Nightly regression',
    description: 'Every night on main, against staging.',
    environment: 'staging',
    milestone: 'Sprint 12',
    plan: 'PLAN-2',
    configurations: [
      { group: 'Browser', name: 'Chrome' },
      { group: 'OS', name: 'Linux' },
    ],
    tags: ['nightly', 'e2e'],
  },
},

Option	Variable	What it sets
`run.name`	`PROBARA_RUN_NAME`	The name. Default: the CI build name (`CI #42`), else `Automated run <date> <time> UTC`
`run.description`	`PROBARA_RUN_DESCRIPTION`	What the run is for
`run.environment`	`PROBARA_ENVIRONMENT`	The environment, by name: created in the project when none has that name
`run.milestone`	`PROBARA_MILESTONE`	The milestone, by display id (`M-3`) or exact name
`run.plan`	`PROBARA_PLAN`	The test plan, by display id (`PLAN-2`) or name: the run starts with the plan’s cases
`run.configurations`	`PROBARA_CONFIGURATIONS`	Configuration values by group and name (`Browser=Chrome,OS=Linux`), each group once
`run.tags`	`PROBARA_RUN_TAGS`	Tags (`nightly,e2e`)

The same run from the environment of a CI step, with the project in the config:

PROBARA_RUN_NAME="Nightly regression" PROBARA_ENVIRONMENT=staging PROBARA_RUN_TAGS=nightly,e2e npx cypress run

A name longer than 200 characters is trimmed, and so is a tag longer than 80; tags beyond 50 are dropped, with one warning. An environment, milestone or plan longer than its limit turns reporting off, naming the field: it has to match a name in Probara, and a trimmed one would match none.

Without the plugin, each spec reports in a run of its own, because Cypress never tells the reporter when the whole run ended (registration).

Names or ULIDs

The environment, milestone and configurations also take ULIDs: run.environmentId, run.milestoneId and run.configurationUlids (PROBARA_ENVIRONMENT_ID, PROBARA_MILESTONE_ID, PROBARA_CONFIGURATION_ULIDS). Set one form of each, never both: both is a configuration problem that turns reporting off. Names are the form to use with an app token, which cannot look ULIDs up.

An unknown environment name creates the environment.
An unknown milestone, plan or configuration makes Probara refuse the report (422, naming the field): nothing is recorded, and the log says why. Check the names in Probara.

Run source

A run records where it came from: the branch, the commit and the CI build URL, read from the variables of GitHub Actions, GitLab CI, CircleCI, Azure Pipelines, Jenkins, Bitbucket Pipelines and Buildkite (what each CI fills in). Override a field with source or its variable, or send none:

Option	Variable	Default
`source.branch`	`PROBARA_BRANCH`	detected from the CI
`source.commit`	`PROBARA_COMMIT`	detected from the CI
`source.buildUrl`	`PROBARA_BUILD_URL`	detected from the CI
`source: false`	—	sends no source

An existing run

run.ulid (PROBARA_RUN_ULID) reports into a run that exists, such as one probara run create made for the machines of a pipeline (sharding) or one a person created in Probara (run selection). The reporter adds its results to that run and leaves it open:

$ PROBARA_RUN_ULID=01J9Z3K4M5N6P7Q8R9S0T1V2W3 npx cypress run
[probara] Sending 3 results of 3 tests (3 passed, 0 failed, 0 skipped, 0 blocked)
[probara] Recorded 3 results (2 new cases, 0 unmatched) in R-1 (open): https://app.probara.net/projects/SHOP/runs/R-1
[probara] The run R-1 of SHOP stays open: close it in Probara, or with probara run close --project SHOP --run-ulid 01J9Z3K4M5N6P7Q8R9S0T1V2W3

The options of a new run are then ignored, with one warning: the run keeps its own name, environment, milestone and configurations.

[probara] Ignored name: a reused run (run.ulid) keeps its own

To get a run’s ULID in Probara, open the run and choose Copy run ULID in Run actions (the menu in the run’s header).

Closing

The run	At the end of `cypress run`
Created by the reporter	Closed, whatever the exit code of the tests
Reused (`run.ulid`, `run.ulids`)	Left open: whoever created it closes it
Any, when a report failed after retries	Left open: the log names it, and the error says what to do

closeRun (PROBARA_CLOSE_RUN) decides for every run: true closes a reused run too, false leaves a created one open. closeRuns decides per project ({ SHOP: true, WEB: false }) when closeRun is not set. Close a run by hand with the Probara CLI:

npx @probara/cli run close --run-ulid 01J9Z3K4M5N6P7Q8R9S0T1V2W3

Statuses

Each attempt is sent with Cypress’s own verdict:

What Cypress reports	Status
A test that passed	`passed`
A test that failed: an assertion, an error, a timeout	`failed`
An attempt Cypress is about to run again (a retry)	`failed`
`it.skip`, or `this.skip()` in the test or a `beforeEach`	`skipped`
A test a failing `beforeEach` kept from running	`skipped`
A failing `beforeEach` or `before`, reported as `"before each" hook for "…"`	`failed`, under that name
A spec that throws as it loads (`An uncaught error was detected outside of a test`)	`failed`
A spec Cypress cannot parse	`failed`, as `Spec failed to run`

The reporter never sends blocked on its own: map a status to it when you need it (status mapping). A result carries the status, the duration and start time Cypress reported, and the error of a failed attempt in its notes, with its message and stack and without the terminal’s colors.

A hook that fails

A beforeEach or before belongs to no test of its own: what it did belongs to the test it runs for. When it fails, Cypress reports the failure under a test it names after the hook and the test it was running, and the reporter keeps that name, which is the one a JUnit import gives too.

A hook that fails once and passes on the retry is one test with two results: the failed attempt and the pass. A hook that fails on every attempt, with retries.runMode: 1:

describe('Checkout', () => {
  beforeEach(() => {
    throw new Error('the cart service is down');
  });

  it('never runs', () => {});
  it('never runs either', () => {});
});

describe('Profile', () => {
  it('still runs', () => {});
});

cypress/e2e/hooks.cy.js > Checkout never runs                             failed
cypress/e2e/hooks.cy.js > Checkout "before each" hook for "never runs"    failed
cypress/e2e/hooks.cy.js > Checkout never runs either                      skipped
cypress/e2e/hooks.cy.js > Profile still runs                              passed

Checkout never runs is the first attempt of the test, which failed because its hook did and which Cypress then retried.
"before each" hook for "never runs" is the failure Cypress could not get past.
Checkout never runs either never ran at all: the reporter reports every test of the suite that reported no attempt as skipped, which a JUnit report cannot, since such a test leaves no trace in it. It is not known to be broken, only known not to have run.
Profile still runs: a failing beforeEach stops the rest of its own describe, never the rest of the spec.

A spec Cypress cannot run

A spec whose module scope throws while Cypress loads it is one failed result, named as Cypress names it, and retried like any test:

cypress/e2e/throws.cy.js > An uncaught error was detected outside of a test

A spec Cypress cannot parse (a syntax error) runs no test at all. The plugin turns the failure Cypress counted into one failed result, so a broken spec never shows green:

cypress/e2e/broken.cy.js > Spec failed to run

Its notes read Cypress could not run this spec: 1 failure(s), no test ran, and its case sits in the suite of the spec alone. Without the plugin, there is no such result.

The exit code is Cypress’s own

Cypress ends a cypress run with the number of its own failed tests, and reporting changes nothing of it: a test that fails twice is one failed test, and a hook failure is a test of its own. What Probara recorded (every attempt, a mapped status, a filtered result, a run that could not be sent) never reaches the exit code.

Status mapping and filter

statusMapping sends a status as another; statusFilter sends no result with the given statuses, after the mapping. Both take passed, failed, skipped and blocked.

reporterOptions: { projectId: 'SHOP', statusMapping: { failed: 'blocked' } },

$ npx cypress run --spec cypress/e2e/cart.cy.js
[probara] Sending 5 results of 5 tests (3 passed, 0 failed, 1 skipped, 1 blocked)

With statusFilter: ['failed'] instead, the failed attempt is not sent at all, and the Sending line says how many were left out:

[probara] Sending 4 results of 4 tests (3 passed, 0 failed, 1 skipped, 0 blocked); 1 left out by statusFilter

As variables: PROBARA_STATUS_MAPPING=failed=blocked,skipped=passed and PROBARA_STATUS_FILTER=skipped,blocked. A value that is not a status, or a status mapped twice, is a configuration problem that turns reporting off. Cypress still fails the command: the mapping and the filter change only what Probara records.

Filtering skipped leaves out a lot: every it.skip and every test a failing hook kept from running.
blocked tells failures that are not the test’s own apart: map failed to blocked when a hook failure should not read as a failing test.

Retries and flaky tests

Every attempt is a result of its own, in the order the attempts ran, each with its own status, duration, error, steps and files. How many attempts Cypress makes is Cypress’s own setting, retries:

module.exports = defineConfig({
  e2e: {
    reporter: '@probara/cypress-reporter',
    reporterOptions: { projectId: 'SHOP' },
    setupNodeEvents(on, config) {
      return probaraNodeEvents(on, config);
    },
    retries: { runMode: 1 },
  },
});

$ npx cypress run --spec cypress/e2e/retry.cy.js
[probara] Sending 2 results of 1 test (1 passed, 1 failed, 0 skipped, 0 blocked)

What Probara shows. The run case keeps the last outcome (passed here), and every attempt stays in its history: a flaky test is a case whose history mixes failures and passes on the same run. The Sending line shows it: more results than tests.
The attempt number. From the second attempt on, the result carries an attempt parameter (2, 3, …), next to browser. Neither is part of the automation key.
No flaky flag. The attempts themselves are the evidence. probara import junit keeps only the last attempt of a retried test, so a suite that moves from the JUnit import to the reporter gains the earlier attempts in the history of each flaky case.
Each attempt starts empty. What the probara.* helpers said belongs to the attempt that said it: the steps, files and metadata of a retry are its own, never inherited.
Screenshots of a retry. Each failed attempt gets the screenshot Cypress took for it.
Cypress’s own summary counts each test once, by its last attempt; Probara counts every attempt.

To send only the last attempt, do not retry in Cypress: no option of the reporter drops attempts.

Results file

When a report cannot be sent (Probara unreachable, a refused token, a report that failed after its retries), its results are lost unless you keep them. Set a results file, and the reporter writes what it could not send to it; probara import results sends it later, into the same runs.

reporterOptions: { projectId: 'SHOP', resultsFile: 'results/probara-results.json' },

Or PROBARA_RESULTS_FILE=results/probara-results.json, relative to the directory cypress run was started in.

$ npx cypress run
[probara] Sending 1 result of 1 test (1 passed, 0 failed, 0 skipped, 0 blocked)
[probara] 1 result was not sent: Probara answered 403 forbidden: <message from Probara>. No run was created or updated
[probara] Wrote the 1 result that was not sent to /work/shop/results/probara-results.json: send them with probara import results /work/shop/results/probara-results.json

Then, from any job that holds the token, send whatever was left:

npx @probara/cli import results 'results/probara-results*.json'

Quote the glob: probara expands it, and it matches the file and its siblings.

Only when needed. The file is written only when something was not sent: when every result went through, there is no file.
Into the same runs. The file names the runs its results belong to, the settings that sent them (project, run, rootDir, createMissingCases) and each result as the reporter built it, so it lands in the same cases from any machine. Never the token.
Never over an earlier file. When a file is already at that path (another machine’s, or an earlier run’s), the reporter writes to the first free sibling: probara-results-2.json, then -3, and so on.
Statuses before the mapping. The file keeps each status before statusMapping, which applies again when the file is sent.

What travels with the file

Files	In the results file
`probara.attach()` bodies and files, the console output	Copied next to it, in `probara-results-attachments/`, and named relative to the file
Screenshots and videos	Named by their absolute path, not copied

Move or download the file and its attachments folder together, and import it from there: a CI artifact another job downloads keeps those attachments. A machine that never had the screenshots and videos cannot send them.

Reporting off: keep everything

With reporting off (PROBARA_ENABLED=false, or no token), the reporter writes every result to the file. Run the tests where there is no token, and send the file from a job that holds it:

PROBARA_ENABLED=false PROBARA_PROJECT=SHOP PROBARA_RESULTS_FILE=results/probara-results.json npx cypress run
npx @probara/cli import results 'results/probara-results*.json'

The same file shows what would be sent without sending it (check what would be sent). The import (several files and globs, --dry-run, exit codes) is in probara import results.

Assign failed results

assignFailedTo names the people who look into failures. Every report asks Probara to assign each run case it leaves failed, and that has no assignee yet, to one of them, in turn, so a failing nightly run lands on someone’s list without anyone triaging it first.

reporterOptions: {
  projectId: 'SHOP',
  assignFailedTo: ['ana@example.com', 'bo@example.com'],
},

Or from the environment: PROBARA_ASSIGN_FAILED_TO=ana@example.com,bo@example.com.

Rule	Why
Only run cases a report leaves `failed`	Passed, skipped and blocked results need no one
Only run cases without an assignee	An existing assignee is never replaced: someone already owns it
Only active members who can run tests in the project	A case cannot be assigned to someone who cannot work on it
The listed members take turns, in the order you list them	Failures are spread, not piled on the first name
An email that matches no such member is skipped	The results are still recorded; Probara’s warning counts them

The last attempt decides. A flaky test that passes on its retry leaves its case passed, so it is never assigned, however many of its attempts failed.
After statusMapping. With statusMapping: { failed: 'blocked' }, nothing is failed, so nothing is assigned.
The warning never names the emails, only how many were skipped: [probara] Probara warned: <message from Probara>.
The list is trimmed and deduplicated. More than 20 emails, or a value that is not an email, turns reporting off, naming assignFailedTo.