# Cypress results, statuses and retries

> The run each cypress run reports into, how attempts, hooks and broken specs become statuses, retries and flaky tests, the results file for what could not be sent, and assigning failed results.

Each `cypress run` reports into one Probara run: a new automated run by default, closed at the end,
or an existing one. Every attempt of every test is a result in it, with Cypress's own verdict. This
page covers the run, the statuses, retries, what happens when results cannot be sent, and who gets
the failures.

## The run

The plugin creates the run, sends every result of every spec into it, and closes it when Cypress
ends the run. These options describe the run it creates:

```js
// cypress.config.js
reporterOptions: {
  projectId: 'SHOP',
  run: {
    name: 'Nightly regression',
    description: 'Every night on main, against staging.',
    environment: 'staging',
    milestone: 'Sprint 12',
    plan: 'PLAN-2',
    configurations: [
      { group: 'Browser', name: 'Chrome' },
      { group: 'OS', name: 'Linux' },
    ],
    tags: ['nightly', 'e2e'],
  },
},
```

| Option               | Variable                  | What it sets                                                                            |
| -------------------- | ------------------------- | --------------------------------------------------------------------------------------- |
| `run.name`           | `PROBARA_RUN_NAME`        | The name. Default: the CI build name (`CI #42`), else `Automated run <date> <time> UTC` |
| `run.description`    | `PROBARA_RUN_DESCRIPTION` | What the run is for                                                                     |
| `run.environment`    | `PROBARA_ENVIRONMENT`     | The environment, by name: created in the project when none has that name                |
| `run.milestone`      | `PROBARA_MILESTONE`       | The milestone, by display id (`M-3`) or exact name                                      |
| `run.plan`           | `PROBARA_PLAN`            | The test plan, by display id (`PLAN-2`) or name: the run starts with the plan's cases   |
| `run.configurations` | `PROBARA_CONFIGURATIONS`  | Configuration values by group and name (`Browser=Chrome,OS=Linux`), each group once     |
| `run.tags`           | `PROBARA_RUN_TAGS`        | Tags (`nightly,e2e`)                                                                    |

The same run from the environment of a CI step, with the project in the config:

```bash
PROBARA_RUN_NAME="Nightly regression" PROBARA_ENVIRONMENT=staging PROBARA_RUN_TAGS=nightly,e2e npx cypress run
```

A name longer than 200 characters is trimmed, and so is a tag longer than 80; tags beyond 50 are
dropped, with one warning. An environment, milestone or plan longer than its limit turns reporting
off, naming the field: it has to match a name in Probara, and a trimmed one would match none.

Without the plugin, each spec reports in a run of its own, because Cypress never tells the reporter
when the whole run ended
([registration](https://docs.probara.net/en/developers/ci-integrations/cypress/configuration/#registration)).

### Names or ULIDs

The environment, milestone and configurations also take ULIDs: `run.environmentId`,
`run.milestoneId` and `run.configurationUlids` (`PROBARA_ENVIRONMENT_ID`, `PROBARA_MILESTONE_ID`,
`PROBARA_CONFIGURATION_ULIDS`). Set one form of each, never both: both is a configuration problem
that turns reporting off. Names are the form to use with an app token, which cannot look ULIDs up.

- An unknown environment **name** creates the environment.
- An unknown milestone, plan or configuration makes Probara refuse the report (`422`, naming the
  field): nothing is recorded, and the log says why. Check the names in Probara.

### Run source

A run records where it came from: the branch, the commit and the CI build URL, read from the
variables of GitHub Actions, GitLab CI, CircleCI, Azure Pipelines, Jenkins, Bitbucket Pipelines and
Buildkite ([what each CI fills in](https://docs.probara.net/en/developers/ci-integrations/cypress/ci-providers/)). Override a
field with `source` or its variable, or send none:

| Option            | Variable            | Default              |
| ----------------- | ------------------- | -------------------- |
| `source.branch`   | `PROBARA_BRANCH`    | detected from the CI |
| `source.commit`   | `PROBARA_COMMIT`    | detected from the CI |
| `source.buildUrl` | `PROBARA_BUILD_URL` | detected from the CI |
| `source: false`   | —                   | sends no source      |

### An existing run

`run.ulid` (`PROBARA_RUN_ULID`) reports into a run that exists, such as one `probara run create`
made for the machines of a pipeline ([sharding](https://docs.probara.net/en/developers/ci-integrations/cypress/sharding/)) or
one a person created in Probara ([run selection](https://docs.probara.net/en/developers/ci-integrations/cypress/run-selection/)).
The reporter adds its results to that run and leaves it open:

```text
$ PROBARA_RUN_ULID=01J9Z3K4M5N6P7Q8R9S0T1V2W3 npx cypress run
[probara] Sending 3 results of 3 tests (3 passed, 0 failed, 0 skipped, 0 blocked)
[probara] Recorded 3 results (2 new cases, 0 unmatched) in R-1 (open): https://app.probara.net/projects/SHOP/runs/R-1
[probara] The run R-1 of SHOP stays open: close it in Probara, or with probara run close --project SHOP --run-ulid 01J9Z3K4M5N6P7Q8R9S0T1V2W3
```

The options of a new run are then ignored, with one warning: the run keeps its own name,
environment, milestone and configurations.

```text
[probara] Ignored name: a reused run (run.ulid) keeps its own
```

To get a run's ULID in Probara, open the run and choose **Copy run ULID** in **Run actions** (the
menu in the run's header).

### Closing

| The run                                 | At the end of `cypress run`                                |
| --------------------------------------- | ---------------------------------------------------------- |
| Created by the reporter                 | Closed, whatever the exit code of the tests                |
| Reused (`run.ulid`, `run.ulids`)        | Left open: whoever created it closes it                    |
| Any, when a report failed after retries | Left open: the log names it, and the error says what to do |

`closeRun` (`PROBARA_CLOSE_RUN`) decides for every run: `true` closes a reused run too, `false`
leaves a created one open. `closeRuns` decides per project (`{ SHOP: true, WEB: false }`) when
`closeRun` is not set. Close a run by hand with the [Probara CLI](https://docs.probara.net/en/developers/cli/):

```bash
npx @probara/cli run close --run-ulid 01J9Z3K4M5N6P7Q8R9S0T1V2W3
```

## Statuses

Each attempt is sent with Cypress's own verdict:

| What Cypress reports                                                                | Status                            |
| ----------------------------------------------------------------------------------- | --------------------------------- |
| A test that passed                                                                  | `passed`                          |
| A test that failed: an assertion, an error, a timeout                               | `failed`                          |
| An attempt Cypress is about to run again (a retry)                                  | `failed`                          |
| `it.skip`, or `this.skip()` in the test or a `beforeEach`                           | `skipped`                         |
| A test a failing `beforeEach` kept from running                                     | `skipped`                         |
| A failing `beforeEach` or `before`, reported as `"before each" hook for "…"`        | `failed`, under that name         |
| A spec that throws as it loads (`An uncaught error was detected outside of a test`) | `failed`                          |
| A spec Cypress cannot parse                                                         | `failed`, as `Spec failed to run` |

The reporter never sends `blocked` on its own: map a status to it when you need it
([status mapping](#status-mapping-and-filter)). A result carries the status, the duration and start
time Cypress reported, and the error of a failed attempt in its notes, with its message and stack
and without the terminal's colors.

### A hook that fails

A `beforeEach` or `before` belongs to no test of its own: what it did belongs to the test it runs
for. When it fails, Cypress reports the failure under a test it names after the hook and the test it
was running, and the reporter keeps that name, which is the one a JUnit import gives too.

A hook that fails **once** and passes on the retry is one test with two results: the failed attempt
and the pass. A hook that fails **on every attempt**, with `retries.runMode: 1`:

```js
// cypress/e2e/hooks.cy.js
describe('Checkout', () => {
  beforeEach(() => {
    throw new Error('the cart service is down');
  });

  it('never runs', () => {});
  it('never runs either', () => {});
});

describe('Profile', () => {
  it('still runs', () => {});
});
```

```text
cypress/e2e/hooks.cy.js > Checkout never runs                             failed
cypress/e2e/hooks.cy.js > Checkout "before each" hook for "never runs"    failed
cypress/e2e/hooks.cy.js > Checkout never runs either                      skipped
cypress/e2e/hooks.cy.js > Profile still runs                              passed
```

- `Checkout never runs` is the first attempt of the test, which failed because its hook did and
  which Cypress then retried.
- `"before each" hook for "never runs"` is the failure Cypress could not get past.
- `Checkout never runs either` never ran at all: the reporter reports every test of the suite that
  reported no attempt as **skipped**, which a JUnit report cannot, since such a test leaves no trace
  in it. It is not known to be broken, only known not to have run.
- `Profile still runs`: a failing `beforeEach` stops the rest of **its own** `describe`, never the
  rest of the spec.

### A spec Cypress cannot run

A spec whose module scope throws while Cypress loads it is one failed result, named as Cypress names
it, and retried like any test:

```text
cypress/e2e/throws.cy.js > An uncaught error was detected outside of a test
```

A spec Cypress **cannot parse** (a syntax error) runs no test at all. The plugin turns the failure
Cypress counted into one failed result, so a broken spec never shows green:

```text
cypress/e2e/broken.cy.js > Spec failed to run
```

Its notes read `Cypress could not run this spec: 1 failure(s), no test ran`, and its case sits in the
suite of the spec alone. Without the plugin, there is no such result.

### The exit code is Cypress's own

Cypress ends a `cypress run` with the number of its own failed **tests**, and reporting changes
nothing of it: a test that fails twice is one failed test, and a hook failure is a test of its own.
What Probara recorded (every attempt, a mapped status, a filtered result, a run that could not be
sent) never reaches the exit code.

### Status mapping and filter

`statusMapping` sends a status as another; `statusFilter` sends no result with the given statuses,
**after** the mapping. Both take `passed`, `failed`, `skipped` and `blocked`.

```js
reporterOptions: { projectId: 'SHOP', statusMapping: { failed: 'blocked' } },
```

```text
$ npx cypress run --spec cypress/e2e/cart.cy.js
[probara] Sending 5 results of 5 tests (3 passed, 0 failed, 1 skipped, 1 blocked)
```

With `statusFilter: ['failed']` instead, the failed attempt is not sent at all, and the `Sending`
line says how many were left out:

```text
[probara] Sending 4 results of 4 tests (3 passed, 0 failed, 1 skipped, 0 blocked); 1 left out by statusFilter
```

As variables: `PROBARA_STATUS_MAPPING=failed=blocked,skipped=passed` and
`PROBARA_STATUS_FILTER=skipped,blocked`. A value that is not a status, or a status mapped twice, is
a configuration problem that turns reporting off. Cypress still fails the command: the mapping and
the filter change only what Probara records.

- **Filtering `skipped` leaves out a lot**: every `it.skip` and every test a failing hook kept from
  running.
- **`blocked` tells failures that are not the test's own apart**: map `failed` to `blocked` when a
  hook failure should not read as a failing test.

## Retries and flaky tests

Every attempt is a result of its own, in the order the attempts ran, each with its own status,
duration, error, steps and files. How many attempts Cypress makes is Cypress's own setting,
`retries`:

```js
// cypress.config.js
module.exports = defineConfig({
  e2e: {
    reporter: '@probara/cypress-reporter',
    reporterOptions: { projectId: 'SHOP' },
    setupNodeEvents(on, config) {
      return probaraNodeEvents(on, config);
    },
    retries: { runMode: 1 },
  },
});
```

```text
$ npx cypress run --spec cypress/e2e/retry.cy.js
[probara] Sending 2 results of 1 test (1 passed, 1 failed, 0 skipped, 0 blocked)
```

- **What Probara shows.** The run case keeps the last outcome (`passed` here), and every attempt
  stays in its history: a flaky test is a case whose history mixes failures and passes on the same
  run. The `Sending` line shows it: more results than tests.
- **The attempt number.** From the second attempt on, the result carries an `attempt` parameter
  (`2`, `3`, ...), next to `browser`. Neither is part of the automation key.
- **No flaky flag.** The attempts themselves are the evidence. `probara import junit` keeps only the
  **last** attempt of a retried test, so a suite that moves from the JUnit import to the reporter
  gains the earlier attempts in the history of each flaky case.
- **Each attempt starts empty.** What the `probara.*` helpers said belongs to the attempt that said
  it: the steps, files and metadata of a retry are its own, never inherited.
- **Screenshots of a retry.** Each failed attempt gets the screenshot Cypress took for it.
- **Cypress's own summary** counts each test once, by its last attempt; Probara counts every
  attempt.

To send only the last attempt, do not retry in Cypress: no option of the reporter drops attempts.

## Results file

When a report cannot be sent (Probara unreachable, a refused token, a report that failed after its
retries), its results are lost unless you keep them. Set a results file, and the reporter writes
what it could not send to it; `probara import results` sends it later, into the same runs.

```js
reporterOptions: { projectId: 'SHOP', resultsFile: 'results/probara-results.json' },
```

Or `PROBARA_RESULTS_FILE=results/probara-results.json`, relative to the directory `cypress run` was
started in.

```text
$ npx cypress run
[probara] Sending 1 result of 1 test (1 passed, 0 failed, 0 skipped, 0 blocked)
[probara] 1 result was not sent: Probara answered 403 forbidden: <message from Probara>. No run was created or updated
[probara] Wrote the 1 result that was not sent to /work/shop/results/probara-results.json: send them with probara import results /work/shop/results/probara-results.json
```

Then, from any job that holds the token, send whatever was left:

```bash
npx @probara/cli import results 'results/probara-results*.json'
```

Quote the glob: `probara` expands it, and it matches the file and its siblings.

- **Only when needed.** The file is written only when something was not sent: when every result
  went through, there is no file.
- **Into the same runs.** The file names the runs its results belong to, the settings that sent
  them (project, run, `rootDir`, `createMissingCases`) and each result as the reporter built it, so
  it lands in the same cases from any machine. Never the token.
- **Never over an earlier file.** When a file is already at that path (another machine's, or an
  earlier run's), the reporter writes to the first free sibling: `probara-results-2.json`, then
  `-3`, and so on.
- **Statuses before the mapping.** The file keeps each status before `statusMapping`, which applies
  again when the file is sent.

### What travels with the file

| Files                                                   | In the results file                                                                  |
| ------------------------------------------------------- | ------------------------------------------------------------------------------------ |
| `probara.attach()` bodies and files, the console output | Copied next to it, in `probara-results-attachments/`, and named relative to the file |
| Screenshots and videos                                  | Named by their absolute path, not copied                                             |

Move or download the file and its attachments folder together, and import it from there: a CI
artifact another job downloads keeps those attachments. A machine that never had the screenshots
and videos cannot send them.

### Reporting off: keep everything

With reporting off (`PROBARA_ENABLED=false`, or no token), the reporter writes **every** result to
the file. Run the tests where there is no token, and send the file from a job that holds it:

```bash
PROBARA_ENABLED=false PROBARA_PROJECT=SHOP PROBARA_RESULTS_FILE=results/probara-results.json npx cypress run
npx @probara/cli import results 'results/probara-results*.json'
```

The same file shows what would be sent without sending it
([check what would be sent](https://docs.probara.net/en/developers/ci-integrations/cypress/troubleshooting/#check-what-would-be-sent)).
The import (several files and globs, `--dry-run`, exit codes) is in
[`probara import results`](https://github.com/cynch-me/probara-sdk/blob/main/packages/cli/docs/commands.md#probara-import-results).

## Assign failed results

`assignFailedTo` names the people who look into failures. Every report asks Probara to assign each
run case it leaves failed, and that has no assignee yet, to one of them, in turn, so a failing
nightly run lands on someone's list without anyone triaging it first.

```js
reporterOptions: {
  projectId: 'SHOP',
  assignFailedTo: ['ana@example.com', 'bo@example.com'],
},
```

Or from the environment: `PROBARA_ASSIGN_FAILED_TO=ana@example.com,bo@example.com`.

| Rule                                                      | Why                                                             |
| --------------------------------------------------------- | --------------------------------------------------------------- |
| Only run cases a report leaves `failed`                   | Passed, skipped and blocked results need no one                 |
| Only run cases without an assignee                        | An existing assignee is never replaced: someone already owns it |
| Only active members who can run tests in the project      | A case cannot be assigned to someone who cannot work on it      |
| The listed members take turns, in the order you list them | Failures are spread, not piled on the first name                |
| An email that matches no such member is skipped           | The results are still recorded; Probara's warning counts them   |

- **The last attempt decides.** A flaky test that passes on its retry leaves its case `passed`, so it
  is never assigned, however many of its attempts failed.
- **After `statusMapping`.** With `statusMapping: { failed: 'blocked' }`, nothing is failed, so
  nothing is assigned.
- **The warning never names the emails**, only how many were skipped:
  `[probara] Probara warned: <message from Probara>`.
- **The list** is trimmed and deduplicated. More than 20 emails, or a value that is not an email,
  turns reporting off, naming `assignFailedTo`.

Related: [Configuration](https://docs.probara.net/en/developers/ci-integrations/cypress/configuration/) ·
[Sharding and projects](https://docs.probara.net/en/developers/ci-integrations/cypress/sharding/) ·
[Troubleshooting](https://docs.probara.net/en/developers/ci-integrations/cypress/troubleshooting/#the-run-stays-open) ·
Source: [runs.md](https://github.com/cynch-me/probara-sdk/blob/main/packages/cypress-reporter/docs/runs.md),
[statuses.md](https://github.com/cynch-me/probara-sdk/blob/main/packages/cypress-reporter/docs/statuses.md),
[specs.md](https://github.com/cynch-me/probara-sdk/blob/main/packages/cypress-reporter/docs/specs.md),
[retries.md](https://github.com/cynch-me/probara-sdk/blob/main/packages/cypress-reporter/docs/retries.md),
[results-file.md](https://github.com/cynch-me/probara-sdk/blob/main/packages/cypress-reporter/docs/results-file.md),
[assign-failed.md](https://github.com/cynch-me/probara-sdk/blob/main/packages/cypress-reporter/docs/assign-failed.md)