# Jest results, statuses and retries

> The run each jest command reports into, how attempts become statuses, retries and flaky tests, the results file for what could not be sent, and assigning failed results.

Each `jest` command reports into one Probara run: a new automated run by default, closed at the
end, or an existing one. Every attempt of every test is a result in it, with Jest's own verdict.
This page covers the run, the statuses, retries, what happens when results cannot be sent, and who
gets the failures.

## The run

These options describe the run the reporter creates:

```js
reporters: [
  'default',
  [
    '@probara/jest-reporter',
    {
      run: {
        name: 'Nightly regression',
        description: 'Every night on main, against staging.',
        environment: 'staging',
        milestone: 'Sprint 12',
        plan: 'PLAN-2',
        configurations: [
          { group: 'Node', name: '22' },
          { group: 'OS', name: 'Linux' },
        ],
        tags: ['nightly', 'unit'],
      },
    },
  ],
],
```

| Option               | Variable                  | What it sets                                                                                |
| -------------------- | ------------------------- | ------------------------------------------------------------------------------------------- |
| `run.name`           | `PROBARA_RUN_NAME`        | The name. Default: the CI build name (`CI #42`), else `Automated run <date> <time> UTC`     |
| `run.description`    | `PROBARA_RUN_DESCRIPTION` | The description, up to 2000 characters (longer is cut, with a warning)                      |
| `run.environment`    | `PROBARA_ENVIRONMENT`     | The environment, by name: created in the project when none has that name                    |
| `run.milestone`      | `PROBARA_MILESTONE`       | The milestone, by display id (`M-3`) or exact name                                          |
| `run.plan`           | `PROBARA_PLAN`            | The test plan, by display id (`PLAN-2`) or exact name: the run starts with the plan's cases |
| `run.configurations` | `PROBARA_CONFIGURATIONS`  | Configuration values by group and name (`Node=22,OS=Linux`), each group once                |
| `run.tags`           | `PROBARA_RUN_TAGS`        | Tags (`nightly,unit`), up to 50                                                             |

The same run from the environment of a CI step, with the project in the config:

```bash
PROBARA_RUN_NAME="Nightly regression" PROBARA_ENVIRONMENT=staging PROBARA_RUN_TAGS=nightly,unit npx jest
```

### Names or ULIDs

The environment, milestone and configurations also take ULIDs: `run.environmentId`,
`run.milestoneId` and `run.configurationUlids` (`PROBARA_ENVIRONMENT_ID`, `PROBARA_MILESTONE_ID`,
`PROBARA_CONFIGURATION_ULIDS`). Set one form of each, never both: both is a configuration problem
that turns reporting off. Names are the form to use with an app token, which cannot look ULIDs up.

- An unknown environment **name** creates the environment.
- An unknown milestone, plan or configuration makes Probara refuse the first report (`422`, naming
  the field): nothing is recorded, and the log says why. Check the names in Probara.

### Run source

A run records where it came from: the branch, the commit and the CI build URL, read from the
variables of GitHub Actions, GitLab CI, CircleCI, Azure Pipelines, Jenkins, Bitbucket Pipelines and
Buildkite ([what each CI fills in](https://docs.probara.net/en/developers/ci-integrations/jest/ci-providers/)). Override a
field with `source` or its variable, or send none:

```js
reporters: ['default', ['@probara/jest-reporter', { source: { branch: 'release/2.4' } }]],
```

| Option            | Variable            | Default              |
| ----------------- | ------------------- | -------------------- |
| `source.branch`   | `PROBARA_BRANCH`    | detected from the CI |
| `source.commit`   | `PROBARA_COMMIT`    | detected from the CI |
| `source.buildUrl` | `PROBARA_BUILD_URL` | detected from the CI |
| `source: false`   | —                   | sends no source      |

The CI build name also names the run when `run.name` is not set.

### An existing run

`run.ulid` (`PROBARA_RUN_ULID`) reports into a run that exists, such as one `probara run create`
made for the shards of a pipeline ([sharding](https://docs.probara.net/en/developers/ci-integrations/jest/sharding/)) or one
a person created in Probara ([run selection](https://docs.probara.net/en/developers/ci-integrations/jest/run-selection/)).
The reporter adds its results to that run and leaves it open:

```text
$ PROBARA_RUN_ULID=01J9Z3K4M5N6P7Q8R9S0T1V2W3 npx jest
[probara] Sending 3 results of 3 tests (3 passed, 0 failed, 0 skipped, 0 blocked)
[probara] Recorded 3 results (2 new cases, 0 unmatched) in R-1 (open): https://app.probara.net/projects/SHOP/runs/R-1
```

The options of a new run are then ignored, with one warning: the run keeps its own name,
environment, milestone and configurations.

```text
[probara] Ignored name: a reused run (run.ulid) keeps its own
```

To get a run's ULID in Probara, open the run and choose **Copy run ULID** in **Run actions** (the
menu in the run's header).

### Closing

| The run                                 | At the end of `jest`                                       |
| --------------------------------------- | ---------------------------------------------------------- |
| Created by the reporter                 | Closed, once every result and file was sent                |
| Reused (`run.ulid`)                     | Left open: whoever created it closes it                    |
| Created in a `--watch` session          | Left open, for the next re-run                             |
| Any, when a report failed after retries | Left open: the log names it, and the error says what to do |

`closeRun` (`PROBARA_CLOSE_RUN`) decides for every run of a command that is not a watch session:
`true` closes a reused run too, `false` leaves a created one open. `closeRuns` decides per project
(`{ SHOP: true, WEB: false }`) when `closeRun` is not set. When a report fails, what was not sent can
go to a [results file](#results-file), which `probara import results` sends later into the same run,
and closes it like the reporter would have.

Close a run by hand with the [Probara CLI](https://docs.probara.net/en/developers/cli/):

```bash
npx @probara/cli run close --run-ulid 01J9Z3K4M5N6P7Q8R9S0T1V2W3
```

With [several Probara projects](https://docs.probara.net/en/developers/ci-integrations/jest/sharding/#several-probara-projects),
each project gets its own run. The name, description, environment (by name) and tags go with every
new run; the milestone, plan and configurations go with the configured project's run only.

## Statuses

Each attempt is sent with Jest's own verdict:

| The attempt                                                                         | Status                          |
| ----------------------------------------------------------------------------------- | ------------------------------- |
| Passed                                                                              | `passed`                        |
| Failed: an assertion, an error, a timeout, a `beforeAll` or `beforeEach` that threw | `failed`                        |
| `test.skip`, `xit`, a skipped `describe`, a test `.only` or `-t` left out           | `skipped`                       |
| `test.todo`                                                                         | `skipped`, with the note `Todo` |
| `test.failing`, and it threw as expected                                            | `passed`                        |
| `test.failing`, but it passed                                                       | `failed`                        |

The reporter never sends `blocked` on its own: map a status to it
([below](#status-mapping-and-filter)) when you need it.

```js
// tests/statuses.test.js
test('passes', () => {});

test('fails', () => {
  expect(1 + 1).toBe(3);
});

test.skip('is skipped', () => {});

test.todo('is not written yet');

test.failing('is a known bug', () => {
  expect(0.1 + 0.2).toBe(0.3);
});
```

```json
[
  { "automationKey": "tests/statuses.test.js > passes", "status": "passed" },
  { "automationKey": "tests/statuses.test.js > fails", "status": "failed" },
  {
    "automationKey": "tests/statuses.test.js > is not written yet",
    "status": "skipped",
    "notes": "Todo"
  },
  { "automationKey": "tests/statuses.test.js > is a known bug", "status": "passed" },
  { "automationKey": "tests/statuses.test.js > is skipped", "status": "skipped" }
]
```

- **Order.** Tests are sent in the order Jest reports them: the tests that ran and the todo tests as
  they end, then the skipped tests of the file, once it ends.
- **Errors** go in the notes of the result: every error of the attempt, with its message and stack,
  without the terminal's colors, after a
  [comment](https://docs.probara.net/en/developers/ci-integrations/jest/metadata/#comment) if the test wrote one.
- **Todo tests** are never lost: they are skipped results with the note `Todo`, so the case shows
  what is still to be written. `probara import junit` records them as passed, because jest-junit
  writes them as passing test cases.
- **Expected failures** are Jest's: a `test.failing` test that throws does what it should, so it
  passes, and the run stays green in Probara as it does in Jest. When the bug is fixed, the test
  passes, Jest fails it, and so does Probara, reminding you to remove `.failing`.
- **Skipped tests have no duration**: Jest does not run them. On Jest 29 they get their file's
  start time; Jest 30 gives each its own.

### A hook that fails after the tests

An `afterAll` hook that throws fails the test file after its tests passed. Jest prints the error and
fails the command; its tests stay green. So that Probara does not show the file as green either, the
reporter sends one more failed result for the file, the one jest-junit writes, with the error in its
notes:

```js
// tests/cleanup.test.js
afterAll(() => {
  throw new Error('Could not drop the test database');
});

test('creates an order', () => {});
```

```json
[
  { "automationKey": "tests/cleanup.test.js > creates an order", "status": "passed" },
  {
    "automationKey": "tests/cleanup.test.js > Test execution failure: could be caused by test hooks like 'afterAll'.",
    "status": "failed"
  }
]
```

Its key is the one `probara import junit` gives the same failure in jest-junit's report, so both
tools record it into the same case.

### A file Jest cannot run

A test file that fails before its tests exist (a syntax error, an import that throws) has no test to
report. The reporter names it in one warning, and reports the other files as usual:

```text
$ npx jest
[probara] Could not report tests/broken.test.js: Jest could not run it (Cannot find module './missing-module' from 'tests/broken.test.js')
[probara] Sending 1 result of 1 test (1 passed, 0 failed, 0 skipped, 0 blocked)
[probara] Recorded 1 result (1 new case, 0 unmatched) in R-1 (closed): https://app.probara.net/projects/SHOP/runs/R-1
```

Fix the file: Jest's own output shows the error in full.

### Status mapping and filter

`statusMapping` sends a status as another; `statusFilter` sends no result with the given statuses,
after the mapping. Both take `passed`, `failed`, `skipped` and `blocked`:

```js
reporters: [
  'default',
  [
    '@probara/jest-reporter',
    { statusMapping: { failed: 'blocked' }, statusFilter: ['skipped'] },
  ],
],
```

With one passing test, one failing test and one `test.todo`:

```text
$ npx jest
[probara] Sending 2 results of 2 tests (1 passed, 0 failed, 0 skipped, 1 blocked); 1 left out by statusFilter
[probara] Recorded 2 results (2 new cases, 0 unmatched) in R-1 (closed): https://app.probara.net/projects/SHOP/runs/R-1
[probara] Filtered out 1 result by their status (statusFilter): not sent
```

As variables: `PROBARA_STATUS_MAPPING=failed=blocked,skipped=passed` and
`PROBARA_STATUS_FILTER=skipped,blocked`. A value that is not a status, or a status mapped twice, is a
configuration problem that turns reporting off. `statusFilter: ['skipped']` leaves out todo tests
too: they are skipped results. The mapping changes what Probara records only: Jest's exit code stays
the tests' own.

## Retries and flaky tests

Every attempt of a test is a result of its own. A test that `jest.retryTimes(2)` retries twice sends
three results, in the order they ran, each with its own status, duration, error, steps and files.

```js
// tests/dashboard.test.js
jest.retryTimes(1);

let attempts = 0;

test('loads the dashboard', () => {
  attempts += 1;
  // Fails the first time, passes on the retry.
  expect(attempts).toBe(2);
});
```

```text
$ npx jest
[probara] Sending 2 results of 1 test (1 passed, 1 failed, 0 skipped, 0 blocked)
[probara] Recorded 2 results (1 new case, 0 unmatched) in R-1 (closed): https://app.probara.net/projects/SHOP/runs/R-1
```

- **What Probara shows.** The run case keeps the last outcome (`passed` here), and every attempt
  stays in its history: a flaky test is a case whose history mixes failures and passes on the same
  run. The log says it too: more results than tests.
- **No flaky flag.** The reporter does not mark attempts as flaky: the attempts themselves are the
  evidence. jest-junit and `probara import junit` keep only the last attempt of a retried test.
- **Each attempt starts empty.** `probara.*` calls apply to the attempt that made them, so a retry
  that stops earlier does not inherit the steps, files or metadata of the first attempt.
- **Jest's summary** counts the test once, by its last attempt; Probara counts every attempt.

By default Jest retries the failed tests of a file after the file's other tests; Jest 30's
`jest.retryTimes(n, { retryImmediately: true })` retries each one right away. Either way each
attempt is sent as it ends, so the order of attempts of one test is always the order they ran.

| Jest                                            | What changes for Probara                                                                                             |
| ----------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- |
| `retryImmediately`, `waitBeforeRetry` (Jest 30) | When each attempt runs; every attempt is still a result                                                              |
| `logErrorsBeforeRetry`                          | What Jest prints; every failed attempt's error is in its own result anyway                                           |
| `test.concurrent` on Jest 29                    | Never retried by Jest: one result                                                                                    |
| `test.concurrent` on Jest 30.0 to 30.4          | Each attempt is a result, but the `probara.*` calls of a retry go to the first attempt: Jest does not say which runs |
| `test.concurrent` on Jest 30.5 and later        | Each attempt is a result, with its own `probara.*` calls                                                             |

To send only the last attempt, do not retry in Jest: no option of the reporter drops attempts. To
keep failures of retried tests from counting, [map them](#status-mapping-and-filter), knowing it
applies to every failure.

## Results file

When a report cannot be sent (Probara unreachable, the network lost, a refused report), its results
are lost unless you keep them. Set a results file, and the reporter writes what it could not send to
it at the end of the run; `probara import results` sends it later, into the same runs.

```js
reporters: ['default', ['@probara/jest-reporter', { resultsFile: 'probara-results.json' }]],
```

Or `PROBARA_RESULTS_FILE=probara-results.json`, relative to the directory `jest` runs in.

```text
$ PROBARA_RESULTS_FILE=probara-results.json npx jest
[probara] Sending 3 results of 3 tests (3 passed, 0 failed, 0 skipped, 0 blocked)
[probara] 3 results were not sent: Probara answered 403 forbidden: <message from Probara>. No run was created or updated
[probara] Wrote the 3 results that were not sent to /work/shop/probara-results.json: send them with probara import results /work/shop/probara-results.json
```

Then, from any job that holds the token, send whatever was left:

```bash
npx @probara/cli import results 'probara-results*.json'
```

Quote the glob: `probara` expands it, and it matches the file and its siblings. When nothing
matches, nothing was left unsent: the import says so and exits 0.

- **Only when needed.** The file is written only when something was not sent: when every result
  went through, there is no file.
- **Into the same runs.** The file names the runs its results belong to: a run the failed report
  left open gets the rest of its results, and a run that was never created is created by the import
  with the same name and settings. Runs are closed like the reporter would have closed them.
- **Consumed by the import.** `probara import results` sends each file on its own, then deletes it
  (and its attachments folder) once every result in it was sent, or rewrites it with only what is
  still unsent. Importing again never sends a result twice, so the import is safe to run in every
  pipeline. `--dry-run` never touches the files.
- **Never touches an earlier file.** When a file is already at that path (another shard's on a
  shared disk, or an earlier run's), the reporter leaves it as it is and writes to the first free
  sibling: `probara-results-2.json`, then `-3`, and so on, each with its own attachments folder.
  The log names the file written, and the glob imports them all, oldest first.
- **Never half written.** Each file is written whole to a temporary file in the same folder first (a
  dot name the glob never matches), then appears under its name at once: an import that runs at the
  same time sees the whole file or none of it.

### Attachments travel with the file

The files a result carries (what `probara.attach()` copied, the console output of `captureOutput`)
are kept next to the results file, in its attachments folder (`probara-results-attachments/` for
`probara-results.json`), and the file names them relative to itself. Move or download the file and
its folder together, anywhere, and import it from there: a CI artifact another job downloads keeps
its attachments.

An upload that fails during the import is not kept for another try: its result was recorded, so it
is no longer in the file, and the folder goes with the file once every result was sent. The import
logs it and exits 1.

### Reporting off: keep everything

With reporting off (`PROBARA_ENABLED=false`, or no token), the reporter writes **every** attempt to
the file, case links included. Run the tests where there is no token (a fork, a machine without
network), and import the file from a job that holds it:

```text
$ PROBARA_ENABLED=false PROBARA_RESULTS_FILE=probara-results.json npx jest
[probara] Wrote 3 results to /work/shop/probara-results.json: send them with probara import results /work/shop/probara-results.json
```

The same file shows what would be sent without sending it
([check what would be sent](https://docs.probara.net/en/developers/ci-integrations/jest/troubleshooting/#check-what-would-be-sent)).

### In CI

Keep the files and their attachments folders as artifacts of the test job, and import them in a
later job (or step) that holds the token:

```yaml
- name: Run Jest tests
  run: npx jest
  env:
    PROBARA_API_TOKEN: ${{ secrets.PROBARA_API_TOKEN }}
    PROBARA_RESULTS_FILE: probara-results.json
- name: Send what Probara did not get
  if: always()
  run: npx --yes @probara/cli@0.2 import results 'probara-results*.json'
  env:
    PROBARA_API_TOKEN: ${{ secrets.PROBARA_API_TOKEN }}
    PROBARA_ENABLED: ${{ secrets.PROBARA_API_TOKEN != '' }}
```

When Probara was only briefly away, the import sends the rest. When it still cannot, the import
exits 1 and keeps what it could not send in each file; the files it sent are gone.

A results file is trusted input: its attachments are uploaded from the paths it names, so import
only the files your own jobs wrote. The import leaves out, with a warning, an attachment whose
relative path leads outside the folder of the file; absolute paths are kept.

### The format

Version 1 of a JSON document: the project, the runs and the settings of the reporter (never the
token), then each result as the reporter built it, before `statusMapping` (which applies when the
file is sent). The full format is in
[core's results file](https://github.com/cynch-me/probara-sdk/blob/main/packages/core/README.md#results-file),
and the import (several files and globs, the JSON summary, exit codes) in
[`probara import results`](https://github.com/cynch-me/probara-sdk/blob/main/packages/cli/docs/commands.md#probara-import-results).

## Assign failed results

`assignFailedTo` names the people who look into failures. Every report asks Probara to assign each
run case it leaves failed, and that has no assignee yet, to one of them, in turn, so a failing
nightly run lands on someone's list without anyone triaging it first.

```js
reporters: [
  'default',
  [
    '@probara/jest-reporter',
    { assignFailedTo: ['ana@example.com', 'bo@example.com', 'cy@example.com'] },
  ],
],
```

Or from the environment: `PROBARA_ASSIGN_FAILED_TO=ana@example.com,bo@example.com`. With
`ana@example.com` and `bo@example.com` members of the project and `cy@example.com` not, two failed
cases go to Ana and Bo, and Probara says one email matched no one:

```text
$ npx jest
[probara] Sending 3 results of 3 tests (1 passed, 2 failed, 0 skipped, 0 blocked)
[probara] Probara warned: assignFailedTo: 1 of 3 emails did not match a member who can be assigned in this project
[probara] Recorded 3 results (3 new cases, 0 unmatched) in R-1 (closed): https://app.probara.net/projects/SHOP/runs/R-1
```

| Rule                                                      | Why                                                             |
| --------------------------------------------------------- | --------------------------------------------------------------- |
| Only run cases a report leaves `failed`                   | Passed, skipped and blocked results need no one                 |
| Only run cases without an assignee                        | An existing assignee is never replaced: someone already owns it |
| Only active members who can run tests in the project      | A case cannot be assigned to someone who cannot work on it      |
| The listed members take turns, in the order you list them | Failures are spread, not piled on the first name                |
| An email that matches no such member is skipped           | The results are still recorded; Probara's warning counts them   |

- **After `statusMapping`.** The status that counts is the one Probara records: with
  `statusMapping: { failed: 'blocked' }`, nothing is failed, so nothing is assigned.
- **The warning never names the emails**, only how many were skipped: the log of a CI job is not the
  place to list who is or is not a member. Check the list against the project's members in Probara.
- **Turns start again with every report**: each report (a chunk of up to 500 results, a shard, a
  watch re-run) gives its first failed case to the first member listed.
- **Every report of the run asks**, with the same list: a run with several Probara projects asks in
  each project's run, and the shards of a shared run each assign the cases they leave failed.
- **A results file keeps the list.** `probara import results` sends it with the results, unless
  `--assign-failed-to` or `PROBARA_ASSIGN_FAILED_TO` names others.

Related: [Configuration](https://docs.probara.net/en/developers/ci-integrations/jest/configuration/) ·
[Sharding and projects](https://docs.probara.net/en/developers/ci-integrations/jest/sharding/) ·
[Troubleshooting](https://docs.probara.net/en/developers/ci-integrations/jest/troubleshooting/#the-run-was-left-open) ·
Source: [runs.md](https://github.com/cynch-me/probara-sdk/blob/main/packages/jest-reporter/docs/runs.md),
[statuses.md](https://github.com/cynch-me/probara-sdk/blob/main/packages/jest-reporter/docs/statuses.md),
[retries.md](https://github.com/cynch-me/probara-sdk/blob/main/packages/jest-reporter/docs/retries.md),
[results-file.md](https://github.com/cynch-me/probara-sdk/blob/main/packages/jest-reporter/docs/results-file.md),
[assign-failed.md](https://github.com/cynch-me/probara-sdk/blob/main/packages/jest-reporter/docs/assign-failed.md)