# Vitest results, statuses and retries

> The run each vitest run reports into, how attempts, hooks and broken files become statuses, retries and repeats, the results file for what could not be sent, and assigning failed results.

Each `vitest run` reports into one Probara run: a new automated run by default, closed at the end,
or an existing one. Every execution of every test is a result in it, with Vitest's own verdict.
This page covers the run, the statuses, retries and repeats, what happens when results cannot be
sent, and who gets the failures.

## The run

These options describe the run the reporter creates:

```ts
reporters: [
  'default',
  [
    '@probara/vitest-reporter',
    {
      run: {
        name: 'Nightly regression',
        description: 'Every night on main, against staging.',
        environment: 'staging',
        milestone: 'Sprint 12',
        plan: 'PLAN-2',
        configurations: [
          { group: 'Node', name: '22' },
          { group: 'OS', name: 'Linux' },
        ],
        tags: ['nightly', 'unit'],
      },
    },
  ],
],
```

| Option               | Variable                  | What it sets                                                                                |
| -------------------- | ------------------------- | ------------------------------------------------------------------------------------------- |
| `run.name`           | `PROBARA_RUN_NAME`        | The name. Default: the CI build name (`CI #42`), else `Automated run <date> <time> UTC`     |
| `run.description`    | `PROBARA_RUN_DESCRIPTION` | The description, up to 2000 characters (longer is cut, with a warning)                      |
| `run.environment`    | `PROBARA_ENVIRONMENT`     | The environment, by name: created in the project when none has that name                    |
| `run.milestone`      | `PROBARA_MILESTONE`       | The milestone, by display id (`M-3`) or exact name                                          |
| `run.plan`           | `PROBARA_PLAN`            | The test plan, by display id (`PLAN-2`) or exact name: the run starts with the plan's cases |
| `run.configurations` | `PROBARA_CONFIGURATIONS`  | Configuration values by group and name (`Node=22,OS=Linux`), each group once                |
| `run.tags`           | `PROBARA_RUN_TAGS`        | Tags (`nightly,unit`), up to 50                                                             |

The same run from the environment of a CI step, with the project in the config:

```bash
PROBARA_RUN_NAME="Nightly regression" PROBARA_ENVIRONMENT=staging PROBARA_RUN_TAGS=nightly,unit npx vitest run
```

Vitest `projects` do not change the run: every project of the config reports into the same run,
and a result names its Vitest project in the `project` parameter when the config has more than one
([Vitest projects](https://docs.probara.net/en/developers/ci-integrations/vitest/sharding/#vitest-projects)).

### Names or ULIDs

The environment, milestone and configurations also take ULIDs: `run.environmentId`,
`run.milestoneId` and `run.configurationUlids` (`PROBARA_ENVIRONMENT_ID`, `PROBARA_MILESTONE_ID`,
`PROBARA_CONFIGURATION_ULIDS`). Set one form of each, never both: both is a configuration problem
that turns reporting off. Names are the form to use with an app token, which cannot look ULIDs up.

- An unknown environment **name** creates the environment.
- An unknown milestone, plan or configuration makes Probara refuse the first report (`422`, naming
  the field): nothing is recorded, and the log says why. Check the names in Probara.

### Run source

A run records where it came from: the branch, the commit and the CI build URL, read from the
variables of GitHub Actions, GitLab CI, CircleCI, Azure Pipelines, Jenkins, Bitbucket Pipelines and
Buildkite ([what each CI fills in](https://docs.probara.net/en/developers/ci-integrations/vitest/ci-providers/#what-ci-detection-fills-in)).
Override a field with `source` or its variable, or send none:

```ts
reporters: ['default', ['@probara/vitest-reporter', { source: { branch: 'release/2.4' } }]],
```

| Option            | Variable            | Default              |
| ----------------- | ------------------- | -------------------- |
| `source.branch`   | `PROBARA_BRANCH`    | detected from the CI |
| `source.commit`   | `PROBARA_COMMIT`    | detected from the CI |
| `source.buildUrl` | `PROBARA_BUILD_URL` | detected from the CI |
| `source: false`   | —                   | sends no source      |

The CI build name also names the run when `run.name` is not set.

### An existing run

`run.ulid` (`PROBARA_RUN_ULID`) reports into a run that exists, such as one `probara run create`
made for the shards of a pipeline
([sharding](https://docs.probara.net/en/developers/ci-integrations/vitest/sharding/#pattern-2-create-the-run-report-from-each-shard-close-it)),
or one a person created in Probara
([run selection](https://docs.probara.net/en/developers/ci-integrations/vitest/run-selection/)). The reporter adds its
results to that run and leaves it open:

```text
$ PROBARA_RUN_ULID=01J9Z3K4M5N6P7Q8R9S0T1V2W3 npx vitest run
[probara] Sending 3 results of 3 tests (3 passed, 0 failed, 0 skipped, 0 blocked)
[probara] Recorded 3 results (2 new cases, 0 unmatched) in R-1 (open): https://app.probara.net/projects/SHOP/runs/R-1
```

The options of a new run are then ignored, with one warning: the run keeps its own name,
environment, milestone and configurations.

```ts
reporters: ['default', ['@probara/vitest-reporter', { run: { name: 'Nightly regression' } }]],
```

```text
$ PROBARA_RUN_ULID=01J9Z3K4M5N6P7Q8R9S0T1V2W3 npx vitest run
[probara] Ignored name: a reused run (run.ulid) keeps its own
[probara] Sending 3 results of 3 tests (3 passed, 0 failed, 0 skipped, 0 blocked)
[probara] Recorded 3 results (2 new cases, 0 unmatched) in R-1 (open): https://app.probara.net/projects/SHOP/runs/R-1
```

To get a run's ULID in Probara, open the run and choose **Copy run ULID** in **Run actions** (the
menu in the run's header).

### Closing

| The run                                       | At the end of `vitest`                                     |
| --------------------------------------------- | ---------------------------------------------------------- |
| Created by the reporter                       | Closed, once every result and file was sent                |
| Reused (`run.ulid`)                           | Left open: whoever created it closes it                    |
| Created in a watch session (`vitest --watch`) | Left open: every re-run of the session reports into it     |
| Any, when a report failed after retries       | Left open: the log names it, and the error says what to do |

`closeRun` (`PROBARA_CLOSE_RUN`) decides for every run of a command that is not a watch session:
`true` closes a reused run too, `false` leaves a created one open. `closeRuns` decides per project
(`{ SHOP: true, WEB: false }`) when `closeRun` is not set. When a report fails, what was not sent
can go to a [results file](#results-file), which `probara import results` sends later into the same
run, and closes it like the reporter would have.

- **Watch mode.** `vitest --watch` (and `vitest` alone in a terminal, which starts watch mode)
  makes one run per Probara project for the whole session: each re-run reports into it, and the
  `Sending` line counts that re-run only. The reporter never closes it, and says once how to. A run
  closed or deleted in Probara during the session gets no more results: the reporter sends that
  re-run's results into a new run (or keeps them in the results file, when one is set), and says so
  ([watch mode](https://docs.probara.net/en/developers/ci-integrations/vitest/configuration/#watch-mode)).
- **Shards with the `blob` reporter.** A shard (`vitest run --shard=1/2`) that writes a blob sends
  nothing and creates no run: `vitest --merge-reports` reports every shard's results into one run,
  which it closes. A shard with the reporter beside `blob` in its reporters says so:
  `Sharded run with the blob reporter: the results are reported by vitest --merge-reports`
  ([sharding](https://docs.probara.net/en/developers/ci-integrations/vitest/sharding/#pattern-1-merge-the-blob-reports)).

Close a run by hand with the [Probara CLI](https://docs.probara.net/en/developers/cli/):

```bash
npx @probara/cli run close --run-ulid 01J9Z3K4M5N6P7Q8R9S0T1V2W3
```

With [several Probara projects](https://docs.probara.net/en/developers/ci-integrations/vitest/sharding/#several-probara-projects),
each project gets its own run. The name, description, environment (by name) and tags go with every
new run; the milestone, plan and configurations go with the configured project's run only.

## Statuses

Each attempt is sent with Vitest's own verdict, as `passed`, `failed` or `skipped`.
`statusMapping` and `statusFilter` can change that afterwards.

| The attempt                                                                           | Status                                                           |
| ------------------------------------------------------------------------------------- | ---------------------------------------------------------------- |
| Passed                                                                                | `passed`                                                         |
| Failed: an assertion, an error, a timeout, a `beforeEach` or `afterEach` that threw   | `failed`                                                         |
| `expect.soft` failures                                                                | `failed`, with every failure in the notes                        |
| `test.skip`, `skipIf`, `runIf`, a skipped `describe`, a test `.only` or `-t` left out | `skipped`                                                        |
| `test.todo`                                                                           | `skipped`, with the note `Todo`                                  |
| `context.skip()`; with a note, `context.skip('why')`                                  | `skipped`; with a note, the note `Skipped: why`                  |
| `test.fails`, and it threw as expected                                                | `passed`                                                         |
| `test.fails`, but it passed                                                           | `failed`, with Vitest's error `Expect test to fail`              |
| A test under a `beforeAll` that threw                                                 | `skipped`                                                        |
| A test Vitest never started, after `--bail` or Ctrl-C                                 | `skipped`, with the note `Not run: the test run was interrupted` |

The reporter never sends `blocked` on its own: map a status to it
([below](#status-mapping-and-filter)) when you need it.

```ts
// tests/statuses.test.ts
import { expect, test } from 'vitest';

test('passes', () => {});

test('fails', () => {
  expect(1 + 1).toBe(3);
});

test.skip('is skipped', () => {});

test.todo('is not written yet');

test('skips itself', (context) => {
  context.skip('needs the payment sandbox');
});

test.fails('is a known bug', () => {
  expect(0.1 + 0.2).toBe(0.3);
});
```

```json
[
  { "automationKey": "tests/statuses.test.ts > passes", "status": "passed" },
  { "automationKey": "tests/statuses.test.ts > fails", "status": "failed" },
  { "automationKey": "tests/statuses.test.ts > is skipped", "status": "skipped" },
  {
    "automationKey": "tests/statuses.test.ts > is not written yet",
    "status": "skipped",
    "notes": "Todo"
  },
  {
    "automationKey": "tests/statuses.test.ts > skips itself",
    "status": "skipped",
    "notes": "Skipped: needs the payment sandbox"
  },
  { "automationKey": "tests/statuses.test.ts > is a known bug", "status": "passed" }
]
```

- **Order.** Results are sent file by file once the run ends, each file's tests in the order they
  are written, and every attempt of a test in the order it ran.
- **Errors** go in the notes of the result: every error of the attempt, `expect.soft` failures
  included, in order, each with its message and stack, without the terminal's colors, after a
  [comment](https://docs.probara.net/en/developers/ci-integrations/vitest/metadata/#comment) if the test wrote one. A
  passed attempt sends no error.
- **Skips are not told apart.** Vitest reports `test.skip`, `skipIf`, `runIf`, `.only` and `-t`
  the same way, and so does its JUnit output: they are skipped results with no note, as
  `probara import junit` records them.
- **Todo tests** are never lost: they are skipped results with the note `Todo`, so the case shows
  what is still to be written.
- **Expected failures** are Vitest's: a `test.fails` test that throws does what it should, so it
  passes. When the bug is fixed, the test passes, Vitest fails it, and so does Probara, reminding
  you to remove `.fails`.
- **An interrupted run** (`--bail`, Ctrl-C) still reports what it ran. A test Vitest never started
  is skipped with the note `Not run: the test run was interrupted`; a test Vitest skipped itself
  because the run was cancelled keeps Vitest's note, `The test run was aborted by the user.` A test
  file Vitest never started has no test to report: one warning counts such files.

### Errors outside a test

An error in a suite's hook or in the file itself fails the file in Vitest, even when its tests
pass. The reporter sends the failed results Vitest's JUnit output writes for it, with the error in
their notes, so Probara does not show the file as green either:

| Where the error happens                                                       | What Probara gets                                                                       |
| ----------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- |
| The `beforeAll` or `afterAll` of a `describe`                                 | One failed result named after the `describe`: `tests/session.test.ts > session`         |
| A `beforeAll` or `afterAll` outside every `describe`                          | One failed result named after the file: `tests/cleanup.test.ts > tests/cleanup.test.ts` |
| The test file fails to load (a syntax error, an import that throws)           | One failed result named after the file: `tests/broken.test.ts > tests/broken.test.ts`   |
| An unhandled error: a rejection no test awaited, an error thrown from a timer | No result: Vitest lists it under its unhandled errors, and one warning counts them      |

```ts
// tests/session.test.ts
import { beforeAll, describe, test } from 'vitest';

describe('session', () => {
  beforeAll(() => {
    throw new Error('Could not seed the users');
  });

  test('renews the token', () => {});
});

test('logs out', () => {});
```

```json
[
  { "automationKey": "tests/session.test.ts > session", "status": "failed" },
  { "automationKey": "tests/session.test.ts > session > renews the token", "status": "skipped" },
  { "automationKey": "tests/session.test.ts > logs out", "status": "passed" }
]
```

The failed results come first in their file. Their keys are the ones `probara import junit` gives
the same failures in Vitest's JUnit output, so both tools record them into the same case, odd names
included: Vitest's JUnit writer names a hook's failure after its own `describe`, not after the
`describe` blocks above it, and a file's failure after the file, which its key then holds twice.

A file that fails to load is one failed result, whatever tests it holds:

```ts
// tests/broken.test.ts
import { expect, test } from 'vitest';
import { total } from './missing-module';

test('adds the items', () => {
  expect(total([1, 2])).toBe(3);
});
```

```json
[
  {
    "automationKey": "tests/broken.test.ts > tests/broken.test.ts",
    "status": "failed"
  }
]
```

Fix the file: Vitest's own output shows the error in full.

### Status mapping and filter

`statusMapping` sends a status as another; `statusFilter` sends no result with the given statuses,
after the mapping. Both take `passed`, `failed`, `skipped` and `blocked`:

```ts
reporters: [
  'default',
  [
    '@probara/vitest-reporter',
    { statusMapping: { failed: 'blocked' }, statusFilter: ['skipped'] },
  ],
],
```

```ts
// tests/filters.test.ts
import { expect, test } from 'vitest';

test('passes', () => {});

test('fails', () => {
  expect(true).toBe(false);
});

test.todo('is not written yet');
```

```text
$ npx vitest run
[probara] Sending 2 results of 2 tests (1 passed, 0 failed, 0 skipped, 1 blocked); 1 left out by statusFilter
[probara] Recorded 2 results (2 new cases, 0 unmatched) in R-1 (closed): https://app.probara.net/projects/SHOP/runs/R-1
[probara] Filtered out 1 result by their status (statusFilter): not sent
```

As variables: `PROBARA_STATUS_MAPPING=failed=blocked,skipped=passed` and
`PROBARA_STATUS_FILTER=skipped,blocked`. A value that is not a status, or a status mapped twice, is
a configuration problem that turns reporting off. `statusFilter: ['skipped']` leaves out todo tests
too: they are skipped results. The mapping changes what Probara records only: Vitest's exit code
stays the tests' own.

## Retries and repeats

Every execution of a test is a result of its own: each attempt Vitest's `retry` gives it, and each
time `repeats` runs it again. A test retried once sends two results, in the order they ran, each
with its own status, duration, errors, steps and files.

```ts
// tests/dashboard.test.ts
import { expect, test } from 'vitest';

let loads = 0;

test('loads the dashboard', { retry: 1 }, () => {
  loads += 1;
  // Fails the first time, passes on the retry.
  expect(loads).toBe(2);
});

test('opens the menu', { repeats: 2 }, () => {});
```

```json
[
  { "automationKey": "tests/dashboard.test.ts > loads the dashboard", "status": "failed" },
  {
    "automationKey": "tests/dashboard.test.ts > loads the dashboard",
    "status": "passed",
    "parameters": { "attempt": "2" }
  },
  { "automationKey": "tests/dashboard.test.ts > opens the menu", "status": "passed" },
  {
    "automationKey": "tests/dashboard.test.ts > opens the menu",
    "status": "passed",
    "parameters": { "repeat": "2" }
  },
  {
    "automationKey": "tests/dashboard.test.ts > opens the menu",
    "status": "passed",
    "parameters": { "repeat": "3" }
  }
]
```

```text
$ npx vitest run
[probara] Sending 5 results of 2 tests (4 passed, 1 failed, 0 skipped, 0 blocked)
[probara] Recorded 5 results (2 new cases, 0 unmatched) in R-1 (closed): https://app.probara.net/projects/SHOP/runs/R-1
```

- **What Probara shows.** The run case keeps the outcome of the last execution (`passed` here),
  and every execution stays in its history: a flaky test is a case whose history mixes failures
  and passes on the same run. The count of the log says it: more results than tests.
- **Parameters tell executions apart.** `attempt` is `2`, `3`... from the second attempt on, and
  `repeat` is `2`, `3`... from the second execution of `repeats` on. Each repeat retries on its
  own, so its attempts start again at 1: the second attempt of the second repeat has
  `{ "repeat": "2", "attempt": "2" }`. They win over `probara.parameters()` of the same name, and
  are never part of the key: every execution goes to the same case.
- **Each attempt starts empty.** `probara.*` calls, steps,
  [files](https://docs.probara.net/en/developers/ci-integrations/vitest/steps-and-attachments/#probaraattach) and
  [console output](https://docs.probara.net/en/developers/ci-integrations/vitest/steps-and-attachments/#console-output)
  belong to the attempt that made them, so a retry does not inherit what the first attempt
  recorded.
- **Errors per attempt.** A failed attempt's notes hold its own errors only, not those of the
  attempts before it.
- **No flaky flag.** The reporter does not mark attempts as flaky: the attempts themselves are the
  evidence. Vitest's JUnit output, and so `probara import junit`, has one result per test, however
  many attempts it took.

### Repeats

`repeats: 2` runs a test three times: once, then twice more. Vitest fails such a test when any of
its executions failed; Probara gets each execution with its own status, and no extra result for
Vitest's verdict, which would count a failure twice. So a repeated test whose last execution
passed shows `passed` in the run while Vitest failed it: its failed executions are in the case's
history.

### When retries run

Vitest retries a failed test right away, before the next test, with its `beforeEach` and
`afterEach` hooks around each attempt. Whatever sets the retries (the `retry` of the config, of a
`describe` or of a test, or `--retry`) and whatever its condition or delay, every attempt is a
result.

| Vitest                                        | What Probara gets                                             |
| --------------------------------------------- | ------------------------------------------------------------- |
| `retry`, `--retry`                            | One result per attempt                                        |
| `repeats`                                     | One result per execution, and per attempt of each execution   |
| `test.fails`                                  | One result: Vitest never retries a `test.fails` test          |
| `test.concurrent` with `retry`                | One result per attempt, each with its own errors              |
| A shard that ran with `--reporter=blob` alone | The final attempt only, with one warning at the merge (below) |

The attempts are recorded by the setup file the reporter adds to every Vitest project. Where it
does not run, Vitest still retries, but only the final attempt of each test can be sent, and one
warning says how many tests that was:
`Vitest retried or repeated 1 test whose earlier attempts were not recorded: only its final attempt is reported`.
That happens when Vitest runs without the reporter in its reporters, such as a shard run with
`--reporter=blob` alone on the command line, which replaces the reporters of the config: name the
reporter beside it, `--reporter=blob --reporter=@probara/vitest-reporter`
([the reporter goes beside `blob`](https://docs.probara.net/en/developers/ci-integrations/vitest/sharding/#the-reporter-goes-beside-blob)).

### Keeping retries out of your numbers

To send only the last attempt, do not retry in Vitest: no option of the reporter drops attempts. To
keep failures of retried tests from counting, map them ([status mapping](#status-mapping-and-filter)),
knowing it applies to every failure.

## Results file

When a report cannot be sent (Probara unreachable, the network lost, a refused report), its results
are lost unless you keep them. Set a results file, and the reporter writes what it could not send
to it at the end of the run; `probara import results` sends it later, into the same runs.

```ts
reporters: ['default', ['@probara/vitest-reporter', { resultsFile: 'probara-results.json' }]],
```

Or `PROBARA_RESULTS_FILE=probara-results.json`, relative to the directory `vitest` runs in.

```text
$ PROBARA_RESULTS_FILE=probara-results.json npx vitest run
[probara] Sending 3 results of 3 tests (3 passed, 0 failed, 0 skipped, 0 blocked)
[probara] 3 results were not sent: Probara answered 403 forbidden: <message from Probara>. No run was created or updated
[probara] Wrote the 3 results that were not sent to /work/shop/probara-results.json: send them with probara import results /work/shop/probara-results.json
```

Then, from any job that holds the token, send whatever was left:

```bash
npx @probara/cli import results 'probara-results*.json'
```

Quote the glob: `probara` expands it, and it matches the file and the siblings described below.
When nothing matches, nothing was left unsent: the import says so and exits 0.

- **Only when needed.** The file is written only when something was not sent: when every result
  went through, there is no file.
- **Into the same runs.** The file names the runs its results belong to: a run the failed report
  left open gets the rest of its results, and a run that was never created is created by the import
  with the same name and settings. Runs are closed like the reporter would have closed them.
- **Consumed by the import.** `probara import results` sends each file on its own, then deletes it
  (and its `<name>-attachments/` folder) once every result in it was sent, or rewrites it with only
  what is still unsent. Importing again never sends a result twice, so the import is safe to run in
  every pipeline. `--dry-run` never touches the files.
- **Never touches an earlier file.** When a file is already at that path (another shard's on a
  shared disk, or an earlier run's), the reporter leaves it as it is and writes to the first free
  sibling: `probara-results-2.json`, then `-3`, and so on, each with its own attachments folder.
  The log names the file written, and the glob imports them all, oldest first.
- **Never half written.** Each file is written whole to a temporary file in the same folder first (a
  dot name the glob never matches), then appears under its name at once: an import that runs at the
  same time sees the whole file or none of it.

A second run that cannot send while the first run's file is still there:

```text
$ PROBARA_RESULTS_FILE=probara-results.json npx vitest run
[probara] Sending 3 results of 3 tests (3 passed, 0 failed, 0 skipped, 0 blocked)
[probara] 3 results were not sent: Probara answered 403 forbidden: <message from Probara>. No run was created or updated
[probara] Wrote the 3 results that were not sent to /work/shop/probara-results.json: send them with probara import results /work/shop/probara-results.json
$ PROBARA_RESULTS_FILE=probara-results.json npx vitest run
[probara] Sending 3 results of 3 tests (3 passed, 0 failed, 0 skipped, 0 blocked)
[probara] 3 results were not sent: Probara answered 403 forbidden: <message from Probara>. No run was created or updated
[probara] Wrote the 3 results that were not sent to /work/shop/probara-results-2.json (/work/shop/probara-results.json already exists): send them with probara import results /work/shop/probara-results-2.json
```

### Attachments travel with the file

Every file a result carries is copied next to the results file, in its `<name>-attachments/` folder
(`probara-results-attachments/`), and the file names them relative to itself: what
`probara.attach()` copied, the files of Vitest's `context.annotate()`, the failure screenshots of
browser tests and the console output of `captureOutput`. The file never points into Vitest's
`attachmentsDir`, which a later run or a clean step may empty. Move or download the file and its
folder together, anywhere, and import it from there: a CI artifact another job downloads keeps its
attachments.

```ts
// tests/receipt.test.ts
import { probara } from '@probara/vitest-reporter';
import { test } from 'vitest';

test('prints the receipt', async () => {
  await probara.attach({ name: 'receipt', body: 'Total: 25.00' });
});
```

```text
$ PROBARA_ENABLED=false PROBARA_RESULTS_FILE=probara-results.json npx vitest run
[probara] Wrote 1 result to /work/shop/probara-results.json: send them with probara import results /work/shop/probara-results.json
$ npx @probara/cli import results probara-results.json
[probara] probara-results.json: 1 result
[probara] Results: 1 (1 passed, 0 failed, 0 skipped, 0 blocked)
[probara] Project: SHOP
[probara] Run: new run "Automated run 2026-09-29 14:05 UTC"
[probara] Base URL: https://app.probara.net
[probara] Missing cases: created
[probara] Attachments: on
[probara] Recorded 1 result (1 new case, 0 unmatched) in R-1 (closed): https://app.probara.net/projects/SHOP/runs/R-1
[probara] Attached 1 file to results (0 skipped, 0 failed)
[probara] Deleted probara-results.json and probara-results-attachments: every result was sent
```

An upload that fails during the import is not kept for another try: its result was recorded, so it
is no longer in the file, and the folder goes with the file once every result was sent. The import
logs it and exits 1.

### Reporting off: keep everything

With reporting off (`PROBARA_ENABLED=false`, or no token), the reporter writes **every** attempt to
the file, case links included. Run the tests where there is no token (a fork, a machine without
network), and import the file from a job that holds it:

```text
$ PROBARA_ENABLED=false PROBARA_RESULTS_FILE=probara-results.json npx vitest run
[probara] Wrote 3 results to /work/shop/probara-results.json: send them with probara import results /work/shop/probara-results.json
```

The same file shows what would be sent without sending it: see
[check what would be sent](https://docs.probara.net/en/developers/ci-integrations/vitest/troubleshooting/#check-what-would-be-sent).

### In CI

Keep the files and their `<name>-attachments/` folders as artifacts of the test job, and import them
in a later job (or step) that holds the token:

```yaml
- name: Run Vitest tests
  run: npx vitest run
  env:
    PROBARA_API_TOKEN: ${{ secrets.PROBARA_API_TOKEN }}
    PROBARA_RESULTS_FILE: probara-results.json
- name: Send what Probara did not get
  if: always()
  run: npx @probara/cli import results 'probara-results*.json'
  env:
    PROBARA_API_TOKEN: ${{ secrets.PROBARA_API_TOKEN }}
```

A later step in the same job works too, as above: when Probara was only briefly away, the import
sends the rest. When it still cannot, the import exits 1 and keeps what it could not send in each
file; the files it sent are gone.

A results file is trusted input: its attachments are uploaded from the paths it names, so import
only the files your own jobs wrote. The import leaves out, with a warning, an attachment whose
relative path leads outside the folder of the file; absolute paths are kept.

### In watch mode

A watch session re-run that is refused because its run was closed is kept in the file too, naming
that run, and the log says how to send it into an open run with `--run-ulid`
([when the run is closed during the session](https://docs.probara.net/en/developers/ci-integrations/vitest/configuration/#when-the-run-is-closed-during-the-session)).

### The format

Version 1 of a JSON document: the project, the runs and the settings of the reporter (the token
never), then each result as the reporter built it, before `statusMapping` (which applies when the
file is sent). The full format is in
[core's results file](https://github.com/cynch-me/probara-sdk/blob/main/packages/core/README.md#results-file),
and the import (several files and globs, the JSON summary, exit codes) is described in
[`probara import results`](https://github.com/cynch-me/probara-sdk/blob/main/packages/cli/docs/commands.md#probara-import-results).

## Assign failed results

`assignFailedTo` names the people who look into failures. Every report asks Probara to assign each
run case it leaves failed, and that has no assignee yet, to one of them, in turn, so a failing
nightly run lands on someone's list without anyone triaging it first.

```ts
reporters: [
  'default',
  [
    '@probara/vitest-reporter',
    { assignFailedTo: ['ana@example.com', 'bo@example.com', 'cy@example.com'] },
  ],
],
```

```ts
// tests/checkout.test.ts
import { expect, test } from 'vitest';

test('pays with a card', () => {
  expect('declined').toBe('paid');
});

test('pays with a voucher', () => {
  expect(0).toBe(25);
});

test('shows the total', () => {});
```

With `ana@example.com` and `bo@example.com` members of the project and `cy@example.com` not, the two
failed cases go to Ana and Bo, and Probara says one email matched no one:

```text
$ npx vitest run
[probara] Sending 3 results of 3 tests (1 passed, 2 failed, 0 skipped, 0 blocked)
[probara] Probara warned: assignFailedTo: 1 of 3 emails did not match a member who can be assigned in this project
[probara] Recorded 3 results (3 new cases, 0 unmatched) in R-1 (closed): https://app.probara.net/projects/SHOP/runs/R-1
```

Or from the environment: `PROBARA_ASSIGN_FAILED_TO=ana@example.com,bo@example.com`.

| Rule                                                      | Why                                                             |
| --------------------------------------------------------- | --------------------------------------------------------------- |
| Only run cases a report leaves `failed`                   | Passed, skipped and blocked results need no one                 |
| Only run cases without an assignee                        | An existing assignee is never replaced: someone already owns it |
| Only active members who can run tests in the project      | A case cannot be assigned to someone who cannot work on it      |
| The listed members take turns, in the order you list them | Failures are spread, not piled on the first name                |
| An email that matches no such member is skipped           | The results are still recorded; Probara's warning counts them   |

- **The last attempt decides.** A retried test that failed and then passed leaves its run case
  passed, so nothing is assigned; each failed attempt is still a result of the case.
- **After `statusMapping`.** The status that counts is the one Probara records: with
  `statusMapping: { failed: 'blocked' }`, nothing is failed, so nothing is assigned.
- **The warning never names the emails**, only how many were skipped: the log of a CI job is not the
  place to list who is or is not a member. Check the list against the project's members in
  Probara.
- **Turns start again with every report**: each report (a chunk of up to 500 results, a shard, a
  watch re-run) gives its first failed case to the first member listed.
- **Every report of the run asks**, with the same list: a run with several Probara projects asks in
  each project's run, and the shards of a shared run each assign the cases they leave failed.
- **A results file keeps the list.** `probara import results` sends it with the results, unless
  `--assign-failed-to` or `PROBARA_ASSIGN_FAILED_TO` names others.

Related: [Configuration](https://docs.probara.net/en/developers/ci-integrations/vitest/configuration/) ·
[Sharding and projects](https://docs.probara.net/en/developers/ci-integrations/vitest/sharding/) ·
[Troubleshooting](https://docs.probara.net/en/developers/ci-integrations/vitest/troubleshooting/#the-run-was-left-open) ·
Source: [runs.md](https://github.com/cynch-me/probara-sdk/blob/main/packages/vitest-reporter/docs/runs.md),
[statuses.md](https://github.com/cynch-me/probara-sdk/blob/main/packages/vitest-reporter/docs/statuses.md),
[retries.md](https://github.com/cynch-me/probara-sdk/blob/main/packages/vitest-reporter/docs/retries.md),
[results-file.md](https://github.com/cynch-me/probara-sdk/blob/main/packages/vitest-reporter/docs/results-file.md),
[assign-failed.md](https://github.com/cynch-me/probara-sdk/blob/main/packages/vitest-reporter/docs/assign-failed.md)