# Playwright results and retries

> The run @probara/playwright-reporter creates, how attempts become statuses, retries and flaky tests, assigning failures, and the results file for what could not be sent.

Each `playwright test` reports into one Probara run: a new automated run by default, closed at the
end, or an existing one. This page covers that run, the status of each attempt, retries, and what
happens to results that could not be sent.

## Run options

These options describe the run the reporter creates:

```ts
reporter: [
  [
    '@probara/playwright-reporter',
    {
      run: {
        name: 'Nightly regression',
        description: 'Every night on main, against staging.',
        environment: 'staging',
        milestone: 'Sprint 12',
        plan: 'PLAN-2',
        configurations: [
          { group: 'Browser', name: 'Chromium' },
          { group: 'OS', name: 'Linux' },
        ],
        tags: ['nightly', 'e2e'],
      },
    },
  ],
],
```

| Option               | Variable                  | What it sets                                                                                |
| -------------------- | ------------------------- | ------------------------------------------------------------------------------------------- |
| `run.name`           | `PROBARA_RUN_NAME`        | The name. Default: the CI build name (`CI #42`), else `Automated run <date> <time> UTC`     |
| `run.description`    | `PROBARA_RUN_DESCRIPTION` | The description, up to 2000 characters (longer is cut, with a warning)                      |
| `run.environment`    | `PROBARA_ENVIRONMENT`     | The environment, by name: created in the project when none has that name                    |
| `run.milestone`      | `PROBARA_MILESTONE`       | The milestone, by display id (`M-3`) or exact name                                          |
| `run.plan`           | `PROBARA_PLAN`            | The test plan, by display id (`PLAN-2`) or exact name: the run starts with the plan's cases |
| `run.configurations` | `PROBARA_CONFIGURATIONS`  | Configuration values by group and name (`Browser=Chromium,OS=Linux`), each group once       |
| `run.tags`           | `PROBARA_RUN_TAGS`        | Tags (`nightly,e2e`), up to 50                                                              |

### Names or ULIDs

The environment, milestone and configurations also take ULIDs: `run.environmentId`,
`run.milestoneId` and `run.configurationUlids` (`PROBARA_ENVIRONMENT_ID`, `PROBARA_MILESTONE_ID`,
`PROBARA_CONFIGURATION_ULIDS`). Set one form of each, never both: both is a configuration problem
that turns reporting off. Names are the form to use with an app token, which cannot look ULIDs up.

- An unknown environment **name** creates the environment.
- An unknown milestone, plan or configuration makes Probara refuse the first report (`422`, naming
  the field): nothing is recorded, and the log says why. Check the names in Probara.

### Run source

A run records where it came from: the branch, the commit and the CI build URL, read from the
variables of the CI. The first provider that matches wins:

| CI                  | Detected by              | Run name                                          | Branch, commit and build URL from                                                                 |
| ------------------- | ------------------------ | ------------------------------------------------- | ------------------------------------------------------------------------------------------------- |
| GitHub Actions      | `GITHUB_ACTIONS=true`    | `GITHUB_WORKFLOW #GITHUB_RUN_NUMBER`              | `GITHUB_HEAD_REF` or `GITHUB_REF_NAME`, `GITHUB_SHA`, the run URL                                 |
| GitLab CI           | `GITLAB_CI=true`         | `CI_PROJECT_NAME #CI_PIPELINE_IID`                | `CI_MERGE_REQUEST_SOURCE_BRANCH_NAME` or `CI_COMMIT_REF_NAME`, `CI_COMMIT_SHA`, `CI_PIPELINE_URL` |
| CircleCI            | `CIRCLECI=true`          | `CIRCLE_PROJECT_REPONAME #CIRCLE_BUILD_NUM`       | `CIRCLE_BRANCH`, `CIRCLE_SHA1`, `CIRCLE_BUILD_URL`                                                |
| Azure Pipelines     | `TF_BUILD=True`          | `BUILD_DEFINITIONNAME #BUILD_BUILDNUMBER`         | The source branch, `BUILD_SOURCEVERSION`, the build results URL                                   |
| Jenkins             | `JENKINS_URL` set        | `JOB_NAME #BUILD_NUMBER`                          | `BRANCH_NAME` or `GIT_BRANCH`, `GIT_COMMIT`, `BUILD_URL`                                          |
| Bitbucket Pipelines | `BITBUCKET_BUILD_NUMBER` | `BITBUCKET_REPO_SLUG #BITBUCKET_BUILD_NUMBER`     | `BITBUCKET_BRANCH`, `BITBUCKET_COMMIT`, the pipeline URL                                          |
| Buildkite           | `BUILDKITE=true`         | `BUILDKITE_PIPELINE_SLUG #BUILDKITE_BUILD_NUMBER` | `BUILDKITE_BRANCH`, `BUILDKITE_COMMIT`, `BUILDKITE_BUILD_URL`                                     |

A tag build sends no branch. Invalid values are dropped, never sent. Override a field with `source`
or its variable, or send none:

```ts
reporter: [['@probara/playwright-reporter', { source: { branch: 'release/2.4' } }]],
```

| Option            | Variable            | Default              |
| ----------------- | ------------------- | -------------------- |
| `source.branch`   | `PROBARA_BRANCH`    | detected from the CI |
| `source.commit`   | `PROBARA_COMMIT`    | detected from the CI |
| `source.buildUrl` | `PROBARA_BUILD_URL` | detected from the CI |
| `source: false`   | —                   | sends no source      |

### An existing run

`run.ulid` (`PROBARA_RUN_ULID`) reports into a run that exists, such as one `probara run create`
made for the shards of a pipeline ([sharding](https://docs.probara.net/en/developers/ci-integrations/playwright/sharding/)).
The options of a new run are then ignored, with one warning: the run keeps its own name,
environment, milestone and configurations.

### Closing the run

A run the reporter creates is closed at the end, once every result and file was sent; a reused one
stays open. `closeRun` (`PROBARA_CLOSE_RUN`) decides for every run. When a report fails, the run is
left open: the log names it, and what was not sent can go to a [results file](#results-file).

## Assign failed results

`assignFailedTo` (`PROBARA_ASSIGN_FAILED_TO=ana@example.com,bo@example.com`) names up to 20 members
of the organization by email. Every report of the run asks Probara to assign each run case it
leaves failed and without an assignee to one of them, in turn; a run case that already has an
assignee keeps it.

- Emails are trimmed and counted once each, ignoring case. A value that is not an email, or more
  than 20 emails, is a configuration problem that turns reporting off.
- An email that matches no member who can be assigned in the project is not an error: Probara
  counts them in a warning, which the reporter logs without naming them.

## Statuses

Each attempt is sent with Playwright's own verdict: what the attempt did compared with what the
test expected.

| The attempt                                                                 | Status    |
| --------------------------------------------------------------------------- | --------- |
| Passed                                                                      | `passed`  |
| Failed: an assertion, an error, a hook or a fixture threw                   | `failed`  |
| Timed out                                                                   | `failed`  |
| Interrupted (the run stopped: `Ctrl+C`, `--max-failures`, a global timeout) | `failed`  |
| `test.skip()`, `test.fixme()`, a skipped `describe`                         | `skipped` |
| `test.fail()`, and it failed as expected                                    | `passed`  |
| `test.fail()`, but it passed                                                | `failed`  |

The reporter never sends `blocked` on its own: map a status to it ([below](#status-mapping-and-filter))
when you need it.

```ts
// tests/statuses.spec.ts
import { expect, test } from '@playwright/test';

test('passes', async () => {});

test('fails', async () => {
  expect(1 + 1).toBe(3);
});

test('is skipped on CI', async () => {
  test.skip(true, 'Needs the payment sandbox');
});

test.fixme('is not written yet', async () => {});

test('is a known bug', async () => {
  test.fail(true, 'SHOP-99: totals round down');
  expect(0.1 + 0.2).toBe(0.3);
});
```

```json
[
  { "automationKey": "statuses.spec.ts > passes", "status": "passed" },
  { "automationKey": "statuses.spec.ts > fails", "status": "failed" },
  {
    "automationKey": "statuses.spec.ts > is skipped on CI",
    "status": "skipped",
    "notes": "Skipped: Needs the payment sandbox"
  },
  { "automationKey": "statuses.spec.ts > is not written yet", "status": "skipped" },
  { "automationKey": "statuses.spec.ts > is a known bug", "status": "passed" }
]
```

- **Errors** go in the notes of the result: every error of the attempt, with its message and
  stack, after a [comment](https://docs.probara.net/en/developers/ci-integrations/playwright/metadata/#comment) if the test
  wrote one.
- **Skip reasons** go in the notes as `Skipped: <reason>`, the same line `probara import junit`
  writes, from `test.skip(condition, reason)` or `test.fixme(condition, reason)`.
- **Expected failures** are Playwright's: a `test.fail()` test that fails does what it should, so
  it passes, and the run stays green in Probara as it does in Playwright. When the bug is fixed,
  the test passes, Playwright fails it, and so does Probara, reminding you to remove `test.fail()`.
- **Timeouts** keep Playwright's message (`Test timeout of 30000ms exceeded.`) in the notes.

### Status mapping and filter

`statusMapping` sends a status as another; `statusFilter` sends no result with the given statuses,
after the mapping. Both take `passed`, `failed`, `skipped` and `blocked`:

```ts
reporter: [
  [
    '@probara/playwright-reporter',
    { statusMapping: { failed: 'blocked' }, statusFilter: ['skipped'] },
  ],
],
```

With one passing, one failing and one skipped test:

```text
$ npx playwright test
[probara] Sending 2 results of 2 tests (1 passed, 0 failed, 0 skipped, 1 blocked); 1 left out by statusFilter
[probara] Recorded 2 results (2 new cases, 0 unmatched) in R-1 (closed): https://app.probara.net/projects/SHOP/runs/R-1
[probara] Filtered out 1 result by their status (statusFilter): not sent
[probara] Attached 1 file to results (0 skipped, 0 failed)
```

As variables: `PROBARA_STATUS_MAPPING=failed=blocked,skipped=passed` and
`PROBARA_STATUS_FILTER=skipped,blocked`. A value that is not a status, or a status mapped twice, is
a configuration problem that turns reporting off. The mapping changes what Probara records only:
Playwright's exit code stays the tests' own.

## Retries and flaky tests

Every attempt of a test is a result of its own. A test retried twice sends three results, in the
order they ran, each with its own status, duration, errors, steps and files.

```ts
// playwright.config.ts (excerpt)
retries: 1,
```

```ts
// tests/flaky.spec.ts
import { expect, test } from '@playwright/test';

test('loads the dashboard', async ({}, testInfo) => {
  // Fails the first time, passes on the retry.
  expect(testInfo.retry).toBe(1);
});
```

```json
[
  { "automationKey": "flaky.spec.ts > loads the dashboard", "status": "failed" },
  { "automationKey": "flaky.spec.ts > loads the dashboard", "status": "passed" }
]
```

```text
$ npx playwright test
[probara] Sending 2 results of 1 test (1 passed, 1 failed, 0 skipped, 0 blocked)
[probara] Recorded 2 results (1 new case, 0 unmatched) in R-1 (closed): https://app.probara.net/projects/SHOP/runs/R-1
[probara] Attached 1 file to results (0 skipped, 0 failed)
```

The file is the error context Playwright wrote for the failed attempt.

- **What Probara shows.** The run case keeps the last outcome (`passed` here), and every attempt
  stays in its history: a flaky test is a case whose history mixes failures and passes on the same
  run. The log says it too: more results than tests.
- **No flaky flag.** The reporter does not mark attempts as flaky: the attempts themselves are the
  evidence, and Playwright's own verdict (flaky, when a retry passed) stays in Playwright's reports.
- **Each attempt starts empty.** `probara.*` metadata applies to the attempt that called it, so a
  retry that stops earlier does not inherit the metadata of the first attempt.
- **Order.** Results are sent in the order Playwright ends the attempts; since the run case keeps
  the last outcome, the final retry is the one it shows.

To send only the last attempt, filter in Playwright instead: no option of the reporter drops
attempts. To keep failures of retried tests out of your numbers, [map them](#status-mapping-and-filter),
knowing the mapping applies to every failure.

## Results file

When a report cannot be sent (Probara unreachable, the network lost, a refused report), its results
are lost unless you keep them. Set a results file, and the reporter writes what it could not send
to it at the end of the run; `probara import results` sends it later, into the same runs.

```ts
reporter: [['@probara/playwright-reporter', { resultsFile: 'probara-results.json' }]],
```

Or `PROBARA_RESULTS_FILE=probara-results.json`, relative to the directory Playwright runs in.

### Keep what could not be sent, send it later

```text
$ PROBARA_RESULTS_FILE=probara-results.json npx playwright test
[probara] Sending 2 results of 2 tests (2 passed, 0 failed, 0 skipped, 0 blocked)
[probara] 2 results were not sent: Probara answered 403 forbidden: <message from Probara>. No run was created or updated
[probara] Wrote the 2 results that were not sent to /work/shop/probara-results.json: send them with probara import results /work/shop/probara-results.json
```

Then, from any job that holds the token, send whatever was left with the
[Probara CLI](https://docs.probara.net/en/developers/cli/):

```bash
npx @probara/cli import results 'probara-results*.json'
```

Quote the glob: `probara` expands it, and it matches the file and the siblings described below.
When nothing matches, nothing was left unsent: the import says so and exits 0.

- **Only when needed.** The file is written only when something was not sent: when every result
  went through, there is no file.
- **Into the same runs.** The file names the runs its results belong to: a run the failed report
  left open gets the rest of its results, and a run that was never created is created by the import
  with the same name and settings. Runs are closed as the reporter would have closed them.
- **Consumed by the import.** `probara import results` sends each file on its own, then deletes it
  (and its `<name>-attachments/` folder) once every result in it was sent, or rewrites it with only
  what is still unsent. Importing again never sends a result twice, so the import is safe to run in
  every pipeline. `--dry-run` never touches the files.
- **Never touches an earlier file.** When a file is already at that path (another shard's on a
  shared disk, or an earlier run's), the reporter leaves it as it is and writes to the first free
  sibling: `probara-results-2.json`, then `-3`, and so on, each with its own attachments folder.
  The log names the file written, and the glob imports them all, oldest first.
- **Never half written.** Each file is written whole to a temporary file in the same folder first
  (a dot name the glob never matches), then appears under its name at once: an import running at
  the same time sees the whole file or none of it, and a run stopped halfway leaves no partial file.
- **Attachments by path.** Files are referenced where Playwright wrote them, by absolute path: keep
  the output folder (`test-results/`) there until the file is sent. In-memory bodies are written
  next to the file, in `<name>-attachments/`, and referenced relative to it, so the file and that
  folder can move together. An upload that fails during the import is not kept for another try:
  its result was recorded, so it is no longer in the file. The import logs it and exits 1.

A second run that cannot send while the first run's file is still there writes a sibling:

```text
$ PROBARA_RESULTS_FILE=probara-results.json npx playwright test
[probara] Sending 2 results of 2 tests (2 passed, 0 failed, 0 skipped, 0 blocked)
[probara] 2 results were not sent: Probara answered 403 forbidden: <message from Probara>. No run was created or updated
[probara] Wrote the 2 results that were not sent to /work/shop/probara-results-2.json (/work/shop/probara-results.json already exists): send them with probara import results /work/shop/probara-results-2.json
```

### Reporting off: keep everything

With reporting off (`PROBARA_ENABLED=false`, or no token), the reporter writes **every** attempt to
the file, case links included. Run the tests where there is no token (a fork, a machine without
network), and import the file from a job that holds one:

```text
$ PROBARA_ENABLED=false PROBARA_RESULTS_FILE=probara-results.json npx playwright test
[probara] Wrote 2 results to /work/shop/probara-results.json: send them with probara import results /work/shop/probara-results.json
```

The same file shows what would be sent without sending it: see
[check what would be sent](https://docs.probara.net/en/developers/ci-integrations/playwright/troubleshooting/#check-what-would-be-sent).

### In CI

Keep the files, their `<name>-attachments/` folders and the output folder as artifacts of the test
job, and import them in a later job that holds the token. A results file and its folder can be
downloaded anywhere; the output folder's files are found at the absolute path they had, so restore
it at that path, or import in the same job:

```yaml
- name: Run Playwright tests
  run: npx playwright test
  env:
    PROBARA_API_TOKEN: ${{ secrets.PROBARA_API_TOKEN }}
    PROBARA_RESULTS_FILE: probara-results.json
- name: Send what Probara did not get
  if: always()
  run: npx --yes @probara/cli@0.2 import results 'probara-results*.json'
  env:
    PROBARA_API_TOKEN: ${{ secrets.PROBARA_API_TOKEN }}
    PROBARA_ENABLED: ${{ secrets.PROBARA_API_TOKEN != '' }}
```

When Probara was only briefly away, the import sends the rest. When it still cannot, the import
exits 1 and keeps what it could not send in each file; the files it sent are gone.

A results file is trusted input: the import uploads the files it names, from any path the job can
read, so import only the files your own jobs wrote. An attachment whose relative path leads outside
the folder of the file is left out with a warning.

### The format

The file is version 1 of a JSON document: the project, the runs and the settings of the reporter
(never the token), then each result as the reporter built it, before `statusMapping`, which applies
when the file is sent. The full format is in
[`@probara/core`'s results file](https://github.com/cynch-me/probara-sdk/blob/main/packages/core/README.md#results-file),
and the import (several files and globs, the JSON summary, exit codes) in
[`probara import results`](https://github.com/cynch-me/probara-sdk/blob/main/packages/cli/docs/commands.md#probara-import-results).

Next: [one run for sharded pipelines](https://docs.probara.net/en/developers/ci-integrations/playwright/sharding/), or
[troubleshooting](https://docs.probara.net/en/developers/ci-integrations/playwright/troubleshooting/#the-run-was-left-open).