# Playwright reporter

> Send every Playwright test result to Probara from CI with @probara/playwright-reporter — requirements, install, configuration and your first run.

`@probara/playwright-reporter` is a [Playwright](https://playwright.dev) reporter that sends every
result of a `playwright test` run to Probara: each attempt with its status, steps, errors and
attachments, linked to its test case. Cases that do not exist yet are created, and the run is
closed once everything is in. You register it in `playwright.config` next to the reporter you read
in the terminal; there is no separate upload step.

- Package: [`@probara/playwright-reporter` on npm](https://www.npmjs.com/package/@probara/playwright-reporter)
  (current version `0.1.0`).
- Source and changelog: [probara-sdk on GitHub](https://github.com/cynch-me/probara-sdk/tree/main/packages/playwright-reporter)
  (Apache-2.0).
- Probara side: the [Playwright integration guide](https://docs.probara.net/en/guides/integrations/playwright/) covers the
  **Playwright** card, its app tokens and what Probara shows.

## Before you start

| You need                                              | Why                                                                                                          |
| ----------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ |
| Node.js 22.12 or later                                | The package's `engines` field. Playwright's main process runs the reporter.                                  |
| `@playwright/test` 1.42 or later                      | The peer range. Files of steps need 1.50, the error context of a failed attempt 1.51.                        |
| A paid plan (**Pro** or **Enterprise**, or any trial) | Reporting from CI is not part of the Free plan. On Free, Probara answers `403 forbidden`.                    |
| Someone with the **Manage integrations** permission   | Creating an app token is part of that permission, held by owners and admins.                                 |
| The **Project ID** of the project to report into      | A capital letter followed by capital letters or digits, such as `SHOP`. Any other value turns reporting off. |

## 1. Create an app token

The reporter signs in with an [app token](https://docs.probara.net/en/developers/authentication/#app-tokens): a token that
belongs to your organization, not to a person. It uses no seat and keeps working when the person
who created it leaves.

1. In **Workspace › Integrations**, select the **Playwright** card.
2. Open the **Access tokens** tab and select **Create an app token**.
3. Enter a **Token name** after what uses it (the repository or the pipeline) and select
   **Create token**.
4. Select **Copy token** in **Copy your new token**. It starts with `probara_app_` and is shown once: store it right away as
   a CI secret named `PROBARA_API_TOKEN`.

The reporter reads the token from the environment. Never put it in `playwright.config`, which is
committed.

## 2. Install the reporter

```bash
npm i -D @probara/playwright-reporter
```

Pin the version in `package.json` (or keep the lockfile) so a new release never changes a
pipeline by surprise: the reporter is in `0.x`, where a minor version may change options
([upgrading](https://docs.probara.net/en/developers/ci-integrations/playwright/migrating/#upgrading)).

## 3. Register it in `playwright.config`

Add it next to the reporter you read in the terminal, with your Project ID (or set
`PROBARA_PROJECT` in the environment instead):

```ts
// playwright.config.ts
import { defineConfig } from '@playwright/test';

export default defineConfig({
  reporter: [['list'], ['@probara/playwright-reporter', { projectId: 'SHOP' }]],
});
```

Everything else has a default: the run is named after the CI build, and its branch, commit and
build URL come from the CI. The package exports the type of its options, `ProbaraPlaywrightOptions`,
if you want the options checked:

```ts
// playwright.config.ts
import { defineConfig } from '@playwright/test';
import type { ProbaraPlaywrightOptions } from '@probara/playwright-reporter';

export default defineConfig({
  reporter: [
    ['list'],
    ['@probara/playwright-reporter', { projectId: 'SHOP' } satisfies ProbaraPlaywrightOptions],
  ],
});
```

## 4. Run your tests with the token

Give the step that runs `npx playwright test` the secret. In GitHub Actions:

```yaml
- name: Run Playwright tests
  run: npx playwright test
  env:
    PROBARA_API_TOKEN: ${{ secrets.PROBARA_API_TOKEN }}
```

The reporter logs on stderr, in `[probara]` lines, and ends with the link to the run:

```text
$ npx playwright test
[probara] Sending 2 results of 2 tests (2 passed, 0 failed, 0 skipped, 0 blocked)
[probara] Recorded 2 results (1 new case, 0 unmatched) in R-1 (closed): https://app.probara.net/projects/SHOP/runs/R-1
```

Without a token and a project, the reporter stays off and quiet: local runs, and builds that get
no secrets (such as pull requests from forks on most CIs), send nothing, with no extra setting. For a complete workflow on your CI, see
[CI providers](https://docs.probara.net/en/developers/ci-integrations/playwright/ci-providers/).

## What you see in Probara

- **An automated test run**, named after the CI build (`Playwright #42` on GitHub Actions), with
  the branch, the commit and a link to the build. It is closed when the reporter is done.
- **One result per attempt**: a retried test shows every attempt in the run case's history, and the
  run case keeps the last outcome.
- **Steps** from `test.step`, nested, with their status, duration and error, and the files attached
  in them.
- **Attachments**: screenshots, videos, traces, the error context and your own
  `testInfo.attach()` files.
- **New test cases** for tests that match none, in suites built from the file and the `describe`
  titles.
- **Attribution to the token**: runs and results show the app and the token name, such as
  **Playwright · e2e-ci**, not a person.

## How Playwright maps to Probara

| Playwright                               | In Probara                                                                |
| ---------------------------------------- | ------------------------------------------------------------------------- |
| The `playwright test` command            | One automated run, named after the CI build, closed at the end            |
| A test (per Playwright project)          | A test case, matched by its automation key or linked by a case id         |
| Each attempt (retries included)          | A result in the run, with its status, duration, start time and errors     |
| `test.step` and the hooks that run steps | The steps of the result, nested, with their status, duration and error    |
| Screenshots, videos, traces, attachments | Files of the result, or of the step they were attached in                 |
| `probara.*` calls in the test            | The title, suite, tags, fields and steps of a new case; result parameters |

The same test in two Playwright projects (`chromium` and `firefox`) is two cases: the project is
part of the automation key ([linking](https://docs.probara.net/en/developers/ci-integrations/playwright/linking/#automation-keys)).

## Reporting never breaks your test run

The reporter never throws into Playwright and never changes its exit code: a failed test fails the
command, a reporting problem does not. A wrong setting, Probara being unreachable or a refused
token is logged on stderr, never with the token, and the tests keep their outcome. There is no
option to fail the command on a reporting error; to keep what could not be sent, set a
[results file](https://docs.probara.net/en/developers/ci-integrations/playwright/results-and-retries/#results-file).

## Reference

| Page                                                                                      | What it covers                                                                         |
| ----------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- |
| [Configuration](https://docs.probara.net/en/developers/ci-integrations/playwright/configuration/)                 | Every option, its variable, type and default; precedence; network, proxies and retries |
| [Linking tests to cases](https://docs.probara.net/en/developers/ci-integrations/playwright/linking/)              | Annotations, case ids in titles, `probara.id()`, automation keys, missing cases        |
| [Metadata](https://docs.probara.net/en/developers/ci-integrations/playwright/metadata/)                           | Title, suite, comment, tags, fields, parameters, ignoring an attempt                   |
| [Steps and attachments](https://docs.probara.net/en/developers/ci-integrations/playwright/steps-and-attachments/) | `test.step`, hooks, `probara.step()`, screenshots, videos, traces, `probara.attach()`  |
| [Results and retries](https://docs.probara.net/en/developers/ci-integrations/playwright/results-and-retries/)     | Run options, statuses, status mapping, retries, the results file                       |
| [Sharding and several projects](https://docs.probara.net/en/developers/ci-integrations/playwright/sharding/)      | One run for every shard, `merge-reports`, results for several Probara projects         |
| [CI providers](https://docs.probara.net/en/developers/ci-integrations/playwright/ci-providers/)                   | GitHub Actions, GitLab CI, Jenkins, CircleCI, Azure Pipelines, Bitbucket, Buildkite    |
| [Troubleshooting](https://docs.probara.net/en/developers/ci-integrations/playwright/troubleshooting/)             | Problems and their solutions, the log, `PROBARA_DEBUG`, checking what would be sent    |
| [Migrating and upgrading](https://docs.probara.net/en/developers/ci-integrations/playwright/migrating/)           | From Qase, from `probara import junit`, versioning and the automation key contract     |

Next: [configure the reporter](https://docs.probara.net/en/developers/ci-integrations/playwright/configuration/), or go
back to the [CI integrations overview](https://docs.probara.net/en/developers/ci-integrations/). Running Jest, Cypress or
Vitest instead? See the [Jest reporter](https://docs.probara.net/en/developers/ci-integrations/jest/), the
[Cypress reporter](https://docs.probara.net/en/developers/ci-integrations/cypress/) or the
[Vitest reporter](https://docs.probara.net/en/developers/ci-integrations/vitest/).