# Troubleshoot the Playwright reporter

> Fix the common problems of @probara/playwright-reporter — nothing reported, 401 and 403 answers, runs left open, unmatched results, missing attachments — and read its logs.

The reporter logs on stderr, in `[probara]` lines; [`PROBARA_DEBUG=true`](#debug-logging) adds
every request. Whatever the problem, Playwright's exit code stays the tests' own: the reporter
never throws into Playwright and has no option to fail the command on a reporting error.

## Where problems show

| You see                                        | Look at                                                                                    |
| ---------------------------------------------- | ------------------------------------------------------------------------------------------ |
| No `[probara]` line at all                     | [Nothing is reported](#nothing-is-reported-and-nothing-is-logged)                          |
| `... Probara reporting is off ...`             | [A configuration problem](#reporting-is-off-a-configuration-problem)                       |
| `... not sent ...`                             | [401 or 403](#probara-answers-401-or-403), [the run was left open](#the-run-was-left-open) |
| `... not recorded (...)`                       | [Unmatched results](#results-were-not-recorded-unmatched)                                  |
| `[probara] probara.x() ...` in a test's output | [A helper warns](#a-helper-warns-that-it-only-works-while-a-test-runs)                     |

## Nothing is reported, and nothing is logged

**Why.** Without a token and a project, the reporter stays off and quiet, on purpose: local runs
and fork builds send nothing. It says so only at debug:

```text
$ PROBARA_DEBUG=true npx playwright test
[probara] Probara reporting is not configured: set PROBARA_API_TOKEN and PROBARA_PROJECT to enable it
```

**Solution.** Give the step that runs `npx playwright test` the `PROBARA_API_TOKEN` secret, and set
the project (`projectId` or `PROBARA_PROJECT`). Check that the reporter is registered in the config
Playwright uses (`reporter: [...]`), or in `--reporter` when you pass one: `--reporter` replaces the
config's reporters. With `merge-reports`, the token and the reporter belong to the merge job
([sharding](https://docs.probara.net/en/developers/ci-integrations/playwright/sharding/#pattern-1-merge-the-blob-reports)).

## Reporting is off: a configuration problem

```text
$ PROBARA_RUN_ULID=R-12 npx playwright test
[probara] Probara reporting is off: PROBARA_RUN_ULID is not a ULID
```

**Why.** A setting has a wrong value, or only one of the token and the project is set. Each problem
is logged, naming the option or variable at fault (never its value), and nothing is sent.

**Solution.** Fix the setting the line names
([options](https://docs.probara.net/en/developers/ci-integrations/playwright/configuration/#options)).

## Probara answers 401 or 403

```text
$ npx playwright test
[probara] Sending 2 results of 2 tests (2 passed, 0 failed, 0 skipped, 0 blocked)
[probara] 2 results were not sent: Probara answered 401 unauthorized: <message from Probara>. No run was created or updated
```

**Why.** `401 unauthorized`: the token is wrong, revoked, or meant for another Probara host
(`baseUrl`). `403 forbidden`: the token cannot report to that project, or the organization is on
the Free plan, where reporting from CI is not available.

**Solution.** Create a new app token from the **Playwright** card
([get started](https://docs.probara.net/en/developers/ci-integrations/playwright/#1-create-an-app-token)) and update the
secret; check the Project ID; check the organization's plan. The
[app token reference](https://docs.probara.net/en/developers/authentication/#app-tokens) lists the exact calls a token may
make.

## The run was left open

The error line ends with `The run R-4 was left open: <url>`, or, when not even the first report
went through, `No run was created or updated`.

**Why.** A report failed after its retries (Probara unreachable, a refused report): the reports
after it are not sent, and a run that already has results stays open, so nothing looks complete
when it is not. The last line names the run.

**Solution.** Set a [results file](https://docs.probara.net/en/developers/ci-integrations/playwright/results-and-retries/#results-file):
what was not sent goes there, with the run it belongs to, and `probara import results` sends it
into that run and closes it. Otherwise close the run by hand with
`npx @probara/cli run close --run-ulid <ulid>`, and run the tests again.

## A job reports after its run was closed

In GitHub Actions the jobs of one workflow run share a run, and a final job closes it. A job that
reports after that close is refused with `409 ci_run_closed`, and the reporter logs that the run was
already closed. It sends nothing more, and the tests keep their exit code.

**Why.** A reporting job is missing from the final job's `needs`, so the final job closed the run
before that job reported. Or a parallel job closed the run at its own end because it lacked
`PROBARA_CLOSE_RUN: false`, and every job after it is refused the same way.

**Solution.** Keep every reporting job in the `needs` of the final job, and set
`PROBARA_CLOSE_RUN: false` on each parallel job. The
[parallel jobs](https://docs.probara.net/en/developers/ci-integrations/playwright/ci-providers/#parallel-jobs-one-run-closed-by-the-final-job)
section has the complete workflow.

## Results were not recorded (unmatched)

```text
$ PROBARA_CREATE_MISSING_CASES=false npx playwright test
[probara] Sending 2 results of 2 tests (2 passed, 0 failed, 0 skipped, 0 blocked)
[probara] Recorded 1 result (0 new cases, 1 unmatched) in R-1 (closed): https://app.probara.net/projects/SHOP/runs/R-1
[probara] 1 result was not recorded (case_not_found): cart.spec.ts > cart > removes an item
```

**Why.** A result names a case that does not exist (a typo in an id, a case of another project), or
matches no case while `createMissingCases` is off. Probara records the others and says which were
left out, with the reason.

**Solution.** Fix the id, create the case, or let the reporter create missing cases (the default).

## A case of another project is not sent

**Why.** A test is linked to `WEB-3` while `WEB` is neither the project nor one of `projects`: the
result would land in the wrong project. One warning per project says so.

**Solution.** List the project in `projects` (`PROBARA_PROJECTS=WEB`)
([several projects](https://docs.probara.net/en/developers/ci-integrations/playwright/sharding/#several-probara-projects)).

## Every run creates new cases

**Why.** The automation keys changed: a test, `describe` or file was renamed or moved, a Playwright
project was renamed, or `rootDir` (`testDir`) moved
([what changes a key](https://docs.probara.net/en/developers/ci-integrations/playwright/linking/#automation-keys)). Running
from another directory does not change keys: they are relative to Playwright's `rootDir`.

**Solution.** Link the tests to their cases with ids
([linking](https://docs.probara.net/en/developers/ci-integrations/playwright/linking/)), so a rename keeps them. Before a
refactor, [compare the keys](#check-what-would-be-sent).

## An attachment is missing

**Why.** The log names every file it skipped and why: over 32 MiB, an image over 10 MiB or 8192 px,
a refused content type, more than 20 files in one result, or a file that was gone when it uploaded
([limits](https://docs.probara.net/en/developers/ci-integrations/playwright/steps-and-attachments/#limits-and-what-is-skipped)).
A file attached inside a step goes to that step, not to the result. Videos and traces only exist
when `use` turns them on.

**Solution.** Keep `test-results/` until the reporter is done; lower the size of screenshots
(`fullPage: false`); attach fewer files per result.

## A helper warns that it only works while a test runs

**Why.** A `probara.*` helper was called outside a test: at the top of a file, in `beforeAll`,
`afterAll` or a worker fixture. Nothing is recorded, and the test goes on.

**Solution.** Call it in the test, `beforeEach`, `afterEach` or a test fixture
([where to call the helpers](https://docs.probara.net/en/developers/ci-integrations/playwright/metadata/#where-to-call-the-helpers)).

## Probara answers 429 (too many requests)

**Why.** The organization's rate limit (60 requests per minute on Free, 1,200 on Pro, and 3,000 on Enterprise) is shared by every token
and pipeline. Attachments cost two requests per result
([rate limit](https://docs.probara.net/en/developers/ci-integrations/playwright/configuration/#rate-limit)).

**Solution.** The reporter waits and retries on its own. For large suites, keep traces and videos
for failures only, lower `attachmentConcurrency`, or spread sharded pipelines in time.

## merge-reports sends nothing

**Why.** The reporter is not among the merge's reporters, or the merge job has no token.

**Solution.** Pass `--reporter @probara/playwright-reporter` (or a `--config` that registers it) to
`npx playwright merge-reports`, and give that job `PROBARA_API_TOKEN` and the project
([sharding](https://docs.probara.net/en/developers/ci-integrations/playwright/sharding/#pattern-1-merge-the-blob-reports)).

## The log

The reporter writes to **stderr** only, every line prefixed with `[probara]`; stdout belongs to
Playwright's own reporters (`list`, `dot`...). A run that reports logs two lines:

```text
$ npx playwright test
[probara] Sending 2 results of 2 tests (2 passed, 0 failed, 0 skipped, 0 blocked)
[probara] Recorded 2 results (1 new case, 0 unmatched) in R-1 (closed): https://app.probara.net/projects/SHOP/runs/R-1
```

| Line                                 | What it says                                                                                                     |
| ------------------------------------ | ---------------------------------------------------------------------------------------------------------------- |
| `Sending N results of M tests (...)` | Before anything is sent: one result per attempt and case, by status after `statusMapping`, and what was left out |
| `Recorded N results (...) in R-1`    | Per run: what Probara recorded, the cases it created, the results it could not match, and the link               |
| `N results were not recorded (...)`  | The unmatched results, by reason, with their cases or keys                                                       |
| `Attached N files to results (...)`  | Uploads: attached, skipped and failed                                                                            |
| `N results were not sent: ...`       | A failed report: the reason, and the run left open                                                               |
| `Wrote ... to <file>`                | What went to the [results file](https://docs.probara.net/en/developers/ci-integrations/playwright/results-and-retries/#results-file)     |

A warning that repeats in every test (a malformed helper call, a skipped attachment) is logged the
first time, naming the test, and at debug afterwards. The token never appears in any line, even
when Probara's answer echoes it. Log lines are for people: do not parse them, since a minor version
may change them.

## Debug logging

`debug: true` (`PROBARA_DEBUG=true`) adds a line for every request (method, path, status, time)
and every retry, and says why reporting is off when it is:

```bash
PROBARA_DEBUG=true npx playwright test
```

Keep it off in normal runs: it is verbose, though it never logs the token or the bodies.

## Check what would be sent

The reporter has no dry run of its own, but a results file is one: run the tests with reporting off
and a results file, then let the CLI print what it would send. Nothing reaches Probara, and no
token is needed:

```text
$ PROBARA_ENABLED=false PROBARA_PROJECT=SHOP PROBARA_RESULTS_FILE=probara-results.json npx playwright test
[probara] Wrote 2 results to /work/shop/probara-results.json: send them with probara import results /work/shop/probara-results.json
$ npx @probara/cli import results probara-results.json --dry-run
passed	SHOP-12	cart.spec.ts > cart > adds an item
passed	-	cart.spec.ts > cart > removes an item
Total: 2 results from 1 file (2 passed, 0 failed, 0 skipped, 0 blocked)
```

The first command logs on stderr; the dry run prints on stdout, one line per result: the status,
the case (`-` when none is named) and the automation key. `--dry-run` never touches the file:
delete it afterwards, or import it for real, because a file left there makes the next run write
`probara-results-2.json`. `--json` prints the whole document instead, each entry as Probara would
receive it ([`probara import results`](https://github.com/cynch-me/probara-sdk/blob/main/packages/cli/docs/commands.md#probara-import-results)).

Use it to compare keys before a refactor or an upgrade: write one file per run, and diff the two
dry runs. Give each run its own file name and start without them: with a file already at the path,
a run writes to a sibling (`before-2.json`), and the dry run of `before.json` would show an older
run, hiding the difference.

```bash
rm -f before.json after.json
PROBARA_ENABLED=false PROBARA_PROJECT=SHOP PROBARA_RESULTS_FILE=before.json npx playwright test
# Change the code, or upgrade the reporter, then:
PROBARA_ENABLED=false PROBARA_PROJECT=SHOP PROBARA_RESULTS_FILE=after.json npx playwright test
npx @probara/cli import results before.json --dry-run > before.txt
npx @probara/cli import results after.json --dry-run > after.txt
diff before.txt after.txt
```

Any line that changed is a test whose key or case link changed: it would get a new case.

Next: [network, retries and proxies](https://docs.probara.net/en/developers/ci-integrations/playwright/configuration/#network),
or the [CLI troubleshooting guide](https://github.com/cynch-me/probara-sdk/blob/main/packages/cli/docs/troubleshooting.md)
for `probara import results`, `run create` and `run close`.