Troubleshoot the Playwright reporter

The reporter logs on stderr, in [probara] lines; PROBARA_DEBUG=true adds every request. Whatever the problem, Playwright’s exit code stays the tests’ own: the reporter never throws into Playwright and has no option to fail the command on a reporting error.

Where problems show

You see	Look at
No [probara] line at all	Nothing is reported
... Probara reporting is off ...	A configuration problem
... not sent ...	401 or 403, the run was left open
... not recorded (...)	Unmatched results
[probara] probara.x() ... in a test’s output	A helper warns

Nothing is reported, and nothing is logged

Why. Without a token and a project, the reporter stays off and quiet, on purpose: local runs and fork builds send nothing. It says so only at debug:

$ PROBARA_DEBUG=true npx playwright test
[probara] Probara reporting is not configured: set PROBARA_API_TOKEN and PROBARA_PROJECT to enable it

Solution. Give the step that runs npx playwright test the PROBARA_API_TOKEN secret, and set the project (projectId or PROBARA_PROJECT). Check that the reporter is registered in the config Playwright uses (reporter: [...]), or in --reporter when you pass one: --reporter replaces the config’s reporters. With merge-reports, the token and the reporter belong to the merge job (sharding).

Reporting is off: a configuration problem

$ PROBARA_RUN_ULID=R-12 npx playwright test
[probara] Probara reporting is off: PROBARA_RUN_ULID is not a ULID

Why. A setting has a wrong value, or only one of the token and the project is set. Each problem is logged, naming the option or variable at fault (never its value), and nothing is sent.

Solution. Fix the setting the line names (options).

Probara answers 401 or 403

$ npx playwright test
[probara] Sending 2 results of 2 tests (2 passed, 0 failed, 0 skipped, 0 blocked)
[probara] 2 results were not sent: Probara answered 401 unauthorized: <message from Probara>. No run was created or updated

Why. 401 unauthorized: the token is wrong, revoked, or meant for another Probara host (baseUrl). 403 forbidden: the token cannot report to that project, or the organization is on the Free plan, where reporting from CI is not available.

Solution. Create a new app token from the Playwright card (get started) and update the secret; check the Project ID; check the organization’s plan. The app token reference lists the exact calls a token may make.

The run was left open

The error line ends with The run R-4 was left open: <url>, or, when not even the first report went through, No run was created or updated.

Why. A report failed after its retries (Probara unreachable, a refused report): the reports after it are not sent, and a run that already has results stays open, so nothing looks complete when it is not. The last line names the run.

Solution. Set a results file: what was not sent goes there, with the run it belongs to, and probara import results sends it into that run and closes it. Otherwise close the run by hand with npx @probara/cli run close --run-ulid <ulid>, and run the tests again.

A job reports after its run was closed

In GitHub Actions the jobs of one workflow run share a run, and a final job closes it. A job that reports after that close is refused with 409 ci_run_closed, and the reporter logs that the run was already closed. It sends nothing more, and the tests keep their exit code.

Why. A reporting job is missing from the final job’s needs, so the final job closed the run before that job reported. Or a parallel job closed the run at its own end because it lacked PROBARA_CLOSE_RUN: false, and every job after it is refused the same way.

Solution. Keep every reporting job in the needs of the final job, and set PROBARA_CLOSE_RUN: false on each parallel job. The parallel jobs section has the complete workflow.

Results were not recorded (unmatched)

$ PROBARA_CREATE_MISSING_CASES=false npx playwright test
[probara] Sending 2 results of 2 tests (2 passed, 0 failed, 0 skipped, 0 blocked)
[probara] Recorded 1 result (0 new cases, 1 unmatched) in R-1 (closed): https://app.probara.net/projects/SHOP/runs/R-1
[probara] 1 result was not recorded (case_not_found): cart.spec.ts > cart > removes an item

Why. A result names a case that does not exist (a typo in an id, a case of another project), or matches no case while createMissingCases is off. Probara records the others and says which were left out, with the reason.

Solution. Fix the id, create the case, or let the reporter create missing cases (the default).

A case of another project is not sent

Why. A test is linked to WEB-3 while WEB is neither the project nor one of projects: the result would land in the wrong project. One warning per project says so.

Solution. List the project in projects (PROBARA_PROJECTS=WEB) (several projects).

Every run creates new cases

Why. The automation keys changed: a test, describe or file was renamed or moved, a Playwright project was renamed, or rootDir (testDir) moved (what changes a key). Running from another directory does not change keys: they are relative to Playwright’s rootDir.

Solution. Link the tests to their cases with ids (linking), so a rename keeps them. Before a refactor, compare the keys.

An attachment is missing

Why. The log names every file it skipped and why: over 32 MiB, an image over 10 MiB or 8192 px, a refused content type, more than 20 files in one result, or a file that was gone when it uploaded (limits). A file attached inside a step goes to that step, not to the result. Videos and traces only exist when use turns them on.

Solution. Keep test-results/ until the reporter is done; lower the size of screenshots (fullPage: false); attach fewer files per result.

A helper warns that it only works while a test runs

Why. A probara.* helper was called outside a test: at the top of a file, in beforeAll, afterAll or a worker fixture. Nothing is recorded, and the test goes on.

Solution. Call it in the test, beforeEach, afterEach or a test fixture (where to call the helpers).

Probara answers 429 (too many requests)

Why. The organization’s rate limit (60 requests per minute on Free, 1,200 on Pro, and 3,000 on Enterprise) is shared by every token and pipeline. Attachments cost two requests per result (rate limit).

Solution. The reporter waits and retries on its own. For large suites, keep traces and videos for failures only, lower attachmentConcurrency, or spread sharded pipelines in time.

merge-reports sends nothing

Why. The reporter is not among the merge’s reporters, or the merge job has no token.

Solution. Pass --reporter @probara/playwright-reporter (or a --config that registers it) to npx playwright merge-reports, and give that job PROBARA_API_TOKEN and the project (sharding).

The log

The reporter writes to stderr only, every line prefixed with [probara]; stdout belongs to Playwright’s own reporters (list, dot…). A run that reports logs two lines:

$ npx playwright test
[probara] Sending 2 results of 2 tests (2 passed, 0 failed, 0 skipped, 0 blocked)
[probara] Recorded 2 results (1 new case, 0 unmatched) in R-1 (closed): https://app.probara.net/projects/SHOP/runs/R-1

Line	What it says
Sending N results of M tests (...)	Before anything is sent: one result per attempt and case, by status after statusMapping, and what was left out
Recorded N results (...) in R-1	Per run: what Probara recorded, the cases it created, the results it could not match, and the link
N results were not recorded (...)	The unmatched results, by reason, with their cases or keys
Attached N files to results (...)	Uploads: attached, skipped and failed
N results were not sent: ...	A failed report: the reason, and the run left open
Wrote ... to <file>	What went to the results file

A warning that repeats in every test (a malformed helper call, a skipped attachment) is logged the first time, naming the test, and at debug afterwards. The token never appears in any line, even when Probara’s answer echoes it. Log lines are for people: do not parse them, since a minor version may change them.

Debug logging

debug: true (PROBARA_DEBUG=true) adds a line for every request (method, path, status, time) and every retry, and says why reporting is off when it is:

PROBARA_DEBUG=true npx playwright test

Keep it off in normal runs: it is verbose, though it never logs the token or the bodies.

Check what would be sent

The reporter has no dry run of its own, but a results file is one: run the tests with reporting off and a results file, then let the CLI print what it would send. Nothing reaches Probara, and no token is needed:

$ PROBARA_ENABLED=false PROBARA_PROJECT=SHOP PROBARA_RESULTS_FILE=probara-results.json npx playwright test
[probara] Wrote 2 results to /work/shop/probara-results.json: send them with probara import results /work/shop/probara-results.json
$ npx @probara/cli import results probara-results.json --dry-run
passed  SHOP-12  cart.spec.ts > cart > adds an item
passed  -  cart.spec.ts > cart > removes an item
Total: 2 results from 1 file (2 passed, 0 failed, 0 skipped, 0 blocked)

The first command logs on stderr; the dry run prints on stdout, one line per result: the status, the case (- when none is named) and the automation key. --dry-run never touches the file: delete it afterwards, or import it for real, because a file left there makes the next run write probara-results-2.json. --json prints the whole document instead, each entry as Probara would receive it (probara import results).

Use it to compare keys before a refactor or an upgrade: write one file per run, and diff the two dry runs. Give each run its own file name and start without them: with a file already at the path, a run writes to a sibling (before-2.json), and the dry run of before.json would show an older run, hiding the difference.

rm -f before.json after.json
PROBARA_ENABLED=false PROBARA_PROJECT=SHOP PROBARA_RESULTS_FILE=before.json npx playwright test
# Change the code, or upgrade the reporter, then:
PROBARA_ENABLED=false PROBARA_PROJECT=SHOP PROBARA_RESULTS_FILE=after.json npx playwright test
npx @probara/cli import results before.json --dry-run > before.txt
npx @probara/cli import results after.json --dry-run > after.txt
diff before.txt after.txt

Any line that changed is a test whose key or case link changed: it would get a new case.

Next: network, retries and proxies, or the CLI troubleshooting guide for probara import results, run create and run close.