Cypress reporter troubleshooting

Each entry starts from what you see, then says why and what to do. The reporter logs in [probara] lines on stdout, among Cypress’s own output; PROBARA_DEBUG=true (debugging) adds every request. Whatever the problem, cypress run still ends with the number of its own failed tests: reporting changes nothing of that.

Nothing is reported, and nothing is logged

$ PROBARA_DEBUG=true npx cypress run
[probara] Probara reporting is not configured: set PROBARA_API_TOKEN and PROBARA_PROJECT to enable it

Why. Without a token and a project, the reporter stays off and quiet, on purpose: local runs and fork builds send nothing. It says so only at debug.

Solution. Give the step that runs cypress run the PROBARA_API_TOKEN secret, and set the project (projectId in reporterOptions, or PROBARA_PROJECT). Check that the reporter is the reporter of the config block Cypress ran (e2e), and that you ran cypress run: cypress open reports nothing (interactive mode).

Reporting is off: a configuration problem

$ PROBARA_RUN_ULID=R-12 npx cypress run
[probara] Probara reporting is off: PROBARA_RUN_ULID is not a ULID

Why. A setting has a wrong value, or only one of the token and the project is set. Each problem is logged, naming the option or variable at fault (never its value), and nothing is sent.

Solution. Fix the setting the line names (options).

Probara answers 401 or 403

$ npx cypress run
[probara] Sending 3 results of 3 tests (3 passed, 0 failed, 0 skipped, 0 blocked)
[probara] 3 results were not sent: Probara answered 401 unauthorized: <message from Probara>. No run was created or updated

Why. 401: the token is wrong, revoked, or from another Probara (baseUrl). 403: the token cannot report to that project, or the organization is on the Free plan (reporting from CI needs a paid plan).

Solution. Create a new app token from the Cypress card (get started) and update the secret; check the Project ID; check the organization’s plan. To keep what was not sent, set a results file.

Results are reported, but the screenshots, the video and the helpers are not

$ npx cypress run
[probara] Results are reported, but the screenshots, the videos and the probara.* helpers are not: add setupNodeEvents(on, config) { return probaraNodeEvents(on, config); } (from '@probara/cypress-reporter/setup') to the Cypress config, and require('@probara/cypress-reporter/support') in the support file

Why. The Cypress config registers the reporter but not the plugin. Every result is still sent, but in one run per spec, each closed as its spec ends, so a cypress run with several specs makes several runs. Without the plugin there is no screenshot, no video and no probara.* helper.

Solution. Register the plugin, and return its config from setupNodeEvents as probaraNodeEvents(on, config) does (registration). Register it once: a second registration in the same run is left with no results.

Another plugin stopped working, or this one did

Why. Cypress keeps one handler per event of setupNodeEvents: when two plugins register the same event with the same on, the one registered last wins and the other never runs, without a warning. With probaraNodeEvents registered last, the other plugin loses its handler; registered first, this one loses its own:

The event this plugin lost	What is missing
`after:run`	Everything: nothing is sent and no run is created, with no `[probara]` line at the end of the run
`after:spec`	The results of every spec, and the failed result of a spec Cypress could not run
`after:screenshot`	The screenshots of failed attempts
`before:spec`	The screenshots are not matched to their spec
`before:run`	The `browser` parameter of every result

Tasks are merged, so the probara task is never lost this way.

Solution. Hand every plugin the on of cypress-on-fix, which runs every handler of an event in turn (with other plugins):

setupNodeEvents(cypressOn, config) {
  const on = require('cypress-on-fix')(cypressOn);
  require('cypress-split')(on, config);
  probaraNodeEvents(on, config);
  return config;
}

The run stays open

Line	Why, and what to do
`The run <R> of <P> stays open: close it in Probara, or with probara run close --project <P> --run-ulid <ULID>`	The run was not created by this `cypress run` (`run.ulid`), or `closeRun` is off: close it with that command, or in Probara
`The run <R> was left open: <url>`	A report failed after its retries (Probara unreachable, a refused report). Keep what was not sent with a results file, or close the run by hand
`The run <R> was already closed or aborted`	Somebody closed it while the run was sending; nothing to do
`Could not close the run <R>: <reason>. It was left open: <url>`	The results are in, but the close failed: close it by hand

A run the reporter did not create (run.ulid, run.ulids) is never closed, unless closeRun says so (closing).

A job reports after its run was closed

In GitHub Actions the jobs of one workflow run share a run, and a final job closes it. A job that reports after that close is refused with 409 ci_run_closed, and the reporter logs that the run was already closed. It sends nothing more, and the tests keep their exit code.

Why. A reporting job is missing from the final job’s needs, so the final job closed the run before that job reported. Or a parallel job closed the run at its own end because it lacked PROBARA_CLOSE_RUN: false, and every job after it is refused the same way.

Solution. Keep every reporting job in the needs of the final job, and set PROBARA_CLOSE_RUN: false on each parallel job. The parallel jobs section has the complete workflow.

Results were not recorded (unmatched)

$ PROBARA_CREATE_MISSING_CASES=false npx cypress run
[probara] Sending 3 results of 3 tests (3 passed, 0 failed, 0 skipped, 0 blocked)
[probara] Recorded 1 result (0 new cases, 2 unmatched) in R-1 (closed): https://app.probara.net/projects/SHOP/runs/R-1
[probara] 2 results were not recorded (case_not_found): cypress/e2e/cart.cy.js > Cart removes an item; cypress/e2e/login.cy.js > Login logs in with a valid password

Why. A result names a case that does not exist (a typo in an id, a case of another project), or matches no case while createMissingCases is off. Probara records the others and says which were left out, with the reason.

Solution. Fix the id, create the case, or let the reporter create missing cases (the default).

Every run creates new cases

Why. The automation keys changed: a spec, describe or it was renamed or moved, cypress run was started from another directory, or keyIncludesFile changed (what changes a key). Coming from probara import junit, the JUnit reports may have been written without the file attribute of their root suite.

Solution. Name the cases in the titles (linking), so a rename keeps them. Before a refactor, compare the keys; coming from the JUnit import, see migrating.

A helper said something that belongs to no test

$ npx cypress run
[probara] Left out 1 probara.* call in cypress/e2e/helpers.cy.js: what it said belongs to no attempt that ran (probara.title() of a hook that runs no test) (first seen in cypress/e2e/helpers.cy.js; repeats are logged at debug)

Why. A probara.* helper ran with no attempt running, such as in the before hook of a describe, which Cypress runs before its first test starts. What it said is never given to another test.

Solution. Call the helpers in the test, in a beforeEach or in an afterEach (where to call the helpers).

A test lost its steps, files or metadata

Why. Two tests of one spec with the same full title (two its with one title in one describe, or a b › c next to a › b c) cannot be told apart, so what their helpers said is left out rather than given to the wrong test. They also share one key, so one case.

Solution. Give every test its own title.

The helpers did nothing

You see	Why, and what to do
`The probara.* helpers do nothing: nothing was found at Cypress.expose('probara') when the spec ran, …`	The config registers no plugin, or `setupNodeEvents` does not return the config `probaraNodeEvents` returns, so Cypress never hands its settings to the browser
`The probara.* helpers could not send what they were told: there is no cy chain in this frame`	A helper ran where there is no `cy` chain, such as a spec’s top-level code. Nothing is recorded
`probara.* could not send what it was told: <reason>`	The `cy.task('probara', …)` of the call was refused. The test keeps running
`probara.attach() takes { name, path } or { name, body }`	A wrong argument to `probara.attach()`. The call is ignored
`probara.attach() could not attach "<name>": its contentType is not a string`	A `contentType` that is not a string. The file is not attached
`probara.attach() could not attach "<name>": it has neither a path nor a body`	An attachment with neither a `path` nor a `body`
`probara.attach() could not read "<path>" (<reason>): it is not attached`	The file of a `path` is missing or unreadable (`ENOENT`, `EISDIR`): the test goes on without it
`probara.step() takes a function as its body`	A body that is not a function
`probara.step() takes its options as an object ({ expected, data })`	Options that are not an object
`probara.<helper>() could not be recorded: <reason>`	A metadata helper (`id`, `title`, `suite`, `comment`, `ignore`, `parameters`, `tags`, `fields`, `link`, `issue`) took a value it cannot use
`probara.<helper>() takes ...`	One warning per wrong argument, such as `probara.id() takes a case id or a list of case ids, such as PRB-12`
`Ignored malformed probara metadata (type "…")`	A message from a support file of another version of the package; reinstall so both sides match

Without the plugin, these warnings are written to the browser console and nowhere else. With it, they reach the run’s log, naming where they came from: [probara] <message> (first seen in cypress/e2e/cart.cy.js › Cart adds an item). A warning about something inside a spec is logged when that spec ends, not at the end of the run.

A screenshot was left out

$ PROBARA_DEBUG=true npx cypress run
[probara] Left out the screenshots of cypress/e2e/cart.cy.js that name no failed test: Checkout -- pays with voucher (failed)

Why. A debug line only. The reporter attaches a screenshot only to the result of the attempt whose name it carries. A screenshot named otherwise (cy.screenshot('my-name'), or a failed attempt Cypress retried with no second screenshot) belongs to no result.

Solution. Let Cypress take the screenshots of failures; attach a screenshot of your own with probara.attach() (screenshots).

Run selection ran every test

Why. With runCasesOnly, the plugin could not tell which tests belong to the run, so every test ran and was reported, rather than none. One warning says why:

Warning	Why
`runCasesOnly needs the run whose tests to run: set run.ulid or PROBARA_RUN_ULID. Every test runs and is reported`	No run to take the tests from
`runCasesOnly: could not read the cases of the run <ULID> (<reason>). Every test runs and is reported`	The run’s cases could not be read: a wrong ULID, a token of another organization, Probara unreachable
`runCasesOnly needs the support file: require('@probara/cypress-reporter/support') in the Cypress support file, or every test runs and is reported`	The support file is missing: without it no test can be skipped

Solution. Follow the warning (when it cannot select). When every test was skipped instead, no test belongs to the run:

[probara] No test matches the cases of the run 01J9Z3K4M5N6P7Q8R9S0T1V2W3: every test was skipped, and none is reported

Results were left out on purpose

You see	Why
`Sending 4 results of 4 tests (3 passed, 0 failed, 1 skipped, 0 blocked); 1 left out by statusFilter`	The status is in `statusFilter`
`Sending 3 results of 4 tests (3 passed, 0 failed, 0 skipped, 0 blocked); 1 ignored with probara.ignore()`	The test called `probara.ignore()`
`Sending 2 results of 3 tests (2 passed, 0 failed, 0 skipped, 0 blocked); 1 linked only to cases of unlisted projects`	Every case the result links belongs to a project `projects` does not list

The Sending line is the summary of the whole run: it is written once, when the run ends.

An attachment is missing

Why. The log names every file it skipped and why: over 32 MiB, an image over 10 MiB or 8192 px, a refused content type, more than 20 files in one result, or a file probara.attach() could not read (limits). A file attached inside a step goes to that step, not to the result.

Solution. Attach a file that exists when you call probara.attach(), keep files under the limits, and attach fewer files per result.

Probara answers 429 (too many requests)

Why. The organization’s rate limit is shared by every token and pipeline. Attachments cost two requests per result, and without the plugin every spec pays for a run of its own (rate limit).

Solution. The reporter waits and retries on its own. For large suites, attach less, lower attachmentConcurrency, or spread split pipelines in time.

Debugging

The log

The reporter and its plugin write to stdout, every line prefixed with [probara]. A cypress run that reports logs two lines, after the last spec and before Cypress’s (Run Finished) table:

$ npx cypress run
[probara] Sending 5 results of 4 tests (3 passed, 2 failed, 0 skipped, 0 blocked)
[probara] Recorded 5 results (2 new cases, 0 unmatched) in R-1 (closed): https://app.probara.net/projects/SHOP/runs/R-1

A warning that repeats in every test is logged the first time, naming where it was first seen, and at debug afterwards. The token never appears in any line, even when Probara’s answer echoes it.

`PROBARA_DEBUG`

debug: true (PROBARA_DEBUG=true) adds a line for every request (method, path, status, time) and every retry, the repeats of warnings, and says why reporting is off when it is:

PROBARA_DEBUG=true npx cypress run

Keep it off in normal runs: it is verbose, though it never logs the token or the bodies.

Check what would be sent

The reporter has no dry run of its own, but a results file is one: run the tests with reporting off and a results file, then let the CLI print what it would send. Nothing reaches Probara, and no token is needed:

$ PROBARA_ENABLED=false PROBARA_PROJECT=SHOP PROBARA_RESULTS_FILE=probara-results.json npx cypress run
[probara] Wrote 3 results to /work/shop/probara-results.json: send them with probara import results /work/shop/probara-results.json
$ npx @probara/cli import results probara-results.json --dry-run
passed  SHOP-12  cypress/e2e/cart.cy.js > Cart adds an item
passed  -  cypress/e2e/cart.cy.js > Cart removes an item
passed  -  cypress/e2e/login.cy.js > Login logs in with a valid password
Total: 3 results from 1 file (3 passed, 0 failed, 0 skipped, 0 blocked)

One line per result: the status, the case (- when none is named) and the automation key. --dry-run never touches the file; delete it afterwards, or a later run writes probara-results-2.json.

To compare keys before a refactor, an upgrade or a switch of keyIncludesFile, write one file per run and diff the two dry runs. Start without the files: with a file already at the path, a run writes to a sibling, and the dry run would show the older run.

rm -f before.json after.json
PROBARA_ENABLED=false PROBARA_PROJECT=SHOP PROBARA_RESULTS_FILE=before.json npx cypress run
# Change the code, or upgrade the reporter, then:
PROBARA_ENABLED=false PROBARA_PROJECT=SHOP PROBARA_RESULTS_FILE=after.json npx cypress run
npx @probara/cli import results before.json --dry-run > before.txt
npx @probara/cli import results after.json --dry-run > after.txt
diff before.txt after.txt

Any line that changed is a test whose key or case link changed: it would get a new case.

Every line the reporter logs

The reporting library the reporter shares with the other reporters and the CLI writes the common lines (Sending, Recorded, not recorded, not sent, Wrote, Probara warned, Probara reporting is off: <problem>). These are the lines of the Cypress reporter itself.

Nothing was reported

Line	Level	Meaning and what to do
`Probara reporting is not configured: set PROBARA_API_TOKEN and PROBARA_PROJECT to enable it`	debug	Nothing is reported, on purpose
`Probara reporting is disabled by <option or variable>`	debug	`enabled: false`, or `PROBARA_ENABLED`
`Probara reporting is off: <problem>`	error	A configuration problem
`Probara reporting is off: the reporter could not start: <reason>`	error	Report it: the reporter never throws into Cypress, and says why it stopped
`Probara reporting is off: the reporter could not be loaded; reinstall @probara/cypress-reporter`	error	Loading the package failed: reinstall it
`Ignored the unknown option "<name>" of @probara/cypress-reporter`	warn	A typo in an option name. Reporting goes on without it
`Results are reported, but the screenshots, the videos and the probara.* helpers are not: ...`	warn	No plugin
`Could not report a spec without setupNodeEvents: <reason>`	error	A spec’s results could not be sent in the run of their own (no plugin). The other specs are unaffected

Some results were left out

Line	Level	Meaning and what to do
`Could not report an attempt of "<test>": <reason>`	error	Report it: that one attempt is lost, and the run goes on
`Left out <n> probara.* call(s) in <spec>: what it said belongs to no attempt that ran (<labels>)`	warn	A helper ran with no test
`Left out the screenshots of <spec> that name no failed test: <names>`	debug	A screenshot belongs to no attempt
`Ignored malformed probara metadata (type "…")`	warn	A message of a support file of another version (helpers did nothing)
`Dropped the issues of probara.issue(): no issueUrlTemplate turns their ids into links`	warn	Set `issueUrlTemplate`
`probara.<helper>() takes ...`, `probara.attach() could not attach "<name>": <reason>`, `probara.step() takes ...`	warn	A wrong argument; the call is ignored and the test is never failed
`Could not write the results file <path>: <reason>`	error	The results file could not be written: check the path and its folder
`Skipped the invalid result of <what>: <reason>`	warn	A result that could not be built; report it
`Skipped an attachment: <reason>`	warn	A file was left out (an attachment is missing)

The run

Line	Level	Meaning and what to do
`Ran only the tests of run <ULID>: <k> of <m> tests match its cases; <s> skipped and not reported`	info	Run selection worked
`No test matches the cases of the run <ULID>: every test was skipped, and none is reported`	warn	No test belongs to the run
`The run <R> of <P> stays open: close it in Probara, or with probara run close --project <P> --run-ulid <ULID>`	info	A reused run, or `closeRun: false` (the run stays open)
`Could not create the Probara run: <reason>` / `Could not create the run: <reason>`	error	The run could not be created; nothing was sent
`Could not close the run <R>: <reason>. It was left open: <url>`	error	Close it by hand
`The run <R> was already closed or aborted`	info	Somebody closed it meanwhile
`<n> results were not sent: <reasons>. <No run was created or updated \| The run <R> was left open: <url>>`	error	Probara refused or could not be reached; set a results file

Report a problem at github.com/cynch-me/probara-sdk/issues, with the [probara] lines of a run with PROBARA_DEBUG=true.

Related: Configuration · Network · Results file · Source: troubleshooting.md, debugging.md