# Cypress reporter troubleshooting > Problems with @probara/cypress-reporter and their solutions, every line the reporter logs, PROBARA_DEBUG, and how to check what would be sent without sending it. Each entry starts from what you see, then says why and what to do. The reporter logs in `[probara]` lines on **stdout**, among Cypress's own output; `PROBARA_DEBUG=true` ([debugging](#debugging)) adds every request. Whatever the problem, `cypress run` still ends with the number of its own failed tests: reporting changes nothing of that. ## Nothing is reported, and nothing is logged ```text $ PROBARA_DEBUG=true npx cypress run [probara] Probara reporting is not configured: set PROBARA_API_TOKEN and PROBARA_PROJECT to enable it ``` **Why.** Without a token and a project, the reporter stays off and quiet, on purpose: local runs and fork builds send nothing. It says so only at debug. **Solution.** Give the step that runs `cypress run` the `PROBARA_API_TOKEN` secret, and set the project (`projectId` in `reporterOptions`, or `PROBARA_PROJECT`). Check that the reporter is the `reporter` of the config block Cypress ran (`e2e`), and that you ran `cypress run`: `cypress open` reports nothing ([interactive mode](https://docs.probara.net/en/developers/ci-integrations/cypress/configuration/#interactive-mode)). ## Reporting is off: a configuration problem ```text $ PROBARA_RUN_ULID=R-12 npx cypress run [probara] Probara reporting is off: PROBARA_RUN_ULID is not a ULID ``` **Why.** A setting has a wrong value, or only one of the token and the project is set. Each problem is logged, naming the option or variable at fault (never its value), and nothing is sent. **Solution.** Fix the setting the line names ([options](https://docs.probara.net/en/developers/ci-integrations/cypress/configuration/#options)). ## Probara answers 401 or 403 ```text $ npx cypress run [probara] Sending 3 results of 3 tests (3 passed, 0 failed, 0 skipped, 0 blocked) [probara] 3 results were not sent: Probara answered 401 unauthorized: . No run was created or updated ``` **Why.** `401`: the token is wrong, revoked, or from another Probara (`baseUrl`). `403`: the token cannot report to that project, or the organization is on the Free plan (reporting from CI needs a paid plan). **Solution.** Create a new app token from the **Cypress** card ([get started](https://docs.probara.net/en/developers/ci-integrations/cypress/#1-create-an-app-token)) and update the secret; check the Project ID; check the organization's plan. To keep what was not sent, set a [results file](https://docs.probara.net/en/developers/ci-integrations/cypress/results-and-retries/#results-file). ## Results are reported, but the screenshots, the video and the helpers are not ```text $ npx cypress run [probara] Results are reported, but the screenshots, the videos and the probara.* helpers are not: add setupNodeEvents(on, config) { return probaraNodeEvents(on, config); } (from '@probara/cypress-reporter/setup') to the Cypress config, and require('@probara/cypress-reporter/support') in the support file ``` **Why.** The Cypress config registers the reporter but not the plugin. Every result is still sent, but in **one run per spec**, each closed as its spec ends, so a `cypress run` with several specs makes several runs. Without the plugin there is no screenshot, no video and no `probara.*` helper. **Solution.** Register the plugin, and return its config from `setupNodeEvents` as `probaraNodeEvents(on, config)` does ([registration](https://docs.probara.net/en/developers/ci-integrations/cypress/configuration/#registration)). Register it once: a second registration in the same run is left with no results. ## Another plugin stopped working, or this one did **Why.** Cypress keeps one handler per event of `setupNodeEvents`: when two plugins register the same event with the same `on`, the one registered last wins and the other never runs, without a warning. With `probaraNodeEvents` registered last, the other plugin loses its handler; registered first, this one loses its own: | The event this plugin lost | What is missing | | -------------------------- | ------------------------------------------------------------------------------------------------- | | `after:run` | Everything: nothing is sent and no run is created, with no `[probara]` line at the end of the run | | `after:spec` | The results of every spec, and the failed result of a spec Cypress could not run | | `after:screenshot` | The screenshots of failed attempts | | `before:spec` | The screenshots are not matched to their spec | | `before:run` | The `browser` parameter of every result | Tasks are merged, so the `probara` task is never lost this way. **Solution.** Hand every plugin the `on` of [cypress-on-fix](https://github.com/bahmutov/cypress-on-fix), which runs every handler of an event in turn ([with other plugins](https://docs.probara.net/en/developers/ci-integrations/cypress/configuration/#with-other-plugins)): ```js setupNodeEvents(cypressOn, config) { const on = require('cypress-on-fix')(cypressOn); require('cypress-split')(on, config); probaraNodeEvents(on, config); return config; } ``` ## The run stays open | Line | Why, and what to do | | --------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | | `The run of

stays open: close it in Probara, or with probara run close --project

--run-ulid ` | The run was not created by this `cypress run` (`run.ulid`), or `closeRun` is off: close it with that command, or in Probara | | `The run was left open: ` | A report failed after its retries (Probara unreachable, a refused report). Keep what was not sent with a results file, or close the run by hand | | `The run was already closed or aborted` | Somebody closed it while the run was sending; nothing to do | | `Could not close the run : . It was left open: ` | The results are in, but the close failed: close it by hand | A run the reporter did not create (`run.ulid`, `run.ulids`) is never closed, unless `closeRun` says so ([closing](https://docs.probara.net/en/developers/ci-integrations/cypress/results-and-retries/#closing)). ## A job reports after its run was closed In GitHub Actions the jobs of one workflow run share a run, and a final job closes it. A job that reports after that close is refused with `409 ci_run_closed`, and the reporter logs that the run was already closed. It sends nothing more, and the tests keep their exit code. **Why.** A reporting job is missing from the final job's `needs`, so the final job closed the run before that job reported. Or a parallel job closed the run at its own end because it lacked `PROBARA_CLOSE_RUN: false`, and every job after it is refused the same way. **Solution.** Keep every reporting job in the `needs` of the final job, and set `PROBARA_CLOSE_RUN: false` on each parallel job. The [parallel jobs](https://docs.probara.net/en/developers/ci-integrations/cypress/ci-providers/#parallel-jobs-one-run-closed-by-the-final-job) section has the complete workflow. ## Results were not recorded (unmatched) ```text $ PROBARA_CREATE_MISSING_CASES=false npx cypress run [probara] Sending 3 results of 3 tests (3 passed, 0 failed, 0 skipped, 0 blocked) [probara] Recorded 1 result (0 new cases, 2 unmatched) in R-1 (closed): https://app.probara.net/projects/SHOP/runs/R-1 [probara] 2 results were not recorded (case_not_found): cypress/e2e/cart.cy.js > Cart removes an item; cypress/e2e/login.cy.js > Login logs in with a valid password ``` **Why.** A result names a case that does not exist (a typo in an id, a case of another project), or matches no case while `createMissingCases` is off. Probara records the others and says which were left out, with the reason. **Solution.** Fix the id, create the case, or let the reporter create missing cases (the default). ## Every run creates new cases **Why.** The automation keys changed: a spec, `describe` or `it` was renamed or moved, `cypress run` was started from another directory, or `keyIncludesFile` changed ([what changes a key](https://docs.probara.net/en/developers/ci-integrations/cypress/linking/#what-changes-a-key)). Coming from `probara import junit`, the JUnit reports may have been written without the `file` attribute of their root suite. **Solution.** Name the cases in the titles ([linking](https://docs.probara.net/en/developers/ci-integrations/cypress/linking/)), so a rename keeps them. Before a refactor, [compare the keys](#check-what-would-be-sent); coming from the JUnit import, see [migrating](https://docs.probara.net/en/developers/ci-integrations/cypress/migrating/#from-the-junit-import). ## A helper said something that belongs to no test ```text $ npx cypress run [probara] Left out 1 probara.* call in cypress/e2e/helpers.cy.js: what it said belongs to no attempt that ran (probara.title() of a hook that runs no test) (first seen in cypress/e2e/helpers.cy.js; repeats are logged at debug) ``` **Why.** A `probara.*` helper ran with no attempt running, such as in the `before` hook of a `describe`, which Cypress runs before its first test starts. What it said is never given to another test. **Solution.** Call the helpers in the test, in a `beforeEach` or in an `afterEach` ([where to call the helpers](https://docs.probara.net/en/developers/ci-integrations/cypress/metadata/#where-to-call-the-helpers)). ## A test lost its steps, files or metadata **Why.** Two tests of one spec with the same full title (two `it`s with one title in one `describe`, or `a b` › `c` next to `a` › `b c`) cannot be told apart, so what their helpers said is left out rather than given to the wrong test. They also share one key, so one case. **Solution.** Give every test its own title. ## The helpers did nothing | You see | Why, and what to do | | ------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `The probara.* helpers do nothing: nothing was found at Cypress.expose('probara') when the spec ran, …` | The config registers no plugin, or `setupNodeEvents` does not return the config `probaraNodeEvents` returns, so Cypress never hands its settings to the browser | | `The probara.* helpers could not send what they were told: there is no cy chain in this frame` | A helper ran where there is no `cy` chain, such as a spec's top-level code. Nothing is recorded | | `probara.* could not send what it was told: ` | The `cy.task('probara', …)` of the call was refused. The test keeps running | | `probara.attach() takes { name, path } or { name, body }` | A wrong argument to `probara.attach()`. The call is ignored | | `probara.attach() could not attach "": its contentType is not a string` | A `contentType` that is not a string. The file is not attached | | `probara.attach() could not attach "": it has neither a path nor a body` | An attachment with neither a `path` nor a `body` | | `probara.attach() could not read "" (): it is not attached` | The file of a `path` is missing or unreadable (`ENOENT`, `EISDIR`): the test goes on without it | | `probara.step() takes a function as its body` | A body that is not a function | | `probara.step() takes its options as an object ({ expected, data })` | Options that are not an object | | `probara.() could not be recorded: ` | A metadata helper (`id`, `title`, `suite`, `comment`, `ignore`, `parameters`, `tags`, `fields`, `link`, `issue`) took a value it cannot use | | `probara.() takes ...` | One warning per wrong argument, such as `probara.id() takes a case id or a list of case ids, such as PRB-12` | | `Ignored malformed probara metadata (type "…")` | A message from a support file of another version of the package; reinstall so both sides match | Without the plugin, these warnings are written to the **browser console** and nowhere else. With it, they reach the run's log, naming where they came from: `[probara] (first seen in cypress/e2e/cart.cy.js › Cart adds an item)`. A warning about something inside a spec is logged when that spec ends, not at the end of the run. ## A screenshot was left out ```text $ PROBARA_DEBUG=true npx cypress run [probara] Left out the screenshots of cypress/e2e/cart.cy.js that name no failed test: Checkout -- pays with voucher (failed) ``` **Why.** A debug line only. The reporter attaches a screenshot only to the result of the attempt whose name it carries. A screenshot named otherwise (`cy.screenshot('my-name')`, or a failed attempt Cypress retried with no second screenshot) belongs to no result. **Solution.** Let Cypress take the screenshots of failures; attach a screenshot of your own with `probara.attach()` ([screenshots](https://docs.probara.net/en/developers/ci-integrations/cypress/steps-and-attachments/#screenshots)). ## Run selection ran every test **Why.** With `runCasesOnly`, the plugin could not tell which tests belong to the run, so every test ran and was reported, rather than none. One warning says why: | Warning | Why | | --------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------- | | `runCasesOnly needs the run whose tests to run: set run.ulid or PROBARA_RUN_ULID. Every test runs and is reported` | No run to take the tests from | | `runCasesOnly: could not read the cases of the run (). Every test runs and is reported` | The run's cases could not be read: a wrong ULID, a token of another organization, Probara unreachable | | `runCasesOnly needs the support file: require('@probara/cypress-reporter/support') in the Cypress support file, or every test runs and is reported` | The support file is missing: without it no test can be skipped | **Solution.** Follow the warning ([when it cannot select](https://docs.probara.net/en/developers/ci-integrations/cypress/run-selection/#when-it-cannot-select)). When every test was skipped instead, no test belongs to the run: ```text [probara] No test matches the cases of the run 01J9Z3K4M5N6P7Q8R9S0T1V2W3: every test was skipped, and none is reported ``` ## Results were left out on purpose | You see | Why | | ---------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------- | | `Sending 4 results of 4 tests (3 passed, 0 failed, 1 skipped, 0 blocked); 1 left out by statusFilter` | The status is in `statusFilter` | | `Sending 3 results of 4 tests (3 passed, 0 failed, 0 skipped, 0 blocked); 1 ignored with probara.ignore()` | The test called `probara.ignore()` | | `Sending 2 results of 3 tests (2 passed, 0 failed, 0 skipped, 0 blocked); 1 linked only to cases of unlisted projects` | Every case the result links belongs to a project `projects` does not list | The `Sending` line is the summary of the whole run: it is written once, when the run ends. ## An attachment is missing **Why.** The log names every file it skipped and why: over 32 MiB, an image over 10 MiB or 8192 px, a refused content type, more than 20 files in one result, or a file `probara.attach()` could not read ([limits](https://docs.probara.net/en/developers/ci-integrations/cypress/steps-and-attachments/#limits-and-what-is-skipped)). A file attached inside a step goes to that step, not to the result. **Solution.** Attach a file that exists when you call `probara.attach()`, keep files under the limits, and attach fewer files per result. ## Probara answers 429 (too many requests) **Why.** The organization's rate limit is shared by every token and pipeline. Attachments cost two requests per result, and without the plugin every spec pays for a run of its own ([rate limit](https://docs.probara.net/en/developers/ci-integrations/cypress/configuration/#rate-limit)). **Solution.** The reporter waits and retries on its own. For large suites, attach less, lower `attachmentConcurrency`, or spread split pipelines in time. ## Debugging ### The log The reporter and its plugin write to stdout, every line prefixed with `[probara]`. A `cypress run` that reports logs two lines, after the last spec and before Cypress's `(Run Finished)` table: ```text $ npx cypress run [probara] Sending 5 results of 4 tests (3 passed, 2 failed, 0 skipped, 0 blocked) [probara] Recorded 5 results (2 new cases, 0 unmatched) in R-1 (closed): https://app.probara.net/projects/SHOP/runs/R-1 ``` A warning that repeats in every test is logged the first time, naming where it was first seen, and at debug afterwards. The token never appears in any line, even when Probara's answer echoes it. ### `PROBARA_DEBUG` `debug: true` (`PROBARA_DEBUG=true`) adds a line for every request (method, path, status, time) and every retry, the repeats of warnings, and says why reporting is off when it is: ```bash PROBARA_DEBUG=true npx cypress run ``` Keep it off in normal runs: it is verbose, though it never logs the token or the bodies. ### Check what would be sent The reporter has no dry run of its own, but a results file is one: run the tests with reporting off and a results file, then let the CLI print what it would send. Nothing reaches Probara, and no token is needed: ```text $ PROBARA_ENABLED=false PROBARA_PROJECT=SHOP PROBARA_RESULTS_FILE=probara-results.json npx cypress run [probara] Wrote 3 results to /work/shop/probara-results.json: send them with probara import results /work/shop/probara-results.json $ npx @probara/cli import results probara-results.json --dry-run passed SHOP-12 cypress/e2e/cart.cy.js > Cart adds an item passed - cypress/e2e/cart.cy.js > Cart removes an item passed - cypress/e2e/login.cy.js > Login logs in with a valid password Total: 3 results from 1 file (3 passed, 0 failed, 0 skipped, 0 blocked) ``` One line per result: the status, the case (`-` when none is named) and the automation key. `--dry-run` never touches the file; delete it afterwards, or a later run writes `probara-results-2.json`. To compare keys before a refactor, an upgrade or a switch of `keyIncludesFile`, write one file per run and diff the two dry runs. Start without the files: with a file already at the path, a run writes to a sibling, and the dry run would show the older run. ```bash rm -f before.json after.json PROBARA_ENABLED=false PROBARA_PROJECT=SHOP PROBARA_RESULTS_FILE=before.json npx cypress run # Change the code, or upgrade the reporter, then: PROBARA_ENABLED=false PROBARA_PROJECT=SHOP PROBARA_RESULTS_FILE=after.json npx cypress run npx @probara/cli import results before.json --dry-run > before.txt npx @probara/cli import results after.json --dry-run > after.txt diff before.txt after.txt ``` Any line that changed is a test whose key or case link changed: it would get a new case. ## Every line the reporter logs The reporting library the reporter shares with the other reporters and the CLI writes the common lines (`Sending`, `Recorded`, `not recorded`, `not sent`, `Wrote`, `Probara warned`, `Probara reporting is off: `). These are the lines of the Cypress reporter itself. ### Nothing was reported | Line | Level | Meaning and what to do | | ------------------------------------------------------------------------------------------------- | ----- | ------------------------------------------------------------------------------------------------------ | | `Probara reporting is not configured: set PROBARA_API_TOKEN and PROBARA_PROJECT to enable it` | debug | [Nothing is reported](#nothing-is-reported-and-nothing-is-logged), on purpose | | `Probara reporting is disabled by

stays open: close it in Probara, or with probara run close --project

--run-ulid ` | info | A reused run, or `closeRun: false` ([the run stays open](#the-run-stays-open)) | | `Could not create the Probara run: ` / `Could not create the run: ` | error | The run could not be created; nothing was sent | | `Could not close the run : . It was left open: ` | error | [Close it by hand](#the-run-stays-open) | | `The run was already closed or aborted` | info | Somebody closed it meanwhile | | ` results were not sent: . was left open: >` | error | Probara refused or could not be reached; set a results file | Report a problem at [github.com/cynch-me/probara-sdk/issues](https://github.com/cynch-me/probara-sdk/issues), with the `[probara]` lines of a run with `PROBARA_DEBUG=true`. Related: [Configuration](https://docs.probara.net/en/developers/ci-integrations/cypress/configuration/) · [Network](https://docs.probara.net/en/developers/ci-integrations/cypress/configuration/#network) · [Results file](https://docs.probara.net/en/developers/ci-integrations/cypress/results-and-retries/#results-file) · Source: [troubleshooting.md](https://github.com/cynch-me/probara-sdk/blob/main/packages/cypress-reporter/docs/troubleshooting.md), [debugging.md](https://github.com/cynch-me/probara-sdk/blob/main/packages/cypress-reporter/docs/debugging.md)