Results and retries

Results

Each attempt of a scenario is one result, in the order the attempts ran:

• A scenario that is not retried sends one result.
• A scenario that fails once and then passes, with --retry, arrives as two results of one key: the failed attempt, then the passed one.
• Each row of a Scenario Outline is a result of its own, recorded on the run case with its values.

npx cucumber-js --retry 2 --format @probara/cucumber-reporter

Every attempt stays in the case’s history. The run case keeps the last outcome.

Statuses

Every result has one of four statuses: passed, failed, skipped or blocked. The reporter sends passed, failed and skipped. blocked comes from status mapping.

Step statuses

Cucumber’s status	Probara’s status	Why
passed	passed	The step ran and passed.
failed	failed	The step ran and failed. Its message is the step’s error.
ambiguous	failed	More than one definition matches the step. Its message says so.
skipped, pending, undefined, unknown	skipped	The step did not run: it is skipped, has no definition, or is pending.

Cucumber’s report spells these in lowercase (its JSON report) or in uppercase (its Messages). The reporter reads both.

Scenario statuses

A scenario, or one row of an outline, is:

1. failed when any of its steps failed, or any of its Before or After hooks failed;
2. otherwise skipped when any of its steps did not run;
3. otherwise passed.

Status mapping and filter

statusMapping sends a status as another, and statusFilter leaves results with those statuses out of the report, after the mapping. For example, PROBARA_STATUS_MAPPING=failed=blocked and PROBARA_STATUS_FILTER=skipped send every failed scenario as blocked, and send no skipped one.

Durations

A step’s duration is its own. A scenario’s duration is the sum of its steps and hooks, in milliseconds. The start of each attempt is sent as its executedAt.

Results file

When Probara cannot take a result, the reporter can keep it in a file to send later. Set resultsFile (PROBARA_RESULTS_FILE=reports/probara-results.json): at the end of the run, every result that was not sent is written there. With reporting off, every result is written there.

The file is never overwritten. When one is already there, the reporter writes to the first free name beside it (probara-results-2.json, and so on), and the summary line says where it went.

To send the file later, run the CLI with the same token and project:

npx @probara/cli import results 'reports/probara-results*.json'

Its results keep their keys and their case links.

Assigning failed results

assignFailedTo (PROBARA_ASSIGN_FAILED_TO=ana@example.com,bo@example.com) assigns each failed result that has no assignee to one of the members it names, in turn.