Cypress run selection

Someone creates a run in Probara (the cases of a release, of a test plan, the ones that failed yesterday) and the pipeline runs exactly those tests. With runCasesOnly, the plugin reads the run’s cases before the first spec, the support file skips every test that belongs to none of them, and the results of the tests that ran go into that run, which stays open.

Quick path

1. Register the plugin and load the support file (get started). Selection needs both; without the variables below, they select nothing.
2. In the job that runs a run’s tests, set PROBARA_RUN_CASES_ONLY=true and PROBARA_RUN_ULID to the run’s ULID. In Probara, open the run, select Run actions (the menu in the run’s header) and choose Copy run ULID; the same option is in the menu of each run in the run list.
3. Check the run in Probara: its cases have the results, and the run stays open for whoever created it.

$ PROBARA_RUN_CASES_ONLY=true PROBARA_RUN_ULID=01J9Z3K4M5N6P7Q8R9S0T1V2W3 npx cypress run
[probara] Ran only the tests of run 01J9Z3K4M5N6P7Q8R9S0T1V2W3: 2 of 3 tests match its cases; 1 skipped and not reported

In GitHub Actions, a workflow started by hand with the run’s ULID as an input:

on:
  workflow_dispatch:
    inputs:
      run_ulid:
        description: The ULID of the Probara run to run the tests of
        required: true

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-node@v5
        with:
          node-version: 22
      - run: npm ci
      - name: Run the tests of a Probara run
        run: npx cypress run
        env:
          PROBARA_API_TOKEN: ${{ secrets.PROBARA_API_TOKEN }}
          PROBARA_RUN_CASES_ONLY: 'true'
          PROBARA_RUN_ULID: ${{ inputs.run_ulid }}

Turn it on per job, with the variables: runCasesOnly: true in the Cypress config turns it on in every job, and a job without a ULID then warns and runs every test.

Which tests belong to the run

A test belongs to the run when one of these holds, checked for each test before it runs:

The test	Example
Its automation key is the key of one of the run’s cases	A case the reporter or probara import junit created
Its title or one of its describe titles names one of the run’s cases	it('SHOP-12 adds an item'), describe('SHOP-12 Cart')

• probara.id() does not count: it runs with the test, after the selection. Name the case in a title instead (linking).
• Keys are the reporter’s: the spec path (with keyIncludesFile) and the full title. A suite whose keys were renamed after the run was created has no key match at all.
• A case of the run with no test simply gets no result: the log counts only the tests.

What is reported

Test	Runs	Reported
Belongs to the run	Yes	Yes, into the run, with every attempt
Belongs to the run, but it.skip	No	Yes, as skipped
Belongs to no case of the run	No	No: it is skipped and left out of the run

A test outside the run is skipped, not run: the support file asks the plugin about each test in a root beforeEach, and calls this.skip() on a test the run does not take. It is not reported, so it adds nothing to the run; Cypress’s own summary counts it as skipped.

What still runs

Cypress loads a spec before the selection can skip its tests, so a spec with no test of the run still runs the module scope of its file (the top-level code, what it imports and what it sets up) and the before and after hooks at the root of the file. Its tests are skipped; the file is not skipped as a whole.

When that is expensive, also pass Cypress only the specs of the run’s tests:

PROBARA_RUN_CASES_ONLY=true PROBARA_RUN_ULID=01J9Z3K4M5N6P7Q8R9S0T1V2W3 npx cypress run --spec 'cypress/e2e/{cart,checkout}.cy.js'

Quote the glob, so the shell does not expand it first.

When it cannot select

Selection never breaks the test run: when it cannot know which tests belong to the run, every test runs and is reported, with one warning.

What happens	The warning
No run.ulid nor PROBARA_RUN_ULID	runCasesOnly needs the run whose tests to run: set run.ulid or PROBARA_RUN_ULID. Every test runs and is reported
The run’s cases cannot be read	runCasesOnly: could not read the cases of the run <ULID> (<reason>). Every test runs and is reported
The support file is missing	runCasesOnly needs the support file: require('@probara/cypress-reporter/support') in the Cypress support file, or every test runs and is reported
No test matches any case of the run	No test matches the cases of the run <ULID>: every test was skipped, and none is reported

The cases cannot be read with a wrong ULID, a token of another organization, or Probara unreachable: the reason is in the line. Without the plugin there is nobody to read the cases or answer the browser, so every test runs and is reported too.

When no test matches, nothing is sent and the run is left as it was: check that the ULID is the one of the run you meant.

Setups

• Split pipelines: each machine reads the run’s cases and runs its share of the selected tests.
• One more request: reading the cases is one request before the first spec, 200 cases per page (network), and then one cy.task() per test, which the plugin answers at once. It is the one read an app token can make: the run case keys endpoint.
• statusFilter: ['skipped'] also leaves out the tests of the run that are it.skip.

Related: Configuration · An existing run · Troubleshooting · Source: run-selection.md