# Cypress run selection

> Run only the Cypress tests of a run created in Probara with runCasesOnly, how tests are matched to the run's cases, what still runs, and what happens when selection is not possible.

Someone creates a run in Probara (the cases of a release, of a test plan, the ones that failed
yesterday) and the pipeline runs exactly those tests. With `runCasesOnly`, the plugin reads the
run's cases before the first spec, the support file skips every test that belongs to none of them,
and the results of the tests that ran go into that run, which stays open.

## Quick path

1. Register the plugin and load the support file
   ([get started](https://docs.probara.net/en/developers/ci-integrations/cypress/#3-register-the-reporter-and-its-plugin)).
   Selection needs both; without the variables below, they select nothing.
2. In the job that runs a run's tests, set `PROBARA_RUN_CASES_ONLY=true` and `PROBARA_RUN_ULID` to
   the run's ULID. In Probara, open the run, select **Run actions** (the menu in the run's header)
   and choose **Copy run ULID**; the same option is in the menu of each run in the run list.
3. Check the run in Probara: its cases have the results, and the run stays open for whoever created
   it.

```text
$ PROBARA_RUN_CASES_ONLY=true PROBARA_RUN_ULID=01J9Z3K4M5N6P7Q8R9S0T1V2W3 npx cypress run
[probara] Ran only the tests of run 01J9Z3K4M5N6P7Q8R9S0T1V2W3: 2 of 3 tests match its cases; 1 skipped and not reported
```

In GitHub Actions, a workflow started by hand with the run's ULID as an input:

```yaml
on:
  workflow_dispatch:
    inputs:
      run_ulid:
        description: The ULID of the Probara run to run the tests of
        required: true

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-node@v5
        with:
          node-version: 22
      - run: npm ci
      - name: Run the tests of a Probara run
        run: npx cypress run
        env:
          PROBARA_API_TOKEN: ${{ secrets.PROBARA_API_TOKEN }}
          PROBARA_RUN_CASES_ONLY: 'true'
          PROBARA_RUN_ULID: ${{ inputs.run_ulid }}
```

Turn it on per job, with the variables: `runCasesOnly: true` in the Cypress config turns it on in
every job, and a job without a ULID then warns and runs every test.

## Which tests belong to the run

A test belongs to the run when one of these holds, checked for each test before it runs:

| The test                                                               | Example                                                  |
| ---------------------------------------------------------------------- | -------------------------------------------------------- |
| Its automation key is the key of one of the run's cases                | A case the reporter or `probara import junit` created    |
| Its title or one of its `describe` titles names one of the run's cases | `it('SHOP-12 adds an item')`, `describe('SHOP-12 Cart')` |

- **`probara.id()` does not count**: it runs with the test, after the selection. Name the case in a
  title instead ([linking](https://docs.probara.net/en/developers/ci-integrations/cypress/linking/)).
- **Keys are the reporter's**: the spec path (with
  [`keyIncludesFile`](https://docs.probara.net/en/developers/ci-integrations/cypress/configuration/#keyincludesfile)) and the
  full title. A suite whose keys were renamed after the run was created has no key match at all.
- **A case of the run with no test** simply gets no result: the log counts only the tests.

## What is reported

| Test                              | Runs | Reported                                  |
| --------------------------------- | ---- | ----------------------------------------- |
| Belongs to the run                | Yes  | Yes, into the run, with every attempt     |
| Belongs to the run, but `it.skip` | No   | Yes, as skipped                           |
| Belongs to no case of the run     | No   | No: it is skipped and left out of the run |

A test outside the run is **skipped**, not run: the support file asks the plugin about each test in a
root `beforeEach`, and calls `this.skip()` on a test the run does not take. It is not reported, so it
adds nothing to the run; Cypress's own summary counts it as skipped.

### What still runs

Cypress loads a spec before the selection can skip its tests, so a spec with no test of the run
still runs the **module scope of its file** (the top-level code, what it imports and what it sets
up) and the `before` and `after` hooks at the root of the file. Its tests are skipped; the file is
not skipped as a whole.

When that is expensive, also pass Cypress only the specs of the run's tests:

```bash
PROBARA_RUN_CASES_ONLY=true PROBARA_RUN_ULID=01J9Z3K4M5N6P7Q8R9S0T1V2W3 npx cypress run --spec 'cypress/e2e/{cart,checkout}.cy.js'
```

Quote the glob, so the shell does not expand it first.

## When it cannot select

Selection never breaks the test run: when it cannot know which tests belong to the run, every test
runs and is reported, with one warning.

| What happens                         | The warning                                                                                                                                         |
| ------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------- |
| No `run.ulid` nor `PROBARA_RUN_ULID` | `runCasesOnly needs the run whose tests to run: set run.ulid or PROBARA_RUN_ULID. Every test runs and is reported`                                  |
| The run's cases cannot be read       | `runCasesOnly: could not read the cases of the run <ULID> (<reason>). Every test runs and is reported`                                              |
| The support file is missing          | `runCasesOnly needs the support file: require('@probara/cypress-reporter/support') in the Cypress support file, or every test runs and is reported` |
| No test matches any case of the run  | `No test matches the cases of the run <ULID>: every test was skipped, and none is reported`                                                         |

The cases cannot be read with a wrong ULID, a token of another organization, or Probara unreachable:
the reason is in the line. Without the plugin there is nobody to read the cases or answer the
browser, so every test runs and is reported too.

When no test matches, nothing is sent and the run is left as it was: check that the ULID is the one
of the run you meant.

## Setups

- **Split pipelines**: each machine reads the run's cases and runs its share of the selected tests.
- **One more request**: reading the cases is one request before the first spec, 200 cases per page
  ([network](https://docs.probara.net/en/developers/ci-integrations/cypress/configuration/#network)), and then one
  `cy.task()` per test, which the plugin answers at once. It is the one read an app token can make:
  the [run case keys endpoint](https://docs.probara.net/en/developers/api/run-cases/#list-a-runs-case-keys).
- **`statusFilter: ['skipped']`** also leaves out the tests of the run that are `it.skip`.

Related: [Configuration](https://docs.probara.net/en/developers/ci-integrations/cypress/configuration/#runcasesonly) ·
[An existing run](https://docs.probara.net/en/developers/ci-integrations/cypress/results-and-retries/#an-existing-run) ·
[Troubleshooting](https://docs.probara.net/en/developers/ci-integrations/cypress/troubleshooting/#run-selection-ran-every-test) ·
Source: [run-selection.md](https://github.com/cynch-me/probara-sdk/blob/main/packages/cypress-reporter/docs/run-selection.md)