Running tests with GitHub Actions

From a project’s runs page you can start your test workflow in GitHub Actions. Probara creates the run, starts the workflow on the branch you choose, and shows the build on the run with its link, status, start, end and duration. The workflow’s test jobs report their results into that same run through the Probara reporters, and a final job closes it.

This needs the GitHub integration installed on the repository, and a workflow prepared once as described below.

Before you start

1. Add the token and the project to the repository

1. In Probara, open Workspace › Integrations, select your reporter’s card (for example Playwright), and create an app token in its Access tokens tab. Copy it: Probara shows it only once.
2. In GitHub, open the repository’s Settings › Secrets and variables › Actions.
3. On the Secrets tab, select New repository secret. Name it PROBARA_API_TOKEN and paste the token.
4. On the Variables tab, select New repository variable. Name it PROBARA_PROJECT and enter the project’s ID, such as SHOP.

Never commit the token to the repository.

2. Prepare the workflow

A workflow Probara can launch follows three rules:

This workflow runs Playwright in four parallel shards when Probara launches it, and on pushes to main and on pull requests. Its final job closes the run when all of the shards finish. Replace the test steps with your own:

name: Probara E2E

run-name: ${{ inputs.probara_run_id && format('Probara {0}', inputs.probara_run_id) || github.workflow }}

on:
  push:
    branches: [main]
  pull_request:
  workflow_dispatch:
    inputs:
      probara_run_id:
        description: Probara run ULID (set by Probara when it launches this workflow)
        required: false
        type: string

env:
  PROBARA_API_TOKEN: ${{ secrets.PROBARA_API_TOKEN }}
  # Fork pull requests get no secrets: reporting stays off there.
  PROBARA_ENABLED: ${{ secrets.PROBARA_API_TOKEN != '' }}
  PROBARA_PROJECT: ${{ vars.PROBARA_PROJECT }}
  # Empty unless Probara launched the workflow: the reporters read an empty ULID as unset.
  PROBARA_RUN_ULID: ${{ inputs.probara_run_id }}

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        shard: [1, 2, 3, 4]
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: npm
      - run: npm ci
      - run: npx playwright install --with-deps
      # The Probara reporter in playwright.config reads the PROBARA_ variables.
      - run: npx playwright test --shard=${{ matrix.shard }}/${{ strategy.job-total }}
        env:
          # Each shard leaves the run open; the final job closes it.
          PROBARA_CLOSE_RUN: false

  close-run:
    needs: test
    if: always()
    runs-on: ubuntu-latest
    steps:
      - uses: actions/setup-node@v4
        with:
          node-version: 22
      - run: npx --yes @probara/cli@0.6 run close

• Shards share one run. On a launch, every shard reports into the run Probara created. On push and pull_request, the shards of one workflow run share a run of their own: the first shard to report creates it.
• The final job closes the run. if: always() closes it even when tests fail. Without that job, the run stays open after the workflow ends; you can still close it from Probara.
• One job only? Add the run close step at the end of that job, with if: always().

On push and pull_request, no run is created first. A re-run of the workflow in GitHub is a new attempt, so it reports into a run of its own.

3. Launch a run

1. Open the project’s Runs and select Run in GitHub Actions. The button appears when a GitHub account is connected and your plan includes integrations.
2. Choose the Repository. The list follows the project’s repositories.
3. Choose the Branch and the Workflow. Only active workflows are listed. To use a branch the list does not show, type its full name in the branch search and pick Use “name”.
4. Optionally enter a Run name. By default it is the workflow and the branch.
5. Select Start run.

Probara creates the run, starts the workflow once and opens the run. If GitHub refuses to start the workflow, Probara shows why and keeps no run. If GitHub does not answer, Probara keeps the run and says “GitHub didn’t confirm the start. If the workflow runs, its build appears on the run.”

Follow the build

The run header shows a GitHub Actions card while the workflow runs:

• the status: Starting, Dispatched, Queued, In progress, then the conclusion, such as Succeeded, Failed or Cancelled;
• Open in GitHub, the link to the workflow run;
• Started, Finished and Duration, which counts up while the workflow runs;
• the attempt, when it is not the first.

The card follows the workflow; it never closes the run. Results appear on the run as the reporters send them. A run’s public share link never shows the build.

Under CI source, the run also records the Repository, Branch and Workflow it was launched with.

Re-run a launched workflow

GitHub re-runs a workflow with the same inputs, so a re-run names the launched run again. Probara gives each re-run attempt a run of its own, named after the launched run with its attempt, such as “Nightly E2E · attempt 2”, and closes it when the attempt finishes in GitHub. The launched run keeps the results and the build of the first attempt.

To test again in a fresh run started from Probara, launch the workflow again instead.

Pull request comments

When a run that records its GitHub Actions pull request is closed or aborted, Probara can comment its results on that pull request, by default only when tests fail. Each comment is a new one and hides the previous one (see Pull request comments).

A launched run starts from a branch, so it records no pull request. On a pull_request workflow, the reporters and the CLI read the pull request number from the GitHub event, so the run its jobs report into records it. Reports sent to the reports API can also carry the repository and the pull request number in run.source.

Troubleshooting

Automate it

Scripts and AI assistants can launch the same way with an API token or Probara MCP: the endpoints are in the GitHub API page, and the MCP tools are list_github_workflows, list_github_branches and launch_github_workflow.