Syncing feature files from Git

Git can be the source of truth for Gherkin cases. A CI job reads your .feature files and makes the cases match them: a new scenario creates a case, a changed scenario updates its case, and a removed scenario archives its case. These are synced cases. You change their scenarios in the .feature file, through a pull request. Everything else about them stays editable in Probara.

Set up the sync

1. In Workspace → Integrations, open the Cucumber card and create an app token for your pipeline. The sync needs a paid plan, as CI reporting does. The CLI page has the details.
2. Save the token as the CI secret PROBARA_API_TOKEN, and set PROBARA_PROJECT to your project code.
3. Run the sync after checkout, on your default branch:

- name: Sync the feature files to Probara
  run: npx @probara/cli sync gherkin "features/**/*.feature"
  env:
    PROBARA_API_TOKEN: ${{ secrets.PROBARA_API_TOKEN }}
    PROBARA_PROJECT: <PROJECT_CODE>

Quote the pattern so your shell leaves it to the CLI. Run the command from the repository root, or pass --root-dir: the paths in the keys and in the file links are relative to it. For a first run, add --dry-run. It prints what a real sync would create, update, move, restore or archive, and writes nothing.

The CI providers page has a complete pipeline that syncs and runs the tests for GitHub Actions, GitLab CI, Jenkins, CircleCI, Azure Pipelines, Bitbucket Pipelines and Buildkite.

What a synced case is

• Its title, description, steps, precondition and the tags its file gives it are read-only, in the editor and in the API. The case shows a Synced from Git badge with its path and line, linked to the file when the sync knows the link.
• Everything else stays editable: assignee, priority, custom fields except the precondition, defects, issue links, comments, plan and run membership, the tags you add, and the suite. The next sync puts the case back in its file’s suite.
• Review does not apply to synced cases. The pull request that changes the file is their review, so sending a synced case for review and restoring one of its earlier versions are refused.

How a scenario maps to a case

Steps keep their English keywords: Given, When, Then, And, But and *. A file written in another language stops the sync.

Keys

Each scenario has a key, <path> > <Feature> > <Scenario>, such as features/checkout/pay.feature > Checkout > Pay with a card. The sync and the results import both build it, so a result finds the case its scenario synced to.

• The path is relative to the repository root and uses / separators.
• A Rule: is not part of the key. A Scenario Outline has one key, and its Examples rows are not part of it.
• Two scenarios with the same key in one file stop the sync. The error names both lines, so rename one of them.
• Renaming a file, its Feature or a scenario changes its key. The sync then creates a case, unless a binding or a move finds the existing one.

Moves, renames and bindings

The sync matches each scenario in this order. Each case is matched once.

1. Binding. A @probara:<ID> tag binds the scenario to that case, whatever its key. Use it when you rename a scenario and change its steps in the same commit. An ID that names no case of the project, or a case of another project, and two scenarios bound to one case, stop the sync, and nothing is written.
2. Key. The case that holds the scenario’s key is updated.
3. Content. A new key moves a synced case that disappeared in the same sync, when the scenario’s steps, Background and Examples are the same. The case keeps its history under the new key. Names, tags, the description and the link do not count, so a rename that leaves the steps unchanged is a move. A move needs exactly one such case. Otherwise the scenario gets a new case, and the old one is archived.

A manual case that a key or a binding finds becomes a synced case. The sync takes the file’s title, description, steps and suite. Its old steps are replaced, so their images are removed and its steps no longer use shared steps.

A file moved to a folder that another job syncs gets a new case in that folder. The old case is archived only by the job that syncs its old folder.

Removed scenarios and scope

A synced case whose scenario is gone is archived, never deleted, and it waits in Trash. Only cases whose file matches one of the sync’s arguments can be archived. Cases of files outside those arguments are not touched, so each folder can have its own job:

npx @probara/cli sync gherkin "features/checkout/**/*.feature"
npx @probara/cli sync gherkin "features/cart/**/*.feature"

• A scenario that comes back restores its archived case. If you archive a synced case yourself, the next sync that still includes its scenario restores it. To keep a synced case archived, remove its scenario from the file.
• A glob that matches no file stops the command, so a typo never archives a folder.
• One sync takes at most 1,000 scenarios and 20 arguments. A larger repository syncs one folder per job.

Links to the files

The CLI links each synced case to its scenario’s line. It builds the link from the CI it runs in: GitHub Actions, GitLab CI, Bitbucket Pipelines, or Azure Pipelines for a repository in Azure Repos. For any other host, pass a template:

npx @probara/cli sync gherkin "features/**/*.feature" --file-url "https://git.example.com/acme/shop/blob/main/{path}#L{line}"

Only https links are sent. Other CI systems send no link, and the case shows its path.

Results from Cucumber

Cucumber results use the same keys, so a run fills in the synced cases it covers. For cucumber-js, the Cucumber reporter sends each scenario’s results at the end of the run, with the files its steps attach. Add it to cucumber.js:

module.exports = {
  default: {
    format: ['@probara/cucumber-reporter'],
    formatOptions: { probara: { projectId: 'SHOP' } },
  },
};

For Cucumber-JVM, playwright-bdd and other tools that write Cucumber JSON, import the report after the run:

npx cucumber-js --format json:results/cucumber.json
npx @probara/cli import cucumber results/cucumber.json

• Each scenario is one result. Each row of a Scenario Outline is one combination of its case, and its result lands on the run case with the row’s values.
• Each step is a result step with its keyword, status, duration and data (a Doc String or a Data Table). The Background steps sit under one Background step.
• A step that passes is Passed. A step that fails, or matches more than one step definition, is Failed. A step that is skipped, pending or undefined did not run, so it is Skipped. A scenario is Failed when a step or a Before or After hook failed, Skipped when a step did not run, and Passed otherwise.
• Keep the feature files in the checkout when you import. A result finds its case only when its feature file is there, because the key includes the path.
• Import from the directory Cucumber ran in, or pass it as --root-dir, and give the sync the same root. The report’s paths start from that directory.
• The import does not send the attachments of a report, such as screenshots. The result keeps its steps, statuses, durations and errors. The Cucumber reporter sends the files a scenario attaches.

The import reads the Cucumber JSON report. Probara is tested against cucumber-js 13. Cucumber-JVM and playwright-bdd write the same layout, which Probara has not tested.

Use the Cucumber JSON report rather than JUnit for BDD suites. A JUnit test has one name and one status, with no step keywords, data or Examples values, so Probara cannot rebuild the key of a scenario, or find the run case of an Outline row, from it.

Coming from another tool

Probara reads one binding tag, @probara:<ID>. The tags of other tools, such as Xray’s @TEST_ABC-12, Zephyr Scale’s @TestCaseKey=ABC-T1, Qase’s @QaseID=12 or TestRail’s @C123, link nothing. The sync keeps them as file tags of the case, so remove them from the files once you have switched.

There are two ways to switch:

• Start from the files. Run the sync with --dry-run, then for real. Each scenario becomes a new case. The cases and results of the old tool stay there.
• Keep the cases you imported. Import the cases from a Qase or TestRail export (see Importing test cases). Each one gets a Probara ID: find it by the case’s title. In the files, replace each old tag with a binding to that ID, such as @QaseID=12 with @probara:SHOP-45, and run the sync. Each bound case becomes a synced case. Its title, steps and suite come from the file, and its fields, comments and defects stay.

Xray and Zephyr Scale exports are not an import source, so start from the files.

Troubleshooting

API and MCP

The sync is POST /api/v1/projects/{projectId}/gherkin-sync, and the sync_gherkin_features MCP tool calls the same endpoint. See the API reference and the MCP tool reference.