Vitest steps and attachments

The steps of a test show under its result in Probara, nested as they ran, each with its status, duration and error. Vitest has no steps of its own: write them with probara.step(), which runs its body and records it. A result also carries the files its attempt attached with probara.attach() and with Vitest’s own context.annotate(), the screenshots Vitest takes of a failure in browser mode and, with captureOutput, the attempt’s console output.

`probara.step()`

import { probara } from '@probara/vitest-reporter';
import { expect, test } from 'vitest';

test('checks out', async () => {
  const cart = await probara.step('Open the cart', async () => ({ items: 2 }));
  await probara.step('Pay', async () => {
    probara.step('Enter the card', () => {
      expect(cart.items).toBe(2);
    });
    await probara.step('Confirm', async () => {
      expect(await Promise.resolve('declined')).toBe('paid');
    });
  });
});

[
  {
    "status": "failed",
    "steps": [
      { "action": "Open the cart", "status": "passed" },
      {
        "action": "Pay",
        "status": "failed",
        "steps": [
          { "action": "Enter the card", "status": "passed" },
          { "action": "Confirm", "status": "failed" }
        ]
      }
    ]
  }
]

probara.step(title, body, options) runs body and returns what it returns, so a step can compute a value the test uses (cart above):

The body	The step	What `probara.step()` does
Returns	`passed`	Returns its value
Throws	`failed`, with the error’s message and stack	Throws the same error: the test fails as usual
Returns a promise (`async`)	Ends when the promise settles: `passed`, or `failed`	Returns a promise that settles the same way
Returns another thenable	Ends when the body returns (see below)	Returns it untouched
None (`probara.step('Seeded')`)	`passed`, with no duration	Returns `undefined`

Nesting follows the calls: a step started inside another step’s body is its child, across await too. Two steps of the same parent run with Promise.all are both its children. An error that leaves a step’s body fails the step, and each parent step it leaves too.
A step that never ended (a promise the test did not await, still pending when the test ends) is failed, with the error The step had not finished when the test ended. Await every probara.step() that has an async body.
Hooks: a step in beforeEach or afterEach is a step of the running test. In beforeAll, afterAll or a describe body no single test runs: the body runs, and the step is not recorded (where to call the helpers).
Retries: each attempt records its own steps; a retry starts with none (retries and repeats).
Browser mode: a browser has no async context to follow, so a step nests under another only when it starts in the synchronous part of the other’s body. A step started after an await in a body is not that step’s child.

Steps of a new case

A result’s steps record what ran. A case’s steps are its specification: an action, the expected result and the test data. The outermost steps of an attempt, in the order they started, are the steps of the case the report creates; expected and data go with them:

import { probara } from '@probara/vitest-reporter';
import { test } from 'vitest';

test('refunds an order', () => {
  probara.step('Open the order', () => {}, { expected: 'The order is shown', data: 'order=1042' });
  probara.step(
    'Refund it',
    () => {
      probara.step('Confirm the dialog', () => {});
    },
    { expected: 'The order is refunded' },
  );
});

[
  {
    "steps": [
      { "action": "Open the order", "expected": "The order is shown", "data": "order=1042" },
      {
        "action": "Refund it",
        "expected": "The order is refunded",
        "steps": [{ "action": "Confirm the dialog" }]
      }
    ],
    "case": {
      "steps": [
        { "action": "Open the order", "expected": "The order is shown", "data": "order=1042" },
        { "action": "Refund it", "expected": "The order is refunded" }
      ]
    }
  }
]

Nested steps (Confirm the dialog) are steps of the result only: how a test works, not what it specifies.
Like every case field, case steps apply only when the report creates the case (created cases only). An existing case keeps its own steps.
expected and data are strings: a step given anything else is recorded with its title only, and one warning says so.

Promises and thenables

A body that returns a promise ends its step when the promise settles. probara.step() returns a new promise that settles the same way, so await probara.step(...) behaves as awaiting the body. A rejection the test never handles is still an unhandled rejection, which Vitest reports, as it would without the step.

A body can also return a thenable that is not a promise: a supertest request, a query builder. Calling its then() would start it (or start it twice), so probara.step() returns it untouched, and the step ends as soon as the body returns, before the request is sent. To time the request inside the step, await it in an async body:

import { probara } from '@probara/vitest-reporter';
import { expect, test } from 'vitest';

// A stand-in for a supertest request: it runs when awaited.
const request = () => ({
  then: (resolve: (response: { status: number }) => void) => resolve({ status: 200 }),
});

test('answers the health check', async () => {
  const response = await probara.step('Call /health', async () => await request());
  expect(response.status).toBe(200);
});

[{ "status": "passed", "steps": [{ "action": "Call /health", "status": "passed" }] }]

`probara.attach()`

Give it a file on disk or a body, and await it. It works from anywhere a test runs: the test body, a beforeEach or afterEach hook, onTestFinished, a step, a helper or a page object the test calls.

import { writeFileSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { probara } from '@probara/vitest-reporter';
import { test } from 'vitest';

test('prints the receipt', async () => {
  const path = join(tmpdir(), 'receipt.txt');
  writeFileSync(path, 'Total: 25.00');
  await probara.attach({ name: 'receipt', path });
  await probara.attach({
    name: 'order',
    body: JSON.stringify({ id: 1042, total: 25 }),
    contentType: 'application/json',
  });
  await probara.attach({ name: 'log', body: 'paid in 1.2 s' });
});

The result gets three files:

receipt.txt text/plain
order.json application/json
log.txt text/plain

Field	With `path`	With `body`
`name`	Required: the name in Probara; the file’s extension is added when it has none	Required: the name in Probara; the content type’s extension is added when it has none
`path`	The file, absolute or relative to the directory `vitest` runs in	—
`body`	—	A string, a `Buffer` or a `Uint8Array`, sent as it is
`contentType`	Optional: guessed from the file name	Optional: guessed from `name`’s extension, else `text/plain` for a string, `application/octet-stream` for bytes

Copied when called. The file (or the body) is copied at once, so a test can delete or overwrite it right after: what Probara gets is what it was at the call.
Await it. In Node the copy is made before the call returns; in browser mode it ends when the promise settles, and a copy that ends after its test ended is not attached, with a warning that says Await probara.attach() before the test ends (probara.attach() in a browser).
A string is sent as it is, as UTF-8 text, never decoded: pass a Buffer or a Uint8Array for binary content.
A file that cannot be attached (it does not exist, it is a folder, it is larger than 32 MiB) is one warning, such as probara.attach() did not attach "receipt": it does not exist; the test goes on, and its result is sent.
Each attempt has its own files: a retried test uploads the files of every attempt, each to its own result.

Files of a step

A file attached while a probara.step() runs, from the step’s body or anything it calls, goes to that step, and Probara shows it under the step:

import { probara } from '@probara/vitest-reporter';
import { test } from 'vitest';

test('uploads an avatar', async () => {
  await probara.step('Choose the file', async () => {
    await probara.attach({ name: 'avatar.png', body: new Uint8Array([137, 80, 78, 71]) });
  });
});

avatar.png image/png

Vitest annotations

The files of Vitest’s context.annotate(message, attachment) are sent with the attempt that made them, named by the annotation’s message. The message itself is not sent: probara.comment() writes a note.

import { test } from 'vitest';

test('exports the report', async ({ annotate }) => {
  await annotate('summary', { body: 'Exported 3 rows', bodyEncoding: 'utf-8' });
});

summary.txt text/plain

Vitest	A text `body`	Binary content
4.1.3 and later, 5	Pass `bodyEncoding: 'utf-8'`: without it, Vitest labels a body base64	A `Uint8Array` body with its `contentType`, or a `path`
3.2, 4.0 to 4.1.2	As it is: no `bodyEncoding` yet	A `path`: Vitest 3 loses some bytes of a binary body, and such a body is left out, with a warning

A path is copied by Vitest into its attachmentsDir, and the file in Probara gets the annotation’s message and the path’s extension: annotate('screen', { path: 'shot.png' }) is screen.png. Vitest refuses annotate() once the test body ended: in afterEach and onTestFinished, use probara.attach().

Failure screenshots

In browser mode, Vitest takes a screenshot when a test fails, and the reporter attaches it to the failed execution, image/png: the first screenshot to the first failed attempt, and so on, or every screenshot to the last failed attempt when their counts differ. attachScreenshots: false (PROBARA_ATTACH_SCREENSHOTS=false) leaves them out. A test that runs in Node has none.

Console output

captureOutput: true (PROBARA_CAPTURE_OUTPUT=true) attaches what each attempt writes to the console: console.log, console.info and console.debug as stdout.log, and console.warn and console.error as stderr.log, text/plain, each only when it has text. It is off by default, and needs nothing else: no setup file, no change to the tests. Vitest still prints the output as usual.

reporters: ['default', ['@probara/vitest-reporter', { captureOutput: true }]],

import { beforeEach, test } from 'vitest';

beforeEach(() => {
  console.info('Seeding the cart');
});

test('prints what it does', () => {
  console.log('Opening the cart');
  console.error('Slow response: 2.1 s');
});

stdout.log text/plain
stderr.log text/plain

Here stdout.log holds Seeding the cart and Opening the cart, and stderr.log holds Slow response: 2.1 s: a test’s beforeEach and afterEach hooks write into its attempt.

What each attempt gets

Output	Where it goes
The test body, its `beforeEach` and `afterEach` hooks	The attempt that wrote it
Each attempt of a retried or repeated test	Its own `stdout.log` and `stderr.log`
A `test.concurrent` test, or a test in a `describe.concurrent`	Left out, with one warning
A retried test whose attempts were not recorded	Left out, with one warning (see when retries run)
The module scope, a `describe` body, `beforeAll`, `afterAll`	Not attached: Vitest gives it to no test

Concurrent tests are left out. Vitest gives each write to the test current in its worker, which is not always the test that wrote it when tests run at once: rather than attach a write to the wrong test, the reporter leaves the output of concurrent tests out, and says how many.
Retries are split by time. An attempt’s files hold what Vitest received from its start to the start of the next attempt. Vitest hands the reporter a test’s writes in batches, so a write made right as a retry starts can land in the file of the attempt before it.

reporters: ['default', ['@probara/vitest-reporter', { captureOutput: true }]],

import { test } from 'vitest';

test.concurrent('finds boots', async () => {
  console.log('searching for boots');
});

test('finds hats', () => {
  console.log('searching for hats');
});

Only the test that runs alone gets its output, stdout.log, and the log says why the other has none:

$ npx vitest run
[probara] The console output of 1 test is left out: Vitest attributes the writes of concurrent tests to the wrong test, and the attempts of a retried or repeated test without recorded attempts cannot be told apart (first seen in the run; repeats are logged at debug)
[probara] Sending 2 results of 2 tests (2 passed, 0 failed, 0 skipped, 0 blocked)
[probara] Recorded 2 results (2 new cases, 0 unmatched) in R-1 (closed): https://app.probara.net/projects/SHOP/runs/R-1
[probara] Attached 1 file to results (0 skipped, 0 failed)

`silent` and `onConsoleLog`

Vitest’s silent and onConsoleLog decide what the terminal shows, not what reporters get: the output they hide is still attached. To keep a test’s output out of Probara, leave captureOutput off.

silent: true,
reporters: ['default', ['@probara/vitest-reporter', { captureOutput: true }]],

import { test } from 'vitest';

test('works quietly', () => {
  console.log('nothing shows in the terminal');
});

stdout.log text/plain

Limits of the output

32 MiB per test and stream. Past it, the stream keeps what fits and ends with the line [probara] The output of this test was cut here: an attachment holds at most 32 MiB. The output of a whole run is not capped.
Kept in memory. The reporter holds the output of the run’s tests until the run ends, when it sends the results: a run whose tests write a lot needs the memory for it.
Sharded runs keep it. A shard’s blob holds its tests’ output, and vitest --merge-reports attaches the same files as a run that was not sharded (sharding).

Limits and what is skipped

Limit	What happens
32 MiB per file or body	A larger one is skipped, with a warning
20 files per result	The files after the first 20 are skipped, with one warning
Images over 10 MiB or 8192 px wide or tall	Skipped with a warning (Probara converts images to WebP)
Executables and scripts	Skipped: Probara refuses their content types
A missing, unreadable or empty file	Skipped with a warning
A result that recorded nothing	Its files are not uploaded

Warnings name the file, and the log ends with how many files were attached, skipped and failed. uploadAttachments: false (PROBARA_UPLOAD_ATTACHMENTS=false) uploads none. Files upload after their report is recorded, while the run is still open, then the run is closed.

Where the copies are kept

The reporter keeps its copies in <attachmentsDir>/probara/<id>/, under Vitest’s attachmentsDir (.vitest-attachments on Vitest 3 and 4, .vitest/attachments on Vitest 5, by default), until they are sent:

The run	Its copies
Every reporter is one Vitest ships (`default`, `verbose`, `junit`…) or this one	Removed once the run is sent
A shard (`--shard`), a run with `blob`, or any other reporter	Kept: a blob may name them, and a merge reads them
Results that could not be sent, with `resultsFile`	Copied next to the results file, in `<name>-attachments/`

Register the reporter by its package name, '@probara/vitest-reporter'. Given by a path or as an object, it counts as another reporter, and its copies are kept.
Carry attachmentsDir with the blobs. The blob of a shard names the files it attached by their place on the shard’s machine: copy the shard’s attachmentsDir along with its blob to the job that runs vitest --merge-reports, at the same path. A file the merge cannot find is skipped with the warning Skipped an attachment: its file is missing or unreadable, and its result is sent all the same (carry the blobs and the attachments).
Vitest’s own copies of annotation paths stay in attachmentsDir: Vitest never removes them, and neither does the reporter. Remove the folder when it grows.

When steps or files are missing

A step is missing. Only probara.step() makes steps: expect calls and helper functions do not. A step called outside a test (beforeAll, a describe body) runs its body and records nothing, with a warning.
Steps are flattened, or fail with The step had not finished. A step started without await ends after the test: await every probara.step() whose body is async.
The case has no steps. Only a case the report creates takes steps. An existing case keeps its own.
A file is missing. The log names every file it left out, and why (an attachment is missing). A file attached inside a step goes to that step, not to the result.