Metadata-Version: 2.5
Name: verifaied
Version: 0.15.0.dev33
Summary: Find what's untested in your code — locally, no account required
Project-URL: Homepage, https://pypi.org/project/verifaied/
Author: Kyle Richards
License: MIT License
        
        Copyright (c) 2026 Kyle Richards
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
License-File: LICENSE
Keywords: agents,ai,coverage,llm,pytest,testing
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.10
Requires-Dist: httpx>=0.27.0
Requires-Dist: rich>=13.7.0
Requires-Dist: tomli>=2.0.1; python_version < '3.11'
Requires-Dist: tomlkit>=0.13.0
Requires-Dist: typer>=0.12.0
Provides-Extra: audit
Requires-Dist: playwright<2,>=1.49; extra == 'audit'
Description-Content-Type: text/markdown

# verifaied

Find what's untested in your code, so you (or your coding agent) can fix
it without waiting for CI.

Two verbs:

- **`verifaied check`** — runs entirely on your machine. No account, no API
  token, no network call. Tells you which Python functions are untested or
  only partially covered, and can hand you a ready-to-paste prompt for each
  one.
- **`verifaied upload`** — syncs the same coverage to the [verifAIed](https://verifaied.app)
  app, which adds history, branch diffing against CI, a web UI, and AI-written
  test prompts. Handles both Python (pytest-cov) and TypeScript/JavaScript
  (vitest/Istanbul). Needs a free account.

## Install

```bash
uv tool install verifaied        # or
pipx install verifaied
```

## `verifaied check` — local, no account

```bash
pytest --cov --cov-branch --cov-report=json
verifaied check
```

```
  12 covered  3 partial  2 untested   (17 functions)

   function        file                   missing
●  apply_topup     app/billing.py:41      42-47
●  refund          app/billing.py:60      61-68
◐  _resolve_plan   app/plans.py:12        18
```

Nothing leaves your machine — the whole analysis is a local AST pass over the
source `coverage.json` already points at.

`check` is Python-only: it reads pytest-cov's `coverage.json`. Point it at a
vitest/Istanbul report and it exits with a note to use `verifaied upload`,
which analyzes TypeScript/JavaScript on the hosted side.

Add `--prompt` to get a concrete, ready-to-paste instruction for each one,
generated from the function's signature and its uncovered lines (no LLM, no
cost, unlimited):

```bash
verifaied check --prompt
```

```
Write a pytest test for `apply_topup` in `app/billing.py`.

This function is currently **untested** — no line of it executes under the
existing suite.

What to do:
- Call `apply_topup(user_id, amount)`.
- Assert on the **return value** — compare it to the exact expected value.
- Cover the lines that never execute: **42-47**.
- It raises `ValueError` — cover that path with `pytest.raises(ValueError)`.
```

Useful flags: `--limit N` (how many to list, `0` for all), `--root <path>` (the
directory the coverage paths are relative to — pytest's rootdir), and
`--fail-under <pct>`, which exits `3` when too few functions are fully covered,
so `check` works as a CI gate.

## `verifaied upload` — sync to the app

Mint an API token from the verifAIed app (Settings → Tokens; a free account is
enough). The CLI ships pointing at the hosted API (`https://api.verifaied.app`);
point it elsewhere via `VERIFAIED_API_URL` if you're running the backend locally
or self-hosting:

```bash
export VERIFAIED_API_TOKEN=vr_live_...
# Only set this if you're not using the hosted API:
export VERIFAIED_API_URL=http://localhost:8000   # local dev
```

From inside any git repo you've connected to verifAIed:

```bash
pytest --cov --cov-report=json --junitxml=junit.xml
verifaied upload
```

That's it — `--repo` is optional. The CLI parses `git remote get-url origin`,
looks the repo up under your account via `/installations/me`, and uses the
resulting UUID. You can also be explicit:

```bash
verifaied upload --repo kyle/verifaied        # owner/name slug
verifaied upload --repo <UUID>                # raw UUID, no lookup
```

The CLI reads your coverage report(s), pulls the source for every file
they reference from your working tree, and posts everything to
`/repositories/<id>/local-coverage`. With no `--coverage` flag it
auto-discovers the report, looking in order for `./coverage.json`
(pytest-cov), then `./coverage/coverage-final.json`, then a bare
`./coverage-final.json` (vitest/Istanbul). Pass `--coverage` one or more
times to name reports explicitly — repeat it to push Python and
TypeScript coverage in a single upload; multiple reports merge per
language on the server. The response prints a summary of untested /
partial / failing functions so you (or your LLM) can fix them on the
next iteration.

### What the upload reports

The summary leads with counts so an LLM (or you) sees the deltas first:

- **functions matched** — functions in the uploaded report the analyzer
  resolved against your source.
- **functions preserved** — functions retained from a previous upload in
  another language for the same branch (only shown when non-zero). A
  Python-only upload never wipes the TypeScript functions a vitest upload
  recorded on the same branch, and vice versa — the branch's total tracked
  functions is _matched + preserved_.
- **untested / partial / failing tests** — what to fix next.

### What gets uploaded

The CLI only sends files that appear in your coverage report — i.e.
exactly the files your test run instrumented (pytest-cov's `"files"`
map, or the top-level file keys in an Istanbul `coverage-final.json`).
There is no directory walk and no glob:

- `.env`, build artefacts, vendored libraries, and anything outside
  your coverage scope are never read.
- Test files themselves are only uploaded if your coverage config
  includes them (e.g. `--cov=tests`).

To audit the exact file list (and total bytes) before anything leaves
your machine, run with `--dry-run`:

```bash
verifaied upload --dry-run
```

This prints the resolved branch / commit / file table and exits without
contacting the backend. Useful when you're about to upload from an
unfamiliar repo, or when narrowing down where an unexpected file is
coming from.

If a sensitive value did make it into a covered source file (a baked-in
API key in a fixture, etc.), open the repo in the verifAIed web UI,
expand **Recently deleted** under the branches grid, and use
**Permanently delete**. That hard-deletes the analysis row and cascades
to the uploaded source — the per-card **Delete** is a soft-delete that
keeps the data around for accidental-deletion recovery.

### TypeScript / JavaScript

`upload` ingests vitest/Istanbul coverage the same way it ingests
pytest-cov's. Generate it with vitest's **Istanbul** provider:

```bash
npm i -D @vitest/coverage-istanbul
vitest run --coverage --coverage.provider=istanbul --coverage.reporter=json \
  --reporter=junit --outputFile.junit=junit.xml
verifaied upload --junit junit.xml
```

The Istanbul provider is required. Istanbul applies your source maps, so
`coverage/coverage-final.json` reports the original `.ts` / `.tsx`
sources — which is what verifAIed matches function definitions against.
vitest's default **v8** provider emits a different report shape that
verifAIed rejects, so `--coverage.provider=istanbul` is not optional. The
`--reporter=junit --outputFile.junit=junit.xml` flags produce the JUnit
XML that feeds the failing-tests panel; hand it to the upload with
`--junit`.

If a report references only compiled output — paths under `dist/`,
`build/`, `out/`, or `.next/` with no `.ts` / `.tsx` / `.jsx` entries —
the upload is refused. That's the source-maps-not-applied case: coverage
landed on bundled JS instead of your sources, and verifAIed can't map it
back. Switching to the Istanbul provider is the fix.

**Supported formats.** pytest-cov `coverage.json` and Istanbul
`coverage-final.json` only. LCOV, Cobertura XML, jest's own format, and
c8 / v8-style JSON are rejected with a clear error naming the two
supported shapes.

### `upload` flags

- `--repo / -r <UUID|owner/name>` — repository to upload to. If omitted,
  the CLI auto-detects from the GitHub `origin` remote.
- `--branch / -b <name>` — branch to attach the upload to (default:
  `git branch --show-current`, then `local`)
- `--coverage <path>` — coverage report to upload. **Repeatable** — pass
  it once per report to combine Python and TypeScript coverage in one
  upload. Accepts pytest-cov `coverage.json` and Istanbul
  `coverage-final.json`. Default: auto-detect `./coverage.json`, then
  `./coverage/coverage-final.json` (or a bare `./coverage-final.json`).
- `--junit <path>` — optional JUnit XML for failing-test detail
- `--commit-sha <sha>` — commit sha for display (default: `git rev-parse HEAD`)
- `--root <path>` — root the coverage paths are relative to (default: cwd)
- `--api-url <url>` — backend base URL (overrides `VERIFAIED_API_URL`)
- `--token <token>` — API token (overrides `VERIFAIED_API_TOKEN`)
- `--dry-run` — print the file list and total bytes that would be
  uploaded, then exit without contacting the backend. Skips the token
  requirement so you can audit without configuring auth.

## `verifaied check-done` — the "am I done?" gate

`check-done` is the **stop condition** for an agent's test-writing loop.
It fetches a verdict for a branch — failing tests plus the coverage of the
functions you changed — and exits accordingly, so a CI job or an agent can
gate on it:

```bash
pytest --cov --cov-report=json --junitxml=junit.xml
verifaied upload            # push the fresh coverage first
verifaied check-done        # then ask: am I done?
```

The verdict composes into `{done, status, gaps}`:

- `done` (bool) — `true` only when there are **no gaps at all**. The
  strict signal to stop.
- `status` — `done` | `done-with-warnings` | `not-done`.
  `done-with-warnings` means the only gaps left are ones the repo owner
  marked non-blocking (per-repo policy, set in the web UI).
- `gaps[]` — each remaining problem with a `ref`, whether it's `blocking`,
  and a `next_action` pointing at the free (`get_baseline_prompt`) or paid
  (`fix_branch`) remedy.
- `not_checked[]` — signals that couldn't be evaluated (e.g. failing tests
  when no `junit.xml` was uploaded), surfaced so a green verdict is never
  silently green.

### Exit codes

- `0` — done (or `done-with-warnings` with `--allow-warnings`)
- `3` — not done: blocking gaps remain (the CI-gate failure code)
- `2` — HTTP/API error (auth, 404, network)
- `1` — usage/config error (no token, bad `--repo`)

### `check-done` flags

- `--repo / -r <UUID|owner/name>` — repository to check (default:
  auto-detect from the GitHub `origin` remote)
- `--branch / -b <name>` — branch to judge (default:
  `git branch --show-current`)
- `--json` — print the raw verdict JSON instead of the human summary
- `--allow-warnings` — treat `done-with-warnings` as passing (exit 0);
  default is strict (only a clean `done` exits 0)
- `--api-url <url>` — backend base URL (overrides `VERIFAIED_API_URL`)
- `--token <token>` — API token (overrides `VERIFAIED_API_TOKEN`)

### As an agent stop condition

Drop this loop into your agent's instructions file so it self-corrects
and knows when to stop:

```text
Test-coverage loop (run until done):
  1. Run tests with coverage:  pytest --cov --cov-report=json --junitxml=junit.xml
  2. Push the results:         verifaied upload
  3. Ask if you're done:       verifaied check-done
  4. If exit code is 0, stop — you're done.
     If exit code is 3, fix each gap in the output (use get_baseline_prompt
     for a free test prompt, or fix_branch for an LLM-written one), then
     go back to step 1.
```

## `verifaied audit` — measure the rendered page

Coverage says a browser *reached* your component. It can't say the button
ended up behind the modal. `verifaied audit` loads your running app at
several viewport widths and measures it — no LLM, no cost:

- **Overflow** — anything spilling out of the viewport, page or element.
- **Contrast** — text below the WCAG AA minimum (4.5:1, or 3:1 for large
  text), skipping anything sitting on a background image, where computed
  styles can't answer honestly.
- **Broken images** — an `<img>` that finished loading with nothing in it.
- **Console errors** — what the page shouted while it rendered.

It drives a real browser, so Playwright ships as an optional extra rather
than a dependency of every command:

```bash
pip install 'verifaied[audit]'
playwright install chromium

verifaied audit http://localhost:5173
```

```
  viewport   findings   console errors
  375x812           2                0
  768x1024          0                0
  1280x800          0                0

375x812  http://localhost:5173/
  element_overflow  div.card  Extends 40px past the right edge of the 375px viewport.
  low_contrast      p.muted   Text contrast is 2.10:1 against its background, below…
```

Findings are recorded against the branch as `visual_defect` gaps — **the
one `check_done` signal that is a warning by default**. An audit doesn't
know which of your choices were deliberate (a horizontally-scrolling
table, a muted caption), so it reports loudly and blocks nothing until you
turn it on under **Definition of done**. Only the most recent audit
counts: re-run it after a fix and a clean pass clears what the last one
raised.

Each viewport's full-page screenshot is stored with the run, so every
finding has its frame beside it.

### Exit codes

- `0` — clean: nothing found at any viewport
- `3` — findings recorded (the CI-gate failure code)
- `2` — the audit could not be run or could not be recorded
- `1` — usage/config error (no token, bad `--repo`, bad `--viewports`, or
  Playwright not installed)

### `audit` flags

- `--repo / -r <UUID|owner/name>` — repository to record against
  (default: auto-detect from the GitHub `origin` remote)
- `--branch / -b <name>` — branch (default: `git branch --show-current`)
- `--viewports <widths>` — comma-separated widths in pixels (default:
  `375,768,1280`)
- `--driver <human|agent|suite>` — who's running it, for the session
  ledger (default: `human`)
- `--api-url <url>` — backend base URL (overrides `VERIFAIED_API_URL`)
- `--token <token>` — API token (overrides `VERIFAIED_API_TOKEN`)

## `verifaied test` — record a flow once, replay it forever

A coverage percentage says which lines ran. It can't say whether the
API-tokens screen still works. `verifaied test` records that flow in a
real browser, saves it as a normal `@playwright/test` spec in your repo,
and replays it with video whenever you ask.

Name the flow in verifAIed (**New test** on the branch's **Tests** page,
which opens a page of its own) — or just record it, and the CLI creates
the row for you:

```bash
# Record. A browser opens; click the flow through, and say what you expect
# from the recording studio in verifAIed. Click Finish recording when done
# (closing the browser by hand does the same thing). Set base_url in
# .verifaied/config.toml and this is just `--url /settings`.
verifaied test record "Settings/API tokens" --url http://localhost:3000/settings

# Start from another recorded flow — it replays first, visibly, and is
# never re-recorded. Repeatable and ordered. It is also somewhere to
# start, so --url is optional: recording begins where the chain ends.
verifaied test record "Settings/API tokens" --pre "Auth/Log in"

# With neither, recording starts at base_url — so a flow that begins on
# your app's home page needs nothing but a name.
verifaied test record "Home/Sign up"

# Change one step without redoing the flow. The first two steps replay,
# then the recorder hands you the browser at exactly that point — with
# the original's remaining steps beside the canvas to copy across.
verifaied test edit "Settings/API tokens" --from 2

# Replay it — headless, with video, trace and browser coverage.
verifaied test run "Settings/API tokens"

# Or replay everything and exit with the worst code.
verifaied test run --all

# Point any of them at another configured environment.
verifaied test run --all --env stage

# What's recorded, and how each last went on this branch.
verifaied test list

# Or leave a recorder running and drive the whole thing from the web.
verifaied test studio
```

A test is named `<feature path>/<name>`, parsed at its **last** slash.
The feature path is optional and may nest as deep as you like, so
`"Log in"` is a test at the root of the tree and
`"Settings/API tokens/Revoke"` is one two folders down — which is why the
name itself may never contain a slash.

The spec lands at `.verifaied/tests/<feature path>/<name>.spec.ts` — yours
to read, edit and commit. Recording is followed immediately by a **replay**,
because the recorder can't capture video and a spec that doesn't pass
isn't a recording: the replay is what produces the video you watch back
in verifAIed, the trace, and the browser coverage for the branch.

The recording window is a real window — resize it and the page resizes
with it, so you can record a flow at the width you care about. While you
click, the steps and assertions appear **live** on the branch's Tests
page in verifAIed, and stay there afterwards as the test's flowchart.

Close the recorder without doing anything and the test stays a **draft** —
nothing is saved, and nothing starts holding your branch to it.

**Editing** a saved flow is `verifaied test edit "<ref>" --from N`, and it
is a branch rather than a re-record: the first `N` steps are kept and
replayed one at a time — the flow ticks down to the branch point like a
debugger — and from there you rebuild with the whole studio. `--from 0`
starts again from nothing and `--from <step count>` adds to the end. The
kept steps are frozen for the session, so to change step 3 you branch at
2: the branch point *is* the edit cursor. Whatever came after it is listed
beside the canvas in verifAIed and can be copied across a step or a group
at a time, each one executed against the live page exactly like a line you
write. Finishing overwrites the spec; cancelling leaves it byte for byte
as it was.

A test this branch has never run, or whose last run failed, shows up as
the `unverified_recorded_test` gap in `verifaied check-done`.

### Saying what you expect

Assertions are authored in the **recording studio** — the page the record
command puts you on, which comes alive when the recording connects — and
not in the page being recorded: Playwright's own in-page toolbar offers
four canned matchers and sits on top of the UI you are pointing at, so
this recorder switches it off.

Click **Pick**, then click an element in the recording browser. That click
is swallowed rather than recorded, so choosing something never becomes a
step of the flow — and the picker is verifAIed's own, so it reaches
disabled controls and elements the page has made click-through, which the
stock one cannot. Escape cancels it. With an element in hand, the builder composes any
`expect` matcher — visible, hidden, enabled, disabled, checked, text,
value, CSS property, attribute, class, count, URL — and there is a paid
box beside it that turns "this button should be red" into the same kind of
line, written from the element's HTML and computed styles.

Either way, **the running page decides**. Every assertion is executed
against the browser you have open before it is kept: one that holds is
written into the spec where you added it and appears in the flowchart with
its own accent, and one that doesn't comes back with Playwright's own
received-vs-expected text instead of quietly becoming a line that asserts
nothing.

### Doing something the recorder can't record

Chromium dispatches no events on a `disabled` control, so clicking one in
the recording browser records *nothing* — which makes "click the disabled
Save button, then check nothing changed" a flow you cannot capture by
clicking. The studio's **Action** builder is the way in: pick the element,
choose a verb (click, double-click, fill, press, check, uncheck, select an
option, hover) and click **Add action**. The action is performed against
the live page and only kept if it ran, exactly like an assertion.

When the picked element is disabled, **Force** is switched on for you —
that's what makes the input land on a control the browser would otherwise
skip — and the composed line says so, `.click({ force: true })`, so the
saved spec reads as what it does. Authored actions appear in the flowchart
with the same accent as authored assertions, and are edited and deleted
the same way.

### Describing a whole flow

**Describe a flow** takes several steps at once, in your own words — "fill
my email and password, sign in, and check the dashboard heading shows".
verifAIed composes the lines against a snapshot of the page as it stands
right now, sent up by the recorder alongside the element you picked, and
hands them back as one ordered batch. Paid, and the one place a model
writes actions as well as assertions: you wrote the steps, and the running
page still gets the last word on every one of them.

The CLI holds the batch and releases it **one step at a time** — the next
line only goes to the browser once the one before it has actually held.
The moment a step is refused the sequence stops: everything behind it is
cancelled without ever running, and the studio shows the one step that
failed with Playwright's own words. So a wrong line costs you a red block
and a retry, not a sequence that carried on into whatever it found next.

### Changing what the recorder captured

While the recording is live, **every block in the flow can be edited or
deleted** — the assertions and actions the studio authored, and the
natural clicks the recorder wrote down as you made them. An edit opens the
same builder that composes a new line, prefilled; a rewrite is run against
the page exactly like the original and takes the original's place in the
flow, so a mis-clicked step is a correction rather than a re-recording.

The one block with no controls is the **navigation the spec starts from**:
a spec without its `goto` cannot replay at all, so removing one would
break the test rather than change it.

Deleting one of the recorder's own lines does not edit the recorder's
file — nothing can, it is rewritten in full on every action — so the CLI
stops emitting that line when it saves the spec instead. That is why the
delete carries the statement's **text** as well as its position: the
recorder mutates lines it has already written (a download or a dialog
rewrites an earlier one; a run of keystrokes collapses into a single
`fill`), and if the text no longer matches when the command is applied,
the delete is **refused and the step comes back** rather than taking a
neighbour with it.

Once the spec is saved it is a plain file in your repo, and your editor is
the better tool for it.

### Running another test inside this one

The rail's **Insert a test** section runs a whole recorded flow *at the
point you are in this one*: pick a test, click **Insert**, and its steps
run against the page in the recording browser. If they run, they join the
flow as one named block and are written into the saved spec as a single
`test.step(...)`:

```ts
await test.step("Run 'Auth/Log in'", async () => {
  await page.goto('http://localhost:3000/login');
  await page.getByLabel('Email').fill('a@b.c');
});
```

That is ordinary Playwright — the saved spec runs for anybody, with or
without verifAIed. Only that test's **own steps** are spliced in, never
its pre-step chain: a chain is set-up built for a fresh browser, and
replaying "log in" in the middle of a flow that is already logged in is at
best a detour. The same test can be inserted as many times as the flow
needs it, because an insert happens *at a position*.

**Keep linked** is the per-insert choice, off by default:

* **Off (snapshot)** — the steps are copied in and now belong to this
  spec. They never change on their own.
* **On (linked)** — the block carries a marker comment naming the test it
  came from and the hash of the body it copied, and verifAIed rewrites it
  when that test is re-recorded. Until then the Tests page says the copy
  is out of date, and the next `verifaied test run` (or the replay after a
  recording) refreshes it in place and tells you it did. Files are only
  rewritten when something actually changed.

**Finish recording** saves the spec and closes the browser, which is the
same thing closing the window by hand does.

### Pre-steps

`--pre "<Feature>/<Name>"` names another **recorded** test to run first.
It replays in the browser before recording starts, so signing in is
something you do once rather than at the head of every spec. Pass it more
than once for a chain; the order is the order they run, a pre-step's own
pre-steps run before it, and a flow shared by two of them runs once.

A chain is also somewhere to start: with one in front of it a test needs
no `--url` at all, and recording begins on the page the chain leaves the
browser on. A test recorded that way has no navigation of its own, so its
chain can't be cleared — verifAIed refuses, and `test run` says so if the
chain is scrubbed by deleting a test it named.

The chain is stored with the test, so later `test record` and `test run`
calls reuse it without the flag. Each spec file in your repo stays a plain
recording of one flow — the composition happens in a throwaway file at run
time, which is what makes editing `Auth/Log in` fix every test that starts
by logging in. You can also pick pre-steps on the **New test** page.

### Environments

A recorded flow runs against a real app *somewhere*, and somewhere is the
thing that changes: your machine, a container stack, staging. Name each
one, and everything that differs about running there lives in that one
table:

```toml
# .verifaied/config.toml — commit this alongside your specs.
[browser-tests]
default = "local"

[browser-tests.env.local]
base_url = "http://localhost:3000"
reset = "make db-reset"

[browser-tests.env.stage]
base_url = "https://stage.example.com"
# No reset. A deployed app is not yours to drop.
```

```bash
verifaied test run "Settings/API tokens"            # default → local
verifaied test run "Settings/API tokens" --env stage
verifaied test record "Settings/API tokens" --env stage --url /settings
verifaied test studio --env stage
```

`--env` works on `record`, `run` and `studio`. Without it you get
`default`, or the only environment you defined; two or more with nothing
choosing between them is an error rather than a guess, and a name that
isn't configured is refused with the ones that are.

The flat `reset` and `base_url` keys still work and read as one unnamed
environment, so a config written before environments existed carries on
unchanged.

**`base_url`** is what keeps a recording from belonging to one machine.
Start a flow at `--url /settings` and every location the recorder writes
down on that origin is saved as a **path**, so the spec in your repo
reads `await page.goto('/settings')` and replays against whatever
`base_url` says on the machine running it. It becomes Playwright's
`baseURL` for the run, and it is also where a recording starts when
nothing else says: no `--url` and no `--pre` records from `/`.

Absolute URLs are left exactly as they are, on both sides: a flow that
genuinely starts on somebody else's login page keeps working, and a spec
full of absolute URLs replays as it always did. `base_url` must be an
origin — scheme and host, no path — and a spec with a relative location
in a repo that names none is refused up front, rather than dying on
Playwright's "invalid URL" halfway through a run.

### Variables — the data that differs, and the data that must be fresh

A login is one flow with three passwords. A "create a token called
*scratch*" flow works once and then meets the token it made. Both are
about **data**, not about where the app is, so both live in the
environment too:

```toml
[browser-tests.env.local.vars]
# Committed, non-secret: the same for everyone who checks this out.
login_email = "test@example.com"
# Secret: this machine's environment supplies it.
login_password = { from_env = "E2E_PASSWORD" }
# Fresh every run: "tok-e2e-9f3ab120".
token_name = { value = "tok-e2e", unique = true }
```

Type a configured value into the form while recording and the spec in
your repo reads:

```ts
await page.getByLabel('Email').fill(process.env.VERIFAIED_VAR_LOGIN_EMAIL);
```

The live flowchart shows it as `${login_email}`, and every run supplies
whatever that environment says. The rewrite is **whole-literal and
exact**: a value that merely appears inside a longer string is somebody's
own text and is left alone, `press` is never touched (its argument is a
key, not data), and `goto`/`toHaveURL` belong to `base_url`.

**`from_env` is a secret.** It never reaches verifAIed: not in the spec,
not in the page snapshot the studio reasons about, not in the text of a
failed assertion — every one of those has the value replaced by
`${name}` on the way out. A missing one stops the run by name, *before*
the reset, because finding out afterwards costs you a database.

**`unique` is what makes a deployed environment work without a reset.**
The value gets a fresh suffix per run — one value for the whole
invocation, so every reference to it agrees — so the flow that creates
`tok-e2e-9f3ab120` can run again tomorrow against the same database. That
is the intended answer for staging and prod, and it is why they need no
reset at all.

Names are lowercase (`[a-z][a-z0-9_]*`), values are at least three
characters (a shorter one would match all over a spec), and anything
malformed is an error naming the variable rather than a silently ignored
line.

#### `verifaied var set` — write one without opening the file

```bash
# Committed with the file.
verifaied var set login_email "test@example.com"

# A secret: only the environment variable's NAME is written down.
verifaied var set login_password --from-env E2E_PASSWORD

# Fresh every run.
verifaied var set token_name "tok-e2e" --unique

# Which environment. A name that doesn't exist yet is created — this is
# how the first one gets written.
verifaied var set login_email "test@stage.com" --env stage
```

It writes into the environment a run would use — `--env`, else the
configured `default`, else the only one defined — and refuses rather than
guessing when there are several and nothing chooses between them. A repo
with no environments at all is told to pass `--env <name>`, because a
variable belongs to one.

Your comments, key ordering and formatting survive: the file is one your
repo commits and reviews, so it is edited rather than regenerated. And
the write is verified by reading it back — if the result wouldn't load,
the original bytes go back and the command fails.

This is what the studio's **Variables** section hands you while you are
recording: name a variable there and the page shows the exact line to run
on the machine with the recorder on it.

**A recording picks the change up on its own.** The config is re-read on
the recorder's own beat — a `stat` per second, and a parse only when the
file actually moved — so running `var set` with the browser still open
adds the name to the studio's list within a couple of seconds and hands
the value straight to the running driver, where the next fill resolves it
instead of typing `undefined`. A `unique` variable keeps the value it
started the recording with, and a config that is momentarily unparseable
(a half-saved buffer) is skipped in silence rather than taking a live
browser down with it.

One thing is not automatic, and cannot be: a `from_env` value is read out
of the environment of the process running the recorder, so exporting it
in *another* shell can never reach it. Export it and restart
`verifaied test record` / `verifaied test studio` — until then the
variable is listed as **unset** and simply isn't offered, rather than
resolving to nothing.

A line that names a variable with no value on this machine is refused
**before** it reaches the browser, with the command that fixes it:
`${api_token} has no value on this machine — run: verifaied var set
api_token "<value>" — it applies automatically, no restart`. Playwright's
own answer would be `value: expected string, got undefined`, which is
true and tells nobody what to do.

#### `verifaied var unset` — remove one

```bash
verifaied var unset login_email

# Which environment. Unlike `var set`, it has to exist already.
verifaied var unset login_email --env stage
```

The same environment precedence, the same comment-preserving edit, and
the same verify-by-reload: if the result wouldn't load, or the variable
still reads back, the original bytes go back and nothing changed. A name
the environment doesn't declare is refused with the ones it does. The
`vars` table is left in place even when it ends up empty — the header and
the comment above it are yours.

This is what the studio's Variables list hands over for its Trash button:
the config is a file on your machine, so the command is the delete.

### Resetting state before each run

A recorded flow runs against a real application with real data. "Create a
token called *scratch*" passes the first time and fails forever after,
because the second run meets the token the first one made. Tell verifAIed
how to put your app back, in the environment where that is safe to do:

```toml
[browser-tests.env.local]
base_url = "http://localhost:3000"
reset = "make db-reset"
```

Commit that file — it belongs to the repo, the same way the specs do, and
everyone who runs the tests should be running the same reset.

The command is a **shell line**, run from `--root` (your repo root by
default) with your terminal's stdio, so you watch it work. There is no
timeout: a database reset can legitimately take minutes, and Ctrl-C still
reaches it. It runs:

- before every recording, chain or no chain — what a flow is recorded
  against has to be what it is later replayed against;
- before the replay that follows a recording, because the recording just
  spent a few minutes changing everything it touched;
- before **each** spec in `verifaied test run`, so every flow meets the
  same starting state.

It is **fail-closed**. A reset that exits non-zero stops the run before
any session is opened — no video, no coverage, no pass recorded — and a
`[browser-tests]` section the CLI cannot read (malformed TOML, a `reset`
that isn't a non-empty string) is an error too, because a typo must never
read as "no reset wanted". No `reset` key means no reset; a *named*
environment without one says so in one dim line, so you are never left
wondering whether the config was read.

**A reset aimed somewhere that isn't your machine is refused.** An
environment with a `reset` and a non-loopback `base_url` is an error at
load unless it also says `reset_remote = true`. Deployed environments are
not meant to need one — use `unique` variables instead — and the opt-in
exists for the one real case, a dedicated throwaway e2e deployment, where
it has to be said out loud.

The config is read fresh at each of those points, so editing it takes
effect on the next recording without restarting `test studio`.

The command lives in your repo rather than in verifAIed because it is a
shell line executed on developer machines: that is exactly the trust model
of an npm script or a git hook, reviewed by the people who work on the
repo. Storing it server-side would turn a compromised account into code
running on every laptop that pulled.

### Where a run happened

Every run says which environment produced it: beside the run in the
ledger, on the Tests page's last-run line, and on the connected
recorder's pill (`Recorder connected · main · runs against stage`). It is
a **label**, not a verdict — `check_done` asks whether a recorded test
passed, not where — so a pass on stage clears the gap exactly as a pass
locally does.

Tests tracked from your own Playwright suite are outside this: their
`playwright.config.ts` owns where they point, and verifAIed only reads
the report they produced.

### `test studio` — record from the web, without the terminal

`verifaied test record` is one recording per terminal command. `verifaied
test studio` is the same recording with the terminal taken out of the
loop: start it once, leave it running, and from then on the whole flow —
name the test, give it a start URL or a chain to start from, click
**Record** — happens on the verifAIed Tests page. The browser still opens on your machine, because
that is where your app is.

```bash
verifaied test studio
```

While it is connected the Tests page shows a **Recorder connected** pill
with the branch it is on, and every test gets a Record button in place of
the command to paste. The Playwright version gate runs before any of that
is announced, so a machine that could never record never claims it can.

**Running from the web.** A connected studio does two jobs, not one. The
**Run** button on a recorded test's page (`repos/:repoId/tests/:testId/view`)
queues a *run* on the same queue a Record queues a recording on, and the
studio replays that test's saved spec **headed** — you clicked it, you are
sitting in front of this machine, so watching the browser do it is the
point. `verifaied test run` from a terminal stays headless. Either way the
replay streams its progress back as it goes, and the flowchart on the page
fills in block by block: a tick on what has passed, a pulse on what is
running, a cross where it stopped. The verdict that lands when the run
finishes is the same one `verifaied test run` produces — the streaming is
what somebody watches, not what is recorded.

A studio older than 0.15.0 cannot run, only record. It says so on every
beat, the service only hands it what it declared, and clicking Run against
one is refused with "the connected studio is too old — restart it after
upgrading verifaied" rather than queueing something that would wait
forever.

**Queueing.** Recordings and runs happen one at a time, oldest first — one
studio can drive one browser, and a spec being re-recorded must not be
replayed at the same time. Clicking Record on a second test while the first is
going queues it, and the web says what it is waiting behind. A queued
recording can be cancelled from the studio page until a recorder takes it;
after that there is a browser opening on your machine, and nothing in the
cloud can stop it. A request that fails is reported with the reason and
the studio carries on — one bad pre-step chain is not a reason for it to
die.

**Branches.** The branch is re-detected on every beat, so switching
branches under a running studio moves what the web shows and what the next
recording lands against. `--branch` pins it instead.

**Ctrl-C** does the safe thing at either moment. Mid-recording, the
interrupt is forwarded to the recorder exactly as it is for `test record`:
the browser closes, the spec is saved, the replay runs, the request
completes — and only then does the studio exit. At the idle prompt it
stops immediately and says goodbye, so the connected pill disappears at
once rather than timing out.

Exit codes: `0` stopped cleanly, `1` usage/config error (no token, bad
`--repo`, unsupported Playwright), `2` the API could not be reached. A
dropped heartbeat is warned about and retried; a recording the studio
could not report on is not something to keep quiet about, so that one
stops it.

### Requirements

`node`, `npx` and `@playwright/test` in your repo (or a subdirectory of
it — the CLI finds the first one that has it, and `--runner-dir`
overrides). This is Node Playwright, not the Python `verifaied[audit]`
extra.

**`test record` and `test studio` only run against Playwright 1.60.x.** Recording drives
Playwright's recorder through an internal API — the same entry point
`playwright codegen` uses — so it refuses any version it hasn't been
verified against rather than opening a browser that might record nothing.
`test run` uses only public API and works on any version.

### Exit codes

- `0` — recorded, or replayed cleanly
- `*` — the replay's own exit code when the spec failed (`test run --all`
  reports the worst across the run)
- `2` — the replay was clean but its final coverage upload failed
- `1` — usage/config error (no token, bad `--repo`, unknown test, no
  Playwright)

### `test` flags

Shared by `record`, `run` and `studio`:

- `--repo / -r <UUID|owner/name>` — repository (default: auto-detect from
  the GitHub `origin` remote)
- `--branch / -b <name>` — branch (default: `git branch --show-current`)
- `--commit <sha>` — commit SHA (default: `git rev-parse HEAD`)
- `--root <dir>` — repo root the spec paths resolve against (default: cwd)
- `--runner-dir <dir>` — where to run `npx playwright` from (default: the
  first directory under `--root` with `@playwright/test` installed)
- `--port <n>` — coverage collector port, 127.0.0.1 only (default: 4571)
- `--interval <seconds>` — seconds between coverage uploads during a
  replay (default: 10)
- `--env <name>` — which environment from `.verifaied/config.toml` to run
  against (default: the configured `default`, or the only one defined)
- `--api-url <url>` / `--token <token>`

`record` also takes `--url <path-or-url>` (optional: a path is resolved
against `base_url`, and with no `--url` recording starts where a `--pre`
chain leaves the browser, or at `base_url` itself when there is no chain
either),
`--description <text>`, `--pre "<Feature>/<Name>"` (repeatable;
replaces the stored chain — omit it to keep what the test already has),
and `--no-replay` to skip the replay. `run` takes `--all` in place of a
test name. `studio` takes `--no-replay` and takes no test name at all —
what it records is whatever the web asks for.

## Agent loop

We've found the most fruitful way to use verifAIed is to put a short
loop into your coding agent's instructions file (`CLAUDE.md`,
`AGENTS.md`, `.cursorrules`, `GEMINI.md`,
`.github/copilot-instructions.md`) so it self-corrects on every change.

The easiest way is to let your agent write that loop into the rules
file for you. Paste the prompt below into your agent — it will inspect
the repo, find the project's real test command, confirm the CLI is
installed and `VERIFAIED_API_TOKEN` is set, and write the loop section
into the matching rules file, tailored to your repo:

```markdown
We've found that adding a short test-coverage loop to your agent's instructions is the most fruitful way to use verifAIed. I want you to write that loop into this repo's agent-instructions file, tailored to how this repo actually runs its tests.

verifAIed is a coverage analysis tool. Its CLI (`verifaied upload`) pushes a local `coverage.json` + `junit.xml` to the server; its MCP server exposes a `check_done` tool that returns a `{done, status, gaps}` verdict for the branch (the loop's stop condition), and a `fix_branch` tool that returns a single prompt describing every untested function, partial branch, and failing test. The agent loop is: run tests → `verifaied upload` → call `check_done` → if `done` is true stop, otherwise fix the gaps (`get_baseline_prompt` per function, or `fix_branch` for one prompt covering everything) → repeat.

Do the following, in order:

# 1. Detect the project setup

- **Test runner & command**: look at `pyproject.toml` (`[tool.pytest.ini_options]`, `[project.scripts]`), `pytest.ini`, `tox.ini`, `setup.cfg`, `Makefile` targets (`test`, `check`), `justfile`, or a top-level `scripts/` directory. Use the test command the project already uses — don't invent a new one.
- **Coverage flags**: pytest must produce `coverage.json` with branch coverage AND per-test contexts, plus `junit.xml`. If the existing command already does that, reuse it. Otherwise build the command:
  ```
  pytest --cov --cov-branch --cov-context=test --junitxml=junit.xml
  coverage json --show-contexts -o coverage.json
  ```
  The `coverage json --show-contexts` step is non-negotiable — pytest-cov's `--cov-report=json` alone drops the per-test contexts, which makes verifAIed mark the upload "Needs attention".
- **Agent-instructions file**: pick the one that matches the agent you (the model) are. If unsure, fall back to the project's existing convention.
  - Claude Code: `CLAUDE.md`
  - Codex CLI / OpenAI: `AGENTS.md`
  - Cursor: `.cursorrules` or `.cursor/rules/*.mdc`
  - Gemini CLI: `GEMINI.md`
  - GitHub Copilot: `.github/copilot-instructions.md`
  - If none exists in the repo, create the one matching your own agent.

# 2. Verify the CLI is installed and configured

- Run `verifaied --help`. If it exits non-zero, install it: `uv tool install verifaied` (preferred) or `pipx install verifaied`.
- Check `echo $VERIFAIED_API_TOKEN`. If empty, stop and tell the user to mint one from the verifAIed app (Settings → Tokens) and export it: `export VERIFAIED_API_TOKEN=vr_live_...`. Do NOT invent a placeholder and do NOT proceed without it.

# 3. Write the loop into the agent-instructions file

Append (or create) a section titled `## Test coverage loop with verifAIed` containing:

- The exact test command(s) you built in step 1 — so future runs of the agent don't have to re-derive them.
- The exact upload command (`verifaied upload`, plus any flags this repo needs — e.g. `--coverage path/to/coverage.json` if your test command lands the JSON outside the repo root).
- An instruction to call the `check_done` tool on the verifAIed MCP server after each upload, and to stop only when it returns `done: true`.
- An instruction that while `check_done` reports gaps, fix them: call `get_baseline_prompt(function_id)` for a free per-function test prompt, or `fix_branch` with `scope: everything` for a single prompt covering every gap, then apply it.
- An instruction to repeat from step 1 until `check_done` reports `done: true`.
- One line: "If `VERIFAIED_API_TOKEN` is unset, stop and ask the user — do not skip this loop."

# 4. Confirm with the user

Show the user the diff of the file you edited (or created) and the test command you wrote in. Do NOT run the loop yourself yet — let the user review first.

# Hard rules

- Do NOT change the test command's behavior beyond adding coverage / junit flags.
- Do NOT add new dependencies if pytest-cov and coverage are already transitive deps.
- Do NOT write a fallback that skips the loop when the token is missing — the agent must stop and ask.
- Do NOT enumerate functions yourself or write tests; `check_done` lists the gaps and `fix_branch` / `get_baseline_prompt` return the prompts that close them.
- Do NOT decide "done" yourself; `check_done` is the stop condition.
- Do NOT replace an existing agent-instructions file — append a new section.
```

### Or paste the instructions in by hand

If you'd rather skip the meta-prompt, drop this section into your
agent's rules file as-is. Swap in your repo's actual test command if
it differs from the pytest default:

```markdown
## Test coverage loop with verifAIed

After implementing or modifying any code in this repo, run this loop until verifAIed reports done:

1. Run the test suite with branch coverage and per-test contexts:
   ```
   pytest --cov --cov-branch --cov-context=test --junitxml=junit.xml
   coverage json --show-contexts -o coverage.json
   ```
2. Upload the results to verifAIed:
   ```
   verifaied upload
   ```
3. Ask if you're done: call the `check_done` tool on the verifAIed MCP server (or run `verifaied check-done`). It returns `{done, status, gaps}` for this branch — failing tests plus untested/partially-covered functions among the ones you changed.
4. If `done` is true, stop — the branch is covered.
5. Otherwise fix the gaps it lists: call `get_baseline_prompt(function_id)` for a free test prompt per function, or `fix_branch` with `scope: everything` for a single LLM-written prompt covering every gap on the branch.
6. Go back to step 1.

Stop only when `check_done` reports `done: true`. If `VERIFAIED_API_TOKEN` is not set, stop and ask the user — do not skip this loop.
```
