Metadata-Version: 2.4
Name: qe-harness
Version: 0.13.0
Summary: Black-box, five-stage validation of conversation-to-action AI assistants: intent, orchestration, MCP tool calls, backend state and user response
Author: Shan Konduru
License-Expression: MIT
Keywords: testing,qa,llm,agents,mcp,evaluation,chatbot
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: mcp>=2.3
Requires-Dist: pydantic>=2.7
Requires-Dist: pyyaml>=6
Requires-Dist: python-dotenv>=1.0
Requires-Dist: websockets>=12
Provides-Extra: azure
Requires-Dist: openai>=1.50; extra == "azure"
Provides-Extra: anthropic
Requires-Dist: anthropic>=2.4; extra == "anthropic"
Provides-Extra: judge
Requires-Dist: openai>=1.50; extra == "judge"
Requires-Dist: anthropic>=2.4; extra == "judge"
Dynamic: license-file

# qe-harness

Black-box, five-stage validation of conversation-to-action AI assistants.

A chat-only test sees the final reply. `qe-harness` tests every turn at five stages, from the outside:

**intent → orchestration → MCP tool calls → backend state → user response**

It never imports the application under test, and it doesn't intercept anything. It signs in as test users and
chats over the app's API, reads the app's per-turn traces, gets an independent record of every tool call from a
recording proxy the sandbox routes its tool traffic through (stdio or HTTP), and compares read-only snapshots of the
business data before and after each turn (see "How the evidence is collected" below). So it catches what a polite reply hides: a write the
customer never approved, an amount no tool returned, a reply that says "done" when nothing was saved.

## Install

```bash
pip install qe-harness            # core
pip install "qe-harness[judge]"   # plus the optional LLM grader (Azure OpenAI or Anthropic)
```

Python 3.11 or newer.

## Quick start

```bash
qe init          # writes qe.yaml (sandbox connection) and qe_catalog.yaml (test cases) in this folder
# edit both files, then provide the two secrets the config names:
#   QE_SANDBOX_TOKEN  sandbox control token
#   QE_TEST_PASSWORD  test users' password
# (environment variables, or a .env file next to qe.yaml)
qe doctor        # connection checks: all green means testing can start
qe run --open    # run every suite, evaluate each turn, open the HTML report
```

`qe` finds its configuration from `--config PATH`, then `$QE_CONFIG`, then `qe.yaml` in the working directory.
Relative paths in it start from the folder holding the file (or from `base_dir:` if set).

## What the app's sandbox must provide

| Need | Endpoint or setting |
|---|---|
| Sign-in as test users | `POST /api/login` `{"username", "password"}`, session cookie |
| Conversations | `POST /api/conversations` → `{"id"}`; WebSocket `/ws?conversation=<id>` with messages `{"type": "user", "text"}` and `{"type": "confirm", "key", "approved"}`; events `ready`, `awaiting_confirmation`, `assistant_message`, `error`, `turn_completed` |
| Per-turn traces | `GET /api/conversations/<id>/traces`: intent (`primary_intent`, `intents`), plan steps, tool calls with arguments and outcomes |
| File uploads (optional) | `POST /api/conversations/<id>/documents?filename=…` |
| Control | `GET /sandbox/status` (versions, clock, toggles) and `POST /sandbox/reset`, with header `x-sandbox-token` |
| Independent tool record | Route tool traffic through `qe_harness.mcp_proxy`: stdio (start the tool server through it) or HTTP (point the app's MCP URL at it). See below |
| Business data | A read-only SQLite replica the harness can snapshot |

## How the evidence is collected (nothing is intercepted)

The harness doesn't sniff network traffic, decrypt TLS or hook into the app. Each source of evidence is something
the client switches on in their sandbox:

```
                     the client's sandbox (each source is switched on by the client)
  +------------+                    +-------------+   MCP    +-------------+   MCP   +-------------+
  |            | (1) chat as test   |             | -------> |  qe proxy   | ------> |             |
  |            |     users          |  app under  |          |  relays     |         | tool server |
  |            | -----------------> |  test       | <------- |  unchanged, | <------ |             |
  |            | <----------------- |             |          |  logs each  |         +------+------+
  | qe-harness |                    +------+------+          |  tool call  |                |
  |            | (2) per-turn trace API    |                 +------+------+                v
  |            | <-------------------------+                        |                +-------------+
  |            | (3) tool-call log, independent of the app's trace  |                |  business   |
  |            | <--------------------------------------------------+                |  data       |
  |            | (4) read-only snapshots of a replica, before and after every turn   |  (replica)  |
  |            | <-------------------------------------------------------------------+-------------+
  +------------+
```

1. **Conversation**: the harness signs in as test users and chats over the app's normal API, like a customer would.
2. **Intent, plan and tool calls**: the app reports them per turn through its trace API (the app describing itself).
3. **Independent tool-call record**: the app's tool traffic is routed *through* `qe_harness.mcp_proxy` by changing
   one setting in the sandbox. The proxy passes every MCP message through unchanged and writes each `tools/call`
   request and its result to a log. If the app's trace leaves out or misreports a tool call, this record disagrees,
   and the harness flags it ("trace and proxy disagree").
4. **Business data**: read-only snapshots of a replica of the system of record, before and after every turn.

### Enabling the proxy

| The app talks to its tool server over | The sandbox changes | Run |
|---|---|---|
| **stdio** (the tool server is a child process) | the command that starts the tool server | `python -m qe_harness.mcp_proxy --log-dir <dir> -- <tool server command>` |
| **HTTP** (MCP streamable HTTP) | the tool-server URL the app connects to, to the proxy's address | `python -m qe_harness.mcp_proxy --http --listen 0.0.0.0:9100 --upstream http://tools:9000/mcp --log-dir <dir>` |

- **stdio**: one proxy process per tool-server process; the customer comes from `COPILOT_CUSTOMER_ID` in its
  environment.
- **HTTP**: a reverse proxy. Requests keep their path, headers (`Mcp-Session-Id`, `Authorization`, …) and body;
  event-stream responses are streamed through as they arrive. The customer comes from a request header the app
  sends (`--customer-header`, default `X-Customer-Id`) or a fixed `--customer`. Each request/response exchange is
  logged with its own id, so concurrent conversations can't be mixed up.
- Both modes write the same log format to `<dir>`, which must match `tools.proxy_log_dir` in the harness config.
- The proxy logs only tool calls and their results. It changes nothing, and if it's removed the app works exactly
  as before.

### Other ways to get an independent record

If routing tool traffic through the proxy isn't possible, these give similar evidence, roughly from most to least
recommended:

- **API gateway or service-mesh access logs** (Kong, Apigee, AWS API Gateway, Istio/Envoy) for the back-end calls the
  tools make: very independent of the app.
- **Database change capture or audit logs** (e.g. Debezium, database audit triggers): an independent record of every
  write; scales better than whole-database snapshots.
- **OpenTelemetry traces** the app already emits for model and tool calls: useful, but like the trace API it's the
  app describing itself.
- **An audit log inside the tool server**: independent of the agent, not of the client's code.
- **Packet capture** (tcpdump, mitmproxy with a trusted CA): a last resort. Traffic is usually TLS, it needs
  network access and the security team's consent, and it sees nothing at all when MCP runs over stdio.

## Commands

| Command | What it does |
|---|---|
| `qe init [--dir D] [--force]` | Starter `qe.yaml` and `qe_catalog.yaml` |
| `qe init --ci github\|azure` | A sample nightly CI pipeline |
| `qe doctor [--quick] [--workers]` | Network, control API, clock, versions, proxy toggle, reset, read-back, sign-in, round trip, trace link, tool capture; `--workers` checks every sandbox |
| `qe status` | Sandbox versions, clock and toggles |
| `qe cases [--suite S]` | List catalog cases |
| `qe run [--suite S…] [--case ID] [--repeats N] [--variants all\|LIST] [--judge] [--parallel N] [--open]` | Reset the sandbox, run the suites, evaluate every turn, write evidence, the HTML report and the defect list; `--parallel N` spreads suites over N sandboxes; `--variants` also runs each case's paraphrases (see Phrasings) |
| `qe run … [--junit FILE] [--flaky fail\|skip\|pass] [--baseline RUN]` | For CI: JUnit XML, how flaky cases count, and fail only on regressions against an accepted run |
| `qe console [--port N] [--no-open]` | Interactive page to build a run (presets, personas, suites, test types, intents or single cases; phrasings, repeats, grader, parallel sandboxes), see the run plan and coverage preview, launch it, watch it live and open its report |
| `qe runs` | Recent runs |
| `qe show RUN CASE` | Evidence for one case: message, approvals, intent, plan, trace and proxy tool calls, data changes, reply, verdicts |
| `qe report RUN [--open]` | Rebuild the self-contained HTML report |
| `qe coverage [RUN] [--trend [--grid] [--since DATE] [--last N \| --all]] [--open]` | Intent coverage for a run (default: the latest finished), or across runs: per intent family the latest and previous pass rate, the change, best and worst, and a sparkline of the last 20 runs (fixed width however many runs there are); `--grid` shows every number, one column per run; `--since` keeps runs from a date on; `--open` opens the report, whose grid holds every run |
| `qe scorecard [RUN] [--all] [--open]` | The run measured against the Intent-Based Testing Catalogue: 71 test types by category, with metrics, failing cases and the six evidence-seam rates |
| `qe defects [RUN] [--open]` | Defect list for the app team (HTML and CSV) |
| `qe junit [RUN] --out FILE [--flaky …] [--baseline …]` | JUnit XML for a finished run |
| `qe mcp serve` | Run the harness as an MCP server for AI assistants |

## What is checked per turn

| Stage | Checks |
|---|---|
| Intent | The router's primary intent is one of the expected intents |
| Orchestration | Writes only after approval; no unexpected approval prompts (approval steps approve only the expected action with matching arguments; anything else is declined and fails here); no repeated identical calls |
| MCP tool call | Required calls succeeded with the expected arguments; forbidden calls absent; **trace and proxy agree** |
| Backend state | Only the expected tables changed; SQL checks; **declined → nothing changed; data changed → a write succeeded** |
| User response | Required facts present, forbidden phrases absent, optional LLM grader for consistency with what the tools did |

Each failure records its first failing stage, so developers know where to look. With `--repeats N`, every case gets
a pass rate with a 95% Wilson range and unstable cases are reported as flaky.

## Phrasings: the same request asked other ways

A case can list `variants:`: paraphrases that keep its intent and every detail (ids, amounts, dates, names) and its
reliance on earlier turns. The catalog's `defaults.variant_styles` names their positions (this catalog: casual,
formal, indirect, noisy).

```bash
qe run --variants all --judge --parallel 3   # the original prompts plus every variant, each a full pass
qe run --variants 0,4                        # only the original and the 4th (noisy) phrasing
```

Every phrasing is a full pass over the suites, in the same conversation context as the original, checked at all
five stages, so a paraphrase that is understood but then handled differently also fails. The variants of a case
count as its runs (a case that fails only in some phrasings is FLAKY). The console, the report and `summary.json`
(`intent_robustness`) add per phrasing style: intent accuracy and end-to-end pass rate with the change from the
original wording; paraphrase invariance (cases where every phrasing got an accepted intent); consistency (same intent
for every phrasing); the most confused intent pairs; and every misread phrasing, linked to its evidence. The defect
list quotes the phrasing that failed.

## Fault injection: what does the assistant say when a tool fails?

The recording proxy can answer chosen tool calls itself, so the call never reaches the tool server and nothing
changes in the backend. A catalog step lists them under `faults:`; the harness writes them to
`<proxy_log_dir>/faults.json` just before that step and removes the file right after (and at the start of every run).

```yaml
faults: [{tool: make_payment, mode: error}]                         # every make_payment call gets an MCP tool error (503)
faults: [{tool: list_claims, mode: error, times: 1}]                # only the first call fails: a retry would work
faults: [{tool: add_rider, mode: timeout, delay_s: 3}]              # waits, then a "timed out" tool error
faults: [{tool: list_drivers, mode: empty}]                         # a successful but empty result
```

Injected results copy the envelope of the tool server's real results (e.g. `resultType`, `_meta`), so the app's MCP
client accepts them whatever protocol version it speaks. The proxy log marks them (`"injected": "<mode>"`), the
case view shows "fault injected by the proxy", and trace and proxy still have to agree. Works over stdio and MCP
HTTP. The FAULTS suites probe false success (claiming a payment or change that failed), invented figures after a
failed lookup, recovery after a one-off error, and empty results.

## Assistants that call REST APIs directly (no MCP)

The same recording and fault injection work for an assistant whose tools call REST APIs. Put one REST proxy in front
of each API and point the app's base URL for that API at it (a configuration change, no code change):

```bash
python -m qe_harness.mcp_proxy --rest --name policy --listen 127.0.0.1:9101     --upstream https://policy-api.sandbox/v1 --openapi policy-openapi.json --log-dir qe_runs/proxy
# app setting: POLICY_API_URL=http://127.0.0.1:9101   (was https://policy-api.sandbox/v1)
```

- Every request is forwarded to the upstream base URL (same method, path, query, headers and body; that host only;
  `..` segments refused) and recorded in the proxy log in the same format as MCP calls, so the five stages, the
  seams, the scorecard and the report work unchanged. The log entry keeps the HTTP details under `rest`.
- **Operation names** come from the API's OpenAPI spec (`operationId`, or `METHOD /template` without one; literal
  paths win over templated ones; the spec's server base path is stripped). With no spec: `METHOD /path` as sent.
- **Arguments**: path parameters, query parameters and the JSON body's fields, merged (numbers stay numbers).
- **Faults** use the same `faults:` steps, with the operation name as `tool`: `error` returns HTTP 503, `timeout` waits
  `delay_s` then returns 504, `empty` returns 200 with `[]` (GET) or `{}`. Faulted requests never reach the API.
- **Trace vs proxy**: when the app's tool names differ from the API's operations, list each tool's operations in the
  config. The seam check then requires every recorded API call to be explained by a tool call in the trace, and every
  successful tool call to have made at least one of its operations (a tool mapped to `[]` makes no API calls):

```yaml
tools:
  proxy_log_dir: qe_runs/proxy
  operations:
    make_payment: [getPolicy, createPayment]
    list_policies: [listPolicies]
```

- **Not recorded**: `--ignore REGEX` (repeatable) relays matching paths without recording or faulting them, e.g.
  health checks or the app's discovery of the API, so they don't count as tool calls.
- **Try it** (in the source repository): `python demo/start_sandbox.py --workers 3 --mcp rest` runs the demo assistant against the insurance
  backend as a REST API (`insurance_copilot/rest_api.py`, OpenAPI at `/v1/openapi.json`) behind this proxy; run the
  harness with `qe --config qe_harness/sandbox.rest.yaml ...` (its `tools.operations` maps each tool to its
  operationId).

Sandbox only, like the MCP HTTP mode: the customer comes from a request header (`--customer-header`, default
`X-Customer-Id`) or `--customer`, and the proxy has no authentication or TLS of its own.

## Test-type scorecard (Intent-Based Testing Catalogue)

`qe scorecard [RUN] [--all] [--open]` and a report panel map a run onto the catalogue's 12 categories and 71 test
types. Each type is **measured** (metrics from this run), **not in this run** (supported, but e.g. no `--variants` or
no fault steps), **planned**, or **not applicable** (listed with a reason under `scorecard.not_applicable` in the
config). Metrics come from two places:

- automatic, from evidence every turn already has: intent accuracy, macro F1 and the top confusion pair (IU-01),
  out-of-scope detection and false accept/reject (IU-04), paraphrase invariance and degradation per phrasing
  (IU-06), argument correctness (ES-01), tool-call and state-graded task success (TT-01, TT-02), graceful failure,
  false success and retries under faults (TT-03), duplicate actions (TT-04), trajectory and plan adherence (AT-01),
  recovery and abandon rate (AT-03), pass^1 / pass^k / flakiness (RC-01), trace completeness (CP-06), p50/p95/p99
  latency (PO-01), timeout handling (PO-04), drift against the previous run of the same shape (CE-01), and the six
  **evidence seams** (C12: trace/proxy divergence, approval vs state, silent writes, phantom successes, reply vs
  action; SE-06 transport parity is verified by running the catalog over stdio and over HTTP);
- tagged cases: `test_types: [SS-05, ...]` on a catalog case counts its pass rate toward those types (and lists
  failing cases next to the type).

The report puts the six seam rates at the top (the category the catalogue notes most eval tools can't run).

The scorecard's **Cases (evidence)** column lists every case whose turns fed a type's metrics (automatic evidence and
tagged cases); cases that failed that type's own check are red, and long lists collapse behind a count. Two more report
sections, reached from the link bar at the top of the report:

- **Stages → test types**: one card per evaluation stage (Intent, Orchestration, MCP tool call, Backend state, User
  response) plus "Across stages" (whole-run checks such as pass^k, latency, drift and cost), listing the catalogue test
  types judged at that stage with their status and main metric, and the stage's own pass rate. A type judged at two
  stages appears under both (`STAGES_OF` in `qe_harness/scorecard.py`).
- **Cases → test types**: per case, the types it is designed to test (its `test_types:` tags) and the types it is
  evidence for (automatic metrics), red where the case failed that type's check; filterable, and repeated in each
  case's evidence view.

Drift (CE-01) compares only with an earlier run of the same shape: the same transport (MCP over stdio or HTTP, or
REST, from the sandbox status's `transport`), repeats, phrasings and cases.


## qe console: build, launch and watch runs from the browser

```bash
qe console                                   # opens http://127.0.0.1:8765/?t=<token>
qe --config qe_harness/sandbox.rest.yaml console --port 0 --no-open
```

A page opened from disk can't start processes, so `qe console` is a small local web server (standard library only)
that serves an interactive page and runs the harness behind it. It listens on 127.0.0.1 only; every API call needs the
random token in the printed URL, and starting a run needs it in the URL (a same-site cookie only opens report pages).

- **Build a run**: presets that follow the catalogue's execution modes (smoke, full regression, phrasing robustness,
  reliability ×3, fault drills, privacy and roles, representatives, intent pack, re-run failures); persona cards (from
  the catalog's test users; labels from `personas:` in the config); scope by suite, test type, intent or single case (a
  case brings the earlier turns of its conversation as setup); phrasing toggles with a live preview of one case in each
  wording; repeats, grader and parallel sandboxes. Harness configs next to the chosen one (e.g. MCP and REST) are offered
  in a selector, with sandbox readiness per worker (checked every 15 s). Selections can be saved by name (in this
  browser). The scope list works from the keyboard (arrows, Space or Enter).
- **Run plan**: a one-line summary, cases, setup turns, turns to judge, an estimate from the last run's turn times,
  the test types and stages the selection would measure, a warning when more sandboxes are chosen than there is work
  for, and the equivalent `qe run` command when the selection is whole suites. Run is disabled while no sandbox answers;
  the panel then shows how to start them (`console.start_hint` in the config).
- **Live run**: progress and time left, one lane per sandbox, and a **conversation replay**: each message as it is
  sent (with its wording), the tool calls from the app's trace, what the recording proxy saw, confirmation prompts,
  backend changes, the reply and the five stage verdicts. Fault-drill turns show the fault being armed and whether the
  assistant retried or gave up. A live grid has one five-stage strip per case and phrasing; clicking a strip (or a
  verdict, or a feed line) opens the **evidence drawer**: what the case expects, the conversation, every stage's
  failures and notes, tool calls, wire calls, faults, backend changes and the grader's verdict. When the run ends, a
  **summary card** shows the pass rate, defects by severity, the top three findings, test-catalogue coverage per
  category, and the change since the last run of the same shape. **Present** (or `#present=1`) switches to a
  full-screen view for a projector; Esc leaves it. Reloading the page re-attaches to the latest run.
  `#preset=robust` (or another preset's key) opens the page with that preset applied; `#tab=live` opens the live view.
- **History**: recent runs with their pass rates, a pass-rate line across them, and reports; a run still marked
  running that this console isn't running is shown as "interrupted?".

Branding and the start hint come from an optional `console:` section of the config:

```yaml
console:
  title: Harbor Mutual · Insurance Copilot
  subtitle: Five-stage black-box validation
  start_hint: python demo/start_sandbox.py --workers 3 --mcp http
```

## Parallel runs (one sandbox per worker)

Every run resets its sandbox, so one sandbox runs one test at a time (a lock enforces it). To go faster, give the
harness several identical sandboxes: list the extra ones under `workers:` in the config, each with its own URL,
read-only data path and proxy log folder.

```yaml
workers:
  - {base_url: http://127.0.0.1:8101, sor_path: data/sandbox-2.db, proxy_log_dir: qe_runs/proxy-2}
  - {base_url: http://127.0.0.1:8102, sor_path: data/sandbox-3.db, proxy_log_dir: qe_runs/proxy-3}
```

```bash
qe doctor --workers              # check every sandbox
qe run --judge --parallel 3      # the main sandbox plus two workers
```

Work is split into (repeat, suite) units; each runs on the next free worker from a fresh reset of that sandbox, and
the results merge into one run and one report. Suites must be independent for this: each is one test user's
conversation, and its data checks should be scoped to that user's records. The sandboxes must report the same
versions and clock, or the run refuses to start.

## Intent coverage

List the app's intent families under `intents:` in `qe.yaml`. Each report shows, per family, the cases that test it,
their pass rate and a status (covered, thin, failing, not tested), plus a families × runs grid of how coverage
changes over time. In the terminal, `qe coverage --trend` gives one line per family that stays the same width however
many runs there are:

```
21 run(s) · 11-03 21:39 → 11-05 09:12 · sparklines oldest → newest, · = not tested
                   latest   prev  change   best  worst  runs
modify_policy         33%    67%     -33    67%    33%  ·▆▆▆▆▆▆▆▆▆▆▆▃▃▃▆▆▆▃▃
general_question     100%   100%       0   100%     0%  ·▁▁▁▁▁▁▁████████████
```

`--since 2026-11-04` keeps only runs from that date on (for example after the catalog changed), `--last N` or `--all`
sets how many runs the sparklines cover (default 20), and `--grid` prints every number, one column per run, as many
recent runs as fit the window. In the report the grid keeps every run and scrolls sideways.

## Defect list for the app team

Every report comes with a defect list next to it (`defects.html` and `defects.csv`, linked from the report header);
`qe defects [RUN] --open` rebuilds and opens it. One entry per case that failed, cases that fail every time first,
then flaky ones. Each entry has:

- what the customer said, what the catalog expected and why, and what went wrong at each stage;
- the app's reply and what it did (intent, tool calls, approval prompts it shouldn't have shown, data changed);
- how often it failed, in this run and across all earlier runs;
- how to reproduce it (which test user, which earlier messages of the conversation);
- a link to the evidence in the report (`report.html#case=ID&repeat=N` opens that case and try).

Severity comes from what went wrong: **critical** when the app did or tried to do something it shouldn't (a
declined stray approval, a forbidden call, an unexpected data change, a trace that disagrees with the proxy);
**major** when it didn't do what was asked, or the grader found the reply contradicts the tools; **minor** when the
action was right but the reply missed something, or the request was misclassified. A catalog case can override it
with `severity: critical | major | minor`. The CSV imports into Jira, Azure Boards or a spreadsheet.

## In CI: JUnit, baselines and nightly pipelines

```bash
qe run --repeats 3 --judge --junit qe-results.xml --flaky skip --baseline qe-baseline.json
```

- `--junit FILE` writes JUnit XML: one test suite per catalog suite, one test case per case, the failure message is
  the first thing that went wrong, with a link to the evidence.
- `--flaky fail|skip|pass` decides how a case that failed only some tries counts (default `fail`).
- `--baseline RUN|latest|summary.json` fails the build only on **regressions**: cases that passed every try in the
  accepted baseline (or are new) and don't now. Cases that already failed there are reported as skipped ("known
  failure"), so they stay visible without turning every build red. Accept a baseline by committing a run's
  `summary.json`.
- The exit code is 0 when nothing counts as failed, 1 when something does, 2 when the run couldn't start.
- `qe junit [RUN] --out FILE` does the same for a run that has already finished.

`qe init --ci github` writes a nightly GitHub Actions workflow (`.github/workflows/qe-nightly.yml`) and
`qe init --ci azure` an Azure DevOps pipeline (`azure-pipelines-qe.yml`). Both install `qe-harness`, wait for the
sandbox with `qe doctor`, run the suites with a baseline, publish the JUnit results and keep the report and defect
list as build artifacts. Add the secrets listed at the top of the file; use a self-hosted runner or agent if the
sandbox is only reachable inside your network.

## In AI assistants (MCP)

```bash
claude mcp add qe-harness -- qe mcp serve
```

For Cursor or VS Code, add a stdio server with command `qe` and args `["mcp", "serve"]`.

| Tool | CLI equivalent | Notes |
|---|---|---|
| `sandbox_status` | `qe status` | read-only |
| `sandbox_doctor` | `qe doctor` | `round_trip`, `workers`; resets the sandbox, so marked as changing state |
| `list_cases`, `get_case` | `qe cases` | read-only |
| `run_tests` | `qe run` | suite, case, repeats, judge, parallel, variants; changes state; one pass of the catalog per call at most (phrasings count as passes); runs in the background |
| `get_run_status`, `list_runs` | `qe runs`, `qe report` | totals, per-case results, report and defect-list paths |
| `get_failure` | `qe show` | masked evidence incl. the grader's verdict, marked as untrusted data |
| `get_intent_coverage`, `get_intent_trend` | `qe coverage [--trend]` | trend has `last`, `since` and the latest/previous/change/best/worst summary |
| `get_defects` | `qe defects` | filterable by severity |
| `get_scorecard` | `qe scorecard` | status and metrics per catalogue test type, the six evidence-seam rates; `include_planned` |
| `check_regressions` | `qe junit --baseline` | regressions against an accepted run (default: the previous run), `flaky` mode, whether a build would pass |

Prompt: `triage_run`. `qe init` (writes project files) has no tool on purpose.

## LLM grader (optional)

`qe run --judge` asks a model whether each reply is consistent with what the tools actually did. It uses its own
settings, separate from the app's: `QE_JUDGE_PROVIDER` = `AZURE` (with `QE_JUDGE_AZURE_API_KEY`,
`QE_JUDGE_AZURE_ENDPOINT`, `QE_JUDGE_AZURE_DEPLOYMENT`) or `ANTHROPIC` (with `ANTHROPIC_API_KEY`, optional
`QE_JUDGE_MODEL`).

If the grader can't give a verdict (missing key, provider down), the turn is **not** counted as passed: it fails at
the user-response stage as "not evaluated", and the defect list shows those cases separately rather than as app
defects. `qe run --judge` also makes one small grading call before it starts, so a broken grader stops the run
straight away.

## Status

Alpha. Fault injection and hand-written paraphrase variants are in (0.5.0); REST APIs can be recorded and faulted as well as MCP (0.6.0, run live end to end in 0.7.0). Not yet included: load and soak testing,
simulated customers and automatic variant generation. The CI baseline is a regression gate; a gate on absolute thresholds (for example "no critical defects")
is not built in yet.

## License

MIT
