Metadata-Version: 2.4
Name: oneport-conformance
Version: 0.3.0
Summary: Verify AI-written code two ways: intent (judge the diff) and behaviour (drive the running app). The conformance gate for AI code.
Author: OnePort
License-Expression: MIT
Project-URL: Repository, https://github.com/oneport/oneport-conformance
Project-URL: Issues, https://github.com/oneport/oneport-conformance/issues
Keywords: ai,code-review,verification,conformance,llm,security
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: click>=8.1.0
Requires-Dist: oneport-account>=0.2.0
Requires-Dist: httpx>=0.24
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0; extra == "dev"
Dynamic: license-file

# oneport-conformance

**Does this change match what the team actually asked for?**

The rest of the OnePort suite answers *"is it safe?"* — no secrets, no CVEs, no
breaking changes. `oneport-conformance` answers the other half: **"is it what we
meant?"** The product team writes down how the software *should* behave; this
tool checks whether a change (often written by an AI) actually built it that way,
and flags where it drifted.

It's the **conformance gate** — the piece that turns OnePort from "a bundle of
scanners" into "the layer that verifies AI did what you intended."

## Two halves of one verifier

Conformance checks *both* meanings of "did the AI do it right":

| Half | Question | How | Command |
|------|----------|-----|---------|
| **Intent** | Is it built the way we said? | LLM judges the **git diff** against your written rules | `check` |
| **Behaviour** | Does the running app actually do it? | **Drives your real running app** and reads the real HTTP response — status, body, timing | `verify` |

The intent half catches architectural drift a running app won't show you ("the
AI called Stripe directly — that violates R2", even though it *works*). The
behaviour half catches the opposite — the class of bug the screen hides: a
silent 500, an endpoint returning empty/mock data, a broken multi-step auth
flow. Run `all` to get both in one verdict.

> The behaviour half verifies at the **HTTP boundary** — see **Limits** for
> exactly what it does and doesn't see. When it can't decide, it says
> **UNKNOWN**, never a quiet pass.

### Behaviour: drive the running app

You describe what should be true; a flow is authored once (one model call), then
**replayed forever with no model in the loop** — a recorded flow is a fixed,
tiny, deterministic read, so today's fix can't silently break last week's flow.

```bash
# author a flow from plain English and save it
oneport-conformance record "log in as admin@x.com / pw, then /api/me returns that user" \
    --out .oneport/flows/login.json --base-url http://localhost:3000

# replay it — deterministic, spends no tokens
oneport-conformance verify --flow .oneport/flows/login.json --base-url http://localhost:3000

# or author + run in one shot
oneport-conformance verify --say "checkout returns 200 and an order id" \
    --base-url http://localhost:3000 --record .oneport/flows/checkout.json
```

A flow is plain JSON — real HTTP steps with assertions about the real response
(`status`, `json_has`, `json_eq`, `body_missing`, `max_ms`), and a `save`/`{{var}}`
mechanism so a login step's token flows into later protected steps. A failed
check returns the **evidence** (the request that fired, the status it returned)
and the **`file:line`** where the route is defined — the pointer that turns
"something broke" into a fix.

```
FAIL — 'checkout' did not do what was claimed
  drove http://localhost:3000 · 1 step(s) · 0 pass · 2 fail · 0 unknown

  [FAIL] checkout succeeds — status is 200   app/routes.py:41
      expected: status 200
      observed: status 500
      evidence: GET /api/checkout -> 500 (8ms)
```

## How it works

1. The team writes an **intent doc** in plain English — `.oneport/intent.md`:

   ```markdown
   # Payments
   - All money movement must go through the ledger service.
   - Never call the Stripe SDK directly from a route handler.

   # Auth
   - Access tokens expire in 24h; refresh tokens in 30d.
   ```

   Each bullet becomes a numbered rule (`R1`, `R2`, …).

2. You run the check against a change:

   ```bash
   oneport-conformance check               # last commit vs the intent
   oneport-conformance check --staged       # staged changes
   oneport-conformance check --base main     # the whole branch
   ```

3. It sends the **enumerated intent + the git diff** through the OnePort managed
   model and returns grounded deviations — each tied to a specific rule id, with
   the file/line, an explanation, and a **confidence score**.

```
DEVIATES — 1 deviation(s) at/above 'medium'
  checked 3 intent rule(s) · model gemini-flash-latest · 812 tokens

  [HIGH] R2 (90% conf) app/routes/pay.py:41
      rule:     Never call the Stripe SDK directly from a route handler.
      observed: route handler calls stripe.PaymentIntent.create(...)
      why:      bypasses the ledger service R1 requires
```

## Honesty (this is the point)

Conformance is **AI judgment**, so the tool is built to never fake certainty:

- It only flags a deviation it can **ground in the diff** and tie to a real rule.
  A finding that cites a non-existent rule is dropped.
- Every deviation and the run carry a **confidence score**; low-confidence
  findings are labelled as prompts for a human, not asserted.
- An unparseable model reply is reported as **inconclusive**, never a pass.
- A missing intent doc or an empty diff is a visible **skip/error**, never green.

> A verification layer that hallucinates breaches is worse than none.

## Waivers

Accept a known, reviewed deviation so it stops blocking — visibly, never
silently. `.oneport/conformance-waivers.json`:

```json
[
  {"rule": "R2", "file": "legacy/pay.py", "reason": "grandfathered", "expires": "2026-12-31"}
]
```

Waived deviations still appear in output, marked `[WAIVED]`. Expired waivers are
ignored, so debt resurfaces on its own.

## Output & CI

- `-f human` (default), `-f json`, `-f sarif` (for GitHub code scanning).
- `--fail-on low|medium|high` — the minimum severity that blocks. Default `medium`.
- **Exit codes:** `0` conforms / nothing to check · `1` blocking deviation(s) ·
  `2` usage or service error.

```bash
oneport-conformance check --base main -f sarif --fail-on high > conformance.sarif
```

## Limits (the behaviour half)

A verification tool that oversells its reach is worse than none, so here is
exactly what the runtime half sees.

| | |
|---|---|
| **Strong** | silent failed requests (a 500 the UI hides), an endpoint returning wrong/empty/mock data, a broken multi-step flow (auth token not honoured), wrong status codes, slow responses |
| **Sees** | the real HTTP **status, body/JSON, and timing** of every step it drives, and values carried across steps |
| **Can't see yet** | in-browser **state stores** (zustand/Redux), the **console**, **React render** streams, the **DOM** — these need an injected JS SDK (roadmap), so app-internal state and client-only bugs are out of scope for now |
| **Not the tool** | cross-browser rendering, visual regressions, sites you don't own — that's Playwright |

When the runtime half can't decide — the app is unreachable, or a body isn't the
JSON an assertion needs — the verdict is **UNKNOWN**, treated as "did not pass",
never a quiet green. The intent half is an **LLM judgment** on a diff: grounded
(a finding citing a non-existent rule is dropped) and confidence-scored, but an
informed second opinion, not a proof.

## Install

```bash
pip install oneport-conformance
onp account login <op_live_...>   # 0-BYOK: runs on OnePort's managed models,
                                  # metered to your account — no API key needed.
                                  # Token: https://www.oneport.co.in/console
```

Replaying a saved flow (`--flow`) uses no model and needs no account.

Part of [OnePort](https://github.com/oneport) — the verification & security
layer for AI-written code.
