Metadata-Version: 2.5
Name: weir-run
Version: 0.1.0
Summary: Canary runner for weir canary judge: plants fake canaries, runs a program's flows against a loopback mock, judges the captured telemetry
Project-URL: Homepage, https://github.com/IdoGol24/weir
Project-URL: Repository, https://github.com/IdoGol24/weir
Project-URL: Issues, https://github.com/IdoGol24/weir/issues
Project-URL: Changelog, https://github.com/IdoGol24/weir/blob/main/CHANGELOG.md
Project-URL: Security, https://github.com/IdoGol24/weir/blob/main/SECURITY.md
License-Expression: Apache-2.0
Classifier: Development Status :: 3 - Alpha
Classifier: Framework :: Pytest
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Security
Classifier: Topic :: Software Development :: Testing
Classifier: Typing :: Typed
Requires-Python: >=3.12
Requires-Dist: click>=8.1
Requires-Dist: opentelemetry-proto>=1.44
Requires-Dist: weir-scan>=0.5.0
Description-Content-Type: text/markdown

# weir-run

The canary runner checks whether a program's telemetry carries the
credentials the program holds. It plants a unique fake credential (a
canary) wherever a real one would live, runs the program's own example
flows against a local mock with telemetry exported to a file, and asks
`weir canary judge` one question: did any planted value reach the capture?
Because the runner knows every value it planted, the answer is an exact
match in seven encoded forms, not a pattern guess.

Its contract, and the judge's, is weir's
[`docs/canary.md`](https://github.com/IdoGol24/weir/blob/main/docs/canary.md).
This page is for the maintainer who adds the check to a project's CI.

## The pieces

```text
weir-run run --config canary.toml
  1. plant   one fresh fake value per placement -> ledger (mode 0600, private temp dir)
  2. mock    127.0.0.1 only: OpenAI-compatible model API, OAuth endpoints,
             OTLP/HTTP receiver, your project's own APIs; refuses every proxied
             request to another host
  3. flows   each under an empty environment plus an allowlist, URLs pointed at the mock
  4. capture OTLP/JSON lines + other telemetry bodies + each flow's stdout and stderr
  5. judge   weir canary judge (weir-scan: opens no socket, starts no process)
  6. delete  the ledger and the capture
exit 0 clean and complete | 1 a canary reached the capture | 2 usage, input or runner error | 3 incomplete
```

- **The judge is weir-scan's**, called only through its command line. The
  runner never imports weir, and weir never imports the runner (both are
  import-linter contracts in the repository's `pyproject.toml`). The runner
  requires weir-scan 0.5.0 or later, the first release that ships
  `weir canary judge`; without a judge it exits 2 before running anything.
- **Local and offline.** The mock binds `127.0.0.1` on a port the OS picks;
  there is no option to bind anything else. Every flow gets the mock as its
  HTTP and HTTPS proxy, and the mock refuses every proxied request: a
  `CONNECT` or an absolute-URL request for any other host gets a 403, and
  nothing is forwarded, opened or resolved. The runner does not sandbox the
  network: a client that ignores proxy settings can connect directly (see
  [Limits of this release](#limits-of-this-release)).
- **Test-time only.** The runner is a dependency of your CI job, never of
  your program. The mock and the receiver exist only for one run.

## Fake data only

- Every value in a run is generated by the runner: the canaries it plants
  and the tokens its OAuth endpoints issue. Never put a real credential in
  a config, a flow's `env` or anywhere a flow reads it.
- Never point a flow at a real provider. Every API a flow calls must be the
  mock: the model API through `OPENAI_BASE_URL`, your project's own APIs
  through `[[mock.endpoint]]`, OAuth through the mock's endpoints.
- The ledger and the capture are deleted when the run ends. Do not copy
  them anywhere, and never commit one: a provider-shaped canary in a public
  repository is reported to the provider as a leaked key. Only the
  value-free report may be kept, for example as a CI artifact.
- In a CI job, check out with `persist-credentials: false` so the job token
  is not in the checkout a flow can read.

## Try the demo

The demo is one synthetic agent flow, standard library only: it gets an
OAuth token, asks the model (which calls its `web_search` tool), calls the
search API, sends the result back, and exports its spans over OTLP/HTTP.

```sh
weir-run run --config demo/canary.toml        # exits 0
weir-run run --config demo/canary-leak.toml   # exits 1
```

`canary-leak.toml` differs only in `DEMO_LEAK = "1"`, which makes the flow
record its search key and its HTTP Basic client credentials as span
attributes, put its request headers (with the access token) in a JSON
string attribute, and print the search key in a debug line. The run
reports each as a `LEAK` line by placement, file, line and form, and the
relayed debug line reads `debug: search key <canary:env.SEARCH_API_KEY>`.

## The config

```toml
[mock]
oauth = true                      # serve the OAuth endpoints (default false)
model_reply = "Done."             # the model's final answer
tool_calls = "each-offered-tool-once"   # or "none"

[mock.credential_fields]          # extra credential fields of built-in endpoints
model = ["header:api-key"]        # endpoints: model, oauth, otlp (otlp: header and query only)

[[mock.endpoint]]                 # one of your project's own APIs
id = "search"                     # served at {mock}/ep/search/...
credential_fields = ["header:x-api-key", "query:api_key", "body:api_key"]
status = 200
reply = '{"results": []}'
content_type = "application/json"
capture = false                   # true: keep request bodies as capture files

[[placement]]
id = "env.SEARCH_API_KEY"         # the variable is the id after "env."
kind = "env"                      # env | code
shape = "generic"                 # see Shapes
required = true                   # incomplete unless exercised or leaked
provider_host = "api.search.example"   # a refused attempt to reach it is evidence

[[placement]]
id = "code.client.api_key"
kind = "code"
variable = "MY_VARIABLE"          # the flow reads this; omit it for CANARY_CODE_CLIENT_API_KEY

[[flow]]
id = "quickstart"
command = ["{python}", "examples/quickstart.py"]   # {python}: the runner's interpreter
cwd = "."                         # relative to this file
timeout_seconds = 300
env = { SEARCH_BASE_URL = "{mock}/ep/search" }     # non-secret settings; {mock} is the mock
skip = "needs a GPU"              # optional: recorded as skipped, which is incomplete
```

Unknown tables and keys are errors (exit 2), so a typo cannot silently
drop a placement. Ids follow the contract's id rule (1 to 128 characters
from `[A-Za-z0-9._:#/-]`) and are unique.

### Placements

This release plants two kinds and mints a third (docs/canary.md,
"Placements and kinds"):

| kind | where the canary lives |
| --- | --- |
| `env` | the flow's environment, under the variable your software reads |
| `code` | a variable the runner names; the flow reads it and passes it explicitly to a constructor or call |
| `mock-oauth` | every authorization code, device code, access, refresh and ID token the mock's OAuth endpoints issue, as `runtime.oauth.<token>#<n>`; nothing to configure |

### Shapes

A canary is a provider's prefix followed by random `[A-Za-z0-9]`, so
software that checks a key's prefix or length accepts it. Each provider
shape matches the key pattern weir's catalog judges that provider's keys
by. No value ever contains `FAKE`, `TEST`, `EXAMPLE`, `DEMO`,
`PLACEHOLDER` or `REDACTED`, in any letter case.

| shape | prefix and body |
| --- | --- |
| `generic` | 40 characters, no prefix |
| `openai` | `sk-proj-` + 48 |
| `anthropic` | `sk-ant-api03-` + 93 |
| `github` | `ghp_` + 36 |
| `google` | `AIza` + 35 |
| `aws-access-key-id` | `AKIA` + 16 of `[A-Z0-9]` |

Values come from the CSPRNG, fresh each run. `--seed N` swaps in a
deterministic stream for reproducing a run locally; never use it in CI.

## What a flow receives

A flow starts from an empty environment, not the job's, and gets exactly:

| variables | value |
| --- | --- |
| `PATH` | the runner's `PATH` |
| `HOME`, `TMPDIR`, `TEMP`, `TMP` | a fresh private directory per flow (`USERPROFILE` too on Windows) |
| `LANG`, `PYTHONUTF8` | `C.UTF-8`, `1` |
| `SYSTEMROOT`, `PATHEXT` | Windows only, from the runner |
| `CANARY_MOCK_URL` | the mock's base URL, `http://127.0.0.1:<port>` |
| `OPENAI_BASE_URL`, `OPENAI_API_BASE` | `<mock>/v1` |
| `OTEL_EXPORTER_OTLP_ENDPOINT`, `OTEL_EXPORTER_OTLP_{TRACES,LOGS,METRICS}_ENDPOINT` | the mock's receiver |
| `OTEL_EXPORTER_OTLP_PROTOCOL` and the per-signal ones | `http/protobuf` |
| `HTTP_PROXY`, `HTTPS_PROXY`, `ALL_PROXY` (both cases) | the mock, which refuses |
| `NO_PROXY`, `no_proxy` | `127.0.0.1,localhost` |
| each placement's variable | its canary |
| the flow's `env` | as written, `{mock}` replaced |

A real provider key in your CI secrets never reaches a flow. A flow's
leftover child processes are stopped when the flow ends.

## The mock

| path | what it is |
| --- | --- |
| `POST /v1/chat/completions` | Chat Completions, plain and streaming. With tools offered, turn one calls every tool once with a minimal schema-valid argument, so each tool's credentials are exercised; once tool results are in the conversation it answers `model_reply` |
| `POST /v1/responses` | Responses, plain and streaming, same script |
| `POST /v1/embeddings`, `GET /v1/models` | fixed answers |
| `POST /v1/traces`, `/v1/logs`, `/v1/metrics` | the OTLP/HTTP receiver: protobuf or JSON, gzip or deflate |
| `GET /oauth/authorize` | redirects with a fresh code |
| `POST /oauth/token` | authorization code, refresh token, password, device code, client credentials, token exchange and JWT bearer grants; a refresh token where the grant usually has one, an ID token when the scope holds `openid` |
| `POST /oauth/device_authorization` | a fresh device code; the user code is not a credential |
| `GET /.well-known/openid-configuration` | discovery metadata pointing at the mock |
| `/ep/<id>/...` | a configured endpoint: its fixed reply |

Every supported grant succeeds, whatever it presents. The OAuth paths exist only with `oauth = true`.
Anything else is answered 404 and listed after the run.

**Credential fields.** Before any of these handles a request, the mock
inspects its credential fields: the whole `Authorization` and
`Proxy-Authorization` headers on every mocked endpoint, each endpoint's
declared fields,
and on the OAuth endpoints the form fields `client_secret`,
`client_assertion`, `assertion`, `code`, `device_code`, `refresh_token`,
`subject_token`, `actor_token` and `password`. A field holding a canary in
any encoded form (a Basic credential included) is `request` evidence that
the placement was exercised. Any other non-empty value is recorded as
`unexpected.<endpoint>.<field>#<n>`, never with its value, and makes the
run incomplete: a flow that sends a credential no canary accounts for has
left the check's premise.

**The capture.** The receiver writes each export request decompressed, as
one OTLP/JSON line (trace and span ids in lowercase hex), under
`capture/otlp/`. Request headers never enter the capture. A body it cannot
decode is kept as its own capture file and a marker line makes the run
incomplete. Request bodies of an endpoint with `capture = true` and each
flow's stdout and stderr are swept capture files.

## Output

- **stderr:** each flow's status and its own stdout and stderr, relayed
  after the run with every encoded form of every canary replaced by
  `<canary:PLACEMENT-ID>`; refused egress and unmocked requests by count.
- **stdout:** the judge's text report: one `LEAK` line per hit, the
  not-exercised placements, the incomplete reasons, the coverage line and
  the result. It names placements and locations, never a value or a
  fragment of one.
- `--report FILE` also writes the judge's `canary-report/1` JSON.
- The exit code is the judge's, or 2 for the runner's own usage and
  configuration errors and any other failure of the run (an unwritable
  `--report`, a judge that times out). A runner fault never exits 1.

A clean result is evidence about the flows that ran and the placements
planted, not a statement that the program is safe. Read the coverage line:
a placement no flow exercised is printed as `not exercised`.

## The pytest plugin

Installing the runner registers a `canary_check` fixture (subprocess mode):

```python
def test_quickstart_exports_no_credentials(canary_check):
    canary_check.mock(oauth=True)
    canary_check.env("SEARCH_API_KEY", required=True)
    variable = canary_check.code("code.client.api_key")
    search = canary_check.endpoint("search", credential_fields=["header:x-api-key"])
    canary_check.run(["{python}", "examples/quickstart.py"], env={"SEARCH_BASE_URL": search})
    # teardown judges; the test fails on a leak or an incomplete run
```

Declare the mock, the placements and the endpoints first; the first `run`
plants the canaries and starts the mock. At teardown the plugin judges,
prints the redacted flow output and the report, deletes the ledger and the
capture, and fails the test unless the result is clean. To assert on the
result yourself, call `result = canary_check.judge()` (`result.code`,
`result.report`). The judge is the `weir` beside the interpreter or on
`PATH`; override the `canary_judge_command` fixture to use another.

## Limits of this release

- Placements: `env`, `code` and `mock-oauth` only. The other kinds in the
  contract (`tool-field`, `mcp`, `settings-file` and the rest) are reserved.
- Model APIs: OpenAI-compatible only.
- The runner does not sandbox the network. A client that ignores proxy
  settings and has a hard-coded host is not stopped by the runner: on a
  machine with network access it connects to that host directly. Run such
  flows in a container or network namespace with no route out.
- No GitHub Action and no in-process pytest mode yet.
