Metadata-Version: 2.4
Name: agent-certify
Version: 0.7.0
Summary: Structural verification of multiagent systems
Author-email: Cognitive Engineering <david.martin@cognitive-engineering.dev>
License-Expression: Apache-2.0
Project-URL: Homepage, https://cognitive-engineering.dev
Project-URL: Repository, https://github.com/zeekmartin/agent-certify
Project-URL: Documentation, https://github.com/zeekmartin/agent-certify#readme
Project-URL: Changelog, https://github.com/zeekmartin/agent-certify/blob/main/CHANGELOG.md
Project-URL: Issues, https://github.com/zeekmartin/agent-certify/issues
Keywords: multiagent,agent,structural-analysis,fragility,graph-theory,ai-safety
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Security
Classifier: Topic :: System :: Monitoring
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.24.0
Requires-Dist: cryptography>=41.0.0
Provides-Extra: mcp
Provides-Extra: otel
Provides-Extra: cursor
Provides-Extra: all
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pytest-cov; extra == "dev"
Requires-Dist: ruff; extra == "dev"
Dynamic: license-file

# agent-certify

[![CI](https://github.com/zeekmartin/agent-certify/actions/workflows/ci.yml/badge.svg)](https://github.com/zeekmartin/agent-certify/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/agent-certify)](https://pypi.org/project/agent-certify/)
[![Python](https://img.shields.io/pypi/pyversions/agent-certify)](https://pypi.org/project/agent-certify/)
[![Downloads](https://img.shields.io/pypi/dm/agent-certify)](https://pypi.org/project/agent-certify/)
[![License](https://img.shields.io/badge/license-Apache%202.0-blue)](LICENSE)
[![Coverage](https://img.shields.io/badge/coverage-%E2%89%A590%25-brightgreen)](https://github.com/zeekmartin/agent-certify/actions/workflows/ci.yml)

Structural verification of multiagent systems.

Scan any multiagent system's communication topology.
Find structural single points of failure before they're exploited.

## Install

```bash
pip install agent-certify
```

The only dependencies are `numpy` and `cryptography`. The MCP server, the
OpenTelemetry export and the Cursor, CrewAI and LangGraph parsers need nothing
else — no framework is imported to read its logs (the `[mcp]`, `[otel]`,
`[cursor]` and `[all]` extras exist but are empty).

## Let your agent scan itself (MCP server)

agent-certify runs as an MCP server, so an agent can check its own structure
while it works:

```bash
claude mcp add agent-certify -- agent-certify mcp
```

or in an MCP client's settings:

```json
{
  "mcpServers": {
    "agent-certify": {"command": "agent-certify", "args": ["mcp"]}
  }
}
```

| Tool | What it returns |
|---|---|
| `topology_summary` | a plain-language health check, written for the agent to read about itself |
| `scan_topology` | FI, nodes, links, bridges, articulation points, every node's role (`perspective` picks the vocabulary) |
| `blast_radius` | what a compromised or failed `node` reaches, and whether the system fragments |
| `check_anomalies` | anomalies against `.agent-certify/baseline.json` (or `baseline_path`) |
| `check_policy` | violations of `.agent-certify/policy.json` (or `policy_path`) |

Every tool takes an optional `session_path`. Without it the server finds the
**current** session: `CLAUDE_CODE_SESSION_ID` (Claude Code sets it for the
processes it starts), then the newest session of the working directory's
project. It never falls back to another project's sessions. A real
`topology_summary` from this repository's own development session:

> Your current session has 12 nodes (1 orchestrator, 10 tools, the human) and
> 11 links. Fragility Index is 1.0 — every connection is a single point of
> failure with no backup path. The orchestrator (claude) is the sole keystone:
> if it fails, the system fragments into 11 isolated pieces. Most-used tool:
> Bash (97 calls, 8 failed). Overall 2.9% of 278 tool calls failed. No
> anomalies detected against the baseline of 25 sessions. No policy
> violations. This is the usual shape of a single-agent session — one
> orchestrator calling each tool directly; redundancy only appears when
> sub-agents share tools.

The server speaks MCP over stdio (newline-delimited JSON-RPC 2.0, protocol
versions 2024-11-05 to 2025-06-18) without the `mcp` package. A tool that
cannot run (no baseline yet, unknown node, missing session) returns a normal
result with `isError: true` and a message saying what to do; protocol errors
are JSON-RPC errors; nothing a request does stops the server.

Tool arguments come from an AI agent, which may be following injected
instructions, so the server treats them as untrusted. Path arguments
(`session_path`, `baseline_path`, `policy_path`) expand `~` but never
environment variables, are resolved (symlinks, `..`), and must lie inside
`~/.claude/projects` or the server's working directory; anything else gets
"Path not allowed". Responses and errors never contain a resolved path, file
contents or exception text: unexpected failures are logged with their
traceback on stderr and reported with a correlation id. Each tool call is
limited to 30 seconds (`initialize`, `ping` and `tools/list` answer directly).

## Quick start

```python
from agent_certify import ClaudeCodeParser, AgentFIScanner, BlastRadiusAnalyzer

# Parse a Claude Code session
parser = ClaudeCodeParser()
graph = parser.parse("~/.claude/projects/-home-user-project/session.jsonl")

# Scan structural fragility
scanner = AgentFIScanner(graph)
result = scanner.scan()
print(f"Fragility Index: {result.fi:.4f}")
print(f"Bridges: {len(result.bridges)}")
print(f"Critical nodes: {result.critical_nodes}")

# Blast radius if 'bash' tool is compromised
blast = BlastRadiusAnalyzer(graph)
report = blast.analyze("bash")
print(f"Direct impact: {len(report.direct_impact)} nodes")
print(f"System fragmented: {report.system_fragmented}")
```

## CLI

```bash
agent-certify scan ~/.claude/projects/                       # every session, plus a merged graph
agent-certify scan ~/.claude/projects/<project>/<session>.jsonl
agent-certify scan <path> --format json|markdown|html [-o FILE]
agent-certify scan <path> --perspective engineering|risk|compliance|monitoring|classic
agent-certify blast ~/.claude/projects/ --node bash

# Baseline and anomaly detection
agent-certify learn ~/.claude/projects/C--dev-myproject/     # writes .agent-certify/baseline.json
agent-certify learn s1.jsonl s2.jsonl s3.jsonl --output ./my-baseline.json
agent-certify scan session.jsonl                             # compares against ./.agent-certify/baseline.json if present
agent-certify scan session.jsonl --baseline path/to/baseline.json
agent-certify scan session.jsonl --no-baseline

# Tell the baseline what you have decided (survives a re-learn)
agent-certify baseline accept --tool Bash                    # stop flagging it
agent-certify baseline reject --tool deploy_production       # keep flagging it, forever
agent-certify baseline accept --link "agent:Build -> deploy_staging"   # that agent only
agent-certify baseline accept --domain api.openai.com        # a destination you allow
agent-certify baseline reject --domain "*.cn"                # a destination you never allow
agent-certify baseline show                                  # what is on record

# Policy: explicit rules, drafted from the baseline, edited by a human
agent-certify suggest-policy                                 # writes .agent-certify/policy.json
agent-certify scan session.jsonl                             # also checks ./.agent-certify/policy.json if present
agent-certify scan session.jsonl --policy path/to/policy.json   # or --no-policy

# Live monitoring
agent-certify watch ~/.claude/projects/C--dev-myproject/     # newest session in the project
agent-certify watch path/to/session.jsonl --interval 10 --perspective risk

# MCP server logs as a second data source
agent-certify scan session.jsonl --include-mcp

# OpenTelemetry spans (OTLP/JSON)
agent-certify scan session.jsonl --otel                      # appends to .agent-certify/traces.jsonl
agent-certify scan session.jsonl --otel-stdout | <collector> # spans on stdout, report on stderr
agent-certify watch ... --otel                               # one span per scan cycle

# Other formats are detected from the content
agent-certify scan path/to/repo/.agent-trace/                # Cursor Agent Trace
agent-certify scan examples/crewai_run.json                  # CrewAI execution log
agent-certify scan examples/langgraph_trace.jsonl            # LangGraph trace
agent-certify learn logs/crew-runs/                          # a folder of crew runs

# Structural baseline and drift detection
agent-certify keygen ~/keys/signing.key                      # mode 600; keep it outside the repository
agent-certify certify <path> --key ~/keys/signing.key -o system.cert.json   # valid 90 days (--validity-days)
agent-certify verify system.cert.json <path> --public-key signing.key.pub   # exit 1 on drift
#   --public-key is required: it is the issuer's key, obtained out of band
#   --expect-system-id ID   --ignore-expiry   --allow-unsigned (skips the signature check; warns)

# In CI: scan exits 2 when a finding reaches --fail-on (critical by default)
agent-certify scan session.jsonl --fail-on warning           # or critical (default), never
```

Node names are matched case-insensitively: `--node bash` finds `tool:Bash`.

## What it measures

The **Fragility Index (FI)** measures the proportion of 
communication links in your multiagent system that have 
no structural backup path.

- **FI = 0**: Fully redundant — every path has an alternative
- **FI = 1**: Fully fragile — every link is a single point of failure
- **Bridges**: edges whose removal disconnects the system

FI is computed on the agent communication graph using triangle 
participation analysis. The same structural measure is used by 
[sal-torch](https://pypi.org/project/sal-torch/) on transformer 
attention graphs (published at PPAM 2026, Springer LNCS).

## Node classification and perspectives

Each node is removed in turn and FI recomputed. That gives every node one
structural **level**:

- **high**: removing this agent/tool raises system FI by ≥ 10% (relative), or
  the node is an articulation point (removing it disconnects the system)
- **medium**: raises FI by 2–10%
- **low**: raises FI by less than 2%, or lowers it

A **perspective** names the levels for the people reading the report. The
measurement is the same under every perspective; only the words (and the
colours in the HTML view) change.

| Perspective | high | medium | low | Use it when |
|---|---|---|---|---|
| `engineering` (CLI default) | KEYSTONE | LOAD-BEARING | PERIPHERAL | designing or refactoring the agent architecture |
| `risk` | HIGH RISK | MODERATE RISK | LOW RISK | threat modelling, security review |
| `compliance` | UNRESTRICTED | BOUNDED | LOGGED | audits, access reviews, regulatory reports |
| `monitoring` | WATCH | TRACK | IGNORE | deciding where to point runtime alerts |
| `classic` | CRITICAL | BUFFER | IMMUNE | v0.1 labels; the Python API default |

```python
result = AgentFIScanner(graph, perspective="risk").scan()
result.node_classifications        # {"claude": "HIGH RISK", ...}
result.node_levels                 # {"claude": "high", ...}
result.labels("compliance")        # same result, other vocabulary
```

## Baselines

`agent-certify learn` reads N past sessions and records what normal looks like:

- mean ± std of FI, node count, link count, bridge count, sub-agent count and
  error rate (errors / tool calls), plus FI's range and the most sub-agents seen
- every tool seen, and the fraction of sessions it appears in
- every invocation link seen (`source -> target`)
- call volume per tool and per link (mean ± std, max, and how many sessions it
  was seen in), which is what `call_volume_spike` compares against
- every network destination seen (`known_domains`)
- the tools, links and destinations you have accepted or rejected by hand, which
  are decisions rather than statistics and so survive a re-learn (see below)

Sub-agent ids are unique per spawn, so links are recorded by role: a sub-agent
counts as `agent:<type>` (`agent:Explore`, `agent:general-purpose`). Standard
deviations are population deviations, so a baseline learned from one session
has std 0.

The baseline is saved to `.agent-certify/baseline.json` in the current
directory (`--output` to change). `scan` picks it up from there automatically.
The baseline is optional: without one, `scan` behaves exactly as without
anomaly detection.

```python
from agent_certify import AgentFIScanner, AnomalyDetector, BaselineLearner

learner = BaselineLearner()
baseline = learner.learn(["~/.claude/projects/C--dev-myproject/"])
learner.save(baseline)

report = AnomalyDetector(baseline).detect(graph, AgentFIScanner(graph).scan())
report.has_critical
print(report)
```

### Deciding what counts as normal

Learning is a statistic. Whether you *accept* what it learned is a decision, and
a decision must not be erased by the next `learn`. Three commands record one:

```bash
# A tool you know about. Stops being an anomaly, without re-learning anything.
agent-certify baseline accept --tool Bash

# A tool that must never become normal. It keeps firing even after a re-learn
# that saw it in every single session, and `suggest-policy` will never draft it
# as a REQUIRED tool.
agent-certify baseline reject --tool deploy_production

# Scoped acceptance: this agent may call this tool. Another agent calling the
# same tool still fires.
agent-certify baseline accept --link "agent:Build -> deploy_staging"

# A network destination, as a host or a wildcard over its subdomains.
agent-certify baseline accept --domain api.openai.com
agent-certify baseline reject --domain "*.cn"

agent-certify baseline show            # or --format json
```

All three are stored in `baseline.json` and **carried forward by
`agent-certify learn`**, which recomputes every statistic and leaves the
decisions standing. `learn --forget-overrides` drops them on purpose.

Scoped acceptance is what makes `--link` worth having. Approving
`agent:Build -> deploy_staging` does not approve `deploy_staging`: if another
caller appears, the tool is flagged again and the message names the callers that
no accepted link covers.

```
[CRITICAL]  New tool never seen: deploy_staging
            Baseline: not in 14 known tools
            2 call(s) by agent:Test:0 — not covered by an accepted link: agent:Test
```

A tool can be accepted and then rejected, or the reverse: the newer decision
wins. "Permanent" means it survives re-learning, not that you cannot change your
mind. Rejection beats a scoped link, so a rejected tool fires whoever calls it.

A link endpoint written with a `kind:` prefix (`tool:Bash`, `agent:Explore`) is
taken as it stands. A bare word is resolved against the baseline — the
orchestrator's id is a bare word (`claude`) and so is a tool name — and falls
back to a tool, which is the usual case, since the reason to scope a link is
normally a tool that is new.

Rejection is deliberately asymmetric: there is no `reject --link`. A link you
never want is a policy FORBIDDEN rule, which `scan --policy` enforces with a
message you write yourself.

`--domain` records the same kind of decision about a network destination, with
the same precedence; see [Network destinations](#network-destinations).

## Anomaly detection

| Check | Flags | Severity |
|---|---|---|
| `new_tool` | a tool never seen in the baseline, or one you rejected | critical |
| `new_link` | an invocation path never seen or accepted (links to a flagged tool are not repeated) | warning |
| `fi_deviation` | FI more than 2σ from the mean | warning; critical past 3σ when FI *rose* |
| `error_rate` | error rate more than 2σ *above* the mean | warning; critical past 3σ |
| `sub_agents` | more sub-agents than ever seen, or 2σ above the mean | warning; critical when above the max *and* past 3σ |
| `node_count` | node count more than 2σ from the mean | info; warning past 3σ |
| `call_volume_spike` | a tool, or one link to it, called more than 3σ above the mean | warning; critical past 5σ |
| `new_domain` | a network destination never seen, and not accepted | critical |
| `rejected_domain` | a destination matching a rejected pattern | critical |

- **critical**: something structurally new or a large move toward fragility. Look now.
- **warning**: a structural change worth a look; often explained by the task.
- **info**: unusual but benign on its own.

A falling FI means added redundancy (usually sub-agents sharing tools). That is
a structural change, not a risk in itself, so it never goes past warning. When
the baseline never varied on a metric (std 0), z is undefined: any difference
is reported with no z-score at the lower severity.

### Call volume

`call_volume_spike` answers a question the other checks do not: not *whether* a
tool was used but *how much*. A tool whose calls jump from 50 to 500 is known,
reaches no new path and may fail no more often than usual — until v0.7.0 it
passed every check, and the injection benchmark measured that as a blind spot
for a release before this closed it.

```
[CRITICAL] Tool Bash called 487 times (baseline: 50 ± 1, 391.3σ)
           called by claude
```

Volume is learned per tool **and per link**, and a link spike replaces the
tool-level one when it adds something:

```
[CRITICAL] Link agent:Explore -> tool:Bash called 401 times (baseline: 1 ± 0, …)
           agent:Explore alone accounts for this; other callers of tool:Bash
           are compared separately
```

- a tool with one caller is reported as the tool: the link would say the same thing;
- a tool with several callers is reported as the link that spiked, so the message
  names who is responsible and the other callers are judged against their own
  links. An agent calling a busy tool its usual number of times is not flagged
  because somebody else hammered it.
- a tool or link the baseline never saw has no volume to deviate from; that it is
  new is `new_tool` / `new_link`'s business.
- only increases are reported. A quiet session is not a risk.

Per-tool statistics count only the sessions where the tool actually appeared —
a tool used rarely but heavily would otherwise have a mean near zero and fire
every time it ran at all. With no deviation to compare against (one session, or
a count that never varied) twice the mean stands in for 3σ, reported at warning
with no z-score.

For a directory, `scan` lists the anomaly count of each session in the
session table.

## How well does detection work?

The checks above are measured, not asserted. `agent_certify.benchmark` injects
faults whose answer is known into clean sessions and scores what comes back:

```bash
python scripts/benchmark.py                      # synthetic sessions, no logs needed
python scripts/benchmark.py --sessions ~/.claude/projects/C--dev-myproject
python scripts/benchmark.py --format json        # for CI
```

```
Anomaly injection benchmark — 6 clean session(s) against a baseline of 6
============================================================================================
anomaly type    expected                   trials  detected     rate  severity
--------------------------------------------------------------------------------------------
new_tool        new_tool/crit                   6         6   100.0%  6 critical
error_spike     error_rate/warn                 6         6   100.0%  6 critical
new_agent       new_link/warn                   6         6   100.0%  6 warning
volume_spike    call_volume_spike/warn          6         6   100.0%  6 critical
forbidden_tool  new_tool/crit                   6         6   100.0%  6 critical
foreign_domain  new_domain/crit                 6         6   100.0%  6 critical
shell_domain    (blind spot)                    6         0     0.0%  not covered by any check
--------------------------------------------------------------------------------------------
detection rate 100.0% (36/36)   false negatives 0.0%   false positives 0.0% (0/6 clean, 0 anomalies)
blind spots: shell_domain — injected and expected to go unreported, no check covers the fault
```

| Injection | Fault | Expected |
|---|---|---|
| `inject_new_tool` | a tool the baseline never saw | `new_tool`, critical |
| `inject_error_spike` | existing calls failed until the error rate reaches `rate` | `error_rate`, warning or worse |
| `inject_new_agent` | an agent of an unseen type calling a known tool | `new_link`, warning or worse |
| `inject_volume_spike` | `factor`× the calls to one tool | `call_volume_spike`, warning or worse |
| `inject_forbidden_tool` | a tool the operator rejected | `new_tool`, critical |
| `inject_foreign_domain` | a destination the baseline never saw | `new_domain`, critical |
| `inject_shell_domain` | a host named inside a shell command argument | **nothing** — a measured blind spot |

Three things in that table are worth more than the 100%:

- **The volume spike was a measured miss for a release.** v0.6.1 reported it as
  a blind spot — no check read call volume — and v0.7.0's `call_volume_spike`
  closed it. The case stays in the suite, which is the point of measuring a gap
  instead of omitting it. The two faults added in v0.7.0 change nothing
  structural at all: the volume spike adds no node, link or failure, and the
  foreign destination leaves the topology byte-for-byte identical.
- **One blind spot is open, and it is injected rather than asserted.**
  Extraction reads hosts from URLs and from fields that name a destination, so
  `{"command": "curl -sS collector.example.net/x"}` reaches no baseline and
  fires no anomaly. `inject_shell_domain` builds that argument object and hands
  it to the production extractor; whatever comes back lands in the graph, which
  today is nothing. It is counted apart from the detection rate so it neither
  inflates it nor drops out of the report, and it turns into a detection by
  itself — no edit to the benchmark — the day extraction reads shell commands.
  "No blind spots left" is a sentence this project will not print unless the
  measurement says it.
- **The baseline is learned from sessions that are not the ones tested.**
  `learn_holdout()` holds out every other session; a baseline learned from
  exactly the sessions it is then run against reports nothing on them by
  construction, and its false positive rate would mean nothing. A baseline that
  does not fit does produce false positives, and the benchmark says so.

The headline numbers come from synthetic sessions
(`agent_certify.benchmark.synthetic`), which are deliberately regular. Run
`--sessions` against your own logs for numbers that describe your system.

## Network destinations

An agent graph says which tools ran. It does not say **where they went** — and
if you have to answer where your data is processed, that is the question. An MCP
server or HTTP tool repointed at another host changes the answer without
changing the topology at all: same nodes, same links, same counts.

So destinations are read from the session's own logs and compared against what
the baseline learned:

```bash
agent-certify baseline accept --domain api.openai.com     # or "*.openai.com" for subdomains
agent-certify baseline reject --domain "*.cn"
agent-certify baseline reject --domain "*.alibaba.com"
agent-certify baseline show                               # includes a domains section
```

```
[CRITICAL] Network destination api.qwen.cn in tool call — not in baseline, matches rejected pattern *.cn
           Baseline: rejected pattern *.cn
           reached by tool:WebFetch — rejected, so it stays an anomaly across re-learns
```

**What counts as a destination.** Logs have no agreed field for this, so
extraction is broad but not indiscriminate:

- **any full URL** in a tool call's arguments — `https://api.openai.com/v1/chat`
  gives `api.openai.com` (`http`, `https`, `ws`, `wss`, `ftp`, `ftps`);
- **a bare host**, but only as the value of a field that names one: `url`,
  `uri`, `endpoint`, `host`, `hostname`, `domain`, `base_url`, `base_uri`,
  `api_url`, `webhook_url`, `server`, `origin`, `proxy`, their plurals, and a
  dozen more — `NETWORK_KEYS` in `agent_certify/domains.py` is the full list.
  `{"host": "db.internal"}` counts; an IP literal counts too.

  One exception, to keep configuration noise out of the report: a value whose
  last label is a common file extension (`notes.txt`, `data.csv`, `app.log` —
  `NOT_HOSTS` in the same module) is read as a filename, not a host. Every
  label on that list is absent from the DNS root, so no reachable name is
  dropped by it; `collector.zip`, `x.sh`, `y.so`, `z.md`, `a.py` and `b.rs` are
  destinations and are reported as such.

A domain-looking word *anywhere* is not a destination. That distinction is what
keeps this usable: a session that reads a file mentioning `example.com`, or
greps for `alibaba.com`, has contacted nobody, and reporting it as egress would
bury the real finding. For the same reason a tool **result** is read only when
that call's own arguments carried a URL — so a `WebFetch` records its redirects,
while a `Read` of a document full of links records nothing.

**Patterns, not addresses.** Nothing is resolved, in either direction. A
pattern is matched against the name the agent used, which is the thing you made
a decision about:

| Pattern | Matches | Does not match |
|---|---|---|
| `api.openai.com` | that host only | `other.openai.com` |
| `*.cn` | `api.unknown.cn`, `deep.sub.unknown.cn` | `cn`, `notcn` |
| `*.alibaba.com` | `x.alibaba.com`, `x.y.alibaba.com` | `alibaba.com`, `evil-alibaba.com` |

A wildcard covers subdomains only: the bare domain is a destination of its own
and deserves its own decision. `.cn` is accepted as a spelling of `*.cn`, and a
whole URL is accepted and reduced to its host. There is no `*` pattern — a host
that is not in the baseline already fires as `new_domain`, so deny-by-default is
the behaviour you get without asking for it.

Decisions follow the same rules as tools: stored in `baseline.json`, carried
forward by `learn`, last write wins, and **reject beats accept** — a rejected
pattern fires for a host the baseline learned in every session.

Destinations are read by the Claude Code, CrewAI, LangGraph and MCP-log parsers,
appear in `scan --format json` as `seen_domains`, and are listed by
`agent-certify learn` and `baseline show`.

## Example workflow

```bash
# 1. Learn what normal looks like from the last few weeks of sessions
agent-certify learn ~/.claude/projects/C--dev-myproject/

# 2. Scan a new session; anomalies print above the node table
agent-certify scan ~/.claude/projects/C--dev-myproject/<new-session>.jsonl

# 3. Triage with the vocabulary your team uses
agent-certify scan <new-session>.jsonl --perspective monitoring --format html -o review.html
```

```
Fragility Index  0.3103   (9/29 links with tri(e)=0; baseline: 0.98 ± 0.09)

⚠  ANOMALIES DETECTED (8):

[CRITICAL] Sub-agents: 3 (baseline: 0.04 ± 0.20, max seen 1)
           Types: agent:Explore, agent:general-purpose

[WARNING]  New communication path: claude -> agent:Explore
           Baseline: not among 35 known links

[WARNING]  FI deviation: 0.31 (baseline: 0.98 ± 0.09, z = -7.8)
           Sub-agents detected — shared tools add triangles; structural change expected
...
```

## Blast radius

`BlastRadiusAnalyzer.analyze(node)` reports:

- **direct_impact**: 1-hop neighbours. A compromised tool can steer its caller
  through poisoned results, so spread is measured in both directions.
- **total_reachable**: everything within 3 hops (`hops=` to change).
- **accessible_tools**: tools the node can invoke itself or through the agents
  it spawns (follows call direction).
- **remaining_fi / system_fragmented**: structure after isolating the node.
  "Fragmented" means isolation splits the system into more pieces than it had.

## Certificates and drift

An `AgentCertificate` is a signed (Ed25519) structural baseline: the nodes, the
directed invocation links, FI and node roles. `graph_hash` covers the topology
only, not call counts. `verify(graph)` names each change: new agent types, new
tool links, links that disappeared, FI movement, role changes. A topology change
can be detected before the behaviour it enables shows up.

Sub-agents are compared by **type**, not by their per-spawn id: a session that
spawns a fresh `Explore` agent for the same tools matches a certificate issued
for another `Explore` agent. A different type is reported as such:

```
New agent type: agent:Research (not in certified baseline)
Missing agent type: agent:Explore (was in certified baseline)
```

Certificates issued before v0.3 have no `identity` field; they keep comparing
agents by id, and their signatures remain valid.

### Trust model

A signature proves something only when it is checked against a key you
already trust. `verify` therefore **requires** the issuer's public key
(`--public-key`, or `public_key=` in Python), obtained through a trusted
channel. The `signer_public_key` written inside a certificate is informational
only: whoever can edit the certificate can replace it. A certificate fails
verification when it is unsigned, signed by another key, edited after
signing, outside its validity window, or issued for another system
(`--expect-system-id`). `--allow-unsigned` skips the signature check
explicitly and prints a warning; use it only for local experiments.

Since v0.5 every certificate carries `not_before` / `not_after` (90 days by
default, `--validity-days`), so an older, more permissive certificate cannot be
replayed indefinitely. Certificates from earlier versions have no validity
window; they still verify, with a warning, and should be re-issued.

The signed bytes are `b"agent-certify/cert/v1\x00"` followed by the
certificate as JSON with sorted keys, compact separators, ASCII escaping and
`fi_score` written as a 10-place decimal string; see `certificate.py` for the
exact rules. Older certificates keep their original encoding, and the format
is part of what is signed, so a certificate cannot be downgraded.

Keys: `agent-certify keygen` creates the private key with mode 600 and refuses
to overwrite an existing file or follow a symlink. `certify` warns when the key
is readable by other users. On Windows the file inherits the folder's ACL, so
keep keys in a folder only you can read. Never commit a key; `*.key` is in
`.gitignore`.

## Policy

Anomalies say *this is unusual*. A policy says *this is not allowed* (or *this
must be there*), whatever the statistics. A session can have both, and `scan`
reports them separately.

```json
{
  "rules": [
    {"kind": "forbidden_tool", "target": "deploy_production", "severity": "critical", "message": "No agent may deploy to production"},
    {"kind": "forbidden_link", "target": "agent:* → *database*", "severity": "critical", "message": "Sub-agents may not reach the database"},
    {"kind": "max_sub_agents", "target": "5", "severity": "warning", "message": "More than 5 sub-agents is unusual"},
    {"kind": "required_tool", "target": "Bash", "severity": "warning", "message": "Bash is expected in every session"}
  ],
  "generated_from": "baseline (25 sessions)",
  "created_at": "2026-10-27T20:00:00Z"
}
```

| kind | target | violated when |
|---|---|---|
| `forbidden_tool` | tool pattern | a matching tool is used |
| `forbidden_link` | `"source → target"` (or `->`) | a matching invocation link exists |
| `max_sub_agents` | integer | more sub-agents than that |
| `required_tool` | tool pattern | no matching tool is used |

Patterns are shell-style globs (`*`, `?`), case-insensitive, matched against a
node's id (`tool:Bash`), type (`agent:Explore`) and name (`Bash`). `agent:*`
matches every sub-agent; a bare name matches exactly (`deploy` does not match
`deploy_production`; `deploy*` does).

`agent-certify suggest-policy` drafts a conservative policy from the baseline:
tools seen in every session become REQUIRED (warning), and the sub-agent count
is capped at the baseline maximum + 1 (warning) and + 3 (critical). FORBIDDEN
rules are an organisational decision, so the draft never contains any; add
them by hand. `suggest-policy` refuses to overwrite an existing policy file
(which may hold your edits) without `--force`.

## Live monitoring

`agent-certify watch` follows a session while Claude Code writes it. Given a
project directory it picks the most recently modified session. Every
`--interval` seconds (default 30) it checks whether the transcript or any of
its sub-agent transcripts grew; if so it re-parses, rescans and prints:

```
Watching C:\Users\me\.claude\projects\H--agent-certify\c603eabc-….jsonl (every 5s, Ctrl+C to stop)
Baseline: loaded (25 sessions)
Policy: loaded (4 rules)
────────────────────────────────────────────────────────────
[11:35:02] Nodes: 7   Links: 6   FI: 1.0000   Calls: 192
  [CRITICAL] POLICY VIOLATION: FORBIDDEN tool used: Edit
[11:35:07] Nodes: 7   Links: 6   FI: 1.0000   Calls: 192
[11:35:27] Nodes: 7   Links: 6   FI: 1.0000   Calls: 196
[11:35:37] Nodes: 7   Links: 6   FI: 1.0000   Calls: 197
[11:35:52] Nodes: 7   Links: 6   FI: 1.0000   Calls: 200
```

(A real run on this repository's own development session, with a demo policy
that forbids `Edit`.)

Each anomaly (critical and warning) and each policy violation is printed once,
when it first appears, and again only if its severity changes. A session file
that does not exist yet is waited for; a half-written last line is skipped
until it is complete. It uses plain size polling, no file-system notification
library. `--scans N` stops after N scans and `--duration S` after S seconds — the only way to end a watch on a session that has gone quiet; in Python, `SessionWatcher(...)`
takes `on_anomaly` / `on_policy_violation` callbacks and has `stop()`.

## MCP server logs

Claude Code also logs every MCP server connection, one JSONL file per server
and process start:

| Platform | Location |
|---|---|
| Windows | `%LOCALAPPDATA%\claude-cli-nodejs\Cache\<project>\mcp-logs-<server>\*.jsonl` |
| Linux | `~/.cache/claude-cli-nodejs/<project>/mcp-logs-<server>/*.jsonl` |
| macOS | `~/Library/Caches/claude-cli-nodejs/<project>/mcp-logs-<server>/*.jsonl` |

Records are `{"timestamp", "sessionId", "cwd", "debug" | "error"}`; tool calls
appear as `Calling MCP tool: <tool>` and failures as `Tool '<tool>' failed …`.
These logs include calls Claude Code makes **on its own**, such as the IDE
server's `getDiagnostics` after an edit, which never appear in the transcript.

`--include-mcp` (or `MCPLogsParser().enrich(graph)`) adds the calls whose
`sessionId` matches the session, as `claude → tool:mcp__<server>__<tool>`. A
tool the transcript already shows keeps its node, and calls already counted
there are not counted twice. The logs do not say which agent made a call, so
all of them are attributed to the orchestrator. No logs (no MCP servers, or a
platform without the cache) is not an error: the graph is returned unchanged.

## OpenTelemetry

`--otel` turns every scan into one span, `agent_certify.scan`, with the
structural metrics as attributes (`agent_certify.fi`, `.node_count`,
`.edge_count`, `.bridge_count`, `.sub_agent_count`, `.anomaly_count`, ...).
Anomalies and policy violations become span events (`anomaly.new_tool`,
`policy_violation.forbidden_tool`, ...). A critical anomaly or any policy
violation sets the span status to ERROR.

Spans are OTLP/JSON, one `{"resourceSpans": [...]}` object per line: the
layout the OpenTelemetry Collector's `file` exporter writes and its
`otlpjsonfile` receiver reads, so a collector can forward them to Jaeger,
Grafana Tempo, Datadog and so on. All scans of one session share a trace id
derived from the session id, so a `watch` run appears as one trace with a span
per cycle; a graph with no session id (a Cursor trace that does not carry one)
gets a random trace id per span instead. No OpenTelemetry SDK is needed.

## Cursor Agent Trace

[Agent Trace](https://agent-trace.dev) (github.com/cursor/agent-trace) records
**which code was written by AI and which by humans**: files, line ranges and
contributors (`human`, `ai` with a `model_id`, `mixed`, `unknown`), usually in
`.agent-trace/traces.jsonl`. It does not record tool calls, MCP servers or
sub-agents, so the graph is a provenance graph:

- the producing tool (`cursor`) is the orchestrator, linked to each model (`agent:<model_id>`)
- each contributor links to the files it wrote (`file:<path>`), weighted by lines

That graph is bipartite (contributors on one side, editor and files on the
other) and has no triangles, so **FI is always 1.0** for it. The useful views
are blast radius (which files a model touched), policy rules such as
`{"kind": "forbidden_link", "target": "agent:* → file:infra/*"}` and anomaly
detection (a new model, new files touched).

Formats are detected from the content, not only the extension:
`.agent-trace/traces.jsonl` is routed to the Cursor parser although it ends in
`.jsonl`, and a CrewAI log is recognised whether it is `.json` or `.jsonl`. The
first 20 records of a file are probed and the first one a parser claims decides;
a file nothing claims is read as a Claude Code transcript, as before. A
directory is scanned for every supported format: Claude Code sessions,
`.agent-trace/` folders, `*.agent-trace` files, and CrewAI or LangGraph logs
(MCP server logs are left out; they enrich sessions via `--include-mcp`). A
`.json` file that holds no records of any format — `package.json`, say — is
ignored.

## CrewAI

A [CrewAI](https://crewai.com) crew writes its execution log as JSON
(`Crew(output_log_file="run.json")`), and its event bus is commonly dumped to
JSONL; both are read, as is a single object wrapping the entries
(`{"events": [...]}`). Records are matched on content, since the layout differs
between CrewAI versions: an entry is read when it names an agent
(`agent`, `agent_role`, `agent_name`, `role`, `agent_key`) and at least one of a
task, a tool, a crew or a status.

| Log content | Graph |
|---|---|
| the crew | `crew` (orchestrator) |
| `inputs` on a kickoff | `human` → `crew` |
| each agent role | `agent:<role>` |
| a task assigned to an agent | `crew` → `agent:<role>`, weight = distinct tasks |
| a tool call | `agent:<role>` → `tool:<name>` |
| `Delegate work to coworker` / `Ask question to coworker` | `agent:<from>` → `agent:<coworker>`, besides the tool edge |
| an `error`, or `status: failed` | error count on that edge |

Node ids are the roles themselves (`agent:Researcher`), so an agent is the same
node across runs and a baseline can be learned over a crew's history. A call is
counted once whether the log holds its start, its completion, or both (paired by
`tool_call_id`/`id`/`run_id`, or by agent + tool + task). Delegation is what
gives these graphs structure: `crew → Writer`, `crew → Researcher` and
`Writer → Researcher` close a triangle, so FI falls below 1 and the delegation
links have a backup path while the tool calls do not.

## LangGraph

[LangGraph](https://langchain-ai.github.io/langgraph/) runs a state machine of
named nodes. Three trace shapes are read, from `.json` or `.jsonl`:

1. **Event stream** — `graph.astream_events(..., version="v2")`:
   `metadata.langgraph_node` says which node a record belongs to and
   `metadata.langgraph_triggers` which node handed control to it.
2. **Debug stream** — `stream_mode="debug"`: `{"type": "task", "payload": {...}}`
   and its `task_result`.
3. **Plain transition records** — what a hand-rolled logger writes:
   `{"node": "agent", "next": "tools", "tool_calls": [...]}`.

| Trace content | Graph |
|---|---|
| the compiled graph | `graph` (orchestrator) |
| the input it was invoked with | `human` → `graph` |
| each node that ran | `agent:<node>` |
| a state transition | `agent:<from>` → `agent:<to>`, weight = times control took that path |
| `__start__` as the source | `graph` → `agent:<node>` |
| a tool invocation | `agent:<node>` → `tool:<name>` |
| `on_tool_error`, or an `error` field | error count on that edge |

A node is modelled as an agent whose type is its name, so `agent:tools` is
stable across runs. A node and the tools it calls stay separate: the `tools`
node of a ReAct graph is `agent:tools`, the tool it ran is `tool:search`.
`__start__` is the orchestrator, `__end__` is dropped (a sentinel, not a
participant), and a node that loops back to itself adds no edge — the repeat
shows in its call count. Consecutive records *about* the same node are one
activation, not a loop, so they move neither count.

Transitions come from `langgraph_triggers`/`triggers`, then from an explicit
`next`/`to`, and otherwise from the order the nodes ran in. That last fallback
is a guess: parallel branches interleave, so a trace without triggers can show a
transition the graph definition does not have. Traces from `astream_events` or
`stream_mode="debug"` carry triggers and do not rely on it.

## How Claude Code sessions map to graphs

| Log content | Graph |
|---|---|
| the main transcript's assistant | `claude` (orchestrator) |
| prompts typed by the person | `human` → `claude` |
| `tool_use` blocks | caller → `tool:<name>` (shared across agents) |
| `Agent` / `Task` calls (or any call with `subagent_type`) | caller → `agent:<id>` |
| `<session>/subagents/agent-<id>.jsonl` | that agent's own tool calls |
| `tool_result` with `is_error: true` | error count on the edge and the tool |

Streamed messages repeat `tool_use` blocks, so calls are de-duplicated by id.
Claude Code transcripts and MCP logs are read line by line; a Cursor
`.agent-trace` file of up to 50 MB is read whole (it may be a single JSON
document), larger ones line by line. Lines that cannot be read — malformed,
nested too deeply, or longer than 10 MB — are skipped and counted: the count
appears in the scan report, in JSON (`skipped_records`), in OTel spans and in
certificates. A warning is printed as soon as one record is skipped — on a
verification tool, an incomplete graph is worth saying out loud however small
the fraction — and the scan report gives that fraction. A directory scan reads
at most 1,000 session files and walks at most 5 folder levels.

A session without sub-agents is a star (one orchestrator, many tools) and has
FI = 1 by construction. Structure becomes informative once sub-agents share
tools: each shared tool closes a triangle with the orchestrator.

## Supported systems

- [x] Claude Code (JSONL logs from ~/.claude/)
- [x] Claude Code MCP server logs (`--include-mcp`)
- [x] Cursor Agent Trace (code provenance; `.agent-trace/`)
- [x] CrewAI (execution logs and event bus; `.json` / `.jsonl`)
- [x] LangGraph (event stream, debug stream or plain transition records)
- [ ] AutoGen (planned)
- [ ] OTel traces (planned)

New sources subclass `agent_certify.parser.BaseParser` and return an `AgentGraph`.

## Links

- [sal-torch](https://pypi.org/project/sal-torch/) — structural analysis of AI models
- [sal-certify](https://cognitive-engineering.dev/sal-certify) — structural AI certification
- [cognitive-engineering.dev](https://cognitive-engineering.dev)

---

**agent-certify** is built by [Cognitive Engineering](https://cognitive-engineering.dev) (Switzerland).
License: [Apache 2.0](LICENSE).

[![sal-torch](https://img.shields.io/pypi/v/sal-torch?label=sal-torch)](https://pypi.org/project/sal-torch/)
[![sal-certify](https://img.shields.io/badge/sal--certify-cognitive--engineering.dev-ffbf00)](https://cognitive-engineering.dev/sal-certify)
