Metadata-Version: 2.5
Name: varly
Version: 1.0.0
Summary: Deterministic runtime verification engine for AI agents.
Project-URL: Homepage, https://github.com/Hugoesin19/air-engine
Project-URL: Repository, https://github.com/Hugoesin19/air-engine
Project-URL: Documentation, https://github.com/Hugoesin19/air-engine#readme
Author: Hugoesin19
License-Expression: MIT
License-File: LICENSE
Keywords: agents,ai,ci,llm,observability,verification
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Typing :: Typed
Requires-Python: >=3.12
Requires-Dist: pyyaml>=6.0
Description-Content-Type: text/markdown

﻿# Varly

> Deterministic contract verification for AI agent runs.

AIR (Analysis Intermediate Representation) is the internal trace format. Varly verifies completed agent runs against YAML contracts — without LLM-as-judge.

## What is it?

varly is a **post-mortem verification infrastructure** for software systems with non-deterministic components (LLMs, multi-agent workflows).

It does not execute your agents. It observes completed executions, translates them into a provider-agnostic intermediate representation (AIR), and evaluates **properties and invariants** defined in contracts — not exact string matches.

The goal is to bring CI-style regression detection to probabilistic systems: deterministic, reproducible diagnostics that can gate merges before production.

## Quick start (5 minutes)

Prerequisites: **Python 3.12+**.

### Install from PyPI (recommended)

```bash
pip install varly
varly verify --demo
```

Expect `PASS` and `violations: 0`.

### Develop from source

Requires [uv](https://docs.astral.sh/uv/).

```bash
git clone https://github.com/Hugoesin19/air-engine.git
cd air-engine
uv sync

# 1) Generate a deterministic mock agent run (no API keys)
uv run python examples/demo_agent/run.py

# 2) Verify against the default policy → expect PASS (exit code 0)
uv run varly verify examples/demo_agent/artifacts/mock_run.json \
  --contract examples/policies/mvp.yaml \
  --source capture
```

You should see `PASS` and `violations: 0`. Run the full test suite with `uv run pytest`.

**Install options:** [docs/INSTALL.md](docs/INSTALL.md) · **60s demo:** `uv run python scripts/demo_60s.py` · **Viewer:** `uv run varly view --trace … --contract … --source capture`

More examples (canonical AIR traces, LangGraph/OpenAI fixtures, CI reports, `diff`) are below.

## Architecture

Seven responsibility domains, unidirectional data flow:

```
Capture → Adapters → AIR → Verification ← Contracts
                              ↓
                        Persistence
```

| Domain | Responsibility |
|--------|----------------|
| **Capture** | Obtain execution telemetry with minimal overhead |
| **Adapters** | Translate external formats into the internal model |
| **AIR** | Immutable, provider-agnostic intermediate representation |
| **Contracts** | Load and validate user-defined properties and invariants |
| **Verification** | Evaluate contracts against the AIR |
| **Persistence** | I/O for traces and diagnostics (no semantic knowledge) |
| **Orchestration** | CLI, policies, and lifecycle coordination |

The core (`core`, `analyzer`) has **zero external dependencies** and knows only about events and graphs — never about OpenAI, LangGraph, or any specific framework.

## More examples

```bash
# Validate structure + metrics
uv run varly validate examples/trace_valid_minimal.json --show-dag

# Verify against the full MVP policy
uv run varly verify examples/trace_valid_minimal.json \
  --contract examples/policy_mvp.yaml --show-metrics

# Generate a deterministic mock agent run with zero API cost
uv run python examples/demo_agent/run.py

# LangGraph / OpenAI telemetry → same verification (via library)
uv run python -c "
from varly.interfaces.library import load_trace, verify
from pathlib import Path
p = Path('examples/policy_mvp.yaml')
for src, path in [
    ('langgraph', 'examples/langgraph_run_minimal.json'),
    ('openai', 'examples/openai_run_minimal.json'),
]:
    d = verify(path, p, source=src)
    print(src, 'PASS' if d.passed else 'FAIL')
"

# Capture log → AIR → verify
uv run python -c "
from pathlib import Path
from varly.interfaces.library import verify
policy = Path('examples/policies/mvp.yaml')
print(verify('examples/demo_agent/artifacts/mock_run.json', policy, source='capture').passed)
"

# Recorded OpenAI Responses shape (no live API)
uv run varly verify examples/fixtures/recorded/openai_responses_search.json \
  --contract examples/policies/mvp.yaml --source openai
```

## Roadmap

| Phase | Focus | Outcome |
|-------|-------|---------|
| **MVP** | Core validation | ✅ AIR + contracts + adapters + CLI |
| **v1** | Source agnosticism + CI | Capture, mock agent, GitHub Action — [Product Roadmap](docs/PRODUCT_ROADMAP.md) |
| **v2** | Normative expressiveness | Formal contract DSL |
| **v3** | Ergonomics | Local GUI and topology editor |
| **v4** | Scalability | Distributed verification |
| **v5** | Research | Early stopping, policies, predictive analysis |

## Reference specification

- [MVP Roadmap](docs/MVP_ROADMAP.md) — Sprints 0–5 (complete)
- [Product Roadmap](docs/PRODUCT_ROADMAP.md) — post-MVP plan (Sprints 6+)
- [Product Development Roadmap](docs/PRODUCT_DEV_ROADMAP.md) — **active** product phases (P1–P5)
- [Next Steps Roadmap](docs/NEXT_STEPS_ROADMAP.md) — adoption steps 1–6 (complete)
- [Install guide](docs/INSTALL.md) — `uv`, `pip`, GitHub Action
- [Onboarding checklist](docs/ONBOARDING.md) — first PASS/FAIL on a fresh machine
- [Diagnostic viewer](docs/VIEWER.md) — `varly view` in the browser
- [Capture recipe](docs/recipes/capture-run-recorder.md) — instrument any agent with `RunRecorder`
- [Changelog](CHANGELOG.md)
- [Architecture specs](docs/architecture/) — formal AIR schema and contract model
- [Architecture Decision Records](docs/adrs/) — foundational design decisions

## Policy packs

Ready-made contracts in `examples/policies/` — swap the YAML file only to change verification strictness:

| File | Use case |
|------|----------|
| `mvp.yaml` | Default CI gate (10 s / 10 000 tokens) |
| `strict.yaml` | Tight limits (500 ms / 100 tokens) |
| `dev.yaml` | Relaxed local runs |

```bash
uv run varly verify examples/trace_valid_minimal.json --contract examples/policies/strict.yaml
uv run varly verify examples/trace_valid_minimal.json --contract examples/policies/mvp.yaml --output diagnostic.json
```

See [docs/policies/README.md](docs/policies/README.md) and [Diagnostic JSON schema](docs/architecture/diagnostic-schema-1.0.0.md).

## CI integration

This repository runs three CI jobs on every push and pull request:

- `quality` — lint, typecheck, and pytest
- `golden-fixtures` — stable CLI exit codes on canonical pass/fail examples
- `mock-agent-pipeline` — deterministic mock agent → capture log → verify (uploads SARIF)

Machine-readable reports:

```bash
uv run varly verify examples/trace_valid_minimal.json \
  --contract examples/policies/mvp.yaml --format json

uv run varly verify examples/trace_valid_minimal.json \
  --contract examples/policies/strict.yaml --format junit --output report.xml

uv run varly verify examples/trace_valid_minimal.json \
  --contract examples/policies/mvp.yaml --format sarif --output report.sarif
```

On GitHub Actions, failed verifies also emit `::error` annotations. The composite action uploads the report as an artifact (and SARIF when `report-format: sarif`).

### Baseline regression gate

Fail CI when a new run introduces violations the golden baseline did not have:

```bash
uv run varly diff \
  examples/trace_valid_minimal.json \
  examples/trace_invalid_missing_tool_return.json \
  --contract examples/policies/mvp.yaml
```

See [docs/workflows/baseline.md](docs/workflows/baseline.md).

Run the same fixture gate locally:

```bash
uv run python scripts/ci/verify_golden_fixtures.py
```

### Reuse in another repository

Copy `.github/actions/verify-trace/` into your project, then call it after your agent writes a trace or capture log:

```yaml
- uses: ./.github/actions/verify-trace
  with:
    trace-file: artifacts/run.json
    contract-file: policies/mvp.yaml
    source: capture
    report-format: junit
    report-file: varly-report.xml
```

Requirements for consumer repos:

- Python 3.12+
- `uv` available in the workflow
- `pyproject.toml` with `varly` installed, or run from a checkout of this repo

For canonical AIR traces, keep `source: air` (default). The action will run `validate` before `verify`.

## Development

```bash
uv sync
uv run ruff check .
uv run ruff format --check .
uv run mypy src
uv run pytest
uv run varly validate examples/trace_valid_minimal.json
uv run varly verify examples/trace_valid_minimal.json --contract examples/policy_mvp.yaml
```

## Product status

varly is an **open-source product under active development**, started as a Final Year Project and evolving toward a production-ready verification tool for AI-agent workflows.

- **MVP:** complete (AIR core, contracts, adapters, CLI)
- **v1:** complete (capture, CI, policy packs, reports, `diff`, viewer)
- **Now:** product phases P1–P5 — [Product Development Roadmap](docs/PRODUCT_DEV_ROADMAP.md)
- **Later:** contract DSL, enterprise ingest, scalable verification

This repository is the source of truth for design and implementation. Contributions and feedback are welcome under the license below.

## Authorship

Designed and implemented by **Hugo** ([@Hugoesin19](https://github.com/Hugoesin19)).

Copyright (c) 2026 Hugoesin19. All rights reserved under the MIT License terms.

If you reference this work academically or commercially, please keep attribution to the original repository: [github.com/Hugoesin19/air-engine](https://github.com/Hugoesin19/air-engine).

> **Note:** The product is **Varly** (`pip install varly`). Rename the GitHub repo to `varly` when you are ready.

## License

Released under the **MIT License** — see [LICENSE](LICENSE).

You may use, modify, and redistribute the software, including in commercial products, provided the copyright notice and license text are retained. The software is provided as-is, without warranty.
