Metadata-Version: 2.5
Name: etalon
Version: 0.0.2
Summary: Qualification evidence for private AI deployments
Project-URL: Homepage, https://github.com/coyos-ai/etalon
Project-URL: Website, https://etalon.sh
Project-URL: Documentation, https://docs.etalon.sh
Author: Coyos d.o.o.
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: evidence,llm,qualification
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.11
Requires-Dist: httpx==0.28.1
Requires-Dist: jinja2==3.1.6
Requires-Dist: jsonschema==4.23.0
Requires-Dist: pydantic>=2.10.6
Requires-Dist: pyyaml==6.0.2
Provides-Extra: dev
Requires-Dist: pytest==9.0.3; extra == 'dev'
Provides-Extra: typesafe
Requires-Dist: typesafe-sdk==0.7.1; extra == 'typesafe'
Description-Content-Type: text/markdown

# Etalon

[![License: Apache-2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
[![pytest](https://github.com/coyos-ai/etalon/actions/workflows/pytest.yml/badge.svg?branch=main)](https://github.com/coyos-ai/etalon/actions/workflows/pytest.yml?query=branch%3Amain)
[![PyPI](https://img.shields.io/pypi/v/etalon?logo=pypi&logoColor=white)](https://pypi.org/project/etalon/)
[![ghcr.io](https://img.shields.io/badge/ghcr.io-coyos--ai%2Fetalon-2496ED?logo=docker&logoColor=white)](https://github.com/coyos-ai/etalon/pkgs/container/etalon)
[![Python](https://img.shields.io/pypi/pyversions/etalon?logo=python&logoColor=white)](https://pypi.org/project/etalon/)

**Website:** [etalon.sh](https://etalon.sh) · **Docs:** [docs.etalon.sh](https://docs.etalon.sh) · **Source:** [github.com/coyos-ai/etalon](https://github.com/coyos-ai/etalon)

Etalon is an open-source runner that produces qualification evidence for a
specific self-hosted model deployment. It runs a versioned pack against an
OpenAI-compatible endpoint on your network and writes an evidence bundle:
deployment fingerprint, raw inputs and outputs, case results, coverage, a
three-state decision (`QUALIFIED`, `NOT_QUALIFIED`, `INDETERMINATE`), and a
self-contained HTML report. The runner sends no telemetry. Deterministic packs
contact only the configured model endpoint. Judge packs also contact their
declared judge transport, which is external TypeSafe Jev when a pack selects
it; a local `--judge-endpoint` keeps judging on your network (see
[Judges](https://docs.etalon.sh/docs/judges)).

Etalon does not determine that a regulation, contract, or internal control has
been met. It records what was evaluated, under which configuration, and how
the pack's own policy read the scores.

## Install

Python 3.11 or newer.

```bash
pip install etalon
```

Or run the container from GitHub Container Registry:

```bash
docker run --rm -it \
  -v "$(pwd)/packs:/app/packs" \
  -v "$(pwd)/evidence:/app/evidence" \
  ghcr.io/coyos-ai/etalon:latest \
  --help
```

`pip install 'etalon[typesafe]'` adds the TypeSafe judge transport. An
air-gapped install from a hashed wheelhouse is documented in
[Offline install](https://docs.etalon.sh/docs/offline-install).

## Quick start

The repository ships a synthetic demo pack and a scripted local fixture, so
you can produce a bundle with no model weights and no external service.

```bash
git clone https://github.com/coyos-ai/etalon.git && cd etalon
python3 -m venv .venv && source .venv/bin/activate
pip install -e .

# terminal A: fixture endpoint, listens on 127.0.0.1 only
python3 examples/mock_endpoint.py --port 8000

# terminal B
etalon pack validate examples/demo-pack

etalon qualify \
  --endpoint http://127.0.0.1:8000/v1 \
  --pack examples/demo-pack \
  --serving-config examples/demo-pack/serving.yaml \
  --system-prompt examples/demo-pack/system_prompt.txt \
  --model etalon-demo-baseline \
  --operator "Demo Operator" \
  --output runs/

# use the run directory that qualify printed
etalon inspect runs/<run-id>
etalon verify runs/<run-id>
etalon report runs/<run-id>
```

`qualify` exits 0 when a bundle was written, whatever the decision. Swap
`--model etalon-demo-quantized` for a `NOT_QUALIFIED` run, or
`etalon-demo-error` for `INDETERMINATE`; the full fixture table is in
[Packs](https://docs.etalon.sh/docs/packs).

Without a checkout, `etalon pack init ./my-demo --from demo` copies the bundled
demo pack out of the installed package, ready to edit and point at your own
endpoint.

No install at all: open
[examples/sample-run/report.html](examples/sample-run/report.html)
(`QUALIFIED`) or
[examples/sample-run-broken/report.html](examples/sample-run-broken/report.html)
(`NOT_QUALIFIED`) in a browser.

## Container

The image `ghcr.io/coyos-ai/etalon` is built from the [Dockerfile](Dockerfile)
in this repository. It runs as the non-root user `etalon` (uid 10001) with
working directory `/app`, and declares two volumes: `/app/packs` for packs and
`/app/evidence` for run output. The entrypoint is `etalon`, so arguments after
the image name are CLI arguments. Tags follow releases (`0.0.2`, `0.0`,
`latest`); `edge` tracks `main`.

```bash
mkdir -p packs evidence

# copy the bundled demo pack into ./packs and validate it
docker run --rm -v "$(pwd)/packs:/app/packs" \
  ghcr.io/coyos-ai/etalon:latest pack init /app/packs/demo --from demo
docker run --rm -v "$(pwd)/packs:/app/packs" \
  ghcr.io/coyos-ai/etalon:latest pack validate /app/packs/demo
```

Qualifying through the container against the local fixture, and the notes on
file ownership and host networking, are in
[Getting started](https://docs.etalon.sh/docs/getting-started#run-the-container).
`docker build -t etalon .` builds the same image from a checkout.

## What you get

A run directory is a verifiable bundle: manifest, deployment fingerprint,
captured inputs and outputs, per-case results, metrics with confidence
intervals, coverage, the decision, requalification triggers, the HTML report,
and `hashes.json` over all of it. `etalon verify` recomputes the hashes and fails
closed on any change. `etalon check` re-reads the deployment and tells you
whether the recorded fingerprint is still current. `etalon compare` puts two
bundles side by side.

Optional Ed25519 signing (`etalon keygen`, `etalon sign`,
`etalon verify --require-signature`) and a GitHub Action that fails a deploy
job on a missing or altered bundle are described in
[Signing](https://docs.etalon.sh/docs/signing).

## Commands

```text
etalon pack init <dir> --from demo
etalon pack validate <pack>
etalon qualify --endpoint URL --pack DIR --operator NAME --output DIR [--resume --run-id RUN_ID]
etalon inspect <run>
etalon verify <run> [--public-key KEY --require-signature]
etalon report <run> [--output FILE]
etalon check <run> --endpoint URL
etalon compare <run-a> <run-b>
etalon keygen | sign | export
```

Flags, exit codes, and the decision rules are in the
[CLI reference](https://docs.etalon.sh/docs/cli) and
[How qualification works](https://docs.etalon.sh/docs/how-qualification-works).

## Documentation

The operator guide lives in [`docs/`](docs/README.md) and is published at
[docs.etalon.sh](https://docs.etalon.sh):

- [Getting started](https://docs.etalon.sh/docs/getting-started) — install, the local fixture, the container
- [How qualification works](https://docs.etalon.sh/docs/how-qualification-works) — pack versus runner, bundle contents, decisions
- [CLI reference](https://docs.etalon.sh/docs/cli)
- [Packs](https://docs.etalon.sh/docs/packs) — the two public fixtures and the fixture model ids
- [Judges](https://docs.etalon.sh/docs/judges) and [TypeSafe judge](https://docs.etalon.sh/docs/typesafe-judge)
- [Offline install](https://docs.etalon.sh/docs/offline-install) — hashed wheelhouse, network namespace disabled
- [Signing](https://docs.etalon.sh/docs/signing) — Ed25519 over `hashes.json`, the verify action, stdlib helpers

## Development

```bash
pip install -e ".[dev]"
pytest
```

Direct dependencies are pinned in `pyproject.toml`; the hashed lock in
`requirements.txt` is what CI and the offline install use. The packs checked
in under `examples/` are public synthetic fixtures.

## Language

Say "qualified under pack X", "evaluated against methodology Y", "evidence
generated by runner Z". Do not write that a run is a legal determination or
that a regulation has been met.

## License

[Apache-2.0](LICENSE). Copyright Coyos d.o.o.
