Metadata-Version: 2.4
Name: pycharter
Version: 0.0.66
Summary: Runtime data contract enforcement for Python: one Pydantic-native contract validates API objects, Kafka streams, and ETL batches in-process — with quality scoring, a versioned contract registry, and ODCS round-trip.
Author-email: Optophi <contact@optophi.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/optophi/pycharter
Project-URL: Documentation, https://optophi.github.io/pycharter/
Project-URL: Repository, https://github.com/optophi/pycharter
Project-URL: Issues, https://github.com/optophi/pycharter/issues
Project-URL: Contributing, https://github.com/optophi/pycharter/blob/main/CONTRIBUTING.md
Keywords: data-contract,data-contracts,runtime-validation,data-quality,data-governance,json-schema,pydantic,odcs,open-data-contract-standard,kafka,streaming,etl,validation,schema-registry,schema
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: Database
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pydantic<3.0,>=2.0.0
Requires-Dist: jsonschema>=4.26.0
Requires-Dist: pandas>=2.0.0
Requires-Dist: numpy>=1.24.0
Requires-Dist: pyyaml>=6.0.0
Requires-Dist: sqlalchemy<3.0,>=2.0.0
Requires-Dist: alembic<2.0,>=1.13.0
Requires-Dist: jsonata-python>=0.6.0
Requires-Dist: httpx<1.0,>=0.24.0
Requires-Dist: networkx>=3.0
Provides-Extra: server
Requires-Dist: fastapi<0.142,>=0.136.3; extra == "server"
Requires-Dist: uvicorn[standard]<1.0,>=0.24.0; extra == "server"
Requires-Dist: starlette>=1.2.1; extra == "server"
Requires-Dist: pydantic-settings<3.0,>=2.14.2; extra == "server"
Requires-Dist: python-multipart>=0.0.6; extra == "server"
Requires-Dist: PyJWT<3.0,>=2.8.0; extra == "server"
Requires-Dist: httpx<1.0,>=0.24.0; extra == "server"
Requires-Dist: aiofiles>=25.1.0; extra == "server"
Requires-Dist: prometheus-client>=0.19.0; extra == "server"
Requires-Dist: psycopg[binary]>=3.3.4; extra == "server"
Provides-Extra: pipeline
Requires-Dist: sshtunnel>=0.4.0; extra == "pipeline"
Requires-Dist: boto3>=1.26.0; extra == "pipeline"
Requires-Dist: google-cloud-storage>=3.12.0; extra == "pipeline"
Requires-Dist: azure-storage-blob>=12.0.0; extra == "pipeline"
Requires-Dist: openpyxl>=3.1.5; extra == "pipeline"
Requires-Dist: lxml>=4.9.0; extra == "pipeline"
Requires-Dist: pyarrow>=14.0.0; extra == "pipeline"
Requires-Dist: pymongo>=4.0.0; extra == "pipeline"
Requires-Dist: openlineage-python>=1.0.0; extra == "pipeline"
Provides-Extra: streaming
Requires-Dist: websockets>=12.0; extra == "streaming"
Requires-Dist: watchfiles>=0.21.0; extra == "streaming"
Requires-Dist: prometheus_client>=0.19.0; extra == "streaming"
Requires-Dist: aiohttp>=3.9.0; extra == "streaming"
Requires-Dist: aiokafka>=0.9.0; extra == "streaming"
Requires-Dist: aio-pika>=9.0.0; extra == "streaming"
Requires-Dist: redis>=5.0.0; extra == "streaming"
Requires-Dist: python-socketio>=5.11.0; extra == "streaming"
Provides-Extra: semantic
Requires-Dist: linkml>=1.7.0; extra == "semantic"
Requires-Dist: linkml-runtime>=1.7.0; extra == "semantic"
Requires-Dist: rdflib>=7.0.0; extra == "semantic"
Requires-Dist: owlrl>=6.0.0; extra == "semantic"
Requires-Dist: neo4j<7.0.0,>=6.2.0; extra == "semantic"
Provides-Extra: extraction
Requires-Dist: tree-sitter<1.0.0,>=0.25.2; extra == "extraction"
Requires-Dist: tree-sitter-python<1.0.0,>=0.23.0; extra == "extraction"
Requires-Dist: tree-sitter-javascript<1.0.0,>=0.23.0; extra == "extraction"
Requires-Dist: tree-sitter-typescript<1.0.0,>=0.23.0; extra == "extraction"
Requires-Dist: tree-sitter-go<1.0.0,>=0.23.0; extra == "extraction"
Requires-Dist: tree-sitter-rust<1.0.0,>=0.24.2; extra == "extraction"
Requires-Dist: tree-sitter-java<1.0.0,>=0.23.5; extra == "extraction"
Requires-Dist: markdown-it-py<5.0.0,>=4.2.0; extra == "extraction"
Requires-Dist: pypdf<8.0.0,>=4.0.0; extra == "extraction"
Requires-Dist: sqlglot>=25.0; extra == "extraction"
Provides-Extra: ai
Requires-Dist: anthropic>=0.34.0; extra == "ai"
Requires-Dist: openai>=2.44.0; extra == "ai"
Requires-Dist: mcp<3,>=1.0; extra == "ai"
Provides-Extra: otel
Requires-Dist: opentelemetry-api<2,>=1.20; extra == "otel"
Provides-Extra: cli
Requires-Dist: rich<16.0,>=13.0.0; extra == "cli"
Provides-Extra: docs
Requires-Dist: mkdocs<2,>=1.5.3; extra == "docs"
Requires-Dist: mkdocs-material[imaging]>=9.7.6; extra == "docs"
Requires-Dist: mkdocstrings[python]>=0.24.0; extra == "docs"
Requires-Dist: pymdown-extensions>=10.0.0; extra == "docs"
Requires-Dist: mike>=2.0.0; extra == "docs"
Provides-Extra: airflow
Requires-Dist: apache-airflow<4,>=3.2.2; extra == "airflow"
Provides-Extra: dagster
Requires-Dist: dagster>=1.7; extra == "dagster"
Provides-Extra: prefect
Requires-Dist: prefect<4,>=3.0; extra == "prefect"
Provides-Extra: postgres
Requires-Dist: asyncpg>=0.29; extra == "postgres"
Requires-Dist: psycopg[binary]>=3.3.4; extra == "postgres"
Provides-Extra: all
Requires-Dist: pycharter[ai,cli,docs,extraction,otel,pipeline,postgres,semantic,server,streaming]; extra == "all"
Provides-Extra: ci
Requires-Dist: pycharter[extraction,pipeline,postgres,semantic,server,streaming]; extra == "ci"
Requires-Dist: testcontainers[postgres]>=4.0.0; extra == "ci"
Requires-Dist: testcontainers[kafka]>=4.0.0; extra == "ci"
Requires-Dist: pytest>=7.0.0; extra == "ci"
Requires-Dist: pytest-asyncio>=0.21.0; extra == "ci"
Requires-Dist: pytest-cov>=4.0.0; extra == "ci"
Requires-Dist: pytest-timeout>=2.1.0; extra == "ci"
Requires-Dist: pytest-xdist>=3.5.0; extra == "ci"
Requires-Dist: pytest-randomly>=3.15.0; extra == "ci"
Requires-Dist: ruff>=0.15.20; extra == "ci"
Requires-Dist: mypy>=1.0.0; extra == "ci"
Requires-Dist: types-PyYAML>=6.0.0; extra == "ci"
Requires-Dist: requests>=2.32.3; extra == "ci"
Requires-Dist: pre-commit>=3.0.0; extra == "ci"
Requires-Dist: mkdocs<2,>=1.5.3; extra == "ci"
Requires-Dist: mkdocs-material[imaging]>=9.7.6; extra == "ci"
Requires-Dist: mkdocstrings[python]>=0.24.0; extra == "ci"
Requires-Dist: pymdown-extensions>=10.0.0; extra == "ci"
Provides-Extra: dev
Requires-Dist: pycharter[all]; extra == "dev"
Requires-Dist: requests>=2.32.3; extra == "dev"
Requires-Dist: pytest>=7.0.0; extra == "dev"
Requires-Dist: pytest-asyncio>=0.21.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
Requires-Dist: pytest-timeout>=2.1.0; extra == "dev"
Requires-Dist: pytest-xdist>=3.5.0; extra == "dev"
Requires-Dist: pytest-benchmark>=4.0.0; extra == "dev"
Requires-Dist: testcontainers[kafka]>=4.0.0; extra == "dev"
Requires-Dist: testcontainers[postgres]>=4.0.0; extra == "dev"
Requires-Dist: opentelemetry-sdk<2,>=1.20; extra == "dev"
Requires-Dist: ruff>=0.15.20; extra == "dev"
Requires-Dist: mypy>=1.0.0; extra == "dev"
Requires-Dist: types-PyYAML>=6.0.0; extra == "dev"
Requires-Dist: pre-commit>=3.0.0; extra == "dev"
Requires-Dist: build>=0.10.0; extra == "dev"
Requires-Dist: twine>=4.0.0; extra == "dev"
Requires-Dist: marimo>=0.9.0; extra == "dev"
Requires-Dist: vulture>=2.11; extra == "dev"
Requires-Dist: deptry>=0.20; extra == "dev"
Requires-Dist: pip-audit>=2.7; extra == "dev"
Dynamic: license-file

# PyCharter

> **PyCharter enforces data contracts where your code runs.**

Define a contract once — from a **Pydantic model** (`pycharter contract
from-model`) or versioned **YAML** — and the same artifact validates API
payloads in-process, gates Kafka consumers record by record, and checks ETL
batches at stage boundaries. Deterministic, Pydantic-native, in your process:
no warehouse round-trip, no separate CI engine deciding after the fact.
**The contract is what actually runs.**

Most contract tooling stops earlier in the lifecycle: datacontract-cli — the
ODCS reference implementation — verifies contracts in CI and batch; Soda and
Great Expectations scan the warehouse; dbt tests run after materialisation.
PyCharter is the **runtime half** of that ecosystem: it executes contracts
inside the producing service, and round-trips **ODCS** (v3.1.0 export, v3.0.x /
v3.1.x import) so the contracts you enforce at runtime slot into the tools you
already use.

**Full documentation:** [https://optophi.github.io/pycharter/](https://optophi.github.io/pycharter/)

[![Python 3.11+](https://img.shields.io/badge/python-3.11+-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)

**[Where PyCharter fits vs datacontract-cli, Pydantic, Pandera, GX, Soda, dbt →](docs/getting-started/comparisons.md#where-pycharter-fits)**

---

## Validate a record in 30 seconds

```bash
pip install pycharter
```

```python
from pycharter import Validator

validator = Validator.from_dict(
    {
        "type": "object",
        "version": "1.0.0",
        "properties": {
            "name": {"type": "string"},
            "age": {"type": "integer"},
        },
        "required": ["name", "age"],
    }
)

result = validator.validate({"name": "Alice", "age": 30})
print(result.is_valid)  # True

result = validator.validate({"name": "Bob", "age": "N/A"})
print(result.errors[0])  # Column 'age': expected integer, found string 'N/A'
```

That's the library at its smallest — no database, no setup. In production
you'd load the same contract from a file (`Validator.from_file("contract.yaml")`)
or a shared store. Failures come back as structured diagnostics (row, column,
expected type, offending value), and the generated model validates as fast as
a hand-written Pydantic model (~0.7 µs/record —
[measured](docs/operations/benchmarks.md)).

Prefer the shell? The same check works on a whole file:

```bash
pycharter validate contract.yaml data.jsonl   # colorized report, exit 0/1
```

---

## Why PyCharter

Teams often maintain a Pydantic model at the API, a Pandera schema in the
pipeline, a Great Expectations suite for quality, and an ODCS YAML as the
"contract" — and they drift. PyCharter unifies authoring and enforcement so
the contract is what actually runs, **where** it runs being the point:

| Tool | Where enforcement runs | Object | Dataset | Quality | Contract | Versioned |
|------|------------------------|:------:|:-------:|:-------:|:--------:|:---------:|
| Pydantic | In-process (single objects) | ● | ○ | ○ | ◐ | ◐ |
| Pandera | In-process (DataFrames) | ◐ | ● | ◐ | ◐ | ◐ |
| Great Expectations | Warehouse / batch jobs | ○ | ● | ● | ◐ | ◐ |
| Soda | Warehouse-side, scheduled scans | ○ | ● | ● | ◐ | ◐ |
| dbt tests | Warehouse, post-materialisation | ○ | ● | ◐ | ◐ | ● |
| datacontract-cli (ODCS ref. impl.) | CI / batch | ○ | ◐ | ◐ | ● | ● |
| **PyCharter** | **In-process runtime: APIs, streams, ETL** | **●** | **●** | **●** | **●** | **●** |

<sub>● full · ◐ partial · ○ none — markers reflect scope, not quality; see the
[full comparison](docs/getting-started/comparisons.md#where-pycharter-fits)</sub>

- **One definition, every surface** — model or YAML → API gate, stream gate,
  ETL steps, quality jobs, ODCS round-trip.
- **Enforces, not just authors** — rejects or coerces bad records *before*
  they land; warehouse-side engines measure what *did* land.
- **Deterministic and auditable** — versioned contracts, reproducible runs,
  thread-safe shared validators.

Choose Pydantic alone for one-service API shape checks, Pandera for
DataFrame validation in one process, GX/Soda for warehouse-centric batch
quality, datacontract-cli for ODCS authoring and CI gates — and PyCharter
when the contract must be the enforced single source of truth across
services. Full narrative and per-tool sections:
**[PyCharter vs. the alternatives](docs/getting-started/comparisons.md)**.

---

## Gate a FastAPI endpoint

```bash
pip install pycharter[server]
```

```python
from fastapi import Depends, FastAPI
from pycharter.contrib.fastapi import contract_dependency

app = FastAPI()
validate_order = contract_dependency("contracts/orders.yaml")  # built once


@app.post("/orders")
async def create_order(order: dict = Depends(validate_order)) -> dict:
    # `order` is the coerced + validated payload.
    return {"status": "accepted", "order_id": order["order_id"]}
```

Contract violations return a structured 422; the same YAML also drives your
pipelines and workers. Recipe:
[Gate a FastAPI endpoint](docs/cookbook/gate-fastapi-endpoint.md).

## Validate a Kafka topic, per record

```python
import asyncio
from pycharter import kafka


async def main():
    async for record in kafka.consume(
        topic="orders",
        bootstrap_servers="localhost:9092",
        group_id="my-service",
        contract_dir="contracts/orders",  # or contract_store=...
    ):
        process(record.payload)  # only records that passed the contract


asyncio.run(main())
```

Invalid records route to a DLQ with a structured envelope; `detect` /
`shadow` / `enforce` modes support safe rollouts, and `pycharter stream run`
runs it as a worker with Prometheus metrics. Start at
[Streaming — start here](docs/guides/streaming-overview.md).

## Gate contract evolution in CI

```bash
pycharter contract diff old.yaml new.yaml --fail-on-breaking
```

Classifies every field/type/required change as breaking or safe
(backward-compat semantics) — drop into pre-commit or CI so a contract
change can't ship without review.

---

## Concepts

| Concept | What it is | When you use it |
|--------|------------|------------------|
| **Schema** | The shape of the data (JSON Schema): types, required fields, nested objects. | When you only need structure. |
| **Data contract** | Schema + **coercion rules** (`"30"` → `30`) + **validation rules** (min/max, allowed values) + optional **metadata** (ownership, governance). | One artifact for structure, transforms, and business rules. |
| **Contract store** | A database (SQLite, PostgreSQL, …) holding versioned contracts so many apps reuse them. | Multiple services needing one source of truth. |
| **Enforcement point** | Where the contract executes: API dependency, stream consumer, ETL step flags (`coerce` / `validate` / `quality`). | Wherever bad data must be stopped. |

```
  Option A: no database   schema/contract in code or YAML → Validator → validate(data)
  Option B: with store    contract in DB → Validator(store=...)       → validate(data)
  Option C: pipelines     extract → [contract] → transform → [contract] → load
```

Start with Option A; add the store when contracts need versioned sharing; add
pipeline/stream enforcement where data moves. Deep dives:
[Concepts](docs/getting-started/concepts.md) ·
[Contract → ETL → quality journey](docs/getting-started/contract-etl-quality-journey.md) ·
[Cookbook](docs/cookbook/index.md).

---

## Artifacts

PyCharter has **six artifacts**. Every one is a YAML file in the same envelope,
so all six are authored, versioned, diffed, and reviewed the same way:

```yaml
api_version: pycharter.io/v1   # the format marker — always this
kind: DataContract             # which artifact this is (one of the six below)
metadata:
  name: orders                 # identity slug
  namespace: example           # identity scope (optional; absent = global)
  version: "1.0.0"             # content version
spec:                          # the body — shape depends on `kind`
  ...
```

Identity is `(metadata.namespace, metadata.name, metadata.version)`, and
`name` is also the display label — there is no separate title field.

| `kind:` | Defines | Run it with |
|---|---|---|
| **`DataContract`** | The rules a record must satisfy: shape, coercions, validations, and what its fields *mean*. | `Validator`, or a `contract:` block on any pipeline / stream step |
| **`Pipeline`** | A batch extract → transform → load flow. | `pycharter pipeline run` |
| **`StreamingWorker`** | A long-running consumer that validates a stream record by record. | `pycharter stream run` |
| **`Bridge`** | A WebSocket firehose republished to Kafka, so it can be consumed by a scaled worker group. No validation, no DLQ. | `pycharter bridge run` |
| **`ConceptScheme`** | A governed vocabulary: the concepts contract fields bind to. | `pycharter db seed`, semantic API, UI |
| **`ConceptSchemeTemplate`** | A reusable starter vocabulary you copy and adapt into a `ConceptScheme`. | `GET /api/v1/semantic/templates`, UI |

Scaffold one with `pycharter contract init`, `pycharter pipeline init`,
`pycharter stream init`, or `pycharter bridge init`; runnable end-to-end
examples live in [`data/seed/examples/`](src/pycharter/data/seed/examples/).
Files written by an older PyCharter still load, and `pycharter migrate` rewrites
them in place — see
[Artifact format versioning](docs/reference/artifact-format-versioning.md).

### 1. `DataContract`

The artifact every enforcement point loads. One file carries the shape
(`json_schema`), the transforms applied **before** validation
(`coercion_rules`), and the business rules applied **after** the types check out
(`validation_rules`):

```yaml
# contracts/orders.yaml
api_version: pycharter.io/v1
kind: DataContract
metadata:
  name: order
  version: "1.0.0"
  status: active
  type: domain_entity
  description: A customer order accepted by the storefront API.
  ownership:
    roles:
      business_owner: [storefront-team]
      technical_owner: [data-platform]

spec:
  json_schema:
    type: object
    title: order
    version: "1.0.0"
    additionalProperties: false
    required: [order_id, customer_id, total_amount, currency_code, status]
    properties:
      order_id:      {type: string, format: uuid}
      customer_id:   {type: string}
      total_amount:  {type: number, minimum: 0}
      currency_code: {type: string, minLength: 3, maxLength: 3}
      status:        {type: string}
      placed_at:     {type: string, format: date-time}

  # Before validation: make messy input typed. "19.99" -> 19.99, "usd" -> "USD".
  coercion_rules:
    version: "1.0.0"
    rules:
      total_amount: coerce_to_float
      currency_code: coerce_to_uppercase
      placed_at: coerce_to_datetime

  # After validation: business rules beyond types.
  validation_rules:
    version: "1.0.0"
    rules:
      total_amount:
        greater_than_or_equal_to: {threshold: 0}
      currency_code:
        matches_regex: {pattern: "^[A-Z]{3}$"}
      status:
        only_allow:
          allowed_values: [pending, paid, shipped, cancelled]
```

That is the `contracts/orders.yaml` the FastAPI gate above loads — and the same
file backs `pycharter validate contracts/orders.yaml orders.jsonl`, the Kafka
consumer, and the pipeline below. Rules are **named, not code**: 21 built-in
coercions and 16 validations, listed in
[built-in coercions and validations](docs/reference/builtin-coercions-and-validations.md)
(add `severity: warning` inside a rule to report instead of reject). Start one
from a real sample with
`pycharter contract sketch --from-json sample.json -o contracts/orders.yaml`.

A contract's four body sections — `json_schema`, `coercion_rules`,
`validation_rules`, and `field_bindings` (below) — can also be **stored
separately and reused by reference**, so two contracts share one set of
validation rules instead of copying them. Authoring them inline, as above, is
the default and always works.

### 2. `Pipeline`

Batch extract → transform → load. The `contract:` block attached to a step *is*
the enforcement point — put it on whichever steps must be gated:

```yaml
# pipelines/orders_ingest/pipeline.yaml
api_version: pycharter.io/v1
kind: Pipeline
metadata:
  name: orders_ingest
  namespace: examples
  version: "1.0.0"
  description: Load the daily orders CSV, normalize it, export validated JSONL.

spec:
  variables:
    INPUT_PATH: data/input/orders.csv
    OUTPUT_PATH: data/exports/orders.jsonl

  steps:
    - id: extract_orders
      type: extract
      source:
        type: file
        path: ${INPUT_PATH}
        format: csv

    - id: normalize_orders
      type: transform
      input: extract_orders
      operations:
        - rename: {order_total: total_amount}
        - defaults: {status: pending}

    - id: load_orders
      type: load
      input: normalize_orders
      target:
        type: file
        path: ${OUTPUT_PATH}
        format: jsonl
        mode: overwrite
      contract:                      # <- the enforcement point
        ref: {type: store, name: order, version: "1.0.0"}
        coerce: true                 # apply coercion_rules
        validate: true               # reject records that fail the contract
        quality: true                # run the contract's quality checks
```

```bash
pycharter pipeline run ./pipelines/orders_ingest --watch
```

Use `ref: {type: file, path: ../../contracts/orders.yaml}` to point at a
contract on disk instead of the store. Attach the block to any step — gate on
extract, after transform, before load, or all three. Sources include `file`,
`http`, and SQL; `${VAR}` placeholders resolve from the `spec.variables:` block
first, then the environment, with `${VAR:-default}` and `${VAR:?required}`
semantics. More: [Building pipelines](docs/tutorials/pipelines.md).

### 3. `StreamingWorker`

The same contract, enforced record by record on a live stream instead of a
batch. `mode` is what makes a rollout safe: `detect` passes every record through
and only counts violations, `shadow` forwards invalid records *and* copies them
to the DLQ, and `enforce` (the default) sends them to the DLQ only:

```yaml
# streams/orders/stream-worker.yaml
api_version: pycharter.io/v1
kind: StreamingWorker
metadata:
  name: orders
  namespace: example
  version: "1.0.0"
  description: Validate the orders.raw topic against the order contract.

spec:
  variables:
    KAFKA_BROKERS: localhost:9092

  streams:
    - name: orders-validator
      source:
        type: kafka                  # kafka | websocket | sse | rabbitmq | sqs
        topic: orders.raw
        bootstrap_servers: ${KAFKA_BROKERS}
        consumer_group: pycharter-orders
        auto_offset_reset: latest
      contract:                      # same block shape as a pipeline step
        ref: {type: store, name: order, version: "1.0.0"}
        coerce: true
        validate: true
      mode: enforce                  # detect | shadow | enforce
      delivery: at_least_once
      load:
        target:
          type: kafka
          topic: orders.validated
          bootstrap_servers: ${KAFKA_BROKERS}
      dlq:
        target:
          type: kafka
          topic: orders.dlq
          bootstrap_servers: ${KAFKA_BROKERS}

  observability:
    health_port: 8080                # /healthz, /readyz, /metrics
    log_format: json
```

```bash
pycharter stream run ./streams/orders/stream-worker.yaml
```

More: [Streaming — start here](docs/guides/streaming-overview.md).

### 4. `Bridge`

A WebSocket is a single connection, so running more workers against one just
duplicates consumption. A bridge republishes the firehose onto a partitioned
Kafka topic, which a scaled `StreamingWorker` group can then consume. It only
moves bytes — no contract, no validation, no DLQ:

```yaml
# bridges/firehose/bridge.yaml
api_version: pycharter.io/v1
kind: Bridge
metadata:
  name: firehose-bridge
  namespace: example
  version: "1.0.0"
  description: Republish a WebSocket firehose to a Kafka topic.

spec:
  variables:
    KAFKA_BROKERS: localhost:9092

  bridges:
    - name: firehose
      source:
        type: websocket
        url: ${WS_URL:?WS_URL is required}
        data_format: json
        ping_interval: 20.0          # keepalive seconds
        idle_timeout: 60.0           # reconnect after this much silence
      sink:
        type: kafka
        topic: market.raw
        bootstrap_servers: ${KAFKA_BROKERS}
```

```bash
pycharter bridge run ./bridges/firehose/bridge.yaml
```

### 5. `ConceptScheme`

Contracts describe the *shape* of a field. A concept scheme describes what
fields **mean**, once, for every contract that uses them:

```yaml
# vocabulary/ecommerce.yaml
api_version: pycharter.io/v1
kind: ConceptScheme
metadata:
  name: ecommerce
  version: "2.0.0"
  description: Governed vocabulary for the storefront domain.

spec:
  concepts:
    - id: Customer
      label: Customer
      concept_type: entity          # a thing with identity
      definition: A person or organization that places orders.
    - id: Order
      label: Order
      concept_type: event
      definition: A customer's request to purchase one or more products.
    - id: MonetaryAmount
      label: Monetary Amount
      concept_type: attribute       # a property or measurement
      definition: A value of money in a stated currency.
    - id: OrderTotal
      label: Order Total
      concept_type: attribute
      broader: MonetaryAmount       # narrower than MonetaryAmount
      definition: Amount payable for an order, including tax and shipping.
```

Contracts then **bind** their fields to those concepts, in the same contract
file as the schema:

```yaml
spec:
  field_bindings:
    order_id:
      primary:
        concept: {concept: Order}
        concept_scheme: "ecommerce:2.0.0"
        role: identity              # this field identifies the record's entity
    customer_id:
      primary:
        concept: {concept: Customer}
        concept_scheme: "ecommerce:2.0.0"
        role: reference             # points at a different entity
    total_amount:
      primary:
        concept: {concept: OrderTotal}
        concept_scheme: "ecommerce:2.0.0"
        role: attribute             # a property of this record (the default)
      annotations:                  # optional extra meaning, zero or more
        - concept: MonetaryAmount
          concept_scheme: "ecommerce:2.0.0"
          relationship: derived_from
```

A binding may also carry `pii: true` (personal data) or `restricted: true`
(raw values must never be captured) — field-level flags that govern what
downstream profiling, extraction, and AI surfaces are allowed to see.

Bindings are what make two contracts that both call a column `total` comparable,
and they are what the MCP server serves to agents: the context an agent receives
about a field is bound to the same contract your validators enforce. A field may
stay unbound — governance tooling simply flags it for review.

### 6. `ConceptSchemeTemplate`

A starter vocabulary — the same idea as a `ConceptScheme`, but shipped as a
reusable pattern you copy and adapt rather than a governed scheme you enforce
against. PyCharter bundles six (`ecommerce`, `healthcare`, `saas`, `party`,
`product_catalog`, `event_participation`); this is the shape if you write your
own:

```yaml
# templates/subscriptions.yaml
api_version: pycharter.io/v1
kind: ConceptSchemeTemplate
metadata:
  name: subscriptions
  version: "1.0.0"
  description: Tenants, plans, and recurring charges.
  category: domain               # domain (a full domain) | pattern (a reusable cluster)
  tags: [billing, saas]

spec:
  concepts:
    - name: Tenant
      concept_type: entity
      definition: An organization that holds a subscription.
      alt_labels: [Account, Workspace]
    - name: Plan
      concept_type: reference_data
      definition: A named tier a tenant can subscribe to.
    - name: Subscription
      concept_type: event
      definition: A tenant's commitment to a plan over a billing period.

  relationships:
    - {source: Tenant, target: Subscription, type: has, label: holds}
    - {source: Subscription, target: Plan, type: references, label: on plan}
```

Browse and fetch the bundled ones with `GET /api/v1/semantic/templates`, or read
them under
[`data/templates/concept_schemes/`](src/pycharter/data/templates/concept_schemes/).

More on the vocabulary layer: [Vocabulary](docs/guides/concepts.md) ·
[Binding any resource to a concept](docs/guides/resource-bindings.md) · a
complete worked example (22 contracts, a scheme, and a pipeline) ships in
[`data/seed/examples/ecommerce/`](src/pycharter/data/seed/examples/ecommerce/).

---

## Installation

```bash
pip install pycharter
```

That's all you need for in-process validation. Optional extras add
capabilities — install only what you use:

| Extra | Adds | Install |
|-------|------|---------|
| `[fastapi]` | `contract_dependency` request gate | `pip install pycharter[server]` |
| `[streaming]` / `[kafka]` | Streaming validation engine / Kafka source | `pip install pycharter[streaming]` |
| `[api]` | REST API server (FastAPI + Uvicorn) | `pip install pycharter[server]` |
| `[ui]` | Pre-built Web UI (no Node.js required) | `pip install pycharter[server]` |
| `[postgres]` | PostgreSQL drivers for the ETL loader, DLQ, and contract store (asyncpg + psycopg) | `pip install pycharter[postgres]` |
| `[mcp]` | MCP server for AI agents | `pip install pycharter[ai]` |
| `[otel]` | OpenTelemetry tracing instruments | `pip install pycharter[otel]` |
| `[observability]` | Prometheus exporter + webhook sinks | `pip install pycharter[server]` |
| `[airflow]` / `[dagster]` | Orchestrator operators | `pip install pycharter[airflow]` |
| `[all]` | The common stack: api, ui, worker, pipeline, postgres, streaming, messaging, extraction, lineage, semantic-export, collab, cli, docs (not the optional `mcp` / `otel` / `observability` / `airflow` / `dagster` integrations — install those explicitly) | `pip install pycharter[all]` |

### Database setup (only for the contract store, API, or UI)

```bash
pycharter db init     # creates schema (default: sqlite:///pycharter.db)
pycharter db seed     # loads reference data
```

Skip this entirely if you only validate dicts or YAML contracts directly.
PostgreSQL/SQLite have managed Alembic migrations; see
[Store backend tiers](docs/reference/store-backend-tiers.md) and the
[Configuration Guide](docs/guides/configuration.md).

---

## ODCS and the contract ecosystem

PyCharter round-trips the
[Open Data Contract Standard](https://bitol-io.github.io/open-data-contract-standard/):
`to_odcs()` emits v3.1.0 documents that validate against the published
schema; `from_odcs()` imports v3.0.x / v3.1.x documents authored anywhere —
so contracts written with datacontract-cli enforce at runtime here, and
contracts authored here flow back into the ODCS toolchain. PyCharter is not
a rival standard; it is an **execution engine** for the standard.

```python
from pycharter.contract_io import from_odcs, to_odcs
from pycharter import Validator

contract = from_odcs(odcs_document)  # any v3.0.x / v3.1.x doc
validator = Validator(contract)  # enforce it immediately
```

How PyCharter composes with datacontract-cli, Soda, GX, and dbt:
[ODCS integration and interop](docs/guides/odcs-and-interop.md).

## Agents and MCP

PyCharter ships an MCP server (`python -m pycharter.mcp`, stdio JSON-RPC)
that serves **governed context packs** — concepts, relationships, and the
contract fields bound to them — to Claude Code, Claude Desktop, or any MCP
client, with role/workspace policy filtering. The context an agent receives
is bound to the same contracts PyCharter enforces at runtime, so what the
agent is told about your data is what your validators actually check.

```bash
pip install pycharter[ai]
export PYCHARTER_DATABASE_URL=postgresql://localhost/pycharter
python -m pycharter.mcp
```

Tools, client config, and the governance model:
[Agent integration (MCP)](docs/guides/agent-integration-mcp.md).

---

## Architecture at a glance

| Service | Input | Output | Journey stage |
|---------|-------|--------|---------------|
| **Contract Parser** | Contract files (YAML/JSON) | `ContractMetadata` | Specification → Parsing |
| **Contract Builder** | Separate artifacts or store | Consolidated contract | Storage → Consolidation |
| **Contract Store** | `ContractMetadata` | Stored metadata (DB) | Parsing → Storage |
| **Pydantic Generator** | JSON Schema | Pydantic models | Storage → Model generation |
| **JSON Schema Converter** | Pydantic models | JSON Schema | (Bidirectional) |
| **Runtime Validator** | Pydantic models + data | `ValidationResult` | Model generation → Validation |
| **ETL Pipelines** | Config files or code | `PipelineResult` | Extract → Transform → Load |
| **Quality Assurance** | Contract + data | `QualityReport` | Validation → Quality monitoring |

Each service is independent yet composable. The full tour — including the
REST API and Web UI wrappers — lives in the
[Core services reference](docs/reference/core-services.md); the layered
design philosophy is in [ARCHITECTURE.md](ARCHITECTURE.md).

## Picking the right API

The canonical path is the **`Validator` class** — create one from a contract
file, dict, directory, or store, and call `validate(record)` /
`validate_batch(records)`. It compiles the Pydantic model once, is
[thread-safe to share](docs/operations/benchmarks.md), and is what
production code should use.

| Use case | Approach | Example |
|----------|----------|---------|
| Production service, many validations | `Validator` class | `Validator.from_file("c.yaml").validate(data)` |
| Pandas DataFrame, split good/bad rows | `validate_dataframe()` | `validate_dataframe(df, "c.yaml", on_error="quarantine")` |
| Shell / CI gate on a data file | `pycharter validate` CLI | `pycharter validate c.yaml data.jsonl` |
| Batch processing | `Validator.validate_batch()` | `validator.validate_batch(rows)` |
| You already have a Pydantic model | Low-level function (deprecated at top level) | `from pycharter.runtime_validator import validate` |

ETL flows use **`Pipeline`**; quality checks use **`QualityCheck`**; contract
storage is pluggable via **`ContractStoreClient`** implementations. Built-in
coercions/validations and custom registration:
[reference](docs/reference/builtin-coercions-and-validations.md).

---

## Documentation

- **Docs site:** [https://optophi.github.io/pycharter/](https://optophi.github.io/pycharter/) — or `pycharter docs serve` locally (`pip install pycharter[docs]`)
- [Start here](docs/getting-started/start-here.md) · [Quick Start tour](docs/getting-started/quickstart.md) · [Cookbook](docs/cookbook/index.md)
- [End-to-end user guide](docs/guides/user-guide-end-to-end.md) · [Admin / governance guide](docs/guides/admin-guide-governance.md)
- [Streaming — start here](docs/guides/streaming-overview.md) · [Configuration](docs/guides/configuration.md) · [Validator benchmarks](docs/operations/benchmarks.md)

## Development and testing

```bash
python -m venv venv && source venv/bin/activate
pip install -e ".[dev]" && pre-commit install
pytest tests/unit            # fast suite; `pytest -m integration` needs Docker
./scripts/ci.sh              # full GHA-parity gate before opening a PR
```

See [CONTRIBUTING.md](CONTRIBUTING.md) for the workflow and
[AGENTS.md](AGENTS.md) for AI-assisted contributions.

## Contributing

Contributions are welcome — see [CONTRIBUTING.md](CONTRIBUTING.md). Report
security issues per [SECURITY.md](SECURITY.md); community expectations are in
[CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md).

## License

MIT — see [LICENSE](LICENSE).

## Links

- **Repository:** [github.com/optophi/pycharter](https://github.com/optophi/pycharter)
- **Issues:** [GitHub Issues](https://github.com/optophi/pycharter/issues)
- **PyPI:** [pypi.org/project/pycharter](https://pypi.org/project/pycharter/)
