Metadata-Version: 2.4
Name: trustpipe
Version: 0.1.0
Summary: AI Data Supply Chain Trust & Provenance Platform
Project-URL: Homepage, https://github.com/pallavi-chandrashekar/trustpipe
Project-URL: Repository, https://github.com/pallavi-chandrashekar/trustpipe
Project-URL: Issues, https://github.com/pallavi-chandrashekar/trustpipe/issues
Author: Pallavi Chandrashekar
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: compliance,data-engineering,data-lineage,data-provenance,data-quality,data-trust,eu-ai-act,mlops
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries
Requires-Python: >=3.10
Requires-Dist: click>=8.0
Requires-Dist: jinja2>=3.0
Requires-Dist: pydantic>=2.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: rich>=13.0
Provides-Extra: airflow
Requires-Dist: apache-airflow>=2.7; extra == 'airflow'
Provides-Extra: all
Requires-Dist: anthropic>=0.30; extra == 'all'
Requires-Dist: dash>=2.14; extra == 'all'
Requires-Dist: evidently>=0.4; extra == 'all'
Requires-Dist: fastapi>=0.100; extra == 'all'
Requires-Dist: numpy>=1.24; extra == 'all'
Requires-Dist: openai>=1.0; extra == 'all'
Requires-Dist: pandas>=2.0; extra == 'all'
Requires-Dist: plotly>=5.0; extra == 'all'
Requires-Dist: pyod>=1.0; extra == 'all'
Requires-Dist: scikit-learn>=1.3; extra == 'all'
Requires-Dist: uvicorn>=0.23; extra == 'all'
Provides-Extra: api
Requires-Dist: fastapi>=0.100; extra == 'api'
Requires-Dist: numpy>=1.24; extra == 'api'
Requires-Dist: pandas>=2.0; extra == 'api'
Requires-Dist: uvicorn>=0.23; extra == 'api'
Provides-Extra: dashboard
Requires-Dist: dash>=2.14; extra == 'dashboard'
Requires-Dist: numpy>=1.24; extra == 'dashboard'
Requires-Dist: pandas>=2.0; extra == 'dashboard'
Requires-Dist: plotly>=5.0; extra == 'dashboard'
Provides-Extra: dbt
Requires-Dist: dbt-core>=1.6; extra == 'dbt'
Provides-Extra: dev
Requires-Dist: anthropic>=0.30; extra == 'dev'
Requires-Dist: dash>=2.14; extra == 'dev'
Requires-Dist: evidently>=0.4; extra == 'dev'
Requires-Dist: fastapi>=0.100; extra == 'dev'
Requires-Dist: httpx>=0.24; extra == 'dev'
Requires-Dist: mypy>=1.5; extra == 'dev'
Requires-Dist: numpy>=1.24; extra == 'dev'
Requires-Dist: openai>=1.0; extra == 'dev'
Requires-Dist: pandas>=2.0; extra == 'dev'
Requires-Dist: plotly>=5.0; extra == 'dev'
Requires-Dist: pyod>=1.0; extra == 'dev'
Requires-Dist: pytest-cov>=4.0; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Requires-Dist: ruff>=0.1; extra == 'dev'
Requires-Dist: scikit-learn>=1.3; extra == 'dev'
Requires-Dist: uvicorn>=0.23; extra == 'dev'
Provides-Extra: drift
Requires-Dist: evidently>=0.4; extra == 'drift'
Requires-Dist: numpy>=1.24; extra == 'drift'
Requires-Dist: pandas>=2.0; extra == 'drift'
Provides-Extra: kafka
Requires-Dist: confluent-kafka>=2.0; extra == 'kafka'
Provides-Extra: llm
Requires-Dist: anthropic>=0.30; extra == 'llm'
Requires-Dist: openai>=1.0; extra == 'llm'
Provides-Extra: pandas
Requires-Dist: numpy>=1.24; extra == 'pandas'
Requires-Dist: pandas>=2.0; extra == 'pandas'
Provides-Extra: poisoning
Requires-Dist: numpy>=1.24; extra == 'poisoning'
Requires-Dist: pandas>=2.0; extra == 'poisoning'
Requires-Dist: pyod>=1.0; extra == 'poisoning'
Requires-Dist: scikit-learn>=1.3; extra == 'poisoning'
Provides-Extra: postgres
Requires-Dist: psycopg[binary]>=3.0; extra == 'postgres'
Provides-Extra: s3
Requires-Dist: boto3>=1.28; extra == 's3'
Provides-Extra: spark
Requires-Dist: numpy>=1.24; extra == 'spark'
Requires-Dist: pandas>=2.0; extra == 'spark'
Requires-Dist: pyspark>=3.4; extra == 'spark'
Provides-Extra: trust
Requires-Dist: numpy>=1.24; extra == 'trust'
Requires-Dist: pandas>=2.0; extra == 'trust'
Description-Content-Type: text/markdown

# TrustPipe

**AI Data Supply Chain Trust & Provenance Platform**

Track data provenance, score trust, detect poisoning, and generate compliance reports — inside your existing data pipelines.

[![Tests](https://github.com/pallavi-chandrashekar/trustpipe/actions/workflows/test.yml/badge.svg)](https://github.com/pallavi-chandrashekar/trustpipe/actions)
[![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
[![License: Apache 2.0](https://img.shields.io/badge/License-Apache%202.0-green.svg)](LICENSE)

## The 3-Line Promise

```python
from trustpipe import TrustPipe          # 1. Import
tp = TrustPipe()                          # 2. Initialize (zero-config)
tp.track(df, name="customers")            # 3. Track
```

That's it. Provenance recorded. Merkle chain extended. Trust scored. Query anytime.

## Why TrustPipe?

| Problem | Impact |
|---------|--------|
| **EU AI Act Article 10** enforcement begins August 2026 | Few orgs have data provenance docs |
| **87% of AI projects fail** — primary cause: data quality | $406M avg annual loss per org |
| **Data poisoning attacks** now target RAG, fine-tuning, MCP tools | Most orgs have zero detection |

No infrastructure exists to solve all three *inside* the data pipeline. TrustPipe does.

## Three Independent Layers

| Layer | What It Does | Works Alone? |
|-------|-------------|:------------:|
| **Provenance** | Track where data came from (Merkle-backed, tamper-evident) | ✓ |
| **Trust** | Score data quality and safety (0-100, six dimensions) | ✓ |
| **Compliance** | Generate regulatory documents (EU AI Act, data cards) | ✓ |

## Quick Start

```bash
pip install trustpipe
```

### Track Data Provenance

```python
from trustpipe import TrustPipe

tp = TrustPipe()

# Track a raw dataset
raw = tp.track(df_raw, name="raw_customers", source="s3://bucket/raw/")

# Track a transformation with parent linkage
clean = tp.track(df_clean, name="clean_customers", parent=raw.id)

# View the full chain
for record in tp.trace("clean_customers"):
    print(f"{record.name} ← {record.source} ({record.row_count} rows)")
```

### Trust Scoring (0-100)

```python
score = tp.score(df, name="customers")
print(score.explain())
# Trust Score: 95/100 (Grade: A+)
#
#   Completeness           ████████████████████ 100.0 (w=0.20)
#   Consistency            ████████████████████ 100.0 (w=0.20)
#   Poisoning Risk         ████████████████████ 100.0 (w=0.15)
#   Freshness              ████████████████████ 100.0 (w=0.15)
#   Drift                  ████████████████░░░░  80.0 (w=0.15)
#   Provenance Depth       ██████████████░░░░░░  70.0 (w=0.15)
```

Six dimensions: **Provenance Depth**, **Freshness**, **Completeness**, **Consistency**, **Drift**, **Poisoning Risk**.

### Poisoning Scan

```python
result = tp.scan(df)
# 0 anomalies / 10000 rows (zscore fallback)
# With PyOD installed: Isolation Forest detection
```

### Drift Detection

```python
score = tp.score(new_data, name="features", reference=baseline_data)
# Drift dimension drops when distributions shift
```

### EU AI Act Compliance Report

```python
report = tp.comply("training_set", regulation="eu-ai-act-article-10")
# Generates full Article 10 report with gap analysis:
# - Data sources & lineage (Art. 10(2)(a))
# - Quality metrics (Art. 10(2)(b))
# - Bias assessment (Art. 10(2)(f))
# - 10-point compliance gap analysis with recommendations
```

Also supports `datacard` and `audit-log` report types.

### Pandas Auto-Tracking

```python
tp.pandas()  # activate once

df = pd.read_csv("data.csv")          # auto-tracked
df.to_parquet("output.parquet")        # auto-tracked
```

### Airflow Integration

```python
from trustpipe.plugins.airflow_plugin import trustpipe_task

@trustpipe_task(tp, name="etl_output", inputs=["raw_data"])
@task
def transform(data):
    return processed_data  # auto-tracked with parent linkage
```

### dbt Integration

```python
from trustpipe.plugins.dbt_plugin import DbtPlugin

dbt = DbtPlugin(tp)
dbt.import_manifest("target/manifest.json")    # imports full lineage
dbt.import_run_results("target/run_results.json")  # tracks execution
```

### REST API

```bash
trustpipe serve --port 8000
# POST /track, GET /trace/{name}, GET /score/{name}
# GET /comply/{name}, GET /verify, GET /export
# Auto-generated docs at /docs
```

### Web Dashboard

```bash
trustpipe dashboard --port 8050
# Overview: trust score gauges per dataset
# Records: searchable/sortable provenance table
# Compliance: gap summary across all datasets
```

### CI/CD Trust Gate

```bash
# Fail CI if trust score < threshold
trustpipe gate training_set --threshold 70
# Exit 0 = PASS, Exit 1 = FAIL
```

GitHub Actions workflow included — see `.github/workflows/trust-gate.yml`.

### Multi-Project Federation

```python
from trustpipe.core.federation import Federation

fed = Federation([prod_tp, staging_tp, ml_tp])
fed.status()                    # unified status across all projects
fed.search("customers")         # find dataset across all projects
fed.verify_all()                # integrity check everywhere
```

### Webhook / Slack Alerts

```python
from trustpipe.alerts.webhook import AlertManager, SlackAlert

alerts = AlertManager()
alerts.add(SlackAlert(webhook_url="https://hooks.slack.com/..."))
alerts.check_score("training_set", score, threshold=70)   # alerts on drop
alerts.check_integrity(verify_result)                       # alerts on failure
```

## CLI Commands

| Command | Purpose |
|---------|---------|
| `trustpipe init` | Initialize project |
| `trustpipe trace <dataset>` | Show provenance chain (tree/table/json) |
| `trustpipe verify` | Verify Merkle chain integrity |
| `trustpipe status` | Project summary |
| `trustpipe score <dataset>` | Trust score (0-100) with dimension breakdown |
| `trustpipe scan <file>` | Poisoning / anomaly scan |
| `trustpipe comply <dataset>` | Compliance report (EU AI Act, datacard, audit-log) |
| `trustpipe export` | Export provenance data (JSON/CSV) |
| `trustpipe gate <dataset>` | CI/CD trust gate (exit 1 if below threshold) |
| `trustpipe dashboard` | Launch web dashboard |
| `trustpipe serve` | Launch REST API server |

## Installation

```bash
# Core only (provenance + CLI, 5 lightweight deps)
pip install trustpipe

# With trust scoring (adds pandas, numpy)
pip install trustpipe[trust]

# With drift detection
pip install trustpipe[drift]

# With poisoning detection
pip install trustpipe[poisoning]

# Plugins
pip install trustpipe[spark]      # PySpark integration
pip install trustpipe[api]        # FastAPI REST server
pip install trustpipe[dashboard]  # Plotly Dash web UI

# Everything
pip install trustpipe[all]

# Development
pip install trustpipe[dev]
```

## Storage Backends

| Backend | Use Case | Config |
|---------|----------|--------|
| **SQLite** (default) | Local dev, single user | Zero-config, `~/.trustpipe/` |
| **PostgreSQL** | Team collaboration | `pip install trustpipe[postgres]` |
| **S3** | Enterprise scale | `pip install trustpipe[s3]` |

## Architecture

```
Your Pipeline (Spark / Airflow / Pandas / dbt / Kafka)
        │
        ▼
┌──────────────────────────────────────────┐
│            TrustPipe SDK                 │
│                                          │
│  Layer 1: Provenance                     │
│  ┌────────────────────────────────────┐  │
│  │ Merkle chain · Lineage DAG · Tags  │  │
│  └────────────────────────────────────┘  │
│                                          │
│  Layer 2: Trust                          │
│  ┌────────────────────────────────────┐  │
│  │ 6 dimensions · Drift · Poisoning   │  │
│  └────────────────────────────────────┘  │
│                                          │
│  Layer 3: Compliance                     │
│  ┌────────────────────────────────────┐  │
│  │ EU AI Act · Data Cards · Audits    │  │
│  └────────────────────────────────────┘  │
│                                          │
│  Plugins: Pandas·Spark·Airflow·dbt·Kafka │
│  Storage: SQLite · PostgreSQL · S3       │
│  Alerts:  Webhook · Slack                │
│  Serve:   REST API · Web Dashboard       │
└──────────────────────────────────────────┘
```

## Design Principles

1. **Zero-config start** — works out of the box with SQLite
2. **3-line integration** — no rewriting existing pipelines
3. **Data fingerprinting only** — NEVER stores your raw data
4. **Not blockchain** — Merkle hash tree (same as git), zero overhead
5. **LLM-enhanced, not LLM-dependent** — core works fully offline
6. **Graceful degradation** — missing optional deps return neutral scores, never crash
7. **Pluggable everything** — storage, scoring, compliance templates, alert destinations

## Testing

```bash
make test          # full suite (118 tests)
make test-quick    # unit tests only
make lint          # ruff check
```

## License

Apache 2.0 — open source, enterprise-friendly.
