Metadata-Version: 2.5
Name: simpleaudit-studio
Version: 0.3.0
Summary: Production AI model audit platform — one-liner demo mode via uvx
License: MIT
License-File: LICENSE
Requires-Python: >=3.11
Requires-Dist: cronsim==2.7
Requires-Dist: django==5.2.4
Requires-Dist: djangorestframework==3.16.0
Requires-Dist: drf-spectacular==0.28.0
Requires-Dist: hatchet-sdk==1.41.0
Requires-Dist: httpx>=0.27
Requires-Dist: pydantic<3,>=2.7
Requires-Dist: python-dotenv==1.1.1
Requires-Dist: requests==2.32.5
Requires-Dist: sentry-sdk[django]>=2.0
Requires-Dist: simpleaudit==0.2.1
Requires-Dist: workos==10.4.0
Provides-Extra: dev
Requires-Dist: factory-boy==3.3.2; extra == 'dev'
Requires-Dist: ruff; extra == 'dev'
Provides-Extra: e2e
Requires-Dist: playwright>=1.45; extra == 'e2e'
Provides-Extra: postgres
Requires-Dist: psycopg[binary]==3.2.13; extra == 'postgres'
Provides-Extra: server
Requires-Dist: gunicorn==23.0.0; extra == 'server'
Description-Content-Type: text/markdown

---
title: SimpleAudit Studio
emoji: 🔍
colorFrom: blue
colorTo: indigo
sdk: docker
app_port: 7860
pinned: false
---

# SimpleAudit Studio
<a href="https://sushantgautam-simpleaudit-studio.hf.space" target="_blank" rel="noopener noreferrer">
  <img alt="SimpleAudit Studio — a platform for reproducible AI model audits" src="https://github.com/user-attachments/assets/d9e0105a-9e2c-4455-a855-ad5854fd604f" />
</a>

[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT)
[![Python](https://img.shields.io/badge/Python-3.11%2B-blue.svg)](https://www.python.org/)
[![PyPI](https://img.shields.io/pypi/v/simpleaudit-studio)](https://pypi.org/project/simpleaudit-studio/)

A self-hostable platform for running reproducible AI model audits using the [SimpleAudit](https://github.com/kelkalot/simpleaudit) engine (Target → Auditor → Judge). Every audit captures frozen, versioned inputs so historical results stay interpretable years later.

## 🚀 Quick Start

```bash
uvx simpleaudit-studio
```

[`uvx`](https://docs.astral.sh/uv/#uvx) installs the [`simpleaudit-studio`](https://pypi.org/project/simpleaudit-studio/) package and runs it in an isolated environment. No Docker, no Postgres, no manual setup. Opens at http://localhost:8000 (login: `studio` / `admin123`). A mock model server is pre-seeded so you can start exploring audit results immediately.

🌐 Or skip the setup entirely — try the live demo: <a href="https://sushantgautam-simpleaudit-studio.hf.space" target="_blank" rel="noopener noreferrer">sushantgautam-simpleaudit-studio.hf.space</a>

Want the **latest unreleased code** from this repo instead of the PyPI release? Same one-liner, pointed at git:

```bash
uvx --from "git+https://github.com/SushantGautam/SimpleAuditStudio" spin
```


### 🤖 Using Real Models

The pre-seeded connections point at a built-in mock server. To run real audits, update your model connections on the **Models** page to any OpenAI-compatible endpoint. A few common options:

| Provider | Base URL | Example |
|----------|----------|---------|
| Ollama (local) | `http://localhost:11434/v1` | Qwen 3, Llama 3.3, Mistral |
| vLLM | `http://your-vllm-server:8000/v1` | Any HF model |
| OpenAI | `https://api.openai.com/v1` | GPT-4o, o3 |
| Together AI | `https://api.together.xyz/v1` | Many open models |
| Groq | `https://api.groq.com/openai/v1` | Fast inference |

Set the API key in the connection form (leave blank for local servers that don't require auth), then start a run from **New Experiment**.

## 🛠️ Local Development

For contributors working on the codebase, use [uv](https://docs.astral.sh/uv/) — no manual venv steps, no Makefile. Two verbs cover everything:

- **`uv sync`** — create/update `.venv` from `pyproject.toml` + `uv.lock`
- **`uv run <cmd>`** — run any command inside `.venv` (auto-syncs first if the lockfile changed)

### Setup (one time)

```bash
git clone https://github.com/SushantGautam/SimpleAuditStudio
cd SimpleAuditStudio
cp .env.local.example .env          # minimal local config (SQLite + mock models)

uv sync --extra dev --extra postgres   # create .venv + install all deps
uv run manage.py setup_local           # migrate + create admin (studio / BOOTSTRAP_PASSWORD from .env) + seed data
```

`setup_local` chains the three idempotent first-run steps (migrate → bootstrap admin → seed scenario packs & model connections) into one command. Re-run it any time — it skips what already exists.

### Daily work

```bash
uv run manage.py dev_server --embedded   # easiest: web UI + API + audit worker, zero-Docker → http://localhost:8000
```

Other options:
```bash
uv run manage.py runserver               # web UI + API only (no worker)
uv run manage.py dev_server              # web UI + worker, uses .env (edit POSTGRES_*/HATCHET_* to point at your infra)
SIMPLEAUDIT_LOCAL_SQLITE=1 uv run manage.py test infra   # tests
```

## 🐳 Self-Hosting

For teams or multi-user setups, use Docker Compose:

```bash
git clone https://github.com/SushantGautam/SimpleAuditStudio
cd SimpleAuditStudio
cp .env.example .env
# edit POSTGRES_PASSWORD and BOOTSTRAP_PASSWORD at minimum
docker compose up -d
```

Services: Web UI (:8000), PostgreSQL, Hatchet queue (:8888), Worker. Optional profile: `--profile mock` (mock model API).

See [docs/deployment.md](docs/deployment.md) for production hardening, backups, and upgrades.

## ✨ What You Can Do

- Build versioned scenario sets and register OpenAI-compatible models
- Run experiments across models, scenario versions and settings, with a review step before launch
- Watch runs live, including per-repetition results
- Monitor models for drift on a schedule (interval or cron)
- Filter, customise and compare runs on an interactive dashboard

## 🏗️ Architecture

```
Browser → Django (gunicorn :8000 in Compose; runserver :7860 in the HF Space / uvx demo)
              ↓
         PostgreSQL 16 (domain + queue DBs)
              ↓
         Hatchet Server (durable job queue)
              ↓
         Worker (CPU pool) → SimpleAudit Engine
              Target → Auditor → Judge
```

## 📚 Documentation

- [Architecture](docs/architecture.md) — system design, service boundaries, deployment topology
- [Domain Model](docs/domain-model.md) — scenarios, scenario sets, audit runs, immutability invariants
- [Deployment](docs/deployment.md) — Docker Compose, production hardening, upgrades

## 🤝 Contributing

Architecture decisions are documented in [docs/architecture.md](docs/architecture.md).
