Metadata-Version: 2.5
Name: ragbt
Version: 0.1.0
Summary: dbt for RAG — declarative, incremental, tested embedding pipelines
Project-URL: Homepage, https://github.com/shivamim/ragbt
Project-URL: Repository, https://github.com/shivamim/ragbt
Project-URL: Issues, https://github.com/shivamim/ragbt/issues
Author: Shivam Shukla
License: Apache-2.0
License-File: LICENSE
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Requires-Python: >=3.10
Requires-Dist: asyncpg>=0.29
Requires-Dist: httpx>=0.27
Requires-Dist: pgvector>=0.2
Requires-Dist: pydantic>=2.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: rich>=13.0
Requires-Dist: sqlalchemy[asyncio]>=2.0
Requires-Dist: tenacity>=8.0
Requires-Dist: typer>=0.9
Provides-Extra: dev
Requires-Dist: build>=1.0; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Requires-Dist: twine>=5.0; extra == 'dev'
Description-Content-Type: text/markdown

<div align="center">

# 🧱 ragbt

> **"dbt for RAG"** — declarative, incremental, tested embedding pipelines

[![PyPI](https://img.shields.io/pypi/v/ragbt.svg)](https://pypi.org/project/ragbt/)
[![Python](https://img.shields.io/pypi/pyversions/ragbt.svg)](https://pypi.org/project/ragbt/)
[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)

</div>

---

## The Problem

Building RAG pipelines today means writing bespoke scripts that:

- **Re-embed everything on every run** — slow and expensive
- **Have no tests** — you only find out your retrieval is broken in production
- **Are impossible to diff or review** in PRs

**ragbt** brings the [dbt](https://www.getdbt.com/) workflow — declarative configs, incremental builds, tests, and lineage — to embedding pipelines.

---

## ✨ Features

| Feature | What it means |
|---------|---------------|
| 🔄 **Incremental runs** | Content-hash based; only changed docs get re-embedded |
| 🧪 **Retrieval tests** | Assert that queries return the right docs before shipping |
| 📊 **Lineage & manifest** | Lineage from source → chunks → vectors |
| 🎯 **Multi-target** | Isolated `dev` / `prod` environments |
| 🔌 **Pluggable** | Swappable chunkers, embedders, and vector stores |
| ⚡ **Fast** | HNSW index on pgvector for sub-second search |

---

## 🚀 Quick Start

### 1. Install

```bash
pip install ragbt
```

Requires **Python 3.10+** and a Postgres database with the [`pgvector`](https://github.com/pgvector/pgvector) extension.

### 2. Set environment variables

```bash
export RAGBT_DATABASE_URL="postgresql+asyncpg://user:pass@host:5432/db"
export VOYAGE_API_KEY="your-api-key"
# or OPENAI_API_KEY="your-key"
```

### 3. Scaffold a project

```bash
ragbt init my_rag_project
cd my_rag_project
```

### 4. Add your docs

Drop Markdown or text files into `data/product_docs/`.

### 5. Run incrementally

```bash
ragbt run
```

Only changed documents are re-chunked and re-embedded. Subsequent runs are near-instant.

### 6. Test retrieval quality

```bash
ragbt test
```

Fails CI if your expected docs don't surface for key queries.

### 7. Generate lineage

```bash
ragbt docs generate
ragbt ls
```

---

## 📁 Project Structure

```
my_rag_project/
├── ragbt_project.yml          # Project metadata
├── sources.yml                # Where raw content lives
├── models/
│   └── product_docs.yml       # Chunking + embedding strategy
├── tests/
│   └── retrieval_tests.yml    # Quality assertions
├── data/
│   └── product_docs/          # Your raw docs
└── manifest.json              # Generated lineage graph
```

---

## 🛠️ Configuration

### `sources.yml` — define content sources

```yaml
sources:
  - name: product_docs
    type: directory
    path: ./data/product_docs
    file_types: [md, txt]
```

### `models/product_docs.yml` — declare chunking & embedding

```yaml
models:
  - name: product_docs_chunks
    source: product_docs
    description: "Customer-facing product documentation"

    chunking:
      strategy: recursive
      chunk_size: 512
      chunk_overlap: 50

    embedding:
      provider: voyage
      model: voyage-3
      dimensions: 1024
      batch_size: 100

    materialization: incremental
    index:
      backend: pgvector
      table: product_docs_vectors
      distance: cosine
```

### `tests/retrieval_tests.yml` — gate your pipeline

```yaml
tests:
  - name: refund_policy_findable
    model: product_docs_chunks
    query: "what is your refund policy"
    expect:
      top_k: 5
      must_contain_doc: "refund-policy.md"
      min_score: 0.75

  - name: no_orphaned_chunks
    model: product_docs_chunks
    type: integrity
```

---

## 🖥️ CLI Reference

| Command | Description |
|---------|-------------|
| `ragbt init [name]` | Scaffold a new project |
| `ragbt run` | Incremental run (default) |
| `ragbt run --full-refresh` | Force re-process everything |
| `ragbt run --target prod` | Run against the prod index |
| `ragbt test` | Run retrieval assertions |
| `ragbt docs generate` | Build `manifest.json` + lineage graph |
| `ragbt ls` | List models and last-run status |

---

## ✅ Supported in v0.1

| Component | Supported |
|-----------|-----------|
| **Sources** | `directory` (local files) |
| **Chunkers** | `recursive`, `markdown` |
| **Embedders** | `voyage`, `openai` |
| **Vector Stores** | `pgvector` (Postgres + pgvector) |
| **Distance Metrics** | `cosine`, `euclidean`, `dot` |
| **Tests** | `retrieval`, `integrity`, `freshness` |

---

## 🗺️ Roadmap

| Feature | Status | Target |
|---------|--------|--------|
| `semantic` chunker | Planned | v0.2 |
| `cohere` embedder | Planned | v0.2 |
| `pinecone` / `weaviate` vector stores | Planned | v0.2 |
| `ragbt plan` — dry-run preview | Planned | v0.2 |
| `ragbt diff` — changed files report | Planned | v0.2 |
| DAG with `ref()` between models | Planned | v0.3 |
| Retrieval metrics (MRR, HitRate) | Planned | v0.3 |
| Automatic RAGBT PR checks | Planned | v0.3 |
| GitHub Actions CI | ✓ Available | v0.1 |
| RAGBT Cloud dashboard | Future | v1.0 |

---

## 🏗️ Architecture

```
sources.yml ──▶ config loader (Pydantic)
                      │
                      ▼
              ┌───────────────┐
              │  state store   │  (Postgres: doc hashes, run history)
              └───────┬───────┘
                      │ diff: what changed?
                      ▼
              ┌───────────────┐
              │  chunker       │  recursive | markdown
              └───────┬───────┘
                      ▼
              ┌───────────────┐
              │  embedder      │  voyage | openai
              └───────┬───────┘
                      ▼
              ┌───────────────┐
              │  vector store  │  pgvector
              └───────┬───────┘
                      ▼
              ┌───────────────┐
              │  test runner   │  golden query → expected chunk assertions
              └───────┬───────┘
                      ▼
              ┌───────────────┐
              │  manifest/docs │  lineage graph, freshness, run history
              └───────────────┘
```

### Core Concepts vs dbt

| dbt concept | ragbt equivalent |
|---|---|
| `sources.yml` | `sources.yml` — where raw content lives |
| `models/*.sql` | `models/*.yml` — declared chunking + embedding strategy |
| Incremental models | Content-hash based incremental runs |
| `dbt test` | `ragbt test` — retrieval assertions |
| `dbt docs generate` | `ragbt docs generate` — lineage graph |
| `target: dev` / `target: prod` | `--target` flag — separate indexes |
| `manifest.json` | `manifest.json` — lineage metadata + run state |
| Jinja macros | Python plugin functions for custom chunkers/embedders |

---

## 🧪 Development

```bash
git clone https://github.com/shivamim/ragbt.git
cd ragbt
pip install -e ".[dev]"
pytest
```

---

## 🐳 Docker

```bash
docker-compose up -d postgres
ragbt init demo && cd demo
ragbt run
ragbt test
```

---

## 📜 License

Apache License 2.0 — see [LICENSE](LICENSE).

---

<div align="center">

Built with ❤️ by [Shivam Shukla](https://github.com/shivamim)

</div>
