Metadata-Version: 2.4
Name: anchorx
Version: 0.1.1
Summary: Grounding layer for AI agents: verifiable, dated, cited knowledge acquisition.
Project-URL: Homepage, https://github.com/syedawais355/anchorx
Project-URL: Repository, https://github.com/syedawais355/anchorx
Project-URL: Issues, https://github.com/syedawais355/anchorx/issues
Author: Syed Muhammad Awais Gillani
License: MIT
License-File: LICENSE
Keywords: ai-agents,claude,grounding,mcp,provenance,rag,retrieval
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing :: Indexing
Requires-Python: >=3.12
Requires-Dist: ebooklib>=0.18
Requires-Dist: fastmcp>=3.0
Requires-Dist: httpx>=0.28
Requires-Dist: lancedb>=0.34
Requires-Dist: model2vec>=0.3
Requires-Dist: numpy>=2.0
Requires-Dist: openpyxl>=3.1
Requires-Dist: pillow>=10.0
Requires-Dist: playwright>=1.40
Requires-Dist: pyarrow>=25.0
Requires-Dist: pypdfium2>=4.0
Requires-Dist: python-docx>=1.1
Requires-Dist: python-pptx>=0.6
Requires-Dist: trafilatura>=2.0
Requires-Dist: yt-dlp>=2024.0
Provides-Extra: asr
Requires-Dist: faster-whisper>=1.0; extra == 'asr'
Provides-Extra: clip
Requires-Dist: open-clip-torch>=2.20; extra == 'clip'
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: reportlab>=4.0; extra == 'dev'
Provides-Extra: diarization
Requires-Dist: pyannote-audio>=3.1; extra == 'diarization'
Provides-Extra: full
Requires-Dist: faster-whisper>=1.0; extra == 'full'
Requires-Dist: open-clip-torch>=2.20; extra == 'full'
Requires-Dist: playwright-stealth>=1.0; extra == 'full'
Requires-Dist: pyannote-audio>=3.1; extra == 'full'
Requires-Dist: pytesseract>=0.3; extra == 'full'
Requires-Dist: sentence-transformers>=3.0; extra == 'full'
Provides-Extra: ocr
Requires-Dist: pytesseract>=0.3; extra == 'ocr'
Provides-Extra: rerank
Requires-Dist: sentence-transformers>=3.0; extra == 'rerank'
Provides-Extra: stealth
Requires-Dist: playwright-stealth>=1.0; extra == 'stealth'
Description-Content-Type: text/markdown

<div align="center">

# ⚓ AnchorX

### The grounding layer for AI agents — verifiable, dated, cited knowledge instead of compressed memory.

[![PyPI](https://img.shields.io/pypi/v/anchorx?color=3775A9&logo=pypi&logoColor=white)](https://pypi.org/project/anchorx/)
[![Python](https://img.shields.io/badge/python-3.12%2B-3776AB?logo=python&logoColor=white)](https://pypi.org/project/anchorx/)
[![Tests](https://github.com/syedawais355/AnchorX/actions/workflows/tests.yml/badge.svg)](https://github.com/syedawais355/AnchorX/actions/workflows/tests.yml)
[![License](https://img.shields.io/badge/license-MIT-blue)](LICENSE)
[![MCP](https://img.shields.io/badge/interface-MCP%20%7C%20CLI-orange)](#3-usage)

**One command in Claude Code:** `claude mcp add anchor -- uvx anchorx`

</div>

---

## 1. The Problem

Most "AI + data" tools are **fetchers**. Give them a URL, they hand back a string, and then they forget everything. That string has:

- ❌ **no date** — you can't tell if it's current or three years stale
- ❌ **no provenance** — you can't open the exact spot it came from
- ❌ **no memory** — the next question starts from zero
- ❌ **no dedup or conflict detection** — ten copies of the same claim look like ten facts
- ❌ **no sense of absence** — the model can't tell you what it *doesn't* have, so it fills the gap by hallucinating

```mermaid
flowchart LR
    U["URL / PDF / video"] --> F["🪣 Fetcher"]
    F --> S["raw string<br/>no date · no source · no memory"]
    S --> L["🤖 LLM"]
    L --> A["answer you cannot verify"]
    style F fill:#3a1a1a,stroke:#a33,color:#fff
    style A fill:#3a1a1a,stroke:#a33,color:#fff
```

When the answer is wrong, you have **no way to check why**.

---

## 2. The Solution

**AnchorX is the layer _above_ the fetchers.** Every piece of knowledge it acquires is normalized into a single schema and stored in a local, embedded corpus — carrying a **resolvable anchor** back to a human-openable source, a publication date, a quality score, and freshness metadata.

```mermaid
flowchart LR
    U["URL / PDF / video / image"] --> A["⚓ AnchorX"]
    A --> D[("Document<br/>+ anchors<br/>+ provenance<br/>+ quality")]
    D --> C[("Local corpus<br/>SQLite + LanceDB")]
    C --> L["🤖 LLM (Claude)"]
    L --> R["answer + [20:47] · p.14 · ¶14<br/>citations you can click open"]
    style A fill:#0d2818,stroke:#2ea44f,color:#fff
    style R fill:#0d2818,stroke:#2ea44f,color:#fff
```

> **Success criterion:** not *"the AI stops being wrong."* It's *"when the AI is wrong, one click shows you exactly why."*

### Architecture at a glance — read it bottom-up

The durable value lives at **L0/L1**; the adapters at **L2** are replaceable parts.

```
┌─────────────────────────────────────────────────────────────────────┐
│  L4  INTERFACE      MCP server  ·  CLI  ·  Python SDK                │
│      ask · search · ingest · research · coverage · fetch · clear     │
└────────────────────────────────┬────────────────────────────────────┘
┌────────────────────────────────▼────────────────────────────────────┐
│  L3  ORCHESTRATION  Budget Governor · Job Queue · Research · Sync    │
└────────────────────────────────┬────────────────────────────────────┘
        ┌────────────────────────┴───────────────────────┐
┌───────▼──────────────────────────┐   ┌─────────────────▼────────────┐
│  L2  ACQUISITION (adapters)       │   │  L2' RETRIEVAL               │
│  youtube · web_deep · web_search  │   │  pre-filter → BM25 + dense   │
│  documents · images               │   │  → RRF → cross-encoder rerank│
└───────────────┬───────────────────┘   └─────────────────┬────────────┘
┌───────────────▼─────────────────────────────────────────▼────────────┐
│  L1  NORMALIZATION   RawPayload → Document → Chunks → Vectors         │
│      provenance stamping · anchor assignment · quality scoring        │
└────────────────────────────────┬────────────────────────────────────┘
┌────────────────────────────────▼────────────────────────────────────┐
│  L0  CORPUS   SQLite (metadata · FTS5 · graph) · LanceDB (vectors)   │
│      blob cache · dedup · freshness · coverage · grounding           │
└─────────────────────────────────────────────────────────────────────┘
```

---

## 3. Usage

### Install in Claude Code (30 seconds)

```bash
claude mcp add anchor -- uvx anchorx
```

No clone, no virtualenv, no Ollama. On first launch AnchorX auto-installs its browser, downloads a tiny CPU embedding model, creates its data directory, and serves **8 tools** to Claude:

`anchor_ingest` · `anchor_ask` · `anchor_search` · `anchor_research` · `anchor_coverage` · `anchor_fetch` · `anchor_refresh` · `anchor_clear`

Then just talk to Claude:

> *"Ingest this PDF and these two articles into Anchor, then answer my question with citations."*
> *"What's my coverage on 'RAG'? Any gaps?"*
> *"Using only that video, what are the first words spoken?"*

### Or use the CLI

```bash
pip install anchorx

anchor ingest https://youtu.be/VIDEO_ID  "D:/reports/invoice.pdf"  https://example.com/post
anchor ask "what is the total on the invoice"        # grounded, cited, offline
anchor ask "how does chunking work" --url youtu.be   # scope to one source
anchor research "latest techniques for scraping SPAs"# discover → acquire → answer
anchor coverage "retrieval augmented generation"     # what do I have, what's missing
anchor clear --all                                   # wipe the corpus
```

### Or the Python SDK

```python
from anchor_ai.adapters.web_deep import acquire, render_page
from anchor_ai.corpus import DocumentStore, coverage

doc = acquire("https://example.com/post", render=render_page)   # L0→L2 ladder
print(doc.title, doc.quality_score, [s.heading for s in doc.sections])

report = coverage(store.conn, "chunking")
print(report.gaps)   # ["No source newer than 2025-12 …", "62% from one domain …"]
```

---

## 4. Why AnchorX

Four principles drive every decision in the design — and each one is a capability the fetcher-based ecosystem doesn't have.

| Principle | What it means | How AnchorX delivers it |
|---|---|---|
| **Provenance is the product** | An answer is only useful if you can open the source and check it | Every chunk carries a resolvable anchor; citations deep-link to `[20:47]`, `p.14`, `¶14` |
| **Pre-filter, not post-filter** | *"Only videos over 50k views after March 2026"* must be enforced **inside** retrieval | Metadata filters push down into **both** BM25 and the vector scan before ranking |
| **Absence is information** | The model must know the scope of what it has | `coverage()` is a first-class tool that reports **gaps**, not a debug endpoint |
| **Adapters are commodity** | Fetching is a solved, high-maintenance problem | Thin adapters wrap proven libraries; swap one in an afternoon, nothing else changes |

**AnchorX vs. a plain fetcher / vanilla RAG:**

| | Fetcher / vanilla RAG | ⚓ AnchorX |
|---|:---:|:---:|
| Clickable, resolvable citations | ✗ | ✅ |
| Publication dates + freshness / staleness | ✗ | ✅ |
| Metadata pre-filter *inside* retrieval | ✗ | ✅ |
| Coverage report + gap detection | ✗ | ✅ |
| Near-duplicate + corroboration graph | ✗ | ✅ |
| Cross-session persistent memory | ✗ | ✅ |
| Flags what it *couldn't* ground | ✗ | ✅ |

---

## 5. Technical Details

### 5.1 Ingestion — from a URL to a grounded, chunked `Document`

```mermaid
flowchart LR
    U["URL / file"] --> R{Router}
    R -->|youtube| Y[YouTube adapter]
    R -->|.pdf/.docx/.csv| DOC[Documents adapter]
    R -->|image| IMG[Images adapter]
    R -->|web| W[web_deep ladder]
    Y & DOC & IMG & W --> N["Normalize → Document<br/>provenance + anchors"]
    N --> C["Structure-aware<br/>chunking"]
    C --> E["Embed (model2vec, CPU)"]
    E --> S[("SQLite + LanceDB<br/>+ blob cache")]
```

### 5.2 Query — filter first, then fuse, then rerank, then ground

```mermaid
flowchart LR
    Q["Question"] --> F["Metadata<br/>pre-filter"]
    F --> B["BM25 (FTS5)"]
    F --> D["Dense (LanceDB)"]
    B --> X["RRF fusion<br/>k = 60"]
    D --> X
    X --> RR["Cross-encoder<br/>rerank → top 8"]
    RR --> A["Answer + citations<br/>+ grounding report"]
```

### 5.3 `research()` — the multi-round acquisition loop (capped at depth 3)

```mermaid
flowchart TD
    A[Question] --> B[Discover candidates]
    B --> C["Score: authority × recency × relevance / cost"]
    C --> D{Within budget?}
    D -->|no| G[Trim to top-N] --> E
    D -->|yes| E[Acquire → normalize → persist]
    E --> H[Retrieve + rerank]
    H --> I[Coverage check]
    I -->|gaps found, depth left| J[Refine queries] --> B
    I -->|sufficient| K[Answer + citations + grounding]
```

### 5.4 Adapters & their acquisition ladders

Adapters escalate through tiers **only when needed** — a static GET costs ~200 ms; a browser costs seconds.

#### 🌐 web_deep — static → embedded → headless → stealth

```mermaid
flowchart LR
    L0["L0 · httpx + trafilatura<br/>~200ms"] --> J{needs JS?<br/>or near-empty?}
    J -->|no| OUT["Document"]
    J -->|yes| L15["L1.5 · __NEXT_DATA__ / JSON-LD<br/>zero browser cost"]
    L15 -->|found| OUT
    L15 -->|no| L2["L2 · Playwright Chromium<br/>network-idle + auto-scroll"]
    L2 --> L3["L3 · stealth · proxy · cookies<br/>(opt-in)"]
    L3 --> OUT
```

#### ▶️ youtube — captions → ASR → diarization

| Tier | Method | Quality |
|---|---|---|
| **T0** | Manual captions (`yt-dlp`) | 0.95 |
| **T1** | Auto captions | 0.70 |
| **T2** | `faster-whisper` + VAD | 0.90 |
| **T3** | + `pyannote` speaker diarization | 0.92 |

Transcripts are cut into paragraphs on pauses, mapped onto **chapters as sections**, and every chunk is anchored to a timestamp — `[12:04]` resolves to `youtu.be/ID?t=724`.

#### 📄 documents — PDF ladder + office + data cards

```mermaid
flowchart LR
    P0["P0 · text-layer check<br/>chars/page"] --> Q{scanned?}
    Q -->|no| P1["P1 · pypdfium2 text<br/>page-anchored"]
    Q -->|yes| P3["P3 · OCR (optional)"]
    P1 & P3 --> DOCX["DOCX / PPTX / EPUB → Markdown"]
    P1 --> TAB["CSV / XLSX → Data Card<br/>(schema · ranges · head)"]
```

Spreadsheets are **never chunked as prose** — they become a queryable *Data Card* (columns, dtypes, null %, ranges, sample rows).

#### 🖼️ images — classify → route → VLM/OCR

```mermaid
flowchart LR
    I["image"] --> PH["pHash dedup"]
    PH --> CL{classify}
    CL -->|chart| VT["VLM → data table"]
    CL -->|diagram| VS["VLM → structure"]
    CL -->|screenshot| OV["OCR + VLM"]
    CL -->|photo| CAP["VLM caption"]
    VT & VS & OV & CAP --> TXT["content_md<br/>(text-retrievable)"]
```

Embedded images inside PDFs become **child documents** linked to their parent page — a chart on p.14 gets its own OCR/VLM description, retrievable by text query.

### 5.5 Anchors — what makes citation real

The difference between "RAG" and **verifiable RAG**:

| Source | Anchor payload | Renders as | Opens to |
|---|---|---|---|
| `youtube` | `{"t": 1247}` | `[20:47]` | `youtu.be/ID?t=1247` |
| `web` | `{"sel": "…", "para": 14}` | `¶14` | URL + scroll-to |
| `document` | `{"page": 14, "bbox": […]}` | `p.14` | PDF page + highlight |
| `image` | `{"region": […]}` | `region` | image + box overlay |
| `search_result` | `{"rank": 3}` | `SERP #3` | original URL |

### 5.6 Coverage & grounding — the differentiator

**`coverage(topic, filters)`** — what the corpus holds *and where it's thin*:

```json
{
  "n_documents": 47,
  "by_source": { "youtube": 12, "web": 28, "document": 5, "image": 2 },
  "date_histogram": { "2024": 31, "2025": 14, "2026": 2 },
  "domain_concentration": 0.62,
  "authority": { "primary": 4, "secondary": 31, "unknown": 12 },
  "gaps": [
    "No source newer than 2025-12 — topic likely evolved",
    "62% of coverage from a single domain — low independence",
    "No primary/official documentation present"
  ]
}
```

Every **`ask()`** returns a grounding report — including **`unsupported_spans`**, the sentences that could *not* be entailed by the retrieved evidence:

```json
{
  "grounding": {
    "chunks_used": 6, "sources_used": 4, "independent_domains": 3,
    "oldest_evidence": "2024-08-11", "newest_evidence": "2026-03-11",
    "unsupported_spans": ["Penguins architected the platform overnight."],
    "warnings": ["2 of 4 sources are stale (>180d)"]
  }
}
```

### 5.7 Quality scoring

One `quality_score ∈ [0,1]` per document — components stored separately so scores can be **re-weighted without re-ingesting**.

```
quality = 0.30 × acquisition_fidelity      # tier: T0=.95 T1=.70 T2=.90 · L0=.90 · P1=.95
        + 0.25 × extraction_completeness    # text ratio · table survival · no truncation
        + 0.20 × authority                  # primary source · domain · named author
        + 0.15 × recency_fit                # half-life decay vs. topic volatility
        + 0.10 × structure                  # headings · resolved sections · valid anchors
```

### 5.8 Project structure

```
anchor_ai/
├── core/          document · protocols · chunking · quality        (L1 keystone)
├── adapters/      youtube · web_deep · web_search · documents · images
├── corpus/        store · vectors · embeddings · retrieval · rerank
│                  blobs · dedup · freshness · coverage · grounding
│                  entailment · image_index                          (L0 durable value)
├── orchestration/ budget · queue · research · sync                  (L3)
├── interfaces/    cli · mcp_server                                  (L4)
└── bootstrap.py   first-run setup (browser + model)
```

### 5.9 Tech stack

| Concern | Choice | Why |
|---|---|---|
| Metadata store | **SQLite + FTS5** | one file, BM25 included, transactional |
| Vectors | **LanceDB** | embedded, native metadata pre-filtering |
| Embeddings | **model2vec** | static CPU embeddings — good semantics, no server, no GPU |
| Static extract | **trafilatura** | best boilerplate removal per CPU cycle |
| Browser | **Playwright** | superior waiting primitives + context pooling |
| YouTube | **yt-dlp** | the only thing that keeps working |
| PDF | **pypdfium2** | fast text layer, escalate only when needed |
| MCP | **fastmcp** | least ceremony |

### 5.10 Optional model backends

Everything above runs with **zero external services**. Heavier capabilities are opt-in extras, each with a graceful fallback:

```bash
pip install "anchorx[full]"     # everything below
pip install "anchorx[asr]"      # faster-whisper       — YouTube T2 ASR
pip install "anchorx[diarization]"  # pyannote.audio   — YouTube T3 speakers
pip install "anchorx[ocr]"      # pytesseract          — scanned docs / screenshots
pip install "anchorx[clip]"     # open-clip            — text ↔ image search
pip install "anchorx[stealth]"  # playwright-stealth   — web_deep L3
```

---

## 6. License

Released under the **MIT License** — see [`LICENSE`](LICENSE) for the full text.

```
MIT License · Copyright (c) 2026 Syed Muhammad Awais Gillani

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction … THE SOFTWARE IS PROVIDED "AS IS",
WITHOUT WARRANTY OF ANY KIND. See LICENSE for the complete text.
```

<div align="center">

**[📦 PyPI](https://pypi.org/project/anchorx/) · [🐙 GitHub](https://github.com/syedawais355/AnchorX) · [🐛 Issues](https://github.com/syedawais355/AnchorX/issues)**

_Built as a grounding layer for Claude Code and any MCP-compatible agent._

</div>
