Metadata-Version: 2.4
Name: anchorx
Version: 0.1.0
Summary: Grounding layer for AI agents: verifiable, dated, cited knowledge acquisition.
Project-URL: Homepage, https://github.com/syedawais355/anchorx
Project-URL: Repository, https://github.com/syedawais355/anchorx
Project-URL: Issues, https://github.com/syedawais355/anchorx/issues
Author: Syed Muhammad Awais Gillani
License: MIT
License-File: LICENSE
Keywords: ai-agents,claude,grounding,mcp,provenance,rag,retrieval
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing :: Indexing
Requires-Python: >=3.12
Requires-Dist: ebooklib>=0.18
Requires-Dist: fastmcp>=3.0
Requires-Dist: httpx>=0.28
Requires-Dist: lancedb>=0.34
Requires-Dist: model2vec>=0.3
Requires-Dist: numpy>=2.0
Requires-Dist: openpyxl>=3.1
Requires-Dist: pillow>=10.0
Requires-Dist: playwright>=1.40
Requires-Dist: pyarrow>=25.0
Requires-Dist: pypdfium2>=4.0
Requires-Dist: python-docx>=1.1
Requires-Dist: python-pptx>=0.6
Requires-Dist: trafilatura>=2.0
Requires-Dist: yt-dlp>=2024.0
Provides-Extra: asr
Requires-Dist: faster-whisper>=1.0; extra == 'asr'
Provides-Extra: clip
Requires-Dist: open-clip-torch>=2.20; extra == 'clip'
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: reportlab>=4.0; extra == 'dev'
Provides-Extra: diarization
Requires-Dist: pyannote-audio>=3.1; extra == 'diarization'
Provides-Extra: full
Requires-Dist: faster-whisper>=1.0; extra == 'full'
Requires-Dist: open-clip-torch>=2.20; extra == 'full'
Requires-Dist: playwright-stealth>=1.0; extra == 'full'
Requires-Dist: pyannote-audio>=3.1; extra == 'full'
Requires-Dist: pytesseract>=0.3; extra == 'full'
Requires-Dist: sentence-transformers>=3.0; extra == 'full'
Provides-Extra: ocr
Requires-Dist: pytesseract>=0.3; extra == 'ocr'
Provides-Extra: rerank
Requires-Dist: sentence-transformers>=3.0; extra == 'rerank'
Provides-Extra: stealth
Requires-Dist: playwright-stealth>=1.0; extra == 'stealth'
Description-Content-Type: text/markdown

<div align="center">

# ⚓ Anchor AI

**A grounding layer for AI agents — verifiable, dated, cited knowledge instead of compressed memory.**

![Python](https://img.shields.io/badge/python-3.12%2B-3776AB?logo=python&logoColor=white)
![Tests](https://github.com/syedawais355/AnchorX/actions/workflows/tests.yml/badge.svg)
![License](https://img.shields.io/badge/license-MIT-blue)
![SQLite](https://img.shields.io/badge/store-SQLite%20%2B%20LanceDB-003B57?logo=sqlite&logoColor=white)
![MCP](https://img.shields.io/badge/interface-MCP%20%7C%20CLI-orange)

</div>

---

Most "AI + data" tools are **fetchers**: give them a URL, they hand back a string, then forget everything. The string has no date, no provenance, no dedup, no memory, and no way to verify a claim afterwards.

**Anchor is the layer _above_ the fetchers.** Every piece of knowledge it acquires carries a resolvable **anchor** back to a human-openable source (`t=1247`, `p.14`, `¶14`, `bbox`), a publication date, a quality score, and freshness metadata. It can tell an agent not just what it knows, but **what it doesn't** — and when an answer is wrong, one click shows you exactly why.

> **Success criterion:** not "the AI stops being wrong." It's _"when the AI is wrong, one click shows you exactly why."_

---

## ⚡ Install in Claude Code (30 seconds)

Once published to PyPI, add Anchor as an MCP server with one command:

```bash
claude mcp add anchor -- uvx --from anchorx anchor-mcp
```

That's it — no clone, no venv, no Ollama. On first launch Anchor auto-installs its browser, downloads a tiny CPU embedding model, and starts serving these tools to Claude:

`anchor_ingest` · `anchor_ask` · `anchor_search` · `anchor_research` · `anchor_coverage` · `anchor_fetch` · `anchor_refresh` · `anchor_clear`

Then just talk to Claude:

> "Ingest this PDF and these 2 articles into Anchor, then answer my question with citations."

**Or from source** (until the PyPI release):

```bash
git clone https://github.com/syedawais355/anchorx.git && cd anchorx
pip install -e .
python -m playwright install chromium
claude mcp add anchor -- "<path>/.venv/Scripts/anchor-mcp"
```

---

## ✨ Why Anchor

| Principle | What it means | Consequence in the design |
|---|---|---|
| **Provenance is the product** | An answer is only useful if you can open the source and check it | Every chunk carries a resolvable anchor; citations deep-link to the exact spot |
| **Pre-filter, not post-filter** | "Only videos over 50k views after March 2026" must be enforced *inside* retrieval | Metadata filters push down into **both** BM25 and the vector scan |
| **Absence is information** | The model must know the scope of what it has | `coverage()` is a first-class tool that reports gaps, not a debug endpoint |
| **Adapters are commodity** | Fetching is a solved, high-maintenance problem | Thin adapters wrap existing libraries; swap one in an afternoon, nothing else changes |

---

## 🏗️ Architecture

Read it **bottom-up** — the durable value lives at L0/L1; the adapters at L2 are replaceable parts.

```
┌─────────────────────────────────────────────────────────────────────┐
│  L4  INTERFACE        MCP server  │  CLI  │  Python SDK              │
│      ask · search · ingest · fetch · coverage · refresh · research   │
└────────────────────────────────┬────────────────────────────────────┘
┌────────────────────────────────▼────────────────────────────────────┐
│  L3  ORCHESTRATION    Budget Governor · Job Queue · Research · Sync   │
└────────────────────────────────┬────────────────────────────────────┘
        ┌────────────────────────┴───────────────────────┐
┌───────▼──────────────────────────┐   ┌─────────────────▼────────────┐
│  L2  ACQUISITION (Adapters)       │   │  L2' RETRIEVAL               │
│  youtube · web_deep · web_search  │   │  pre-filter → BM25 + dense   │
│  documents · images               │   │  → RRF → cross-encoder rerank│
└───────────────┬───────────────────┘   └─────────────────┬────────────┘
┌───────────────▼─────────────────────────────────────────▼────────────┐
│  L1  NORMALIZATION    RawPayload → Document → Chunks → Vectors        │
│      provenance stamping · anchor assignment · quality scoring        │
└────────────────────────────────┬────────────────────────────────────┘
┌────────────────────────────────▼────────────────────────────────────┐
│  L0  CORPUS    SQLite (metadata · FTS5 · graph) · LanceDB (vectors)  │
│      blob cache · dedup · freshness · coverage · grounding           │
└─────────────────────────────────────────────────────────────────────┘
```

---

## 🔄 How it works

### Ingestion — from a URL to a grounded, chunked `Document`

```mermaid
flowchart LR
    U["URL / file"] --> R{Router}
    R --> L["Adapter ladder<br/>(escalate only when needed)"]
    L --> N["Normalize → Document<br/>provenance + anchors"]
    N --> C["Structure-aware<br/>chunking"]
    C --> E["Embed<br/>(bge-m3 / offline)"]
    E --> S[("SQLite + LanceDB<br/>+ blob cache")]
```

### Query — filter first, then fuse, then rerank, then ground

```mermaid
flowchart LR
    Q["Question"] --> F["Metadata<br/>pre-filter"]
    F --> B["BM25<br/>(FTS5)"]
    F --> D["Dense<br/>(LanceDB)"]
    B --> X["RRF fusion<br/>k=60"]
    D --> X
    X --> RR["Cross-encoder<br/>rerank → top 8"]
    RR --> A["Answer + citations<br/>+ grounding report"]
```

### `research()` — the multi-round acquisition loop (capped at depth 3)

```mermaid
flowchart TD
    A[Question] --> B[Discover candidates]
    B --> C[Score: authority × recency × relevance / cost]
    C --> D{Within budget?}
    D -->|no| G[Trim to top-N] --> E
    D -->|yes| E[Acquire → normalize → persist]
    E --> H[Retrieve + rerank]
    H --> I[Coverage check]
    I -->|gaps found, depth left| J[Refine queries] --> B
    I -->|sufficient| K[Answer + citations + grounding]
```

---

## ⚓ Anchors — what makes citation real

Every chunk resolves back to a human-openable location. This is the difference between "RAG" and **verifiable RAG**.

| Source | Anchor payload | Renders as | Opens to |
|---|---|---|---|
| `youtube` | `{"t": 1247}` | `[20:47]` | `youtu.be/ID?t=1247` |
| `web` | `{"sel": "…", "para": 14}` | `¶14` | URL + scroll-to |
| `document` | `{"page": 14, "bbox": […]}` | `p.14` | PDF page + highlight |
| `image` | `{"region": […]}` | `region` | image + box overlay |
| `search_result` | `{"rank": 3}` | `SERP #3` | original URL |

---

## 🔌 Adapters & acquisition ladders

Adapters escalate through tiers **only when needed** — a static GET costs ~200ms; a browser costs seconds.

<table>
<tr><th>Adapter</th><th>Ladder</th></tr>
<tr><td><b>web_deep</b></td><td>

| Tier | Method | Notes |
|---|---|---|
| L0 | `httpx` + `trafilatura` | static fetch (~200ms) |
| L1 | `needs_js()` classifier | detect SPA shells |
| L1.5 | `__NEXT_DATA__` / JSON-LD | zero-browser content |
| L2 | Playwright Chromium | JS render + stability wait |
| L3 | + stealth / proxy / cookies | opt-in only |

</td></tr>
<tr><td><b>youtube</b></td><td>

| Tier | Method | Quality |
|---|---|---|
| T0 | manual captions | 0.95 |
| T1 | auto captions | 0.70 |
| T2 | `faster-whisper` + VAD | 0.90 |
| T3 | + `pyannote` diarization | 0.92 |

</td></tr>
<tr><td><b>documents</b></td><td>

| Tier | Method |
|---|---|
| P0/P1 | `pypdfium2` text layer |
| P3 | OCR (injectable) |
| — | DOCX · PPTX · EPUB → Markdown |
| — | CSV · XLSX → **Data Card** (never chunked as prose) |

</td></tr>
<tr><td><b>images</b></td><td>

pHash dedup → classify → route → **VLM** (chart→table, diagram, caption) + **OCR**. Embedded images become child `Document`s linked to their parent page.

</td></tr>
<tr><td><b>web_search</b></td><td>

Discovery only (nothing persisted unless asked): **SearXNG · Brave · Exa**, with a snippet-sufficiency check.

</td></tr>
</table>

---

## 📊 Coverage & grounding — the differentiator

**`coverage(topic, filters)`** tells you what the corpus holds *and where it's thin*:

```json
{
  "n_documents": 47,
  "by_source": { "youtube": 12, "web": 28, "document": 5, "image": 2 },
  "date_histogram": { "2024": 31, "2025": 14, "2026": 2 },
  "domain_concentration": 0.62,
  "authority": { "primary": 4, "secondary": 31, "unknown": 12 },
  "gaps": [
    "No source newer than 2025-12 — topic likely evolved",
    "62% of coverage from a single domain — low independence",
    "No primary/official documentation present"
  ]
}
```

Every **`ask()`** returns a grounding report — including **`unsupported_spans`**, the sentences that could *not* be entailed by the retrieved evidence:

```json
{
  "grounding": {
    "chunks_used": 6, "sources_used": 4, "independent_domains": 3,
    "oldest_evidence": "2024-08-11", "newest_evidence": "2026-03-11",
    "unsupported_spans": ["Penguins architected the platform overnight."],
    "warnings": ["2 of 4 sources are stale (>180d)"]
  }
}
```

---

## 🎯 Quality scoring

A single `quality_score ∈ [0,1]` per document — components stored separately so scores can be **re-weighted without re-ingesting**.

```
quality = 0.30 × acquisition_fidelity      # tier: T0=.95 T1=.70 T2=.90 · L0=.90 · P1=.95
        + 0.25 × extraction_completeness    # text ratio · table survival · no truncation
        + 0.20 × authority                  # primary source · domain · named author
        + 0.15 × recency_fit                # half-life decay vs. topic volatility
        + 0.10 × structure                  # headings · resolved sections · valid anchors
```

---

## 📦 Installation

Requires **Python 3.12+**.

```bash
git clone https://github.com/syedawais355/anchorx.git
cd Anchor-AI

python -m venv .venv
.venv\Scripts\Activate.ps1        # Windows (PowerShell)
# source .venv/bin/activate       # macOS / Linux

pip install -e ".[dev]"
python -m playwright install chromium   # for web_deep L2/L3
```

**Optional model backends** (each has an offline fallback, so nothing here is required to run):

```bash
pip install -e ".[asr]"          # faster-whisper (YouTube T2)
pip install -e ".[diarization]"  # pyannote.audio (YouTube T3)
pip install -e ".[rerank]"       # sentence-transformers cross-encoder
pip install -e ".[clip]"         # open-clip text↔image search
pip install -e ".[ocr]"          # pytesseract
pip install -e ".[stealth]"      # playwright-stealth (web_deep L3)
```

---

## 🚀 Usage

### CLI

```bash
anchor ingest https://example.com/post https://youtu.be/VIDEO_ID
anchor ask "how does structure-aware chunking work?" --min-quality 0.6 --after 2026-01-01
anchor search "provenance"
```

`anchor ask` runs **entirely offline** against the corpus — fast, free, and network-free.

### MCP server

```bash
anchor-mcp     # exposes anchor_ask / coverage / search / ingest / fetch / refresh
```

### Python

```python
from anchor_ai.adapters.web_deep import acquire, render_page
from anchor_ai.corpus import DocumentStore, VectorStore, HashingEmbedder, coverage

doc = acquire("https://example.com/post", render=render_page)  # L0→L2 ladder
print(doc.title, doc.quality_score, [s.heading for s in doc.sections])

report = coverage(store.conn, "chunking")
print(report.gaps)
```

---

## 🗂️ Project structure

```
anchor_ai/
├── config.py                   # resolved storage paths
├── core/                       # L1 — the keystone
│   ├── document.py             # Document · Chunk · Section · Anchor · provenance
│   ├── protocols.py            # SourceAdapter · Filters · Candidate · CostEstimate
│   ├── chunking.py             # structure-aware splitter (never splits code/tables)
│   └── quality.py              # 5-part weighted scoring
├── adapters/                   # L2 — replaceable acquisition
│   ├── youtube/                # discover · acquire (captions) · asr · diarization · postprocess
│   ├── web_deep/               # static · detect · embedded · browser · extract (ladder)
│   ├── web_search/             # searxng · brave · exa
│   ├── documents/              # pdf · office (docx/pptx/epub) · tabular (data cards)
│   └── images/                 # phash · classify · vlm · ocr · asset linking
├── corpus/                     # L0 — durable value
│   ├── store.py                # SQLite + FTS5 + migrations + DocumentStore
│   ├── vectors.py              # LanceDB (native metadata pre-filter)
│   ├── embeddings.py           # Ollama + offline embedders
│   ├── retrieval.py            # filter → BM25 + dense → RRF → fuse
│   ├── rerank.py               # cross-encoder reranker
│   ├── blobs.py                # content-addressed raw-payload cache
│   ├── dedup.py                # MinHash near-duplicate + corroboration edges
│   ├── freshness.py            # per-source TTLs + refresh
│   ├── coverage.py             # corpus composition + gaps
│   ├── grounding.py            # citations + grounding report
│   ├── entailment.py           # unsupported-span detection
│   └── image_index.py          # CLIP text↔image index
├── orchestration/              # L3
│   ├── budget.py               # cost governor + graceful degradation
│   ├── queue.py                # resumable SQLite job queue
│   ├── research.py             # multi-round research loop
│   └── sync.py                 # incremental channel/site sync
├── interfaces/                 # L4
│   ├── cli.py                  # anchor CLI
│   └── mcp_server.py           # anchor-mcp server
└── tests/                      # 387 tests
```

---

## 🧰 Tech stack

| Concern | Choice | Why |
|---|---|---|
| Metadata store | **SQLite + FTS5** | one file, BM25 included, transactional |
| Vectors | **LanceDB** | embedded, native metadata pre-filtering |
| Static extract | **trafilatura** | best boilerplate removal per CPU cycle |
| Browser | **Playwright** | superior waiting primitives + context pooling |
| YouTube | **yt-dlp** | the only thing that keeps working |
| ASR | **faster-whisper** + VAD | 4× faster, word timestamps, kills hallucination |
| PDF | **pypdfium2** | fast text layer, escalate only when needed |
| MCP | **fastmcp** | least ceremony |

---

## 🧪 Testing

```bash
pytest            # 387 tests — every adapter, ladder, and pipeline
```

Heavy/optional backends are behind lazy imports with deterministic offline fallbacks, so the full suite runs green with no model servers — while real Chromium renders, PDF/office round-trips, and vector search are genuinely exercised.

---

## 🛣️ Build phases

| Phase | Scope | Status |
|---|---|---|
| **P1** | Schema · corpus · chunking · hybrid retrieval · `youtube` · `web_deep` · CLI | ✅ |
| **P2** | MCP server · pre-filters · `coverage()` · grounding · dedup · freshness | ✅ |
| **P3** | `documents` (PDF/office/data cards) · `web_search` · job queue · budget governor | ✅ |
| **P4** | `images` (VLM+OCR) · embedded-asset linking · `research()` · entailment | ✅ |
| **P5** | CLIP index · diarization · stealth render · incremental sync | ✅ |

---

## 📄 License

Released under the **MIT License** — see [`LICENSE`](LICENSE) for the full text.

```
MIT License

Copyright (c) 2026 Syed Muhammad Awais Gillani

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
```
