Metadata-Version: 2.4
Name: fastref
Version: 0.1.0
Summary: Fast offline documentation reference desk and skill generator for AI agents
Keywords: ai-agents,documentation,offline-docs,sqlite-fts5,skills,bm25
Author: Rafael Darder
Author-email: Rafael Darder <darder@gmail.com>
License-Expression: MIT
Requires-Dist: litert-lm-api>=0.18.0
Requires-Python: >=3.12
Description-Content-Type: text/markdown

# fastref 📚⚡

> Fast offline documentation reference desk and skill generator for AI coding agents.

`fastref` gives AI coding agents an instant, local offline reference desk. It indexes documentation from any tool, framework, or site into a high-performance **SQLite FTS5 database with BM25 ranking**, and automatically generates self-contained **Agent Skills** (under `~/.agents/skills/` or `./.agents/skills/`).

Coding agents can query authoritative documentation and code examples locally in sub-milliseconds without hallucinating APIs, wasting web search latency, or blowing context windows.

---

## The Ingestion Pattern

Instead of fragile ad-hoc scrapers, `fastref` unifies documentation ingestion around two core patterns:

1. **Harvest & Assemble (`fastref harvest`)**: For tools and websites that publish an `llms.txt` file (or website root). `fastref` spiders and crawls linked markdown documentation, resolves relative references, checks external domain boundaries, and bundles pages into clean, structured documentation chunks.
2. **Fetch & Index Structured JSON (`fastref add`)**: For tools and documentation systems that generate machine-readable search indexes (such as Fuse.js, Algolia, or MkDocs-style search JSON endpoints). `fastref` ingests both flat arrays and nested section trees directly.

---

## Features

- **Dual Ingestion Engine**:
  - **`llms.txt` Spider & Harvester**: Discovers and indexes documentation hierarchies from `llms.txt` files or site roots, featuring domain boundary controls, crawl depth limits, and link preview (`--dry-run`).
  - **Structured JSON Ingestor**: Automatically normalizes both flat lists and nested hierarchical search indexes into a unified schema.
- **Hybrid Search (BM25 + Dense Vectors)**:
  - **Lexical BM25**: Native SQLite FTS5 for exact keywords, function names, and prefix matching.
  - **On-Device Semantic Vectors**: Google's **LiteRT-LM** with **EmbeddingGemma 2 Text-270M** (quantized to 157 MB, 256d normalized Matryoshka embeddings) for conceptual retrieval without external APIs or background daemons.
  - **Reciprocal Rank Fusion (RRF)**: Merges lexical and semantic rankings for optimal recall and precision.
- **Zero-Daemon Architecture**: In-process inference starts in ~180ms, eliminating background sockets, daemons, and orphaned processes.
- **Prompt Injection & SSRF Hardened**: Defends against indirect prompt injection by stripping invisible unicode/HTML comments, sanitizing skill frontmatter, blocking private/metadata network IPs, and framing retrieved docs in untrusted reference envelopes.
- **Automatic Agent Skill Generation**: Creates clean `SKILL.md` definitions with security boundary notices and automatically bridges them into active agent harnesses (`~/.agents/skills/`, `~/.gemini/config/skills/`, `~/.claude/skills/`, `~/.config/goose/skills/`).
- **Harness-Agnostic**: Compatible with Antigravity, Claude Code, Cursor, Codex, Goose, Aider, and any agent that supports shell commands or standard skill directories.
- **XDG-Compliant Caching**: Keeps database artifacts neatly organized in `$XDG_CACHE_HOME/fastref` (`~/.cache/fastref`).

---

## Installation

```bash
# Using pip
pip install fastref

# Or with uv
uv tool install fastref
```

---

## Quick Start

### 1. Ingest Documentation & Generate a Skill

#### Pattern A: Harvest from `llms.txt` or Site Root

```bash
# Harvest from an llms.txt index
fastref harvest mytool https://example.com/llms.txt

# Or point directly to the website root (auto-detects /llms.txt)
fastref harvest mytool https://example.com

# Preview discovered links and external domains without fetching
fastref harvest mytool https://example.com/llms.txt --dry-run
```

#### Pattern B: Ingest Pre-Built Structured Search JSON

```bash
# Index structured search index JSON directly from a URL or local file
fastref add mytool https://example.com/search_index.json
fastref add mytool ./path/to/search_index.json
```

By default, both commands:
1. Compile an SQLite FTS5 database to `~/.cache/fastref/<name>.db`.
2. Archive the raw manifest/index to `~/.agents/skills/<name>/resources/raw_index.json`.
3. Write `~/.agents/skills/<name>/SKILL.md` with trigger rules for AI agents.
4. Auto-bridge the skill into installed agent environments (`~/.gemini/config/skills/`, `~/.claude/skills/`, `~/.config/goose/skills/`).

### 2. Search Documentation

`fastref search` provides three adaptive profiles:
- **`find`** (default when vectors are enabled): Returns the top focused code & API snippets (default limit: 3) without catalog clutter.
- **`directory`** (default when vectors are disabled): Returns a catalog of matching topic titles (capped at 20) plus top snippets.
- **`lucky`**: Prints the top matching document in full, followed by 5 related topic links.

```bash
# Focused search (find profile: top snippets, hybrid RRF when vectors enabled)
fastref search mytool "authentication"
fastref search mytool "middleware" --limit 5

# Directory profile: view matching topic catalog + top snippets
fastref search mytool "jwt_validation" --directory

# Lucky profile: print full content of top match + related topic links
fastref search mytool "installation" --lucky

# Show full content of all matches directly
fastref search mytool "config" --full

# Retrieve full documentation by exact URL or topic ID
fastref get mytool "/docs/auth#jwt"
fastref get mytool "/docs/users" "/docs/roles"
```

### 3. Optional: Enable On-Device Vector & Hybrid Search

Vector features are disabled by default to keep the CLI lightweight and zero-noise. When enabled, `fastref` uses Google's **LiteRT-LM** and downloads the **EmbeddingGemma 2 Text-270M** model (157 MB).

```bash
# Enable vector features (downloads the model artifact on first run)
fastref vector enable

# Backfill vector embeddings for an already indexed library
fastref vector index mytool

# Check vector status and indexed libraries
fastref vector status

# When vectors are enabled, fastref search automatically defaults to hybrid RRF:
fastref search mytool "how to handle webhooks asynchronously"

# Or query explicitly via semantic or lexical mode:
fastref search mytool "reactive state management" --semantic
fastref search mytool "on_mount_hook" --lexical

# Disable vector features at any time
fastref vector disable
```

### 4. Explore Documentation Structure

```bash
# View table of contents chapters and topic counts
fastref toc mytool

# Filter chapters matching a keyword
fastref toc mytool auth
```

### 5. Manage Indexed Documentation

```bash
# List all indexed tools, chunk counts, vector status, and database sizes
fastref list

# Remove an indexed tool (both cache database and skill folder)
fastref remove mytool
```


---

## How Agents Use `fastref`

When `fastref` generates a skill, it writes a clean `SKILL.md` into your agent's skill directory. When you ask your agent to implement a feature or debug code using an indexed tool:

1. **Discovery**: The agent runs `fastref search <name> "<query>"` to locate exact components, function signatures, or concepts.
2. **Context-Efficient Retrieval**: The agent inspects targeted snippets or pulls complete topic documentation via `--lucky` or `fastref get <name> <url>`, without loading megabytes of irrelevant HTML into context.
3. **Accurate Implementation**: The agent codes against authoritative local documentation and tested code examples.

---

## CLI Reference

### `fastref harvest <name> <source>`
Harvest documentation from an `llms.txt` file or website root:
- `<name>`: Identifier for the tool or library.
- `<source>`: URL or local path to `llms.txt` (or website root URL).
- `--max-docs <N>`: Maximum documents to spider and index (default: 60).
- `--max-depth <D>`: Maximum spider crawl depth (default: 2).
- `--dry-run`: Preview discovered links and domain categorization without fetching or indexing.
- `--allow-urls <prefixes>`: Comma-separated URL or domain prefixes to allow (e.g. `https://raw.githubusercontent.com/...`).
- `--allow-all-urls`: Allow fetching documentation from any external URL or domain.
- `--allow-remote`: Allow network fetching when source is a local file.
- `--allow-private-ips`: Allow fetching from private, loopback, or local network IPs (e.g. localhost during dev).
- `-p`, `--project`: Install the skill into the local project directory (`./.agents/skills/`) instead of global user skills.
- `--skills-dir <dir>`: Custom destination directory for the generated skill.
- `--cache-dir <dir>`: Custom destination directory for the SQLite database.
- `--description <desc>`: Custom description for the YAML frontmatter in `SKILL.md`.
- `--no-skill`: Only build the SQLite index without generating an agent skill.

### `fastref add <name> <source>`
Index documentation from a pre-built machine-readable search index JSON:
- `<name>`: Identifier for the tool or library.
- `<source>`: HTTP/HTTPS URL or local path to `search_index.json`.
- `--allow-private-ips`: Allow fetching from private, loopback, or local network IPs.
- `-p`, `--project`: Install the skill into the local project directory (`./.agents/skills/`) instead of global user skills.
- `--skills-dir <dir>`: Custom destination directory for the generated skill.
- `--cache-dir <dir>`: Custom destination directory for the SQLite database.
- `--description <desc>`: Custom description for the YAML frontmatter in `SKILL.md`.
- `--no-skill`: Only build the SQLite index without generating an agent skill.

### `fastref search <name> <query>`
Search documentation with BM25 relevance ranking:
- `<name>`: Identifier for the indexed tool.
- `<query>`: Search query (supports SQLite FTS5 match expressions).
- `--lucky`: Print the entire top match immediately and list remaining candidate snippets.
- `--full`: Include full topic content for all matches directly.
- `--limit <N>`: Maximum results to return (default: 8).

### `fastref get <name> <url1> [url2 ...]`
Fetch and display the full documentation text for one or more documents by URL or topic ID.

### `fastref toc <name> [filter]`
Display Table of Contents chapters, topic counts, and sample titles, optionally filtered by keyword.

### `fastref list`
List all indexed tools, their chunk counts, and database sizes.

### `fastref remove <name>`
Remove an indexed tool's SQLite database cache and associated skill directories.

---

## License

MIT License.
