Metadata-Version: 2.5
Name: mcp-trove-crunchtools
Version: 1.0.1
Summary: Self-hosted local file indexing MCP server with semantic search
Author: crunchtools.com
License-Expression: AGPL-3.0-or-later
License-File: LICENSE
Keywords: embeddings,mcp,rag,semantic-search,sqlite
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: GNU Affero General Public License v3 or later (AGPLv3+)
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Python: >=3.11
Requires-Dist: av>=14
Requires-Dist: defusedxml>=0.7
Requires-Dist: fastembed>=0.4
Requires-Dist: fastmcp>=2.0
Requires-Dist: olefile>=0.47
Requires-Dist: openpyxl>=3.1
Requires-Dist: pydantic>=2.0
Requires-Dist: pymupdf4llm<2,>=0.0.17
Requires-Dist: python-docx>=1.0
Requires-Dist: python-pptx>=1.0
Requires-Dist: sqlite-vec>=0.1
Requires-Dist: striprtf>=0.0.26
Requires-Dist: xlrd>=2.0
Provides-Extra: dev
Requires-Dist: mypy>=1.13; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.24; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.8; extra == 'dev'
Provides-Extra: vision
Requires-Dist: google-genai>=1.0; extra == 'vision'
Requires-Dist: openai>=1.0; extra == 'vision'
Provides-Extra: vision-gemini
Requires-Dist: google-genai>=1.0; extra == 'vision-gemini'
Provides-Extra: vision-openai
Requires-Dist: openai>=1.0; extra == 'vision-openai'
Description-Content-Type: text/markdown

# mcp-trove-crunchtools

Self-hosted local file indexing MCP server with semantic search. Index any local directory (pCloud ~/AutoSync/, rclone mounts, ~/Documents/, anything) and search over the contents using hybrid vector + keyword search.

## Features

- **Hybrid search** — Combines semantic vector similarity with FTS5 keyword matching
- **Content categories** — images and videos (captioned), PDF, text, documents (DOCX, DOC, ODT, RTF), spreadsheets (XLSX, XLS, ODS, CSV) and presentations (PPTX, ODP) by default; web pages, data files and source code on request. Every other file can still be found by its name
- **Local-first** — No cloud services, no per-seat fees, your data stays on your machine
- **Lightweight embeddings** — Uses fastembed (ONNX runtime) instead of PyTorch (~22MB vs ~2GB)
- **Incremental indexing** — SHA-256 checksum-based change detection
- **Background mode** — `--index` CLI mode for systemd timer automation

## Install

### uvx (recommended)

```bash
uvx mcp-trove-crunchtools
```

### pip

```bash
pip install mcp-trove-crunchtools
```

### Container

```bash
podman run -v trove-data:/data -v ~/Documents:/docs:ro quay.io/crunchtools/mcp-trove
```

## Claude Code Integration

```bash
claude mcp add mcp-trove-crunchtools -- uvx mcp-trove-crunchtools
```

## Tools (10)

### Search (2)

| Tool | Description |
|------|-------------|
| `trove_search` | Hybrid search: vector similarity, keywords (bm25) and folder/file names, fused by rank. One result per file, its best chunk, with `score`, `sources` and `matches`. `path` scopes it to a path prefix. |
| `trove_similar` | Find files similar to a given indexed file using its average embedding. |

### Index Management (3)

| Tool | Description |
|------|-------------|
| `trove_index` | Index a specific file or directory. Skips unchanged files (checksum-based). |
| `trove_reindex` | Force re-index ignoring checksums. If no path given, reindexes everything. |
| `trove_remove` | Remove a file or directory from the index. |

### Status (5)

| Tool | Description |
|------|-------------|
| `trove_status` | Index statistics: total files, chunks, disk usage, model info. |
| `trove_list` | List indexed files with metadata (size, type, chunk count). |
| `trove_get_chunks` | Show the text chunks for a specific indexed file. |
| `trove_log` | Activity log of indexing runs: start/finish times, file counts, errors. Persisted in SQLite, survives container crashes. |
| `trove_quality` | Per-file error tracking from indexing runs: which files failed, why, and whether a later index resolved it. |

## Environment Variables

| Variable | Default | Description |
|----------|---------|-------------|
| `TROVE_DB` | `~/.local/share/mcp-trove/trove.db` | SQLite database path |
| `TROVE_PATHS` | (none) | Directories to index in background mode, separated by `:` (`;` on Windows) |
| `TROVE_INBOX_PATHS` | (none) | Index paths that empty out by design (an upload inbox), separated like `TROVE_PATHS`; the shrink and removal guards skip them |
| `TROVE_INDEX_WORKERS` | `20` | Images and videos extracted at once. PDFs run one at a time and text files two, whatever this is set to |
| `TROVE_MANIFEST_DIR` | (none) | Directory of per-path file listings the host writes before a run (see Background Indexing); unset, each run walks its trees |
| `TROVE_EMBED_THREADS` | `2` | ONNX threads for embedding |
| `TROVE_EMBEDDING_MODEL` | `BAAI/bge-small-en-v1.5` | fastembed model name |
| `TROVE_EXCLUDE_PATTERNS` | `*.iso,*.zip,...` | Comma-separated glob patterns, matched against file names and the names of the directories above them (`Old Backup` skips that whole tree). Entries already indexed that a pattern now matches are removed on the next run |
| `TROVE_INDEX_TYPES` | `images,videos,pdf,text,documents,spreadsheets,presentations` | Content categories to read and index. Videos and audio are captioned from samples (10 frames and a 20 s sound clip, in one vision request), never sent whole, with their duration, recording time, GPS and tags indexed alongside. Also available: `audio` (sends a 20 s clip of each file's sound to the vision provider, so opt-in), `web` (html), `data` (json, yaml, xml, ...), `code` (source, config and log files). An unknown name stops the server at startup. Spreadsheets are indexed by the first 200 rows of each sheet, presentations by slide text and notes |
| `TROVE_INDEX_LOCATION` | `false` | Index where photos and videos were taken, from EXIF and container GPS. Anyone who can search the index can read it, so it is opt-in |
| `TROVE_INDEX_OTHER` | `name` | Every file outside those categories: `name` indexes its name, folder and extension without reading it, so it can still be found; `skip` leaves it out. Files over 50 MB in a category also get the name entry. Switching a category on or off re-reads or removes its files on the next run |
| `TROVE_CHUNK_SIZE` | `1000` | Characters per text chunk |
| `TROVE_CHUNK_OVERLAP` | `200` | Overlap between chunks |
| `TROVE_PDF_MAX_PAGES` | `5` | Pages indexed from the start of each PDF, enough to tell what the document is about while keeping memory flat on large manuals; `0` or negative indexes every page |
| `TROVE_MAX_TEXT_CHARS` | `5000000` | Characters of extracted text kept per file; the rest is dropped with a warning so one huge document cannot flood a batch with chunks; must be positive, other values fall back to the default with a warning |
| `TROVE_VISION_MAX_CALLS_PER_RUN` | `3000` | Vision API calls one index run may make; images and videos past it are deferred to the next run, not failed. `0` means no limit |
| `TROVE_VISION_PDF_SCANS` | `false` | Caption the first page of scanned PDFs through the vision backend. Sends an image of the page to the provider, so it is opt-in |
| `TROVE_VISION_FALLBACK_MODEL` | (none) | OpenRouter only: a second model tried when the first is rate limited. Unset by default, since the usual fallback costs more |
| `TROVE_REQUIRE_MOUNT` | (none) | Mount point the index paths live on. `--index` checks it is mounted and answers a listing within 30 s before indexing, and fails the run otherwise |
| `TROVE_MAX_EXTRACT_TIMEOUTS` | `5` | Timed-out file extractions after which a run aborts as a hung source |
| `TROVE_BACKUP_DIR` | `<db dir>/backups` | Where `--index` keeps a daily `VACUUM INTO` copy of the database |
| `TROVE_BACKUP_KEEP` | `3` | Daily database copies kept; `0` disables backups |
| `TROVE_STATUS_FILE` | `<db dir>/last_run.json` | Summary of the last `--index` run, for monitoring |
| `TROVE_STATUS_FILE_MODE` | `600` | Octal file mode of the summary. Use `644` when a monitor reads it as another user; it can carry error text that names files |
| `TROVE_VISION_BACKEND` | `none` | Vision captioning backend: `gemini`, `openai`, `openrouter`, `ollama`, or `none` to disable |
| `TROVE_VISION_MODEL` | per backend | Vision model. Defaults to `gemini-2.5-flash`, `gpt-4o-mini`, `google/gemini-3.1-flash-lite` (openrouter) or `llava` depending on the backend; empty when disabled |
| `TROVE_VISION_PROMPT` | (see below) | Caption prompt sent with each image |
| `TROVE_VISION_TIMEOUT` | `120` | Seconds allowed per vision API call |

Image and video captioning is off until `TROVE_VISION_BACKEND` is set. The
`openrouter` backend reads `OPENROUTER_API_KEY_FILE` or `OPENROUTER_API_KEY`,
handles video as well as images, and only routes to zero-retention providers. The
default caption prompt is:

> Generate a concise caption for this image suitable for search indexing.
> Include: what you see, location if identifiable, time of day, any readable
> text or signs.

## Background Indexing

Set up a systemd timer to keep your index fresh:

```bash
TROVE_PATHS=~/Documents:~/AutoSync mcp-trove-crunchtools --index
```

On a network-backed source, walking the tree is the slow part: one listing per
directory and one stat per file, each a round trip. Point `TROVE_MANIFEST_DIR`
at a directory holding one rclone `lsjson` listing per index path, at that path
under the directory with `.lsjson` appended, and the scan reads it instead. For
the index path `/pcloud/Documents/Academic`:

```bash
mkdir -p /manifests/pcloud/Documents
rclone lsjson -R --fast-list --files-only --no-mimetype pcloud:Documents/Academic \
  > /manifests/pcloud/Documents/Academic.lsjson
```

A listing over 6 hours old, missing or unreadable falls back to the walk. With a
manifest, files it no longer lists are removed from the index, unless a tree of
100 or more files would lose over 10% at once, which aborts the path instead.

An `--index` run is built to be left alone:

- Each file is stored in one transaction, so a run killed part way (OOM, reboot) leaves no half-indexed file behind.
- Before indexing, it runs `PRAGMA quick_check`, keeps a daily backup, marks runs abandoned for more than 12 hours as `aborted`, and probes `TROVE_REQUIRE_MOUNT`.
- A path aborts instead of indexing when it lists fewer than half the files indexed under it (an unmounted source looks like deleted files), or when extractions keep timing out (a hung source). `trove_reindex` on the path overrides the file-count check; paths in `TROVE_INBOX_PATHS` are exempt.
- The exit code is `0` only when every path completed with no more than 10% of its files failing. Any other outcome exits `1`, so the systemd unit fails visibly. `TROVE_STATUS_FILE` records what happened, path by path.
- The database records its embedding model and refuses to open under a different one, because vectors from two models cannot be searched together.
- Vision captions are cached by file content (SHA-256; a fingerprint of size and three 1 MB spans for video and audio) and vision model, so a renamed or copied photo reuses its caption. A cached caption is kept while its content is indexed and for 90 days after. A file that failed permanently, or 5 times transiently, is not retried until it changes; `trove_reindex` on its path retries it.
- With `TROVE_VISION_PDF_SCANS=true`, a PDF whose first 5 pages hold under 20 characters of text (a scan) has its first page captioned through the vision backend, under the same budget and cache. It is off by default because it sends an image of the page to the vision provider.
- `trove_reindex` with no path re-reads every file, so it requires `force=true`. Cached captions are reused unless `TROVE_VISION_MODEL` changed.

## Operator commands

| Command | What it does |
|---------|--------------|
| `mcp-trove-crunchtools --index` | Index `TROVE_PATHS` once and exit, for a systemd timer |
| `mcp-trove-crunchtools --reembed` | Rebuild every vector under `TROVE_EMBEDDING_MODEL` from the stored chunks: resumable, swapped in one transaction, nothing re-read |
| `mcp-trove-crunchtools --eval queries.json` | Recall@5 and MRR over `[{"query": ..., "expect": [path fragment, ...]}]` |

[docs/runbook.md](docs/runbook.md) covers monitoring states, restoring a backup, changing
the embedding model and everyday changes.

## Stability

From 1.0.0 these are a stable interface under [SemVer](https://semver.org/):
the MCP tool names, their parameters and result fields; the command-line
flags above; every `TROVE_*` variable in the table; and the database, whose
schema changes only through numbered migrations that run on open
(`PRAGMA user_version`). Removing or renaming any of them, or a migration
that cannot be applied in place, is a major release. Adding is a minor one.

## License

AGPL-3.0-or-later
