Metadata-Version: 2.4
Name: rubberduck-index
Version: 1.3.3
Summary: Local project indexer for RubberDuck Semantic Intelligence MCP
Author: RubberDuck Team
License-Expression: MIT
Project-URL: Homepage, https://rubberduck.com
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Requires-Dist: requests>=2.28
Requires-Dist: pathspec>=0.11
Requires-Dist: watchdog>=3.0

# rubberduck-index

Local project indexer for [RubberDuck Semantic Intelligence](https://rubberduck.com). Syncs your source code to the RubberDuck server for deep semantic analysis — enabling LLMs to query definitions, data flow, call chains, and more.

Works with **Python, JavaScript/TypeScript, Vue, C/C++, Go and Rust**. The indexer itself is language-agnostic: it uploads your source and the server decides what it can analyze, so new language support arrives without upgrading this package.

## How It Works

1. **Scan** — Finds source files, computes SHA-256 hashes, respects `.gitignore`
2. **Diff** — Sends hashes to server; server replies with which files need uploading
3. **Upload** — Sends only changed files (JSON for small batches, gzip tar for large)
4. **Analyze** — Server stores files in user-scoped directories and builds code analysis models
5. **Query** — LLMs use MCP tools (`analyze_code`, `trace_variable`, `call_chain`, etc.)

## Install

```bash
pip install rubberduck-index
```

The file watcher is included — there is no separate extra to install.

### Requirements

- Python 3.9+
- `requests` (HTTP client) — installed automatically
- `pathspec` (`.gitignore`-compatible pattern matching) — installed automatically
- `watchdog` (file watching) — installed automatically

## Quick Start

```bash
# 1. Initialize a project (first-time setup)
cd ~/my-project
rubberduck-index init
# Prompts for your auth token (one-time setup)
# Auto-detects project name from directory

# 2. Sync changes (incremental — only uploads what changed)
rubberduck-index sync

# 3. Watch for changes (auto-sync on file save)
rubberduck-index watch -d    # daemon mode (background)
rubberduck-index stop        # stop the daemon

# 4. Check status
rubberduck-index status      # local + server status
rubberduck-index list        # all your projects on server
```

## Commands

### `init`

Initialize a project directory for indexing. Creates `.rubberduck/config.json`, prompts for your token, scans files, and performs the first sync.

```bash
cd ~/my-project
rubberduck-index init [OPTIONS]
```

| Flag | Description |
|------|-------------|
| `--project` | Project name on the server (default: current directory name) |
| `--token` | Bearer token (skips interactive prompt) |
| `--server` | Override server URL (or set `RUBBERDUCK_SERVER`; default: `https://semantic.rubberduck.com`) |
| `--directory` | Project directory (default: current directory) |
| `--include` | File patterns to include (default: `**/*` — every language) |
| `--no-watch` | Skip starting the background watcher (it starts by default) |
| `--watch`, `-w` | Accepted for compatibility; the watcher already starts by default |

**Typical usage — no flags needed:**

```bash
cd ~/my-project
rubberduck-index init
# Enter your token: ****
# Scanning project... 42 file(s)
# Uploading... Synced 42 file(s), 42 analysis model(s) built
```

### `sync`

Sync changed files to the server. Compares local hashes with server manifest and uploads only what's different.

```bash
rubberduck-index sync [--force]
```

| Flag | Description |
|------|-------------|
| `--force` | Force full re-upload (ignore hash comparison) |

### `watch`

Watch for file changes and auto-sync. Uses OS-native file system events (FSEvents on macOS, inotify on Linux).

```bash
rubberduck-index watch [-d]
```

| Flag | Description |
|------|-------------|
| `--daemon`, `-d` | Run in background. Stop with `rubberduck-index stop`. |

Changes are debounced (default 500ms) and batched before uploading.

### `stop`

Stop the background watcher daemon.

```bash
rubberduck-index stop
```

### `status`

Show index status for the current project — local file count vs. server state.

```bash
rubberduck-index status
```

### `doctor`

Explain which files the indexer filtered out, and why. Compares what git tracks against
what the indexer keeps, then groups the difference by cause so that expected drops
(binaries, build output, lockfiles) are separated from anything worth investigating.

```bash
rubberduck-index doctor                      # grouped summary with samples
rubberduck-index doctor --all                # every filtered path
rubberduck-index doctor --directory ~/repo   # any checkout, no init required
rubberduck-index doctor --include '**/*.c' '**/*.h'   # test a narrower scope
```

`doctor` is strictly read-only: it needs no `.rubberduck/config.json`, writes nothing to
the tree and never contacts the server, so it can be pointed at a throwaway clone. Pass
the same `--include` set your indexing used, or results won't match.

A raw "filtered N files" count is misleading on its own — a repository is full of images
and licences that should be dropped. Lines marked `!` are the ones to look at.

### `list`

List all your indexed projects on the server.

```bash
rubberduck-index list
```

### `remove`

Remove a project from the server (deletes synced files and analysis data).

```bash
rubberduck-index remove [--project NAME]
```

## Authentication

The server requires a Bearer token. Get your token from your RubberDuck admin.

**Three ways to provide your token (in priority order):**

1. **Interactive prompt** (default on `init`):
   ```bash
   rubberduck-index init
   # Enter your token: ****
   # Token is saved to .rubberduck/config.json — you won't be asked again
   ```

2. **Environment variable** (any command):
   ```bash
   export RUBBERDUCK_TOKEN=your-token
   rubberduck-index init
   ```

3. **`--token` flag** (init only, for scripting):
   ```bash
   rubberduck-index init --token your-token
   ```

The token is saved in `.rubberduck/config.json` after init. All subsequent commands (`sync`, `watch`, `status`, etc.) read it from there automatically.

The `.rubberduck/` directory is automatically added to `.gitignore` to prevent accidental token commits.

## Configuration

All config lives in `.rubberduck/config.json` (created by `init`):

```json
{
  "server": "https://semantic.rubberduck.com",
  "project": "my-app",
  "token": "your-bearer-token",
  "include": ["**/*"],
  "exclude_defaults": true,
  "max_file_size": 50000000,
  "watch_debounce_ms": 500
}
```

| Field | Default | Description |
|-------|---------|-------------|
| `server` | `https://semantic.rubberduck.com` | MCP server URL |
| `project` | directory name | Project name on the server |
| `token` | — | Bearer token for auth |
| `include` | `["**/*"]` | Glob patterns for files to index (all languages by default) |
| `exclude_defaults` | `true` | Use built-in exclude list (see below). `false` relies on `.gitignore` + `.rubberduck/ignore` alone — credentials stay excluded either way. |
| `max_file_size` | `50000000` (50MB) | Skip files larger than this |
| `watch_debounce_ms` | `500` | Debounce interval for file watcher |
| `request_timeout_s` | `300` | Max silence from the server before a request is considered dead |
| `total_timeout_s` | `0` | Wall-clock ceiling for one sync. `0` = unlimited |
| `upload_chunk_size` | `10` | Files per upload request |
| `max_attempts` | `3` | Attempts per chunk before giving up |

### Custom ignore patterns

Create `.rubberduck/ignore` with `.gitignore`-style patterns:

```
# Extra excludes
tests/fixtures/**
docs/**
*.generated.py
```

### Built-in excludes

**Dependencies & build output** — `node_modules`, `bower_components`, `vendor`, `third_party`, `Pods`, `Carthage`, `Godeps`, `/deps`, `.venv`, `venv`, `__pycache__`, `.tox`, `.mypy_cache`, `.pytest_cache`, `.ruff_cache`, `.eggs`, `*.egg-info`, `dist`, `build`, `target`, `/out`, `coverage`, `.next`, `.nuxt`, `.svelte-kit`, `.turbo`, `.vite`, `.parcel-cache`, `.nyc_output`, `.git`, `.hg`, `.svn`, `.DS_Store`

**Credentials** (never uploaded, even with `exclude_defaults: false`) — `.env*`, `*.pem`, `*.key`, `*.pfx`, `*.p12`, `*.jks`, `*.keystore`, `id_rsa*`, `credentials.json`, `secrets.y*ml`, `.npmrc`, `.pypirc`, `.netrc`, `.htpasswd`

**Non-source noise** — lockfiles, `*.min.js`, `*.min.css`, `*.map`, `*.snap`, `*.csv`, `*.tsv`, `*.parquet`, `*.sqlite`, `*.db`, `*.ipynb`, `*.svg`, `*.log`

**Binaries** are detected by content (null bytes), not by extension.

Tests, Markdown, `.d.ts` declarations, Dockerfiles and config files are **not** excluded — they carry real signal for code analysis.

### Large repositories

Indexing cost is dominated by server-side analysis, not transfer — on the order of
seconds per file. A repository of several thousand files therefore takes hours, and the
client is built to survive that:

- **Chunked uploads.** Files go up in small batches, so one request finishes well inside
  `request_timeout_s` instead of timing out mid-flight.
- **Per-chunk retry.** Timeouts, dropped connections and `429`/`5xx` are retried with
  exponential backoff and jitter, honouring `Retry-After`. Client errors (`401`, `413`,
  `422`…) fail immediately rather than burning attempts.
- **Retry budget.** Retries are capped across the whole sync, so a struggling server
  isn't hammered by a fresh burst for every chunk.
- **Resume.** Chunks the server accepted are durable. If a sync does stop, re-running
  `rubberduck-index sync` compares the manifest and continues from where it left off —
  it never restarts from zero.
- **Idempotency.** Each chunk carries a stable key derived from its contents, so a retry
  after a lost response can be recognised as a duplicate rather than re-analysed.

Progress reporting shows the measured rate and an estimated finish time:

```
  [████████░░░░░░░░░░░░░░░░░░░░░░] 27% 2000/7394 files 0.43 files/s, ~3h29m left
```

**On timeouts.** `total_timeout_s` defaults to `0` (unlimited) because a large repository
legitimately runs for hours and a fixed ceiling would abort real work. A hung server is
caught by `request_timeout_s`, which bounds how long the client will wait *in silence* —
that is a different thing from how long a sync may take. Setting `request_timeout_s` to
`0`/`None` is not supported: without it, a stalled server can hold the client
indefinitely.

## Examples

### Index a project

```bash
cd ~/my-project
rubberduck-index init
```

### Index only specific languages or directories

```bash
rubberduck-index init --include "**/*.py"                  # Python only
rubberduck-index init --include "src/**/*.ts" "src/**/*.tsx"
```

### Point at a different server

```bash
export RUBBERDUCK_SERVER=https://dev.semantic.rubberduck.com
rubberduck-index init
```

### Use with Cursor / Claude Code

After indexing, tell the LLM in Cursor or Claude Code:

```
Load the project "my-app" and trace the data flow of the `request` variable.
```

The LLM will use MCP tools:
1. `load_repo(repo="local/my-app")` — loads code for analysis
2. `analyze_code(statement="trace data flow of request", analysis_id="...")` — queries the code model
3. Returns facts about definitions, assignments, and flow paths

## Troubleshooting

**"No .rubberduck/config.json found"**
Run `rubberduck-index init` first, or `cd` into the project directory.

**"Token is required"**
Get your token from your RubberDuck admin, then run `rubberduck-index init` again.

**"Request body too large"**
Your project exceeds 2GB. Use `--include` patterns to reduce file count.

**"No matching files found"**
Check your `include` patterns in `.rubberduck/config.json`. The default `**/*` indexes
every language, so this usually means everything was filtered out by `.gitignore`,
`.rubberduck/ignore`, or a narrowed `include` from an older version — a config written by
a pre-1.2 release still says `["**/*.py"]` and will skip non-Python files.

**Watcher not detecting changes**
`watchdog` ships as a dependency, so reinstall if it's missing: `pip install --force-reinstall rubberduck-index`. On Linux, check inotify limits: `sysctl fs.inotify.max_user_watches`.
