Metadata-Version: 2.5
Name: git-rag
Version: 0.1.0
Summary: Retrieval-augmented search over an Obsidian vault managed as a git repository
Project-URL: Homepage, https://github.com/dogfoodlab-io/git-rag
Project-URL: Repository, https://github.com/dogfoodlab-io/git-rag
Project-URL: Issues, https://github.com/dogfoodlab-io/git-rag/issues
Author-email: raz <mehraz.hakim@gmail.com>
License-Expression: MIT
License-File: LICENSE
Keywords: git,mcp,obsidian,pgvector,rag,retrieval
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.12
Requires-Dist: httpx>=0.27.0
Requires-Dist: mcp<2,>=1.9
Requires-Dist: psycopg[binary,pool]>=3.2.0
Requires-Dist: python-dotenv>=1.0.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: rich>=13.0.0
Requires-Dist: sentence-transformers>=3.0.0
Requires-Dist: typer>=0.15.0
Description-Content-Type: text/markdown

# git-rag

Retrieval-augmented search over an Obsidian vault that is managed as a git repository.
Postgres + pgvector store the index; git commits drive the updates; a Claude Code skill
and MCP server give agents a query surface.

Source: [dogfoodlab-io/git-rag](https://github.com/dogfoodlab-io/git-rag)

## Agent instructions (copy-paste)

Paste the block below into Cursor, Claude Code, ChatGPT, or any coding agent. Fill in the
two paths first.

```text
You are setting up git-rag (repo: https://github.com/dogfoodlab-io/git-rag) so I can
search my Obsidian vault with hybrid RAG (Postgres + pgvector) and expose it to agents
via MCP.

My absolute vault path (git repo, preferably branch main):
  VAULT_PATH=<<<ABS_PATH_TO_OBSIDIAN_VAULT>>>

Preferred install location for this tool:
  INSTALL_DIR=<<<ABS_PATH_E.G._~/dev/git-rag>>>

Do the following end-to-end without asking me to run commands unless something needs a
secret or GUI approval (Docker Desktop, API keys):

1) Prerequisites: ensure `git`, `uv` (https://docs.astral.sh/uv/), and Docker are available.
   If Docker is unavailable, use a local Postgres with the `vector` and `pg_trgm` extensions
   and set DATABASE_URL accordingly (default Docker URL uses host port 5433).

2) Clone and install:
   git clone https://github.com/dogfoodlab-io/git-rag.git "$INSTALL_DIR"
   cd "$INSTALL_DIR"
   cp .env.example .env
   Set in .env:
     VAULT_PATH=<my vault path>
     VAULT_BRANCH=main
     DATABASE_URL=postgresql://vault:vault@localhost:5433/git_rag
     EMBED_PROVIDER=local
     EMBED_MODEL=sentence-transformers/all-MiniLM-L6-v2
     EMBED_DIM=384
   (Optional API embeds instead of local:)
     EMBED_PROVIDER=openrouter
     EMBED_MODEL=openai/text-embedding-3-small
     EMBED_DIM=1536
     OPENROUTER_API_KEY=<ask me>
   uv sync
   docker compose up -d
   uv run git-rag init
   uv run git-rag index
   uv run git-rag hooks install
   uv run git-rag status   # confirm current=true

3) If the vault is not already a git repo on main:
   cd "$VAULT_PATH" && git init -b main && git add -A && git commit -m "initial vault"

4) Wire MCP for agents (stdio). Prefer Cursor project or user MCP config.

   Cursor — merge into ~/.cursor/mcp.json or the project's .cursor/mcp.json:
   {
     "mcpServers": {
       "git-rag": {
         "command": "uv",
         "args": ["run", "--directory", "INSTALL_DIR", "git-rag", "mcp"],
         "env": {
           "GIT_RAG_ENV_FILE": "INSTALL_DIR/.env"
         }
       }
     }
   }
   Replace INSTALL_DIR with the absolute path from step 2.

   Claude Desktop — same server block under mcpServers in
   ~/Library/Application Support/Claude/claude_desktop_config.json (macOS).

   Claude Code — register the same stdio server / or rely on CLI:
     uv run --directory INSTALL_DIR git-rag context "QUESTION"

5) After MCP is connected, use tools (never invent vault facts):
   - vault_context: primary — token-budgeted pack with citations
   - vault_search: ranked hits / filters
   - vault_note: full note by path
   - vault_related, vault_tags, vault_status
   Citation rules: every claim cites path + line range; if retrieval is empty, say so and
   do not answer from general knowledge as if it came from my notes.
   If vault_status shows current=false, run: uv run git-rag index

6) Verify with:
   uv run git-rag query "test" -k 3
   uv run git-rag status --json
   Confirm the MCP server starts: uv run git-rag mcp (stdio; leave running under the client)

Report back: install path, whether index is current, document/chunk counts, and the exact
MCP JSON you wrote (with absolute paths).
```

### MCP-only (already installed)

If the tool is already cloned and indexed, paste this instead:

```text
Configure git-rag as an MCP stdio server and use it for questions about my notes.

Install dir: <<<ABS_PATH_TO_git-rag_CLONE>>>
Env file:    <<<ABS_PATH_TO_git-rag_CLONE>>>/.env

1) Add MCP server "git-rag":
   command: uv
   args: ["run", "--directory", "<install dir>", "git-rag", "mcp"]
   env: { "GIT_RAG_ENV_FILE": "<install dir>/.env" }

   Cursor: ~/.cursor/mcp.json or .cursor/mcp.json
   Claude Desktop: claude_desktop_config.json → mcpServers

2) Restart the MCP host / reload servers.

3) For note questions, call vault_context first (then vault_search / vault_note as needed).
   Cite path + lines on every claim. If empty, say nothing was found — do not invent.
   If results look stale, check vault_status and run git-rag index.
```

## How it works

```
commit on main ──> git hook ──> git diff --name-status -M <last_indexed>..HEAD
                                          │
                        ┌─────────────────┴─────────────────┐
                        │  A  new note        → chunk+embed │
                        │  M  edited note     → re-chunk    │
                        │  R  renamed/moved   → move row    │
                        │  D  deleted note    → cascade del │
                        └─────────────────┬─────────────────┘
                                          v
                        chunk hash lookup in embedding_cache
                                          v
                        embed only the chunks never seen before
                                          v
                     documents + chunks (HNSW vector + GIN tsvector)
                                          v
                   git-rag query / context  ──> vault-search skill / MCP
```

GitHub Actions can run the same `git-rag index` against a reachable `DATABASE_URL`
when the vault repo is pushed.

The index always corresponds to a real commit. Content is read with `git show <sha>:<path>`
rather than from the working tree, so an unsaved or half-written note never enters the index.

### Why this design

**Git is the change feed.** `git diff --name-status -M` is the only thing that reliably
reports renames as renames. Comparing filesystem mtimes cannot distinguish "moved" from
"deleted and created", which would throw away and recompute every embedding in the note.
The last indexed commit is stored in `index_state`, so the delta is exact and resumable.

**Embeddings are cached by content hash, not by path.** `embedding_cache` is keyed on
`(model, sha256(title + heading + chunk text))` and deliberately outlives the documents that
reference it. Consequences:

- Moving a note between folders costs zero embeddings.
- Renaming a note costs zero embeddings unless the rename changes the note's derived title.
- Reorganising one note's headings only re-embeds the chunks that actually changed.
- Deleting and later restoring a note is free.

The hash includes the title and heading breadcrumb because both are part of the embedded
text. Keying on body alone would serve a stale vector after a title change.

**Retrieval is hybrid.** A vector KNN and a Postgres full-text query run in parallel and are
fused with Reciprocal Rank Fusion. Vector search alone misses exact identifiers (ticket
numbers, people, acronyms); full-text alone misses paraphrase. Personal notes need both,
because you rarely remember the words you originally wrote.

**The filename is searchable, separately from the title.** Every chunk stores a
`context` breadcrumb of `filename > title > heading`, which feeds both the embedding and the
tsvector. This matters because the two routinely disagree: a note named
`2026-06-24 - RoA Doc Mgmt Review.md` may have an H1 of `Resourcing for Extension`. Indexing
only the H1 makes the note unfindable by the name its author actually remembers — and in
Obsidian the filename is the primary identifier, since that is what wikilinks resolve against.

**Every note is retrievable.** Obsidian vaults are full of stubs whose entire content is a
title or a single heading. Those produce no body chunks, so the chunker emits a synthetic
filename+title+headings chunk. Without it, roughly 30% of a real vault is invisible to search.

## Setup

Requires Docker and [uv](https://docs.astral.sh/uv/).

### Install from PyPI

```bash
python -m pip install git-rag
git-rag --help
```

The distribution installs the `git_rag` Python package and the `git-rag` and
`git-rag-mcp` console commands. For example:

```python
import git_rag

print(git_rag.__version__)
```

For development from a clone, continue with the uv-based setup below.

```bash
cp .env.example .env      # then set VAULT_PATH
uv sync
docker compose up -d      # or: docker-compose up -d
uv run git-rag init
uv run git-rag index    # first run downloads the embedding model (~90 MB)
```

The vault must be a git repository with a `main` branch:

```bash
cd "/path/to/vault" && git init -b main && git add -A && git commit -m "initial vault"
```

Then install the triggers:

```bash
uv run git-rag hooks install
```

This installs four hooks into the vault's `.git/hooks`:

| Hook | Fires when | Why it matters |
| --- | --- | --- |
| `post-commit` | you commit locally | the main trigger |
| `post-merge` | `git pull` brings in commits | picks up edits made on another device |
| `post-checkout` | you switch back to `main` | recovers after branch work |
| `post-rewrite` | rebase / amend | history rewrites force a safe rebuild |

Indexing runs detached and behind a lock, so it never adds latency to a git command and
concurrent hooks cannot race. Each run appends to `.git-rag/index.log`. Hooks no-op unless
`HEAD` is the branch named in `VAULT_BRANCH`, so scratch-branch commits are never indexed.

## Querying

```bash
uv run git-rag query "why did we pick Postgres" -k 8   # ranked hits with scores
uv run git-rag context "why did we pick Postgres"      # token-budgeted agent context
uv run git-rag note "Meetings/2026-02-11"              # one note in full
uv run git-rag related "Projects/Alpha.md"             # nearest notes
uv run git-rag tags                                    # topic overview
uv run git-rag status                                  # is the index current?
```

Useful flags: `--mode vector|text|hybrid`, `--path "Meetings/"`, `--tag project-x`,
`--exclude "Archive/"`, `--expand` (include adjacent chunks), `--json`.

### Agent access (MCP)

```bash
uv run git-rag mcp
# or: uv run git-rag-mcp
```

See [`examples/mcp.cursor.json`](examples/mcp.cursor.json) and
[`examples/mcp.claude-desktop.json`](examples/mcp.claude-desktop.json).

Tools: `vault_search`, `vault_context`, `vault_note`, `vault_related`, `vault_tags`, `vault_status`.

`.claude/skills/vault-search/SKILL.md` teaches Claude Code when to reach for the vault and
how to cite results. It prefers MCP tools when available, requires citations on every claim,
and forbids answering from general knowledge when retrieval comes back empty.

### GitHub Actions

Index updates on vault push via the reusable workflow (needs a reachable Postgres+pgvector
`DATABASE_URL` secret — not your laptop Docker):

```yaml
# in the vault repo — see examples/vault-repo-index.yml
jobs:
  index:
    uses: dogfoodlab-io/git-rag/.github/workflows/reusable-index.yml@main
    with:
      git_rag_repository: dogfoodlab-io/git-rag
    secrets:
      DATABASE_URL: ${{ secrets.DATABASE_URL }}
```

Copy [`examples/vault-repo-index.yml`](examples/vault-repo-index.yml) into the vault repo, or
use the composite action under `.github/actions/index`. Prefer `EMBED_PROVIDER=openai`,
`openrouter`, or `voyage` in CI for faster cold starts; cache `~/.cache` if using `local`.

## Configuration

| Variable | Purpose |
| --- | --- |
| `VAULT_PATH` | absolute path to the git-managed vault |
| `VAULT_BRANCH` | branch that triggers indexing (default `main`) |
| `DATABASE_URL` | Postgres DSN; host port is 5433 to avoid colliding with a local Postgres |
| `EMBED_PROVIDER` | `local` (default), `voyage`, `openai`, or `openrouter` |
| `EMBED_MODEL` / `EMBED_DIM` | must agree; `init` refuses a mismatch |
| `EMBED_BASE_URL` | optional OpenAI-compatible API root (also accepts `OPENAI_BASE_URL`) |
| `CHUNK_TOKENS` / `CHUNK_OVERLAP` | chunk budget, default 512 / 64 |
| `VAULT_IGNORE` | comma-separated globs never indexed |
| `GIT_RAG_ENV_FILE` | use an alternate config, e.g. a second vault |
| `GIT_RAG_CA_BUNDLE` | CA bundle for TLS-intercepting networks (see below) |

### OpenRouter

```bash
# .env
EMBED_PROVIDER=openrouter
EMBED_MODEL=openai/text-embedding-3-small
EMBED_DIM=1536
OPENROUTER_API_KEY=sk-or-...
# optional: EMBED_BASE_URL=https://openrouter.ai/api/v1

uv run git-rag init --reset && uv run git-rag index
```

Model IDs are OpenRouter slugs (e.g. `openai/text-embedding-3-small`). Match `EMBED_DIM`
to the model (1536 for `text-embedding-3-small` unless you request fewer dimensions).

`local` is the default because personal notes should not need to leave the machine, and
on-device embedding makes reindexing free. Switching provider or model changes vector
geometry, so `init` refuses to mix and you must rebuild:

```bash
uv run git-rag init --reset && uv run git-rag index
```

### Corporate TLS interception

If the model download fails with `CERTIFICATE_VERIFY_FAILED`, a proxy (Netskope, Zscaler)
is re-signing TLS. Export the macOS trust store and point the config at it:

```bash
mkdir -p .git-rag
{ security find-certificate -a -p /System/Library/Keychains/SystemRootCertificates.keychain
  security find-certificate -a -p /Library/Keychains/System.keychain; } > .git-rag/ca-bundle.pem
echo "GIT_RAG_CA_BUNDLE=$PWD/.git-rag/ca-bundle.pem" >> .env
```

## Tests

```bash
./tests/integration.sh
```

Drives a real git repo through assertions on its own throwaway database: add, edit,
rename, cross-folder move, rename-plus-edit in one commit, delete, off-branch commits,
rewritten history, ignore globs, embedding reuse, retrieval, and an installed git hook
indexing a commit end-to-end. Set `KEEP=1` to keep the fixture for inspection.

## Schema

| Table | Contents |
| --- | --- |
| `documents` | one row per note: path (unique), title, content hash, frontmatter, tags, wikilinks |
| `chunks` | heading-scoped passages with `vector` embedding, generated `tsvector`, line range |
| `embedding_cache` | `(model, content_sha) -> vector`; outlives documents |
| `index_state` | last indexed commit, model identity, dimension |

Indexes: HNSW (cosine) on `chunks.embedding`, GIN on `chunks.tsv`, GIN on `documents.tags`,
trigram on `documents.path`. Chunks cascade-delete with their document.

## Operational notes

- `git-rag index --full` rebuilds every note, ignoring content hashes. Use it after
  changing chunking or embedding logic; a plain `index` would consider everything unchanged.
- A full rebuild also prunes notes that no longer exist in the tree.
- `git-rag status` compares the indexed commit to the vault's `HEAD`, which is the fastest
  way to spot a hook that silently stopped firing.
- One malformed note is logged and skipped rather than aborting the run.
