Metadata-Version: 2.4
Name: talkpipe-vault
Version: 1.0.0b4
Summary: AI-powered personal information assistant
Author-email: Travis Bauer <tlbauer@sandia.gov>
Maintainer-email: Travis Bauer <tlbauer@sandia.gov>
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/sandialabs/talkpipe-vault
Project-URL: Repository, https://github.com/sandialabs/talkpipe-vault
Project-URL: Documentation, https://github.com/sandialabs/talkpipe-vault#readme
Project-URL: Bug Tracker, https://github.com/sandialabs/talkpipe-vault/issues
Keywords: ai
Classifier: Development Status :: 3 - Alpha
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Operating System :: OS Independent
Classifier: Topic :: Text Processing
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.11.4
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: talkpipe[all]>=1.0.0b5
Requires-Dist: fastapi[standard]>=0.109.1
Requires-Dist: uvicorn>=0.24.0
Requires-Dist: jinja2>=3.1.6
Requires-Dist: watchdog[watchmedo]
Requires-Dist: textual>=8.0.0
Provides-Extra: dev
Requires-Dist: pytest>=9.0.3; extra == "dev"
Requires-Dist: pytest-mock>=3.10.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
Requires-Dist: pytest-asyncio>=0.23.0; extra == "dev"
Requires-Dist: httpx>=0.24.0; extra == "dev"
Requires-Dist: ruff<0.17,>=0.16.3; extra == "dev"
Requires-Dist: mypy<3,>=2.3; extra == "dev"
Requires-Dist: pre-commit>=4.0; extra == "dev"
Requires-Dist: bandit>=1.7.7; extra == "dev"
Requires-Dist: safety>=2.3.0; extra == "dev"
Provides-Extra: test
Requires-Dist: pytest>=9.0.3; extra == "test"
Requires-Dist: pytest-mock>=3.10.0; extra == "test"
Requires-Dist: pytest-cov>=4.0.0; extra == "test"
Requires-Dist: pytest-asyncio>=0.23.0; extra == "test"
Requires-Dist: httpx>=0.24.0; extra == "test"
Provides-Extra: security
Requires-Dist: bandit>=1.7.7; extra == "security"
Requires-Dist: safety>=2.3.0; extra == "security"
Dynamic: license-file

<p align="center">
  <img src="docs/talkpipe_vault.jpg" alt="TalkPipe Vault Logo" width="300">
</p>

# TalkPipe Vault

> Turn folders of documents into a searchable, question-answerable vault — on your own machine.

[![Python 3.11.4+](https://img.shields.io/badge/python-3.11.4+-blue.svg)](https://www.python.org/downloads/)
[![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
[![Development Status](https://img.shields.io/badge/status-alpha-yellow.svg)](https://github.com/sandialabs/talkpipe-vault)

<p align="center">
<img src="docs/vault-homepage.png" alt="Talkpipe Homepage" width="100%">
</p>
<p align="center">
<img src="docs/vault-search.png" alt="vault search example" width="100%">
</p>

## What is TalkPipe Vault?

TalkPipe Vault is a web application that indexes your documents — notes,
papers, reports, an Obsidian vault, a project archive — into a local
[LanceDB](https://lancedb.com/) vector database and lets you explore them
three ways:

- **Semantic search** — find documents by meaning, not just words
- **Keyword search** — precise full-text queries with boolean operators
- **Ask** — single-turn Q&A with answers grounded in your documents

Everything runs locally by default. The built-in embedding model (model2vec)
runs in-process with no server or API key; generated answers can come from
any LLM provider TalkPipe supports — a local [Ollama](https://ollama.com/)
server, OpenAI, Anthropic — and TalkPipe plugins can add others. Your
documents are only ever sent to the provider you choose.

It is built on the [TalkPipe](https://github.com/sandialabs/talkpipe)
pipeline framework and doubles as a real-world example of composing document
processing, vector search, and RAG from reusable components — see the
[Advanced Guide](docs/ADVANCED.md) if that side interests you.

**Status:** alpha, under active development. The PyPI release can lag this
README — if something described here is missing from a `pip install`,
install from source ([Development setup](docs/ADVANCED.md#development-setup))
to get the documented behavior.

## Run the web app

### Option 1: pip install

```bash
# A virtual environment avoids PEP 668 "externally managed" errors on
# recent Linux distributions.
python -m venv .venv
source .venv/bin/activate      # Windows: .venv\Scripts\activate

pip install talkpipe-vault
vault-server
```

Open http://127.0.0.1:8002, then:

1. **Vaults & Documents** — pick the folder (or glob pattern) to index. A
   vault name is suggested for you; one click creates the vault and indexes
   into it. The first index downloads the default embedding model from
   Hugging Face (about 250 MB on disk, cached afterward).
2. **Search** and **Ask** away.

Answers on the Ask page need a chat provider — any one that TalkPipe
supports. Pick it on the **Settings** page: a local Ollama server (the
default setting; enter its URL under **Connections & credentials**), or
OpenAI or Anthropic (enter an API key there — no environment variables
needed). The default setting is Ollama, so until a reachable provider is
configured, Ask shows a connection error that explains how to fix it. To
try the Ask page without any provider, select the built-in scripted
responder (**eliza**) as the chat source on the Settings page — it needs
no server or key but is only useful for checking that the plumbing works.

### Option 2: Container (Podman or Docker)

```bash
podman run --rm -p 8002:8002 \
  -v vault_data:/app/data \
  -v ~/Documents:/documents:ro,Z \
  -e TALKPIPE_OLLAMA_SERVER_URL=http://host.containers.internal:11434 \
  ghcr.io/sandialabs/talkpipe-vault:latest
```

Then open http://127.0.0.1:8002 (use `127.0.0.1`, not `localhost` — rootless
podman publishes ports IPv4-only). Docker users can substitute `docker run`
with the same arguments.

What each piece does:

- `-v vault_data:/app/data` — persistent storage for vaults, settings, and
  the embedding-model cache, so the model downloads once and your data
  survives container recreation. On start the container reopens the vault
  you last used; the first run starts on the Vaults & Documents page.
- `-v ~/Documents:/documents:ro,Z` — host documents to index; the folder
  picker only sees what you mount. Mount `~` instead to browse your whole
  home directory. Keep `:Z` on SELinux Linux hosts (e.g. Fedora); **drop it
  on macOS and Windows**, where it makes podman try to relabel every mounted
  file.
- The `TALKPIPE_OLLAMA_SERVER_URL` line is **optional** and only matters if
  you use the default Ollama chat setting — drop it if you configure OpenAI
  or Anthropic in the browser instead. Without any provider, search and
  indexing still work; for Ask, either configure a provider or select the
  scripted **eliza** responder on the Settings page.

**macOS/Windows notes:** containers run inside the podman machine VM
(Podman Desktop sets this up). In PowerShell, replace the `\` line
continuations with backticks and write the documents path explicitly
(`-v C:\Users\you\Documents:/documents:ro`). Before indexing a large
collection, give the VM more memory than its default (often 2 GB) — a big
ingestion peaks around 1.5–2 GB and an over-limit kill is silent
(exit code 137, `oom=true` in `podman inspect`):

```bash
podman machine stop
podman machine set --memory 4096    # MiB; use 8192 for very large collections
podman machine start
```

A compose service and instructions for deriving your own customized image
(different default models, extra packages) are in the
[Advanced Guide](docs/ADVANCED.md#containers).

## The terminal interface (`vault-tui`)

Everything above is also available without a browser — in an SSH session, a
tmux window, or on a headless machine — through `vault-tui`, installed
alongside `vault-server`. It is new and not yet in a PyPI release, so install
it from source — either straight from the repository:

```bash
python -m venv .venv
source .venv/bin/activate      # Windows: .venv\Scripts\activate
pip install "git+https://github.com/sandialabs/talkpipe-vault.git"
```

or from a clone with `pip install .` (contributors use the editable
`pip install -e ".[dev]"` in [Development setup](docs/ADVANCED.md#development-setup)).

```bash
vault-tui ~/my-vault      # open (or create) a vault — the index folder, not your documents
vault-tui --resume        # reopen the most recently used vault
vault-tui                 # start on the Vault tab and choose one there
```

A vault is a folder that holds the search index; the documents live wherever
they already are and are named on the Vault tab. Point `vault-tui` at a folder
of documents by mistake and it asks before turning that folder into a vault.
Opening a vault loads the embedding model, which on a first run means
downloading it (about 250 MB for the default model2vec model) — the Vault tab
says so while it waits, and the header reads "opening…" until the vault is
ready.

Enter in either path field on the Vault tab runs **Index documents** (it
opens the vault when only the vault path is filled).

It runs in-process (no server needed) and uses the same vault files, recent
list, model settings and credentials as the web interface, so you can switch
between the two freely. Tabs mirror the web pages:

| Key | Tab | What you can do |
|-----|-----|-----------------|
| `F2` | Vault | Open/create a vault, browse to a documents folder, index it (with progress), open or delete recent vaults, edit the retrieval filter |
| `F3` | Search | Semantic search; the detail pane follows the highlighted result — `Enter` loads the full chunk, `o` shows/opens the source document, `c` copies the chunk, Copy All copies every result |
| `F4` | Keywords | Full-text search (Whoosh syntax), and building/rebuilding the full-text index |
| `F5` | Ask | Question answering with the answer, its "Answered by" line and the source chunks it used; optional keyword boost |
| `F6` | Settings | Configuration status (Re-test), embedding/chat model settings, connections & credentials |
| `F1` / `Ctrl+R` / `Ctrl+Q` | | Help / reload the vault and settings (after indexing or editing `~/.talkpipe.toml` outside the app) / quit (`Ctrl+C` only reminds you of `Ctrl+Q`; while an indexing run is in progress `Ctrl+Q` asks first, because quitting abandons it) |

`--show-source-paths` shows file paths in results, as for `vault-server`.
On a shared machine, `TALKPIPE_VAULT_ROOT` and `TALKPIPE_DOCUMENT_ROOTS`
confine where vaults and documents may live for both interfaces — see
[Confining paths on a shared machine](docs/ADVANCED.md#confining-paths-on-a-shared-machine).
Long operations (embedding, Ask, indexing) run in the background and report
progress in the tab that started them. As in the browser, Ask needs a chat
provider: enter the Ollama URL or an API key under **Connections &
credentials** on the Settings tab (`F6`) — it is the last field on the tab,
and one `Shift+Tab` from the top (the Re-test button `F6` lands on) jumps
straight to it; `PageUp`/`PageDown` scroll the tab — and press **Save
connection settings**, which re-tests the configuration by itself. A bare
`host:11434` is completed to `http://host:11434` when saved. Alternatively
export `TALKPIPE_OLLAMA_SERVER_URL` before starting — but a URL saved on the
Settings tab takes precedence over the variable, so clear the field to go
back to it (the configuration status names which one is in effect). The
OpenAI base URL field points the OpenAI provider at any OpenAI-compatible
endpoint — see [Provider notes](docs/ADVANCED.md#provider-notes). While an
answer is being generated, `Esc` stops waiting for it. "Index
documents" adds to the open vault — tick **Overwrite existing index** to
replace it; re-indexing the same folder without it duplicates every chunk
(the summary line says so when it happens, and the box unticks itself after
a replace run). Indexing never updates the full-text index: the header
shows "keywords out of date" until you rebuild it on the Keywords tab, and
a keyword search that finds nothing says so. Long chunk text scrolls once
you `Tab` into its pane; the question box grows as a long question wraps,
and `PageUp`/`PageDown` in it scroll a long answer (the answer pane is also
four `Tab` stops from the question box). After a search the result list has
the focus, so `F3`/`F4` (or `Shift+Tab`) return to the query field before
you type the next query. If the open vault's folder disappears from disk
(deleted or unmounted outside the app), Search, Ask and `Ctrl+R` say so
instead of quietly recreating it empty. The **Retrieval filter** button on the Vault tab edits the same
per-vault ChatterLang script as the web page, with an example in the
dialog (its **Help** button adds the result shape and more recipes); a
saved filter does nothing until you tick **Enabled on this machine**. The
script syntax is in the Advanced Guide under
[Writing a retrieval filter](docs/ADVANCED.md#writing-a-retrieval-filter).

## The web interface

- **Vaults & Documents** — choose the documents to index and the vault to
  index them into, on one page with a built-in folder browser. With no vault
  open, a vault name is suggested from the documents folder and created on
  submit; with one open, the same form adds documents to it. Recent vaults
  are remembered for one-click reopening, and indexing shows live progress.
- **Settings** — choose embedding and chat providers/models, with a live
  **Configuration status** panel that tests your selection (and can download
  an uncached embedding model via Re-test), plus **Connections &
  credentials** for API keys and the Ollama URL — no environment variables
  required.
- **Semantic Search** — vector similarity search over your documents.
- **Keyword Search** — boolean and phrase queries. Matching is
  case-insensitive but on exact word tokens (`apple` won't match `apples`);
  use semantic search for meaning-based lookups.
- **Ask** — single-turn Q&A with source citations you can open and copy.
  Once a full-text index exists, a **Boost retrieval with keyword search**
  checkbox appears: the chat model distills your question into index
  keywords, the keyword matches are merged with the vector-search results,
  and the combined context is used to answer. The "Answered by" line under
  the answer confirms whether the boost ran and how many keyword hits were
  merged (keyword hits that duplicate vector results are combined, so the
  boost can be active even when the source list looks unchanged).

Every search result and Ask citation has an **Open** link that fetches the
original document from the server — PDFs, images, and plain text open right
in the browser; other formats download. Because the file is streamed over
HTTP, this works the same when the server runs in a container (where your
documents live at a container-side mount path the browser can't reach
directly).

## Configuring models

The Settings page is the primary way to configure models; choices persist
and apply immediately. The defaults are just starting points — embeddings:
`model2vec` / `minishlab/potion-retrieval-32M` (in-process, no key or
server); chat: `ollama` / `mistral-small` — and both dropdowns list every
provider registered with TalkPipe: model2vec, Ollama, OpenAI, and Anthropic
out of the box, plus any provider added by an installed TalkPipe plugin,
which appears there automatically. API keys are entered in the browser, not
the environment.

One behavior worth knowing: the embedding model is a property of the indexed
data — embeddings are only comparable to queries embedded by the same model —
so each vault records the embedder it was built with and reopens with it,
regardless of the current default. Chat models can be switched freely at any
time.

Configuration is also possible via `~/.talkpipe.toml` or `TALKPIPE_*`
environment variables; the full reference (precedence, all keys, templates,
provider notes) is in the [Advanced Guide](docs/ADVANCED.md#model-configuration).

## More documentation

The [Advanced Guide](docs/ADVANCED.md) covers:

- Command-line indexing with `makevectordatabase` and the full `vault-server`
  flag reference
- The compose service and deriving a customized container image
- The complete model configuration reference
- Architecture, the reusable TalkPipe sources/segments, and building your own
  pipelines
- Vault storage layout
- Development setup
- The experimental directory-monitoring components

## Requirements

- **Python** 3.11.4+ (pip install path)
- **Ollama** (optional) for local chat answers; **OpenAI/Anthropic API key**
  (optional) for cloud models. Embeddings work out of the box with neither.

## Contributing

Contributions are welcome. Before submitting: `pytest` passes, and
`ruff check .`, `ruff format --check .`, and `mypy` are clean -- CI fails on
any finding from those three (`ruff check --fix . && ruff format .` fixes
most; `pre-commit install` runs them on every commit). See
[Development setup](docs/ADVANCED.md#development-setup).

### Development environment

Local development uses [uv](https://docs.astral.sh/uv/) against the committed
`uv.lock`, so contributors share one reproducible set of versions:

```bash
uv sync --extra dev
uv run pytest
```

**CI does not use the lockfile.** It installs with pip (`pip install -e
'.[dev]'`) and resolves dependencies fresh, on purpose: that is what someone
running `pip install talkpipe-vault` gets, so the build breaks when *they*
would break. A dependency problem that only the lockfile hides is one we want
CI to see.

Two consequences worth remembering:

- `uv.lock` is a development convenience. It pins nothing for users and is not
  a security control — the version floors in `pyproject.toml` are what
  actually protect an install. Fix a vulnerable dependency by raising its
  floor, not by refreshing the lock.
- The lock must still stay honest. CI runs `uv lock --check`, which installs
  nothing and fails only when `uv.lock` and `pyproject.toml` have drifted
  apart. If you change dependencies, run `uv lock` and commit the result.

## License

Apache License 2.0 — see [LICENSE](LICENSE).

## Authors

- **Travis Bauer** — *Initial development* — [Sandia National Laboratories](https://www.sandia.gov/)

## Acknowledgments

Built with [TalkPipe](https://github.com/sandialabs/talkpipe); vector storage
by [LanceDB](https://lancedb.com/); file monitoring with
[Watchdog](https://github.com/gorakhargosh/watchdog).
