Metadata-Version: 2.4
Name: querdex
Version: 0.3.1
Summary: Reasoning-first document intelligence system
Author: Animay Tiwari
Author-email: Animay Tiwari <animaytiwari@outlook.com>
License: MIT
Keywords: document,indexing,llm,rag,retrieval
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing :: Indexing
Requires-Python: >=3.11
Requires-Dist: beautifulsoup4>=4.12.3
Requires-Dist: markdown-it-py>=3.0.0
Requires-Dist: networkx>=3.4.2
Requires-Dist: numpy>=1.26
Requires-Dist: pydantic>=2.9.0
Requires-Dist: pymupdf>=1.24.0
Requires-Dist: python-docx>=1.1.2
Provides-Extra: anthropic
Requires-Dist: anthropic>=0.49.0; extra == 'anthropic'
Provides-Extra: dev
Requires-Dist: mypy>=2.0; extra == 'dev'
Requires-Dist: pytest>=8.3.3; extra == 'dev'
Requires-Dist: ruff>=0.8.6; extra == 'dev'
Provides-Extra: embeddings
Requires-Dist: sentence-transformers>=3.0.0; extra == 'embeddings'
Provides-Extra: openai
Requires-Dist: openai>=1.67.0; extra == 'openai'
Description-Content-Type: text/markdown

# Querdex

[![PyPI version](https://img.shields.io/pypi/v/querdex)](https://pypi.org/project/querdex/)
[![Total downloads](https://static.pepy.tech/badge/querdex)](https://pepy.tech/projects/querdex)
[![Downloads/month](https://img.shields.io/pypi/dm/querdex)](https://pypistats.org/packages/querdex)
[![Python versions](https://img.shields.io/pypi/pyversions/querdex)](https://pypi.org/project/querdex/)
[![CI](https://github.com/its-animay/querdex/actions/workflows/ci.yml/badge.svg)](https://github.com/its-animay/querdex/actions/workflows/ci.yml)
[![License: MIT](https://img.shields.io/badge/license-MIT-green.svg)](https://opensource.org/licenses/MIT)

**Reasoning-first document intelligence system.**

Querdex indexes any document into a hierarchical tree, then uses a two-tier LLM search to answer questions with cited sources. It works without an LLM (keyword heuristics), and optionally plugs in Anthropic or OpenAI for higher-quality results.

---

## Table of Contents

- [How it works](#how-it-works)
- [Installation](#installation)
- [Quick Start (CLI)](#quick-start-cli)
- [LLM Setup](#llm-setup)
- [Structured Extraction](#structured-extraction)
- [Knowledge Graph](#knowledge-graph)
- [CLI Reference](#cli-reference)
- [Python API](#python-api)
- [Supported File Types](#supported-file-types)
- [Environment Variables](#environment-variables)

---

## How it works

```
Document
   │
   ▼
Ingestion ──► parse into pages/sections (Section[])
   │
   ▼
Indexing ───► build hierarchical tree (TreeNode) + entity map + knowledge graph
   │
   ▼
Storage ────► persist to SQLite (sections, tree, entities, graph, query cache)
   │
   ▼
Query
  ├─ Tier 1: LLM (or keyword) batch-prune of tree nodes
  ├─ Tier 2: LLM (or heuristic) per-node relevance scoring
  ├─ Retrieval: pull section text for selected nodes
  └─ Answer: LLM synthesizes answer with source citations
   │
   ▼
Adaptive ───► update node summaries based on query feedback (runs in background)
```

Three query routes are selected automatically:
- **single_doc** — standard hierarchical search on one document
- **multi_doc** — virtual super-tree across up to 3 documents
- **graph** — entity-seeded graph walk for relationship queries ("how does X relate to Y?")

---

## Installation

**Base install** (no LLM, uses keyword heuristics):
```bash
pip install querdex
```

**With Anthropic (Claude):**
```bash
pip install querdex[anthropic]
```

**With OpenAI (GPT):**
```bash
pip install querdex[openai]
```

**With local embeddings** (dense semantic node scoring, runs offline):
```bash
pip install querdex[embeddings]
```

**Development:**
```bash
git clone <repo>
cd querdex
uv sync --extra dev
# or with an LLM provider:
uv sync --extra dev --extra anthropic
uv sync --extra dev --extra openai
```

**Requirements:** Python 3.11+

---

## Quick Start (CLI)

### 1. Index a document

```bash
querdex index ./report.pdf --doc-id annual-report
```

Output:
```
Indexed doc_id=annual-report version=1
Nodes=12 max_depth=3
```

### 2. Query it

```bash
querdex query --doc-id annual-report --query "What was the Q3 revenue?"
```

Output:
```
Query ID: 3f8a1c...
Intent: single_doc | Cache hit: False
Q3 revenue was $1.2B, up 8% year-over-year (Revenue Analysis, pages 4-6).
```

### 3. Multi-turn conversation (session)

```bash
# First turn
querdex query --doc-id annual-report \
  --query "What were the risk factors?" \
  --session-id session_001

# Second turn — context from first turn is carried over
querdex query --doc-id annual-report \
  --query "Which of those risks materialised?" \
  --session-id session_001
```

### 4. Re-index an updated document

When the document changes, Querdex only rebuilds the affected parts:
```bash
querdex index ./report_v2.pdf --doc-id annual-report
```

### 5. Delete a document

```bash
querdex delete --doc-id annual-report
```

### Custom database path

By default the database is stored at `./index_store/querdex.db`. To change it:
```bash
querdex --db /path/to/my.db index ./report.pdf --doc-id demo
querdex --db /path/to/my.db query --doc-id demo --query "summary?"
```

---

## LLM Setup

Without any LLM configured, Querdex falls back to keyword/heuristic matching — it always produces an answer, just less precise.

### Anthropic (Claude)

```bash
export QUERDEX_LLM_PROVIDER=anthropic
export QUERDEX_LLM_API_KEY=sk-ant-...

# Optional: override model defaults
export QUERDEX_LLM_TIER1_MODEL=claude-haiku-4-5-20251001   # fast, cheap (batch prune)
export QUERDEX_LLM_TIER2_MODEL=claude-sonnet-4-6            # powerful (deep reasoning + answers)
```

### OpenAI (GPT)

```bash
export QUERDEX_LLM_PROVIDER=openai
export QUERDEX_LLM_API_KEY=sk-...

# Optional: override model defaults
export QUERDEX_LLM_TIER1_MODEL=gpt-4o-mini   # fast, cheap
export QUERDEX_LLM_TIER2_MODEL=gpt-4o         # powerful
```

**How the two tiers are used:**

| Tier | Model | Purpose |
|------|-------|---------|
| Tier 1 | cheap/fast | Single batched call to prune all tree nodes to the relevant few |
| Tier 2 | powerful | Per-node deep reasoning to confirm relevance + score confidence |
| Answer | powerful | Synthesise a cited answer from the retrieved section text |

---

## Structured Extraction

Pull structured facts out of any indexed document — with **source grounding**: every extraction carries the exact section, page, and character span it came from, so nothing is silently hallucinated.

The schema is defined **by example**, not by code. Describe what you want and (optionally) show one or two examples:

```bash
querdex extract --doc-id demo \
  --prompt "Extract revenue figures and executive names" \
  --examples examples.json \
  --html review.html
```

`examples.json`:
```json
[
  {
    "text": "Alice Chen reported revenue of $5M in Q1.",
    "extractions": [
      {"extraction_class": "metric", "extraction_text": "revenue of $5M", "attributes": {"period": "Q1"}},
      {"extraction_class": "person", "extraction_text": "Alice Chen"}
    ]
  }
]
```

The classes and attribute keys in your examples define the output schema. Every result is aligned back to the source text (exact → fuzzy matching); model output that cannot be located is kept but flagged `unaligned` so you can review it instead of trusting it.

`--html` writes a self-contained review page: the full document with color-coded highlights per extraction class, toggleable legend, attribute tooltips, and a click-to-jump list of all extractions.

Long documents are chunked and processed in parallel; use `--passes 2` to trade extra LLM calls for higher recall. Without an LLM configured, extraction degrades to literal matching of your example texts.

Python API:

```python
from querdex.extraction import ExtractionTask, ExtractionExample, ExampleExtraction

task = ExtractionTask(
    description="Extract revenue figures and executive names",
    examples=[...],
)
run = engine.extract_document("demo", task, passes=1)
for e in run.extractions:
    print(e.extraction_class, repr(e.extraction_text), e.section_id, e.char_start, e.alignment)
```

---

## Knowledge Graph

Build a queryable **entity graph** across your indexed documents: canonical entities as nodes, typed relationships as edges, with automatic theme clustering and hub ranking.

Every edge is **grounded** — it carries the document, page, character span, and the exact sentence that justifies it, so any relationship can be traced back to its source.

```python
engine = build_engine("./index_store/querdex.db")
graph = engine.build_knowledge_graph()          # or pass doc_ids=[...]

for hub in graph.hubs[:5]:                       # most connected entities
    print(hub.name, hub.score)

for community in graph.communities:              # detected themes
    print(community.label, community.entity_ids)

for relation in graph.relations:
    print(relation.source_entity_id, relation.predicate, relation.target_entity_id)
    print("  evidence:", relation.evidence)      # the sentence it came from
```

Entities are resolved and canonicalized (aliases such as "Aurora" fold into "Aurora Financial Group") and typed as `organization`, `person`, `place`, `metric`, `date`, or `concept`. Relations are marked `stated` when a sentence directly asserts them and `inferred` when derived from co-occurrence.

### Explaining connections

Ask how any two entities are related and get the path back, with the source sentence behind **every hop**:

```bash
querdex graph                                        # build it once
querdex explain --from "Asia-Pacific" --to "Singapore"
```

```
Asia-Pacific --[was fastest growing region at]--> 22 percent
             --[revenue growth led by financial services customers in]--> Singapore

2 hop(s), weakest link 0.70
  1. Asia-Pacific -> 22 percent   (stated, 0.70)
     "Asia-Pacific was the fastest growing region at 22 percent revenue growth, led by ..."
     aurora p18 chars 0-133
```

```python
path = engine.explain_connection("Asia-Pacific", "Singapore")
print(path.as_sentence(), path.confidence)       # confidence = weakest hop
for hop in path.hops:
    print(hop.predicate, hop.evidence, hop.doc_id, hop.page_number)

sub = engine.entity_subgraph(["Nimbus Analytics"], hops=2)   # scoped neighbourhood
```

### Visualizing

```bash
querdex graph --html graph.html --report GRAPH_REPORT.md --json graph.json
```

`--html` writes a self-contained interactive page — nodes sized by connectivity and coloured by community, solid edges for `stated` relations and dashed for `inferred`, hover to isolate a neighbourhood, and click any node or edge to read its grounded evidence. No external requests, so it works offline.

Works with no LLM configured. With one, relationships are additionally extracted as typed subject/predicate/object triples — accepted only when both endpoints resolve to known entities and the evidence is located in the source, so the model cannot introduce an ungrounded edge. Graphs persist to the store and are retrievable with `store.latest_knowledge_graph()`.

---

## CLI Reference

```
querdex [--db PATH] <command> [options]
```

| Command | Description |
|---------|-------------|
| `index <file>` | Index a document. Auto-detects format from extension. |
| `query` | Query an indexed document. |
| `extract` | Run schema-by-example structured extraction over an indexed document. |
| `graph` | Build an entity knowledge graph; optionally write HTML, Markdown, or JSON. |
| `explain` | Explain how two entities are connected, citing each hop. |
| `delete` | Remove a document and all its data from the store. |

### `index`

```
querdex index <file_path> [--doc-id ID]
```

| Argument | Default | Description |
|----------|---------|-------------|
| `file_path` | required | Path to the document to index |
| `--doc-id` | auto-generated from filename+hash | Stable identifier for this document |

### `query`

```
querdex query --doc-id ID --query TEXT [--session-id ID]
```

| Argument | Default | Description |
|----------|---------|-------------|
| `--doc-id` | required | Document to query |
| `--query` | required | Natural language question |
| `--session-id` | none | Enables multi-turn context (pass same ID across turns) |

### `extract`

```
querdex extract --doc-id ID --prompt TEXT [--examples FILE] [--passes N] [--html FILE]
```

| Argument | Default | Description |
|----------|---------|-------------|
| `--doc-id` | required | Document to extract from |
| `--prompt` | required | Natural language description of what to extract |
| `--examples` | none | JSON file with few-shot examples (defines the output schema) |
| `--passes` | `1` | Extraction passes; more passes improve recall |
| `--html` | none | Write an interactive HTML review page to this path |

### `graph`

```
querdex graph [--doc-id ID ...] [--html FILE] [--report FILE] [--json FILE]
```

| Argument | Default | Description |
|----------|---------|-------------|
| `--doc-id` | all documents | Limit to these documents (repeatable) |
| `--html` | none | Write a self-contained interactive graph page |
| `--report` | none | Write a Markdown report (hubs, communities, relations) |
| `--json` | none | Write the full graph as JSON |

### `explain`

```
querdex explain --from ENTITY --to ENTITY [--max-hops N]
```

| Argument | Default | Description |
|----------|---------|-------------|
| `--from` | required | Source entity name (aliases and partial names resolve) |
| `--to` | required | Target entity name |
| `--max-hops` | `5` | Maximum path length |

### `delete`

```
querdex delete --doc-id ID
```

---

## Python API

For integration into your own application:

```python
import asyncio
from querdex.services import build_engine

# build_engine reads QUERDEX_LLM_* env vars automatically
engine = build_engine("./index_store/querdex.db")

# Index a document
doc = asyncio.run(engine.index_document("./report.pdf", doc_id="annual-report"))
print(f"Indexed: {doc.doc_id} | nodes={doc.stats.total_nodes}")

# Query
result = engine.query_document("annual-report", "What was Q3 revenue?")
print(result.answer)
print(f"Confidence: {result.confidence:.0%}")
for source in result.source_nodes:
    print(f"  Source: {source.title}, pages {source.pages}")

# Multi-turn query
result2 = engine.query_document(
    "annual-report",
    "What caused that increase?",
    session_id="my-session-001",
)

# Re-index after the document changes
doc_v2 = asyncio.run(engine.reindex_document("./report_v2.pdf", doc_id="annual-report"))

# Delete
engine.store.delete_document("annual-report")

# Always close when done
engine.store.close()
```

### Passing an LLM client directly

```python
from querdex.llm.anthropic_client import AnthropicLLMClient
from querdex.services.engine import QuerdexEngine
from querdex.storage import SQLiteStore

llm = AnthropicLLMClient(
    api_key="sk-ant-...",
    tier1_model="claude-haiku-4-5-20251001",
    tier2_model="claude-sonnet-4-6",
)
store = SQLiteStore("./querdex.db")
engine = QuerdexEngine(store, llm_client=llm)
```

### Using the FakeLLMClient in tests

```python
from querdex.llm.fake_client import FakeLLMClient
from querdex.query.answering import AnswerGenerator

fake = FakeLLMClient(
    default='{"answer": "Revenue was $1.2B.", "confidence": 0.9}'
)
gen = AnswerGenerator(llm_client=fake)
answer, confidence, sources = gen.generate("What was revenue?", chunks)
```

---

## Supported File Types

| Extension | Parser | Notes |
|-----------|--------|-------|
| `.txt` | TextParser | Plain text, split by paragraphs |
| `.md`, `.markdown` | MarkdownParser | Heading-aware section splitting |
| `.html`, `.htm` | HTMLParser | Strips tags, extracts text blocks |
| `.docx` | DOCXParser | Microsoft Word, paragraph-level |
| `.pdf` | PDFParser | Page-level; OCR optional (see below) |
| `.py` | PythonCodeParser | Function/class level chunking |
| `.js`, `.ts`, `.jsx`, `.tsx` | JSCodeParser | Function-level chunking |
| `.csv` | CSVParser | Row-batched sections |
| `.db`, `.sqlite` | SQLiteParser | Table-level sections |
| `.mp3`, `.wav`, `.m4a`, `.mp4`, `.mov` | AudioVideoParser | Transcript-based (requires Whisper or similar) |
| `.url` | URLParser | Fetches and parses the web page at that URL |
| URL string | URLParser | Pass a URL string directly as the file path |

### PDF OCR

For scanned PDFs, enable OCR via environment variables:

```bash
# Tesseract (local)
export QUERDEX_OCR_ENABLED=true
export QUERDEX_OCR_PROVIDER=tesseract         # default when OCR enabled
export QUERDEX_TESSERACT_CMD=tesseract        # path to tesseract binary

# Cloud OCR (custom endpoint)
export QUERDEX_OCR_ENABLED=true
export QUERDEX_OCR_PROVIDER=cloud
export QUERDEX_OCR_ENDPOINT=https://your-ocr-api.com/v1/ocr
export QUERDEX_OCR_API_KEY=your-key
```

---

## Environment Variables

| Variable | Default | Description |
|----------|---------|-------------|
| `QUERDEX_LLM_PROVIDER` | _(none)_ | `anthropic` or `openai`. If unset, heuristic mode is used. |
| `QUERDEX_LLM_API_KEY` | _(none)_ | API key for the selected provider |
| `QUERDEX_LLM_TIER1_MODEL` | `claude-haiku-4-5-20251001` / `gpt-4o-mini` | Fast model for batch node pruning |
| `QUERDEX_LLM_TIER2_MODEL` | `claude-sonnet-4-6` / `gpt-4o` | Powerful model for deep reasoning and answers |
| `QUERDEX_EMBEDDING_PROVIDER` | _(none)_ | `local` or `openai`. Enables dense semantic node scoring; unset uses LLM/heuristic scoring. |
| `QUERDEX_EMBEDDING_MODEL` | `all-MiniLM-L6-v2` / `text-embedding-3-small` | Embedding model for the selected provider |
| `QUERDEX_EMBEDDING_API_KEY` | _(none)_ | API key for hosted embedding providers (falls back to `QUERDEX_LLM_API_KEY`) |
| `QUERDEX_OCR_ENABLED` | `false` | Enable OCR for scanned PDFs |
| `QUERDEX_OCR_PROVIDER` | `tesseract` | `tesseract` or `cloud` |
| `QUERDEX_TESSERACT_CMD` | `tesseract` | Path to Tesseract binary |
| `QUERDEX_OCR_ENDPOINT` | _(none)_ | Endpoint URL for cloud OCR provider |
| `QUERDEX_OCR_API_KEY` | _(none)_ | API key for cloud OCR provider |

---

## License

MIT
