Metadata-Version: 2.5
Name: omnichunk
Version: 2.2.0
Summary: Structure-aware deterministic chunking for code, prose, and markup.
Project-URL: Homepage, https://github.com/oguzhankir/omnichunk
Project-URL: Documentation, https://oguzhankir.github.io/omnichunk/
Project-URL: Issues, https://github.com/oguzhankir/omnichunk/issues
Project-URL: Source, https://github.com/oguzhankir/omnichunk
Project-URL: Changelog, https://github.com/oguzhankir/omnichunk/blob/main/CHANGELOG.md
License: MIT
License-File: LICENSE
Keywords: chunking,code-analysis,embedding,llm,rag,semantic-search,tree-sitter
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Typing :: Typed
Requires-Python: >=3.10
Provides-Extra: all
Requires-Dist: langchain-core<2,>=0.2.11; extra == 'all'
Requires-Dist: langchain-text-splitters<2,>=0.2.2; extra == 'all'
Requires-Dist: line-profiler>=4.1.0; extra == 'all'
Requires-Dist: llama-index-core<0.15,>=0.10.68.post1; extra == 'all'
Requires-Dist: numpy>=1.24.0; extra == 'all'
Requires-Dist: opentelemetry-api>=1.21.0; extra == 'all'
Requires-Dist: py-spy>=0.3.14; extra == 'all'
Requires-Dist: pypdf>=3.0; extra == 'all'
Requires-Dist: python-docx>=1.0; extra == 'all'
Requires-Dist: tiktoken>=0.5.0; extra == 'all'
Requires-Dist: tree-sitter-bash>=0.23.0; extra == 'all'
Requires-Dist: tree-sitter-c-sharp>=0.23.0; extra == 'all'
Requires-Dist: tree-sitter-c>=0.23.0; extra == 'all'
Requires-Dist: tree-sitter-cpp>=0.23.0; extra == 'all'
Requires-Dist: tree-sitter-elixir>=0.3.0; extra == 'all'
Requires-Dist: tree-sitter-go>=0.23.0; extra == 'all'
Requires-Dist: tree-sitter-java>=0.23.0; extra == 'all'
Requires-Dist: tree-sitter-javascript>=0.23.0; extra == 'all'
Requires-Dist: tree-sitter-kotlin>=0.23.0; extra == 'all'
Requires-Dist: tree-sitter-php>=0.23.0; extra == 'all'
Requires-Dist: tree-sitter-python>=0.23.0; extra == 'all'
Requires-Dist: tree-sitter-ruby>=0.23.0; extra == 'all'
Requires-Dist: tree-sitter-rust>=0.23.0; extra == 'all'
Requires-Dist: tree-sitter-scala>=0.23.0; extra == 'all'
Requires-Dist: tree-sitter-sql>=0.3.0; extra == 'all'
Requires-Dist: tree-sitter-swift>=0.0.1; extra == 'all'
Requires-Dist: tree-sitter-typescript>=0.23.0; extra == 'all'
Requires-Dist: tree-sitter>=0.23.0; extra == 'all'
Provides-Extra: all-languages
Requires-Dist: tree-sitter-bash>=0.23.0; extra == 'all-languages'
Requires-Dist: tree-sitter-c-sharp>=0.23.0; extra == 'all-languages'
Requires-Dist: tree-sitter-c>=0.23.0; extra == 'all-languages'
Requires-Dist: tree-sitter-cpp>=0.23.0; extra == 'all-languages'
Requires-Dist: tree-sitter-elixir>=0.3.0; extra == 'all-languages'
Requires-Dist: tree-sitter-go>=0.23.0; extra == 'all-languages'
Requires-Dist: tree-sitter-java>=0.23.0; extra == 'all-languages'
Requires-Dist: tree-sitter-javascript>=0.23.0; extra == 'all-languages'
Requires-Dist: tree-sitter-kotlin>=0.23.0; extra == 'all-languages'
Requires-Dist: tree-sitter-php>=0.23.0; extra == 'all-languages'
Requires-Dist: tree-sitter-python>=0.23.0; extra == 'all-languages'
Requires-Dist: tree-sitter-ruby>=0.23.0; extra == 'all-languages'
Requires-Dist: tree-sitter-rust>=0.23.0; extra == 'all-languages'
Requires-Dist: tree-sitter-scala>=0.23.0; extra == 'all-languages'
Requires-Dist: tree-sitter-sql>=0.3.0; extra == 'all-languages'
Requires-Dist: tree-sitter-swift>=0.0.1; extra == 'all-languages'
Requires-Dist: tree-sitter-typescript>=0.23.0; extra == 'all-languages'
Requires-Dist: tree-sitter>=0.23.0; extra == 'all-languages'
Provides-Extra: code
Requires-Dist: tree-sitter-go>=0.23.0; extra == 'code'
Requires-Dist: tree-sitter-java>=0.23.0; extra == 'code'
Requires-Dist: tree-sitter-javascript>=0.23.0; extra == 'code'
Requires-Dist: tree-sitter-python>=0.23.0; extra == 'code'
Requires-Dist: tree-sitter-rust>=0.23.0; extra == 'code'
Requires-Dist: tree-sitter-typescript>=0.23.0; extra == 'code'
Requires-Dist: tree-sitter>=0.23.0; extra == 'code'
Provides-Extra: dev
Requires-Dist: hypothesis>=6.0.0; extra == 'dev'
Requires-Dist: maturin>=1.7.0; extra == 'dev'
Requires-Dist: mutmut<3.0.0,>=2.4.4; extra == 'dev'
Requires-Dist: mypy==1.20.2; extra == 'dev'
Requires-Dist: nltk>=3.8.1; extra == 'dev'
Requires-Dist: numpy>=1.24.0; extra == 'dev'
Requires-Dist: opentelemetry-sdk>=1.21.0; extra == 'dev'
Requires-Dist: pre-commit>=3.7.0; extra == 'dev'
Requires-Dist: pypdf>=3.0; extra == 'dev'
Requires-Dist: pytest-cov>=4.0.0; extra == 'dev'
Requires-Dist: pytest-xdist>=3.6.0; extra == 'dev'
Requires-Dist: pytest>=8.0.0; extra == 'dev'
Requires-Dist: python-docx>=1.0; extra == 'dev'
Requires-Dist: ruff==0.15.12; extra == 'dev'
Requires-Dist: scipy>=1.10.0; extra == 'dev'
Requires-Dist: semchunk>=2.0.0; extra == 'dev'
Requires-Dist: tiktoken>=0.5.0; extra == 'dev'
Requires-Dist: tree-sitter-c-sharp>=0.23.0; extra == 'dev'
Requires-Dist: tree-sitter-c>=0.23.0; extra == 'dev'
Requires-Dist: tree-sitter-cpp>=0.23.0; extra == 'dev'
Requires-Dist: tree-sitter-go>=0.23.0; extra == 'dev'
Requires-Dist: tree-sitter-java>=0.23.0; extra == 'dev'
Requires-Dist: tree-sitter-javascript>=0.23.0; extra == 'dev'
Requires-Dist: tree-sitter-kotlin>=0.23.0; extra == 'dev'
Requires-Dist: tree-sitter-php>=0.23.0; extra == 'dev'
Requires-Dist: tree-sitter-python>=0.23.0; extra == 'dev'
Requires-Dist: tree-sitter-ruby>=0.23.0; extra == 'dev'
Requires-Dist: tree-sitter-rust>=0.23.0; extra == 'dev'
Requires-Dist: tree-sitter-swift>=0.0.1; extra == 'dev'
Requires-Dist: tree-sitter-typescript>=0.23.0; extra == 'dev'
Requires-Dist: tree-sitter>=0.23.0; extra == 'dev'
Provides-Extra: docs
Requires-Dist: mkdocs-material>=9.5.0; extra == 'docs'
Requires-Dist: mkdocs>=1.6.0; extra == 'docs'
Requires-Dist: mkdocstrings[python]>=0.25.0; extra == 'docs'
Requires-Dist: pymdown-extensions<12.2,>=10.0; (python_version < '3.11') and extra == 'docs'
Requires-Dist: pymdown-extensions>=10.0; (python_version >= '3.11') and extra == 'docs'
Provides-Extra: docx
Requires-Dist: python-docx>=1.0; extra == 'docx'
Provides-Extra: formats
Requires-Dist: pypdf>=3.0; extra == 'formats'
Requires-Dist: python-docx>=1.0; extra == 'formats'
Provides-Extra: graph
Provides-Extra: langchain
Requires-Dist: langchain-core<2,>=0.2.11; extra == 'langchain'
Requires-Dist: langchain-text-splitters<2,>=0.2.2; extra == 'langchain'
Provides-Extra: llamaindex
Requires-Dist: llama-index-core<0.15,>=0.10.68.post1; extra == 'llamaindex'
Provides-Extra: mcp
Provides-Extra: otel
Requires-Dist: opentelemetry-api>=1.21.0; extra == 'otel'
Provides-Extra: pdf
Requires-Dist: pypdf>=3.0; extra == 'pdf'
Provides-Extra: pgvector
Requires-Dist: psycopg[binary]<4,>=3.2.1; extra == 'pgvector'
Provides-Extra: pinecone
Provides-Extra: profiling
Requires-Dist: line-profiler>=4.1.0; extra == 'profiling'
Requires-Dist: py-spy>=0.3.14; extra == 'profiling'
Provides-Extra: qdrant
Requires-Dist: qdrant-client<2,>=1.12; extra == 'qdrant'
Provides-Extra: rust
Requires-Dist: maturin>=1.7.0; extra == 'rust'
Provides-Extra: scipy
Requires-Dist: scipy>=1.10.0; extra == 'scipy'
Provides-Extra: semantic
Requires-Dist: numpy>=1.24.0; extra == 'semantic'
Provides-Extra: sentence-transformers
Requires-Dist: sentence-transformers>=2.2.0; extra == 'sentence-transformers'
Provides-Extra: supabase
Provides-Extra: tiktoken
Requires-Dist: tiktoken>=0.5.0; extra == 'tiktoken'
Provides-Extra: transformers
Requires-Dist: transformers>=4.30.0; extra == 'transformers'
Provides-Extra: vectordb
Provides-Extra: weaviate
Description-Content-Type: text/markdown

<div align="center">
  <img src="https://raw.githubusercontent.com/oguzhankir/omnichunk/main/assets/omnichunk-logo.png" alt="omnichunk" width="360">
  <br><br>
  <a href="https://pypi.org/project/omnichunk/"><img src="https://img.shields.io/pypi/v/omnichunk?v=3" alt="PyPI"></a>
  <a href="https://github.com/oguzhankir/omnichunk/actions/workflows/ci.yml"><img src="https://github.com/oguzhankir/omnichunk/actions/workflows/ci.yml/badge.svg?v=3" alt="CI"></a>
  <a href="https://codecov.io/gh/oguzhankir/omnichunk"><img src="https://codecov.io/gh/oguzhankir/omnichunk/branch/main/graph/badge.svg?precision=2" alt="Codecov (Cobertura includes branch data from pytest --cov-branch)"></a>
  <a href="https://pypi.org/project/omnichunk/"><img src="https://img.shields.io/pypi/pyversions/omnichunk?v=3" alt="Python"></a>
  <a href="https://github.com/oguzhankir/omnichunk/blob/main/LICENSE"><img src="https://img.shields.io/pypi/l/omnichunk?v=3" alt="License"></a>
</div>

# Omnichunk

Structure-aware chunking and index preparation for RAG. Turn code, prose and
mixed documents into traceable chunks with explicit size budgets, source ranges
and context such as scopes, imports, headings and notebook cells.

**Our goal:** make reliable chunking the simplest part of building a retrieval
system. Preserve useful structure, explain every source span, update indexes
without stale records, and demonstrate quality on reproducible retrieval tasks.
The [roadmap](ROADMAP.md) defines the evidence required to get there.

**Status:** the current source version is **2.1.4**, with hardening changes tracked
in the [changelog](CHANGELOG.md). The package is classified as beta. Its enterprise
readiness and superiority to other chunkers are goals to validate, not current
certifications. Read the [migration guide](docs/migrations/v2.md) before upgrading
existing indexes.

[Documentation](https://oguzhankir.github.io/omnichunk/) ·
[Roadmap](ROADMAP.md) · [Examples](examples/README.md) ·
[Changelog](CHANGELOG.md) · [Contributing](CONTRIBUTING.md)

## Installation

Python 3.10 or newer:

```bash
pip install omnichunk
pip install "omnichunk[code]"           # six primary Tree-sitter grammars
pip install "omnichunk[tiktoken]"       # named OpenAI tokenizers
pip install "omnichunk[all-languages]"  # additional Tree-sitter grammars
pip install "omnichunk[formats]"        # PDF and DOCX text extraction
```

The base package has no mandatory third-party dependencies. Install `code` for
AST parsing and `semantic` for NumPy-based semantic utilities. Without a grammar,
code uses a diagnosed fallback. Structural chunking requires no external service;
optional model/tokenizer integrations may download assets or use your callbacks.

Other extras include `transformers`, `sentence-transformers`, `scipy`, `langchain`,
`llamaindex`, `qdrant`, `pgvector`, `otel`, `docs`, `dev` and optional Rust build tooling (`rust`).
See [installation details](docs/install.md) for the full grouping and limitations.

## Quick start

```python
from omnichunk import Chunker

source = 'import os\n\ndef hello(name: str) -> str:\n    return f"hello {name}"\n'
chunker = Chunker(max_chunk_size=256, size_unit="chars")
chunks = chunker.chunk("example.py", source)

for item in chunks:
    print(item.byte_range, item.context.breadcrumb)
    print(item.contextualized_text)
```

For token sizing, configure the tokenizer explicitly after installing its extra:

```python
chunker = Chunker(
    max_chunk_size=512,
    size_unit="tokens",
    tokenizer="cl100k_base",
    context_mode="full",
)
```

Every public chunking path validates the final rendered payload. The default
`overflow_policy="split"` enforces the selected budget; `"error"` rejects oversized
units and `"preserve"` explicitly reports structural overflow in metadata.
Context that will not fit is omitted by default, or rejected with
`context_overflow="error"`. Token mode requires an explicit tokenizer/counter;
`tokenizer="approximate"` is available as a clearly labelled estimate.

## Implemented capabilities

- **Code structure:** Tree-sitter boundaries, declarations, scope, sibling/import
  context and structural splitting. The `code` extra covers Python, JavaScript,
  TypeScript, Rust, Go and Java. Optional grammars cover C/C++/C#, Ruby, PHP, Kotlin,
  Swift, SQL, Bash, Scala and Elixir; extraction depth depends on the grammar.
- **Prose and mixed documents:** Markdown headings/lists/tables, delegated fenced
  code and markup, plaintext, JSON/YAML/TOML, HTML/XML and hybrid Python cells.
  Loaders also handle RST, notebooks, LaTeX and optional PDF/DOCX text extraction.
- **Semantic utilities:** caller-supplied embeddings, adaptive boundary detection,
  embedding cache/validation, local embedding helper, topic shifts, dense/sparse
  TF-IDF and MMR reranking. Structural engines still handle non-prose routing.
- **Retrieval utilities:** hierarchical chunk trees, incremental diffs, SQLite
  storage, token-budget selection, exact/near deduplication, entity–chunk graphs,
  centrality/community analysis and heuristic/callback-based propositions.
- **Integration:** synchronous/async/batch APIs, iterators, CLI, JSONL/CSV,
  native LangChain splitter and LlamaIndex node parser, document conversion,
  vendor-shaped vector payloads,
  parser/formatter registration and optional OpenTelemetry.
- **Recoverable indexing:** serializable change plans, a SQLite delivery journal,
  acknowledged batches, persistent embedding reuse and scoped deletion. Optional
  Qdrant and pgvector sinks resume pending writes after a process restart.

Use `language_capabilities()` to distinguish detected languages, installed
grammars, extraction queries and fallback behavior. The
[contract reference](docs/reference/contracts.md) defines guarantees and limits.
Graph helpers provide entity relationships; they are not a complete GraphRAG
retriever. PDF/DOCX support extracts text; it does not provide OCR or full layout.

For recoverable index updates, follow the [indexing guide](docs/indexing.md) and run
`python examples/14_recoverable_indexing.py`. It demonstrates a failed delivery,
restart/replay, metadata updates without re-embedding and deletion. Delivery is
at least once: applications own authorization, and sinks must make repeated
upserts/deletes safe. Acknowledged delivery does not make a multi-batch remote
update atomically visible to readers.

## Files, directories and CLI

```python
from omnichunk import Chunker

chunker = Chunker(max_chunk_size=512, size_unit="chars")
chunks = chunker.chunk_file("README.md")
results = chunker.chunk_directory("./src", glob="**/*.py", concurrency=4)

for result in results:
    if result.error:
        print(result.filepath, result.error)
    else:
        print(result.filepath, len(result.chunks))
```

Pass binary PDF/DOCX paths to `chunk_file()` rather than reading them as strings.

```bash
omnichunk ./src --glob "**/*.py" --max-size 512 --size-unit chars --format jsonl > chunks.jsonl
omnichunk README.md --max-size 256 --size-unit chars --stats
omnichunk README.md --format csv --output chunks.csv
omnichunk eval chunks.jsonl --metrics all
```

For reconstruction evaluation, `--source` must name the same canonical source
that produced the chunks; combining unrelated files with one source is invalid.
Current offline coherence/coverage metrics are heuristics, not retrieval scores.

For source-labeled retrieval evaluation, use the
[retrieval benchmark](benchmarks/README.md#retrieval-evaluation-and-tuning). It
compares chunking settings under shared budgets and reports ranking/evidence
metrics on a versioned, multilingual synthetic corpus. Tune on the development
split before evaluating held-out documents; the initial offline lexical baseline
does not establish production retrieval quality.

## Source ranges and context

Each `Chunk` contains `text`, `contextualized_text`, `byte_range`, `line_range`,
context metadata and size counts. Byte ranges are zero-based, half-open UTF-8
intervals. Line ranges are zero-based and inclusive. For ordinary text:

```python
chunks = chunker.chunk("example.py", source)
raw = source.encode("utf-8")
for item in chunks:
    assert raw[item.byte_range.start:item.byte_range.end].decode("utf-8") == item.text
```

Use chunks from the same `source` in this check. Structured loaders return ranges
in their canonical `LoadedDocument.text`, which may differ from notebook JSON or
original PDF/DOCX bytes. `contextualized_text` adds derived context and is not an
exact source slice. Default `coverage_policy="retrieval"` may omit trivia;
`chunk_with_manifest()` reports skipped byte ranges for canonical text.
`coverage_policy="lossless"` covers every canonical byte, including whitespace-only
input. With overlap disabled, concatenating lossless chunks reconstructs the
source. Each chunk records its logical source ID, revision and configuration
fingerprint. See the [source contract](docs/reference/contracts.md).

## Async, iterators and exports

```python
import asyncio
from omnichunk import Chunker

chunker = Chunker(max_chunk_size=256, size_unit="chars")
chunks = asyncio.run(chunker.achunk("example.py", source))

for item in chunker.stream("example.py", source):
    print(item.text)
```

Iterators accept a complete source string and retain source/AST state. `stream()`
and `chunk()` share boundaries, overlap and budget policies; streaming reports
`total_chunks=-1`. `astream(buffer_size=8)` bounds queued output and supports
cooperative cancellation. `stream_upsert()` keeps a bounded export batch and
streams ordinary-file chunks; structured extraction and semantic analysis still
retain per-document state. It produces rows for your database client.

Vector helpers produce plain data structures; compute embeddings and use the
vendor's client separately. JSONL/CSV and framework conversions are available on
`Chunker`; runnable demonstrations live in [examples](examples/README.md).

## Indexing and compatibility

Content/occurrence IDs and separate embedding fingerprints detect edits and
context/configuration changes. `ChunkStore` scopes data to named collections,
updates atomically and preserves last good state after failures. Apply diff
removals before additions; `updated` is an advisory subset of `added`.
Versioned JSON round-trips full chunk metadata. The
[migration guide](docs/migrations/v2.md) includes legacy conversion and separate
index rebuild/rollback examples.

The local store transaction does not include a remote vector database. Persist
pending deletions/upserts and retry them in your application; `stream_upsert()`
yields payloads and does not acknowledge database writes. Framework exports carry
native document IDs; Weaviate objects use deterministic UUIDs derived from chunk
IDs. Use `weaviate_object_id(chunk_id)` when deleting those objects. Existing
indexes created with autogenerated IDs need a rebuild or explicit reconciliation.

`TokenBudgetOptimizer` budgets `contextualized_text`, including derived context.
Reserve additional space for prompt instructions, citation labels and separators.
Token counts follow the configured chunking tokenizer; estimated counts remain
estimates and should not be treated as exact model limits.

Semantic caches isolate provider, model revision and preprocessing. Give opaque
callbacks an explicit stable fingerprint/revision for durable reuse. Parser and
formatter plugins can be scoped through `PluginRegistry`.

`omnichunk serve --rpc` exposes custom HTTP JSON-RPC with a resolved allowed root,
Host/Origin checks, request limits and loopback defaults. Non-loopback binding
requires authentication. The deprecated `--mcp` alias retains that RPC behavior;
standard MCP lifecycle support remains a later optional integration.

## Development and evaluation

```bash
pip install -e ".[dev,docs,langchain,llamaindex]"
pytest -q
ruff check src tests
ruff format --check src tests
mypy --python-version 3.12 src/omnichunk
python scripts/check_benchmarks.py --run-quality
mkdocs build --strict
```

Default pytest includes bounded Hypothesis source invariants and excludes only
tests marked `slow`. Coverage measures all Python source with a 90% aggregate
line-and-branch gate. The [roadmap](ROADMAP.md) separates local verification from
the release matrix and publication checks.

Match mypy's target to your Python interpreter; CI checks Python 3.10–3.13 with
the optional dependency versions installable on each interpreter.

[Benchmark instructions](benchmarks/README.md) explain reproducibility and the
current comparison limitations. No universal speed or retrieval-quality leadership
claim has been established.

Maintainer and project process are consolidated in [CONTRIBUTING.md](CONTRIBUTING.md).
See also the [security policy](SECURITY.md) and [code of conduct](CODE_OF_CONDUCT.md).
