Metadata-Version: 2.4
Name: fpf-chunking
Version: 0.3.0
Summary: FPF-guided (holon / bounded-context) semantic document chunking
Author: Valentin Alekseev
Author-email: Valentin Alekseev <valentin.alekseev@gmail.com>
License-Expression: MIT
Requires-Dist: anthropic>=0.120.2
Requires-Dist: openai>=2.50.0
Requires-Dist: google-genai>=1.0.0 ; extra == 'gemini'
Requires-Python: >=3.11
Provides-Extra: gemini
Description-Content-Type: text/markdown

# fpf-chunking

FPF-guided (holon / bounded-context) semantic document chunking: parses a
document's structure, uses an LLM to validate which structural boundaries
make good chunk boundaries, and assembles the result into token-bounded
chunks — without ever exceeding the LLM's context window on a single
oversized structural unit.

## Install

    pip install fpf-chunking

Add `[gemini]` if you want to use Gemini (`pip install fpf-chunking[gemini]`)
— Anthropic and OpenAI support is included in the base install.

## Usage

Boundary validation and embeddings each take a small adapter that wraps
your provider's real SDK client — see `fpf_chunking.adapters` for
`AnthropicAdapter`, `OpenAIAdapter`, and `GeminiAdapter`. Mix and match
providers freely; nothing in the pipeline is tied to a specific one.
`OpenAIAdapter` implements both `CompletionClient` (`.generate()`, chat
completions) and `EmbeddingClient` (`.embed()`) — use one instance per
role, or the same instance for both if the same model serves them.

From a file, using Anthropic for boundary validation and OpenAI for
embeddings:

```python
from anthropic import Anthropic
from openai import OpenAI
from fpf_chunking import MarkdownLoader, chunk_document, embed
from fpf_chunking.adapters import AnthropicAdapter, OpenAIAdapter

doc = MarkdownLoader().load("path/to/doc.md")

completion_client = AnthropicAdapter(
    Anthropic(api_key="..."), model="claude-sonnet-4-5-20250929"
)
chunks = chunk_document(doc, completion_client=completion_client, max_chunk_tokens=400)

embedding_client = OpenAIAdapter(OpenAI(api_key="..."), model="text-embedding-3-small")
embedded = embed(chunks, client=embedding_client)
```

`OpenAIAdapter` also works unmodified against anything that speaks the
OpenAI-compatible chat/embeddings wire format — e.g. a
[LiteLLM](https://github.com/BerriAI/litellm) proxy in front of Gemini,
Anthropic, or any other provider LiteLLM supports:

```python
from openai import OpenAI
from fpf_chunking import chunk_document
from fpf_chunking.adapters import OpenAIAdapter

# base_url points at your LiteLLM proxy; api_key is LiteLLM's own proxy
# key (master or virtual), not a real provider credential -- LiteLLM
# holds the actual upstream key itself, server-side.
client = OpenAI(base_url="http://litellm:4000", api_key="sk-litellm-...")
completion_client = OpenAIAdapter(client, model="gemini-3.6-flash")  # LiteLLM model_list name

chunks = chunk_document(doc, completion_client=completion_client, max_chunk_tokens=400)
```

From content already in memory (e.g. fetched over the network by your own
wrapper — no filesystem path required):

```python
from fpf_chunking import Document, chunk_document

doc = Document(doc_id="my-doc", text=buffer_from_network)
chunks = chunk_document(doc, completion_client=completion_client, max_chunk_tokens=400)
```

Or run the whole pipeline on Gemini, sharing one client across both roles:

```python
from google import genai
from fpf_chunking import chunk_document, embed
from fpf_chunking.adapters import GeminiAdapter

client = genai.Client(api_key="...")
completion_client = GeminiAdapter(client, model="gemini-2.5-flash")
embedding_client = GeminiAdapter(client, model="gemini-embedding-001")

chunks = chunk_document(doc, completion_client=completion_client, max_chunk_tokens=400)
embedded = embed(chunks, client=embedding_client)
```

See the module docstrings for the full API: `parse_holons`,
`extract_candidate_boundaries`, `split_oversized_holons`, `assemble_chunks`,
`FPFBoundaryValidator`, `fixed_size_chunks` (naive baseline), and
`semantic_chunks` (embedding-similarity baseline) are all exported for
callers who want the individual pipeline stages instead of the composed
`chunk_document`. `CompletionClient`/`EmbeddingClient` (also exported) are
the two protocols every adapter implements, if you want to write your own
for another provider.
