Metadata-Version: 2.4
Name: fast-pruner
Version: 0.1.0
Summary: A structure-aware local prompt compressor.
Project-URL: Homepage, https://github.com/saurabhdorle/fast-pruner
Project-URL: Repository, https://github.com/saurabhdorle/fast-pruner
Project-URL: Issues, https://github.com/saurabhdorle/fast-pruner/issues
Project-URL: Changelog, https://github.com/saurabhdorle/fast-pruner/blob/main/CHANGELOG.md
License: Apache-2.0
License-File: LICENSE
Requires-Python: >=3.10
Requires-Dist: pydantic>=2.0.0
Requires-Dist: tiktoken>=0.6.0
Requires-Dist: tokenizers>=0.19.0
Provides-Extra: dev
Requires-Dist: black>=24.0.0; extra == 'dev'
Requires-Dist: fastapi>=0.110.0; extra == 'dev'
Requires-Dist: pytest-cov>=5.0.0; extra == 'dev'
Requires-Dist: pytest>=8.0.0; extra == 'dev'
Requires-Dist: uvicorn[standard]>=0.29.0; extra == 'dev'
Provides-Extra: fastapi
Requires-Dist: fastapi>=0.110.0; extra == 'fastapi'
Requires-Dist: uvicorn[standard]>=0.29.0; extra == 'fastapi'
Description-Content-Type: text/markdown

# ⚡ Fast-Pruner ⚡

[![PyPI Version](https://shields.io)](https://pypi.org)
[![License: Apache 2.0](https://shields.io)](https://opensource.org)
[![Build Status](https://shields.io)]()

`fast-pruner` is a **structure-aware, multi-strategy context optimizer and prompt compressor** designed for modern LLMOps pipelines. It intercepts requests locally, cleans unstructured conversational history arrays, protects strict rules/JSON configurations, and helps mitigate the "Lost in the Middle" attention degradation phenomenon—**with no LLM API calls required for the `EPHEMERAL_DROP` and `SEMANTIC_EXTRACTIVE` strategies, and no added per-token API cost**.

> **A note on "offline":** the compression pipeline itself never calls an external LLM (unless you opt into `LLM_GENERATIVE`). However, the default tokenizer (`tiktoken`) downloads its encoding files from a remote source the first time a given encoding is used, unless they're already cached locally. In fully air-gapped environments, pre-populate `TIKTOKEN_CACHE_DIR` before first use.

---

## 🔥 Key Technical Capabilities

* **🧠 Structure-Aware Context Firewall**: Automatically parses payloads to quarantine immutable boundaries (System guidelines, validation objects, Tool/JSON configurations) into an un-deletable safety buffer while isolating variable history logs for trimming.
* **🎯 Multi-Engine Token Pruning Pipeline**:
  * `EPHEMERAL_DROP`: A high-speed, chronological reverse-accumulation truncation engine that drops oldest turns while preserving recent transactional continuity.
  * `SEMANTIC_EXTRACTIVE`: A zero-cost, sentence-level relevance pruning engine that scores each sentence against the active user query. By default it uses **cosine similarity over word-overlap vectors** (lexical, not embedding-based — fast and dependency-free, but won't catch paraphrases with no shared vocabulary), followed by a factual-density filter that targets soft grammatical fillers while preserving numbers, symbols, and named entities. Pass an `embedding_fn` callback to swap in embedding-based (meaning-level) similarity from any model you already use — see [Embedding Injection](#4-embedding-based-relevance-scoring-semantic_extractive--embedding_fn) below.
* **🤖 Dependency-Injected Generative Condensation (`LLM_GENERATIVE`)**: Want abstractive summarization? Inject **any LLM** via a simple function callback — `fast-pruner` never imports a vendor SDK, so it works with any model you can call from Python. We suggest pointing this at a small, low-cost model like `gemini-2.5-flash` or `gpt-4o-mini` to keep summarization cheap, but that's a recommendation, not a requirement — see the **Model & Provider Compatibility** section below.
* **🏎️ Zero-Cost Offline Cross-Vendor Token Factory**: Built-in dynamic router estimates token counts across **OpenAI, Google Gemini, Anthropic Claude, and Hugging Face** in local memory. Since Claude's tokenizer isn't publicly available, counts for Claude models are approximated using an OpenAI-compatible encoding plus a **15% safety padding heuristic** — treat this as a conservative estimate for budgeting purposes, not an exact count. For precise Claude token counts, use [Anthropic's token counting API](https://platform.claude.com/docs/en/build-with-claude/token-counting).
* **🏗️ Attention Layout Engine Optimizer**: Solves performance decay inside deep context windows by mathematically sorting payload strings, anchoring system schemas at the absolute top index and the live user execution query at the bottom (maximizing LLM attention matching).

---

## 📦 System Architecture Layout

```text
       ┌────────────────────────────────────────────────────────┐
       │   Raw Inbound Payload Array (Universal JSON Format)    │
       └───────────────────────────┬────────────────────────────┘
                                   │
                                   ▼
                   ┌───────────────────────────────┐
                   │  ContextParser Framework      │
                   └───────┬───────────────┬───────┘
                           │               │
       [Protected Anchors] │               │ [Transient Backlog]
                           ▼               ▼
    ┌─────────────────────────────┐ ┌─────────────────────────────┐
    │ System Instructions & Rules │ │ Unstructured Chat History   │
    └──────────────┬──────────────┘ └──────────────┬──────────────┘
                   │                               │
                   │                               ▼
                   │                ┌─────────────────────────────┐
                   │                │  ContextCompressor Engine   │
                   │                │  (Local Cosine Vector Math) │
                   │                └──────────────┬──────────────┘
                   │                               │ [Pruned Backlog]
                   │                               ▼
                   │                ┌─────────────────────────────┐
                   │                │   Factual Token Extraction   │
                   │                └──────────────┬──────────────┘
                   │                               │
                   └───────────────┬───────────────┘
                                   │
                                   ▼
                   ┌───────────────────────────────┐
                   │    LayoutOptimizer Matrix     │
                   │  (Fixes "Lost in the Middle") │
                   └───────────────┬───────────────┘
                                   │
                                   ▼
       ┌────────────────────────────────────────────────────────┐
       │  Streamlined, High-Density API Wire Payload Array       │
       └────────────────────────────────────────────────────────┘
```

---

## 📦 Installation & Local Workspace Setup

Install the library directly via `pip` once published, or fetch it straight from your remote repository:

```bash
# Install directly from the PyPI open-source index
pip install fast-pruner

# Alternative: Install the absolute latest engineering commit via GitHub
pip install git+https://github.com/saurabhdorle/fast-pruner.git
```

For local development modifications:
```bash
git clone https://github.com/saurabhdorle/fast-pruner.git
cd fast-pruner
pip install -e ".[dev]"
```

---

## 🧩 Model & Provider Compatibility

`fast-pruner` interacts with LLMs and embedding models in two very different ways — one is fully open, the other has a fixed support list:

| Component | Model support | Why |
|---|---|---|
| **`llm_call` (`LLM_GENERATIVE` strategy)** | ✅ **Any LLM.** You pass a plain Python callback (`Callable[[str], str]`) — `fast-pruner` never imports a vendor SDK. Any model you can call from Python works: OpenAI, Anthropic, Gemini, Cohere, a local Ollama/vLLM model, a fine-tuned open-weights model, anything. | Dependency injection — `fast-pruner` just sends it a prompt string and reads back a string. |
| **`embedding_fn` (`SEMANTIC_EXTRACTIVE` + embeddings)** | ✅ **Any embedding model.** Same pattern: `Callable[[List[str]], List[List[float]]]`. OpenAI, Cohere, Voyage, a local `sentence-transformers`/`fastembed` model — all work. | Same dependency-injection design. |
| **`model_name` (token counting only)** | ⚠️ **Best-effort for a fixed set of families**, not universal. Model names containing `claude`, `gemini`, `llama`/`mistral`/`gemma`/`hf` are routed to a matching counter; everything else falls back to OpenAI's `cl100k_base`/`o200k_base` encoding via `tiktoken`. | Token counting needs to know the actual tokenizer/vocabulary a model uses — that can't be dependency-injected the same way, since it's about *how text is split*, not just *what function processes it*. |

**In short:** the model names you see in the examples below (`gpt-4o-mini`, `gemini-2.5-flash`, etc.) are just illustrative — they are **not** a restriction. Use whichever LLM or embedding model you already have in your pipeline for `llm_call` and `embedding_fn`.

The one place model choice *does* matter is `model_name` passed to `ContextCompressor(...)` — that only controls which token-counting heuristic is used for computing `max_history_tokens` budgets. If your model doesn't match one of the recognized name substrings above, token counts will silently fall back to an OpenAI-style estimate, which may be inaccurate for architecturally different tokenizers (this mainly affects the budget math, not the pruning/summarization quality itself). If you need precise counts for an unsupported model, count tokens yourself and pass a pre-computed budget, or contribute a new branch to `LocalTokenizerFactory` (see `CONTRIBUTING.md`).

---

## 🛠️ Quickstart Usage Implementations

### 1. Direct Script Pipeline Integration (Programmatic Choice Toggling)

```python
from fast_pruner import ContextParser, ContextCompressor, LayoutOptimizer, CompressionStrategy

raw_messages = [
    {"role": "system", "content": "You are a professional stock analysis assistant."},
    {"role": "system", "content": "CRITICAL CONFIGURATION: Always structure output matching: {'ticker': str, 'action': str}."},
    {"role": "user", "content": "I went out for lunch today and had an amazing sandwich. By the way, check the metrics for Apple."},
    {"role": "assistant", "content": "Apple is showing support boundaries at $175. It is cloudy in London today by the way."},
    {"role": "user", "content": "Analyze the Apple support lines."}
]

parsed = ContextParser.parse_messages(raw_messages)
compressor = ContextCompressor(model_name="claude-3-5-sonnet", relevance_threshold=0.25)

compressed = compressor.compress(
    parsed, 
    max_history_tokens=100, 
    strategy=CompressionStrategy.SEMANTIC_EXTRACTIVE
)

final_payload = LayoutOptimizer.assemble(compressed)
```

### 2. Live Automated FastAPI Middleware Integration

Requires the optional `fastapi` extra: `pip install fast-pruner[fastapi]`.

```python
import uvicorn
from fastapi import FastAPI, Request
from fast_pruner import FastPrunerMiddleware

app = FastAPI(title="Enterprise LLM Gateway Proxy")

app.add_middleware(
    FastPrunerMiddleware,
    max_history_tokens=300,
    strategy="semantic_extractive",
    model_name="gpt-4o"
)

@app.post("/v1/chat/completions")
async def chat_gateway_proxy(request: Request):
    payload = await request.json()
    return {"status": "success", "received_payload": payload["messages"]}
```

---

## 🔌 Ecosystem Integrations & Advanced Framework Patterns

### 1. LangChain Integration
`fast-pruner` seamlessly integrates with LangChain memory structures by processing raw dictionary exports and transforming them back into message wrapper tokens:

```python
from langchain_core.messages import messages_to_dict, messages_from_dict
from langchain_openai import ChatOpenAI
from fast_pruner import ContextParser, ContextCompressor, LayoutOptimizer

# 1. Fetch historical objects straight out of LangChain memory trackers
langchain_messages = memory.chat_history.messages
raw_dicts = [msg.dict() for msg in langchain_messages]

# 2. Execute Fast-Pruner local contextual refinement pipeline
parsed = ContextParser.parse_messages(raw_dicts)
compressor = ContextCompressor(model_name="gpt-4o")
compressed = compressor.compress(parsed, max_history_tokens=300, strategy="semantic_extractive")
optimized_dicts = LayoutOptimizer.assemble(compressed)

# 3. Restructure payload back to native LangChain classes for invocation
optimized_langchain_messages = messages_from_dict(optimized_dicts)
llm = ChatOpenAI(model="gpt-4o")
response = llm.invoke(optimized_langchain_messages)
```

### 2. LlamaIndex Integration
Refine dense vector context windows generated from high-volume RAG indexing queries right before handing the data array off to the agent execution track:

```python
from llama_index.core.base.llms.types import ChatMessage
from llama_index.llms.openai import OpenAI
from fast_pruner import ContextParser, ContextCompressor, LayoutOptimizer

# 1. Extract raw dictionary payloads out of standard LlamaIndex history states
raw_dicts = [msg.dict() for msg in llama_chat_history]

# 2. Run local context parsing and semantic chunk compression pass
parsed = ContextParser.parse_messages(raw_dicts)
compressor = ContextCompressor(model_name="gpt-4o")
compressed = compressor.compress(parsed, max_history_tokens=250, strategy="semantic_extractive")
optimized_dicts = LayoutOptimizer.assemble(compressed)

# 3. Map values back to target ChatMessage structures
optimized_llama_history = [ChatMessage(**msg) for msg in optimized_dicts]
llm = OpenAI(model="gpt-4o")
response = llm.chat(optimized_llama_history)
```

### 3. Real-Time Streaming Chat Implementation
Because the matrix optimizations execute completely offline in local memory in under 15 milliseconds, you can run `fast-pruner` directly inside token streaming loops without injecting initial response latency delays or delaying the user experience:

```python
import openai
from fast_pruner import ContextParser, ContextCompressor, LayoutOptimizer

async def stream_chat_gateway(full_chat_history: list):
    # Process and compress history before initiating the streaming connection
    parsed = ContextParser.parse_messages(full_chat_history)
    compressor = ContextCompressor()
    compressed = compressor.compress(parsed, max_history_tokens=500, strategy="semantic_extractive")
    optimized_payload = LayoutOptimizer.assemble(compressed)

    # Dispatch high-density, context-aware layout to OpenAI network streams
    response_stream = openai.chat.completions.create(
        model="gpt-4o",
        messages=optimized_payload,
        stream=True
    )

    # Stream individual chunks back to the client interface instantly
    for chunk in response_stream:
        if chunk.choices.delta.content:
            yield chunk.choices.delta.content
```

### 4. Embedding-Based Relevance Scoring (`SEMANTIC_EXTRACTIVE` + `embedding_fn`)
By default, `SEMANTIC_EXTRACTIVE` scores relevance using fast, dependency-free word-overlap vectors. If you already have an embedding model in your pipeline — OpenAI, Cohere, a local `sentence-transformers`/`fastembed` model, or anything else — you can inject it to get true meaning-level (not just literal word-overlap) relevance scoring. `fast-pruner` never imports an embedding library itself; you bring the function.

The callback signature is `Callable[[List[str]], List[List[float]]]` — it receives a batch of texts and must return one vector per text, **in the same order**. `fast-pruner` batches the active query together with every sentence in the history into a single call, so you get one round-trip instead of N.

```python
from fast_pruner import ContextParser, ContextCompressor, LayoutOptimizer, CompressionStrategy
import openai

client = openai.OpenAI()

def openai_embedding_fn(texts: list[str]) -> list[list[float]]:
    response = client.embeddings.create(model="text-embedding-3-small", input=texts)
    return [item.embedding for item in response.data]

parsed = ContextParser.parse_messages(raw_messages)
compressor = ContextCompressor(model_name="gpt-4o", relevance_threshold=0.4)

compressed = compressor.compress(
    parsed,
    max_history_tokens=300,
    strategy=CompressionStrategy.SEMANTIC_EXTRACTIVE,
    embedding_fn=openai_embedding_fn
)

final_payload = LayoutOptimizer.assemble(compressed)
```

**A note on `relevance_threshold`:** the default (`0.25`) was tuned for word-overlap vectors. Embedding models typically produce a narrower cosine-similarity band for related-but-not-identical sentences (often ~0.3–0.6, depending on the model), so you'll likely want to tune `relevance_threshold` when switching from the default lexical scorer to embeddings — start around `0.4`–`0.5` and adjust based on your own compression/recall tradeoff.

### 5. Advanced Pattern: Generative LLM Condensation Engine (`LLM_GENERATIVE`)
If your application requires abstractive context rewriting rather than extraction filters, you can leverage the `LLM_GENERATIVE` strategy. By using dependency injection, you pass a simple lambda or custom execution wrapper callback. We strongly recommend pointing this to an ultra-fast, low-cost model like `gemini-2.5-flash`.

Here is a full production example using the official Google GenAI SDK (`google-genai`):

```python
import os
from google import genai
from fast_pruner import ContextParser, ContextCompressor, LayoutOptimizer, CompressionStrategy

# 1. Initialize the official Google GenAI Client
client = genai.Client(api_key=os.environ["GEMINI_API_KEY"])

# 2. Define the exact execution wrapper signature expected by Fast-Pruner
def gemini_condensation_runner(prompt_text: str) -> str:
    response = client.models.generate_content(
        model='gemini-2.5-flash',
        contents=prompt_text,
    )
    return response.text

# 3. Load your conversational chat history array
raw_history = [
    {"role": "user", "content": "I went out for a long lunch today. Let's look at the financial profiles for Google."},
    {"role": "assistant", "content": "Alphabet Inc. (GOOGL) is demonstrating strong trend momentum. Also, it is very warm today in Paris."},
    {"role": "user", "content": "Check the GOOGL indicators."}
]

parsed = ContextParser.parse_messages(raw_history)
compressor = ContextCompressor(model_name="gemini-1.5-pro")

# 4. Inject the runner function cleanly directly into the compression pipeline loop
compressed_ctx = compressor.compress(
    parsed,
    max_history_tokens=200,
    strategy=CompressionStrategy.LLM_GENERATIVE,
    llm_call=gemini_condensation_runner  # High-precision callback mapping
)

# 5. Pack layout for final delivery to your core application track
final_payload = LayoutOptimizer.assemble(compressed_ctx)
```

---

### 💡 Short-Form Ecosystem Adaptations (`LLM_GENERATIVE`)

If you are already running standard orchestrators, use these inline snippets to inject your existing model dependencies directly into the compression pass:

* **LangChain Inline**:
```python
compressed = compressor.compress(parsed, max_history_tokens=300, strategy=CompressionStrategy.LLM_GENERATIVE, llm_call=lambda p: chat_model.invoke(p).content)
```

* **LlamaIndex Inline**:
```python
compressed = compressor.compress(parsed, max_history_tokens=250, strategy=CompressionStrategy.LLM_GENERATIVE, llm_call=lambda p: str(llama_llm.complete(p)))
```

* **Standard / Streaming Python SDKs**:
```python
compressed = compressor.compress(parsed, max_history_tokens=500, strategy=CompressionStrategy.LLM_GENERATIVE, llm_call=your_custom_llm_runner)
```

---

## 📈 Real-Time Token Analytics Dashboard
`fast-pruner` incorporates built-in console tracking parameters that trigger metrics transparency logs automatically upon compilation:

```text
--------------------------------------------------
FAST-PRUNER CONTEXT METRICS ENGINE REPORT
 -> Strategy Executed   : semantic_extractive
 -> Original History    : 68 tokens
 -> Optimized History   : 9 tokens
 -> Tokens Extracted    : 59 tokens
 -> Footprint Reduction : 86.76%
--------------------------------------------------
```

---

## 🧪 Operational Automated Verification Suite

To verify performance alignments, execute our synchronized testing framework directly using native shell calls:

```bash
python -m pytest tests/ --noconftest -o cache_dir=%TEMP% -s
```

## 🤝 Contributing
Contributions are welcome — see [`CONTRIBUTING.md`](CONTRIBUTING.md) for setup, testing, and PR guidelines.

## 📝 Changelog
See [`CHANGELOG.md`](CHANGELOG.md) for release history.

## 📄 License
Distributed under the Apache License 2.0. See `LICENSE` for details.
