Metadata-Version: 2.5
Name: piighost
Version: 1.7.1
Summary: Protect personal data (PII) in your LLM prompts. piighost hides sensitive values from the model, then restores the real values in the response, so tools and the user still get the real data. Pluggable detectors (regex, NER, LLM), LangChain and Pydantic AI integrations, and a companion OpenAI/Anthropic proxy.
Project-URL: Homepage, https://github.com/Athroniaeth/piighost
Project-URL: Documentation, https://athroniaeth.github.io/piighost/
Project-URL: Repository, https://github.com/Athroniaeth/piighost
Project-URL: Issues, https://github.com/Athroniaeth/piighost/issues
Project-URL: Changelog, https://github.com/Athroniaeth/piighost/blob/master/CHANGELOG.md
Author: Athroniaeth
License-Expression: MIT
License-File: LICENSE
Keywords: agents,ai,anonymization,de-identification,gliner,langchain,langgraph,llm,ner,nlp,pii,presidio,privacy,pseudonymization,spacy
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Security
Classifier: Topic :: Text Processing :: Linguistic
Classifier: Typing :: Typed
Requires-Python: >=3.11
Requires-Dist: typing-extensions>=4.16.0
Provides-Extra: all
Requires-Dist: aiosqlite>=0.19; extra == 'all'
Requires-Dist: argon2-cffi>=23.1; extra == 'all'
Requires-Dist: asyncpg>=0.29; extra == 'all'
Requires-Dist: cryptography>=44; extra == 'all'
Requires-Dist: gliner2>=1.2; extra == 'all'
Requires-Dist: httpx>=0.28; extra == 'all'
Requires-Dist: langchain-core>=0.3; extra == 'all'
Requires-Dist: langchain>=1.2; extra == 'all'
Requires-Dist: llama-index-core>=0.14; extra == 'all'
Requires-Dist: mistralai>=1.0; extra == 'all'
Requires-Dist: opentelemetry-api>=1.30; extra == 'all'
Requires-Dist: peft<1,>=0.10; extra == 'all'
Requires-Dist: presidio-analyzer>=2.2; extra == 'all'
Requires-Dist: pydantic-ai>=2.0; extra == 'all'
Requires-Dist: pydantic-settings>=2.0; extra == 'all'
Requires-Dist: pydantic>=2.6; extra == 'all'
Requires-Dist: rapidfuzz>=3.9; extra == 'all'
Requires-Dist: redis>=5; extra == 'all'
Requires-Dist: spacy>=3.7; extra == 'all'
Requires-Dist: sqlalchemy[asyncio]>=2.0; extra == 'all'
Requires-Dist: torch>=2.0; extra == 'all'
Requires-Dist: transformers>=4.40; extra == 'all'
Requires-Dist: typer>=0.12; extra == 'all'
Provides-Extra: argon2
Requires-Dist: argon2-cffi>=23.1; extra == 'argon2'
Provides-Extra: client
Requires-Dist: httpx>=0.28; extra == 'client'
Provides-Extra: config
Requires-Dist: pydantic-settings>=2.0; extra == 'config'
Requires-Dist: pydantic>=2.6; extra == 'config'
Requires-Dist: typer>=0.12; extra == 'config'
Provides-Extra: crypto
Requires-Dist: cryptography>=44; extra == 'crypto'
Provides-Extra: fuzzy
Requires-Dist: rapidfuzz>=3.9; extra == 'fuzzy'
Provides-Extra: gliner2
Requires-Dist: gliner2>=1.2; extra == 'gliner2'
Requires-Dist: peft<1,>=0.10; extra == 'gliner2'
Provides-Extra: langchain
Requires-Dist: langchain>=1.2; extra == 'langchain'
Provides-Extra: llama-index
Requires-Dist: llama-index-core>=0.14; extra == 'llama-index'
Provides-Extra: llm
Requires-Dist: langchain-core>=0.3; extra == 'llm'
Requires-Dist: langchain>=1.2; extra == 'llm'
Provides-Extra: middleware
Requires-Dist: langchain>=1.2; extra == 'middleware'
Provides-Extra: mistral
Requires-Dist: mistralai>=1.0; extra == 'mistral'
Provides-Extra: observation
Requires-Dist: opentelemetry-api>=1.30; extra == 'observation'
Provides-Extra: presidio
Requires-Dist: presidio-analyzer>=2.2; extra == 'presidio'
Provides-Extra: pydantic-ai
Requires-Dist: pydantic-ai>=2.0; extra == 'pydantic-ai'
Provides-Extra: redis
Requires-Dist: redis>=5; extra == 'redis'
Provides-Extra: spacy
Requires-Dist: spacy>=3.7; extra == 'spacy'
Provides-Extra: sqlalchemy
Requires-Dist: aiosqlite>=0.19; extra == 'sqlalchemy'
Requires-Dist: asyncpg>=0.29; extra == 'sqlalchemy'
Requires-Dist: sqlalchemy[asyncio]>=2.0; extra == 'sqlalchemy'
Provides-Extra: transformers
Requires-Dist: torch>=2.0; extra == 'transformers'
Requires-Dist: transformers>=4.40; extra == 'transformers'
Description-Content-Type: text/markdown

# PIIGhost

[![CI](https://github.com/Athroniaeth/piighost/actions/workflows/ci.yml/badge.svg)](https://github.com/Athroniaeth/piighost/actions/workflows/ci.yml)
[![codecov](https://codecov.io/gh/Athroniaeth/piighost/branch/master/graph/badge.svg)](https://codecov.io/gh/Athroniaeth/piighost)
[![PyPI version](https://img.shields.io/pypi/v/piighost.svg)](https://pypi.org/project/piighost/)
[![Python versions](https://img.shields.io/pypi/pyversions/piighost.svg)](https://pypi.org/project/piighost/)
[![License: MIT](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)
[![Security: bandit](https://img.shields.io/badge/security-bandit-yellow.svg)](https://github.com/PyCQA/bandit)
[![Discord](https://img.shields.io/badge/Discord-join-5865F2?logo=discord&logoColor=white)](https://discord.gg/vFg9GHQR2s)

`piighost` is a Python library that protects your personal data (PII) in conversations with LLMs through de-identification. Sensitive values are hidden before they are sent, then restored in the response. LangChain, Pydantic AI, LlamaIndex and Claude Code integrations are provided, together with an OpenAI and Anthropic API connector.

This de-identification spots PII with pluggable detectors (regex, NER, LLM) and replaces each value with a placeholder, the token that takes its place. For example:

- `John Doe` becomes `<<PERSON:1>>`
- `john.doe@example.com` becomes `<<EMAIL:1>>`

This placeholder stays the same from one message to the next with the conversational pipeline, which keeps the mapping between a value and its placeholder across the whole conversation. If `john.doe@example.com` reappears three messages later, the placeholder is still `<<EMAIL:1>>`, which lets the LLM follow the thread.

The LLM therefore only receives de-identified text. When it returns placeholders, for example by answering `Hello <<PERSON:1>>`, `piighost` replaces them with the real values. The user sees `John Doe` and never sees the de-identification.

The same mechanism protects agents that call tools. With the LangChain middleware, a tool that needs the real email address receives it in clear, while the LLM that supplies it only writes `<<EMAIL:1>>`.

<p align="center">
  <picture>
    <source media="(prefers-color-scheme: dark)" srcset="docs/assets/deid-chat-dark.gif">
    <img alt="A user chats with an agent, PII values are replaced by placeholders before reaching the LLM and restored afterwards for the user and for tool calls." src="docs/assets/deid-chat-light.gif" width="760">
  </picture>
</p>

*The LLM only sees placeholders. The tool receives the real address, the user gets a clear-text reply, and your agent code stays the same.*

> [!NOTE]
> This retained mapping makes the de-identification a pseudonymization under the GDPR, not a definitive anonymization. With the conversational pipeline, the real values stay stored for the duration of the conversation and must be protected accordingly.

## Why PIIGhost

Most PII tooling stops at detection. Presidio, GLiNER, spaCy, and regex catalogs all find entities in text, and they do it well. The hard part for an LLM agent is everything after detection: swapping values without wrecking the model's reasoning, keeping one value mapped to one token across a conversation, handing tools the real value while the model sees only the token, and putting the originals back in the reply. That orchestration is what `piighost` is.

**What `piighost` adds on top:**

- **Pluggable detectors:** regex catalogs (generic, US, EU, FR), NER (GLiNER2, spaCy, Transformers), an LLM detector, plus exact-match, composite, and chunked detectors (chunking splits text that overruns a model's context window), and you keep the one you trust (Presidio plugs in through an extra).
- **Reversible, transparent tokens:** each value becomes a stable id like `<<PERSON:1>>` and is put back automatically, so the end user reads `john.doe@example.com` and never sees a token. Label-only, masked, and keyed-hash factories are available too.
- **Consistent across a conversation:** the same value keeps the same token for the whole thread, backed by in-process, Redis, or SQLAlchemy memory (Redis and SQL can encrypt values at rest and hash keys).
- **Agent integrations with a tool boundary:** LangChain middleware, Pydantic AI hooks, and LlamaIndex. The tool receives the real value while the model sees only the token, with token-by-token streaming restoration.
- **A customizable staged pipeline:** detect, link, resolve overlaps, expand, de-identify, and an optional guard rail that refuses a reply with residual PII (a detector, an LLM, or Mistral moderation). Swap in fuzzy matching to tolerate typos or add your own stage.
- **Config-driven and self-hostable:** build a whole pipeline from a TOML/JSON file with a CLI to validate it, run it in your process, or as a service through the companion [piighost-api](https://github.com/Athroniaeth/piighost-api) (OpenAI- and Anthropic-compatible proxies).
- **Typed and observable:** ships `py.typed` and a minimal core with everything heavy behind extras, plus OpenTelemetry per-stage spans (viewable in Langfuse or Jaeger) with optional payload redaction.
- **Scope, live text and conversations:** `piighost` protects a running conversation message by message, not a static dataset.

For how it stacks up against Presidio, LangChain, the cloud APIs, and others, see [How PIIGhost compares](https://athroniaeth.github.io/piighost/comparison/).

### Limitations and trade-offs

- **The token does not embed the encrypted value, on purpose.** Unlike a format-preserving encryption token (where the ciphertext *is* the token, e.g. Google DLP), `piighost` uses an id (`<<PERSON:1>>`) backed by a cache. The reason: a token that carries the ciphertext can be captured today and cracked in 20 years ("harvest now, decrypt later", the quantum threat to classical crypto), whereas an id reveals nothing on its own. In return, you need a cache to hold the token-to-value mapping, so a memory backend to deploy, share across workers, and persist in production.
- **That cache stores the real values, so reversibility is pseudonymization, not anonymization (GDPR).** The real values stay stored for the duration of the conversation. The library gives you the means to protect them (AES-GCM encryption of the values, Argon2id hashing of the keys), but the database architecture itself must be secured in production once you use Redis or PostgreSQL.
- **No dataset anonymization.** No k-anonymity, l-diversity, differential privacy, or tabular data. `piighost` protects live text and conversations, not a whole dataset. For that, see ARX, Amnesia, or Google DLP.
- **No checksum validation (Luhn / IBAN / NIR), by choice.** The `RegexDetector` matches on shape alone so it never lets a real value mangled by OCR leak (a checksum would reject it and it would pass in clear). In exchange, it sometimes flags a string that only looks like PII, which costs nothing beyond one extra token.

## Quickstart

```bash
pip install piighost   # or: uv add piighost
```

### De-identify a text

`ExactMatchDetector` de-identifies a dictionary of known values without downloading a model.

```python
import asyncio

from piighost.components.detector import ExactMatchDetector
from piighost.pipeline import AnonymizationPipeline

detector = ExactMatchDetector({"John Doe": "PERSON", "john.doe@example.com": "EMAIL"})
pipeline = AnonymizationPipeline(detector)

result = asyncio.run(pipeline.anonymize("Write to John Doe at john.doe@example.com."))
print(result.text)  # Write to <<PERSON:1>> at <<EMAIL:1>>.
```

### Conversations and agents (LangChain)

The middleware wraps a conversational pipeline and handles every agent turn for you, so the same de-identification applies without any change to your agent logic.

```bash
pip install 'piighost[langchain]'   # or: uv add 'piighost[langchain]'
```

```python
import asyncio

from langchain.agents import create_agent
from langchain.chat_models import init_chat_model
from langchain_core.messages import HumanMessage
from langchain_core.tools import tool

from piighost.components.detector import ExactMatchDetector
from piighost.integrations.langchain import PIIAnonymizationMiddleware
from piighost.pipeline import ThreadAnonymizationPipeline

SYSTEM_PROMPT = (
    "Some inputs contain placeholders like <<PERSON:1>> that stand in for real "
    "values withheld for privacy. Treat each placeholder as the real value, never "
    "comment on its format, and pass it to tools unchanged."
)


@tool
def send_mail(to: str, body: str) -> str:
    """Send an email to `to` with the given body."""
    print(f"[tool] send_mail received to={to!r}")
    return "Email successfully sent."


async def main() -> None:
    # This example calls OpenAI, so set OPENAI_API_KEY in your environment first.
    labels = {"Patrick Dupont": "PERSON", "patrick@acme.com": "EMAIL"}
    detector = ExactMatchDetector(labels)
    pipeline = ThreadAnonymizationPipeline(detector)
    middleware = PIIAnonymizationMiddleware(pipeline)
    # gpt-5.6-terra is a reasoning model; reasoning_effort="none" lets it call
    # function tools over chat/completions.
    model = init_chat_model("openai:gpt-5.6-terra", reasoning_effort="none")
    # The system prompt tells the model to treat placeholders as real values and
    # pass them to tools unchanged, so it does not balk at the tokens.
    agent = create_agent(
        model=model,
        system_prompt=SYSTEM_PROMPT,
        tools=[send_mail],
        middleware=[middleware],
    )
    config = {"configurable": {"thread_id": "demo-thread"}}

    message = HumanMessage(
        "Use the send_mail tool to send a welcome note to Patrick Dupont at patrick@acme.com."
    )
    result = await agent.ainvoke({"messages": [message]}, config=config)
    print(f"user sees: {result['messages'][-1].content!r}")


if __name__ == "__main__":
    asyncio.run(main())
```

This is the **LangChain** integration, but it is only one option. `piighost` also has connectors for [Pydantic AI](https://athroniaeth.github.io/piighost/examples/pydantic-ai/) and [LlamaIndex](https://athroniaeth.github.io/piighost/examples/llama-index/), and the companion [piighost-api](https://github.com/Athroniaeth/piighost-api) exposes OpenAI- and Anthropic-compatible proxies, so you can move de-identification to the HTTP boundary with only a base URL change.

For a real detector and the conversational pipeline, see the [Quickstart](https://athroniaeth.github.io/piighost/getting-started/quickstart/) and the [LangChain integration](https://athroniaeth.github.io/piighost/examples/langchain/).

## Documentation

**[Full documentation](https://athroniaeth.github.io/piighost/)**

<details>
<summary>Browse the docs by section</summary>

- **Get started**
    - [installation](https://athroniaeth.github.io/piighost/getting-started/installation/)
    - [quickstart](https://athroniaeth.github.io/piighost/getting-started/quickstart/)
    - [first pipeline](https://athroniaeth.github.io/piighost/getting-started/first-pipeline/)
- **How-to**
    - [basic usage](https://athroniaeth.github.io/piighost/examples/basic/)
    - [LangChain integration](https://athroniaeth.github.io/piighost/examples/langchain/)
    - [Pydantic AI integration](https://athroniaeth.github.io/piighost/examples/pydantic-ai/)
    - [ready-made detectors](https://athroniaeth.github.io/piighost/examples/detectors/)
- **Reference**
    - [pipeline](https://athroniaeth.github.io/piighost/reference/pipeline/)
    - [middleware](https://athroniaeth.github.io/piighost/reference/langchain/)
    - [detectors](https://athroniaeth.github.io/piighost/reference/detectors/)
    - [CLI](https://athroniaeth.github.io/piighost/reference/cli/)
- **Concepts**
    - [why de-identify](https://athroniaeth.github.io/piighost/why-anonymize/)
    - [architecture](https://athroniaeth.github.io/piighost/architecture/)
    - [placeholder factories](https://athroniaeth.github.io/piighost/placeholder-factories/)
    - [security](https://athroniaeth.github.io/piighost/security/)

</details>

## Project

- **Community**: [Discord](https://discord.gg/vFg9GHQR2s) to get help, report bugs, request features, and discuss de-identification
- **Contributing**: [contribution guide](https://athroniaeth.github.io/piighost/community/contributing/) and [report a bug](https://athroniaeth.github.io/piighost/community/bug-reports/)
- **Ecosystem**:
    - **[Presentation site](https://piighost.athroniaeth.cloud/?utm_source=github&utm_medium=readme&utm_campaign=piighost)**: an overview of the project
    - **[piighost-api](https://github.com/Athroniaeth/piighost-api)**: the inference API server
    - **[piighost-chat](https://github.com/Athroniaeth/piighost-chat)**: an example chat interface with human-in-the-loop
- **License**: [MIT](LICENSE)
