Metadata-Version: 2.5
Name: counterfix
Version: 0.0.1
Summary: Find why an LLM call failed and get a fix that was tested to work.
Project-URL: Homepage, https://github.com/KANSADWALA/CounterFix
Project-URL: Issues, https://github.com/KANSADWALA/CounterFix/issues
Author: Shubham Kansadwala
License-Expression: MIT
License-File: LICENSE
Keywords: counterfactual,debugging,evaluation,hallucination,llm,reliability
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Requires-Dist: json-repair>=0.30
Requires-Dist: jsonschema>=4.20
Requires-Dist: pydantic>=2.7
Requires-Dist: tenacity>=8.2
Provides-Extra: anthropic
Requires-Dist: anthropic>=0.40; extra == 'anthropic'
Provides-Extra: cache
Requires-Dist: diskcache>=5.6; extra == 'cache'
Provides-Extra: cli
Requires-Dist: rich>=13.7; extra == 'cli'
Requires-Dist: typer>=0.12; extra == 'cli'
Provides-Extra: dev
Requires-Dist: mypy>=1.11; extra == 'dev'
Requires-Dist: pytest-recording>=0.13; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Provides-Extra: jev
Requires-Dist: typesafe-sdk>=0.7; extra == 'jev'
Provides-Extra: langchain
Requires-Dist: langchain-core>=0.3; extra == 'langchain'
Provides-Extra: litellm
Requires-Dist: litellm>=1.50; extra == 'litellm'
Provides-Extra: llamaindex
Requires-Dist: llama-index-core>=0.11; extra == 'llamaindex'
Provides-Extra: openai
Requires-Dist: openai>=1.40; extra == 'openai'
Provides-Extra: otel
Requires-Dist: opentelemetry-sdk>=1.27; extra == 'otel'
Provides-Extra: report
Requires-Dist: jinja2>=3.1; extra == 'report'
Provides-Extra: semantic
Requires-Dist: torch>=2.2; extra == 'semantic'
Requires-Dist: transformers>=4.44; extra == 'semantic'
Description-Content-Type: text/markdown

# counterfix

**Find why an LLM call failed, and get a fix that was tested to work.**

Most LLM evaluation tools tell you *that* a model failed. counterfix is built to tell you
*why*, and to hand back a fix it has already checked. When a call fails, it changes one
thing at a time (adds context, removes a false premise, rewords the prompt, gives the
model a tool, raises `max_tokens`) and re-runs the call. The change that fixes the
failure is both the diagnosis and the prescription. We call this **counterfactual
diagnosis**.

> **Status: pre-alpha (v0.0.1).** The core records, the `guard()` wrapper and the seed
> failure set are in place. Detectors and the counterfactual engine are being built.
> APIs will change.

## What it plugs into

counterfix is middleware: it sits between your code and any text-in, text-out LLM.

```python
from openai import OpenAI
from counterfix import guard

client = guard(OpenAI())  # same API as before
response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Why did Einstein win his third Nobel Prize?"}],
)
print(response.choices[0].message.content)
print(response.diagnosis.summary())  # what counterfix found
```

The same wrapper works with any OpenAI-compatible endpoint, for example Groq or a local
Ollama server:

```python
client = guard(OpenAI(base_url="http://localhost:11434/v1", api_key="ollama"))
```

Design rules:

- **Fail open.** If counterfix itself errors, your call still returns normally.
- **Your errors stay yours.** Exceptions from the provider reach your code unchanged.
- **Safe defaults.** Live mode only flags failures and makes no extra LLM calls.
- **Light core.** The core depends on 4 small libraries; providers and heavy tools are extras.

## Failure types (v1)

| Code | Failure |
| --- | --- |
| F1 | Truncation |
| F2 | Format break (JSON, schema) |
| F3 | Over-refusal |
| F4 | Hallucination from missing knowledge |
| F5 | False-premise acceptance |
| F6 | Dropped instruction |
| F7 | Prompt fragility |
| F8 | Computation or token-level error |
| F9 | Ambiguous request |

API errors, rate limits and provider content filters are tracked separately, so they are
never counted as the model's fault.

## Install (development)

```bash
git clone https://github.com/KANSADWALA/CounterFix.git
cd CounterFix
pip install -e ".[dev]"
pytest
```

Optional extras: `openai`, `anthropic`, `litellm`, `langchain`, `llamaindex`, `otel`,
`semantic`, `cache`, `cli`, `report`, `jev`.

## Roadmap

1. Foundations and seed failure set ✅
2. Core records and the `guard()` plug ✅
3. Cheap live detectors (F1 to F3) and the prompt linter
4. LLM-judge and sampling detectors
5. Diagnose mode and HTML report
6. Counterfactual engine with verified fixes
7. Model failure profiles
8. More plugs: decorator, Anthropic, LiteLLM, LangChain, OpenTelemetry
9. Evaluation on public benchmarks
10. Release and paper

## License

MIT
