Metadata-Version: 2.5
Name: counterfix
Version: 0.1.0
Summary: Find why an LLM call failed and get a fix that was tested to work.
Project-URL: Homepage, https://github.com/KANSADWALA/CounterFix
Project-URL: Issues, https://github.com/KANSADWALA/CounterFix/issues
Author: Shubham Kansadwala
License-Expression: MIT
License-File: LICENSE
Keywords: counterfactual,debugging,evaluation,hallucination,llm,reliability
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Requires-Dist: json-repair>=0.30
Requires-Dist: jsonschema>=4.20
Requires-Dist: pydantic>=2.7
Requires-Dist: tenacity>=8.2
Provides-Extra: anthropic
Requires-Dist: anthropic>=0.40; extra == 'anthropic'
Provides-Extra: cache
Requires-Dist: diskcache>=5.6; extra == 'cache'
Provides-Extra: cli
Requires-Dist: rich>=13.7; extra == 'cli'
Requires-Dist: typer>=0.12; extra == 'cli'
Provides-Extra: dev
Requires-Dist: mypy==2.4.0; extra == 'dev'
Requires-Dist: pytest-recording>=0.13; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff==0.17.0; extra == 'dev'
Requires-Dist: types-jsonschema; extra == 'dev'
Provides-Extra: docs
Requires-Dist: mkdocs-material>=9.5; extra == 'docs'
Provides-Extra: jev
Requires-Dist: typesafe-sdk>=0.7; extra == 'jev'
Provides-Extra: langchain
Requires-Dist: langchain-core>=0.3; extra == 'langchain'
Provides-Extra: litellm
Requires-Dist: litellm>=1.50; extra == 'litellm'
Provides-Extra: llamaindex
Requires-Dist: llama-index-core>=0.11; extra == 'llamaindex'
Provides-Extra: mcp
Requires-Dist: mcp>=2.0; extra == 'mcp'
Provides-Extra: openai
Requires-Dist: openai>=1.40; extra == 'openai'
Provides-Extra: otel
Requires-Dist: opentelemetry-sdk>=1.27; extra == 'otel'
Provides-Extra: report
Requires-Dist: jinja2>=3.1; extra == 'report'
Provides-Extra: semantic
Requires-Dist: torch>=2.2; extra == 'semantic'
Requires-Dist: transformers>=4.44; extra == 'semantic'
Description-Content-Type: text/markdown

# counterfix

**Find why an LLM call failed, and get a fix that was tested to work.**

Most LLM evaluation tools tell you *that* a model failed. counterfix is built to tell you
*why*, and to hand back a fix it has already checked. When a call fails, it changes one
thing at a time (raises `max_tokens`, asks for bare JSON, adds the source text, tells the
model to check the question's premises, repeats a missed instruction) and re-runs the
call. A change counts only if it works in 2 of 3 re-runs. The change that fixes the
failure is both the diagnosis and the prescription. We call this **counterfactual
diagnosis**.

> **Status: alpha (v0.1.0).** Everything below works today: live checks, judge checks,
> verified fixes, reports, regression tests, profiles. APIs may still change.
>
> **Known limits.** Verified fixes are common for cut-off answers and broken JSON, rare
> for invented facts (3 of 14 on our seed set) and none yet for over-refusals or false
> premises: there counterfix says so and lists what to try instead. Shipped model profiles
> are small (54 probes; qwen2.5:3b only partly measured). The labels behind our accuracy
> numbers are mostly reference labels, with a small human spot check still to do. See
> [`docs/results.md`](docs/results.md).

## Install

```bash
pip install "counterfix[openai]"
```

The core needs no API key. The `openai` extra is only for wrapping OpenAI-compatible
clients (OpenAI, Groq, Gemini, OpenRouter, Ollama, vLLM, ...).

## Quickstart

### 1. See it work in 10 seconds (no key)

```python
import counterfix
from counterfix import CallRecord, Message

record = CallRecord(
    messages=[Message(role="user", content="Answer in exactly one word: sky colour?")],
    response_text="The sky is usually blue on a clear day.",
)
print(counterfix.diagnose(record, judge=False).explain())
```

```text
F6 Dropped instruction (90%)
  why: instructions not met: exactly 1 word (has 9); a one-word answer (has 9 words)
  fix: Repeat the broken instructions at the end of the prompt as a short checklist. ...
```

`judge=False` runs only the live checks: free, instant, no extra calls. More cases in
[`examples/quickstart.py`](examples/quickstart.py).

### 2. Wrap your client (one line)

```python
from openai import OpenAI
import counterfix

client = counterfix.guard(OpenAI())  # same API as before
response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Why did Einstein win his third Nobel Prize?"}],
)
print(response.choices[0].message.content)
print(response.diagnosis.summary())  # live checks, already run on this call
```

Any OpenAI-compatible endpoint works, for example a local Ollama server:

```python
client = counterfix.guard(OpenAI(base_url="http://localhost:11434/v1", api_key="ollama"))
```

The Anthropic SDK works the same way (`messages.create`, sync or async; `tool_use`
arguments are checked against each tool's `input_schema`):

```python
from anthropic import Anthropic

client = counterfix.guard(Anthropic())
```

Already on LiteLLM or LangChain? Add a callback; it only observes, never changes a reply:

```python
litellm.callbacks = [counterfix.litellm_callback()]
llm.invoke("...", config={"callbacks": [counterfix.langchain_callback()]})
```

Not using any of these? Decorate the function that calls your model (sync or
async). It takes a prompt string or chat messages and returns text or a response:

```python
@counterfix.watch(model="llama3.2")
def ask(prompt: str) -> str:
    return my_local_model(prompt)


answer = ask("Answer in one word: sky colour?")  # returned unchanged
print(ask.last_diagnosis.summary())
```

### 3. Ask a judge for the deeper look

```python
report = counterfix.diagnose(response)  # adds judge checks: false premise, invented facts, ...
print(report.explain())
```

The judge comes from your own key (a free `GROQ_API_KEY` or `GEMINI_API_KEY` is enough;
see [Bring your own key](#bring-your-own-key)). Run `counterfix setup` to see what it
found. Full script: [`examples/diagnose_with_judge.py`](examples/diagnose_with_judge.py).

### 4. Get a fix that was tested to work

```python
result = counterfix.fix(response)  # a guarded response, its diagnosis, or a CallRecord
print(result.explain())
```

```text
found: F1 Truncation (95%)
plain retry: passed 0/2
tried raise_max_tokens: passed 0/2 -> no (still: F1 Truncation)
tried raise_max_tokens_more: passed 2/3 -> VERIFIED
fix: Raise max_tokens a lot (long answers, or reasoning that uses the budget)
  change: {'max_tokens': 4096}
7 re-run(s); stopped at the first verified fix
```

For each failure, counterfix changes one thing (raise `max_tokens`, ask for bare JSON,
check the question's premises, repeat missed instructions, add the source text, ...),
re-runs the call, and re-checks it. A change counts as the fix only if the failure is
gone, with no new failure, in 2 of 3 re-runs, so luck does not count. It first re-runs
the call unchanged: if a plain retry already passes, no change is claimed. The verified
change is in `result.fix` (`changed` params, or full new `messages`) and on the finding's
prescription (`verified=True`). `max_runs` caps the re-runs (15); judge calls come on top.
When nothing passes, `result.escalation` lists what is left (retrieval, a larger model, a
code tool, a clarifying question).

### 5. Log calls, then report on them

```python
client = counterfix.guard(
    OpenAI(), config=counterfix.Config(on_diagnosis=counterfix.log_to("calls.jsonl"))
)
```

```bash
counterfix diagnose calls.jsonl                      # terminal summary, live checks only (free)
counterfix diagnose calls.jsonl --judge --html report.html --json results.jsonl
counterfix diagnose calls.jsonl --fail-on-failure    # exit code 1 if any call failed (CI)
```

The HTML report is one self-contained file you can send to a teammate: totals per failure
type, then each call with its prompt, answer, why it was flagged and the suggested fix.
The JSON lines (`{"record": ..., "diagnosis": ...}`) suit log pipelines, and can be read
back with `counterfix diagnose` or `counterfix.load_log()`.

### 6. Judge checks on production traffic, without slowing it down

```python
config = counterfix.Config(
    sample_rate=0.05,  # every flagged call + 5% of the rest get judge checks
    max_judged_per_hour=60,  # cost cap; calls over it are skipped, never delayed
    on_judged=counterfix.log_to("judged.jsonl"),
)
client = counterfix.guard(OpenAI(), config=config)
```

Judge checks run on one background thread, so your call returns as fast as before. If the
queue fills up, new calls are skipped rather than waited on; `counterfix.background.stats`
counts what was judged and skipped. Call `counterfix.background.flush()` before a short
script exits if you want the queued checks to finish.

### 7. Keep fixed failures fixed (pytest)

```bash
counterfix fix calls.jsonl --judge --out fixes.jsonl
counterfix export-tests fixes.jsonl --out tests/test_prompt_regressions.py
pytest tests/test_prompt_regressions.py
```

Each verified fix becomes a test case that fails if the failure comes back after a prompt
edit or a model change. In your own tests, `counterfix.testing.assert_ok(response)` fails
with the explanation when counterfix finds a failure.

### More

| | |
| --- | --- |
| Streams (`stream=True`) | every chunk passes straight through; the whole answer is checked at the end |
| `with counterfix.context(source=..., reference=...)` | tell the checks what the model was given (RAG) or the right answer |
| `counterfix profile / compare / drift / route` | measure models, compare an upgrade on your own calls, alert on drift, pick the cheapest safe model per task |
| `counterfix.otel.export()`, `counterfix.otel.SpanChecker()` | send diagnoses to OpenTelemetry, or check the GenAI spans you already have |
| `counterfix mcp` | an MCP server so coding agents can lint prompts and check calls |

Full docs: `docs/` (build with `mkdocs serve`). Measured accuracy and fix rates:
[`docs/results.md`](docs/results.md). A Colab-ready notebook on a free model:
[`examples/quickstart.ipynb`](examples/quickstart.ipynb).

## What it checks

Live checks make no extra LLM calls. They catch truncation (including reasoning models that
spend the whole budget thinking), broken JSON or schema mismatches, broken tool-call
arguments, refusals, empty answers, provider content filters, and broken instructions that
code can verify exactly: sentence, word and bullet counts, letter case, reply language
(by script), British or American spelling, banned words, "end with a question", "only the
number", "one word" and similar (in the spirit of IFEval).

### Optional auto-retry: a fix that was re-run

```python
from counterfix import Config, guard

client = guard(OpenAI(base_url=GROQ_URL, api_key=key), config=Config(mode="retry"))
r = client.chat.completions.create(
    model="qwen/qwen3.8-27b",
    messages=[{"role": "user", "content": "List the planets of the solar system, one per line."}],
    max_tokens=20,
)
for f in r.diagnosis.retried_from:
    print("before:", f.failure_type.value, f.evidence[0].detail)
for attempt in r.diagnosis.retries:
    print("retry: ", attempt.changes, attempt.outcome)
print("after: ", r.diagnosis.summary())
```

```text
before: F1 the answer stopped at the max_tokens limit (20)
retry:  {'max_tokens': 80} fixed
after:  no failure found by 5 checks
```

You get back the fixed response (all eight planets), and the record of what changed.
Retry keeps whichever response is better, records what it tried, and never raises if the
retry fails. Add your own fixes per model with `RuleStore` and `FixRule`.

### Deeper checks with a judge (diagnose mode)

Some failures need meaning to spot: invented facts, false premises, content instructions
("mention the price"), ambiguity, wrong arithmetic. These checks ask a judge model, so they
cost extra calls and never run in live mode; `counterfix.diagnose()` adds them when you
want the deeper look (on failures, on a sample of traffic, or in tests). `diagnose()` takes
a response from a guarded client or a `CallRecord`.

One mistake gets one cause: a wrong calculation is reported as F8, not also as an invented
fact (F4), and a cut-off answer as F1, not also as dropped instructions (F6). The findings
that were set aside are kept in `report.suppressed`.

The F8 check never trusts the judge's arithmetic: the judge writes one expression and
counterfix evaluates it with a small whitelist-only evaluator. A sampling check
(`SamplingCheck`) re-asks the model and flags answers that change between samples.
An optional Jev back-end (`counterfix[jev]`) plugs into the same judge interface.

### Bring your own key

counterfix never ships API keys. Live checks need none: they wrap your own client and make
no calls of their own. Judge checks use a judge you choose, in this order:

1. `counterfix.diagnose(response, judge="groq:qwen/qwen3.8-27b")`
2. `counterfix.configure(judge=...)`, once per process. A list of specs makes a voting
   panel, e.g. `["groq:qwen/qwen3.8-27b", "gemini:gemini-3.5-flash-lite", "ollama:llama3.2"]`
3. the `COUNTERFIX_JUDGE` environment variable (comma-separated for a panel)
4. a key it finds: `GROQ_API_KEY` or `GEMINI_API_KEY` (both have free tiers),
   `OPENAI_API_KEY`, `OPENROUTER_API_KEY`, or a model in a local Ollama. A different
   model family from the one being checked is preferred.
5. otherwise the checked model judges itself, with a `SelfJudgeWarning`, because judges are
   lenient with their own mistakes.

See what counterfix finds on your machine (key values are never printed):

```bash
counterfix setup          # keys, local models, and the judge that will be used
counterfix setup --test   # also asks the judge one tiny question
```

### Lint prompts before you send them

```python
from counterfix import lint

for issue in lint("Return ONLY JSON with keys name and age for: Priya, 29."):
    print(issue.rule, issue.title, "->", issue.suggestion)
```

Or from the command line, e.g. in CI:

```bash
counterfix lint prompts/*.txt --fail-on-warning
```

| Rule | Flags | Tends to cause |
| --- | --- | --- |
| L1 | A constraint buried in the middle of a long prompt | F6 |
| L3 | A factual question with no "I don't know" option | F4, F5 |
| L4 | JSON/YAML/CSV asked for in words only, no example or JSON mode | F2 |
| L7 | Pasted outside text with no delimiters | prompt injection |
| L2 | Instructions that conflict (needs `--judge`) | F6, F7 |
| L5 | System prompt and user message disagree (needs `--judge`) | F6, F3 |
| L6 | Request missing a key detail (needs `--judge`) | F9 |

Design rules:

- **Fail open.** If counterfix itself errors, your call still returns normally.
- **Your errors stay yours.** Exceptions from the provider reach your code unchanged.
- **Safe defaults.** Live mode only flags failures and makes no extra LLM calls.
- **Light core.** The core depends on 4 small libraries; providers and heavy tools are extras.

## Failure types (v1)

| Code | Failure |
| --- | --- |
| F1 | Truncation |
| F2 | Format break (JSON, schema) |
| F3 | Over-refusal |
| F4 | Hallucination from missing knowledge |
| F5 | False-premise acceptance |
| F6 | Dropped instruction |
| F7 | Prompt fragility |
| F8 | Computation or token-level error |
| F9 | Ambiguous request |

API errors, rate limits and provider content filters are tracked separately, so they are
never counted as the model's fault.

## Development

```bash
git clone https://github.com/KANSADWALA/CounterFix.git
cd CounterFix
pip install -e ".[dev]"
pytest
```

Optional extras: `openai`, `anthropic`, `litellm`, `langchain`, `llamaindex`, `otel`,
`semantic`, `cache`, `cli`, `report`, `jev`.

## Roadmap

Done:

- `guard()` for any OpenAI-compatible client, live checks, optional auto-retry ✅
- Prompt linter (`counterfix lint`) ✅
- Judge checks, `diagnose()`, bring your own key, `counterfix setup` ✅
- Reports: `log_to()`, `counterfix diagnose LOG` with terminal, JSON and HTML output ✅
- Background judge checks on a sample of traffic, with a cost cap ✅
- `@counterfix.watch` decorator for any function, sync or async ✅
- Anthropic SDK in `guard()`, LiteLLM and LangChain callbacks ✅
- Verified fixes: `counterfix.fix()` (counterfactual re-runs, 2 of 3 must pass) ✅
- Prompt regression tests: `counterfix export-tests` and a pytest plugin ✅
- Model profiles, `compare`, `drift`, `route`; fixes ordered by what worked per model ✅
- OpenTelemetry export and GenAI span reader; streams; RAG context; MCP server ✅

Next:

1. First release on PyPI (v0.1.0)
2. A larger probe suite from public datasets (after a licence check per dataset)
3. A public dataset of diagnosed failures with their verified fixes

## License

MIT
