Metadata-Version: 2.4
Name: ninjaeval
Version: 0.1.0
Summary: Show NinjaEval what happens inside your AI bot: tool calls and LLM calls, recorded only for NinjaEval tests.
License: MIT
Project-URL: Homepage, https://ninjaeval.com
Project-URL: Documentation, https://ninjaeval.com/docs/developer-guide.html#show-ninjaeval-inside-your-bot
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: otel
Requires-Dist: opentelemetry-api>=1.20; extra == "otel"
Dynamic: license-file

# ninjaeval (Python)

Show [NinjaEval](https://ninjaeval.com) what happens inside your AI bot (every tool call and LLM call), so it can grade
them and show them side by side with production in its debugger. No dependencies.

```python
import ninjaeval
ninjaeval.instrument()                     # records OpenAI, Anthropic and LiteLLM chat calls automatically

@ninjaeval.tool                            # records arguments, result or error, and duration
def get_order(order_id: str): ...

@app.post("/chat")
def chat(req):
    with ninjaeval.capture(req.headers) as run:     # active only for NinjaEval test requests
        reply = agent.run(req.message)
    return run.attach({"reply": reply})             # adds the "ninjaeval" block for tests, nothing otherwise
```

- **Tests only.** NinjaEval marks its requests with `X-NinjaEval-Test`. Without that header, every hook does nothing
  and your response is unchanged.
- **Never breaks your app.** Errors inside the SDK are swallowed, and your tools' own errors are recorded and re-raised
  as usual.
- **Redaction.** `capture(headers, redact=lambda text: mask_emails(text))` is applied to everything recorded.
- **Other clients:** call `ninjaeval.record_turn(model=..., messages_in=..., output=..., tool_calls=[...],
  prompt_tokens=..., completion_tokens=..., latency_ms=...)` yourself.
- **OpenTelemetry (optional):** with `pip install "ninjaeval[otel]"`, each tool and LLM call is also a span, a child of
  the test's `traceparent`.
- Works with sync and async code (FastAPI, Flask, Django, Starlette). Contract: the
  [`ninjaeval` block v1](https://ninjaeval.com/docs/developer-guide.html#show-ninjaeval-inside-your-bot).
