Metadata-Version: 2.4
Name: ledgelm
Version: 0.5.0
Summary: Thin Python reporter for LedgeLM eval results
Requires-Python: >=3.10
Requires-Dist: httpx<1,>=0.27
Requires-Dist: pydantic<3,>=2.9
Provides-Extra: all
Requires-Dist: langchain-core<2,>=1; extra == 'all'
Requires-Dist: langgraph<2,>=1; extra == 'all'
Provides-Extra: langchain
Requires-Dist: langchain-core<2,>=1; extra == 'langchain'
Provides-Extra: langgraph
Requires-Dist: langgraph<2,>=1; extra == 'langgraph'
Description-Content-Type: text/markdown

# LedgeLM Python reporter

This package reports eval results and provides optional local helpers for running user-owned
functions, LangChain runnables, and LangGraph graphs or nodes. It never hosts execution or owns
models, credentials, datasets, or judging policy. LedgeLM handles storage, comparison, and GitHub
Checks server-side.

```python
from ledgelm import report, flush

report(
    name="response_grounding",
    type="judge",
    score=0.82,
    passed=True,
    cost=0.004,
	usage={
		"target": {"input_tokens": 812, "output_tokens": 146},
		"judge": {"input_tokens": 1042, "output_tokens": 87},
	},
	input={"messages": [{"role": "user", "content": "Can I return this purchase?"}]},
	output="Yes. Refunds are available within 30 days with a receipt.",
	expected_output="Explain the 30-day refund policy and receipt requirement.",
    provenance={
        "test_case_id": "refund-42",
        "target_id": "support-claude-sonnet",
        "target": {"provider": "anthropic", "model": "claude-sonnet"},
        "dataset": {"name": "support-golden", "version": "2026-07"},
    },
    metadata={"rationale": "The answer cites the supplied policy."},
)

verdict = flush()
```

Set `LEDGELM_API_URL` and `LEDGELM_API_TOKEN` in GitHub Actions. Commit, PR,
branch, run, and workflow fields are detected from the GitHub Actions
environment. Pass `flush(api_url="https://your-tunnel.example")` (or the same
argument to `async_flush`) to override the environment for a local or alternate
deployment. Use `await async_flush()` in async programs.

Both flush variants upload batches in chunks. Transport, authentication, and
validation failures during upload emit a warning and return an `unknown`
verdict without removing buffered results, so LedgeLM availability cannot
break the eval job.

## Evaluation helpers

The generic helper has no framework dependency:

```python
from ledgelm import evaluate

outcome = evaluate(
    name="support-answer",
    input={"question": "Can I return this?"},
    run=lambda case: agent(case["question"]),
    assess=lambda context: {"passed": "30 days" in context.output},
)
```

Install `ledgelm[langchain]`, `ledgelm[langgraph]`, or `ledgelm[all]` for optional framework
dependencies, then use `evaluate_runnable()`, `evaluate_graph()`, or `evaluate_node()` from
`ledgelm.integrations`. Async counterparts are available for each integration. All helpers report
one result but never call `flush()`.

The Pydantic wire models in `src/ledgelm/_generated.py` are generated from the
same Zod-owned OpenAPI document as the TypeScript reporter. Regenerate them
from the repository root with `bun run schema:python`; do not edit that file by
hand.

For a GitHub Actions matrix, do not flush in each shard. Use the generated
`ReportArtifact` model to write one typed JSON artifact per shard, then let the
LedgeLM fan-in action flush the complete matrix:

```python
from pathlib import Path
from ledgelm import ReportArtifact

artifact = ReportArtifact(schema_version=1, shard="quality", results=results)
Path("ledgelm-results.json").write_text(artifact.model_dump_json(), encoding="utf-8")
```
