Metadata-Version: 2.5
Name: cognifity-verdict-eval
Version: 0.1.0a9
Summary: Verdict eval engine: judge, drift detection, Bradley-Terry comparator, regression injector
Project-URL: Homepage, https://cognifity.ai/verdict
Project-URL: Documentation, https://github.com/cognifityai/verdict/tree/main/docs
Project-URL: Repository, https://github.com/cognifityai/verdict
Project-URL: Issues, https://github.com/cognifityai/verdict/issues
Author: Cognifity AI
License-Expression: Apache-2.0
License-File: LICENSE
License-File: NOTICE
Keywords: drift-detection,evaluation,llm,llm-as-judge,statistics
Classifier: Development Status :: 3 - Alpha
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Requires-Dist: cognifity-verdict<0.2,>=0.1.0a9
Requires-Dist: numpy>=1.26
Requires-Dist: scikit-learn>=1.4
Requires-Dist: scipy>=1.11
Provides-Extra: dev
Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
Requires-Dist: pytest>=8; extra == 'dev'
Requires-Dist: pyyaml>=6; extra == 'dev'
Provides-Extra: semantic
Requires-Dist: sentence-transformers>=3.0; extra == 'semantic'
Description-Content-Type: text/markdown

# Verdict Eval

PyPI distribution: `cognifity-verdict-eval`. Python import: `verdict_eval`.

The Verdict eval engine. LLM-as-judge with binary rubric, intent clustering,
non-parametric drift detection per cluster per dimension (Fisher's exact test
for binary PASS/FAIL dimensions, Mann-Whitney U for continuous metrics),
Bradley-Terry pairwise comparator for cross-LLM evaluation, and a synthetic
regression injector for verifying the pipeline catches what it should.

## Pairwise result contract

`PairwiseJudge.compare()` separates preference from execution state. A usable
`PairwiseJudgment` has `status == PairwiseStatus.VALID`, `is_usable == True`,
and a verdict of `A_BETTER`, `B_BETTER`, `TIE`, or `INCONSISTENT`. Exactly one
complete `[[A]]`, `[[B]]`, or `[[C]]` marker is required in each position-swap
round. Missing, empty, truncated, repeated, or conflicting markers produce
`PairwiseStatus.INVALID`; provider failures produce `PairwiseStatus.ERROR`.
Both unusable states carry `verdict=None` and must not be converted to ties.

Ensembles preserve one component record per configured judge and vote using
only usable components. An aggregate can remain usable when at least one
component is usable, but failed components remain visible in
`component_judgments`. A total component failure is unusable. The alignment
harness reports pair and component coverage separately and fails its evidence
gate when either is incomplete.

This does not change captured traces, spans, or storage schemas. Existing
successful 0.1.0a3 positional construction retains its original field order;
the status fields were appended. Consumers should check `is_usable` before
reading `verdict`:

```python
from verdict_eval import PairwiseJudge, PairwiseStatus

judgment = PairwiseJudge(provider=provider, model=model).compare(
    query=query,
    response_a=response_a,
    response_b=response_b,
)
if judgment.status is not PairwiseStatus.VALID:
    raise RuntimeError("pairwise comparison was not usable")
winner = judgment.verdict
```

The versioned registry requires a deliberate `verdict-cluster fit --strategy`
choice; no registry strategy is silently selected. Exact-key `explicit`
clustering is supported. Automatic `semantic` clustering and the semantic
fallback inside `hybrid` are experimental opt-in alpha features. Their frozen
quality evaluation missed one preregistered fragmentation gate (largest
nonoutlier cluster `30.1047%`, maximum `30%`) and must not be described as
generally validated. `verdict-cluster inspect` reports the strategy and this
experimental status. Local semantic work uses the frozen
`sentence-transformers/all-MiniLM-L6-v2` model; runtime download is forbidden.
The legacy trace clustering pipeline remains a separate methodology.

### Supported explicit registry workflow

Stamp a bounded, redaction-safe routing key around the provider request that
owns the intent. The context is token-restoring and the raw key is stored only
as the existing trace routing tag:

```python
import verdict

with verdict.intent_context("billing.v1"):
    response = provider.messages.create(...)
```

Use the real tenant ID for tenant-owned traces. For tenantless Memory/SQLite
stores only, use the reserved local scope `__verdict_local__`; that literal is
not a customer tenant ID. Existing a7 databases start with pending derived
analysis fields, so normalize bounded pages until the JSON result says
`"complete": true`:

```bash
verdict-cluster --storage sqlite:///verdict.db --tenant tenant-a --actor ops \
  normalize --limit 1000
verdict-cluster --storage sqlite:///verdict.db --tenant tenant-a --actor ops \
  fit --strategy explicit --target-workload agent \
  --cutoff 2026-08-22T00:00:00Z
```

Take `version_id` from the fit result, then assign and validate the immutable
preview before activation:

```bash
verdict-cluster --storage sqlite:///verdict.db --tenant tenant-a --actor ops \
  assign --version "$VERSION" --through-cutoff 2026-08-22T00:00:00Z
verdict-cluster --storage sqlite:///verdict.db --tenant tenant-a --actor ops \
  validate --version "$VERSION"
verdict-cluster --storage sqlite:///verdict.db --tenant tenant-a --actor ops \
  activate --version "$VERSION" --expected-generation 0
verdict-pipeline --storage sqlite:///verdict.db --registry-mode active \
  --tenant-id tenant-a
```

Active mode is always pinned to the tenant's active pointer. `inspect` returns
bounded version, cluster, stable display-name, assignment, and event data plus
truncation flags; its `--*-limit` and `--*-offset` options page immutable detail
within hard output ceilings. `rename` changes only a stable display name; `rollback`
requires a previously activated version and the current expected generation.
CLI failures use a closed safe code such as `analysis_index_pending`,
`model_unavailable`, `validation_failed`, or `generation_conflict`; raw storage
and provider exception text is not printed. Semantic and hybrid commands also
require a reviewed local `--model-path` and remain experimental.

Registry shadow analysis is disabled pending the tenant-isolation correction in
[issue #24](https://github.com/cognifityai/verdict/issues/24). Validate an
inactive preview with `verdict-cluster validate`; do not analyze it through the
drift pipeline before activation.

Drift is a batch comparison over each captured trace's `started_at` time. The
runner defaults to a 24-hour current window and a 7-day baseline separated by a
24-hour gap, with at least 30 judgments per `(cluster, dimension)` window. A
signal must clear the BH-adjusted p-value gate and the Cliff's delta effect-size
gate. On binary PASS/FAIL data the default `0.147` delta is a 14.7 percentage-
point sensitivity floor. These defaults require workload-specific validation.

Pipeline reruns use the latest attempt per trace for one complete evaluator
identity: provider, model list, rubric name/version, behavior-relevant
configuration, expected dimensions, and effective prompt/rubric fingerprint. A
latest error is excluded from PASS/FAIL and can be retried. Other evaluator
definitions are retained but not pooled. Persisted drift signals carry the same
fingerprint. Each completed analysis atomically persists a `DriftRun` marker and
its exact signal set, including zero-signal runs; latest-run consumers exclude
legacy ungrouped signals. Optional fixed human-labeled sentinel runs store independent judge-
health aggregates. A healthy status requires both the independent-example floor
and the 95% Wilson-interval lower bound to clear the configured threshold. An
example passes only when every declared label matches; label agreement is a
separate diagnostic, not the gate's statistical unit. Legacy label-only records
remain unavailable for health gating. Any sentinel execution error prevents a
`healthy` result: too few usable examples remain `insufficient_data`;
otherwise the result is `degraded`. When a sentinel file is supplied,
the runner persists the health record and exits 2 before production judgments or
drift unless status is `healthy`.

Probe weights enter suite and category quality gates once per probe. A probe
passes only when every declared expectation passes. Weighted expectation
agreement and its per-dimension breakdown remain separate diagnostics; adding
expectations cannot make a failing probe count less in the quality gate.
The bundled weighted suite is version `2.1`; its direct prompt-injection probe
defines the quoted-text instruction precedence independently for both safety
and instruction-following judgments. New `ProbeRun` and `ProbeResult`
artifacts stamp metric-schema version `3` and one-dimension judge-method version
`2`; historical artifacts without those fields remain version `1` when loaded
through the dataclasses, so scheduled comparisons cannot silently cross the
methodology boundary. Each expectation is judged with a one-dimension rubric
and records its effective evaluator fingerprint. A caller-supplied `Judge` or
`JudgeEnsemble` is narrowed consistently while preserving its rubric, provider,
model, temperature, and token configuration. Probe expectation verdicts accept
only the exact labels `PASS` and `FAIL`; malformed programmatic or YAML suite
definitions fail during construction instead of being normalized into a scored
outcome. Target or follow-up execution errors emit an `ERROR` result for every
declared expectation, so outages remain in every dimension denominator. The
scheduled CLI requires a 100% weighted probe pass rate by default, exits
1 below the configured threshold, and exits 2 on provider/judge execution errors.
Non-positive, non-finite, or non-numeric weights in historical result JSON
contribute zero rather than crashing or corrupting an aggregate. Historical
dimension entries whose `passed` field is not a literal boolean fail closed.
Current artifacts with missing, unnamed, duplicate, non-dictionary, or
contradictory expectation rows cannot pass the probe gate. The user-signal correlator
reports usable sample size, Wilson raw-agreement bounds, and deterministic
bootstrap intervals for both Cohen's kappa and Gwet's coefficient. It refuses to
call low-data output calibrated, excludes `UNCLEAR` judge results from its binary
confusion matrix, and requires an explicit evaluator selection when identities
are mixed. Exact duplicate usable rows collapse per trace; contradictory usable
rows are excluded and counted rather than resolved by input order. Conditional
disagreement rates use the judge-PASS denominator for leniency and the
judge-FAIL denominator for strictness.
Probe JSON artifacts apply Verdict's best-effort pattern redaction to
captured target text, judge reasoning, and provider errors before returning the
serializable run result.

```python
from verdict_eval import (
    DEFAULT_RUBRIC,
    CorruptionInjector,
    DriftDetector,
    Judge,
    PairwiseJudge,
    PairwiseStatus,
)
```

See the [repository README](https://github.com/cognifityai/verdict#readme),
[ADR-002](https://github.com/cognifityai/verdict/blob/main/docs/adrs/002-judge-methodology.md),
and the [verification scripts](https://github.com/cognifityai/verdict/tree/main/scripts).

Apache 2.0.
