Metadata-Version: 2.4
Name: mellea-jev-adapter
Version: 0.7.0
Summary: Unofficial TypeSafe Jev Noul verifier and Choice classifier for Mellea
License: MIT
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: httpx2<3,>=2.0.0
Requires-Dist: pydantic<3,>=2.12.0
Requires-Dist: typesafe-sdk<0.8.0,>=0.7.5
Provides-Extra: mellea
Requires-Dist: mellea<0.9.0,>=0.7.0; extra == "mellea"
Provides-Extra: openai
Requires-Dist: openai<4,>=3.26.0; extra == "openai"
Provides-Extra: dev
Requires-Dist: openai<4,>=3.26.0; extra == "dev"
Requires-Dist: mypy<2,>=1.15; extra == "dev"
Requires-Dist: pytest<10,>=8.3; extra == "dev"
Requires-Dist: pytest-cov<8,>=6; extra == "dev"
Requires-Dist: ruff<0.17,>=0.16; extra == "dev"
Provides-Extra: laya
Requires-Dist: laya-mlx<0.2.0,>=0.1.0; (sys_platform == "darwin" and platform_machine == "arm64") and extra == "laya"
Dynamic: license-file

# Mellea × Jev

[![CI](https://github.com/SoundBlaster/Jev4Mellea/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/SoundBlaster/Jev4Mellea/actions/workflows/ci.yml)
[![Version](https://img.shields.io/github/v/tag/SoundBlaster/Jev4Mellea?label=version)](https://github.com/SoundBlaster/Jev4Mellea/tags)
![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-blue?logo=python&logoColor=white)
![Mellea 0.7–0.8](https://img.shields.io/badge/Mellea-0.7%E2%80%930.8-6f42c1)
![TypeSafe SDK 0.7.x](https://img.shields.io/badge/TypeSafe%20SDK-0.7.x-6f42c1)
[![Coverage gate: 80%](https://img.shields.io/badge/coverage%20gate-%E2%89%A580%25-success)](https://github.com/SoundBlaster/Jev4Mellea/blob/main/pyproject.toml#L68-L70)

A small Python adapter that brings TypeSafe Jev's semantic checks into
[Mellea](https://github.com/generative-computing/mellea). Use Jev to verify
generated text, classify it into your labels, or rate it on a scale. Mellea
continues to manage generation and repair.

- **Verify requirements** with Jev Noul and configurable accept/reject thresholds.
- **Classify text** with TypeSafe Choice and caller-defined categories.
- **Score text** on an ordered scale with TypeSafe Score.
- **Ask several questions in one request** and inspect returned token usage.
- **Connect checks to Mellea `Requirement`s** so they can participate in sampling.
- **Check images with Jev** using explicit PNG/JPEG attachments (experimental).

This is an unofficial, synchronous adapter. Jev evaluates supplied content; it does not
generate or repair it.

## Quick start

Install from a checkout and set an API key from the
[TypeSafe console](https://console.typesafe.ai/). Live requests may incur
charges.

```bash
git clone https://github.com/SoundBlaster/Jev4Mellea.git
cd Jev4Mellea
make install
export TYPESAFE_API_KEY='your-key'
```

The Makefile defaults to Python 3.13. To use another supported interpreter,
run `make install PYTHON=python3.11` (or set `PYTHON` to your installed version).

Ask whether a candidate meets a positive requirement:

```python
from mellea_jev import JevClient, JevVerifier

with JevClient() as jev:
    verifier = JevVerifier(
        jev,
        "The answer gives the museum's opening time from the reference.",
        reference="The museum opens at 10:00 and closes at 18:00.",
    )
    verdict = verifier.evaluate("The museum opens at 10:00.")
    print(verdict.outcome, verdict.p_yes)
```

`outcome` is `pass`, `fail`, or `uncertain`. Use `verifier.as_requirement()` to
attach the same check to a Mellea generation flow. Once the project dependencies
are installed, `make demo` runs without a key or network access using a mocked
Jev response.

## Examples

### Check an image with Jev (experimental)

Attach prepared PNG/JPEG bytes to Noul, Choice, Score, or a mixed `system_one`
request. Refer to each named image as `images.<name>` in the question:

```python
from mellea_jev import ImageInput, JevClient

image = ImageInput.from_file("scene.png")
with JevClient(model="jev-1.14.0") as jev:
    result = jev.noul(
        state={},
        images={"scene": image},
        question="Is a pedestrian crossing visible in `images.scene`?",
    )
    print(result.p_yes, result.model, result.request_id)
```

To check generated text against an image, compose `WithImages` with the existing
Mellea helpers. The image is explicit context; the candidate remains text:

```python
from mellea_jev import ImageInput, JevClient, JevVerifier, WithImages

with JevClient(model="jev-1.14.0") as jev:
    visual = WithImages(jev, {"scene": ImageInput.from_file("scene.png")})
    verifier = JevVerifier(
        visual,
        "The candidate describes only objects visible in `images.scene`.",
    )
    verdict = verifier.evaluate("The photograph shows cars on a street.")
    requirement = verifier.as_requirement()
```

`WithImages` also works with `JevClassifier` and `JevScorer`, including their
`as_requirement()` methods. It copies the image mapping and uses the provider's
existing lifetime. To attach a Mellea `ImageBlock`, explicitly convert it with
`ImageInput.from_mellea(block)`. The adapter does not scan conversation history
or fetch `ImageUrlBlock` URLs. TypeSafe and OpenAI providers implement the
separate `ImageProvider` protocol; other providers remain text-only.

Passing `images` explicitly opts that request into TypeSafe's experimental image
API. The SDK creates multipart attachments; results retain the existing types,
threshold policy, model, request ID, and usage. The `images` state key is reserved
when attachments are supplied; a collision raises `ValueError`. SDK types stay
internal, and images are uploaded to the explicitly configured provider endpoint.

Prepare correctly oriented pixels before upload: PNG/JPEG, at most 16 MiB per
image, each dimension 256–1568 pixels, and 78,400–2,458,624 pixels in total per
image. The API also limits aggregate image bytes to 32 MiB. The adapter checks
file signatures and the per-image byte limit; the API checks decoding, dimensions,
and aggregate limits. It does not resize or apply EXIF rotation. See the
[TypeSafe image contract](https://docs.typesafe.ai/images) for current limits.
Bounding boxes and image-valued question criteria are outside this adapter's
current image interface.

From a checkout, send one potentially billable request with three questions
(Noul, Choice, and Score):

```bash
make image-live-check IMAGE=/path/to/prepared-scene.png
```

This command explicitly uploads the selected file using `TYPESAFE_API_KEY`;
`IMAGE_MODEL` defaults to `jev-1.14.0`. Ordinary tests and CI use mocks.
The example is [`examples/live_image_check.py`](examples/live_image_check.py).

`OpenAIProvider` accepts the same `images` argument and `WithImages` composition
for Noul, Choice, and Score. It sends named base64 PNG/JPEG parts using the
official Decisions SDK, while text-only calls keep their original JSON input:

```python
from mellea_jev import ImageInput, OpenAIProvider

with OpenAIProvider() as provider:
    result = provider.noul(
        state={},
        images={"scene": ImageInput.from_file("scene.png")},
        question="Is a pedestrian crossing visible in `images.scene`?",
    )
```

Image availability depends on the model and endpoint. Check a gateway's
capabilities before uploading; unsupported requests raise an HTTP error and
are not rerouted automatically. The [CoreInfra guide](docs/coreinfra.md) lists
its supported image route.

### Jev through a Proxy

For the CoreInfra Hub, the [connection guide](docs/coreinfra.md) covers both
protocols, cross-model routing, supported images, and an opt-in compatibility
matrix. The `coreinfra_provider` factory composes the existing SDK providers.

Use your Proxy's key and explicitly select its TypeSafe-compatible API root. The SDK adds
`/v1/systemone`; do not include that suffix in `base_url`.

```python
import os
from mellea_jev import JevClient, NoulQuestion

with JevClient(
    api_key=os.environ["PROXY_API_KEY"],
    base_url="https://proxy.example/typesafe/api",
    model="jev-latest",
) as jev:
    result = jev.system_one(
        state={"text": "Hello! Thank you for your message."},
        questions={"greeting": NoulQuestion("Does the text contain a polite greeting?")},
    )
    print(result.answers["greeting"])
```

If your Proxy changes the operation path itself, pass the complete URL instead:

```python
with JevClient(
    api_key=os.environ["PROXY_API_KEY"],
    endpoint_url="https://proxy.example/models/jev/evaluate",
    model="jev-latest",
) as jev:
    result = jev.noul(state={"text": "Hello!"}, question="Is the text a greeting?")
```

`endpoint_url` is used as the exact operation address, without adding
`/v1/systemone`. It applies to Noul, Choice, Score, and batched `system_one`
requests. Supply either `base_url` or `endpoint_url`; passing both raises
`ValueError`, even if `base_url` is the official default.

The adapter retains its dictionary `state` contract. Without an explicit
`base_url` or `endpoint_url`, it uses the official TypeSafe endpoint and ignores
`TYPESAFE_BASE_URL`. Requests never follow redirects or retry automatically.
Replace the example URL with your Proxy's API root. From a checkout,
`make proxy-live-check` runs one potentially billable verification with
`PROXY_API_KEY` and either `PROXY_BASE_URL` or `PROXY_ENDPOINT_URL` set in the
environment; CI uses mocks. For an exact operation URL:

```bash
make proxy-live-check PROXY_ENDPOINT_URL=https://proxy.example/models/jev/evaluate
```

`PROXY_PROVIDER` defaults to `typesafe`; `PROXY_MODEL` can override the model.
The two URL variables are mutually exclusive.

### Use OpenAI Decisions API

From a checkout, install the optional official SDK with
`pip install -e '.[openai]'` and set `OPENAI_API_KEY`.
The Decisions API is in public beta. This provider supports Noul verification,
Choice classification, and Score rubrics, with one question per request.

```python
from mellea_jev import JevVerifier, OpenAIProvider

with OpenAIProvider() as provider:
    verifier = JevVerifier(
        provider,
        "The candidate gives the opening time supported by the reference.",
        reference="The museum opens at 10:00.",
        accept_at=0.90,
        reject_at=0.10,
    )
    verdict = verifier.evaluate("The museum opens at 10:00.")
    print(verdict.outcome, verdict.p_yes, verdict.request_id)
    requirement = verifier.as_requirement()
```

The same provider works with the existing classifier and Mellea requirement:

```python
from mellea_jev import JevClassifier, OpenAIProvider

with OpenAIProvider() as provider:
    classifier = JevClassifier(
        provider,
        "Which team should handle the request?",
        criteria={"billing": "Payments and refunds", "technical": "Product errors"},
    )
    result = classifier.classify("I was charged twice.")
    print(result.choice, result.confidence, result.probabilities)
    requirement = classifier.as_requirement("billing", minimum_confidence=0.80)
```

Choice preserves the SDK's confidence separately from the label probabilities.
Labels must be strings and the response must contain the complete configured distribution.

Score uses the same ordered rubric and inclusive bounds as other providers:

```python
from mellea_jev import JevScorer, OpenAIProvider

with OpenAIProvider() as provider:
    scorer = JevScorer(
        provider, "Rate the severity of the issue.", criteria=["Low", "Medium", "High"]
    )
    result = scorer.evaluate("The button is slightly misaligned.")
    print(result.score, result.confidence, result.legend)
    requirement = scorer.as_requirement(maximum_score=1.0, minimum_confidence=0.80)
```

The score is a fractional position from zero to `len(criteria) - 1`, checked
against the probability-weighted level average. Confidence remains a separate value.

The provider sends the state dictionary as JSON text in `input`. Optional Noul
`criteria` descriptions are appended as JSON to the predicate instructions;
Decisions predicates do not have a separate outcome-criteria field. Refusals
raise `OpenAIRefusalError`; transport, HTTP, and invalid-response errors raise
`OpenAIProviderError` subclasses. These errors never become accepted verdicts.
Requests use the official endpoint by default, without automatic retries or redirects.
For a Proxy, pass an explicit `base_url` (including the API
prefix such as `/v1`) and the Proxy's key:

```python
import os

with OpenAIProvider(
    api_key=os.environ["PROXY_API_KEY"],
    base_url="https://proxy.example/v1",
) as provider:
    result = provider.noul(state={"candidate": "Hello!"}, question="Is it a greeting?")
```

For a Proxy with a different operation path, use its complete endpoint:

```python
with OpenAIProvider(
    api_key=os.environ["PROXY_API_KEY"],
    endpoint_url="https://proxy.example/models/luna/decide",
    model="gpt-6-luna",
) as provider:
    result = provider.noul(state={"candidate": "Hello!"}, question="Is it a greeting?")
```

`endpoint_url` applies to Noul, Choice, and Score without appending `/decisions`.
Supply either `base_url` or `endpoint_url`; passing both raises `ValueError`.
Both URL options require an absolute HTTP(S) address without embedded
credentials, query parameters, or fragments. Nested and percent-encoded paths,
explicit ports, and trailing slashes are supported. URLs are configured per
provider instance; there is no endpoint auto-discovery or protocol conversion.

The gateway must implement the OpenAI Decisions request/response contract; support for
Chat Completions alone is insufficient. The configured URL receives the supplied
key. HTTP and HTTPS URLs are supported; `OPENAI_BASE_URL` is ignored.
Thresholds require evaluation on your own data before comparing them with Jev.
The command-line examples support mutually exclusive `--base-url` and
`--endpoint-url` flags. A custom URL requires `--api-key-env` to explicitly
select the variable containing the Proxy key.

### Use ScaleDown Decisions

`ScaleDownProvider` connects the same Mellea verifier, classifier, and scorer
contracts to ScaleDown's Decisions endpoint. It supports all three primitives
and pins the documented `classify-1` model. Set `SCALEDOWN_API_KEY` or pass
`api_key`; no SDK extra is needed.

```python
import os

from mellea_jev import JevClassifier, JevScorer, JevVerifier, ScaleDownProvider

with ScaleDownProvider(api_key=os.environ["SCALEDOWN_API_KEY"]) as provider:
    verification = JevVerifier(
        provider,
        "The answer follows the refund policy.",
        reference="Refunds are available within 30 days.",
    ).evaluate("The customer can get a refund within 30 days.")
    routing = JevClassifier(
        provider,
        "Which team should handle this ticket?",
        criteria={"billing": "Payments and refunds", "account": "Sign-in and profiles"},
    ).classify("I was charged twice.")
    urgency = JevScorer(
        provider,
        "How urgent is this ticket?",
        criteria=["Can wait", "Fix this week", "Blocks revenue now"],
    ).evaluate("Checkout is failing for every customer.")
    print(verification.outcome, routing.choice, urgency.score)
```

ScaleDown reads `state.text` and `state.document`; this adapter encodes the full
Mellea state dictionary as JSON in `state.text`, so `candidate` and `reference`
remain available. Noul outcome criteria are included in the instructions, and
structured Choice descriptions are serialized as JSON strings. ScaleDown Choice
and Score confidence is the selected label or top level's probability mass; tune
`minimum_confidence` for this meaning rather than reusing a threshold from a
provider with a different confidence definition. Score supports 2–10 levels.

The default URL is `https://api.scaledown.xyz/v1/scaledown`. Pass `base_url` for a
gateway root or `endpoint_url` for an exact operation URL. Requests use the
`x-api-key` header, ignore `SCALEDOWN_BASE_URL`, do not follow redirects, and are
not retried automatically. ScaleDown reports its USD charge through
`result.usage.cost_usd`; the response currently does not document a request ID.

### Use SubQ Decisions

`SubQProvider` connects the same Mellea verifier, classifier, and scorer
contracts to the [SubQ Decisions API](https://docs.subq.ai/endpoints/decisions-api/).
It supports all three primitives through the
preview model `subq-snap-preview`. Set `SUBQ_API_KEY` or pass `api_key`.

```python
import os

from mellea_jev import JevClassifier, JevScorer, JevVerifier, SubQProvider

with SubQProvider(api_key=os.environ["SUBQ_API_KEY"]) as provider:
    verification = JevVerifier(
        provider,
        "The answer follows the refund policy.",
        reference="Refunds are available within 30 days.",
    ).evaluate("The customer can get a refund within 30 days.")
    routing = JevClassifier(
        provider,
        "Which team should handle this ticket?",
        criteria={"billing": "Payments and refunds", "account": "Sign-in and profiles"},
    ).classify("I was charged twice.")
    urgency = JevScorer(
        provider,
        "How urgent is this ticket?",
        criteria=["Can wait", "Fix this week", "Blocks revenue now"],
    ).evaluate("Checkout is failing for every customer.")
    print(verification.outcome, routing.choice, urgency.score)
```

SubQ accepts object state directly. Pass `base_url` for a gateway root or
`endpoint_url` for an exact operation URL. Requests use Bearer authentication;
environment proxies are ignored, redirects are disabled, and requests are not
retried. The model is in public preview. Review SubQ's
[data retention policy](https://docs.subq.ai/using-the-api/data-retention/)
before sending sensitive state. Run one opt-in check with `make subq-live-check`;
the labeled Noul evaluator also supports `make live-evaluate EVAL_PROVIDER=subq`.

For one potentially billable Noul check through an OpenAI-compatible Proxy:

```bash
make proxy-live-check PROXY_PROVIDER=openai PROXY_MODEL=gpt-6-luna \
  PROXY_ENDPOINT_URL=https://proxy.example/models/luna/decide
```

Set `PROXY_API_KEY` locally before running. The same check can be launched directly:

```bash
python examples/live_check.py --provider openai --model gpt-6-luna \
  --endpoint-url https://proxy.example/models/luna/decide --api-key-env PROXY_API_KEY
```

The smoke command makes one request and exits with `0` for pass, `1` for fail,
`2` for uncertain, or `3` if the check could not complete. Invalid CLI arguments
exit with `2` before constructing a provider.

From a development checkout, set `OPENAI_API_KEY` locally and run:

```bash
make openai-live-test
```

This explicitly opts in to up to three potentially billable requests to the
official endpoint: one each for Noul, Choice, and Score. The command stops at
the first failure; requests have no automatic retries. To run just one primitive:

```bash
RUN_LIVE_OPENAI=1 .venv/bin/python -m pytest -q -x --tb=short \
  tests/test_live_openai.py -k choice
```

These are contract smoke tests: they check valid probabilities, complete labels
or score levels, response model and request ID, and Choice/Score token usage.
They do not assert a particular class, confidence threshold, or semantic score.
Refusal and API failures fail the test. Ordinary tests skip live requests even
when a key is present; direct pytest runs require both `RUN_LIVE_OPENAI=1` and
`OPENAI_API_KEY`. See the [official Decisions guide](https://developers.openai.com/api/docs/guides/decisions).

### Verify a requirement with Noul

Noul returns `p_yes`, the probability that a positively phrased requirement is
satisfied. The adapter uses two configurable thresholds; the space between
them is uncertain and is never silently accepted.

```python
from mellea_jev import JevClient, JevVerifier

with JevClient() as jev:
    verifier = JevVerifier(
        jev,
        "The candidate states the opening time supported by the reference.",
        reference="The museum opens at 10:00.",
        criteria={
            "true": "The candidate gives 10:00 as the opening time.",
            "false": "The candidate omits or contradicts the opening time.",
        },
        accept_at=0.90,
        reject_at=0.10,
        repair_hint="Use the opening time stated in the reference.",
    )
    verdict = verifier.evaluate("The museum opens at 10:00.")

    if verdict.outcome == "uncertain":
        print("Route for another check or human review")
```

The threshold values are application policy, not accuracy guarantees. Noul does
not return a separate confidence field or a textual explanation.

To use the verifier as a Mellea requirement:

```python
requirement = verifier.as_requirement()
```

For generation and repair with a real Mellea session, see
[`examples/mellea_ollama.py`](examples/mellea_ollama.py). That example uses
Ollama for generation and Jev for verification.

### Classify into configured categories with Choice

Choice selects one label from the supplied criteria and returns its confidence
and the probability of every label. Descriptions may be strings, JSON objects,
arrays, or `None`.

```python
from mellea_jev import JevClient, JevClassifier

criteria = {
    "billing": {"what": "Payments, invoices, or refunds", "examples": ["duplicate charge"]},
    "technical": "Product errors or problems using the service",
    "other": None,
}

with JevClient() as jev:
    classifier = JevClassifier(
        jev,
        "Choose the best category for this support message.",
        criteria=criteria,
    )
    result = classifier.classify("I was charged twice for my subscription.")
    print(result.choice, result.confidence, result.probabilities)
```

To require a Mellea candidate to be assigned to a particular category:

```python
requirement = classifier.as_requirement(
    "billing",
    minimum_confidence=0.75,
)
```

Confidence thresholds are caller policy. TypeSafe supports up to 255 Choice
labels.

### Rate text on an ordered scale with Score

Score returns a probability-weighted position on an ordered scale. The result
can fall between two levels.

```python
from mellea_jev import JevClient, JevScorer

levels = ["Cosmetic", "Workaround exists", "Blocking"]

with JevClient() as jev:
    scorer = JevScorer(
        jev,
        "How severe is the reported issue?",
        criteria=levels,
    )
    result = scorer.evaluate("The export button crashes and there is no workaround.")
    print(result.score, result.confidence, result.probabilities)
```

The result can also become a Mellea requirement with inclusive score bounds:

```python
requirement = scorer.as_requirement(
    minimum_score=1.0,
    maximum_score=2.0,
    minimum_confidence=0.6,
)
```

Score supports 2–10 ordered string descriptions.

### Batch questions and read usage metadata

`JevClient.system_one()` sends named Noul, Choice, and Score questions in one
request. Each answer remains a typed result. The returned usage counts are
informational and do not affect validation.

```python
from mellea_jev import ChoiceQuestion, JevClient, NoulQuestion

with JevClient() as jev:
    result = jev.system_one(
        state={"candidate": "I was charged twice and cannot log in."},
        questions={
            "urgent": NoulQuestion("Does the message convey urgency?"),
            "team": ChoiceQuestion(
                "Which team should handle this?",
                {"billing": "Payments and invoices", "technical": "Product errors"},
            ),
        },
    )

    print(result.answers["urgent"].p_yes)
    print(result.answers["team"].choice)
    if result.usage is not None:
        print(result.usage.input_tokens, result.usage.output_tokens)
```

For one-off checks without the Mellea helper classes, call the client methods
directly. Each method sends its own request:

```python
with JevClient() as jev:
    yes_no = jev.noul(
        state={"candidate": "The museum opens at 10:00."},
        question="Does the candidate state the opening time?",
    )
    category = jev.choice(
        state={"candidate": "I was charged twice."},
        question="Choose a category.",
        criteria={"billing": "Payments", "technical": "Product errors"},
    )
    rating = jev.score(
        state={"candidate": "The export is broken."},
        question="Rate the impact.",
        criteria=["minor", "major"],
    )
```

The single-question result objects also expose optional usage metadata.

### Use a check during Mellea generation

For an existing Mellea session `m`, pass the adapter's requirement to
`instruct()`. Keep the Jev client open until sampling finishes because the
requirement calls it during validation:

```python
from mellea.stdlib.sampling import RepairTemplateStrategy
from mellea_jev import JevClient, JevVerifier, accepted_text

with JevClient() as jev:
    verifier = JevVerifier(
        jev,
        "The candidate states the opening time supported by the source.",
        reference="The museum opens at 10:00.",
    )
    sampled = m.instruct(
        "State the museum's opening time using this source: {{source}}",
        user_variables={"source": "The museum opens at 10:00."},
        requirements=[verifier.as_requirement()],
        strategy=RepairTemplateStrategy(loop_budget=3, concurrency_budget=1),
        return_sampling_results=True,
    )
    answer = accepted_text(sampled)
```

`accepted_text()` checks the final validation state before returning text. Do
not return `sampled.result` directly after failed or incomplete sampling.

### Run checks with local Laya-MLX

On Apple Silicon macOS, install the optional backend and load a Laya checkpoint.
The first load may download the model weights; inference then runs locally.

```bash
pip install -e '.[laya]'
```

```python
import laya_mlx
from mellea_jev import JevClassifier
from mellea_jev.providers import LayaProvider

agent = laya_mlx.load("aac6fef/laya-mlx")
classifier = JevClassifier(
    LayaProvider(agent),
    "Which team should handle this request?",
    criteria={"billing": "Payments and refunds", "technical": "Bugs and outages"},
)
print(classifier.classify("I was charged twice.").choice)
```

`LayaProvider` accepts an already loaded agent and does not import Laya or MLX
when the base package is imported. It maps Laya's Noul, Choice, and Score
answers into the same response contracts used by `JevClient`. Review the
selected model checkpoint's license separately; the Laya-MLX runtime is
Apache-2.0 licensed. Laya computes Choice confidence from normalized entropy,
so calibrate `minimum_confidence` for the selected backend rather than copying
a threshold from another provider. See the [Laya-MLX API and platform notes](https://github.com/mizorewww/laya-mlx)
and [confidence implementation](https://github.com/mizorewww/laya-mlx/blob/main/laya_mlx/common.py).

### Evaluate a provider on labeled examples

For image work, the [synthetic crosswalk reference](examples/evaluation/visual/README.md)
provides twelve labeled PNG scenes with geometry-derived positive/negative
cases and asset hashes. Run `make visual-reference` to check the bundle offline.
Its visual format is separate from the text evaluator below. The
[visual runner](examples/evaluation/visual/README.md#evaluate-jev-and-luna)
compares Jev and Luna with shared quality and latency reports:

```bash
# Explicit opt-in: up to twelve billable image uploads; use a new receipt path.
make visual-live-evaluate EVAL_PROVIDER=typesafe EVAL_MODEL=jev-1.14.0 \
  EVAL_PREDICTIONS=/tmp/jev-visual.jsonl
# Replay offline without constructing a provider.
make visual-evaluate EVAL_PREDICTIONS=/tmp/jev-visual.jsonl
```

See the visual reference guide for Luna, Proxy configuration, and multi-provider
comparison. The [October 15 visual comparison](examples/evaluation/visual/live-comparison-2026-10-15.md)
records 24 successful calls and their quality/latency metrics. These diagrams do
not establish quality on real photographs or parking-law decisions.

`examples/evaluate.py` reports false-acceptance, false-rejection, and uncertain
rates for every provider, returned model, and threshold pair. The checked-in
museum example is a small format demonstration, not a quality benchmark. Add
representative, non-sensitive examples for your own task before drawing quality
conclusions. See the [initial live Jev and Laya run](examples/evaluation/live-benchmark-2026-09-21.md)
for a four-example smoke benchmark and its limitations.
The [live Proxy smoke run](examples/evaluation/live-proxy-benchmark-2026-10-08.md)
records the same dataset through a TypeSafe-compatible Proxy, with saved
predictions and offline-reproducible metrics.
The [ScaleDown live smoke run](examples/evaluation/scaledown-live-benchmark-2026-10-14.md)
records four predictions and threshold comparisons; it demonstrates API
compatibility, not production quality.

ScaleDown uses the same runner. `SCALEDOWN_API_KEY` is read only when
`--live` is selected; custom gateways require an explicitly selected key
environment variable:

```bash
make live-evaluate EVAL_PROVIDER=scaledown EVAL_THRESHOLD=0.10,0.90
```

The default model is `classify-1`. Add `EVAL_PREDICTIONS=...` to save the
provider outputs for offline threshold comparisons; snapshots contain no API key
or endpoint URL.

The dataset is versioned JSONL: the first line describes the positive Noul
requirement; each following line labels one candidate as `accept` or `reject`.
An optional `reference` is sent with that candidate. For example:

```jsonl
{"type":"dataset","format_version":1,"name":"support-policy","version":"1.0.0","requirement":"The answer follows the refund policy."}
{"type":"example","id":"in-policy","candidate":"...","reference":"...","expected":"accept"}
{"type":"example","id":"out-of-policy","candidate":"...","reference":"...","expected":"reject"}
```

Inference is opt-in. This command sends each example to Jev and saves the raw
predictions, so it may incur charges and transmits dataset text to TypeSafe:

```bash
python examples/evaluate.py examples/evaluation/museum_opening.jsonl \
  --live --provider typesafe --model jev-latest \
  --threshold 0.10,0.90 --save-predictions /tmp/museum-predictions.jsonl
```

OpenAI uses the same Noul dataset and metric definitions. Install the optional
`openai` extra and set `OPENAI_API_KEY` before this opt-in, potentially billable run:

```bash
python examples/evaluate.py examples/evaluation/museum_opening.jsonl \
  --live --provider openai --model gpt-6-luna \
  --threshold 0.10,0.90 --save-predictions /tmp/openai-predictions.jsonl
```

For an OpenAI-compatible Proxy, pass its API prefix; the SDK appends `/decisions`:

```bash
make live-evaluate EVAL_PROVIDER=openai EVAL_MODEL=gpt-6-luna \
  EVAL_BASE_URL=https://proxy.example/openai/v1 EVAL_API_KEY_ENV=PROXY_API_KEY \
  EVAL_PROVIDER_LABEL=openai-proxy-a EVAL_PREDICTIONS=/tmp/openai-proxy-predictions.jsonl
```

For a TypeSafe-compatible Proxy, explicitly select its API root and the
environment variable containing its key. The SDK adds `/v1/systemone` to the
root. This example sends the dataset to your configured Proxy:

```bash
python examples/evaluate.py examples/evaluation/museum_opening.jsonl \
  --live --provider typesafe --model jev-latest \
  --base-url https://proxy.example/typesafe/api --api-key-env PROXY_API_KEY \
  --provider-label typesafe-proxy-a \
  --save-predictions /tmp/proxy-predictions.jsonl
```

Replace the placeholder URL and set `PROXY_API_KEY` before running. The
equivalent Makefile command is:

```bash
make live-evaluate EVAL_PROVIDER=typesafe \
  EVAL_BASE_URL=https://proxy.example/typesafe/api EVAL_API_KEY_ENV=PROXY_API_KEY \
  EVAL_PROVIDER_LABEL=typesafe-proxy-a EVAL_PREDICTIONS=/tmp/proxy-predictions.jsonl
```

When a Proxy changes the operation path, replace `--base-url` with
`--endpoint-url`, or `EVAL_BASE_URL` with `EVAL_ENDPOINT_URL`. The full URL is
used without adding a suffix:

```bash
make live-evaluate EVAL_PROVIDER=openai EVAL_MODEL=gpt-6-luna \
  EVAL_ENDPOINT_URL=https://proxy.example/models/luna/decide EVAL_API_KEY_ENV=PROXY_API_KEY \
  EVAL_PROVIDER_LABEL=openai-proxy-a EVAL_PREDICTIONS=/tmp/openai-proxy-predictions.jsonl
```

`--provider` selects the implementation; `--provider-label` sets the `provider`
field in saved predictions and metric groups. With an explicit custom URL,
the default label is `typesafe-proxy` or `openai-proxy`; otherwise it is the implementation name.
Use distinct labels for distinct Proxy sources, even if they return the same
model name. Both URL flags require `--api-key-env` and are mutually exclusive. Connection flags
require live TypeSafe or OpenAI mode; labels require live
mode and can also distinguish Laya checkpoints. URLs and keys are not saved in
prediction snapshots. Without connection flags, TypeSafe still uses its
official endpoint and `TYPESAFE_API_KEY`; OpenAI uses its official endpoint and
`OPENAI_API_KEY`. Offline scoring remains unchanged. Thresholds need calibration
for each provider and task; identical numeric thresholds do not establish comparable quality.

To compare multiple threshold pairs or reproduce a report, load the saved
predictions offline. No provider is constructed and no request is sent:

```bash
python examples/evaluate.py examples/evaluation/museum_opening.jsonl \
  --predictions /tmp/museum-predictions.jsonl \
  --threshold 0.10,0.90 --threshold 0.20,0.80
```

Use `--live --provider laya --model aac6fef/laya-mlx` to run the same labeled
examples through Laya-MLX on a supported Apple Silicon setup. Loading that
checkpoint may download model weights. Prediction snapshots contain `p_yes`,
provider, and returned model; keep them with the dataset version. The report
also includes the SHA-256 of the exact dataset file, so a changed file cannot be
silently paired with old predictions. A snapshot records the provider, returned
model, raw probability, and elapsed call time in milliseconds for each example.
New snapshots use prediction format version 2; dataset format version remains 1:

```jsonl
{"type":"prediction_set","format_version":2,"dataset":"support-policy","dataset_version":"1.0.0","dataset_sha256":"..."}
{"type":"prediction","id":"in-policy","provider":"typesafe","model":"jev-1.13.0","p_yes":0.98,"elapsed_ms":304.2}
```

The JSON report includes a separate `latency` section, grouped once by provider
and returned model, regardless of how many threshold pairs you compare. Each row
contains `measured_count`, `missing_count`, `min_ms`, `max_ms`, `mean_ms`,
`stddev_ms`, `median_ms`, `p90_ms`, `p95_ms`, and `p99_ms`. Standard deviation
describes the measured population. Percentiles use linear interpolation at
position `(n - 1) * p` in sorted measurements; the method and units are also
included in the report. With one measurement, every percentile equals that
measurement and standard deviation is zero.

Timing uses a monotonic high-resolution clock around each successful
`JevVerifier.evaluate()` call. It includes request preparation, network round
trip (for remote providers), inference, and response validation. It excludes
provider construction, local-model loading, snapshot saving, and report
generation. Calls run sequentially through one provider instance; the first
call is included, so DNS, TLS, or other connection setup can affect it. No
warm-up calls or repetitions are added. This measures the caller's response
latency; it does not isolate server inference time. A failed evaluation stops
the run under the existing error policy and does not contribute a success timing.

Offline replay reproduces the same timing summary without another API call.
Version 1 snapshots still load; their timing is unavailable. Missing timing
(omitted `elapsed_ms` or `null` in version 2) is counted in `missing_count` and
excluded from the distribution. All timing statistics are `null` if no
measurements exist. Numeric durations must be finite and nonnegative. Older
readers cannot read version 2 snapshots.

Tail percentiles from a small dataset are descriptive: with 24 calls, p99 is
close to the slowest observation. Publish the measurement count and use repeated
runs under the same network conditions before drawing conclusions about p95,
p99, or provider speed. Historical snapshots without timing cannot reconstruct
per-call latency from a total runner duration.

The report defines false-acceptance rate as false accepts divided by expected
rejects, false-rejection rate as false rejects divided by expected accepts, and
uncertain rate as uncertain predictions divided by all examples. It also
reports counts and denominators; missing classes are rejected during dataset
loading. Always publish the dataset version, sample count, provider/model,
thresholds, and limitations alongside any observed quality rates. Do not commit
private or sensitive examples or prediction snapshots.

## Requirements and compatibility

- Python **3.11 or newer**.
- Mellea **0.7.x or 0.8.x** for the `Requirement` integration.
- TypeSafe API access and `TYPESAFE_API_KEY` for live Jev requests. Mocked tests and `make demo` need no key.
- The TypeSafe provider uses the official TypeSafe Python SDK **>=0.7.5,<0.8.0**
  and its synchronous HTTPX2 transport. Request compression is explicitly disabled
  to preserve compatibility with existing gateways.
- `laya-mlx` is optional and supported by its upstream project on Apple Silicon macOS.
- OpenAI verification requires the optional `openai` extra (SDK **3.26.0 or newer, below 4**)
  and `OPENAI_API_KEY`. `OpenAIProvider` supports Noul, Choice, and Score;
  `gpt-6-luna` is its default model.
- ScaleDown Decisions requires `SCALEDOWN_API_KEY` for live requests.
  `ScaleDownProvider` supports Noul, Choice, and Score through `classify-1`.
- SubQ Decisions requires `SUBQ_API_KEY` for live requests.
  `SubQProvider` supports Noul, Choice, and Score through the preview model
  `subq-snap-preview`.

The CI compatibility matrix runs the package checks with these combinations:

| Python | Mellea integration |
| --- | --- |
| 3.11 | 0.7.0, 0.8.0 |
| 3.12 | 0.7.0, 0.8.0 |
| 3.13 | 0.7.0, 0.8.0 |
| 3.14 | 0.7.0, 0.8.0 |

The Mellea extra accepts `>=0.7.0,<0.9.0`. Development commands default to 0.8.0;
run `make check MELLEA_VERSION=0.7.0` to check the older supported release.
CI checks both listed releases; other minor versions are not declared compatible.

The Mellea requirement callback is synchronous, so a Jev request can block the
event loop. This package does not provide an async client. If sampling has
already seen a failed candidate, Mellea may return a failed sampling result
instead of propagating a later Jev error or uncertain verdict; inspect the
final result with `accepted_text()`. See [API notes](API_NOTES.md) for external
contracts and [test report](TEST_REPORT.md) for the evidence behind the current
prototype status.

The Mellea helpers depend on small structural protocols: `NoulProvider`,
`ChoiceProvider`, and `ScoreProvider` (or the combined `PrimitiveProvider`). A
custom backend can implement only the primitive it needs; it does not need to
inherit from a package class. Its response must expose the fields in
`NoulResponse`, `ChoiceResponse`, or `ScoreResponse`. The current criteria
shapes follow the TypeSafe request model, and each provider may impose its own
limits. Batched requests remain a TypeSafe feature. Use `JevClient` as the
existing compatible name, or import `TypeSafeProvider` explicitly from
`mellea_jev.providers`. See [provider contracts](src/mellea_jev/contracts.py).

## Development and further reading

```bash
make help    # list the repository commands
make check   # run lint, formatting, type, test, coverage, and whitespace checks
make demo    # run without API keys or network access
```

GitHub CI runs the same `make check` gate: Ruff linting and formatting,
strict Mypy checks, a maximum cyclomatic complexity of 16, and at least 80%
branch-aware coverage. Live service requests remain opt-in and are not part of CI.

- [API notes and source references](API_NOTES.md)
- [Test report and validation limits](TEST_REPORT.md)
- [Development commands](Makefile)
- [Release process](RELEASING.md)
- [Changelog](CHANGELOG.md)
- [Roadmap](roadmap.md)
- [Offline demo](examples/offline_demo.py)
- [Live Jev example](examples/live_check.py)

Before using the adapter with private data, account for the fact that candidate
text and any supplied reference are sent to TypeSafe. The adapter does not log
request bodies or API keys, and it does not follow redirects or retry requests
automatically. See the [API notes](API_NOTES.md) for details.

Licensed under MIT. This project is unofficial and is not affiliated with Mellea, IBM, or TypeSafe.
