Metadata-Version: 2.4
Name: mellea-jev-adapter
Version: 0.3.0
Summary: Unofficial TypeSafe Jev Noul verifier and Choice classifier for Mellea
License: MIT
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: httpx2<3,>=2.0.0
Requires-Dist: pydantic<3,>=2.12.0
Requires-Dist: typesafe-sdk<0.8.0,>=0.7.0
Provides-Extra: mellea
Requires-Dist: mellea<0.9.0,>=0.7.0; extra == "mellea"
Provides-Extra: openai
Requires-Dist: openai<4,>=3.26.0; extra == "openai"
Provides-Extra: dev
Requires-Dist: openai<4,>=3.26.0; extra == "dev"
Requires-Dist: mypy<2,>=1.15; extra == "dev"
Requires-Dist: pytest<10,>=8.3; extra == "dev"
Requires-Dist: pytest-cov<8,>=6; extra == "dev"
Requires-Dist: ruff<0.17,>=0.16; extra == "dev"
Provides-Extra: laya
Requires-Dist: laya-mlx<0.2.0,>=0.1.0; (sys_platform == "darwin" and platform_machine == "arm64") and extra == "laya"
Dynamic: license-file

# Mellea × Jev

[![CI](https://github.com/SoundBlaster/Jev4Mellea/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/SoundBlaster/Jev4Mellea/actions/workflows/ci.yml)
[![Version](https://img.shields.io/github/v/tag/SoundBlaster/Jev4Mellea?label=version)](https://github.com/SoundBlaster/Jev4Mellea/tags)
![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-blue?logo=python&logoColor=white)
![Mellea 0.7–0.8](https://img.shields.io/badge/Mellea-0.7%E2%80%930.8-6f42c1)
![TypeSafe SDK 0.7.x](https://img.shields.io/badge/TypeSafe%20SDK-0.7.x-6f42c1)
[![Coverage gate: 80%](https://img.shields.io/badge/coverage%20gate-%E2%89%A580%25-success)](https://github.com/SoundBlaster/Jev4Mellea/blob/main/pyproject.toml#L68-L70)

A small Python adapter that brings TypeSafe Jev's semantic checks into
[Mellea](https://github.com/generative-computing/mellea). Use Jev to verify
generated text, classify it into your labels, or rate it on a scale. Mellea
continues to manage generation and repair.

- **Verify requirements** with Jev Noul and configurable accept/reject thresholds.
- **Classify text** with TypeSafe Choice and caller-defined categories.
- **Score text** on an ordered scale with TypeSafe Score.
- **Ask several questions in one request** and inspect returned token usage.
- **Connect checks to Mellea `Requirement`s** so they can participate in sampling.

This is an unofficial, synchronous adapter. Jev evaluates text; it does not
generate or repair it.

## Quick start

Install from a checkout and set an API key from the
[TypeSafe console](https://console.typesafe.ai/). Live requests may incur
charges.

```bash
git clone https://github.com/SoundBlaster/Jev4Mellea.git
cd Jev4Mellea
make install
export TYPESAFE_API_KEY='your-key'
```

The Makefile defaults to Python 3.13. To use another supported interpreter,
run `make install PYTHON=python3.11` (or set `PYTHON` to your installed version).

Ask whether a candidate meets a positive requirement:

```python
from mellea_jev import JevClient, JevVerifier

with JevClient() as jev:
    verifier = JevVerifier(
        jev,
        "The answer gives the museum's opening time from the reference.",
        reference="The museum opens at 10:00 and closes at 18:00.",
    )
    verdict = verifier.evaluate("The museum opens at 10:00.")
    print(verdict.outcome, verdict.p_yes)
```

`outcome` is `pass`, `fail`, or `uncertain`. Use `verifier.as_requirement()` to
attach the same check to a Mellea generation flow. Once the project dependencies
are installed, `make demo` runs without a key or network access using a mocked
Jev response.

## Examples

### Jev through a Proxy

Use your Proxy's key and explicitly select its TypeSafe-compatible API root. The SDK adds
`/v1/systemone`; do not include that suffix in `base_url`.

```python
import os
from mellea_jev import JevClient, NoulQuestion

with JevClient(
    api_key=os.environ["PROXY_API_KEY"],
    base_url="https://proxy.example/typesafe/api",
    model="jev-latest",
) as jev:
    result = jev.system_one(
        state={"text": "Hello! Thank you for your message."},
        questions={"greeting": NoulQuestion("Does the text contain a polite greeting?")},
    )
    print(result.answers["greeting"])
```

The adapter retains its dictionary `state` contract. Without an explicit
`base_url`, it uses the official TypeSafe endpoint and ignores
`TYPESAFE_BASE_URL`. Requests never follow redirects or retry automatically.
Replace the example URL with your Proxy's API root. From a checkout,
`make proxy-live-check` runs one potentially billable verification with
`PROXY_API_KEY` and `PROXY_BASE_URL` set in the environment; CI uses mocks.

### Use OpenAI Decisions API

From a checkout, install the optional official SDK with
`pip install -e '.[openai]'` and set `OPENAI_API_KEY`.
The Decisions API is in public beta. This provider supports Noul verification,
Choice classification, and Score rubrics, with one question per request.

```python
from mellea_jev import JevVerifier, OpenAIProvider

with OpenAIProvider() as provider:
    verifier = JevVerifier(
        provider,
        "The candidate gives the opening time supported by the reference.",
        reference="The museum opens at 10:00.",
        accept_at=0.90,
        reject_at=0.10,
    )
    verdict = verifier.evaluate("The museum opens at 10:00.")
    print(verdict.outcome, verdict.p_yes, verdict.request_id)
    requirement = verifier.as_requirement()
```

The same provider works with the existing classifier and Mellea requirement:

```python
from mellea_jev import JevClassifier, OpenAIProvider

with OpenAIProvider() as provider:
    classifier = JevClassifier(
        provider,
        "Which team should handle the request?",
        criteria={"billing": "Payments and refunds", "technical": "Product errors"},
    )
    result = classifier.classify("I was charged twice.")
    print(result.choice, result.confidence, result.probabilities)
    requirement = classifier.as_requirement("billing", minimum_confidence=0.80)
```

Choice preserves the SDK's confidence separately from the label probabilities.
Labels must be strings and the response must contain the complete configured distribution.

Score uses the same ordered rubric and inclusive bounds as other providers:

```python
from mellea_jev import JevScorer, OpenAIProvider

with OpenAIProvider() as provider:
    scorer = JevScorer(
        provider, "Rate the severity of the issue.", criteria=["Low", "Medium", "High"]
    )
    result = scorer.evaluate("The button is slightly misaligned.")
    print(result.score, result.confidence, result.legend)
    requirement = scorer.as_requirement(maximum_score=1.0, minimum_confidence=0.80)
```

The score is a fractional position from zero to `len(criteria) - 1`, checked
against the probability-weighted level average. Confidence remains a separate value.

The provider sends the state dictionary as JSON text in `input`. Optional Noul
`criteria` descriptions are appended as JSON to the predicate instructions;
Decisions predicates do not have a separate outcome-criteria field. Refusals
raise `OpenAIRefusalError`; transport, HTTP, and invalid-response errors raise
`OpenAIProviderError` subclasses. These errors never become accepted verdicts.
Requests use the official endpoint by default, without automatic retries or redirects.
For a Proxy, pass an explicit `base_url` (including the API
prefix such as `/v1`) and the Proxy's key:

```python
import os

with OpenAIProvider(
    api_key=os.environ["PROXY_API_KEY"],
    base_url="https://proxy.example/v1",
) as provider:
    result = provider.noul(state={"candidate": "Hello!"}, question="Is it a greeting?")
```

The gateway must implement the Decisions `/decisions` contract; support for
Chat Completions alone is insufficient. The configured URL receives the supplied
key. HTTP and HTTPS URLs are supported; `OPENAI_BASE_URL` is ignored.
Thresholds require evaluation on your own data before comparing them with Jev.

From a development checkout, set `OPENAI_API_KEY` locally and run:

```bash
make openai-live-test
```

This explicitly opts in to up to three potentially billable requests to the
official endpoint: one each for Noul, Choice, and Score. The command stops at
the first failure; requests have no automatic retries. To run just one primitive:

```bash
RUN_LIVE_OPENAI=1 .venv/bin/python -m pytest -q -x --tb=short \
  tests/test_live_openai.py -k choice
```

These are contract smoke tests: they check valid probabilities, complete labels
or score levels, response model and request ID, and Choice/Score token usage.
They do not assert a particular class, confidence threshold, or semantic score.
Refusal and API failures fail the test. Ordinary tests skip live requests even
when a key is present; direct pytest runs require both `RUN_LIVE_OPENAI=1` and
`OPENAI_API_KEY`. See the [official Decisions guide](https://developers.openai.com/api/docs/guides/decisions).

### Verify a requirement with Noul

Noul returns `p_yes`, the probability that a positively phrased requirement is
satisfied. The adapter uses two configurable thresholds; the space between
them is uncertain and is never silently accepted.

```python
from mellea_jev import JevClient, JevVerifier

with JevClient() as jev:
    verifier = JevVerifier(
        jev,
        "The candidate states the opening time supported by the reference.",
        reference="The museum opens at 10:00.",
        criteria={
            "true": "The candidate gives 10:00 as the opening time.",
            "false": "The candidate omits or contradicts the opening time.",
        },
        accept_at=0.90,
        reject_at=0.10,
        repair_hint="Use the opening time stated in the reference.",
    )
    verdict = verifier.evaluate("The museum opens at 10:00.")

    if verdict.outcome == "uncertain":
        print("Route for another check or human review")
```

The threshold values are application policy, not accuracy guarantees. Noul does
not return a separate confidence field or a textual explanation.

To use the verifier as a Mellea requirement:

```python
requirement = verifier.as_requirement()
```

For generation and repair with a real Mellea session, see
[`examples/mellea_ollama.py`](examples/mellea_ollama.py). That example uses
Ollama for generation and Jev for verification.

### Classify into configured categories with Choice

Choice selects one label from the supplied criteria and returns its confidence
and the probability of every label. Descriptions may be strings, JSON objects,
arrays, or `None`.

```python
from mellea_jev import JevClient, JevClassifier

criteria = {
    "billing": {"what": "Payments, invoices, or refunds", "examples": ["duplicate charge"]},
    "technical": "Product errors or problems using the service",
    "other": None,
}

with JevClient() as jev:
    classifier = JevClassifier(
        jev,
        "Choose the best category for this support message.",
        criteria=criteria,
    )
    result = classifier.classify("I was charged twice for my subscription.")
    print(result.choice, result.confidence, result.probabilities)
```

To require a Mellea candidate to be assigned to a particular category:

```python
requirement = classifier.as_requirement(
    "billing",
    minimum_confidence=0.75,
)
```

Confidence thresholds are caller policy. TypeSafe supports up to 255 Choice
labels.

### Rate text on an ordered scale with Score

Score returns a probability-weighted position on an ordered scale. The result
can fall between two levels.

```python
from mellea_jev import JevClient, JevScorer

levels = ["Cosmetic", "Workaround exists", "Blocking"]

with JevClient() as jev:
    scorer = JevScorer(
        jev,
        "How severe is the reported issue?",
        criteria=levels,
    )
    result = scorer.evaluate("The export button crashes and there is no workaround.")
    print(result.score, result.confidence, result.probabilities)
```

The result can also become a Mellea requirement with inclusive score bounds:

```python
requirement = scorer.as_requirement(
    minimum_score=1.0,
    maximum_score=2.0,
    minimum_confidence=0.6,
)
```

Score supports 2–10 ordered string descriptions.

### Batch questions and read usage metadata

`JevClient.system_one()` sends named Noul, Choice, and Score questions in one
request. Each answer remains a typed result. The returned usage counts are
informational and do not affect validation.

```python
from mellea_jev import ChoiceQuestion, JevClient, NoulQuestion

with JevClient() as jev:
    result = jev.system_one(
        state={"candidate": "I was charged twice and cannot log in."},
        questions={
            "urgent": NoulQuestion("Does the message convey urgency?"),
            "team": ChoiceQuestion(
                "Which team should handle this?",
                {"billing": "Payments and invoices", "technical": "Product errors"},
            ),
        },
    )

    print(result.answers["urgent"].p_yes)
    print(result.answers["team"].choice)
    if result.usage is not None:
        print(result.usage.input_tokens, result.usage.output_tokens)
```

For one-off checks without the Mellea helper classes, call the client methods
directly. Each method sends its own request:

```python
with JevClient() as jev:
    yes_no = jev.noul(
        state={"candidate": "The museum opens at 10:00."},
        question="Does the candidate state the opening time?",
    )
    category = jev.choice(
        state={"candidate": "I was charged twice."},
        question="Choose a category.",
        criteria={"billing": "Payments", "technical": "Product errors"},
    )
    rating = jev.score(
        state={"candidate": "The export is broken."},
        question="Rate the impact.",
        criteria=["minor", "major"],
    )
```

The single-question result objects also expose optional usage metadata.

### Use a check during Mellea generation

For an existing Mellea session `m`, pass the adapter's requirement to
`instruct()`. Keep the Jev client open until sampling finishes because the
requirement calls it during validation:

```python
from mellea.stdlib.sampling import RepairTemplateStrategy
from mellea_jev import JevClient, JevVerifier, accepted_text

with JevClient() as jev:
    verifier = JevVerifier(
        jev,
        "The candidate states the opening time supported by the source.",
        reference="The museum opens at 10:00.",
    )
    sampled = m.instruct(
        "State the museum's opening time using this source: {{source}}",
        user_variables={"source": "The museum opens at 10:00."},
        requirements=[verifier.as_requirement()],
        strategy=RepairTemplateStrategy(loop_budget=3, concurrency_budget=1),
        return_sampling_results=True,
    )
    answer = accepted_text(sampled)
```

`accepted_text()` checks the final validation state before returning text. Do
not return `sampled.result` directly after failed or incomplete sampling.

### Run checks with local Laya-MLX

On Apple Silicon macOS, install the optional backend and load a Laya checkpoint.
The first load may download the model weights; inference then runs locally.

```bash
pip install -e '.[laya]'
```

```python
import laya_mlx
from mellea_jev import JevClassifier
from mellea_jev.providers import LayaProvider

agent = laya_mlx.load("aac6fef/laya-mlx")
classifier = JevClassifier(
    LayaProvider(agent),
    "Which team should handle this request?",
    criteria={"billing": "Payments and refunds", "technical": "Bugs and outages"},
)
print(classifier.classify("I was charged twice.").choice)
```

`LayaProvider` accepts an already loaded agent and does not import Laya or MLX
when the base package is imported. It maps Laya's Noul, Choice, and Score
answers into the same response contracts used by `JevClient`. Review the
selected model checkpoint's license separately; the Laya-MLX runtime is
Apache-2.0 licensed. Laya computes Choice confidence from normalized entropy,
so calibrate `minimum_confidence` for the selected backend rather than copying
a threshold from another provider. See the [Laya-MLX API and platform notes](https://github.com/mizorewww/laya-mlx)
and [confidence implementation](https://github.com/mizorewww/laya-mlx/blob/main/laya_mlx/common.py).

### Evaluate a provider on labeled examples

`examples/evaluate.py` reports false-acceptance, false-rejection, and uncertain
rates for every provider, returned model, and threshold pair. The checked-in
museum example is a small format demonstration, not a quality benchmark. Add
representative, non-sensitive examples for your own task before drawing quality
conclusions. See the [initial live Jev and Laya run](examples/evaluation/live-benchmark-2026-09-21.md)
for a four-example smoke benchmark and its limitations.
The [live Proxy smoke run](examples/evaluation/live-proxy-benchmark-2026-10-08.md)
records the same dataset through a TypeSafe-compatible Proxy, with saved
predictions and offline-reproducible metrics.

The dataset is versioned JSONL: the first line describes the positive Noul
requirement; each following line labels one candidate as `accept` or `reject`.
An optional `reference` is sent with that candidate. For example:

```jsonl
{"type":"dataset","format_version":1,"name":"support-policy","version":"1.0.0","requirement":"The answer follows the refund policy."}
{"type":"example","id":"in-policy","candidate":"...","reference":"...","expected":"accept"}
{"type":"example","id":"out-of-policy","candidate":"...","reference":"...","expected":"reject"}
```

Inference is opt-in. This command sends each example to Jev and saves the raw
predictions, so it may incur charges and transmits dataset text to TypeSafe:

```bash
python examples/evaluate.py examples/evaluation/museum_opening.jsonl \
  --live --provider typesafe --model jev-latest \
  --threshold 0.10,0.90 --save-predictions /tmp/museum-predictions.jsonl
```

OpenAI uses the same Noul dataset and metric definitions. Install the optional
`openai` extra and set `OPENAI_API_KEY` before this opt-in, potentially billable run:

```bash
python examples/evaluate.py examples/evaluation/museum_opening.jsonl \
  --live --provider openai --model gpt-6-luna \
  --threshold 0.10,0.90 --save-predictions /tmp/openai-predictions.jsonl
```

For an OpenAI-compatible Proxy, pass its API prefix; the SDK appends `/decisions`:

```bash
make live-evaluate EVAL_PROVIDER=openai EVAL_MODEL=gpt-6-luna \
  EVAL_BASE_URL=https://proxy.example/openai/v1 EVAL_API_KEY_ENV=PROXY_API_KEY \
  EVAL_PROVIDER_LABEL=openai-proxy-a EVAL_PREDICTIONS=/tmp/openai-proxy-predictions.jsonl
```

For a TypeSafe-compatible Proxy, explicitly select its API root and the
environment variable containing its key. The SDK adds `/v1/systemone` to the
root. This example sends the dataset to your configured Proxy:

```bash
python examples/evaluate.py examples/evaluation/museum_opening.jsonl \
  --live --provider typesafe --model jev-latest \
  --base-url https://proxy.example/typesafe/api --api-key-env PROXY_API_KEY \
  --provider-label typesafe-proxy-a \
  --save-predictions /tmp/proxy-predictions.jsonl
```

Replace the placeholder URL and set `PROXY_API_KEY` before running. The
equivalent Makefile command is:

```bash
make live-evaluate EVAL_PROVIDER=typesafe \
  EVAL_BASE_URL=https://proxy.example/typesafe/api EVAL_API_KEY_ENV=PROXY_API_KEY \
  EVAL_PROVIDER_LABEL=typesafe-proxy-a EVAL_PREDICTIONS=/tmp/proxy-predictions.jsonl
```

`--provider` selects the implementation; `--provider-label` sets the `provider`
field in saved predictions and metric groups. With an explicit `--base-url`,
the default label is `typesafe-proxy` or `openai-proxy`; otherwise it is the implementation name.
Use distinct labels for distinct Proxy sources, even if they return the same
model name. An explicit `--base-url` requires `--api-key-env`. Connection flags
require live TypeSafe or OpenAI mode; labels require live
mode and can also distinguish Laya checkpoints. URLs and keys are not saved in
prediction snapshots. Without connection flags, TypeSafe still uses its
official endpoint and `TYPESAFE_API_KEY`; OpenAI uses its official endpoint and
`OPENAI_API_KEY`. Offline scoring remains unchanged. Thresholds need calibration
for each provider and task; identical numeric thresholds do not establish comparable quality.

To compare multiple threshold pairs or reproduce a report, load the saved
predictions offline. No provider is constructed and no request is sent:

```bash
python examples/evaluate.py examples/evaluation/museum_opening.jsonl \
  --predictions /tmp/museum-predictions.jsonl \
  --threshold 0.10,0.90 --threshold 0.20,0.80
```

Use `--live --provider laya --model aac6fef/laya-mlx` to run the same labeled
examples through Laya-MLX on a supported Apple Silicon setup. Loading that
checkpoint may download model weights. Prediction snapshots contain `p_yes`,
provider, and returned model; keep them with the dataset version. The report
also includes the SHA-256 of the exact dataset file, so a changed file cannot be
silently paired with old predictions. A snapshot records the provider, returned
model, and raw probability for each example:

```jsonl
{"type":"prediction_set","format_version":1,"dataset":"support-policy","dataset_version":"1.0.0","dataset_sha256":"..."}
{"type":"prediction","id":"in-policy","provider":"typesafe","model":"jev-1.13.0","p_yes":0.98}
```

The report defines false-acceptance rate as false accepts divided by expected
rejects, false-rejection rate as false rejects divided by expected accepts, and
uncertain rate as uncertain predictions divided by all examples. It also
reports counts and denominators; missing classes are rejected during dataset
loading. Always publish the dataset version, sample count, provider/model,
thresholds, and limitations alongside any observed quality rates. Do not commit
private or sensitive examples or prediction snapshots.

## Requirements and compatibility

- Python **3.11 or newer**.
- Mellea **0.7.x or 0.8.x** for the `Requirement` integration.
- TypeSafe API access and `TYPESAFE_API_KEY` for live Jev requests. Mocked tests and `make demo` need no key.
- The TypeSafe provider uses the official TypeSafe Python SDK and its synchronous HTTPX2 transport.
- `laya-mlx` is optional and supported by its upstream project on Apple Silicon macOS.
- OpenAI verification requires the optional `openai` extra (SDK **3.26.0 or newer, below 4**)
  and `OPENAI_API_KEY`. `OpenAIProvider` supports Noul, Choice, and Score;
  `gpt-6-luna` is its default model.

The CI compatibility matrix runs the package checks with these combinations:

| Python | Mellea integration |
| --- | --- |
| 3.11 | 0.7.0, 0.8.0 |
| 3.12 | 0.7.0, 0.8.0 |
| 3.13 | 0.7.0, 0.8.0 |
| 3.14 | 0.7.0, 0.8.0 |

The Mellea extra accepts `>=0.7.0,<0.9.0`. Development commands default to 0.8.0;
run `make check MELLEA_VERSION=0.7.0` to check the older supported release.
CI checks both listed releases; other minor versions are not declared compatible.

The Mellea requirement callback is synchronous, so a Jev request can block the
event loop. This package does not provide an async client. If sampling has
already seen a failed candidate, Mellea may return a failed sampling result
instead of propagating a later Jev error or uncertain verdict; inspect the
final result with `accepted_text()`. See [API notes](API_NOTES.md) for external
contracts and [test report](TEST_REPORT.md) for the evidence behind the current
prototype status.

The Mellea helpers depend on small structural protocols: `NoulProvider`,
`ChoiceProvider`, and `ScoreProvider` (or the combined `PrimitiveProvider`). A
custom backend can implement only the primitive it needs; it does not need to
inherit from a package class. Its response must expose the fields in
`NoulResponse`, `ChoiceResponse`, or `ScoreResponse`. The current criteria
shapes follow the TypeSafe request model, and each provider may impose its own
limits. Batched requests remain a TypeSafe feature. Use `JevClient` as the
existing compatible name, or import `TypeSafeProvider` explicitly from
`mellea_jev.providers`. See [provider contracts](src/mellea_jev/contracts.py).

## Development and further reading

```bash
make help    # list the repository commands
make check   # run lint, formatting, type, test, coverage, and whitespace checks
make demo    # run without API keys or network access
```

GitHub CI runs the same `make check` gate: Ruff linting and formatting,
strict Mypy checks, a maximum cyclomatic complexity of 16, and at least 80%
branch-aware coverage. Live service requests remain opt-in and are not part of CI.

- [API notes and source references](API_NOTES.md)
- [Test report and validation limits](TEST_REPORT.md)
- [Development commands](Makefile)
- [Release process](RELEASING.md)
- [Changelog](CHANGELOG.md)
- [Roadmap](roadmap.md)
- [Offline demo](examples/offline_demo.py)
- [Live Jev example](examples/live_check.py)

Before using the adapter with private data, account for the fact that candidate
text and any supplied reference are sent to TypeSafe. The adapter does not log
request bodies or API keys, and it does not follow redirects or retry requests
automatically. See the [API notes](API_NOTES.md) for details.

Licensed under MIT. This project is unofficial and is not affiliated with Mellea, IBM, or TypeSafe.
