Metadata-Version: 2.4
Name: jtc-client
Version: 0.1.0
Summary: A standalone asynchronous client for Judges That Code tool services.
Author: Goncalo Faria
License-Expression: MIT
Keywords: llm,judge,rubric,evaluation,literegistry
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Typing :: Typed
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: aiohttp<4,>=3.9
Provides-Extra: test
Requires-Dist: pytest>=8; extra == "test"
Provides-Extra: publish
Requires-Dist: build>=1.2; extra == "publish"
Requires-Dist: twine>=5; extra == "publish"
Dynamic: license-file

# jtc-client

`jtc-client` is a small, standalone Python client for Judges That Code tool
services. The first public client is `JudgeClient`, which calls the `/judge`
endpoint exposed by a JTC/LiteRegistry gateway.

The package intentionally contains no model-serving or dataset code. Its only
runtime dependency is `aiohttp`.

## Install

```bash
pip install jtc-client
```

For local development:

```bash
pip install -e ".[test,publish]"
```

## Judge one output

```python
import asyncio
import json

from jtc_client import JudgeClient


async def main() -> None:
    client = JudgeClient(
        "http://127.0.0.1:1212/judge",
        service_model_path="judge",
        timeout=600,
        max_retries=3,
    )
    response = await client.verify_output(
        input="What is the capital of France?",
        output="The capital of France is Paris.",
        rubrics=[
            "The response correctly identifies Paris as the capital of France."
        ],
        model="/models/my-rubric-judge",
    )
    print(json.dumps(response, indent=2))


asyncio.run(main())
```

`model` selects the LLM that performs the judgment. `service_model_path`
selects the `/judge` service pool through the gateway. They are intentionally
separate fields.

## Lion-4B example

See [`examples/lion4b.py`](examples/lion4b.py) for the concrete Lion-4B judge
configuration used by JTC.

The example expects a local LiteRegistry gateway on port `1212`. For the
existing Jupiter registry, launch one with:

```bash
literegistry gateway \
  --registry redis://jupiter-cs-aus-183.reviz.ai2.in:59936 \
  --port 1212
```

The Lion-4B model pool and the service registered as
`rubrichub-lion4b-step800-per-rubric-judge` must already be alive in that
registry.

## Request contract

The client sends this JSON object to `/judge`:

```json
{
  "input": "original task",
  "output": "candidate response",
  "rubrics": ["one or more textual criteria"],
  "model": "/registered/judge/model",
  "model_path": "judge-service-name"
}
```

The complete JSON object returned by the server is returned unchanged. It
normally contains one judgment and trace per supplied rubric.

## Build and publish

```bash
python -m build
python -m twine check dist/*
python -m twine upload dist/*
```

Publishing requires a PyPI account and token. Verify that the distribution
name `jtc-client` is available before the first upload.
