Metadata-Version: 2.4
Name: nufi-python
Version: 1.1.0
Summary: Python SDK and CLI for the NuFi platform.
Author: Dudaji Inc.
License-Expression: MIT
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Classifier: Typing :: Typed
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: httpx>=0.25.0
Requires-Dist: PyYAML>=6.0
Requires-Dist: pydantic>=2.0.0
Requires-Dist: tuspy==1.1.0
Requires-Dist: typer>=0.16.0
Requires-Dist: rich>=13.0.0
Provides-Extra: evaluator
Requires-Dist: pyarrow>=24.0.0; extra == "evaluator"
Dynamic: license-file

# NuFi Python SDK/CLI

The distribution is `nufi-python`; it installs the `nufi` Python package and `nufi`
command.

```bash
uv build --clear
python -m pip install ./dist/nufi_python-1.1.0-py3-none-any.whl
nufi --version
```

```python
from nufi import NuFi
```

For verified-wheel installation, model registration, Dataset selection, evaluation,
Leaderboard ranking, compatibility, and troubleshooting, follow the
[SDK and CLI workflow guide](../docs/sdk-cli-guide/README.md).

## Installed quickstart

Create a profile, log in, and select the project once:

```bash
nufi profile create dev --base-domain nufi.local
nufi profile use dev
nufi login
nufi project use team-a
```

The CLI workflow is composable; every long-running identity can be resumed with its
dedicated `wait` command:

```bash
nufi model version create qwen2 v1 --volume models --path /qwen2
nufi compile create qwen2 --version v1 --wait
nufi compile wait COMPILE_NAME_OR_ID
nufi dataset list
nufi evaluation run create METHOD_NAME --model qwen2 \
  --model-version v1 --artifact original --wait
nufi evaluation run wait RUN_ID
nufi leaderboard create quality --method METHOD_NAME \
  --rank-metric exact_match --rank-direction desc
nufi leaderboard submission create quality --model qwen2 \
  --model-version v1 --artifact original --wait
nufi leaderboard submission wait quality SUBMISSION_ID
nufi leaderboard entries quality
```

The same flow is available without local orchestration in Python:

```python
from nufi import NuFi

with NuFi.from_profile() as client:
    compile_run = client.compiles.create(model_name="qwen2", version="v1")
    compiled = client.compiles.wait(compile_run.id)

    evaluation_run = client.evaluations.create(
        evaluation_method_id="METHOD_ID",
        model_artifact_id=9,
        cpu="4",
        memory="8Gi",
        accelerator_count=1,
    )
    evaluated = client.evaluations.wait(evaluation_run.id)

    leaderboard = client.leaderboards.create(
        "quality",
        evaluation_method_id="METHOD_ID",
        ranking_policy={
            "primary": {
                "metric": {"source": "aggregateMetrics", "name": "exact_match"},
                "direction": "desc",
            }
        },
    )
    submission = client.leaderboards.submit(
        leaderboard.id,
        model_artifact_id=9,
    )
    completed = client.leaderboards.wait(leaderboard.id, submission.id)
    entries = client.leaderboards.entries(leaderboard.id)
```

## Local Run

Install dependencies:

```bash
uv sync
```

Run the CLI locally:

```bash
uv run nufi --help
uv run nufi profile
```

## Evaluations

Evaluation Method metadata is unconditional and v2-only. Method config is owned and
validated by the API; the SDK forwards the mapping and any caller-supplied Method type
without a client-side allowlist.

`evaluation method templates` currently exposes lm-eval templates, metrics, and
filters. It is an optional discovery command and is not needed when you already have a
Method config.

```python
from nufi import NuFi

client = NuFi.from_profile()
project = "team-a"
config = {
    "schemaVersion": "nufi.evaluation.method.v2",
    "runnerType": "lm_eval",
    "template": "short_answer",
    "datasetMapping": {"input": "question", "reference": "answer"},
    "nativeConfig": {
        "output_type": "generate_until",
        "doc_to_text": "{{input}}",
        "doc_to_target": "{{reference}}",
        "metric_list": [{"metric": "exact_match"}],
    },
}

templates = client.evaluations.methods.templates(project=project)
preview = client.evaluations.methods.preview(
    "DATASET_ID", "VERSION_ID", project=project, config=config
)
method = client.evaluations.methods.create(
    "DATASET_ID", "VERSION_ID", "gsm8k-exact-match", project=project, config=config
)
run = client.evaluations.create(
    project=project,
    evaluation_method_id=method.id,
    model_artifact_id=9,
    cpu="1",
    memory="2Gi",
    accelerator_count=1,
)
result = client.evaluations.get(run.id, project=project)
client.evaluations.cancel(run.id, project=project)
client.close()
```

The CLI accepts Dataset, Dataset Version, and Evaluation Method names or UUIDs. Run
arguments remain UUIDs because Runs have no name. Select a Model Artifact with
`--model`, `--model-version`, and `--artifact`; `--model-artifact-id` remains available
for automation. Config, default-runtime, and runner-config files may be JSON or YAML
and must have a mapping at the document root.

```yaml
# method.yaml
schemaVersion: nufi.evaluation.method.v2
runnerType: lm_eval
template: short_answer
datasetMapping:
  input: question
  reference: answer
nativeConfig:
  output_type: generate_until
  doc_to_text: "{{input}}"
  doc_to_target: "{{reference}}"
  metric_list:
    - metric: exact_match
```

```bash
uv run nufi evaluation method templates --project team-a
uv run nufi evaluation method validate gsm8k v1 --config method.yaml --project team-a
uv run nufi evaluation method create gsm8k v1 gsm8k-exact-match \
  --config method.yaml --project team-a
uv run nufi evaluation method list --dataset gsm8k --version v1 --project team-a
uv run nufi evaluation method archive gsm8k-exact-match --yes --project team-a
uv run nufi evaluation method restore gsm8k-exact-match --project team-a

uv run nufi evaluation run create gsm8k-exact-match \
  --model qwen2 --model-version v1 --artifact original \
  --project team-a
uv run nufi evaluation run list --dataset gsm8k --version v1 --project team-a
uv run nufi evaluation run get RUN_ID --project team-a
uv run nufi evaluation run cancel RUN_ID --yes --project team-a
```

Run creation defaults to `--cpu 4`, `--memory 8Gi`, and `--accelerator-count 1`;
pass any option to override its default.

Run detail table output shows aggregate and diagnostic summaries only. Use explicit
`--output json` to include the authorized bounded diagnostic input, output, reference,
score, and error fields returned by the API. Additive response fields are preserved;
legacy Method/Run responses missing required v2 fields are rejected.

Run tests:

```bash
uv run pytest
```

Build the package:

```bash
uv build --clear
uv run python scripts/verify_distribution.py
```

The verifier reports the CI source revision from `CI_COMMIT_SHA` and the SHA-256
digest of each artifact for the operator-controlled release handoff.
