Metadata-Version: 2.4
Name: nufi-python
Version: 1.1.2.post1
Summary: Python SDK and CLI for the NuFi platform.
Author: Dudaji Inc.
License-Expression: MIT
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Classifier: Typing :: Typed
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: httpx>=0.25.0
Requires-Dist: PyYAML>=6.0
Requires-Dist: pydantic>=2.0.0
Requires-Dist: tuspy==1.1.0
Requires-Dist: typer>=0.16.0
Requires-Dist: rich>=13.0.0
Provides-Extra: evaluator
Requires-Dist: pyarrow>=24.0.0; extra == "evaluator"
Dynamic: license-file

# NuFi Python SDK/CLI

`nufi-python` is the Python SDK and CLI for a NuFi installation. It requires Python
3.11 or newer and installs the `nufi` Python package and `nufi` command.

## Installation

```bash
python -m pip install nufi-python
nufi --version
nufi --help
```

Install the optional evaluator dependencies when writing a custom evaluator that reads
NuFi Parquet datasets:

```bash
python -m pip install "nufi-python[evaluator]"
```

## Configure access

Ask your NuFi Administrator for the installation's base domain and a Project you can
access. Create a CLI Profile, log in, and select the project once:

```bash
nufi profile create production --base-domain nufi.example
nufi profile use production
nufi login
nufi project use team-a
```

`login` uses the device flow. Add `--no-browser` on a headless host and open the
printed verification URL elsewhere.

## CLI quickstart

The CLI workflow is composable; every long-running identity can be resumed with its
dedicated `wait` command:

```bash
nufi model version create qwen2 v1 --volume models --path /qwen2
nufi compile create qwen2 --version v1 --wait
nufi compile wait COMPILE_NAME_OR_ID
nufi dataset list
nufi evaluation run create METHOD_NAME --model qwen2 \
  --model-version v1 --artifact original --wait
nufi evaluation run wait RUN_ID
nufi leaderboard create quality --method METHOD_NAME \
  --rank-metric exact_match --rank-direction desc
nufi leaderboard submission create quality --model qwen2 \
  --model-version v1 --artifact original --wait
nufi leaderboard submission wait quality SUBMISSION_ID
nufi leaderboard entries quality
```

## Python SDK quickstart

The same flow is available in Python:

```python
from nufi import NuFi

with NuFi.from_profile() as client:
    compile_run = client.compiles.create(model_name="qwen2", version="v1")
    compiled = client.compiles.wait(compile_run.id)

    evaluation_run = client.evaluations.create(
        evaluation_method_id="METHOD_ID",
        model_artifact_id=9,
        cpu="4",
        memory="8Gi",
        accelerator_count=1,
    )
    evaluated = client.evaluations.wait(evaluation_run.id)

    leaderboard = client.leaderboards.create(
        "quality",
        evaluation_method_id="METHOD_ID",
        ranking_policy={
            "primary": {
                "metric": {"source": "aggregateMetrics", "name": "exact_match"},
                "direction": "desc",
            }
        },
    )
    submission = client.leaderboards.submit(
        leaderboard.id,
        model_artifact_id=9,
    )
    completed = client.leaderboards.wait(leaderboard.id, submission.id)
    entries = client.leaderboards.entries(leaderboard.id)
```

## Model Evaluation

Dataset import and Dataset Profiling complete independently. Wait for the Dataset
Profile before previewing or creating an Evaluation Method:

```bash
nufi dataset version create mmlu mmlu-aa-test --source huggingface \
  --repository cais/mmlu --subset abstract_algebra --split test --wait
nufi dataset version profile mmlu mmlu-aa-test --wait --output json
```

The SDK uses the import result's `dataset_version_id` for the same workflow. A failed
Dataset Profile is returned as a terminal object so the caller can report its server
error without sending an Evaluation Method request:

```python
from nufi import NuFi

client = NuFi.from_profile()
dataset_id = "DATASET_ID"
imported = client.datasets.versions.create_from_huggingface(
    dataset_id,
    "mmlu-aa-test",
    repository="cais/mmlu",
    subset="abstract_algebra",
    split="test",
    wait=True,
)
if not imported.dataset_version_id:
    raise RuntimeError("Dataset import completed without a Dataset Version")

version_id = imported.dataset_version_id
profile = client.datasets.versions.wait_for_profile(dataset_id, version_id)
if profile.status != "ready":
    raise RuntimeError(profile.error or "Dataset Profiling failed")

config = {
    "schemaVersion": "nufi.evaluation.method.v2",
    "runnerType": "lm_eval",
    "template": "multiple_choice",
    "datasetMapping": {
        "input": "question",
        "choices": "choices",
        "reference": "answer",
    },
    "nativeConfig": {
        "output_type": "multiple_choice",
        "doc_to_text": "{{input}}",
        "metric_list": [{"metric": "acc"}],
    },
}
preview = client.evaluations.methods.preview(
    dataset_id, version_id, config=config
)
method = client.evaluations.methods.create(
    dataset_id, version_id, "mmlu-aa", config=config
)
client.close()
```

Evaluation Method metadata is unconditional and v2-only. Evaluation Method config is
owned and validated by the API; the SDK forwards the mapping and any caller-supplied
Evaluation Method type without a client-side allowlist.

The `nufi.evaluator.run` helper exits non-zero when an Evaluation Run or result
submission fails. NuFi determines Evaluation Run success from Kubernetes Job success
and durable API acceptance; evaluator output is diagnostic only.

`evaluation method templates` currently exposes lm-eval templates, metrics, and
filters. It is an optional discovery command and is not needed when you already have an
Evaluation Method configuration.

```python
from nufi import NuFi

client = NuFi.from_profile()
project = "team-a"
config = {
    "schemaVersion": "nufi.evaluation.method.v2",
    "runnerType": "lm_eval",
    "template": "short_answer",
    "datasetMapping": {"input": "question", "reference": "answer"},
    "nativeConfig": {
        "output_type": "generate_until",
        "doc_to_text": "{{input}}",
        "doc_to_target": "{{reference}}",
        "metric_list": [{"metric": "exact_match"}],
    },
}

templates = client.evaluations.methods.templates(project=project)
preview = client.evaluations.methods.preview(
    "DATASET_ID", "VERSION_ID", project=project, config=config
)
method = client.evaluations.methods.create(
    "DATASET_ID", "VERSION_ID", "gsm8k-exact-match", project=project, config=config
)
run = client.evaluations.create(
    project=project,
    evaluation_method_id=method.id,
    model_artifact_id=9,
    cpu="1",
    memory="2Gi",
    accelerator_count=1,
)
result = client.evaluations.get(run.id, project=project)
client.evaluations.cancel(run.id, project=project)
client.close()
```

The CLI accepts Dataset, Dataset Version, and Evaluation Method names or UUIDs.
Evaluation Run arguments remain UUIDs because Evaluation Runs have no name. Select a
Model Artifact with
`--model`, `--model-version`, and `--artifact`; `--model-artifact-id` remains available
for automation. Config, default-runtime, and runner-config files may be JSON or YAML
and must have a mapping at the document root.

```yaml
# method.yaml
schemaVersion: nufi.evaluation.method.v2
runnerType: lm_eval
template: short_answer
datasetMapping:
  input: question
  reference: answer
nativeConfig:
  output_type: generate_until
  doc_to_text: "{{input}}"
  doc_to_target: "{{reference}}"
  metric_list:
    - metric: exact_match
```

```bash
nufi evaluation method templates --project team-a
nufi evaluation method validate gsm8k v1 --config method.yaml --project team-a
nufi evaluation method create gsm8k v1 gsm8k-exact-match \
  --config method.yaml --project team-a
nufi evaluation method list --dataset gsm8k --version v1 --project team-a
nufi evaluation method archive gsm8k-exact-match --yes --project team-a
nufi evaluation method restore gsm8k-exact-match --project team-a

nufi evaluation run create gsm8k-exact-match \
  --model qwen2 --model-version v1 --artifact original \
  --project team-a
nufi evaluation run list --dataset gsm8k --version v1 --project team-a
nufi evaluation run get RUN_ID --project team-a
nufi evaluation run cancel RUN_ID --yes --project team-a
```

Evaluation Run creation defaults to `--cpu 4`, `--memory 8Gi`, and
`--accelerator-count 1`; pass any option to override its default.

Evaluation Run detail table output shows aggregate and diagnostic summaries only. Use
explicit `--output json` to include the authorized bounded diagnostic input, output,
reference, score, and error fields returned by the API. Additive response fields are
preserved; legacy Evaluation Method/Evaluation Run responses missing required v2
fields are rejected.
