Metadata-Version: 2.5
Name: benchloop_client
Version: 0.2.23
Summary: Benchloop CLI — command line tool for benchmark analysis data (metrics, trajectory, reports, artifacts).
Requires-Python: >=3.10
Requires-Dist: httpx>=0.25.2
Requires-Dist: tomli>=1.1.0; python_version < '3.13'
Requires-Dist: typer>=0.9.4
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == 'dev'
Description-Content-Type: text/markdown

# benchloop-client (`bl`, `bmcli`)

Command line tool for fetching and downloading benchmark analysis data
(run metrics, trajectory, metadata, pre-analysis reports, task/job lists,
and artifacts) by `group_id` / `job_id` / download `task_id`, and for
submitting Behavior Reward Model (`brm`) analysis tasks.

> PyPI package name is `benchloop_client`; the installed commands are `bl` and `bmcli`.

## Install package

```
pip install benchloop_client

bl --api-key $BENCH_LOOP_API_KEY overview get -j $JOB_ID
bl --api-key $BENCH_LOOP_API_KEY metrics get -j $JOB_ID

# Option 2: config ~/.benchloop/config.toml by cli, and run bl without --api-key option:
bl config set --api-key $BENCHLOOP_API_KEY --host https://benchmark-analysis.alibaba-inc.com

bl overview get -j $JOB_ID
bl metrics get -j $JOB_ID

```

The CLI checks PyPI for a newer release in the background at most once every
24 hours. Update notices include the current/latest versions and the upgrade
command. Notices are written only to stderr, so JSON output remains pipeable.

## Install and run from source

```bash
# with uv (recommended)
uv sync
uv run bl --help

# or editable install with pip
python -m venv .venv && source .venv/bin/activate
pip install -e .
bl --help
```

## Auth & config

Authentication uses a personal api key sent as the `X-API-Key` header.

Precedence: CLI flag > env var > `~/.benchloop/config.toml` > default.

- Env: `BENCHLOOP_API_KEY` (or `BENCHMARK_API_KEY`), `BENCHLOOP_HOST`, `BENCHLOOP_OUTPUT_DIR`
- Default host (prod): `https://benchmark-analysis.alibaba-inc.com`
- Pre host: `https://pre-benchmark-analysis.alibaba-inc.com`

```bash
bl config set --api-key $BENCHLOOP_API_KEY --host https://benchmark-analysis.alibaba-inc.com
```

## Usage

Two-segment `bl <resource> <action>` form. `get` prints a JSON array to
stdout (pipeable to `jq`); `download` writes the standard file layout.
Global options such as `--api-key`, `--host`, `--timeout`, `--retry`,
`--concurrency`, `--quiet`, and `--verbose` must be placed before the resource
name. Use `--concurrency 1` for a per-file byte progress bar, or a higher value
for concurrent requests/downloads.

```bash
bl metrics get -j J123 | jq '.[0]'
bl metrics download -g G1 --scope all -o ./out
bl --concurrency 10 trajectory download -g G1,G2 -o ./out
bl --concurrency 10 trajectory download --task-id TASK_ID -o ./out
bl metadata download --task-id TASK_ID -o ./out
bl report download -j J123 -o ./out
bl report get -g G123
bl report download -g G123 -o ./out
bl jobs get -g G1 --all          # get supports --all (auto-paginate)
bl jobs download -g G1           # download is always full
bl artifacts download -g G1 -o ./out
bl artifacts download --task-id TASK_ID -o ./out
bl pull -g G1 --with metrics,trajectory,jobs,artifacts -o ./out
bl brm run -g G1
bl brm run -g G1 --steps 10,20 --sample-size 512 --scoring-limit 32
bl brm get --task-id UID
bl brm get --task-id UID --scope overall
bl brm get --task-id UID --scope detail
bl brm get --task-id UID --scope status
bl brm status --task-id UID
bl brm list -g G1
```

Group downloads always auto-paginate and include every Job in the Group. In
particular, `trajectory`, detailed `metrics`, `report`, and `artifacts` enumerate
the complete Job list instead of relying on the backend Group endpoints that
return at most 1000 Jobs. Existing aggregate files keep their original paths:

```text
out/G1/trajectory.json
out/G1/detail_metrics.json
out/G1_report.md
out/G1/J123_report.md
out/G1/J123/<artifact-name>
out/TASK_ID_trajectories.zip
out/TASK_ID_metadata.zip
out/TASK_ID/J123.tgz
```

`report download -g` writes both the Group summary report and each Job report.
`pull -g` uses the same complete download behavior for every selected resource.
Use `--skip-existing` to resume file-based downloads after an interruption.

IDs accept repeated flags, comma-separated values, and `-` to read from stdin.
`-g` and `-j` are mutually exclusive in a single command. For trajectory and
artifact downloads, `--task-id` is a third mutually exclusive input mode.

### Download trajectories by task ID

The benchmark analysis Jobs page can turn the current filters into a trajectory
download task. Copy the command shown by the page and run it locally:

```bash
bl trajectory download --task-id TASK_ID -o ./out
```

The CLI fetches the task's complete job ID list, downloads job trajectories with
fixed concurrency, and prints count/ETA progress to stderr. It creates
`./out/TASK_ID_trajectories.zip` with this layout:

```text
trajectories/<jobId>_trajectory.json
metadata/<jobId>.json
manifest.json
```

Metadata is fetched from `/openapi/jobs/{job_id}/meta` and archived alongside
the trajectory. If trajectory download succeeds but metadata fails, the
trajectory is still archived, the metadata error is recorded in
`manifest.metadata_failures`, and the command exits with code `1`. If only some
jobs fail, the successful files are still archived. If every job fails, no empty
ZIP is created.

### Download metadata by task ID

Use `metadata download` when you only need per-job metadata for a download task:

```bash
bl metadata download --task-id TASK_ID -o ./out
```

The CLI creates `./out/TASK_ID_metadata.zip` with this layout:

```text
metadata/<jobId>.json
manifest.json
```

Partial metadata failures are recorded in `manifest.failures`; successful
metadata files are still archived and the command exits with code `1`.

### Download artifacts

Artifacts can be downloaded by Job, Group, or download task:

```bash
bl artifacts download -j J123 -o ./out
bl artifacts download -g G1 -o ./out
bl artifacts download --task-id TASK_ID -o ./out
bl artifacts download -g G1 --include "*.tgz" --exclude "debug-*" -o ./out
bl artifacts download -g G1 --list-only
```

For Job and Group mode, the CLI still lists artifacts first via
`/openapi/jobs/{job_id}/artifacts`, but binary download is routed through the
backend proxy endpoint `/openapi/jobs/{job_id}/artifacts/download` instead of
directly using OSS presigned URLs. In task mode, every job from the task
snapshot is downloaded as `./out/TASK_ID/<jobId>.tgz`.

When an artifact is missing, the backend may return either HTTP 404 or an HTML
error page. The CLI treats both as a missing artifact and writes a valid empty
`.tgz` placeholder instead of saving the error page, so downstream batch
extraction can continue.

Artifact downloads show byte-level progress. On a TTY the progress is refreshed
in place; in CI or redirected stderr it falls back to periodic `[progress]`
lines. `--quiet` suppresses progress output.

### Submit a Behavior Reward Model (`brm`) task

`brm run` creates a Behavior analysis task for exactly one Group. The CLI first
calls `GET /openapi/behavior/steps?group_id=...` to detect whether the Group
has step sampling, then posts `POST /openapi/behavior/tasks`.

```bash
# No-step groups (all_jobs sampling): omit --steps
bl brm run -g G1

# Step groups: --steps is required (comma-separated or repeatable)
bl brm run -g G1 --steps 10,20
bl brm run -g G1 --steps 10 --steps 20 --sample-size 256 --scoring-limit 64

# Pipe the task id
bl brm run -g G1 --steps 10,20 | jq '.[0].uid'
```

Defaults: `--sample-size 512` (step mode only), `--scoring-limit 32` (always
sent). In step mode, omitting `--steps` is a usage error and the available
steps are listed on stderr. Passing `--steps` on a no-step Group is also a
usage error.

On success, stderr logs a short confirmation and stdout prints a JSON array
(one object) that includes at least `uid`, plus `group_id` and
`samplingMode` (`step` or `all_jobs`). Use `.[0].uid` (not necessarily
`task_id`) for follow-up `brm status` / `brm get` calls.

### Query a Behavior Reward Model (`brm`) task

`brm get` queries task status and/or analysis by Behavior task uid (from
`brm run`). Multiple `--task-id` values are supported (repeatable or
comma-separated); stdout is always a JSON array, one element per id, matching
`metrics get -j ...`.

```bash
# Default --scope all (status + overall + detail); samplingMode only here
bl brm get --task-id UID
bl brm get --task-id UID --scope all | jq '.[0] | {status, overall, detail, samplingMode}'

# overall: analysis.summary + categoryScores
bl brm get --task-id UID --scope overall | jq '.[0] | {summary, categoryScores}'

# detail: analysis.behaviorPatterns
bl brm get --task-id UID --scope detail | jq '.[0].behaviorPatterns'

# status only; `brm status` is equivalent
bl brm get --task-id UID --scope status
bl brm status --task-id UID
bl brm status --task-id UID1,UID2
```

`--scope` values:

| scope | source | payload |
| --- | --- | --- |
| `overall` | `GET .../analysis` | `summary`, `categoryScores` |
| `detail` | `GET .../analysis` | `behaviorPatterns` |
| `status` | `GET .../tasks/{uid}` | task status / progress |
| `all` (default) | both endpoints | nested `status`, `overall`, `detail`, plus top-level `samplingMode` |

`brm status` always uses the status endpoint (same as `brm get --scope status`).

### List Behavior Reward Model (`brm`) tasks by Group

`brm list` calls `GET /openapi/behavior/tasks?group_id=...` and prints a JSON
array (one element per `-g`), matching `jobs get -g ...`:

```bash
bl brm list -g G1
bl brm list -g G1,G2
bl brm list -g G1 | jq '.[0].tasks[].uid'
```

Each element looks like `{ "group_id", "total", "tasks": [ ... ] }`. Task
objects keep the backend fields and also set `task_id` from `uid` when present.

## Exit codes

| code | meaning |
| ---- | ------- |
| 0 | all success |
| 1 | partial failure (some ids failed) |
| 2 | usage / parameter error |
| 3 | auth failure (HTTP 401) |
| 4 | network / timeout |

stdout carries data only; logs and progress go to stderr. `--quiet` suppresses
non-error logs, `--verbose` adds debug logs.


## Release new package

```
uv build
twine check dist/benchloop_client-*.whl
twine upload dist/benchloop_client-*.whl --verbose
```
