Metadata-Version: 2.4
Name: sixtytwo-cli
Version: 0.3.18
Summary: Sixtytwo CLI: `sixtytwo rent` reserves reliability-backed GPUs; `sixtytwo` qualifies, monitors, and NCCL-benchmarks your own GPU clusters, with Slurm/SkyPilot integration.
Author: Sixtytwo, Inc.
License-Expression: LicenseRef-Sixtytwo-Commercial
Keywords: gpu,nccl,skypilot,slurm,cluster,benchmarking
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Operating System :: POSIX :: Linux
Classifier: Operating System :: MacOS
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: System :: Distributed Computing
Classifier: Topic :: System :: Monitoring
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: rich<15,>=14.0
Requires-Dist: fastapi<1,>=0.115
Provides-Extra: server
Requires-Dist: fastapi<1,>=0.115; extra == "server"
Requires-Dist: uvicorn<1,>=0.34; extra == "server"
Requires-Dist: geoip2<6,>=4.8; extra == "server"
Requires-Dist: cryptography<46,>=43; extra == "server"
Provides-Extra: sentry
Requires-Dist: sentry-sdk[fastapi]<3,>=2.18; extra == "sentry"
Provides-Extra: serverless
Requires-Dist: runpod<2,>=1.7; extra == "serverless"
Requires-Dist: modal<2,>=1.0; extra == "serverless"
Provides-Extra: gpu
Requires-Dist: nvidia-ml-py3>=7.352.0; extra == "gpu"
Requires-Dist: torch>=2.11; extra == "gpu"
Provides-Extra: skypilot
Requires-Dist: skypilot<0.12,>=0.9.0; extra == "skypilot"
Provides-Extra: skypilot-aws
Requires-Dist: skypilot[aws]<0.12,>=0.9.0; extra == "skypilot-aws"
Provides-Extra: skypilot-gcp
Requires-Dist: skypilot[gcp]<0.12,>=0.9.0; extra == "skypilot-gcp"
Provides-Extra: skypilot-lambda
Requires-Dist: skypilot[lambda]<0.12,>=0.9.0; extra == "skypilot-lambda"
Provides-Extra: skypilot-runpod
Requires-Dist: skypilot[runpod]<0.12,>=0.9.0; extra == "skypilot-runpod"
Dynamic: license-file

# sixtytwo

`sixtytwo-cli` is the command-line client for [sixtytwo](https://sixtytwo.ai),
built for the terminal and for coding agents. It covers two jobs:

- **Rent reliability-backed GPUs.** `sixtytwo rent` reserves GPUs that sixtytwo
  has already qualified, watches them while you run, and refunds the unused time
  if hardware goes bad.
- **Qualify and monitor your own clusters.** `sixtytwo` runs hardware checks,
  NCCL benchmarks, straggler detection, and live fault monitoring on GPUs you
  already have, on both NVIDIA and AMD hardware, and keeps a per-node trust
  score from every test, fault, and recovery.

`sixtytwo rent` stays light (no torch), so it runs anywhere. The qualification
and monitoring commands pull in the GPU stack only when you ask for it.

## Quick start

```bash
pip install sixtytwo-cli

# Rent a qualified GPU
sixtytwo rent login                    # paste a token from sixtytwo.ai/account
sixtytwo rent catalog
sixtytwo rent H100_SXM -n 1 -H 2       # 1x H100 for 2 hours

# Or check the GPUs you already have
sixtytwo init
sixtytwo doctor
```

## Install

Requirements: Python 3.11 or newer; macOS (Apple silicon), Linux x86_64/aarch64, or Windows.

Optional extras:

```bash
pip install 'sixtytwo-cli[gpu]'        # nvidia-ml-py + torch for local GPU checks
pip install 'sixtytwo-cli[skypilot]'   # SkyPilot cloud adapter (see below)
pip install 'sixtytwo-cli[server]'     # local dashboard + collector daemon
```

## Rent GPUs

```bash
sixtytwo rent catalog                  # list SKUs and prices
sixtytwo rent H100_SXM -n 8 -H 4       # reserve 8x H100 for 4 hours, wait for READY
sixtytwo rent ls                       # list my reservations
sixtytwo rent status <reservation-id>
sixtytwo rent ssh <reservation-id>     # open a shell on a READY reservation
```

Every node is validated before you get it and monitored while you use it. Add
`--region us|eu|asia` to pin a geography; billing follows actual usage. The
storefront at [sixtytwo.ai](https://sixtytwo.ai) covers the same flow in the
browser.

## Qualify and monitor your own GPUs

These commands read a `sixtytwo.yaml`, so run `init` once first. It detects this
machine (and your scheduler, if any) and writes the config into the current
directory; everything below loads it.

```bash
sixtytwo init                          # writes ./sixtytwo.yaml (run this first)
sixtytwo doctor                        # validate this box: drivers, GPUs, topology
sixtytwo doctor --all-nodes --json     # validate every node in the cluster
sixtytwo test --quick --all            # fast per-node qualification
sixtytwo test --full gpu-01,gpu-02     # full suite, including NCCL
sixtytwo launch --pre-check python train.py   # qualify, then run with live monitoring
sixtytwo nodes                         # trust scores and fault history
```

`doctor --all-nodes` reports each node's GPU inventory, not just the local
host's. Results land in a local trust registry, so a node's history (tests,
faults, recoveries) follows it over time.

Straggler detection runs at both stages. `test --full` flags a GPU that is
slower than its node peers (an identical kernel timed per GPU) or below its
model's reference baseline, before you commit a run to it. During a run,
`launch` watches per-rank step times and flags a rank that falls behind the
others, so a GPU that degrades into a straggler mid-job is caught too.

### Slurm clusters

Run `init` on the login node (or inside an allocation). It detects Slurm, fills
the node list from `sinfo`/`scontrol`, and sets `remote.mode: slurm`, so `test`
and `doctor --all-nodes` dispatch each node's checks through `srun`. Confirm the
GPU partition and per-node GPU count in `sixtytwo.yaml` (the per-node count
can't be detected from a GPU-less login node):

```yaml
cluster:
  slurm:
    partition: gpu        # your GPU partition
    gpus_per_node: 8
```

On a large or multi-partition cluster `init` lists every node it can see, so
trim `cluster.nodes` to the GPU nodes you actually want to qualify before
running `doctor --all-nodes` or `test --all`.

### SSH clusters (no scheduler)

Any set of boxes you can ssh into can be treated as a cluster: set
`remote.mode: ssh` and list the nodes, where entries accept `user@host:port`
so cloud pods with one public port per node (RunPod, Vast) work directly:

```yaml
cluster:
  nodes:
    - root@203.0.113.7:45663
    - root@203.0.113.9:11542
remote:
  mode: ssh
  ssh_key_path: ~/.ssh/id_ed25519
```

`doctor --all-nodes` and `test --all --parallel 8` then drive every node over
ssh.

## SkyPilot

There are two ways to combine sixtytwo with
[SkyPilot](https://skypilot.readthedocs.io):

```bash
# 1. Qualify a cluster you launched on any cloud SkyPilot supports
sixtytwo skypilot qualify my-cluster --mode quick

# 2. Provision sixtytwo's own GPUs through SkyPilot with `cloud: sixtytwo`
pip install 'sixtytwo-cli[skypilot]'
sixtytwo skypilot install              # register sixtytwo as a SkyPilot cloud
sky show-gpus --cloud sixtytwo
```

`sixtytwo skypilot install` is opt-in: it wires the adapter into the active
virtualenv so `sky launch` can reserve reliability-backed GPUs with
`cloud: sixtytwo`. Run `sixtytwo skypilot --help` for the full set of
subcommands.

## Topology and fabric health

The full test suite maps how a node is actually wired and checks the fabric
it sits on:

- **GPU topology**: the NVLink and XGMI link matrix on every node, with
  PCIe-only or mixed-link wiring flagged before it slows a collective.
- **InfiniBand fabric**: the fabric is mapped to show which leaf switch each
  node hangs off (one sweep per cluster, reused across nodes), and port
  counters are classified so congestion reads as congestion rather than a
  cable fault.
- **GPU to NIC affinity**: an NCCL probe across all GPUs verifies
  rail-optimized placement.
- **Diagnosis with opt-in repairs**: failures map to a remediation plan
  (restart fabric manager, reset wedged GPUs, fix clock sync). Everything is
  off by default and dry-run first; fixes only execute when
  `recovery.in_place_fixes.enabled` is set, stop at the first verified
  repair, and never escalate to a disruptive action without a fresh
  diagnosis.

## Hosted dashboards

Stream telemetry to [sixtytwo.ai](https://sixtytwo.ai) and watch the fleet
from a browser:

```bash
sixtytwo connect --source ssh --name my-cluster   # register (token from the account page)
sixtytwo monitor                                  # stream per-GPU telemetry
sixtytwo install-agent                            # or install as a systemd service
```

The cluster page shows live per-node tiles, a node x GPU utilization heat
map, and a 24h history; on Slurm clusters the agent also attributes GPU
hours to users and accounts, feeding a chargeback report that exports to
CSV.

## Metrics for Grafana

`sixtytwo metrics serve` exposes a Prometheus `/metrics` endpoint backed by the
local trust registry: trust scores, fault counters, recovery downtime, and
per-check status. `sixtytwo metrics export-grafana` prints a curated dashboard
you can import as is.

```bash
sixtytwo metrics serve --host 0.0.0.0 --port 9620
sixtytwo metrics export-grafana --output sixtytwo-overview.json
```

## License

Commercial. The bundled `LICENSE` governs use. Learn more at
[sixtytwo.ai](https://sixtytwo.ai).
