Metadata-Version: 2.4
Name: tuning-client
Version: 0.1.1
Summary: Python SDK and neural-network tuning demo for the hyperopt.dev AutoML service
License-Expression: MIT AND Apache-2.0
Project-URL: Homepage, https://hyperopt.dev
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: tuning_demo/benchmarking/LICENSE-FLAML.txt
License-File: tuning_demo/benchmarking/LICENSE-Ray.txt
License-File: tuning_demo/benchmarking/THIRD_PARTY.md
Requires-Dist: httpx<1,>=0.27
Requires-Dist: websockets<18,>=15
Provides-Extra: demo
Requires-Dist: numpy<3,>=1.26; extra == "demo"
Requires-Dist: scikit-learn<2,>=1.5; extra == "demo"
Requires-Dist: torch<3,>=2.5; extra == "demo"
Provides-Extra: custom
Requires-Dist: pandas<3,>=2.2; extra == "custom"
Requires-Dist: pyarrow<24,>=18; extra == "custom"
Dynamic: license-file

# Tuning Client Python SDK and demo

Tuning Client is the Python SDK and demo for the
[hyperopt.dev AutoML service](https://hyperopt.dev).

Use the Python SDK to request tuning suggestions while keeping model training,
datasets, and model weights on your computer. The optional demo trains small
PyTorch models and compares tuning strategies on classification, regression,
and teaching datasets.

**Code provenance:** The SDK and demo code were generated with ChatGPT and human
reviewed. Review does not guarantee correctness; evaluate the code and benchmark
results for your use case. The demos are examples, not production ML benchmarks.

## Installation

Requires Python 3.11 or newer. Use a virtual environment. Install the SDK alone:

```sh
python -m pip install tuning-client
```

The demo runs on CPU and does not require NVIDIA/CUDA libraries. On Linux,
install the CPU build of PyTorch first to avoid the large GPU dependencies
selected by the default PyPI distribution:

```sh
python -m pip install 'torch>=2.5,<3' --index-url https://download.pytorch.org/whl/cpu
python -m pip install 'tuning-client[demo]'
```

The installed CPU build satisfies the demo's PyTorch requirement. On macOS,
install `tuning-client[demo]` directly. See the
[PyTorch installation guide](https://pytorch.org/get-started/locally/) for other
platforms. If a failed download filled the disk, `python -m pip cache purge`
removes pip's cached downloads; `--no-cache-dir` avoids caching the next install.

## Run the demo

Create an account at [hyperopt.dev](https://hyperopt.dev/signup) and verify your email:

```sh
python -m tuning_demo --api-url https://hyperopt.dev --email you@example.com --dataset breast-cancer
python -m tuning_demo.stats --api-url https://hyperopt.dev --email you@example.com \
  --datasets breast-cancer --runs 2
# The installed command is equivalent:
tuning-client-demo --api-url https://hyperopt.dev --email you@example.com --dataset breast-cancer
python -m tuning_demo --list-datasets
```

The demo prompts for your password. You can instead set `COMPUTE_ACCESS_TOKEN`,
or `COMPUTE_EMAIL` and `COMPUTE_PASSWORD`. Set `COMPUTE_API_URL` to avoid passing
`--api-url` each time. Connections use HTTPS and verified TLS.

Service effort defaults to **medium**. Use `--effort easy` for lower effort;
the observation budget stays the same. `--interactive` offers another dataset
after each run. Sessions created by the demo are released automatically on
completion or failure. `--session-id ID` uses an existing unused session that
you remain responsible for releasing; `--keep-session` retains a new session.

Results are saved in `./results/`. Downloaded datasets are cached in
`~/.cache/tuning-client/`. Override these with `--output-dir` and `--data-dir`.
Optional usage events include a run ID, session ID, workload category, and
progress event; use `--no-telemetry` to disable them. Training data, file paths,
and model weights are not included in those events.

## SDK usage

```python
import os
from tuning_client import TuningClient

with TuningClient(os.environ['COMPUTE_API_URL'],
                   access_token=os.environ['COMPUTE_ACCESS_TOKEN']) as api:
    session_id = api.create_session()
    try:
        stream = api.tuning_session(session_id)
        try:
            configurations = [{'learning_rate': rate} for rate in
                              [0.001, 0.002, 0.003, 0.004, 0.005,
                               0.006, 0.007, 0.008, 0.009, 0.010]]
            stream.initialize(configurations, max_steps=10, seed=42)
            suggestion = stream.suggest()
            # Train the suggested configuration locally, then report validation loss:
            # stream.report(suggestion['idx'], validation_loss)
            # Continue with suggest/report or report_and_suggest as needed.
        finally:
            stream.close()
    finally:
        api.terminate_session(session_id)
```

`initialize` and `reset` default to `effort="medium"`. A report records one
observation; `max_steps` limits observations per model, not optimizer updates.
Use `status()` to inspect progress and rankings. `reset(...)` starts a fresh
run after the previous run completes. `api.list_sessions()` lists your sessions;
`api.wait_for_termination(session_id)` waits for a release to complete.

## Datasets and training budgets

Each demo starts with ten sampled model configurations and allows up to 50
validation observations across them, with at most ten observations per model.
After tuning, all models finish to the same update cap for comparison.
Classification ranks use validation cross-entropy; regression ranks use MSE in
original target units. Lower is better. Test data is used only for final reporting.

| Dataset | Updates per observation | Maximum updates per model | Source |
| --- | ---: | ---: | --- |
| `iris` | 10 | 100 | scikit-learn |
| `wine` | 200 | 2,000 | scikit-learn |
| `breast-cancer` | 30 | 300 | scikit-learn |
| `digits` | 60 | 600 | scikit-learn |
| `glass` | 100 | 1,000 | OpenML dataset 41 |
| `vehicle` | 60 | 600 | OpenML dataset 54 |
| `diabetes` | 10 | 100 | scikit-learn |
| `friedman` | 200 | 2,000 | Generated regression data |
| `linear-regression` | 200 | 2,000 | Generated regression data |
| `toy-lm` | 30 | 300 | Generated language teaching task |
| `toy-vla` | 200 | 2,000 | Generated vision/language teaching task |

Glass and vehicle download on first use. Ordinary datasets use a 70/15/15
train/validation/test split; wine uses 55/30/15. Identical language contexts are
grouped into one split. Preprocessing is fitted on training data only.

`--block-steps N` sets updates per observation and a maximum of `10*N` updates.
The defaults are bounded workload profiles, not guarantees of convergence.
Version 2 uses loss-based classification ranks; these are not comparable with
version 1 accuracy-error ranks. `--seed` sets all seeds; `--config-seed`,
`--split-seed`, and `--training-seed` allow independent overrides.

## Repeated benchmarks

```sh
python -m tuning_demo.stats --api-url https://hyperopt.dev --email you@example.com \
  --datasets iris wine --runs 2
```

Without dataset/run options, this runs all eleven datasets ten times, starting
at seed 42. Each trial starts with fresh models and tuning state. Sessions are
reused per dataset; `--session-scope run` reuses one session across datasets.
Service benchmarks run sequentially.

Local reference schedulers need no service account:

```sh
python -m tuning_demo --scheduler asha --dataset digits
python -m tuning_demo.stats --scheduler flaml --datasets iris wine --runs 2 --workers 2
```

Available schedulers are `service` (default), `asha`, `median`, `pbt`, `pb2`,
and `flaml`. `--workers` applies only to local reference schedulers; it cannot
parallelize service benchmarks. Reference statistics default to available CPU
cores, capped by trial count; use `--workers 1` to limit resource use.
See [benchmark details](tuning_demo/benchmarking/README.md) for algorithm settings.

Results include configurations, seeds, observation histories, timings, losses,
and rankings at the tuning cutoff and after final training. Statistics report
the mean final rank of the model selected at the cutoff and the best final rank
among models already capped at that cutoff. Ties receive ranks such as 1, 1, 3;
unavailable measurements and failed trials do not count as zero. A failed service
trial skips the remaining trials for that dataset. The command exits nonzero
on trial or cleanup failure and preserves partial results.

Timing includes client training and service round trips; it is not a pure
measurement of service compute. The service ranking uses latest reported losses,
not predicted fully trained losses. PBT/PB2 compare evolved checkpoint lineages,
not the original fixed configurations.

## Custom datasets

```sh
python -m pip install 'tuning-client[demo,custom]'
python -m tuning_demo --api-url https://hyperopt.dev --custom-data ./data.csv \
  --target label --task classification --drop-columns id
python -m tuning_demo --api-url https://hyperopt.dev --custom-data ./data.parquet \
  --target price --task regression
```

Use one row per independent example, unique column names, a non-missing scalar
target, and numeric or categorical input columns. Drop identifiers and columns
that leak the target. Custom datasets default to 100 updates per observation.
Files stay local. Use an appropriate split outside this demo for time series,
grouped observations, or other data that requires special leakage controls.

## License and third-party notices

Original SDK and demo code is licensed under the [MIT License](LICENSE).

The benchmark implementations include adaptations of Ray Tune (Apache-2.0)
and FLAML (MIT). Their license texts and attribution are included in the package;
see [third-party notices](tuning_demo/benchmarking/THIRD_PARTY.md).
