Metadata-Version: 2.3
Name: ml-dash
Version: 0.8.0
Summary: ML experiment tracking and data storage
Keywords: machine-learning,experiment-tracking,mlops,data-storage
Author: Ge Yang, Tom Tao
License: MIT License
         
         Copyright (c) 2025 Ge Yang, Tom Tao
         
         Permission is hereby granted, free of charge, to any person obtaining a copy
         of this software and associated documentation files (the "Software"), to deal
         in the Software without restriction, including without limitation the rights
         to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
         copies of the Software, and to permit persons to whom the Software is
         furnished to do so, subject to the following conditions:
         
         The above copyright notice and this permission notice shall be included in all
         copies or substantial portions of the Software.
         
         THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
         IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
         FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
         AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
         LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
         OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
         SOFTWARE.
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Dist: httpx>=0.27.0
Requires-Dist: rich>=13.0.0
Requires-Dist: cryptography>=42.0.0
Requires-Dist: params-proto>=3.0.0
Requires-Dist: keyring>=25.0.0 ; extra == 'auth'
Requires-Dist: pytest>=8.0.0 ; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.23.0 ; extra == 'dev'
Requires-Dist: pyjwt>=2.8.0 ; extra == 'dev'
Requires-Dist: numpy>=1.24 ; extra == 'dev'
Requires-Dist: ruff>=0.3.0 ; extra == 'dev'
Requires-Dist: mypy>=1.9.0 ; extra == 'dev'
Requires-Dist: imageio>=2.31.0 ; extra == 'video'
Requires-Dist: imageio-ffmpeg>=0.4.9 ; extra == 'video'
Requires-Dist: scikit-image>=0.21.0 ; extra == 'video'
Requires-Python: >=3.9
Project-URL: Documentation, https://docs.dash.ml
Project-URL: Homepage, https://dash.ml
Project-URL: Source, https://github.com/fortyfive-labs/ml-dash
Provides-Extra: auth
Provides-Extra: dev
Provides-Extra: video
Description-Content-Type: text/markdown

# ML-Dash

A simple and flexible SDK for ML experiment tracking and data storage with background buffering for high-performance training.

## Features

### Core Features
- **Three Usage Styles**: Pre-configured singleton (dxp), context manager, or direct instantiation
- **Dual Operation Modes**: Remote (API server) or local (filesystem)
- **OAuth2 Authentication**: Secure device flow; the SDK reads the token the `ml-dash` CLI stores
- **Auto-creation**: Automatically creates namespace, project, and folder hierarchy
- **Upsert Behavior**: Updates existing experiments or creates new ones
- **Experiment Lifecycle**: Automatic status tracking (RUNNING, COMPLETED, FAILED, CANCELLED)
- **Organized File Storage**: Prefix-based file organization with unique snowflake IDs
- **Rich Metadata**: Tags, bindrs, descriptions, and custom metadata support
- **Simple API**: Minimal configuration, maximum flexibility

### Performance Features (New in 0.6.7)
- **Background Buffering**: Non-blocking I/O operations eliminate training interruptions
- **Automatic Batching**: Time-based (5s) and size-based (100 items) flush triggers
- **Track API**: Time-series data tracking for robotics, RL, and sequential experiments
- **Numpy Image Support**: Direct saving of numpy arrays as PNG/JPEG images
- **Parallel Uploads**: ThreadPoolExecutor for efficient file uploads

## Installation

This repo is the **Python SDK** — the library you `import ml_dash` from. The
`ml-dash` **command line** is a separate program, distributed on npm; see
[The CLI moved to npm](#the-cli-moved-to-npm) below.

### SDK (this package)

<table>
<tr>
<td>Using uv (recommended)</td>
<td>Using pip</td>
</tr>
<tr>
<td>

```bash
uv add ml-dash
```

</td>
<td>

```bash
pip install ml-dash
```

</td>
</tr>
</table>

### CLI

Version 0.1.1, published as `@dreamlake/ml-dash`. With Node.js >= 20.19:

```bash
npm install -g @dreamlake/ml-dash
```

Or a self-contained binary, needing neither Node.js nor Python:

```bash
# macOS / Linux
curl -fsSL https://pub-42e1dcc7de574d4a92984865fdc95f10.r2.dev/install.sh | sh
```

```powershell
# Windows PowerShell
irm https://pub-42e1dcc7de574d4a92984865fdc95f10.r2.dev/install.ps1 | iex
```

On Alpine the musl builds need `apk add --no-cache libstdc++`.

The package is scoped because the unscoped `ml-dash` name is not claimable on
npm, but **the command is still `ml-dash`**. Either channel updates itself, in
place and through the channel it was installed from:

```bash
ml-dash update --check   # report whether a newer release exists
ml-dash update           # install it
```

## The CLI moved to npm

Up to and including 0.6.27, `pip install ml-dash` also put an `ml-dash`
executable on your PATH. From **0.7.0** it does not: the command line has been
rewritten in TypeScript and ships on its own — as the npm package
`@dreamlake/ml-dash` or the standalone installer above — so it installs and
updates without touching your training environment's Python dependencies.
Upgrading the Python package to 0.7.0 removes the old executable; install the
new CLI alongside it.

Nothing about the SDK changes. `pip install ml-dash` is still how you get
`ml_dash`, and every `Experiment`, `files`, `tracks`, and `storage` API is
untouched.

|  | Before | Now |
|---|---|---|
| SDK (`import ml_dash`) | `pip install ml-dash` | `pip install ml-dash` |
| CLI (`ml-dash login`, `upload`, …) | `pip install ml-dash` | `npm install -g @dreamlake/ml-dash`, or the [standalone installer](#cli) |

The command names and arguments carry over, and **the two share a credential
store**: the CLI writes its token to the same OS keyring entry (and the same
`~/.dash/` fallback files) that `ml_dash.auth.token_storage` reads, so
`ml-dash login` still authenticates your Python code and an existing valid
login is normally picked up as-is. Re-run `ml-dash login` if it is not.

Command *behaviour* is not identical — the rewrite changes and fixes a number
of things — and the CLI's own release notes are where those differences are
described.

If you were importing `ml_dash.cli` or `ml_dash.cli_commands.*` directly from
Python — an unsupported path, but a real one — those modules are gone. Shell
out to the `ml-dash` binary instead.

Releases up to 0.6.27 keep the old Python CLI; pin `ml-dash<0.7` only if you
cannot install the new one yet.

## Quick Start

### 1. Authenticate (Required for Remote Mode)

```bash
npm install -g @dreamlake/ml-dash   # one time; requires Node.js >= 20.19
                                    # (or use the standalone installer above)
ml-dash login
```

This opens your browser for secure OAuth2 authentication. Your credentials are
stored securely in your system keychain — where the Python SDK reads them.

### 2. Start Tracking Experiments

#### Option A: Use the Pre-configured Singleton (Easiest)

```python
from ml_dash.auto_start import dxp

# Start experiment (uploads to https://api.dash.ml by default)
with dxp.run:
    dxp.log("Training started", level="info")
    dxp.params.set(learning_rate=0.001, batch_size=32)

    for epoch in range(10):
        loss = train_one_epoch()
        dxp.metrics("train").log(loss=loss, epoch=epoch)
```

#### Option B: Create Your Own Experiment

```python
from ml_dash import Experiment

with Experiment(
  prefix="alice/my-project/my-experiment",
  dash_url="https://api.dash.ml",  # token auto-loaded
).run as experiment:
  experiment.log("Hello!", level="info")
  experiment.params.set(lr=0.001)
```

#### Option C: Local Mode (No Authentication Required)

```python
from ml_dash import Experiment

with Experiment(
  project="my-project", prefix="my-experiment", dash_root=".dash"
).run as experiment:
  experiment.log("Running locally", level="info")

```

## New Features in 0.6.7

### 🚀 Background Buffering (Non-blocking I/O)

All write operations are now buffered and executed in background threads:

```python
with Experiment("my-project/exp").run as experiment:
    for i in range(10000):
        # Non-blocking! Returns immediately
        experiment.log(f"Step {i}")
        experiment.metrics("train").log(loss=loss, accuracy=acc)
        experiment.files("frames").save_image(frame, to=f"frame_{i}.jpg")

    # All data automatically flushed when context exits
```

Configure buffering via environment variables:
```bash
export ML_DASH_BUFFER_ENABLED=true
export ML_DASH_FLUSH_INTERVAL=5.0
export ML_DASH_LOG_BATCH_SIZE=100
```

### 📊 Track API (Time-Series Data)

Perfect for robotics, RL, and sequential experiments. Supports **timestamp inheritance** for synchronized multi-modal data and **track slicing** for efficient querying.

```python
with Experiment("robotics/training").run as experiment:
    for step in range(1000):
        # First track auto-generates timestamp
        experiment.tracks("robot/position").append(
            step=step,
            x=position[0],
            y=position[1],
            z=position[2]
        )

        # Other tracks inherit same timestamp with _ts=-1 (synchronized!)
        experiment.tracks("camera/left").append(width=640, height=480, _ts=-1)
        experiment.tracks("robot/velocity").append(vx=0.1, vy=0.0, _ts=-1)
        experiment.tracks("sensors/lidar").append(ranges=[1.5, 2.0], _ts=-1)

    experiment.tracks.flush()

    # Query synchronized data with floor-match timestamp queries
    pose_slice = experiment.tracks("robot/position").slice(0.0, 10.0)

    # Single timestamp query
    entry = pose_slice.findByTime(5.5)  # Returns entry at timestamp 5.0

    # Batch queries
    entries = pose_slice.findByTime([1.0, 3.5, 7.0])  # Returns list of 3 entries

    # Iterate through time range
    for entry in pose_slice:
        print(entry["timestamp"], entry["x"], entry["y"])
```

See [the Tracks guide](https://docs.dash.ml/guides/tracks) for complete documentation.

### 🖼️ Numpy Image Support

Save numpy arrays directly as images (PNG/JPEG):

```python
import numpy as np

with Experiment("vision/training").run as experiment:
    # From MuJoCo, OpenCV, PIL, etc.
    pixels = renderer.render()  # numpy array

    # Save as PNG (lossless)
    experiment.files("frames").save_image(pixels, to="frame.png")

    # Save as JPEG with quality control
    experiment.files("frames").save_image(pixels, to="frame.jpg", quality=85)

    # Auto-detection also works
    experiment.files("frames").save(pixels, to="frame.jpg")
```

See [CHANGELOG.md](CHANGELOG.md) for complete release notes.

## Development Setup

### Installing Dev Dependencies

To contribute to ML-Dash or run tests, install the development dependencies:

<table>
<tr>
<td>Using uv (recommended)</td>
<td>Using pip</td>
</tr>
<tr>
<td>

```bash
uv sync --extra dev
```

</td>
<td>

```bash
pip install -e ".[dev]"
```

</td>
</tr>
</table>

This installs:
- `pytest>=8.0.0` - Testing framework
- `pytest-asyncio>=0.23.0` - Async test support
- `ruff>=0.3.0` - Linter and formatter
- `mypy>=1.9.0` - Type checker

The Sphinx toolchain is gone with the docs tree — see
[Documentation](#documentation) below.

### Running Tests

<table>
<tr>
<td>Using uv</td>
<td>Using pytest directly</td>
</tr>
<tr>
<td>

```bash
uv run pytest
```

</td>
<td>

```bash
pytest
```

</td>
</tr>
</table>

### Documentation

The docs live at **[docs.dash.ml](https://docs.dash.ml)** and are built from
[`dreamlake-ai/dash-workspace`](https://github.com/dreamlake-ai/dash-workspace)
(`docs/`), not from this repo. The Sphinx tree that used to sit in `docs/` and
the Docusaurus site in `website/` have both been retired — they had drifted
apart from each other and from the SDK, and one of them documented `_ts` as a
required argument that raises `ValueError` when it is optional and raises
`ConfigurationError`.

Doc changes therefore go to that repo. It runs a `check:sdk` guard that
verifies every python snippet against the installed `ml_dash`, so a signature
change here will fail the docs build there rather than sit unnoticed.

The `skills/dash-docs/` directory is a vendored copy of the generated agent
skill. Never edit it by hand:

```bash
make sync-skill    # refresh from docs.dash.ml
make check-skill   # fail if the vendored copy is stale
```

For maintainers, to build and publish a new release: `uv build && uv publish`
(this publishes the SDK only — the CLI is released from its own npm repo).
