Metadata-Version: 2.5
Name: expr_tracker
Version: 0.2.6
Summary: Local-first experiment tracking with queryable history and expression-based alerts on your training metrics
Project-URL: Homepage, https://hspk.github.io/expr_tracker/
Project-URL: Documentation, https://hspk.github.io/expr_tracker/
Project-URL: Repository, https://github.com/HSPK/expr_tracker
Project-URL: Issues, https://github.com/HSPK/expr_tracker/issues
Project-URL: Changelog, https://github.com/HSPK/expr_tracker/releases
Author-email: HSPK <whxway@whu.edu.cn>
License-Expression: MIT
License-File: LICENSE
Keywords: alerting,deep-learning,experiment-tracking,machine-learning,mlops,monitoring,training,wandb
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: System :: Monitoring
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: click>=8.1.0
Requires-Dist: loguru>=0.7.3
Requires-Dist: pydantic>=2.0
Provides-Extra: all
Requires-Dist: nvidia-ml-py>=12.0; extra == 'all'
Requires-Dist: pandas>=1.5; extra == 'all'
Requires-Dist: polars>=0.20; extra == 'all'
Requires-Dist: trackio>=0.4.0; extra == 'all'
Requires-Dist: wandb>=0.21.0; extra == 'all'
Provides-Extra: gpu
Requires-Dist: nvidia-ml-py>=12.0; extra == 'gpu'
Provides-Extra: pandas
Requires-Dist: pandas>=1.5; extra == 'pandas'
Provides-Extra: polars
Requires-Dist: polars>=0.20; extra == 'polars'
Provides-Extra: trackio
Requires-Dist: trackio>=0.4.0; extra == 'trackio'
Provides-Extra: wandb
Requires-Dist: wandb>=0.21.0; extra == 'wandb'
Description-Content-Type: text/markdown

# Experiment Tracker

[![PyPI](https://img.shields.io/pypi/v/expr-tracker)](https://pypi.org/project/expr-tracker/)
[![Python](https://img.shields.io/pypi/pyversions/expr-tracker)](https://pypi.org/project/expr-tracker/)
[![License](https://img.shields.io/pypi/l/expr-tracker)](LICENSE)
[![Docs](https://img.shields.io/badge/docs-github.io-blue)](https://hspk.github.io/expr_tracker/)

A local-first experiment tracker. Metrics land in a JSONL file you own, stay
queryable while the run is live, and can trigger alerts from an expression language.
`wandb` and `trackio` are optional mirrors, not requirements.

📖 **Documentation: <https://hspk.github.io/expr_tracker/>**

```python
import expr_tracker as et

et.init(project="demo", name="run-1", alert_rules=["zscore(loss[50]) > 3 => error: spike"])
for step in range(1000):
    et.log({"loss": loss, "lr": lr})
et.finish()

et.history(50)                    # the last 50 steps, as dicts
et.history(-1, output_type="pd")  # everything, as a DataFrame
```

## Why

- **The file is the source of truth.** One JSON object per step, appended to
  `metrics.jsonl`. No server, no database, no vendor.
- **History is queryable during the run.** `et.history(n)` answers from an in-memory
  cache and touches the file only for what it evicted — 227&nbsp;µs for
  `history(50)` whether the run has 1,000 steps or 100,000.
- **Alerts are expressions, not callbacks.** `zscore(loss[50]) > 3 or isnan(loss)`
  is parsed, validated, and evaluated against a rolling window. Rules can be replayed
  over a finished run to tune thresholds before you trust them.
- **It stays out of the way.** `log()` costs ~26&nbsp;µs. A failed disk, a dead
  webhook or an unserialisable value degrades with a warning; none can stop training.

## Install

```bash
uv add expr_tracker                 # local-first: click, loguru, pydantic only
uv add "expr_tracker[wandb]"        # mirror to Weights & Biases
uv add "expr_tracker[trackio]"      # mirror to trackio
uv add "expr_tracker[pandas]"       # history(output_type="pandas")
uv add "expr_tracker[all]"          # everything
```

Only the JSONL history is built in. A missing extra is reported with the exact
install command; it never crashes a run.

## Features

| | |
| --- | --- |
| [Logging](https://hspk.github.io/expr_tracker/guide/logging/) | One line per step. Several `log()` calls for one step merge into one row, wandb-compatible `step`/`commit` semantics, numpy and pydantic values handled. |
| [History](https://hspk.github.io/expr_tracker/guide/history/) | `et.history(n)` during or after the run, offline reads of any run directory, dict/pandas/polars output, bounded in-memory cache with observable hit rate. |
| [Alerts](https://hspk.github.io/expr_tracker/guide/alerts/) | An expression DSL with rolling windows, three-valued logic (no false alarms during warm-up), a rule state machine, and a watchdog that catches a hung run. |
| [Channels](https://hspk.github.io/expr_tracker/guide/alerts/#channels) | Lark, Slack, DingTalk, WeCom, generic webhook, email — with rate limiting, dedup, retries and per-channel routing. |
| [Artifacts](https://hspk.github.io/expr_tracker/guide/artifacts/) | Versioned file sets, deduplicated by content, shared across a project's runs, with lineage. |
| [Spans](https://hspk.github.io/expr_tracker/guide/spans/) | Time the parts of a step, and their parts. Each duration becomes a metric, so alerts and queries work on it unchanged; `et trace` exports the timeline for Perfetto. `print_fn` prints the tree live, and plugins attach CPU and GPU cost to each region. |
| [Streams](https://hspk.github.io/expr_tracker/guide/streams/) | Independent producers — a data worker and a training loop — each with their own step cursor and file inside one run. |
| [Distributed](https://hspk.github.io/expr_tracker/guide/distributed/) | Per-rank shards so concurrent appends cannot corrupt step order; only rank 0 alerts by default. |
| [CLI](https://hspk.github.io/expr_tracker/guide/cli/) | `et history`, `et trace`, `et rules explain`, `et rules test`, `et alert`. |

## Examples

Every one runs offline, with no account and no network.

| | |
| --- | --- |
| [`quickstart.py`](examples/quickstart.py) | The sixty-second tour: log a run, merge eval into the training step, read the history back while it is open and again afterwards. |
| [`alert_rules.py`](examples/alert_rules.py) | Four rules against four faults — a loss spike, a non-finite loss, a stalled curve and a regression that must persist. `--fault none` fires nothing, which is the point. |
| [`profile_step.py`](examples/profile_step.py) | Where a step goes. Nested spans become metrics, a plugin adds CPU cost, and the tree exports to Perfetto. |
| [`early_stopping.py`](examples/early_stopping.py) | Querying your own history mid-run to decay the learning rate on a plateau and stop when it stops paying. |
| [`checkpoints.py`](examples/checkpoints.py) | Checkpoints as versioned artifacts, deduplicated by content and fetched later by alias. |
| [`multiprocess_pipeline.py`](examples/multiprocess_pipeline.py) | Four data producers and four trainers as eight processes in one run, with bounded staleness. Each worker gets its own stream and its own lane in the trace, and the blocking spans show which side is the bottleneck. |

```bash
uv run python examples/quickstart.py
uv run python examples/alert_rules.py --fault spike
uv run python examples/multiprocess_pipeline.py --produce-ms 10 --train-ms 40
```

## wandb compatibility

Migrating an existing script is usually one line:

```python
# import wandb as et
import expr_tracker as et
```

`init`, `log`, `finish`, `alert`, `log_artifact`, `use_artifact`, `Artifact`,
`define_metric`, `run.summary`, `run.step`, `run.dir` and `run.url` keep their wandb
names and signatures. See the
[compatibility table](https://hspk.github.io/expr_tracker/guide/backends/#wandb-compatibility).

## Development

```bash
uv sync --all-extras
uv run pytest                                   # everything
uv run pytest -m "not slow and not benchmark"   # the fast suite
uv run pytest -m benchmark -s                   # timing and memory report
uv run pytest --cov=expr_tracker                # coverage
uv run ruff check src tests
uv run ruff format src tests
```

### Test layout

| File | Covers |
| --- | --- |
| `test_history`, `test_expr_*`, `test_alert_*`, `test_writer_durability`, … | per-module unit tests |
| `test_correctness.py` | value and type round trips, randomised commit sequences, ordering invariants |
| `test_cache.py` | that the cache really serves reads: zero-IO assertions, eviction boundaries, warm/cold parity |
| `test_failure_modes.py` | degradation: write failures, read-only dirs, encoder blow-ups, dead alert backends |
| `test_e2e.py` | full runs, resume, crash recovery, offline reads, CLI |
| `test_scenarios.py` | live cross-process reads, alerts during eviction, out-of-order resume |
| `test_hot_paths.py` | contracts and defaults of `et.log` / `et.history` / summary / alerts |
| `test_value_encoding.py` | numpy, pydantic, datetime, Path, Enum round trips; output types; query bounds |
| `test_expr_properties.py` | DSL properties: render round-trip stability, precedence, the whole `M` builder |
| `test_trace.py` | Chrome Trace export: lane layout, stream and step selection, the CLI |
| `test_spans.py` | nesting, aggregation, decorator and async forms, errors, thread and task isolation |
| `test_examples.py` | the shipped examples run, and their backpressure claims hold |
| `test_span_plugins.py` | `print_fn` output and indentation, the plugin protocol, failure isolation, CPU/GPU built-ins |
| `test_streams.py` | stream naming and validation, isolation, resolution order, backend grouping, two-process runs |
| `test_distributed.py` | rank shards, `alert_on_rank`, real multi-process runs |
| `test_wandb.py` | real wandb in offline mode: parameter mapping, step alignment, artifacts |
| `test_trackio.py` | trackio contract, resume mapping, real end-to-end |
| `test_lark.py` | Lark card construction; real delivery when `ET_LARK_TEST_WEBHOOK` is set |
| `test_stress.py` (`slow`) | 100k-row writes, concurrency, cache thrash, write-failure recovery |
| `test_benchmark.py` (`benchmark`) | throughput, tail latency, query cost, memory stability |

### Docs

```bash
uv run --group docs mkdocs serve    # preview at localhost:8000
uv run --group docs mkdocs build    # build into site/
```

Published to GitHub Pages by `.github/workflows/docs.yaml` on every push to `main`.
Internals: [`docs/design.md`](docs/design.md) (data model and key invariants) and
[`docs/architecture.md`](docs/architecture.md) (module map, read/write paths,
concurrency model).

## License

[MIT](LICENSE)
