Metadata-Version: 2.5
Name: jitter-ml
Version: 0.1.1
Summary: Root-cause analysis tool for on-device ML latency
Author-email: Ayush Dubey <ayushdubey3740@gmail.com>
License: MIT
License-File: LICENSE
Keywords: android,benchmark,latency,machine-learning,onnx,statistics,tflite
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Requires-Dist: httpx>=0.25.0
Requires-Dist: numpy>=1.24.0
Requires-Dist: pydantic>=2.0.0
Requires-Dist: rich>=13.0.0
Requires-Dist: scipy>=1.10.0
Requires-Dist: typer>=0.9.0
Provides-Extra: dev
Requires-Dist: build>=1.0.0; extra == 'dev'
Requires-Dist: fastapi>=0.109.0; extra == 'dev'
Requires-Dist: hypothesis>=6.90.0; extra == 'dev'
Requires-Dist: jsonschema>=4.21.0; extra == 'dev'
Requires-Dist: mypy>=1.8.0; extra == 'dev'
Requires-Dist: pytest-cov>=4.1.0; extra == 'dev'
Requires-Dist: pytest>=8.0.0; extra == 'dev'
Requires-Dist: ruff>=0.3.0; extra == 'dev'
Requires-Dist: twine>=5.0.0; extra == 'dev'
Requires-Dist: uvicorn>=0.27.0; extra == 'dev'
Provides-Extra: onnx
Requires-Dist: onnxruntime>=1.15.0; extra == 'onnx'
Provides-Extra: tflite
Requires-Dist: ai-edge-litert>=1.0.0; extra == 'tflite'
Description-Content-Type: text/markdown

# JITTER

**Root-cause analysis for on-device ML latency.**

Jitter reruns your model benchmark under four controlled conditions and tells you *why* it is slow or inconsistent on a real phone: warmup, thermal throttling, delegate fallback, background load, or a genuine regression.

> **Status: Alpha release (v0.1.1).** Published on PyPI as `jitter-ml`. Controlled reruns, statistical validation harness, CLI, and report schema v0.1 are implemented. Physical on-device hardware telemetry and field validation are under active development.

---

## The problem

A single latency number from a phone benchmark is nearly meaningless:

- The first iterations are slow (JIT, caches, lazy initialization).
- The phone heats up and the SoC throttles, so run 50 is slower than run 5.
- NNAPI can silently fall back to the CPU, so the accelerator you requested is not the one that ran.
- Other processes steal time in bursts.
- And occasionally, your change really did make the model slower.

Existing tools measure well. They do not tell you which of these five things you are looking at. Jitter does.

## How it works

Each diagnosis is four controlled reruns of the same benchmark:

| # | Rerun | Held constant | Varied | Isolates |
|---|-------|---------------|--------|----------|
| 1 | Cold vs warm | model, input, delegate | fresh process vs warmed-up process | one-time startup cost |
| 2 | Cooldown | model, input, delegate | phone rests until thermal state recovers | thermal throttling |
| 3 | Delegate swap | model, input | CPU, GPU, NNAPI; records the delegate that actually executed | silent delegate fallback |
| 4 | Quiet device | model, input, delegate | background load controlled | device noise |

The outputs are compared statistically and classified:

| Label | Signature |
|-------|-----------|
| `warmup_drift` | First *k* iterations exceed the steady-state median by more than 3 MAD, then converge |
| `thermal_throttling` | Monotonic rolling-median increase, thermal status at or above moderate, recovery after cooldown |
| `delegate_fallback` | Requested delegate differs from executed delegate, or a step change in median with unchanged variance |
| `background_noise` | Heavy-tailed spikes (high p99/median ratio) that vanish in the quiet-device rerun |
| `regression` | The difference persists across all four reruns |

Each verdict carries a confidence score and the evidence that produced it.

## Components

```mermaid
flowchart LR
    A[Python library + CLI] -- ADB / LAN --> B[Android runner]
    B -- latency, thermal status, battery temp, executed delegate --> C[(report.json)]
    C --> D[Static web dashboard]
    A --> C
```

1. **Python library and CLI** (`jitter`): defines benchmarks, orchestrates the four reruns, computes statistics, classifies the result.
2. **Android runner** (Kotlin, Jetpack Compose, MVI): runs the model with LiteRT or ONNX Runtime Mobile in a foreground service and logs per-iteration latency, `PowerManager` thermal status and headroom, battery temperature, and the delegate that actually ran.
3. **Web dashboard** (Next.js, static): load report files to compare runs and read the verdict. Reports are parsed in the browser. There is no backend, no account and no upload.

## Usage

```bash
pip install jitter-ml

# run the four controlled reruns on an Android device via ADB
jitter run --model mobilenet_v2 --runner adb --serial <device-serial> --output report.json

# or generate a synthetic benchmark report
jitter simulate --failure-mode thermal_throttling --output report.json

# compare two runs statistically
jitter compare before.json after.json --run-a 0 --run-b 1

# evaluate root cause on a benchmark report
jitter diagnose report.json
```

Example output (simulated):

```
$ jitter run --model mobilenet_v3.tflite --runner adb --serial pixel-7a --output report.json
[1/4] cold vs warm .......... warmup 6 iters, steady-state from iter 7
[2/4] cooldown .............. median 14.1 ms after 240 s rest (was 33.0 ms)
[3/4] delegate swap ......... requested GPU, executed GPU, no fallback
[4/4] quiet device .......... noise floor +/- 0.4 ms

$ jitter diagnose report.json
verdict: thermal_throttling   confidence 0.93
drift: +2.3x over 90 s (95% CI 2.1 to 2.5x)
```

Planned exit codes for CI: `0` ok, `1` error, `2` regression detected.

## Statistics

- **Warmup detection:** change-point detection on a rolling median (CUSUM).
- **Confidence intervals:** bootstrap (percentile or BCa), 10,000 resamples.
- **Comparing runs:** Mann-Whitney U, Hodges-Lehmann shift estimate, Cliff's delta as the effect size.
- **Reporting:** median, p95 and p99 with intervals, never a bare mean.

The statistics core will be validated against synthetic latency distributions with known ground truth before any real-device claim is made.

## Report format (draft)

```json
{
  "schema_version": "0.1",
  "model": { "name": "mobilenet_v3.tflite", "sha256": "..." },
  "device": { "model": "...", "android_api": 34 },
  "runs": [
    {
      "rerun": "cold_warm",
      "delegate_requested": "NNAPI",
      "delegate_executed": "GPU",
      "samples_ms": [14.2, 14.0, 13.9],
      "telemetry": { "thermal_status": [0, 0, 1], "battery_temp_c": [31.2, 31.4, 31.9] }
    }
  ],
  "diagnosis": { "label": "thermal_throttling", "confidence": 0.93, "evidence": [] }
}
```

## Validation plan

The claim that matters is how often the engine names the right cause. The plan:

1. Inject known faults on purpose: heat the device with a load, force CPU fallback, add background bursts.
2. Run the diagnosis and record the confusion matrix across the five labels.
3. Publish the accuracy numbers, including the failures, alongside a case study on at least two physical phones.

Emulators are not used: they have no real thermal behavior and no real NNAPI.

## Roadmap

- [ ] Statistics core and desktop CLI (ONNX Runtime), tested on synthetic distributions
- [ ] Trace recorder, report schema v0.1, replay
- [ ] Android runner with the telemetry above
- [ ] Diagnosis engine with fault-injection validation
- [ ] Static web dashboard
- [ ] PyPI release, CI, documentation site
- [ ] Case study across physical devices and delegates

**Out of scope for now:** model conversion, a cloud device farm, user accounts, iOS.

## Related work

- Android Macrobenchmark and the TFLite `benchmark_model` tool measure latency well. Jitter's contribution is controlled reruns, root-cause classification and the statistics around them.
- [Flakerad](https://github.com/AyushDubey23/flakerad), by the same author, applies the same idea (controlled reruns to infer the cause) to flaky tests.

## Contributing

The project is in Alpha (v0.1.1). Issues with ideas, device-specific thermal or delegate quirks, and statistical critique are the most useful contributions right now.

## License

MIT

## Author

Ayush ([@AyushDubey23](https://github.com/AyushDubey23))
