Metadata-Version: 2.4
Name: vayrm
Version: 1.1.0
Summary: Adaptive Compute Runtime
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: psutil>=5.9.0
Requires-Dist: cryptography>=41.0.0
Requires-Dist: PyYAML>=6.0.1
Requires-Dist: aiohttp>=3.9.0
Requires-Dist: qrcode>=7.4
Requires-Dist: pillow>=10.0.0
Requires-Dist: python-dotenv>=1.0.0
Requires-Dist: rich>=13.0.0
Requires-Dist: requests>=2.31.0
Requires-Dist: fastapi>=0.100.0
Requires-Dist: uvicorn>=0.23.0
Requires-Dist: networkx>=3.0
Requires-Dist: torch>=2.14.1
Requires-Dist: transformers<5,>=4.57
Requires-Dist: datasets<5,>=4.8
Requires-Dist: accelerate<2,>=1.15
Requires-Dist: tokenizers<1,>=0.22
Requires-Dist: safetensors<1,>=0.8
Requires-Dist: torchvision<1,>=0.29
Requires-Dist: pyarrow>=15
Requires-Dist: numpy>=1.26
Requires-Dist: openpyxl>=3.1
Provides-Extra: macos
Provides-Extra: windows
Provides-Extra: linux
Provides-Extra: dev
Requires-Dist: pytest>=7.0.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
Requires-Dist: mypy>=1.0.0; extra == "dev"
Requires-Dist: ruff>=0.0.260; extra == "dev"
Requires-Dist: httpx<1,>=0.28; extra == "dev"

# PALADIN Adaptive Compute Runtime (ACR)

> **PALADIN ACR** is a heterogeneous, adaptive distributed-compute
> runtime combining local NodeBrain intelligence, capability-aware
> scheduling, adaptive workload reallocation, direct node-to-node
> training communication, persistent lifecycle/storage management,
> observability, and Tailscale-based production networking.

------------------------------------------------------------------------

## Documentation index

| Topic | Page |
|---|---|
| Endpoints, wire messages, fields, auth, errors (authoritative; checked by `tests/test_endpoints_contract.py`) | [docs/ENDPOINTS.md](docs/ENDPOINTS.md) |
| Install and first run | [docs/getting-started.md](docs/getting-started.md) |
| Environment-variable reference | [docs/configuration.md](docs/configuration.md) |
| Deployment, ports, troubleshooting | [docs/deployment.md](docs/deployment.md) |
| Architecture | [docs/architecture.md](docs/architecture.md), [docs/integrated_architecture.md](docs/integrated_architecture.md) |
| NodeBrain / execution / scheduler / transport | [docs/node.md](docs/node.md), [docs/execution.md](docs/execution.md), [docs/scheduler.md](docs/scheduler.md), [docs/transport.md](docs/transport.md) |
| Training sessions, topology, capability status | [docs/TRAINING_SESSIONS.md](docs/TRAINING_SESSIONS.md), [docs/TRAINING_POLICY_ENGINE.md](docs/TRAINING_POLICY_ENGINE.md), [docs/DISTRIBUTED_EXECUTION.md](docs/DISTRIBUTED_EXECUTION.md) |
| Memory / device / GPU admission and resource optimisation | [docs/RESOURCE_OPTIMIZATION.md](docs/RESOURCE_OPTIMIZATION.md) |
| Simulation and benchmarking | [docs/HETEROGENEOUS_SIMULATION.md](docs/HETEROGENEOUS_SIMULATION.md), [research/greedy_vs_paladin_vs_optimality.md](research/greedy_vs_paladin_vs_optimality.md) |
| Platform notes | [docs/cross-platform.md](docs/cross-platform.md), [docs/platforms/macos/README.md](docs/platforms/macos/README.md), [docs/platforms/windows/README.md](docs/platforms/windows/README.md), [docs/macos_dev_storage.md](docs/macos_dev_storage.md) |

There is no root-level `endpoints.md`; the endpoint contract is `docs/ENDPOINTS.md` (a second copy would drift).

## Quick start

Requirements: Python 3.10-3.13 (the suite is run with 3.11), a dependency **profile** matching the machine (OS, CPU architecture, CPU vs NVIDIA GPU: `requirements/<profile>.txt`, documented in [pip-requirements.md](pip-requirements.md)), and Tailscale for real cross-machine runs. A plain `pip install -e .` takes PyPI's default torch build (Windows: CPU-only; Linux: CUDA 13 plus many GB of `nvidia-*` wheels), so workers are created with the profile-aware bootstrap below. Every node (ROOT and each worker) needs its own venv: installing on ROOT does not install anything remotely, and a worker lacking e.g. `pyarrow` rejects work with `missing_dependency` (pip-requirements.md §4.6). Per-OS setup (CPU vs GPU nodes, commands, prerequisites, validation status): [macOS](docs/platforms/macos/README.md), [Windows](docs/platforms/windows/README.md), [Linux](docs/platforms/linux/README.md). Per-role commands (ROOT/CPU, CUDA, ROCm, MPS, dev), the worker device policy (`ACR_WORKER_POLICY`, `ACR_CPU_DEFAULT_MODELS`, default: a MacBook Air stays in the CPU pool unless `ACR_WORKER_POLICY=accelerator`) and the external prerequisites are in pip-requirements.md §4.4-4.5. *Installed* does not mean *hardware-validated*: see pip-requirements.md §10.

```bash
git clone <repo-url> && cd adaptive-runtime
python scripts/bootstrap_env.py --dry-run     # shows the selected profile, why, and the exact pip commands (changes nothing)
python scripts/bootstrap_env.py --dev         # creates .venv with the right torch build + test tools, installs the project with --no-deps, verifies
source .venv/bin/activate                     # Windows: .\.venv\Scripts\Activate.ps1   (console scripts: `acr` and `vayrm`)

# ROOT (control plane + storage gateway + terminal dashboard)
vayrm root                          # optional: --host 0.0.0.0 --port <p>; needs Tailscale
# or:  ACR_ROLE=root acr root

# worker
vayrm node --root <root-ip>:<root-port>
# or:  ACR_ROLE=node ACR_ROOT_HOST=<ip> ACR_ROOT_CONTROL_PORT=<port> acr node

acr diagnostics | vayrm diagnostics | vayrm doctor     # environment / state checks
python -m acr.node.preflight --role cpu|cuda|rocm|mps|root|dev   # role-aware venv/package/device/policy check; exit 1 = cannot serve that role
```

### Vayrm Python API

`vayrm` is the distribution name in [pyproject.toml](pyproject.toml) (console scripts `vayrm` and `acr`); it is installed from this repository as shown above, and whether a PyPI release exists is not claimed here. Importing `vayrm` sets `USE_PALADIN=1`. Public symbols (`vayrm.__all__`): the thread wrappers `Root`, `Node`, `SubRoot` from [vayrm/public.py](vayrm/public.py), the launch modules `root`, `node`, `subroot` ([vayrm/core/](vayrm/core/)), and the namespaces `scheduler`, `runtime`, `telemetry`, `network`.

```python
import vayrm

# Same behaviour as `vayrm root --host H --port P` / `vayrm node --root IP:PORT [--gpu]` (needs Tailscale).
vayrm.Root(host="0.0.0.0", port=0).start()                       # daemon thread; returns immediately
n = vayrm.Node(root="<tailscale-ip>:<root-port>", interval=1.0, gpu=False)
n.start()
vayrm.SubRoot(root="<tailscale-ip>:<root-port>", host=None, port=None, interval=1.0).start()

n.join(timeout=5)                    # then: n.is_alive(), n.error (exception or SystemExit), n.exit_code
```

`start()` runs `vayrm.core.<role>.start(...)` on a **daemon thread**, so the calling process must stay alive (e.g. `threading.Event().wait()`); there is no `stop()`. A launch function that raises or calls `sys.exit` ends only that thread: the wrapper records it in `.error` / `.exit_code` (a refused launch is not an exception in the caller, so check them after `join()`). `local` and other extra keyword arguments are forwarded to the launch function, which accepts and ignores them. `root=` is optional for `Node`/`SubRoot` (a warning is printed if neither it nor `ACR_ROOT_HOST` is set), but an explicit one must be a Tailscale address: loopback, LAN and wildcard hosts make the launch function exit with status 1.

**`gpu=True` / `vayrm node --gpu`** is the same pre-join gate as `acr node --gpu` (`acr.node.gpu_mode.prepare_gpu_worker`; Windows NVIDIA workers only): it runs before any identity or channel exists, and on any refusal the worker exits with status 2, never joins and never falls back to CPU. On macOS/Linux it refuses (`GPU_FLAG_UNSUPPORTED_PLATFORM`); a MacBook Air uses the CPU pool by default, and Apple GPU (MPS) work needs `ACR_WORKER_POLICY=accelerator`. Vayrm adds no scheduler or device policy of its own: placement, device selection and GPU-requirement enforcement are ROOT/NodeBrain's. Evidence: `tests/test_vayrm_public_api.py` and `tests/test_vayrm_gpu_bridge.py` (mocked policy, nothing started); end-to-end MPS execution through ROOT -> NodeAgent -> trainer is `tests/e2e/test_mps_training_e2e.py` (real MPS execution on Apple Silicon, with a *simulated* capability report as scheduler input). What each entry relies on: [docs/ENDPOINTS.md §12](docs/ENDPOINTS.md); the standalone experimental FastAPI app (`acr/api/server.py`, not started by Vayrm): [docs/ENDPOINTS.md §8.2.1](docs/ENDPOINTS.md). Release history: [CHANGELOG.md](CHANGELOG.md).

### GPU workers: entry points, gates and platform status

A GPU is reported *execution-ready* only after four separate checks pass in the interpreter that will train: (1) NVIDIA hardware seen, (2) a CUDA build of PyTorch, (3) CUDA runtime initialises, (4) a real allocation/kernel executes. `--gpu` prints these four stages and refuses (exit 2, "did NOT join the network") at the first failure; an installed driver or a visible adapter alone is never enough, and an NVIDIA card with a CPU-only PyTorch stays a CPU worker. Drivers are never installed or changed by ACR, and no command here upgrades PyTorch: pick the build for your driver from the [official selector](https://pytorch.org/get-started/locally/) and create the environment with `python scripts/bootstrap_env.py --dry-run` (shows the exact commands, changes nothing) or `pip install -r requirements/<profile>.txt` ([pip-requirements.md](pip-requirements.md)).

| Entry point | Gate / policy | Notes |
|---|---|---|
| `python -m acr.entrypoint --role node --gpu` | `acr.node.gpu_mode.prepare_gpu_worker`, before any config, identity or registration | **Windows NVIDIA workers only**; any other OS refuses with `GPU_FLAG_UNSUPPORTED_PLATFORM` |
| `acr node --gpu` | same gate (`acr/cli.py`) | same scope; this console-script form did not accept `--gpu` before |
| `vayrm node --gpu`, `vayrm.Node(gpu=True)` | same gate, called first (`vayrm/core/node.py`) | Vayrm adds no policy of its own |
| `ACR_WORKER_POLICY=accelerator\|cpu\|auto` | worker policy may only *restrict* validated devices | a MacBook Air stays in the CPU pool unless `accelerator` |
| `ACR_TRAINING_REQUIRE_GPU=cuda\|mps\|any` (ROOT) | binding requirement stamped on each assignment; a worker without the backend refuses with `gpu_requirement_unsatisfied` and ROOT blocks the session — never a silent CPU run | optional accelerators (no requirement) are scoring preferences |

Validate a node before joining: `python -m acr.node.preflight --role cuda|mps|cpu` (exit 1 = cannot serve that role), and on NVIDIA hosts compare `nvidia-smi` with `python -c "import torch;print(torch.__version__, torch.version.cuda, torch.cuda.is_available())"`. A CPU-only `torch.version.cuda` of `None` with a visible NVIDIA GPU means the wrong wheel is installed in that venv.

| Platform | Status | Evidence |
|---|---|---|
| Apple MPS (M1 MacBook Air) | **PASS** | live probe validated `mps` by a real allocation; real training on `mps:0` through ROOT → NodeAgent → trainer; required-MPS without a usable MPS blocks (`tests/e2e/test_mps_training_e2e.py`, `tests/test_acr_api_server.py`). The e2e scheduler input is a *simulated* capability report. |
| CPU | **PASS** | same e2e files; independent of an available accelerator |
| Windows NVIDIA CUDA | **NOT TESTED** in the latest validation | gate and policy are covered by mocked tests (`tests/node/test_gpu_mode.py`); earlier real-hardware records are in the capability table above and `reports/`, not re-run here |
| Linux NVIDIA CUDA, AMD ROCm, NCCL collectives, multi-GPU, cross-machine/Tailscale | **NOT TESTED** | no such environment available; adapters and policy exist but are unverified on hardware |

### Releasing Vayrm

`pyproject.toml` is the only version source (currently `1.1.0`); the newest `CHANGELOG.md` section must name the same version (`tests/test_release_metadata.py`). **Nothing has been tagged or published**: do not assume `pip install vayrm==1.1.0` works until the package is actually on the index and verified there.

**Standard path - GitHub Actions** (`.github/workflows/publish.yml`): push a tag `vX.Y.Z` -> the build job installs the committed lock (`uv sync --frozen --all-extras`), runs `pytest tests/`, builds the wheel and sdist, checks that the tag equals the package version and that both artifacts report it, runs `twine check` -> the publish job (tag pushes only; a manual dispatch from a branch builds and tests but never publishes) uploads with **PyPI Trusted Publishing** (OIDC, no token). It needs, outside this repository: a trusted publisher on PyPI naming this repository, the workflow `publish.yml` and the environment `pypi`, and - to require a human decision before production uploads - required reviewers on the GitHub environment `pypi`. Neither setting can be verified from the code; both are **unverified**. **Billing caveat:** when the account's Actions billing is blocked, every job fails in about two seconds with zero steps and a billing annotation; none of the workflow's checks ran, so that is "unverified", not a pass, and not a code failure.

**Local validation (works without hosted runners; never publishes, tags, commits or edits tracked files):**

```bash
python scripts/release_check.py --version 1.1.0 --out-dir /path/outside/the/repo     # validates the commit HEAD points to
```

It checks the commit and version (pyproject + changelog), `uv lock --check`, installs with the workflow's exact `uv sync --frozen --all-extras` into a **private** environment (never `.venv`), runs the mandatory tests (release metadata and scripts, Vayrm public API and GPU bridge, endpoint contract/provenance, API server), builds the sdist and wheel from a `git archive` of the commit, rebuilds the wheel from the sdist and compares contents, verifies every artifact reports the version, runs `twine check`, and verifies the required modules and the `acr`/`vayrm` entry points. Each line is `PASS`, `FAIL`, `BLOCKED` (no disk/network/tool) or `NOT TESTED`; exit 0 only if every mandatory check passes (1 = failure, 2 = a mandatory check was blocked). `--smoke-install` additionally installs the wheel into a fresh environment. The result and the SHA-256 of the exact artifacts go to `release-manifest.json`. A local run on one machine says nothing about Windows, Linux, CUDA, ROCm, NCCL, multi-GPU or cross-machine behaviour (the script prints these as `NOT TESTED`) and is not a hosted-CI result.

**Manual publication fallback (a deliberate operator decision; separate from validation):** only after (1) a clean, reviewed release commit is checked out and pushed, (2) `release_check.py` ended `ALL MANDATORY CHECKS PASS` for that commit, (3) the changelog and release notes were reviewed, run

```bash
python scripts/release_publish.py --manifest /path/outside/the/repo/release-manifest.json --repository pypi            # verification only
python scripts/release_publish.py --manifest /path/outside/the/repo/release-manifest.json --repository pypi --upload   # asks for a typed confirmation, then a token
```

The first form uploads nothing. Before any upload the script re-hashes the artifacts against the manifest (it will not upload anything that differs from what was validated), requires `HEAD` to be the validated commit with clean release paths, and confirms from the index that the version is not already published (a published version can never be re-uploaded; if the index cannot be reached it refuses). `--repository` has no default (`testpypi` is available but is never used implicitly). The token is a **project-scoped** PyPI API token (PyPI offers only account-scoped tokens for a project's first upload: replace it with a project-scoped one immediately afterwards), entered at a hidden prompt or read from the environment variable `VAYRM_PYPI_TOKEN`; it is passed to `twine` through the child process environment only, never on a command line, never written to a file and scrubbed from any output. Never put a token in source, scripts, documentation, workflow YAML or a committed environment file. This fallback does not run automatically when the Actions publication fails, and neither path creates the Git tag: tagging and any GitHub release are separate manual steps.

Checkpoints: resumable state is written to `checkpoints/<session>[/<trial>]/checkpoint-<step>/` (each trial has its own; `results/` holds only the session record and metrics). `ACR_CHECKPOINT_EVERY_STEPS=N` saves every N optimizer steps in addition to the per-epoch and final saves (0/unset = per epoch + end; a non-integer or negative value blocks the start with a visible reason), `ACR_CHECKPOINT_KEEP` bounds how many are retained (default 2); see [docs/TRAINING_SESSIONS.md](docs/TRAINING_SESSIONS.md) §8.

Scheduling classes: ROOT classifies each worker `accelerator` / `cpu` / `unknown` from its validated capability report (never from the OS, model name or raw GPU telemetry; `unknown` workers are CPU-only) and each job `accelerator` / `cpu` / `any`; GPU-required work is refused explicitly rather than run on a CPU, and CPU workers never join a GPU DDP/FSDP group (docs/TRAINING_SESSIONS.md §6c, docs/DISTRIBUTED_EXECUTION.md).

Dashboard keys (dataset `d`, start/pause/resume `t` + Enter, reconcile `r`, registry `g`, quit `q`) are listed under [Root Dashboard Controls](#root-dashboard-controls); there is no HTTP API for training. All configuration is environment variables ([docs/configuration.md](docs/configuration.md)).

Tests (local, no network access required; Tailscale tests skip when Tailscale is not connected):

```bash
.venv/bin/python -m pytest -q tests -p no:cacheprovider
.venv/bin/python -m pytest -q tests/test_endpoints_contract.py   # endpoint doc <-> code contract
```

Known failing tests (not regressions of documentation): the checkpoint-lineage, recovery-limit and related files noted above fail because their implementation is not in this tree (26 under `tests/training`, 19 under `tests/cluster` at the time of writing), and `tests/cluster/test_rendezvous_port.py` cannot be collected.

## Feature status

| Status | Features |
|---|---|
| **Implemented and exercised by tests** (in committed code) | ROOT/NodeBrain over the PALADIN transport, registration/auth/heartbeats; scheduling with explainable scoring and hard feasibility; Storage Gateway + Dev Storage; model/dataset registry; persistent multi-trial sessions; DDP (`DATA_PARALLEL`), FSDP on `cpu`/gloo, `SINGLE_NODE`, `LOW_COMMUNICATION`; the PREPARE/READY/START handshake; node memory admission (host RAM / one device) |
| **Implemented in the working tree, not yet committed** | worker runtime preflight (`USE_TF=0`, Keras/TF clash detection), recovery decision layer, durable session/attempt journal, dispatch retry, GPU requirement propagation and per-device admission (section above), `ACR_RECOVERY_POLICY` |
| **Implemented in the working tree, tested on CPU only** | `causal_lm` trainer (`acr/training/causal_lm.py`, [docs/CAUSAL_LM_TRAINER.md](docs/CAUSAL_LM_TRAINER.md)): next-token training of Hugging Face decoders (GPT-2) on a text column, single node. Unit/integration tests use a tiny GPT-2 on CPU. The input pipeline is reusable (`acr/training/data_pipeline.py`: sharded, integrity-checked token cache keyed by dataset revision/tokenizer/preprocessing version; bounded prefetch; deterministic batch plan), the trainer has configurable AdamW/schedule, a once-only held-out test evaluation, opt-in validation-based best-model selection and early stopping (single node), and the exported `best_model_uri` is only a *selected* model when selection ran. Measured here: shard cache 1.12x faster than the old tokenizer cold and 54x warm; a tokenizer worker pool and prefetch gave **no** speed-up (benchmarks in the trainer doc). One 20-step local CPU smoke run of the real GPT-2/WikiText-103 completed (see the trainer doc); no remote worker, accelerator (MPS/CUDA) or distributed run has been verified |
| **Simulated only** | all scheduling-policy comparisons (HRRN, size-aware, greedy baselines), the Alibaba `cluster-trace-gpu-v2026` replays, virtual CUDA/MPS worker matrices, fault-injection against ROOT in-process fixtures - see [docs/HETEROGENEOUS_SIMULATION.md](docs/HETEROGENEOUS_SIMULATION.md) |
| **Experimental / off by default** | `ACR_TOPOLOGY_MODE=adaptive`, `ACR_NODE_MODE`, `ACR_POLICY_MODE`, `ACR_SHADOW_SCHEDULING`, `ACR_RECOVERY_POLICY` |
| **Present but inert** | `acr/training/async_checkpoint.py` (`ACR_ASYNC_CHECKPOINT`) is not wired into the training loop |
| **Not implemented** | checkpoint lineage / lineage-safe resume, no-progress recovery cap (`ACR_MAX_RECOVERIES_WITHOUT_PROGRESS`), `ACR_LEGACY_CHECKPOINTS`; HSDP, TP, PP, CP and hybrids (`UNSUPPORTED` with reason codes); HTTP training/control API; authentication on the storage gateway, discovery and dashboard HTTP APIs |
| **Hardware-unverified** | CUDA training, the NCCL/FSDP-on-CUDA path, NVIDIA device-initialisation behaviour, multi-GPU admission, a real heterogeneous (macOS MPS + Windows/Linux CUDA) cluster through a live ROOT, real Tailscale between machines. Everything about these is design plus virtual-worker evidence, not measurement |

**Limitations in one place.** One training session runs at a time (no live multi-job scheduler); ROOT never relays tensors or checkpoints; control and storage endpoints are unauthenticated and bind `0.0.0.0` by default ([docs/ENDPOINTS.md §9](docs/ENDPOINTS.md)); performance claims for the closed-loop decision layer are not made until baselines and ablations are measured.

## Current Implementation Summary

> The authoritative endpoint/message contract is **[docs/ENDPOINTS.md](docs/ENDPOINTS.md)**.
> It is checked against the code by `tests/test_endpoints_contract.py`.

| Component | Role (current code) |
|---|---|
| **ROOT** (`acr/cluster/`, `paladin/apps/root/`) | Control plane: node registry over the PALADIN transport, scheduling, the training session/attempt orchestrator, the model/dataset catalog, the terminal dashboard. ROOT carries only small control messages; it never relays tensors, checkpoints or datasets. |
| **NodeBrain / NodeAgent** (`acr/node/`) | Per-node intelligence: measures its own CPU/RAM/GPU/VRAM, validates and runs training assignments, reports state, checkpoints and failures to ROOT. |
| **Dev Storage / Storage Gateway** (`acr/storage/`) | Persistence plane. ROOT hosts an HTTP storage API (`/health`, `/manifest`, `/list/<ns>`, `GET/PUT/DELETE/HEAD /<ns>/<key>`) over Dev Storage for `datasets`, `models`, `checkpoints`, `artifacts`, `experiments`, `results`. Resources are addressed by logical URIs (`models/<name>`, `datasets/<name>`), never local paths. |
| **Peer / data plane** (`acr/communication/peer_transfer.py`) | Optional node↔node transfer of checkpoints, dataset/model files and model deltas (`STAT/LIST/READ/DELTA`, HMAC-authenticated). Disabled unless `ACR_PEER_SECRET` or `ACR_PEER_SECRET_FILE` is provisioned. |
| **PALADIN** (`paladin/`) | Transport/identity layer under ACR: node identity, registration/authentication, heartbeats, `COMM_JSON` messaging, and its own small dashboard HTTP APIs. |
| **Vayrm** (`vayrm/`) | Launcher/facade: `vayrm root`, `vayrm node --root <ip:port>`, `vayrm subroot`, `vayrm diagnostics|doctor`, plus re-exports of `acr.*`. It does **not** call PALADIN over HTTP. It requires Tailscale, then starts `ACRRoot`/`ACRNode` and wires the storage endpoint (ROOT) and `PeerConnectionManager` + peer data plane (Node) exactly as `acr.entrypoint` does. |

**Model / dataset flow.** ROOT discovers resources from Storage into a versioned registry (state, fingerprint, compatibility). Training is started from the ROOT dashboard; the session pins resource versions/fingerprints and sends logical URIs only. NodeBrain resolves them from Storage (or from a peer that already holds them) and verifies the bytes against the pin.

**Training flow.** `training_session_id` (wire name `session_id` on assignments) is stable for a session; every attempt gets a new `execution_attempt_id` (`<session>-a<N>`). ROOT sends `PREPARE` → node answers `READY` → ROOT sends `TRAINING_START` once every attempt is ready → node reports progress, `CHECKPOINTED`, `INTERRUPTED`, `FAILED`/`REJECTED` (with structured `reason_code`) or `COMPLETED`. Checkpoints are published to `checkpoints/<session>[/<trial>]/…/checkpoint-<step>` (the manifest records the topology that wrote it) and restored from a new attempt after recovery or migration. Execution policies: `SINGLE_NODE`, `DATA_PARALLEL`, `FSDP`, `LOW_COMMUNICATION`. Full sequence and fields: [docs/ENDPOINTS.md §6](docs/ENDPOINTS.md).

**Persistent multi-trial sessions and adaptive parallelism.** `t` starts a persistent session: one `training_session_id`, many trials (`trial_001 … trial_N`, default budget 30, `ACR_SESSION_MAX_TRIALS`). ROOT (`acr/training/session_manager.py`, `acr/cluster/trial_controller.py`) decides hyperparameters, node allocation and — optionally — the parallelism topology of each trial, prunes plateaued trials, and keeps three independent winners (BEST QUALITY / EFFICIENCY / BALANCED, each pointing at the checkpoint and model artifact of the trial that produced it). NodeBrain still does all the computing; ROOT never touches a tensor. The session artifact is written into Dev Storage under `results/<session_id>/training_session/`. Topology control is explicit: `ACR_TOPOLOGY_MODE=planner` (default — today's behaviour, nothing forced), `adaptive` (capability-aware, memory-constrained, measurement-driven selection) or `fixed:<DDP|FSDP|SINGLE>` (the baselines).

| Strategy | Support state |
|---|---|
| DDP (`DATA_PARALLEL`) | SUPPORTED (baseline) |
| FSDP (PyTorch FSDP2) | SUPPORTED on `cpu`/gloo — verified with real multi-process runs; the `cuda`/nccl path is implemented but **not verified on hardware**; an Apple-silicon node serves FSDP on its CPU rank (MPS never joins a gloo group); `UNSUPPORTED reason=device_backend_unsupported` only when the rank's device has no collective path |
| FSDP CPU offload | **Explicit opt-in** (`ACR_FSDP_CPU_OFFLOAD=1` in the *worker's* environment; default off). Applied only to a CUDA rank (FSDP2 `CPUOffloadPolicy`: parameters, gradients and the optimizer step move to the CPU — a memory-for-speed trade, not a scheduling fix); on a CPU rank it is recorded as `not applicable`. The CUDA offload path is **not hardware-verified**; the evidence field is `cpu_offload{requested,applied,note}` |
| CPU task pool (input pipeline) | **Opt-in, off by default** (`ACR_CPU_TASKS=1`; `ACR_CPU_TASK_WORKERS`, `ACR_CPU_TASK_QUEUE`, `ACR_CPU_TASK_TIMEOUT_S`, `ACR_CPU_TASK_THREADS`): the tabular trainer's Arrow→numpy encode step runs in bounded worker processes (`acr/training/cpu_tasks.py`) with backpressure, timeouts, cancellation, duplicate/stale-result protection and an inline fallback; the model it trains is bit-identical to the synchronous pipeline (tested). **Measured on the MacBook Air (60k rows, 6 runs): no speed-up for this workload** — baseline 2.58 s vs pool 3.14 s median — because encoding costs ~0.5 ms/batch; it pays only for heavier CPU-side work. Not wired into the causal-LM tokenizer (profiled: tokenization is ≈ 0.3 % of CPU training cost per token, nothing to offload). Evaluation passes are cached (`ACR_EVAL_CACHE_MB`, default 64, `0` = off): −18 % wall time, identical digests. GPU assistance (`ACR_GPU_ASSIST`) is a decision policy only (`decide_gpu_assist`), **not wired into training and not hardware-validated** |
| CPU intake endpoint | **Opt-in, off by default** (`ACR_CPU_INTAKE=1` + authentication: **per-node credentials** — `python -m acr.cluster.cpu_intake_auth provision --node N --out-dir D`, then `ACR_CPU_INTAKE_CREDENTIAL_FILE` on the worker — or, legacy, the shared `ACR_CPU_INTAKE_SECRET[_FILE]`; dynamic port, advertised in `RootAnnouncement.cpu_intake_port`): CPU-oriented workers register their reported capabilities, lease allow-listed operations (`data.sha256`, `tabular.encode_batch`, `aux.standardize`), and return validated results; nodes holding a training rank are never leased work. HMAC-signed requests bound to the credential's node id, size limits, leases/retries/idempotency/generation checks; workers run tasks in killable child processes beside an independent heartbeat (deadline/cancel/lease loss ⇒ process killed, no late result). **Transport:** TLS 1.2+ with certificate and hostname verification (optional mutual TLS); `ACR_CPU_INTAKE_TLS=auto` (default) serves/uses plaintext only on loopback and **fails closed otherwise** — an existing intake on a non-loopback address needs `ACR_CPU_INTAKE_TLS_CERT/KEY` on ROOT and `ACR_CPU_INTAKE_TLS_CA` on workers (or an explicit `ACR_CPU_INTAKE_TLS=off` on both); provisioning, rotation and migration steps in [docs/ENDPOINTS.md §2.4](docs/ENDPOINTS.md). **Credential migration:** an existing shared-secret deployment keeps working unchanged (reported as `shared_legacy`, with a warning); to move, `migrate --nodes a,b,c`, distribute the files, set `ACR_CPU_INTAKE_ALLOW_SHARED_SECRET=1` during the window, then retire the shared secret. CPU-only setup: provision a credential per CPU worker, start ROOT and workers with `ACR_CPU_INTAKE=1`; no CUDA/NCCL/GPU telemetry needed. Routes, schemas and errors: [docs/ENDPOINTS.md §2.4](docs/ENDPOINTS.md). Verified with CPU-only tests; nothing feeds it automatically from training yet |
| CPU→GPU assist | **Implemented through the existing CPU intake; opt-in on both sides; real execution verified on ONE backend only (NVIDIA CUDA, native Windows, RTX 2050).** ROOT: `ACR_CPU_INTAKE=1` + `ACR_GPU_ASSIST=1`; worker: `ACR_GPU_ASSIST_WORKER=1`. `AuxRequest`→`plan_aux`→`AuxBroker` (`acr/training/gpu_assist_contract.py`) plan a task on operation support, a registered self-tested accelerator, **measured** headroom (heartbeat), memory reserve, link bandwidth and transfer-adjusted benefit (unknown ⇒ CPU); a gpu-planned task can be leased only by a qualified accelerator worker (`acr/node/accel_backend.py`: CUDA / ROCm / MPS adapters), falls back to a CPU task after a short window, on failure or on any refused result, and ROOT records where it **really** ran (`executed_on`) and spot-checks accelerator results against the CPU reference. Only `aux.standardize` has an accelerator implementation. ROCm and MPS adapters are implemented but **unverified (no hardware)**; no training stage submits auxiliary tasks yet. Measured end to end, the GPU path was **not faster** than the CPU worker (wire/validation overhead dominates; see docs/DISTRIBUTED_EXECUTION.md) — no speed-up is claimed. Setup: docs/ENDPOINTS.md §2.4.1 |
| HSDP | UNSUPPORTED — `execution_adapter_missing` (needs a 2-D DeviceMesh, ≥ 4 ranks, a shard-group size on the wire) |
| TP | UNSUPPORTED — `execution_adapter_missing` |
| PP | UNSUPPORTED — `pipeline_partitioner_missing` |
| CP | UNSUPPORTED — `model_not_compatible` (the tabular MLP has no sequence dimension) |
| hybrids (FSDP+TP, …) | UNSUPPORTED — `composition_adapter_missing` |

Also implemented (opt-in via `ACR_TOPOLOGY_MODE=adaptive`, `ACR_NODE_MODE`, `ACR_POLICY_MODE`): node count chosen by the *marginal utility* of each added node; gradient accumulation (SINGLE/DDP) and checkpoint-interval policies reacting to observed or forecast memory pressure; forecasts stored with horizon, actual and error; measured exposed FSDP all-gather/reduce-scatter time. Activation checkpointing and overlap efficiency are UNSUPPORTED/not measured. Status of every capability (IMPLEMENTED / VERIFIED / MEASURED / HYPOTHESIS / UNSUPPORTED): [docs/TRAINING_SESSIONS.md §1](docs/TRAINING_SESSIONS.md).

DDP, FSDP, TP, PP, CP, hyperparameter search, early stopping and checkpointing are established techniques; what this runtime adds is the closed-loop decision layer, and **no performance or novelty claim is made** until fixed-strategy baselines and ablations have been measured (see [docs/TRAINING_SESSIONS.md](docs/TRAINING_SESSIONS.md) §10–11 and [docs/research/ADAPTIVE_PARALLELISM_RESEARCH.md](docs/research/ADAPTIVE_PARALLELISM_RESEARCH.md)). **No new HTTP endpoint**: see [docs/ENDPOINTS.md §13](docs/ENDPOINTS.md).

**Validation status (precise).**

| Level | Status |
|---|---|
| Unit / integration / in-process end-to-end (`pytest -q tests`) | Passing locally (see the test run recorded with each change) |
| Separate processes over loopback (`scripts/real_network_validation.py`; real `vayrm root` + `vayrm node` incl. `RootAnnouncement` → node storage endpoint → logical `models/`/`datasets/` inventory) | Validated locally (loopback) |
| Real Tailscale between machines | **Tailscale validation pending** (Tailscale integration tests skip when Tailscale is not connected) |
| Physical multi-machine / heterogeneous hardware | **Not yet validated** |

**Known security limitations (hardening backlog; not production-secure networking).** The Storage Gateway, the discovery locator (`GET /discovery`) and PALADIN's dashboard APIs have no application-level authentication, and `acr.entrypoint` (and the Storage Gateway under `vayrm root`) bind `0.0.0.0` by default; reachability is limited only by the network/host firewall. See [docs/ENDPOINTS.md §9](docs/ENDPOINTS.md).

------------------------------------------------------------------------

**Recovery, heterogeneity and simulation (2026-10-09).** After a rank loss ROOT tears the gang down and rebuilds it under *fresh* attempt ids and a fresh rendezvous; each NodeBrain fences its earlier attempt before the replacement reports `READY`, and ROOT drops late messages of superseded attempts. Checkpoints keep their topology metadata when fetched from a peer, FSDP ranks agree on the restore mode, and NodeBrains report liveness well inside ROOT's stale window. The topology selector builds each DDP/FSDP/SINGLE candidate from the nodes that report the strategy and fit it (a node that cannot run collectives no longer blocks the others) and every trial records which live-report change altered the plan (`plan_explanation`). The simulator (`acr/simulation/`) replays GPU-cluster traces (Alibaba `cluster-trace-gpu-v2026` subset) through the current `plan_distribution` with deterministic failure schedules; all of it is SIMULATED. Details and the evidence for every claim: `docs/RESOURCE_OPTIMIZATION.md` §6-7, `docs/HETEROGENEOUS_SIMULATION.md` and `reports/paladin_remaining_fixes_and_benchmarks.md`.

**Bounded recovery and experimental aids - what is actually in the code.** (Corrected 2026-10-13: earlier text here described *checkpoint lineage* - `lineage_id`, lineage-safe resume, `ACR_MAX_RECOVERIES_WITHOUT_PROGRESS`, `ACR_LEGACY_CHECKPOINTS` and the `CHECKPOINTS_SKIPPED` / `CHECKPOINT_LEGACY_ACCEPTED` / `OPTIMIZER_NOT_RESTORED` events. **None of that exists in this source tree**: the implementation was held in a local git stash and only its failing tests are present (`tests/training/test_checkpoint_lineage.py`, `tests/cluster/test_recovery_limit.py`, ...). Treat lineage-safe resume and the no-progress recovery cap as **NOT IMPLEMENTED** here.) What the working tree does contain, all uncommitted at the time of writing (see [Feature status](#feature-status)): a recovery decision layer (`acr/training/recovery_decision.py`: deterministic failures are not retried on the same node/pair, bounded retries with backoff, per-session worker exclusion), a durable session/attempt journal (`acr/training/session_store.py`, `trial_record.py`), the worker runtime preflight and the GPU admission path below. Feature flags (all **off by default**):

| Flag | Status | Effect |
|---|---|---|
| `ACR_RECOVERY_POLICY=1` | experimental, uncommitted module | ROOT consults `acr/training/recovery_policy.py` from `interrupt()`: may block with a reason code or restart from scratch; capacity verdicts are advisory only |
| `ACR_ASYNC_CHECKPOINT=1` (`ACR_ASYNC_CHECKPOINT_MAX_BYTES`) | **module present but not wired into the training loop** (nothing imports `acr/training/async_checkpoint.py`) | no effect today; do not rely on it |
| `ACR_SHADOW_SCHEDULING=1` (`ACR_SHADOW_POLICY`, default `root_hrrn`) | experimental | read-only: records what a replay policy would have chosen next to ROOT's real placement and the assumptions behind it |

**GPU requirements and per-device admission (2026-10-13; working tree, uncommitted, simulation-verified only).** A placement request can now carry the accelerator it needs end to end: `WorkloadRequest.gpu_backend` (`cuda`/`mps`), `gpu_count` and `gpu_memory_per_gpu_bytes` (or, if silent, they are derived from the training spec: an operator-fixed `ACR_TRAINING_DEVICE=cuda|mps`, `required_gpu`, `required_gpu_count`, `required_gpu_memory`). ROOT filters and scores with them (backend-agnostic GPU work is no longer accepted by a worker whose PyTorch runtime is CPU; VRAM is compared per device; busy GPUs and GPUs that report no VRAM utilisation are not read as idle), stamps `required_gpu*` onto each assignment, and the node refuses at PREPARE what it cannot run: the preflight initialises every needed device and checks device count and per-device free VRAM, and memory admission reserves per device. Reason codes and fields: [docs/ENDPOINTS.md §6](docs/ENDPOINTS.md); variables: [docs/configuration.md](docs/configuration.md); design and measurements: [docs/RESOURCE_OPTIMIZATION.md §8](docs/RESOURCE_OPTIMIZATION.md) and [docs/HETEROGENEOUS_SIMULATION.md](docs/HETEROGENEOUS_SIMULATION.md). **Verified only against virtual workers and unit tests; no real CUDA device, NCCL run or heterogeneous real-worker run has validated it.**

**GPU requirement enforcement, accelerator report, session timeline (2026-10-13).** A GPU requirement is now *binding* end to end: `ACR_TRAINING_REQUIRE_GPU=cuda|mps|any` marks a session's accelerator as required (optionally `ACR_TRAINING_GPU_FALLBACK=cpu` for optional acceleration, recorded, never silent). The trainer's device policy honours it (previously a CPU-only PyTorch on an NVIDIA worker silently trained on CPU), ROOT's `required_gpu*` stamp now survives into the assignment, and a worker that cannot comply refuses with `gpu_requirement_unsatisfied`; ROOT blocks the session once every eligible worker refused. Nodes publish a normalized accelerator report (`hardware_capability["accelerators"]`: staged CUDA checks, per-device VRAM, explicit `AVAILABLE/UNAVAILABLE/UNKNOWN/NOT_APPLICABLE/UNVERIFIED` states), ROOT records a correlated session timeline and persists `artifacts/sessions/<id>/…` with read-back verification. **Hardware-verified only for a single Windows worker with an RTX 3050 6 GB (torch 2.14.1+cu130). NCCL, MPS, multi-GPU and multi-node execution are not verified.** Details, limits and the verification matrix: [docs/GPU_VALIDATION.md](docs/GPU_VALIDATION.md); evidence: [reports/paladin_validation/](reports/paladin_validation/).

**Scheduling evidence and what it does not show.** Queue-ordering policies (HRRN, size-aware) and the lower-bound study are **simulation-only** (Alibaba `cluster-trace-gpu-v2026` slices on assumed virtual clusters). ROOT runs one training session at a time, so no multi-job scheduling policy is live. In the simulation HRRN-ordered policies lower mean completion time and max stretch against the memory-safe `greedy_mem` baseline while p95 and makespan are mixed, `root` placement alone adds little over a speed-ordered greedy, the original `greedy` baseline over-commits memory, and policies sit 24-32 % above the tightest valid makespan lower bound (a gap to a lower bound is not distance to the optimum). Consolidated, source-linked comparison with all caveats: **[research/greedy_vs_paladin_vs_optimality.md](research/greedy_vs_paladin_vs_optimality.md)**. Latest reports: runtime recovery (`reports/paladin_runtime_recovery_20261010T1529.md`, local, not committed), training resilience (`reports/paladin_training_resilience_20261010T1432.md`, local, not committed), fair evaluation (`reports/paladin_fair_eval_20261010T1313.md`, local, not committed), makespan bounds (`reports/paladin_makespan_bounds_20261010T1343.md`, local, not committed).

## Root Dashboard Controls

The Root node exposes a live terminal dashboard (`acr/ui/root_dashboard.py`)
over the authoritative `RootBrain` / `TrainingOrchestrator` state. It has no
HTTP API of its own and no separate scheduler or training state.

| Key | Action |
|---|---|
| `d` | Focus/unfocus the datasets list; `j`/`k` move, `Enter` selects the dataset (`t` never guesses a dataset). |
| `t` then `Enter` | Context-dependent: no session or a finished one → start training (needs a resolved storage endpoint and a selected dataset); an active session → pause it (checkpoint retained); a paused session → resume it (same `training_session_id`, new `execution_attempt_id`). |
| `r` then `Enter` | Reconcile storage, catalog, nodes, sessions and attempts. Does not reset training progress. |
| `g` | Registry lifecycle mode for models/datasets (`i` info, `v` validate, `x` retire, `z` quarantine, `u` restore; each dangerous operation needs `Enter` to confirm, any other key cancels). |
| `v` | Validate the selected dataset/model (dry-run report). |
| `j` / `k` / `c` | Select next/previous worker; clear selection. |
| `q` | Stop the dashboard. |

`s` and `m` are not supported commands (they are rejected).
Set `ACR_DASHBOARD=0` to disable the dashboard.

### Training session states

`QUEUED → PREPARING → READY → STARTING → RUNNING`, with `RECOVERING` after an
interruption, `PAUSED` (operator) and the terminal states `COMPLETED` / `FAILED`.

### Checkpoint & resume

The node's training loop writes `checkpoints/<session_id>/checkpoint-<step>`
and uploads it through the Storage Gateway; ROOT records the logical
`checkpoint_uri`. On critical stop, migration or failure a **new** attempt is
dispatched with that `checkpoint_uri` (and `checkpoint_peers`, so a peer that
already holds it can serve it before Dev Storage). The session id never changes.
A checkpoint is published atomically (written to `checkpoint-<step>.tmp`, per-file sha256 manifest, verified, then renamed), so a torn or pending write is never selected. The resumed attempt restores model, optimizer, step/epoch/batch position and the torch/numpy/python RNG state; a resume that cannot restore optimizer state is reported as weights-only, and a per-rank-sharded checkpoint under another layout is refused (resharding is not implemented).

------------------------------------------------------------------------

## 1. System Architecture

PALADIN is intentionally separated into control, compute, training-data,
persistence, and network/security planes.

``` mermaid
flowchart TB
    TS["TAILSCALE<br/>Production Network / Security Layer"]

    ROOT["ROOT<br/>Control Plane"]
    RB["RootBrain<br/>Cluster State"]
    SCH["Adaptive Scheduler"]
    TC["Training Coordinator"]
    DASH["Dashboard / API"]
    EP["Dynamic Endpoint Registry"]

    A["NODE A<br/>NodeBrain"]
    B["NODE B<br/>NodeBrain"]
    C["NODE C<br/>NodeBrain"]

    P2P["NODE ↔ NODE<br/>Training Data Plane"]
    STORAGE["Storage API<br/>Persistence Plane"]
    OMV["Persistent Storage / OMV"]

    TS --> ROOT
    TS --> A
    TS --> B
    TS --> C
    TS --> STORAGE

    ROOT --> RB
    RB --> SCH
    SCH --> TC
    ROOT --> DASH
    ROOT --> EP

    ROOT --> A
    ROOT --> B
    ROOT --> C

    A <--> P2P
    B <--> P2P
    C <--> P2P

    ROOT --> STORAGE
    STORAGE --> OMV
```

### Architectural planes

| Plane            | Components                              | Responsibility                                                 |
|------------------|-----------------------------------------|----------------------------------------------------------------|
| Control          | Root, RootBrain, Scheduler, Coordinator | Membership, scheduling, allocation, coordination               |
| Compute          | NodeBrain, local workers                | Hardware/runtime observation and execution                     |
| Training data    | Node ↔ Node                             | Gradients, tensors, model synchronization                      |
| Persistence      | Storage API, OMV/backend                | Datasets, models, checkpoints, artifacts, experiments, results |
| Observability    | Dashboard, telemetry                    | Runtime state, health, allocations, endpoints                  |
| Network/security | Tailscale                               | Private production connectivity and network boundary           |

**Root is not the training-data relay.** High-frequency
gradients/tensors/model traffic stays on the direct Node ↔ Node path.

------------------------------------------------------------------------

## 2. Core Runtime Loop

``` mermaid
flowchart LR
    H["Hardware / Runtime"] --> N["NodeBrain"]
    N --> CV["CapabilityVector"]
    W["Workload"] --> WS["WorkloadSignature"]

    CV --> M["Capability Matching"]
    WS --> M
    M --> S["Adaptive Scheduler"]
    S --> A["Allocation"]
    A --> E["Execution"]

    E --> O["Observed Performance"]
    O --> D["Degradation / Straggler Detection"]
    D --> C["AdaptiveTrainingController"]
    C --> G["Candidate Allocations"]
    G --> U["Utility / Communication Cost"]
    U --> S
```

The runtime continuously follows:

**Observe → Represent → Match → Allocate → Execute → Measure → Detect →
Adapt → Reallocate**

This turns scheduling into a feedback-driven runtime process rather than
a one-time placement decision.

------------------------------------------------------------------------

# 3. Design Principles

### Root = control plane

The Root performs cluster-level scheduling, allocation, coordination,
endpoint publication, and dashboard/API exposure.

### NodeBrain = local intelligence

Each worker observes its own hardware, runtime, health, capabilities,
and execution behavior.

### Node ↔ Node = training data plane

Workers communicate directly for high-frequency gradients, tensors,
model synchronization, and training payloads.

### Storage = persistence plane

Datasets, models, checkpoints, artifacts, experiments, and results are
persisted through the Storage API. Storage is not the training hot path.

### Tailscale = production network/security layer

Production communication uses Tailscale. Machine-specific Tailscale IPs
are not hard-coded; nodes/services dynamically discover their current
Tailscale address.

### Dynamic endpoints

Root, Dashboard, and Storage have independently allocated runtime ports.

``` text
Current Tailscale IP
        +
Dynamic service port
        ↓
Advertised endpoint
```

### Level-1 hierarchy

The current operational hierarchy is:

``` text
Root
 ├── Node
 ├── Node
 └── Node
```

Sub-root/hierarchical scheduling is reserved for future scope.

------------------------------------------------------------------------

# 4. Phase 1–2 — Architecture & NodeBrain Foundation

### Implemented

- Repository and architecture audit.
- Clear boundaries between PALADIN communication, ACR runtime, and Vayrm
  integration.
- NodeBrain local intelligence.
- Cross-platform hardware/runtime discovery.
- CPU/GPU/memory/health telemetry.
- Normalized node capability representation.
- Lightweight continuous monitoring.
- Capability-based node classification.

``` mermaid
flowchart LR
    NB["NodeBrain"] --> D["Discovery"]
    NB --> CAP["Capabilities"]
    NB --> TEL["Telemetry"]
    NB --> HEALTH["Health"]
    NB --> SELF["Local Self Model"]
    CAP --> ASSESS["Local Assessment"]
    TEL --> ASSESS
    HEALTH --> ASSESS
    ASSESS --> STATE["Node State"]
```

Platform-specific implementations are separated so Linux, macOS, and
Windows can expose a common capability model.

------------------------------------------------------------------------

# 5. Phase 3 — Workload & Capability Modeling

Two complementary quantitative representations are used.

``` text
Node
 ├── CPU
 ├── GPU / accelerator
 ├── Memory
 ├── Storage
 ├── Network
 └── Runtime capability
        ↓
CapabilityVector
```

``` text
Workload
 ├── Compute requirements
 ├── Memory requirements
 ├── Accelerator requirements
 ├── I/O characteristics
 └── Communication sensitivity
        ↓
WorkloadSignature
```

Matching:

``` mermaid
flowchart LR
    C["CapabilityVector"] --> MATCH["Capability Matcher"]
    W["WorkloadSignature"] --> MATCH
    MATCH --> OK["Compatible Nodes"]
    MATCH --> NO["Rejected / Incompatible Nodes"]
    OK --> SCH["Scheduler"]
```

This prevents heterogeneous machines from being treated as
interchangeable resources.

------------------------------------------------------------------------

# 6. Phase 4 — Binary Communication Foundation

Implemented communication primitives include:

- binary telemetry representation
- compact framing
- typed message/frame categories
- telemetry/control/gradient payload types
- bounded payload handling
- malformed-frame protection
- extensible framing

Conceptually:

``` text
┌──────────────┬──────────────┬──────────────┬───────────────┐
│ Type / Magic │ Version      │ Length       │ Payload       │
└──────────────┴──────────────┴──────────────┴───────────────┘
```

Bounded framing and partial-read handling provide predictable behavior
over stream-oriented transports.

------------------------------------------------------------------------

# 7. Phase 5–6 — Runtime & Scheduler Integration

``` mermaid
flowchart LR
    NR["Node Reports"] --> RB["RootBrain"]
    RB --> CS["Cluster State"]
    CS --> SCH["Adaptive Scheduler"]
    SCH --> DEC["Scheduling Decision"]
    DEC --> RF["Resource / Allocation Layer"]
    RF --> EXEC["Runtime Execution"]
    EXEC --> OBS["Observed Result"]
    OBS --> RB
```

Implemented concepts include:

- AdaptiveScheduler
- capability-gated scheduling
- scheduler/node-event integration
- RootBrain cluster state
- runtime execution mapping
- resource-aware filtering
- scheduling decision propagation

The core boundary is:

``` text
Scheduler decides
      ↓
Runtime executes
      ↓
Telemetry reports
      ↓
Scheduler adapts
```

------------------------------------------------------------------------

# 8. Phase 7 — Distributed Training Foundation

PALADIN includes a real PyTorch distributed-training path.

``` mermaid
sequenceDiagram
    participant R as Root
    participant A as Node A
    participant B as Node B
    participant C as Node C

    R->>A: Allocation / shard assignment
    R->>B: Allocation / shard assignment
    R->>C: Allocation / shard assignment

    A->>A: Forward + Backward
    B->>B: Forward + Backward
    C->>C: Forward + Backward

    A->>R: Gradient
    B->>R: Gradient
    C->>R: Gradient

    R->>R: Validate + Aggregate
    R->>R: Optimizer Step

    R->>A: Updated Model
    R->>B: Updated Model
    R->>C: Updated Model
```

Implemented:

- multi-worker shared-model training
- real forward/backward execution
- separate worker data shards
- gradient extraction
- shape/dtype/device validation
- model version and global-step management
- duplicate/stale gradient rejection
- optimizer-step ownership
- gradient aggregation
- model synchronization
- multi-worker validation

State progression:

``` text
Model N
  ↓
Worker gradients
  ↓
Validation
  ↓
Aggregation
  ↓
Optimizer step
  ↓
Model N+1
  ↓
Synchronization
```

------------------------------------------------------------------------

# 9. Phase 8 — Direct Training Transport

``` mermaid
flowchart LR
    ENGINE["Training Engine"]
    IF["TrainingTransport"]

    LOCAL["Local Test Transport"]
    TCP["TCP Transport"]
    TAIL["Tailscale TCP"]

    ENGINE --> IF
    IF --> LOCAL
    IF --> TCP
    IF --> TAIL
    TAIL --> PEER["Peer Node"]
```

Implemented:

- transport abstraction
- local loopback validation
- TCP transport
- Tailscale TCP transport
- binary tensor/gradient exchange
- model synchronization
- framing and partial-read reconstruction
- bounded queues/backpressure
- reconnect/backoff
- concurrent multi-worker validation

Production path:

``` text
Node A → Tailscale → Node B
        gradients / tensors / model data
```

The Root remains the control plane rather than becoming a mandatory
relay for every training payload.

------------------------------------------------------------------------

# 10. Phase 9 — Degradation Detection

Runtime adaptation is driven by observed execution behavior.

Tracked information includes:

- iteration time
- throughput
- compute time
- communication time
- observed performance
- expected performance
- node health

``` mermaid
flowchart LR
    RAW["Runtime Metrics"] --> EWMA["EWMA Performance Estimate"]
    EWMA --> EXPECT["Expected vs Observed"]
    EXPECT --> STRAG["Straggler / Degradation Detection"]
    STRAG --> STATE["Degradation / Recovery State"]
    STATE --> CTRL["Adaptive Controller"]
```

Implemented:

- `NodePerformanceState`
- EWMA performance estimation
- expected-vs-observed throughput
- iteration-time tracking
- straggler detection
- degradation/recovery states
- event-driven detection
- anti-thrashing state transitions

The adaptation model is not reduced to a single hard-coded utilization
threshold. It considers sustained observed behavior and the relationship
between expected and actual performance.

------------------------------------------------------------------------

# 11. Phase 10 — Adaptive Workload Reallocation

``` mermaid
flowchart TD
    OBS["Observed Performance"] --> DET["Degradation Detection"]
    DET --> GEN["Candidate Allocation Generation"]

    GEN --> A["Candidate A"]
    GEN --> B["Candidate B"]
    GEN --> C["Candidate C"]

    A --> U["Utility Evaluation"]
    B --> U
    C --> U

    U --> COMM["Communication Cost"]
    U --> GOOD["Expected Cluster Goodput"]
    U --> CAP["Capability / Capacity"]

    COMM --> DEC["Allocation Decision"]
    GOOD --> DEC
    CAP --> DEC

    DEC --> VER["Allocation Epoch / Version"]
    VER --> APPLY["Apply Reallocation"]
    APPLY --> TC["Training Coordinator"]
    TC --> EXEC["Workers"]
    EXEC --> OBS
```

Implemented:

- `AdaptiveTrainingController`
- dynamic candidate allocation generation
- utility-based allocation selection
- communication-aware decisions
- allocation epochs/versioning
- hysteresis/cooldown
- dynamic workload redistribution
- node disappearance handling
- node recovery handling
- TrainingCoordinator integration
- heterogeneous multi-worker adaptive validation

The intended behavior is:

``` text
Detect persistent imbalance
        ↓
Generate alternatives
        ↓
Estimate utility/cost
        ↓
Version allocation
        ↓
Apply
        ↓
Cooldown
        ↓
Observe again
```

This reduces oscillation and prevents stale allocation decisions from
being applied indefinitely.

------------------------------------------------------------------------

# 12. Lifecycle & Persistent Storage

The lifecycle layer coordinates:

``` mermaid
flowchart TB
    LIFE["Training Lifecycle"]
    LIFE --> DATA["Dataset Manager"]
    LIFE --> MODEL["Model Manager"]
    LIFE --> HP["Hyperparameters"]
    LIFE --> EXP["Experiment Manager"]
    LIFE --> CK["Checkpoint Lifecycle"]
    LIFE --> RES["Result Manager"]

    DATA --> API["Storage API"]
    MODEL --> API
    EXP --> API
    CK --> API
    RES --> API

    API --> DS["Datasets"]
    API --> MO["Models"]
    API --> CH["Checkpoints"]
    API --> AR["Artifacts"]
    API --> EX["Experiments"]
    API --> RS["Results"]
```

Supported lifecycle responsibilities include:

- dataset management
- model management
- experiment tracking
- result management
- checkpoint creation
- checkpoint metadata
- training-state restoration
- lifecycle coordination

Checkpoint metadata can include:

``` text
model version
global step
epoch
dataset
allocation
optimizer state
hyperparameters
world/node information
metrics
```

Model-loading paths include formats such as:

- Safetensors
- `.pt`
- `.pth`
- `.h5`
- `.hdf5`
- `.bin`

------------------------------------------------------------------------

# 13. Storage API & Dynamic Storage Endpoint

Storage is a separate plane:

``` mermaid
flowchart LR
    ROOT["Root"]
    CLIENT["Dashboard / Client"]
    API["Storage API"]
    PERSIST["Persistent Storage / OMV"]

    ROOT --> API
    CLIENT --> API
    API --> PERSIST
```

Namespaces are organized around:

``` text
/datasets
/models
/checkpoints
/artifacts
/experiments
/results
```

Supported operation categories include:

``` text
GET / PUT / DELETE / HEAD / LIST
```

The Storage API receives its own dynamically allocated port.

### Endpoint model

``` text
Tailscale IP
   │
   ├── Root dynamic port
   ├── Dashboard dynamic port
   └── Storage dynamic port
```

No service should depend on a machine-specific hard-coded Tailscale IP.

------------------------------------------------------------------------

# 14. Dashboard & Observability

The Dashboard is an observability/control surface over authoritative
runtime state.

It is intended to expose:

- node membership
- node capabilities
- health
- cluster state
- current allocations
- scheduler state
- telemetry
- service endpoints
- storage status

The scheduler/cluster state remains authoritative; the dashboard should
not invent duplicate allocation state.

``` text
Root / Scheduler
       ↓
Authoritative runtime state
       ↓
Dashboard API
       ↓
Dashboard
```

------------------------------------------------------------------------

# 15. Dynamic Endpoint & Network Architecture

``` mermaid
flowchart TB
    IP["Current Tailscale IP"]

    ROOTP["Dynamic Root Port"]
    DASHP["Dynamic Dashboard Port"]
    STORP["Dynamic Storage Port"]

    ROOT["Root"]
    DASH["Dashboard"]
    STORAGE["Storage API"]

    IP --> ROOTP --> ROOT
    IP --> DASHP --> DASH
    IP --> STORP --> STORAGE
```

Important invariant:

``` text
ROOT_PORT != DASHBOARD_PORT != STORAGE_PORT
```

Endpoint discovery is runtime-based:

``` text
Discover local Tailscale IP
        ↓
Bind service to appropriate interface
        ↓
Obtain runtime port
        ↓
Health-check
        ↓
Advertise endpoint
```

Hard-coded development machine addresses are not part of the production
architecture.

------------------------------------------------------------------------

# 16. Node Failure, Degradation & Recovery

``` mermaid
stateDiagram-v2
    [*] --> HEALTHY
    HEALTHY --> DEGRADED: sustained performance deviation
    DEGRADED --> HEALTHY: performance recovery
    DEGRADED --> UNAVAILABLE: node failure
    UNAVAILABLE --> RECOVERING: node returns
    RECOVERING --> HEALTHY: reconciliation complete
    DEGRADED --> REALLOCATION: adaptive controller
    UNAVAILABLE --> REALLOCATION: node removed
    REALLOCATION --> HEALTHY: new allocation active
```

The runtime accounts for:

- temporary degradation
- sustained straggling
- communication problems
- node disappearance
- node recovery
- allocation version changes

Hysteresis/cooldown prevents immediate repeated reallocations.

------------------------------------------------------------------------

# 17. Resource Fabric

The resource-fabric layer represents and manages heterogeneous resource
capacity.

``` text
Node
 │
 ├── CPU
 ├── GPU / accelerator
 ├── Memory
 ├── Network
 ├── Storage
 ├── Health / thermal state
 └── Runtime capability
       ↓
Capability / Capacity Model
       ↓
Filtering / Scoring
       ↓
Offer / Claim / Lease
       ↓
Allocation
```

Relevant mechanisms include:

- capacity models
- normalization
- filtering
- scoring
- resource offers
- resource claims
- quotas
- leases
- health
- preemption
- reconciliation
- cooldown

------------------------------------------------------------------------

# 18. Communication & Security Boundaries

``` mermaid
flowchart TB
    TS["TAILSCALE"]

    ROOT["Root"]
    N1["Node A"]
    N2["Node B"]
    N3["Node C"]
    ST["Storage API"]

    TS --- ROOT
    TS --- N1
    TS --- N2
    TS --- N3
    TS --- ST

    ROOT -. control .-> N1
    ROOT -. control .-> N2
    ROOT -. control .-> N3

    N1 <-->|training data| N2
    N2 <-->|training data| N3
    N1 <-->|training data| N3

    ROOT --> ST
```

Security/architecture invariants:

- Tailscale is mandatory for production networking.
- No machine-specific hard-coded Tailscale IPs.
- No insecure localhost/LAN production fallback.
- Control and training data paths are separated.
- Storage is separate from the training hot path.
- Binary framing is bounded.
- Network queues apply backpressure.
- Storage namespace/path protections are enforced.
- Root does not become an authoritative training/model database.

Known deviations from these invariants (hardening backlog): the Storage Gateway, discovery locator and PALADIN dashboard APIs are unauthenticated and bind `0.0.0.0` by default; see `docs/ENDPOINTS.md` §9.

------------------------------------------------------------------------

# 19. Platform Architecture

``` mermaid
flowchart TB
    ACR["Common ACR Runtime"]

    L["Linux Provider"]
    M["macOS Provider"]
    W["Windows Provider"]

    ACR --> L
    ACR --> M
    ACR --> W

    L --> LH["Linux CPU / GPU / Runtime"]
    M --> MH["Apple Silicon / CPU / MPS"]
    W --> WH["Windows CPU / GPU / Runtime"]
```

Platform-specific hardware/runtime behavior is isolated from common
scheduling logic.

The target is one common capability model across heterogeneous
environments.

------------------------------------------------------------------------

# 20. Repository Structure

``` text
adaptive-runtime/
├── acr/
│   ├── cluster/
│   ├── communication/
│   ├── core/
│   ├── execution/
│   ├── integration/
│   ├── lifecycle/
│   ├── network/
│   ├── node/
│   ├── peers/
│   ├── platforms/
│   ├── registry/
│   ├── resource_fabric/
│   ├── resources/
│   ├── runtime/
│   ├── schedulers/
│   ├── scheduling/
│   ├── simulation/
│   ├── state/
│   ├── storage/
│   ├── telemetry/
│   ├── training/
│   ├── transfer/
│   ├── ui/
│   └── workloads/
├── ai/
├── apps/
│   ├── cli/
│   ├── node/
│   └── root/
│       └── api/
├── cluster/
├── core/
├── protocol/
├── vayrm/
├── docs/
└── tests/
```

### Major runtime areas

| Directory                 | Role                                                 |
|---------------------------|------------------------------------------------------|
| `acr/node/`               | NodeBrain and node-local intelligence                |
| `acr/resources/`          | Resource/capability state                            |
| `acr/workloads/`          | Workload signatures and descriptions                 |
| `acr/scheduling/`         | Adaptive allocation                                  |
| `acr/schedulers/`         | Scheduler strategies                                 |
| `acr/resource_fabric/`    | Resource capacity/allocation                         |
| `acr/training/`           | Distributed training                                 |
| `acr/training/transport/` | Training transport                                   |
| `acr/lifecycle/`          | Dataset/model/checkpoint/experiment/result lifecycle |
| `acr/storage/`            | Storage backends/API                                 |
| `acr/telemetry/`          | Runtime telemetry                                    |
| `acr/communication/`      | Binary communication/framing                         |
| `acr/platforms/`          | Linux/macOS/Windows support                          |
| `acr/ui/`                 | Runtime dashboards                                   |
| `acr/registry/`           | Cluster/runtime registries                           |
| `vayrm/`                  | Runtime/network integration facade                   |

------------------------------------------------------------------------

# 21. Architecture Integrity

The current design intentionally preserves these boundaries:

- `vayrm/` is treated as an integration/runtime dependency and is not
  rewritten as part of PALADIN feature work unless explicitly required.
- PALADIN core is not replaced by ACR.
- Tailscale is not bypassed for production networking.
- No local production fallback exposes runtime services insecurely.
- No Root-side authoritative training/model database is introduced.
- Training payloads are separated from Root control traffic.
- Storage is not the high-frequency training transport.
- Sub-root architecture remains future scope.
- Dynamic endpoint discovery replaces machine-specific IP assumptions.

------------------------------------------------------------------------

# 22. Validation Status

| Phase                                        | Status                      |
|----------------------------------------------|-----------------------------|
| Phase 1 — Architecture / audit               | **COMPLETE**                |
| Phase 2 — NodeBrain foundation               | **COMPLETE**                |
| Phase 3 — Capability / workload modeling     | **COMPLETE**                |
| Phase 4 — Binary communication               | **COMPLETE**                |
| Phase 5 — Runtime integration                | **COMPLETE where verified** |
| Phase 6 — Scheduler integration              | **COMPLETE where verified** |
| Phase 7 — Distributed training               | **VALIDATED**               |
| Phase 8 — Node ↔ Node TCP transport          | **VALIDATED**               |
| Phase 9 — Degradation detection              | **VALIDATED**               |
| Phase 10 — Adaptive reallocation             | **VALIDATED**               |
| Phase 11 — Physical heterogeneous validation | **NEXT**                    |

Previously validated development areas include:

- multi-worker training
- gradient validation/aggregation
- model synchronization
- adaptive allocation scenarios
- TCP training transport
- Tailscale transport abstraction
- dynamic endpoint discovery
- dynamic Storage API port allocation
- Storage API development GET/PUT
- lifecycle tests
- Vayrm compatibility tests

**Physical heterogeneous hardware validation remains the next major
validation step.**

------------------------------------------------------------------------

# 23. Phase 11 — Real Hardware Validation & Benchmarking

The next phase should turn runtime validation into reproducible research
evidence.

## Hardware matrix

A representative test matrix should include combinations such as:

``` text
macOS / Apple Silicon
Linux / NVIDIA
Windows / CPU or NVIDIA
```

Actual hardware should be recorded as experimental configuration rather
than hard-coded into the runtime.

## Baselines

Useful baselines include:

``` text
1. Static allocation
2. Round-robin allocation
3. Capability-aware allocation
4. Adaptive PALADIN allocation
```

## Metrics

Measure:

- iteration time
- samples/sec
- effective throughput
- compute time
- communication time
- synchronization time
- node utilization
- straggler duration
- degradation detection latency
- reallocation latency
- recovery time
- checkpoint overhead
- total training time
- allocation changes

## Controlled degradation experiment

``` mermaid
sequenceDiagram
    participant R as Root
    participant F as Fast Node
    participant S as Degraded Node

    R->>F: Initial allocation
    R->>S: Initial allocation

    F->>R: Performance telemetry
    S->>R: Performance telemetry

    Note over S: Controlled slowdown

    S->>R: Reduced observed throughput
    R->>R: EWMA / degradation detection
    R->>R: Candidate allocation generation
    R->>R: Utility + communication evaluation
    R->>F: Updated allocation
    R->>S: Updated allocation

    F->>R: New performance
    S->>R: New performance
```

The experiment should report the complete timeline and quantitative
changes, not only the final result.

------------------------------------------------------------------------

# 24. Research Context

PALADIN intersects with established systems research areas:

- introspective scheduling
- goodput-aware scheduling
- heterogeneous distributed training
- straggler mitigation
- adaptive resource allocation
- distributed training
- feedback-controlled runtime systems

Representative references include:

- **Gandiva**, OSDI 2018 — introspective cluster scheduling for deep
  learning.
- **Pollux**, OSDI 2021 — co-adaptive scheduling for goodput-oriented
  deep learning.
- **SADDLE**, 2025 — feedback-controlled heterogeneous distributed
  training.
- **PyTorch DistributedDataParallel** — established distributed-training
  infrastructure.

PALADIN should distinguish established mechanisms from the proposed
system-level contribution.

### Established concepts

``` text
EWMA
Straggler detection
Adaptive allocation
Gradient aggregation
Distributed training
Capability-aware scheduling
```

### Proposed integrated system contribution

``` text
Heterogeneous adaptive runtime architecture
        +
NodeBrain local intelligence
        +
Capability/workload representation
        +
Adaptive runtime reallocation
        +
Direct Node ↔ Node training plane
        +
Dynamic secure endpoints
        +
Persistent lifecycle/storage plane
```

Individual mechanisms should not be presented as novel merely because
they are implemented here. Novelty should be supported through
literature review and reproducible experimental comparison.

------------------------------------------------------------------------

# 25. Runtime Lifecycle

``` mermaid
flowchart TD
    START["Startup"]
    NET["Discover Tailscale"]
    EP["Allocate Dynamic Endpoints"]
    REG["Register Services / Nodes"]
    CAP["Collect Capabilities"]
    WORK["Describe Workload"]
    MATCH["Capability Matching"]
    ALLOC["Initial Allocation"]
    EXEC["Execute"]
    MON["Monitor"]
    DET["Detect Degradation"]
    ADAPT["Adaptive Reallocation"]
    CK["Checkpoint"]
    RES["Persist Results"]
    REC["Recovery / Reconciliation"]
    STOP["Shutdown"]

    START --> NET
    NET --> EP
    EP --> REG
    REG --> CAP
    CAP --> WORK
    WORK --> MATCH
    MATCH --> ALLOC
    ALLOC --> EXEC
    EXEC --> MON
    MON --> DET
    DET --> ADAPT
    ADAPT --> EXEC
    EXEC --> CK
    CK --> RES
    MON --> REC
    REC --> ALLOC
    RES --> STOP
```

------------------------------------------------------------------------

# 26. Final Architecture at a Glance

``` mermaid
flowchart TB
    TS["TAILSCALE<br/>Private Production Network"]

    subgraph CP["CONTROL PLANE"]
        ROOT["ROOT"]
        RB["RootBrain"]
        SCH["Adaptive Scheduler"]
        TC["Training Coordinator"]
        DASH["Dashboard"]
    end

    subgraph NODES["HETEROGENEOUS COMPUTE"]
        A["Node A<br/>NodeBrain"]
        B["Node B<br/>NodeBrain"]
        C["Node C<br/>NodeBrain"]
    end

    P2P["DIRECT NODE ↔ NODE<br/>Gradients • Tensors • Model Sync"]

    subgraph STORE["PERSISTENCE"]
        API["Storage API"]
        DB["Datasets / Models / Checkpoints / Artifacts / Results"]
    end

    TS --- CP
    TS --- NODES
    TS --- P2P
    TS --- STORE

    ROOT --> RB
    RB --> SCH
    SCH --> TC
    ROOT --> DASH

    ROOT --> A
    ROOT --> B
    ROOT --> C

    A <--> P2P
    B <--> P2P
    C <--> P2P

    ROOT --> API
    API --> DB
```

## One-sentence architecture

> **The Root decides, NodeBrain observes local conditions, heterogeneous
> nodes execute work, nodes exchange training data directly, Storage
> persists lifecycle artifacts, and observed runtime behavior
> continuously feeds back into adaptive scheduling decisions.**

------------------------------------------------------------------------

## Current Status

**Phases 1–10:** Implemented/validated where explicitly verified.

**Current focus:** **Phase 11 — real heterogeneous hardware validation
and baseline benchmarking.**

**Current hierarchy:** **Root → Node.**

**Training hot path:** **Node ↔ Node.**

**Persistence path:** **Storage API → persistent storage.**

**Production network:** **Tailscale.**

**Future scope:** hierarchical sub-roots, broader physical validation,
reproducible benchmark suite, and research evaluation.


# 27. Root Dashboard Architecture

The Root Dashboard is the operational view over the Root control plane. It does not become a second scheduler; it reads authoritative cluster/runtime state and presents it through the API/UI.

```mermaid
flowchart TB
    USER["Operator / Researcher"]

    subgraph DASH["ROOT DASHBOARD"]
        UI["Dashboard UI"]
        API["Dashboard API"]
        ENDPOINTS["Endpoint View"]
        NODES["Node View"]
        CLUSTER["Cluster View"]
        ALLOC["Allocation View"]
        SCHED["Scheduler View"]
        TEL["Telemetry View"]
        STORAGE["Storage View"]
    end

    subgraph ROOT["ROOT CONTROL PLANE"]
        RB["RootBrain"]
        SCH["Adaptive Scheduler"]
        TC["Training Coordinator"]
        STATE["Authoritative Cluster / Allocation State"]
        REG["Endpoint Registry"]
    end

    subgraph N["COMPUTE NODES"]
        NA["Node A / NodeBrain"]
        NB["Node B / NodeBrain"]
        NC["Node C / NodeBrain"]
    end

    subgraph STORE["PERSISTENCE"]
        SA["Storage API"]
        PS["Persistent Storage"]
    end

    USER --> UI
    UI --> API

    API --> ENDPOINTS
    API --> NODES
    API --> CLUSTER
    API --> ALLOC
    API --> SCHED
    API --> TEL
    API --> STORAGE

    ENDPOINTS --> REG
    NODES --> STATE
    CLUSTER --> STATE
    ALLOC --> STATE
    SCHED --> SCH
    TEL --> RB

    RB --> STATE
    SCH --> STATE
    TC --> STATE

    RB --> NA
    RB --> NB
    RB --> NC

    STORAGE --> SA
    SA --> PS
```

## Dashboard information flow

```mermaid
flowchart LR
    NODE["NodeBrain"] --> REPORT["Node Reports / Telemetry"]
    REPORT --> ROOT["Root / RootBrain"]

    ROOT --> STATE["Authoritative Runtime State"]

    STATE --> API["Dashboard API"]

    API --> NODES["Nodes"]
    API --> CLUSTER["Cluster"]
    API --> ALLOC["Allocations"]
    API --> SCHED["Scheduler"]
    API --> TEL["Telemetry"]
    API --> EP["Endpoints"]
    API --> ST["Storage"]

    NODES --> UI["Dashboard UI"]
    CLUSTER --> UI
    ALLOC --> UI
    SCHED --> UI
    TEL --> UI
    EP --> UI
    ST --> UI
```

### Root Dashboard responsibilities

```text
                 ROOT DASHBOARD
                       │
        ┌──────────────┼──────────────┐
        ↓              ↓              ↓
      Nodes         Cluster        Scheduler
        │              │              │
   capabilities      health       decisions
   health            state        allocations
   telemetry         membership   adaptation
        │              │              │
        └──────────────┼──────────────┘
                       ↓
                  Operator View
```

The dashboard can expose:

- current Root endpoint
- Dashboard endpoint
- Storage endpoint
- registered nodes
- node capabilities
- node health
- current allocations
- allocation epochs/versions
- scheduler state
- training state
- telemetry
- storage availability
- cluster-level runtime state

### Dashboard authority boundary

```text
NodeBrain
    ↓
Node report
    ↓
RootBrain / Cluster State
    ↓
Scheduler / Allocation State
    ↓
Dashboard API
    ↓
Dashboard UI
```

The Dashboard **observes and presents authoritative state**. It does not create an independent allocation database.

---



## Root Dashboard — Mermaid recreation of the terminal dashboard

> This diagram is deliberately a **dashboard/wireframe reconstruction**, not just an architecture graph. It mirrors the major regions of the attached Root terminal UI: header → live-worker table → three-column lower dashboard → events/pipeline/resource/prediction panels.

```mermaid
flowchart TB
    %% =========================================================
    %% HEADER
    %% =========================================================
    H["<b>ACR PALADIN ROOT</b><br/>node_id = ROOT &nbsp; | &nbsp; platform = macOS &nbsp; | &nbsp; PALADIN: RUNNING &nbsp; | &nbsp; workers = 3"]
    HS["<b>ROOT SELF</b>"]

    H --> HS

    %% =========================================================
    %% LIVE WORKERS - dashboard table
    %% =========================================================
    subgraph LIVE["LIVE WORKERS"]
        direction TB

        WH["<b>PEER</b> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <b>PLATFORM</b> &nbsp;&nbsp;&nbsp; <b>STATUS</b> &nbsp;&nbsp; <b>FRESH</b> &nbsp;&nbsp; <b>AGE</b> &nbsp;&nbsp; <b>CPU</b> &nbsp;&nbsp; <b>MEM</b> &nbsp;&nbsp; <b>GPU</b> &nbsp;&nbsp; <b>PREDICTION</b> &nbsp;&nbsp; <b>RISK</b> &nbsp;&nbsp; <b>SCHEDULER</b>"]

        W1["<b>WORKER A</b><br/>macOS<br/>ONLINE • FRESH<br/>CPU / MEM / GPU telemetry<br/>STABLE • SAFE • ELIGIBLE"]

        W2["<b>WORKER B</b><br/>Windows<br/>ONLINE • FRESH<br/>CPU / MEM / GPU telemetry<br/>STABLE • SAFE • ELIGIBLE"]

        W3["<b>WORKER C</b><br/>Windows<br/>ONLINE • FRESH<br/>CPU / MEM / GPU telemetry<br/>STABLE • SAFE • ELIGIBLE"]

        WH --- W1
        WH --- W2
        WH --- W3
    end

    HS --> LIVE

    %% =========================================================
    %% LOWER DASHBOARD
    %% =========================================================
    subgraph DASH["ROOT DASHBOARD"]
        direction LR

        %% LEFT COLUMN
        subgraph LEFT["LEFT COLUMN"]
            direction TB

            SUB["<b>SUBROOT HIERARCHY</b><br/><br/>No hierarchical<br/>subroots discovered"]

            TOPO["<b>PEER / P2P TOPOLOGY</b><br/><br/><b>LOCAL NODE</b><br/>node_id : ROOT<br/>role : ROOT<br/>tailscale_ip : dynamic<br/>endpoint : dynamic<br/><br/><b>PEERS</b><br/>Worker A • Worker B • Worker C"]

            EVENTS["<b>EVENTS</b><br/><br/>TELEM • PeerUpdated<br/>health_ok = true<br/>COMM_ACK<br/>PeerHealthMessage<br/>RootAnnouncement<br/>PeerDirectoryMessage"]

            SUB --- TOPO
            TOPO --- EVENTS
        end

        %% CENTER COLUMN
        subgraph CENTER["CENTER COLUMN"]
            direction TB

            ID["<b>IDENTITY & CONSTRAINTS</b><br/><br/>NODE: selected worker<br/>OS / platform<br/><br/><b>HARD CONSTRAINTS</b><br/>none / reported constraints<br/><br/><b>SOFT CONSTRAINTS</b><br/>none / reported constraints"]

            FABRIC["<b>RESOURCE FABRIC</b><br/><br/>Worker A → CPU / MEM / confidence / freshness<br/>Worker B → CPU / MEM / confidence / freshness<br/>Worker C → CPU / MEM / confidence / freshness<br/><br/><b>CLUSTER RESOURCE STATE</b>"]

            PIPE["<b>PIPELINE VALIDATION</b><br/><br/>PASS  PALADIN connected<br/>PASS  Endpoint valid<br/>PASS  Telemetry fresh<br/>PASS  NodeKnowledge updated<br/>PASS  Resource Fabric updated"]

            ID --- FABRIC
            FABRIC --- PIPE
        end

        %% RIGHT COLUMN
        subgraph RIGHT["RIGHT COLUMN"]
            direction TB

            PRED["<b>PREDICTIVE PREEMPTION — CLUSTER</b><br/><br/><b>ALL NODES NORMAL</b><br/><br/>DETAIL SOURCE: NodeBrain<br/>STATUS: NORMAL<br/>RESOURCE: CPU / MEMORY / GPU<br/>CURRENT: utilization<br/><br/>TRAJECTORY: STABLE / VOLATILE<br/>CONFIDENCE: prediction confidence<br/>FRESHNESS: telemetry freshness<br/>TIME TO LIMIT: predicted horizon"]

            HARNESS["<b>EXPERIMENT HARNESS</b><br/><br/>Decision Engine<br/>Experiment state<br/>Allocation / scheduling decision<br/>No decision engine attached"]

            PRED --- HARNESS
        end
    end

    LIVE --> DASH

    %% =========================================================
    %% INFORMATION FLOW INTO PANELS
    %% =========================================================
    W1 -. "telemetry" .-> FABRIC
    W2 -. "telemetry" .-> FABRIC
    W3 -. "telemetry" .-> FABRIC

    W1 -. "prediction" .-> PRED
    W2 -. "prediction" .-> PRED
    W3 -. "prediction" .-> PRED

    W1 -. "peer health" .-> TOPO
    W2 -. "peer health" .-> TOPO
    W3 -. "peer health" .-> TOPO

    W1 -. "events" .-> EVENTS
    W2 -. "events" .-> EVENTS
    W3 -. "events" .-> EVENTS

    %% =========================================================
    %% STYLING — terminal-like visual grouping
    %% =========================================================
    classDef header fill:#171717,stroke:#7c3aed,stroke-width:2px,color:#ffffff;
    classDef worker fill:#102018,stroke:#22d3ee,stroke-width:1.5px,color:#ffffff;
    classDef panel fill:#171717,stroke:#777777,stroke-width:1.5px,color:#ffffff;
    classDef identity fill:#171717,stroke:#d946ef,stroke-width:2px,color:#ffffff;
    classDef resource fill:#171717,stroke:#7c3aed,stroke-width:2px,color:#ffffff;
    classDef prediction fill:#171717,stroke:#a3e635,stroke-width:2px,color:#ffffff;
    classDef validation fill:#171717,stroke:#eab308,stroke-width:2px,color:#ffffff;
    classDef event fill:#171717,stroke:#64748b,stroke-width:1.5px,color:#ffffff;

    class H,HS header;
    class WH,W1,W2,W3 worker;
    class SUB,TOPO panel;
    class EVENTS event;
    class ID identity;
    class FABRIC resource;
    class PRED prediction;
    class PIPE,HARNESS validation;
```

### Dashboard layout correspondence

```mermaid
flowchart TB
    A["ROOT HEADER<br/>ACR PALADIN ROOT • identity • platform • runtime"] 
    B["LIVE WORKERS<br/>worker table"]

    subgraph ROW["LOWER DASHBOARD ROW"]
        direction LR
        C["LEFT<br/>Subroot Hierarchy<br/>Peer / P2P Topology<br/>Events"]
        D["CENTER<br/>Identity & Constraints<br/>Resource Fabric<br/>Pipeline Validation"]
        E["RIGHT<br/>Predictive Preemption — Cluster<br/>Experiment Harness"]
    end

    A --> B --> ROW
```

The important distinction is that this Mermaid is **not replacing the dashboard with a generic architecture diagram**. It documents the dashboard as a visual operator surface: the top worker table and the lower panel layout are represented explicitly, while the dotted links show where NodeBrain telemetry, predictions, peer state and events populate those panels.

# 28. NodeBrain Architecture

NodeBrain is the local intelligence component running on every compute node.

Its responsibility is to continuously understand:

- what hardware exists
- what resources are currently available
- what the node is capable of
- how the node is performing
- whether the node is healthy
- what the node observes about peers/runtime state
- what local execution is doing

```mermaid
flowchart TB
    NB["NODEBRAIN"]

    subgraph DISC["DISCOVERY"]
        HW["Hardware Discovery"]
        RT["Runtime Discovery"]
        OS["Platform Provider"]
    end

    subgraph MODEL["LOCAL MODEL"]
        SELF["Self Model"]
        CAP["Capability Vector"]
        SNAP["Resource Snapshot"]
        STATE["Node State"]
    end

    subgraph OBS["OBSERVATION"]
        TEL["Telemetry"]
        PERF["Performance"]
        HEALTH["Health"]
        PEER["Peer View / Peer Health"]
    end

    subgraph DEC["LOCAL ASSESSMENT"]
        ASSESS["Assessment"]
        TEMP["Temporal State"]
        MACHINE["State Machine"]
    end

    subgraph PUB["RUNTIME OUTPUT"]
        PUBSTATE["State Publisher"]
        REPORT["Node Reports"]
        EXEC["Local Execution / Workload"]
    end

    NB --> HW
    NB --> RT
    NB --> OS

    HW --> CAP
    RT --> SELF
    OS --> SELF

    CAP --> SNAP
    SELF --> SNAP

    SNAP --> ASSESS
    TEL --> ASSESS
    PERF --> ASSESS
    HEALTH --> ASSESS
    PEER --> ASSESS

    ASSESS --> TEMP
    TEMP --> MACHINE
    MACHINE --> STATE

    STATE --> PUBSTATE
    PUBSTATE --> REPORT
    STATE --> EXEC

    EXEC --> PERF
    EXEC --> TEL
    EXEC --> HEALTH
```

## NodeBrain internal feedback loop

```mermaid
flowchart LR
    HARDWARE["Hardware"] --> DISC["Discovery"]
    DISC --> CAP["Capability Model"]

    RUNTIME["Runtime"] --> SNAP["Resource Snapshot"]
    CAP --> SNAP

    EXEC["Local Workload"] --> METRIC["Observed Metrics"]
    SNAP --> EXEC

    METRIC --> PERF["Performance State"]
    PERF --> ASSESS["Node Assessment"]

    CAP --> ASSESS
    SNAP --> ASSESS
    ASSESS --> STATE["Node State"]

    STATE --> REPORT["Node Report"]
    REPORT --> ROOT["Root"]

    ROOT --> ALLOC["Allocation / Work Assignment"]
    ALLOC --> EXEC
```

## NodeBrain responsibilities by layer

| Layer | NodeBrain responsibility |
|---|---|
| Discovery | Discover platform, hardware, runtime and accelerator capabilities |
| Capability | Build normalized capability representation |
| Resource state | Track current CPU/GPU/memory/network/storage state |
| Telemetry | Collect lightweight runtime observations |
| Performance | Track iteration/throughput/performance information |
| Health | Detect local and peer health conditions |
| Assessment | Combine capability + state + performance evidence |
| State machine | Represent node lifecycle/degradation/recovery state |
| Publishing | Send relevant state to Root |
| Execution awareness | Understand local workload execution and feedback |

---


## Node-side NodeBrain — runtime/telemetry reconstruction

The node-side view is represented from the NodeBrain/communication trace: each worker maintains local resource intelligence, publishes health/prediction knowledge, receives the Root announcement and peer directory, and exchanges health information directly with other workers.

```mermaid
flowchart TB
    NODE["COMPUTE NODE<br/>Windows / macOS / Linux"]

    subgraph NB["NODEBRAIN — LOCAL INTELLIGENCE"]
        direction TB

        DISC["Hardware & Runtime Discovery<br/>CPU • Memory • GPU/accelerator • OS • runtime"]
        CAP["Capability Model<br/>capacity • device type • execution capability"]
        SNAP["Resource Snapshot<br/>current CPU • memory • GPU"]
        PRED["Predictive Resource Assessment<br/>predicted utilization • trajectory • confidence<br/>time-to-threshold • predicted shortage • urgency"]
        HEALTH["Node Health / Readiness<br/>communication status • resource health • resource pressure"]
        KNOW["NodeKnowledge<br/>peer/runtime observations • endpoint • confidence • freshness • generation"]
        PUB["State Publisher<br/>Root reporting + peer health broadcast"]

        DISC --> CAP
        CAP --> SNAP
        SNAP --> PRED
        SNAP --> HEALTH
        PRED --> KNOW
        HEALTH --> KNOW
        KNOW --> PUB
    end

    subgraph ROOTLINK["ROOT CONTROL / COORDINATION"]
        RA["RootAnnouncement<br/>root_node_id • root_endpoint • generation"]
        PD["PeerDirectoryMessage<br/>worker endpoints • status • endpoint validity"]
        ROOT["Root / RootBrain"]
    end

    subgraph P2P["DIRECT NODE ↔ NODE P2P"]
        PA["Peer A"]
        PB["Peer B"]
        PC["Peer C"]
        PH["PeerHealthMessage<br/>resource health • resource pressure<br/>knowledge_dict • confidence • freshness"]
    end

    NODE --> NB

    ROOT --> RA
    RA --> KNOW
    ROOT --> PD
    PD --> KNOW

    PUB --> ROOT
    PUB --> PH
    PH --> PA
    PH --> PB
    PH --> PC

    PA --> PH
    PB --> PH
    PC --> PH
```

### NodeBrain resource-prediction loop

The trace shows NodeBrain repeatedly producing a local assessment containing **current utilization, predicted utilization, trajectory, confidence, freshness, time-to-threshold, predicted-shortage state and urgency**.

```mermaid
flowchart LR
    S["Resource Snapshot"] --> E["Prediction / Estimation"]
    E --> T["Trajectory"]
    E --> C["Confidence"]
    E --> F["Freshness"]
    E --> L["Time to Threshold"]
    E --> PS["Predicted Shortage"]
    E --> U["Urgency"]

    T --> K["NodeKnowledge"]
    C --> K
    F --> K
    L --> K
    PS --> K
    U --> K

    K --> ROOT["Root"]
    K --> PEERS["Peer Nodes"]
```

### Node-side communication flow observed in runtime

```mermaid
sequenceDiagram
    participant NB as NodeBrain
    participant R as Root
    participant P as Peer Nodes

    R->>NB: RootAnnouncement(root_node_id, root_endpoint, generation)
    R->>NB: PeerDirectoryMessage(peers)
    NB->>NB: Update local peer topology
    NB->>NB: Sample CPU / memory / GPU
    NB->>NB: Predict trajectory + confidence + freshness
    NB->>R: NodeKnowledge / telemetry update
    NB->>P: PeerHealthMessage
    P-->>NB: COMM_ACK
    NB-->>R: COMM_ACK / health acknowledgement
    R-->>NB: Updated cluster / peer information
```

### NodeBrain state carried toward Root and peers

```mermaid
flowchart TB
    K["NodeKnowledge"]

    K --> RES["Resource Health<br/>CPU • Memory • GPU"]
    K --> PRED["Prediction<br/>current • predicted • trajectory"]
    K --> CONF["Confidence"]
    K --> FRESH["Freshness"]
    K --> PRESS["Resource Pressure"]
    K --> ENDPOINT["Endpoint<br/>dynamic Tailscale address + service port"]
    K --> STATUS["Communication / Health Status"]
    K --> GEN["Generation / State Version"]

    RES --> REPORT["Health / Telemetry Report"]
    PRED --> REPORT
    CONF --> REPORT
    FRESH --> REPORT
    PRESS --> REPORT
    ENDPOINT --> REPORT
    STATUS --> REPORT
    GEN --> REPORT

    REPORT --> ROOT["Root"]
    REPORT --> PEERS["Peer Nodes"]
```

### Root ↔ NodeBrain ↔ Peer relationship

```mermaid
flowchart LR
    ROOT["ROOT<br/>Control + Scheduling Plane"]

    N1["NodeBrain A"]
    N2["NodeBrain B"]
    N3["NodeBrain C"]

    ROOT -->|RootAnnouncement| N1
    ROOT -->|PeerDirectory| N1
    ROOT -->|RootAnnouncement| N2
    ROOT -->|PeerDirectory| N2
    ROOT -->|RootAnnouncement| N3
    ROOT -->|PeerDirectory| N3

    N1 -->|NodeKnowledge / telemetry| ROOT
    N2 -->|NodeKnowledge / telemetry| ROOT
    N3 -->|NodeKnowledge / telemetry| ROOT

    N1 <-->|PeerHealthMessage| N2
    N2 <-->|PeerHealthMessage| N3
    N3 <-->|PeerHealthMessage| N1

    N1 --> TRAIN["Direct Node ↔ Node Training Data Plane"]
    N2 --> TRAIN
    N3 --> TRAIN
```

**Interpretation:** Root distributes coordination information and consumes node state; NodeBrain owns local observation/prediction; worker nodes exchange peer-health knowledge directly. High-frequency training traffic remains on the direct node-to-node path rather than being routed through the Root dashboard/control plane.

# 29. NodeBrain ↔ Root Interaction

NodeBrain does not replace the Root scheduler.

The relationship is:

```mermaid
sequenceDiagram
    participant H as Node Hardware
    participant N as NodeBrain
    participant R as RootBrain
    participant S as Adaptive Scheduler
    participant E as Local Executor

    H->>N: Hardware / runtime state
    N->>N: Build capability + resource model
    N->>R: Node registration / state report
    R->>S: Cluster state
    S->>S: Capability matching
    S->>S: Allocation decision
    S->>R: Versioned allocation
    R->>N: Work / allocation
    N->>E: Local execution
    E->>N: Runtime metrics
    N->>N: Performance / health assessment
    N->>R: Updated node state
    R->>S: Updated cluster evidence
```

This creates the distributed feedback relationship:

```text
             ROOT
              │
       allocation decision
              ↓
          NodeBrain
              │
        local execution
              ↓
       local observation
              │
              ↓
             ROOT
              │
       adaptive decision
              ↓
          NodeBrain
```

---

# 30. Root + NodeBrain Combined Architecture

The most important control relationship in PALADIN is:

```mermaid
flowchart TB
    subgraph ROOT["ROOT — GLOBAL CONTROL / SCHEDULING"]
        RB["RootBrain"]
        SCH["Adaptive Scheduler"]
        RF["Resource Fabric"]
        TC["Training Coordinator"]
        STATE["Cluster / Allocation State"]
        API["Dashboard API"]
    end

    subgraph A["NODE A — LOCAL INTELLIGENCE"]
        NA["NodeBrain"]
        CA["Capability"]
        TA["Telemetry"]
        EA["Assessment"]
        XA["Execution"]
    end

    subgraph B["NODE B — LOCAL INTELLIGENCE"]
        NB["NodeBrain"]
        CB["Capability"]
        TB["Telemetry"]
        EB["Assessment"]
        XB["Execution"]
    end

    subgraph C["NODE C — LOCAL INTELLIGENCE"]
        NC["NodeBrain"]
        CC["Capability"]
        TCAP["Telemetry"]
        EC["Assessment"]
        XC["Execution"]
    end

    RB --> STATE
    STATE --> SCH
    SCH --> RF
    RF --> TC
    STATE --> API

    SCH --> NA
    SCH --> NB
    SCH --> NC

    NA --> CA
    NA --> TA
    NA --> EA
    EA --> NA
    NA --> XA

    NB --> CB
    NB --> TB
    NB --> EB
    EB --> NB
    NB --> XB

    NC --> CC
    NC --> TCAP
    NC --> EC
    EC --> NC
    NC --> XC

    NA --> RB
    NB --> RB
    NC --> RB
```

The conceptual division is:

```text
ROOT
 ├── Global view
 ├── Cluster state
 ├── Scheduling
 ├── Allocation
 └── Coordination

NODEBRAIN
 ├── Local view
 ├── Capability
 ├── Telemetry
 ├── Performance
 ├── Health
 └── Execution awareness
```

Neither side replaces the other:

```text
NodeBrain = local evidence + local intelligence
Root      = global coordination + scheduling
```

---

# 31. Complete Root–Node–Training Relationship

```mermaid
flowchart TB
    ROOT["ROOT"]
    RB["RootBrain"]
    SCH["Adaptive Scheduler"]
    TC["Training Coordinator"]

    NA["Node A<br/>NodeBrain"]
    NB["Node B<br/>NodeBrain"]
    NC["Node C<br/>NodeBrain"]

    P2P["DIRECT NODE ↔ NODE<br/>Training Data Plane"]

    ROOT --> RB
    RB --> SCH
    SCH --> TC

    TC --> NA
    TC --> NB
    TC --> NC

    NA <--> P2P
    NB <--> P2P
    NC <--> P2P

    NA --> RB
    NB --> RB
    NC --> RB

    P2P -. gradients .-> P2P
    P2P -. tensors .-> P2P
    P2P -. model sync .-> P2P
```

This emphasizes the central PALADIN separation:

```text
                ROOT
                 │
       CONTROL / ALLOCATION
                 │
        ┌────────┼────────┐
        ↓        ↓        ↓
      Node A   Node B   Node C
     NodeBrain NodeBrain NodeBrain
        │        │        │
        └────────┼────────┘
                 ↓
          DIRECT TRAINING
           NODE ↔ NODE
```

---

# 32. Dashboard + NodeBrain Operational View

The operator-visible system can be understood as:

```mermaid
flowchart LR
    subgraph LOCAL["NODE"]
        NB["NodeBrain"]
        CAP["Capabilities"]
        TEL["Telemetry"]
        PERF["Performance"]
        HEALTH["Health"]
    end

    subgraph ROOT["ROOT"]
        RB["RootBrain"]
        SCH["Scheduler"]
        ALLOC["Allocation"]
    end

    subgraph UI["ROOT DASHBOARD"]
        NODEUI["Node Status"]
        CLUI["Cluster Status"]
        ALUI["Allocation View"]
        TEUI["Telemetry"]
        EPUI["Endpoints"]
    end

    NB --> CAP
    NB --> TEL
    NB --> PERF
    NB --> HEALTH

    CAP --> RB
    TEL --> RB
    PERF --> RB
    HEALTH --> RB

    RB --> SCH
    SCH --> ALLOC

    RB --> NODEUI
    RB --> CLUI
    ALLOC --> ALUI
    TEL --> TEUI
    RB --> EPUI
```

This provides a direct explanation of what the operator sees versus what the runtime actually does:

```text
NodeBrain
   ↓
collects evidence

RootBrain
   ↓
aggregates evidence

Adaptive Scheduler
   ↓
makes allocation decisions

Dashboard
   ↓
visualizes authoritative state
```
