Metadata-Version: 2.5
Name: slurm-emulator
Version: 0.9.6
Summary: SLURM command emulator with time manipulation for testing periodic limits
Project-URL: Homepage, https://waldur.com
Project-URL: Documentation, https://docs.waldur.com
Project-URL: Repository, https://github.com/waldur/slurm-emulator
Author-email: Waldur Team <info@waldur.com>
License-Expression: MIT
License-File: LICENSE
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Testing
Classifier: Topic :: System :: Systems Administration
Requires-Python: >=3.9
Requires-Dist: click>=8.0.0
Requires-Dist: fastapi>=0.100.0
Requires-Dist: jinja2>=3.1.6
Requires-Dist: pydantic>=2.0.0
Requires-Dist: python-dateutil>=2.8.0
Requires-Dist: python-multipart>=0.0.20
Requires-Dist: pyyaml>=6.0
Requires-Dist: uvicorn>=0.20.0
Provides-Extra: dev
Requires-Dist: httpx>=0.24.0; extra == 'dev'
Requires-Dist: mypy>=1.0.0; extra == 'dev'
Requires-Dist: pre-commit>=3.0.0; extra == 'dev'
Requires-Dist: pytest-cov>=4.0.0; extra == 'dev'
Requires-Dist: pytest>=7.0.0; extra == 'dev'
Requires-Dist: ruff>=0.12.1; extra == 'dev'
Requires-Dist: tomli>=2.0; (python_version < '3.11') and extra == 'dev'
Requires-Dist: types-python-dateutil; extra == 'dev'
Requires-Dist: types-pyyaml; extra == 'dev'
Requires-Dist: types-requests; extra == 'dev'
Provides-Extra: ssh
Requires-Dist: asyncssh>=2.14.0; extra == 'ssh'
Description-Content-Type: text/markdown

# SLURM Emulator - Time Travel Edition

A comprehensive SLURM command emulator with time manipulation capabilities for testing periodic limits and decay calculations.

## Features

- 🎮 **Interactive CLI** - Full command-line interface with time travel
- ⏰ **Time Manipulation** - Advance time by days, months, or quarters
- 💾 **Usage Injection** - Add specific node-hour usage at any time point
- 🔄 **Decay Calculations** - 15-day half-life fairshare decay simulation
- 🎯 **QoS Management** - Threshold-based QoS switching (normal → slowdown → blocked)
- 📊 **Periodic Limits** - Quarterly allocation with carryover logic
- 🎬 **Scenario Runner** - Complete SLURM_PERIODIC_LIMITS_SEQUENCE.md validation
- 🖥️ **Web Dashboard** - Browser console to view status and drive the emulator ([docs](docs/web-ui.md))
- 🔌 **API Integration** - REST API for waldur-site-agent integration
- 💾 **State Management** - Checkpoint/restore functionality for testing
- ☸️ **Helm Chart** - Single-command Kubernetes deploy of both API planes ([docs](docs/kubernetes.md))

## Quick Start

### Installation

```bash
# Clone the repository
git clone https://github.com/waldur/slurm-emulator.git
cd slurm-emulator

# Install dependencies using uv
uv sync
```

### Interactive CLI (CMD-based)

```bash
# Run with default configuration
uv run slurm-emulator

# Run with SLURM configuration file
uv run slurm-emulator --config examples/slurm.conf

# Advanced features work the same way
uv run slurm-emulator --config examples/custom_slurm.conf

# Validate configuration only
uv run slurm-emulator --validate-only --config /etc/slurm/slurm.conf

🎮 SLURM Emulator - Time Travel Edition (CMD Interface)
Type 'help' or '?' for commands. TAB for auto-completion.
Type 'help <command>' for detailed help on specific commands.

# Note: native emulator commands use underscores (time_advance, account_create,
# usage_inject, ...). The SLURM passthrough commands (sacctmgr, sacct, sinfo,
# sshare) take their arguments with spaces, as on a real cluster.

[default] slurm-emulator> help
# Shows all available commands

[default] slurm-emulator> help time_advance
# Shows detailed help for specific command

[default] slurm-emulator> time_advance 2 months
⏭️  Advanced 2 months

[default] slurm-emulator> account_create test-account "Test Account" 1000
✅ Created account test-account with 1000Nh allocation

[default] slurm-emulator> usage_inject user1 200 test-account
💾 Injected 200.0Nh usage for user1 in test-account at 2024-01-01 00:00:00

[default] slurm-emulator> time_advance 2 months
⏭️  Advanced 2 months
⏰ New time: 2024-03-01 00:00:00

[default] slurm-emulator> usage_inject user1 400 test-account
💾 Injected 400.0Nh usage for user1 in test-account at 2024-03-01 00:00:00

[default] slurm-emulator> limits_calculate test-account
📊 Periodic Limits for test-account:
   Period: 2024-Q1
   Base allocation: 1000Nh
   Total allocation: 1000.0Nh
   Fairshare: 333
   QoS threshold: 1200.0Nh
   Grace limit: 1200.0Nh
   Billing minutes: 60000
```

### Complete Sequence Scenario

Run the full scenario from SLURM_PERIODIC_LIMITS_SEQUENCE.md:

```bash
[default] slurm-emulator> scenario_run sequence --interactive

🎬 Starting SLURM Periodic Limits Sequence Scenario
============================================================

⏸️  Press Enter to execute Step 1: Initial Q1 setup...

📍 Step 1: Initial Q1 2024 Setup
   Setting up 1000Nh quarterly allocation with 20% grace period
   ⚖️  Set fairshare to 333
   🚫 Set GrpTRESMins to 72000 billing-minutes
   🎯 QoS threshold set to 1200.0Nh
   💾 Checkpoint 'initial_setup' created

# ... continues through all 9 steps of the sequence
```

### Direct SLURM Commands

The emulator intercepts and emulates real SLURM commands:

```bash
[default] slurm-emulator> sacctmgr add account test-account description="Test"
 Adding Account(s)
  test-account
 Settings
  Parent     = root
  Description = Test

[default] slurm-emulator> sacctmgr modify account test-account set fairshare=333
 Modified account...
  test-account
 Settings
  fairshare=333

[default] slurm-emulator> sacctmgr modify account test-account set GrpTRESMins=billing=72000
 Modified account...
  test-account
 Settings
  GrpTRESMins=billing=72000

[default] slurm-emulator> sacct --accounts=test-account --starttime=2024-01-01 --endtime=2024-12-31
JobID           JobName  Partition    Account  AllocCPUS      State ExitCode
------------ ---------- ---------- ---------- ---------- ---------- --------
1                 job_1    compute test-acco+         64  COMPLETED      0:0

[default] slurm-emulator> sacct --accounts=test-account -S 2024-01-01 --format=Account,ReqTRES,Elapsed,User --noheader --parsable2
test-account|cpu=64,mem=512G,node=1,billing=64,gres/gpu=4|08:00:00|user1
```

### Energy accounting and `sreport`

Federated allocation portals ask sites for the **energy consumed per SLURM
account** (optionally per user) each month, taken from SLURM's own account
aggregation — `sreport cluster AccountUtilizationByUser -T energy` — rather
than from job rows. The emulator reproduces that pipeline:

- Every usage record carries an `energy` TRES in **joules** (the unit real
  slurmdbd uses for `TRES_ENERGY`). It comes from a per-node power model
  unless a scenario seeds an exact figure:

  ```
  joules = node_hours × 3600 × node_watts(partition) + gpu_hours × 3600 × gpu_watts
  ```

  | Variable | Default | Meaning |
  |----------|---------|---------|
  | `SLURM_EMULATOR_NODE_POWER_W` | `500` | watts per node for partitions not listed below |
  | `SLURM_EMULATOR_PARTITION_POWER_W` | – | per-partition overrides, e.g. `compute=400,gpu=900` |
  | `SLURM_EMULATOR_GPU_POWER_W` | `300` | extra watts per allocated GPU (`GRES/gpu` hours) |

  Jobs completed through the scheduler (`sbatch` / `POST /job/submit`) use
  the same model with their partition; `usage_inject` records run in
  `compute`.

- `POST /api/submit-report` accepts `energy` (joules) in `usage` or in each
  `users` entry, plus an optional `partition`; an explicit value bypasses the
  model so a month's total can be asserted exactly. The
  `regular_access_energy` scenario seeds one month of CPU + GPU usage with
  known energy for this purpose.

- `sreport cluster AccountUtilizationByUser` is emulated with the real
  client's grammar — `start=`/`end=` (same time specs as `sacct`, window on
  the *simulated* clock, `end` exclusive, defaults yesterday→today),
  `accounts=`, `users=`, `cluster=`/`-M`/`-a`, `format=`, `-T`/`--tres=`
  (comma list; `cpu`, `mem`, `node`, `billing`, `gres/gpu`, `energy`, `ALL`),
  `-t Seconds|Minutes|Hours|Percent|SecPer|MinPer|HourPer`, `-p`/`-P`/`-n`.
  One row per association and TRES, the account total first, then its users,
  then sub-accounts:

  ```bash
  $ sreport cluster AccountUtilizationByUser start=2024-03-01 end=2024-04-01 \
      -T energy -t Seconds --parsable2 -n accounts=regular_access
  default|regular_access|||energy|424800000
  default|regular_access|alice||energy|180000000
  default|regular_access|bob||energy|244800000
  ```

  Two things to know from the real source
  (`src/sreport/sreport.c`): there is **no `-t Joules`** — the energy TRES is
  stored in the same slot the time units divide, so joules are what
  `-t Seconds` prints (`-t Minutes` gives joules/60; an unknown format prints
  `unknown time format X` and the report continues in Minutes); and unknown
  TRES names are silently dropped — only an empty list fails
  (`sreport: fatal: No valid TRES given`, exit 1). Only this one report is
  emulated; the header block is printed unless `-n`, also in parsable mode.

- `sacct` gains the real `ConsumedEnergy` / `ConsumedEnergyRaw` columns
  (joules per job, scaled K/M/G unless `--noconvert`). The ReqTRES/AllocTRES
  string does not include `energy`, so the site agent's
  `--format=Account,ReqTRES,Elapsed,User` output is unchanged. `sshare
  GrpTRESRaw` sums it like any other TRES.

## Web Dashboard

A lightweight, browser-based control console is mounted on the API server at
**`http://localhost:8080/ui/`**. It shows live status (time/period, accounts,
usage, QoS, jobs, cluster config) and provides full control — advance/set time,
create/edit accounts, inject usage, apply periodic settings, inline QoS editing,
add/remove account users, and a **scenario editor** to build/adjust and run
scenarios. It shares the same in-memory state as the CLI and JSON API.

```bash
# Start the server (Basic-auth credentials via env vars)
SLURM_EMULATOR_UI_USER=admin SLURM_EMULATOR_UI_PASSWORD=secret \
  uv run uvicorn emulator.api.emulator_server:app --host 0.0.0.0 --port 8080
# then open http://localhost:8080/ui/
```

All `/ui` routes are protected by HTTP Basic auth
(`SLURM_EMULATOR_UI_USER` / `SLURM_EMULATOR_UI_PASSWORD`, default `admin`/`admin`
with a startup warning). See **[docs/web-ui.md](docs/web-ui.md)** for a full
walkthrough with screenshots.

![SLURM Emulator dashboard](docs/screenshots/01-dashboard.png)

## API Integration

Start the API server for waldur-site-agent integration:

```bash
# From the slurm-emulator directory
uv run uvicorn emulator.api.emulator_server:app --host 0.0.0.0 --port 8080
```

### API Endpoints

- `POST /api/apply-periodic-settings` - Apply periodic limits settings
- `POST /api/downscale-resource` - Set QoS to slowdown
- `POST /api/restore-resource` - Restore QoS to normal
- `POST /api/submit-report` - Submit usage reports
- `GET /api/status` - Get emulator status
- `POST /api/time/advance` - Advance emulator time

### Example API Usage

```bash
# Apply periodic settings (from Waldur Mastermind)
curl -X POST http://localhost:8080/api/apply-periodic-settings \\
  -H "Content-Type: application/json" \\
  -d '{
    "resource_id": "slurm_account_123",
    "fairshare": 333,
    "grp_tres_mins": {"billing": 72000},
    "qos_threshold": {"billing": 1000}
  }'

# Submit usage report (from site agent)
curl -X POST http://localhost:8080/api/submit-report \\
  -H "Content-Type: application/json" \\
  -d '{
    "resource_id": "slurm_account_123",
    "usage": {"billing": 167},
    "billing_period": "2024-01-01",
    "date": "2024-01-31T23:59:59Z",
    "users": {
      "user1": {"billing": 100},
      "user2": {"billing": 67}
    }
  }'

# Advance time for testing
curl -X POST "http://localhost:8080/api/time/advance?months=3"
```

## SLURM REST API Emulation (slurmrestd)

The emulator also serves the Slurm 26.05 REST API (`slurmrestd`,
data parser `v0.0.45`) on port 6820, backed by the same state as the
CLI commands and the control API:

```bash
uv run slurmrestd-emulator
# or: uv run uvicorn emulator.api.slurmrestd.app:app --host 0.0.0.0 --port 6820
```

### Endpoint families

- `/slurmdb/v0.0.45/...` — accounting: `accounts`, `users`,
  `associations`, `qos`, `tres`, `clusters`, `jobs` (one job per usage
  record, matching `sacct` output), `ping`, `diag`, `config`. Write
  support (POST/DELETE) covers everything Waldur drives via `sacctmgr`.
- `/slurm/v0.0.45/...` — controller read paths: `jobs` (+ `DELETE
  /job/{job_id}` as the `scancel` equivalent), `nodes`, `partitions`
  (static topology matching `sinfo`), `shares`, `ping`, `diag`, `conf`;
  `reservations`/`licenses` are empty stubs.
- `/openapi.json`, `/openapi`, `/openapi/v3` — generated self-description.

Responses use the real envelope (`meta`/`errors`/`warnings`, payload
keys and field names from the v0.0.45 data parser). Unsupported URL
versions (e.g. `v0.0.45`), unknown paths, and auth failures reject
with slurmrestd's plain-text errors and exit statuses.

### Authentication

Requests need an `X-SLURM-USER-TOKEN` header (or `Authorization:
Bearer`); `X-SLURM-USER-NAME` optionally names the user. By default
any non-empty token is accepted. Set `SLURM_EMULATOR_JWT_KEY` to
enforce real HS256 verification. Mint tokens via the control API
(`scontrol token` stand-in):

```bash
curl -X POST http://localhost:8080/api/token \
  -H "Content-Type: application/json" -d '{"username": "alice"}'

curl http://localhost:6820/slurmdb/v0.0.45/accounts/ \
  -H "X-SLURM-USER-TOKEN: <token>"
```

### State sharing and concurrency

Both API servers and the CLI commands persist to the same JSON state
files (`/tmp/slurm_emulator_db.json`, `/tmp/slurm_emulator_time.json`;
override with `SLURM_EMULATOR_STATE_FILE` / `SLURM_EMULATOR_TIME_FILE`).
The slurmrestd app reloads state on every request and saves after
writes — file locking prevents torn writes, but concurrent writers are
last-writer-wins. Note the control API on 8080 loads state once at
startup, so it can serve stale reads after REST/CLI writes.

The Docker image runs both servers (ports 8080 and 6820) via
`scripts/docker-entrypoint.sh`.

## Run on Kubernetes (via Helm)

A published Helm chart deploys the emulator as a single-replica `Deployment` +
`ClusterIP` Service exposing the control API (8080), the slurmrestd plane
(6820), and optionally the SSH plane (2222). Consumers in the same cluster
reach it at `http://<release>-slurm-emulator.<ns>.svc.cluster.local:8080`.

```bash
helm repo add slurm-emulator https://waldur.github.io/slurm-emulator/
helm install se slurm-emulator/slurm-emulator \
  --namespace se --create-namespace --wait
helm test se -n se        # probes both API planes
```

Set `auth.uiPassword` (dashboard) and `auth.jwtKey` (slurmrestd) before exposing
it anywhere shared — both default to accepting anything. `persistence.enabled=true`
keeps the clock, accounts, and usage records across restarts.

See [`docs/kubernetes.md`](docs/kubernetes.md) for the full operator guide
(credentials, persistence, SSH plane, Ingress/Gateway API, troubleshooting). The
chart source lives at [`charts/slurm-emulator/`](charts/slurm-emulator) — also
installable from disk via `helm install se ./charts/slurm-emulator`.

## SSH Filesystem Plane

For clients that expect a login node in addition to `slurmrestd` (notably
FireCREST v2), the emulator ships an asyncssh server that provides filesystem
operations and dispatches the Slurm CLI commands (`sacctmgr`, `sacct`, `sinfo`,
`sshare`, `scancel`, `sbatch`) against the same shared state:

```bash
uv run --extra ssh slurm-ssh-emulator   # asyncssh server on port 2222
```

It shares the same JSON state files as the CLI, control API, and slurmrestd.

### NSS identity (`SLURM_EMULATOR_NSS=1`)

By default the emulator has no uid/gid model: every user is `1000` and its
own group. Setting `SLURM_EMULATOR_NSS=1` makes it resolve user names
through the OS name service switch (`getpwnam`/`getgrgid`, the calls real
Slurm makes) — inside the Docker image that is libc → sssd → LDAP, with a
reference `examples/nss/sssd.conf` and a one-user `examples/nss/seed.ldif`.
With the mode on:

- `id` over SSH prints the coreutils shape FireCREST's `/status/userinfo`
  parses — `uid=9001(hpc_9001) gid=9001(hpc_9001) groups=9001(hpc_9001)` —
  and honours `-u`/`-g`/`-G`/`-n`; an unknown user exits 1;
- `POST /slurm/v0.0.45/job/submit` and `sbatch` record the submitter's
  uid/gid/group on the job (`user_id`/`group_id`/`group_name`,
  `scontrol show job` `UserId=name(uid) GroupId=group(gid)`, sacct
  `UID`/`GID`/`Group`) and refuse a name the OS cannot resolve
  (`ESLURM_USER_ID_UNKNOWN`, HTTP 422, like the real `USER_ID` parser);
- `sacctmgr add user` for an unresolvable name stops with
  ` There is no uid for user 'x'` (exit 1) unless `-i`/`--immediate` is given,
  mirroring real sacctmgr's prompt; the REST `/slurmdb` user upsert is not
  gated, so a site agent can create accounts before the directory entry exists;
- `sreport` fills `Proper Name` from the gecos field;
- independently of NSS, `SLURM_EMULATOR_ACCOUNTING_ENFORCE=associations` (the
  default everywhere; `slurm.conf` itself leaves it unset, but every managed
  production cluster sets it) makes
  `POST /job/submit` and `sbatch` refuse a user with no association for the
  requested — or, when none is requested, default — account:
  `ESLURM_INVALID_ACCOUNT` "Invalid account or account/partition combination
  specified" (HTTP 422), so a person removed from a project can no longer
  submit against it. **Behaviour change in 0.10:** earlier releases let any
  user submit and fell back to `root`; set
  `SLURM_EMULATOR_ACCOUNTING_ENFORCE=none` to get that back. The check is
  cluster-scoped, so a deployment whose site agent uses `cluster_name: linux`
  must set `SLURM_EMULATOR_CLUSTER_NAME=linux` (chart `clusterName`) or the
  agent's rows land on a cluster the submit never looks at. As in slurmdbd, a
  user's default association cannot be removed while other associations remain
  (`You can not remove the default account of a user`), and removing the last
  one removes the user;
- when the emulator runs as root (the Docker image does), shell commands over
  SSH run *as the login user* — uid, gid, supplementary groups, `$HOME` —
  so `id`, `stat`, `chown` and file ownership behave like a real login node;
  a leading coreutils `timeout N` wrapper (which FireCREST puts in front of
  every SSH command) is stripped before matching the emulated Slurm binaries.

Try it with the FireCREST UI stack:

```bash
cd examples/firecrest/ui
docker compose -f docker-compose.yml -f docker-compose.nss.yml up -d
ldapadd -x -H ldap://localhost:3890 -D cn=admin,dc=example,dc=org -w admin -f ../../nss/seed.ldif
docker compose exec slurm id hpc_9001
```

| Variable | Default | Meaning |
|----------|---------|---------|
| `SLURM_EMULATOR_NSS` | unset | `1` resolves users through the OS NSS (sssd → LDAP in the image) |
| `SLURM_EMULATOR_CLUSTER_NAME` | `default` | slurm.conf `ClusterName`: the cluster associations, submitted jobs and `/slurm/…/conf` are filed under; set it to the site agent's `cluster_name` so the association check on submit sees the agent's rows |
| `SLURM_EMULATOR_NSS_CACHE_TTL` | `60` | seconds a resolved identity (uid/gid/groups) is trusted before NSS is asked again; misses are retried after 5 s |
| `SLURM_EMULATOR_ACCOUNTING_ENFORCE` | `associations` | `AccountingStorageEnforce`: with `associations` (or `limits`/`qos`/`safe`/`wckeys`) a submit by a user with no association for the account fails with `ESLURM_INVALID_ACCOUNT`; `none` = permissive pre-0.10 fallback to the default account, then `root` |
| `SLURM_EMULATOR_SSSD_CONF` | `/etc/slurm-emulator/sssd.conf` | sssd config the entrypoint installs as a private root-only copy (falls back to `/etc/sssd/sssd.conf`) |

In Kubernetes the chart exposes it as `nss.enabled` + `nss.sssdConfSecret`;
sssd runs as root, so the option excludes a non-root `securityContext`.

### Running FireCREST v2 against the emulator

The scheduler plane (slurmrestd) plus the SSH filesystem plane let the emulator
stand in for a real cluster for [eth-cscs/firecrest-v2](https://github.com/eth-cscs/firecrest-v2).
See [`examples/firecrest/`](examples/firecrest/) for the conformance matrix, a
docker-compose overlay, and contract/integration test harnesses.

## Waldur Site Agent Integration

Configure waldur-site-agent to use the emulator:

```yaml
# waldur-site-agent-config.yaml
offerings:
  - name: "SLURM HPC Cluster - Emulator"
    backend_type: "slurm"
    backend_settings:
      # Enable emulator mode
      emulator_mode: true
      emulator_base_url: "http://localhost:8080"

      # Override SLURM commands to use emulator
      command_prefix: ["python", "/path/to/slurm-emulator/emulator/commands/dispatcher.py"]

      # Periodic limits configuration
      periodic_limits:
        enabled: true
        limit_type: "GrpTRESMins"
        tres_billing_enabled: true
        tres_billing_weights:
          CPU: 0.015625
          Mem: 0.001953125G
          "GRES/gpu": 0.25
        fairshare_decay_half_life: 15
        api_endpoints:
          apply_periodic_settings: "http://localhost:8080/api/apply-periodic-settings"
          downscale_resource: "http://localhost:8080/api/downscale-resource"
          restore_resource: "http://localhost:8080/api/restore-resource"
```

## SLURM Configuration Support

The emulator now supports real SLURM configuration files to match actual deployment behavior:

### Loading Configuration

```bash
# Use system SLURM configuration
uv run slurm-emulator --config /etc/slurm/slurm.conf

# Use custom configuration
uv run slurm-emulator --config examples/slurm.conf

# Validate configuration
uv run slurm-emulator --validate-only --config slurm.conf
```

### Supported Configuration Parameters

The emulator parses and applies these SLURM configuration parameters:

**Priority and Decay Settings:**
- `PriorityDecayHalfLife` - Fairshare decay half-life (e.g., "15-00:00:00")
- `PriorityUsageResetPeriod` - Usage reset period ("None" for manual reset)
- `PriorityWeightFairShare` - Fairshare weight for priority calculations
- `PriorityWeightQOS` - QoS weight for priority calculations
- `FairShareDampeningFactor` - Dampening factor for fairshare

**TRES Billing:**
- `TRESBillingWeights` - Billing weights (e.g., "CPU=0.015625,Mem=0.001953125G,GRES/gpu=0.25")

**Priority Flags:**
- `PriorityFlags` - Priority calculation flags (e.g., "MAX_TRES,NO_NORMAL_ASSOC")

### Example Configuration

```bash
# SLURM Configuration
PriorityDecayHalfLife   = 15-00:00:00
PriorityUsageResetPeriod = None # manual reset via sacctmgr RawUsage=0
PriorityWeightFairShare = 259200
PriorityWeightQOS       = 500000
FairShareDampeningFactor = 3
TRESBillingWeights="CPU=0.015625,Mem=0.001953125G,GRES/gpu=0.25"
PriorityFlags=MAX_TRES,NO_NORMAL_ASSOC
```

## Understanding Decay Calculations

The emulator implements SLURM's fairshare decay using the configured half-life:

```python
# Decay formula matches SLURM's implementation
decay_factor = 2 ** (-days_elapsed / half_life_days)

# With default 15-day half-life, after 90 days (1 quarter):
decay_factor = 2 ** (-90 / 15) = 0.0156 (1.56%)

# With 7-day half-life, after 90 days:
decay_factor = 2 ** (-90 / 7) = 0.000135 (0.01%)
```

**Example with 15-day half-life**: User consumes 2000 hours in Q1. After Q1 ends (90 days later):
- Original impact: 2000 hours
- Decayed impact: 2000 × 0.0156 = 31 hours equivalent
- Q2 allocation: 1000 + (1000 - 31) = 1969 hours available

## Key Commands Reference

### Time Manipulation
```bash
time_show                             # Show current time and period
time_advance <amount> <unit>          # Advance time (units: days, months, quarters)
time_set YYYY-MM-DD [HH:MM:SS]       # Set specific date/time

# Examples:
time_advance 2 months
time_advance 30 days
time_set 2024-05-20
```

### Usage Simulation
```bash
usage_inject <user> <amount> [account]  # Inject node-hour usage
usage_show [account] [period]           # Show usage summary with user breakdown

# Examples:
usage_inject user1 200 test-account
usage_show test-account
usage_show test-account 2024-Q1
```

### Account Management
```bash
account_create <name> [description] [allocation]  # Create account
account_list                                      # List all accounts with status
account_show <name>                               # Show detailed account info
account_delete <name>                             # Delete account

# Examples:
account_create test "Test Account" 1000
account_show test
account_list
```

### QoS Management
```bash
qos_show [account]                    # Show QoS status and details
qos_set <account> <qos>              # Set QoS level (normal/slowdown/blocked)
qos_check [account]                  # Check thresholds and auto-update QoS

# Examples:
qos_check test-account
qos_set test-account slowdown
qos_show test-account
```

### Limits Calculation
```bash
limits_calculate [account]           # Calculate and display periodic limits

# Example:
limits_calculate test-account
```

### Scenario Management
```bash
scenario_list [type]                 # List scenarios (optionally filter by type)
scenario_describe <name>             # Show detailed description and learning objectives
scenario_steps <name>                # Show step-by-step command breakdown
scenario_run <name>                  # Run scenario automatically
scenario_run <name> --interactive    # Run with confirmation prompts
scenario_run <name> --step-by-step   # Run with detailed step output
scenario_search <query>              # Search scenarios by keyword

# Examples:
scenario_list qos_management
scenario_describe qos_thresholds
scenario_run qos_thresholds --step-by-step
scenario_search decay
```

### Configuration Management
```bash
config_show                          # Show current SLURM configuration
config_reload <path>                 # Hot-reload configuration file

# Examples:
config_show
config_reload examples/slurm.conf
```

### State Management
```bash
cleanup_all                          # Clean all accounts and reset to fresh state
cleanup_scenario <name>              # Clean specific scenario accounts
cleanup_account <name>               # Clean specific account completely

# Examples:
cleanup_all
cleanup_scenario qos_thresholds
cleanup_account test-account
```

### SLURM Commands
```bash
sacctmgr <args>                      # Run sacctmgr command
sacct <args>                         # Run sacct command
sinfo <args>                         # Run sinfo command
sshare <args>                        # Run sshare command
sreport <args>                       # Run sreport command (cluster AccountUtilizationByUser)

# Examples:
sacctmgr list accounts
sacctmgr modify account test set fairshare=333
sacct --accounts=test --format=Account,User,Elapsed --noheader --parsable2
sshare -A test --parsable2
sreport cluster AccountUtilizationByUser start=2024-01-01 end=2024-02-01 -T energy -t Seconds -P -n
```

## Testing Scenarios

### Basic Usage Pattern
```bash
# Setup with specific configuration
uv run slurm-emulator --config examples/slurm.conf

# In emulator CLI:
time_set 2024-01-01
account_create test-account "Test" 1000

# Month 1: Light usage
usage_inject user1 100 test-account
time_advance 1 months

# Month 2: Heavy usage
usage_inject user1 600 test-account
limits_calculate test-account
qos_check test-account

# Quarter transition
time_advance 1 months
limits_calculate test-account
```

### Configuration Testing
```bash
# Test different decay rates
uv run slurm-emulator --config examples/custom_slurm.conf

# Compare configurations
uv run slurm-emulator --validate-only --config examples/slurm.conf
uv run slurm-emulator --validate-only --config examples/custom_slurm.conf
```

### Decay Validation
```bash
# Q1: Heavy usage
time_set 2024-01-01
account_create test-account "Test" 1000
usage_inject user1 1500 test-account

# Q2: Check decay impact
time_set 2024-04-01
limits_calculate test-account
# Should show ~23Nh effective previous usage (1500 * 0.0156)
```

### QoS Threshold Testing
```bash
# Setup with 1000Nh allocation (1200Nh threshold with 20% grace)
account_create test-account "Test" 1000
qos_show test-account  # Should show "normal"

usage_inject user1 1100 test-account
qos_check test-account  # Should show approaching threshold

usage_inject user1 200 test-account  # Total: 1300Nh
qos_check test-account  # Should trigger slowdown QoS
```

## Architecture

```
slurm-emulator/
├── emulator/
│   ├── core/
│   │   ├── time_engine.py          # Time manipulation
│   │   ├── database.py             # In-memory state
│   │   ├── slurm_config.py         # SLURM config parsing
│   │   ├── scheduler.py            # Submitted-job lifecycle
│   │   └── usage_simulator.py      # Usage injection
│   ├── commands/
│   │   ├── sacctmgr.py             # sacctmgr emulator
│   │   ├── sacct.py                # sacct emulator
│   │   ├── sshare.py               # sshare emulator
│   │   ├── print_fields.py         # SLURM-style column formatting
│   │   └── dispatcher.py           # Command routing / script entry points
│   ├── periodic_limits/
│   │   ├── calculator.py           # Decay & carryover
│   │   └── qos_manager.py          # QoS management
│   ├── scenarios/
│   │   ├── sequence_scenario.py    # Complete scenario
│   │   ├── scenario_registry.py    # Scenario discovery & running
│   │   └── limits_configuration_scenarios.py
│   ├── cli/
│   │   ├── main.py                 # Interactive CLI entry point
│   │   └── cmd_cli.py              # CMD-based CLI
│   └── api/
│       ├── emulator_server.py      # Control REST API (port 8080)
│       ├── ui/                     # HTMX + Jinja2 web dashboard (/ui)
│       ├── slurmrestd/             # slurmrestd emulation (port 6820)
│       └── ssh/                    # SSH filesystem + CLI plane (port 2222)
├── scripts/
│   ├── release.py                  # Release management
│   ├── changelog.sh                # Changelog generation
│   ├── generate_changelog_data.py  # Commit data collection
│   └── prompts/
│       └── changelog-prompt.md     # Changelog prompt template
├── examples/
│   └── firecrest/                  # FireCREST v2 integration example
└── tests/                          # Test suites
```

## Development

### Running Tests

```bash
uv run pytest
```

### Launching as a specific Slurm release

```bash
SLURM_EMULATOR_SLURM_VERSION=25.11 uv run slurmrestd-emulator   # serves /slurm/v0.0.44/, release 25.11.7
SLURM_EMULATOR_SLURM_VERSION=24.11 uv run pytest                 # run the suite as 24.11
```

Tracked releases: `24.11`, `25.05`, `25.11`, `26.05` (default), `master`. The
choice drives the slurmrestd URL prefix, `meta.slurm.release`, and the few
response shapes that changed between releases (`emulator/slurm_version.py`).

### Parity with real Slurm

Every emulated behaviour cites SchedMD's source as `slurm://<path>#<symbol>[@versions]`
and is checked against several Slurm releases (26.05 primary; 25.11, 25.05, 24.11 and
`master`). `uv run scripts/slurm_src.py update` builds the local source cache,
`uv run scripts/check_slurm_refs.py --summary` verifies the references. See
[`docs/slurm-parity.md`](docs/slurm-parity.md).

### Releasing

```bash
# Full release: update version, generate changelog, tag, push
uv run scripts/release.py release X.Y.Z

# Skip changelog generation
uv run scripts/release.py release X.Y.Z --skip-changelog
```

Pushing the tag triggers GitLab CI/CD for testing and PyPI publishing.

### Adding New Scenarios

```python
# Create new scenario class
class CustomScenario:
    def __init__(self, time_engine, database):
        self.time_engine = time_engine
        self.database = database

    def run_scenario(self):
        # Implement scenario steps
        pass
```

### Extending Commands

```python
# Add new SLURM command support
class NewCommandEmulator:
    def handle_command(self, args):
        # Implement command logic
        return "command output"
```

## Troubleshooting

### State Persistence

Emulator state is saved to:
- `/tmp/slurm_emulator_time.json` - Current time
- `/tmp/slurm_emulator_db.json` - Database state

### Common Issues

**"Account not found"**: Create account first with `account_create`
**"No usage records"**: Inject usage with `usage_inject`
**"Time not advancing"**: Check time with `time` command
**"API connection failed"**: Ensure server is running on port 8080

### Reset Emulator

```bash
rm /tmp/slurm_emulator_*.json
slurm-emulator
# Start fresh
```

## License

MIT License - See LICENSE file for details.
