Metadata-Version: 2.5
Name: cryptolabs-sre-agent
Version: 0.4.0
Summary: CryptoLabs SRE reverse-tunnel agent — connects your server to CryptoLabs AI SRE
Project-URL: Homepage, https://cryptolabs.co.za
Project-URL: Discord, https://discord.gg/7yeHdf5BuC
Author-email: CryptoLabs <support@cryptolabs.co.za>
License-Expression: MIT
Requires-Python: >=3.10
Requires-Dist: aiohttp>=3.9
Requires-Dist: click>=8.1
Requires-Dist: websockets>=13.0
Provides-Extra: cli
Requires-Dist: keyring>=25; extra == 'cli'
Requires-Dist: segno==1.6.6; extra == 'cli'
Description-Content-Type: text/markdown

# CryptoLabs SRE Agent

A lightweight reverse-tunnel agent that connects your server to the
[CryptoLabs AI SRE](https://www.cryptolabs.co.za/ai-sre/) investigation
platform — **no SSH keys, no port forwarding, no firewall changes**.

## Quick Start

```bash
# Install (requires Python 3.10+)
pipx install cryptolabs-sre-agent

# Connect using the token from your CryptoLabs SRE dashboard
# Option 1: token file (recommended — not visible in process list)
echo "YOUR_TOKEN" > /etc/sre-agent.token
chmod 600 /etc/sre-agent.token
sudo $(which sre-agent) connect --token-file /etc/sre-agent.token

# Option 2: environment variable
export SRE_AGENT_TOKEN="YOUR_TOKEN"
sudo $(which sre-agent) connect

# Option 3: CLI flag (token visible in `ps aux` — use for quick testing only)
sudo $(which sre-agent) connect --token YOUR_TOKEN
```

## How it Works

1. **Install** — `pipx install cryptolabs-sre-agent`
2. **Run** — `sudo $(which sre-agent) connect --token-file /etc/sre-agent.token`
3. The agent opens an outbound WebSocket (`wss://`) to `sre.ai.cryptolabs.co.za`
4. The token links this agent to your registered server on the dashboard
5. Commands are sent by the SRE system, executed locally, results returned

## Security

### Network
- **TLS enforced** — agent refuses non-`wss://` connections in production
- **Outbound only** — no inbound ports opened on your machine
- **Heartbeat timeout** — server drops idle connections after 90s of silence

### Authentication
- **HMAC-SHA256 signed tokens** — cryptographically bound to a single server
- **Token hashed in storage** — only a SHA-256 hash is stored server-side (not the raw token)
- **Rate limiting** — 10 failed auth attempts per IP triggers a 5-minute lockout
- **10-second auth deadline** — connections that don't authenticate are dropped

### Execution Safety
- **Server-side allowlist** — commands are validated by the SRE API's command
  allowlist *before* reaching the agent
- **Agent-side blocklist** — defense-in-depth: the agent independently blocks
  catastrophic commands (`rm -rf /`, `mkfs`, `dd` to disk, fork bombs,
  `halt`/`poweroff`, writes to `/etc/shadow`, `/root/.ssh/authorized_keys`)
- **Output truncation** — stdout/stderr capped at 1 MB per field
- **Concurrent limit** — max 10 commands in parallel to prevent resource exhaustion
- **Command timeout** — every command has a configurable timeout (default 300s)

### Token Handling
- **`--token-file` recommended** — avoids exposing token in process list
- **Environment variable** (`SRE_AGENT_TOKEN`) — alternative to file
- **CLI flag** (`--token`) — agent warns when this is used

### Revocation
- Deleting the server from the dashboard instantly disconnects the agent
- Token revocation via API (`DELETE /agent/token/{server_id}`)
- Agent auto-shuts down when server sends disconnect message

## Approving a fix

A fix that CryptoLabs AI SRE proposes does not run on your server until **root on that
server** approves that exact plan. The agent enforces this itself for every command of a fix
run, so a compromised web account (WordPress, the admin API, a stolen agent token) cannot start
a fix. Investigation commands are not affected.

```bash
sudo sre-agent approvals                       # the fixes waiting for approval on this server
sudo sre-agent approve <incident_id>           # review the plan, type this server's host name
sudo sre-agent approve <incident_id> --override-risk-review   # only if the risk review rejected it
```

- `approve` shows the incident, the risk review verdict and summary, and every step with its
  full command (control and invisible characters are shown as escapes). You confirm by typing
  this server's own host name. The plan hash is calculated on the server from the steps you
  were shown, never taken from CryptoLabs.
- If the risk review rejected the plan, `approve` refuses unless you add
  `--override-risk-review`. That decision is yours, on your server.
- The approval is one file, `/var/lib/sre-agent/approvals/<incident_id>.json` (directory 0700,
  file 0600). It allows **one run** of **that plan** within **2 hours**. A changed plan, a second
  run or an expired approval is refused with exit code `-3` and
  `not approved on this box: run sudo sre-agent approve <incident_id>`; nothing is started.
- The file is written before CryptoLabs is told. If CryptoLabs refuses the approval (an HTTP 4xx
  answer), the file is removed again and its message is printed. If CryptoLabs does not answer
  (a timeout, a connection error or a 5xx), the file stays, because the fix may already be running;
  run `sudo sre-agent approvals` to see the status. It expires by itself after 2 hours.
- Needs the agent token in `/etc/sre-agent.token` (or `SRE_AGENT_TOKEN_FILE`) and `https` (the API
  base is `SRE_RELAY_BASE`, as for `setup`). CryptoLabs starts a fix only on a server whose
  connected agent reports that it enforces this, so update the agent first.

## Remote terminal (local consent)

The agent can open interactive root shells and run commands for the machine owner
through CryptoLabs (`srecli`). **It is off by default and only you can turn it on, on
the machine itself.**

```bash
sudo sre-agent terminal on      # allow (prints exactly what this allows)
sudo sre-agent terminal off     # refuse
sre-agent terminal status       # prints on/off and, when off, why
```

- Consent is the file `/etc/sre-agent.conf` (`[terminal]` / `enabled = true`). It counts
  only if it is a regular file owned by root and not writable by group or others;
  anything else (missing, wrong owner, unreadable, `enabled` not `true`) means off.
  The agent re-reads it for every request and every 5 seconds during a session; turning
  it off ends open sessions.
- Each session uses its own outbound `wss://` connection; the agent never takes a
  host or URL from the server. Sessions end when the connection drops, on timeout
  (PTY: 30 min idle, 8 h maximum), or when you turn consent off. At most 3 run at once.
- Read-only commands run inside a `systemd-run` sandbox (no writes outside a private
  `/tmp`, no signals to other processes, no docker, no kernel/mount/clock calls).
  Read-only is not confidential: output can contain secrets.
- This switch protects against mistakes and policy, not against a compromised CryptoLabs
  server, which can already run root commands through the agent. The hard stop is
  `sudo systemctl stop sre-agent`.

Protocol: `docs/contracts/2026-10-01-terminal-agent-protocol.html`.

## srecli: the owner's command-line client

`srecli` runs on **your own computer** (not on the servers) and reaches the machines
you registered with CryptoLabs AI SRE. It never needs root and is installed from the
same package as the agent:

```bash
pipx install 'cryptolabs-sre-agent[cli]'   # [cli] adds keyring (the macOS Keychain) and segno (the QR code)
srecli login --code 7HQ2M-9PZ4X-C1VB8-NN3KD-6RT0Y   # a one-time code from SSH on wk01, see below
srecli servers                              # hostname, online, terminal setting, adopted, linked, agent version, ready
srecli show venter-h100x4-01                # details of one server (a hostname or a server id)
srecli exec venter-h100x4-01 -- nvidia-smi -L
srecli elevate venter-h100x4-01             # ask for write access; approve it on your phone
srecli exec venter-h100x4-01 -w --timeout 120 -- systemctl restart nginx
srecli connect venter-h100x4-01             # live root shell (needs the same approval); Ctrl-] or Enter ~ . leaves
srecli logout                               # revokes the key in use (SRECLI_KEY first, else the stored key)
```

- `exec SERVER [-w] [--timeout N] -- COMMAND...`: the words after `--` are joined with single
  spaces and run as one shell command. Output goes to your stdout and stderr, and piped stdin is
  streamed. Leaving on purpose decides the status: Ctrl-] in `connect` is `0`, Ctrl-C is `130`,
  SIGTERM is `143`, SIGHUP is `129` (Enter `~` `.` leaves like Ctrl-]), whatever the server sends
  afterwards. The exit status of everything else is the remote one, as this table says (operator contract, section 8; the
  rules are applied in this order, and the first that fits decides):

  | Status | When |
  | --- | --- |
  | `124` | The command timed out, whatever exit code or signal it ended with. Checked first, so a timed-out command never reads as success. |
  | `255` | The session was cut short (it ended for any reason but a normal exit: the key or the grant ended, the setting was lowered, the command was closed) and the server's answer would give `0`. A cut-short session never reads as success. |
  | the command's own status | The command ended by itself with exit status N: N. |
  | `128` + N | The command (or its unit) ended by signal N, for example `143` for SIGTERM, which is what a read-only command that was stopped reports. |
  | `255` | The server's final answer carries neither an exit status nor a signal: the connection was lost without a final answer, the broker gave up waiting for the agent (a command that ignores SIGTERM for more than 3 s; the reason, for example `grant ended`, is on stderr), or the agent reported an error. |
  | `126` | Refused (reason and message on stderr), for example blocked, no grant, or the terminal is off. |
  | `2` | A usage error, or a hostname that two servers share. |
  | `1` | Any other local error, including "not logged in", `Your key is not valid. Run: srecli login --code CODE (get a code over SSH on wk01)`, `Rate limited, try again later`, and a server whose remote terminal is not enabled. |

  One status is not in this table, because it is srecli's own and not an answer of the server: `141`
  (128 + SIGPIPE), when the reader of your output goes away (a broken pipe, for example `| head`) and the
  command had not ended by itself (see the reader bullet below).

  A `0` therefore means the command ran to its end and said `0`; `143` or `255` after a revoke
  means it was stopped, not that it succeeded.
- A slow reader never ends the session. A pipe into `less`, Ctrl-S on a terminal or a slow disk
  only makes srecli hold the output it cannot write yet: in memory, then in a private temporary
  file (mode 0600, removed when srecli ends, however it ends), up to 1 GiB, in order. What is held
  is counted at what it costs (a stream of one-byte frames counts their bookkeeping too), and the
  limit applies to memory and file together. srecli keeps reading the socket meanwhile, so the
  connection stays up. Past 1 GiB it sends close and exits `1` with "local output could not keep
  up". Ctrl-C or SIGTERM still ends it at once.
- A reader that goes away (a broken pipe, for example `| head`) is treated like SIGPIPE in a local
  pipeline, with no race on when srecli happened to read the final answer: srecli stops writing, sends
  close if the command is still running (srecli sends close once, and no more input after it), waits up to
  5 s in total for the final answer, then drops the connection without waiting for the close handshake, and
  exits quietly: with `124` when that answer says the command timed out (a timeout still comes first),
  with the command's own status when it says the command exited, else with `141` (128 + SIGPIPE).
  A command that was still running is stopped by the close, so `srecli exec box -- dmesg | head -1`
  gives the command's status only if the command had ended by itself. Any other write failure (no
  space left on the device, an I/O error) exits `1` with "local output failed".
- `connect`: Ctrl-], or Enter then `~` then `.` (as in ssh, for keyboards where Ctrl-] cannot be
  typed), leaves with `0`. The start of the session counts as after Enter, so `~.` as the very first
  keys leaves too. A `~` that does not follow Enter, or that is followed by another
  character, is sent unchanged, and `~~` sends one `~`. On start it prints one line, on stderr: how to
  leave, and that background jobs end with the session.
- Without `-w` a command must be on the read-only command list. Write commands and `connect`
  need an approved grant for the key you are using, which `srecli elevate` requests. The grant
  belongs to that key only, and it is approved on your phone with a code from your authenticator app.
- `login --code CODE` exchanges a one-time login code for a key. You get the code over SSH on wk01
  (`docker exec -it sre-api python -m app.terminal_admin login-code <user_id>`): it works once, for
  10 minutes. Case, dashes and spaces in what you type are ignored; anything that is not 25
  characters of the login-code alphabet (0-9 and A-Z without I, L, O and U) exits `2` before any
  request. `srecli login` without `--code` exits `2` and says how to get one. There is no polling
  and no browser. If the server does not accept the code, srecli exits `1` and says the code is not
  valid or has expired, and, if you minted it a moment ago, that someone else may have used it
  first: on wk01 run `terminal_admin keys list <user_id>` and revoke any key you do not know.
- `elevate` prints the approval address, a text QR code of it (made locally with segno) and
  "Approve on your phone: open the link or scan the code, then type the 6-digit code from your
  authenticator app". The code is drawn for a dark terminal; on a light one, use the link. Without
  segno (the `[cli]` extra) it still prints the address, and says so.
  The address is checked before it is printed: it must be exactly
  `https://sre.ai.cryptolabs.co.za/terminal/ui/grant/<grant id>` (the origin is compiled in) with the
  grant id of the answer, a `g_` and 32 lowercase hex characters; anything else exits `1` with "the
  server sent an unexpected approval address", and nothing is printed or polled. A second `elevate`
  from the same key supersedes the first, so srecli only ever shows the request it has just made. If
  another key of your account has a request waiting, the server answers `pending_exists` and
  srecli exits `1`, naming that key and when its request closes, with no address: wait, or on wk01
  run `terminal_admin keys revoke <key_id>`. A grant that ends denied or revoked prints its reason
  (for example `superseded`, `too many wrong codes`, `tamper alarm`). A server that is not
  adopted, an account with no authenticator and a locked account each exit `1` with what to do.
- Browser: `srecli` never opens a browser, for an approval or for anything else, and `BROWSER` is
  ignored. `SRECLI_URL` is for tests: it is honoured only when it names a loopback host, any other
  value exits `2` and nothing is sent, and the approval address is then checked against that
  loopback origin instead.
- `--json` on `servers` and `show` prints the API answer as JSON. `servers` shows hostname, online,
  terminal setting, adopted, linked, agent version, ready; `adopted` is whether you adopted the
  server over SSH, and `linked` is when it was attached to your account.
- Text from the server (hostnames, OS info, labels, messages) has escape sequences, control
  characters and every invisible or unassigned Unicode character (categories Cf, Co, Cs, Cn: direction
  marks, zero-width and tag characters) removed, and line breaks turned into spaces, before it is
  printed. A missing value prints as `-`. `--json` keeps the raw values, escaped by JSON.
- An error answer that is not JSON (for example from nginx) is reported as the HTTP status and a
  short generic text. A refused session upgrade maps like the REST answer: 401 is the key hint,
  429 is rate limited, 503 is "not enabled".
- `srecli elevate` waits for your approval and stops with exit `1` ("the request expired") 30 s after
  the request's `request_expires_at` (at most an hour from the request in any case). It retries a 429
  or 503 until then; a grant status it does not know is an error.
- `srecli` never sends a key that is not `sre_pk_` plus 43 letters and digits, and ignores
  proxy environment variables (`HTTPS_PROXY` and the like) for both REST and the session.
- `srecli login` checks that the key can be stored before it sends the code, because a login code
  works once. If storing fails after a key was issued, it revokes that key at once and exits `1`;
  you then need a new code. With a keyring it tries a throwaway entry first and falls back to the
  file if the keyring refuses. If the key is stored but the older copy in the other store cannot be
  removed, that is a warning, not a failed login.
- A credential store that exists but cannot be read (macOS Keychain locked, no GUI access, a changed
  Python binary) is reported as that, with the reason, and exits `1`. It is never reported as "not
  logged in", and `srecli login` does not fall back to the file while the Keychain may hold a key it
  cannot read, so one computer never ends up with two live keys. An empty Keychain is still "not
  logged in".
- A certificate that does not verify is reported with the reason OpenSSL gave, and exits `1`. Only
  when the issuer cannot be found locally (OpenSSL verify codes 2, 19 and 20: an issuer certificate or
  a self-signed root that is not trusted here) does it say this is probably a local trust-store
  problem and how to fix it (python.org Python on macOS: run "Install Certificates.command"). A
  self-signed server certificate (code 18), an expired certificate, a wrong host name or any other
  reason says that the server's certificate is not valid and not to continue: it is not a problem of
  your trust store.
- The key is stored in the macOS Keychain (service `srecli`) when `keyring` is installed, else
  in `~/.config/srecli/credentials` (mode 0600, with a warning). It is never put on a command
  line or printed. `SRECLI_KEY` in the environment takes precedence over a stored key.
  `srecli` warns when the key expires within 7 days; get a new login code over SSH and run
  `srecli login --code CODE` again to renew.
- `srecli logout` revokes the key in use: `SRECLI_KEY` first, then the stored key. It deletes the
  stored copy only if it is that key, so logging out an assistant's `SRECLI_KEY` never touches the key
  you stored for yourself. If the revoke fails, nothing is deleted and you can retry.

### Using srecli with an AI coding assistant

- Give the assistant its own key: run `srecli login --code` with its own login code on a separate
  account or computer, or set `SRECLI_KEY` for the assistant's process. A read key can list servers
  and run read-only commands and nothing else.
- Writes need **your** approval: the assistant can run `srecli elevate`, but the grant only
  starts after you type a code from your authenticator app on your phone, it lasts at most
  60 minutes (default 30), and it applies to the assistant's key only, never to your own key.
- Read output can contain secrets (environment variables, config files, logs). Anything
  `srecli exec` prints is visible to the assistant. Run `srecli logout` with the
  assistant's `SRECLI_KEY` set when you are done: that revokes the assistant's key and leaves yours.

## Environment Variables

| Variable | Description |
|----------|-------------|
| `SRE_AGENT_TOKEN` | Alternative to `--token` flag |
| `SRE_AGENT_TOKEN_FILE` | Alternative to `--token-file` flag |
| `SRE_RELAY_URL` | Override the relay URL (default: `wss://sre.ai.cryptolabs.co.za/agent/ws`) |
| `SRE_RELAY_BASE` | Base URL of the SRE API for `setup`, `approvals` and `approve` (default: `https://sre.ai.cryptolabs.co.za`) |

## Running as a Service

```bash
# systemd (Linux) — using token file for security
sudo tee /etc/systemd/system/sre-agent.service << 'EOF'
[Unit]
Description=CryptoLabs SRE Agent
After=network-online.target
Wants=network-online.target

[Service]
ExecStart=/root/.local/bin/sre-agent connect --token-file /etc/sre-agent.token
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl enable --now sre-agent
```

## Support

Need help? Join our Discord: https://discord.gg/7yeHdf5BuC

## Development

```bash
cd services/sre-agent
pip install -e ".[dev]"
pytest
```
