Metadata-Version: 2.4
Name: jetstan-agent
Version: 0.2.22
Summary: Production-grade OTA agent for Linux-based vehicles — MQTT communication backbone.
Author: Nihar Sharma
License: MIT License
        
        Copyright (c) 2026 Nihar Sharma
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Project-URL: Repository, https://github.com/PICODE-Labs/JETSTAN-PICODE
Keywords: ota,mqtt,embedded,automotive,raspberry-pi
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: System :: Networking
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: paho-mqtt==2.1.0
Requires-Dist: python-dotenv==1.2.2
Provides-Extra: dev
Requires-Dist: pytest>=8.0.0; extra == "dev"
Requires-Dist: pytest-mock>=3.14.0; extra == "dev"
Dynamic: license-file

# Installing Jetstan Agent as a systemd service

This turns the `jetstan-agent` PyPI package into a real, self-healing
Linux daemon: it starts on boot, restarts itself if it crashes or hangs,
and pulls + installs new versions automatically when told to over MQTT.

## First-time install on a fresh Pi

Two commands. Raspberry Pi OS (Debian 12+) blocks plain `pip install`
system-wide by default (PEP 668) — `pipx` is the correct tool here, not a
workaround: it gives the agent's CLI its own isolated environment, and
(unlike a plain `pip install --user` as root) `--global` installs put it
somewhere every user's `$PATH` — including `sudo`'s — actually finds it,
with no manual symlinking:

```bash
sudo apt update && sudo apt install -y pipx
sudo pipx install --global jetstan-agent
```

Then run the installer, supplying your MQTT credentials directly so the
service is fully configured and running after this one command — no SSH
round-trip to edit a file:

```bash
sudo jetstan-agent-install \
  --mqtt-host <your-cluster>.hivemq.cloud \
  --mqtt-username <username> \
  --mqtt-password '<password>'
```

(Single-quote the password — protects special characters like `!` from
the shell.)

`--mqtt-password` is visible in shell history and briefly in `ps` output
while the command runs. If that matters for your setup (shared machine,
scripted provisioning, etc.), pipe it in instead with `--mqtt-password-stdin`:

```bash
echo '<password>' | sudo jetstan-agent-install \
  --mqtt-host <your-cluster>.hivemq.cloud \
  --mqtt-username <username> \
  --mqtt-password-stdin
```

Either way, the password ends up in `/etc/jetstan/agent.env` (mode `600`,
owned by the `jetstan` service account) — that part is unavoidable, the
running service has to read it from somewhere.

That's it. The installer creates a dedicated `jetstan` system user, a venv
at `/opt/jetstan/venv`, `/etc/jetstan/agent.env` (populated with what you
passed), and the `jetstan-agent` systemd service — enabled, started, and
verified actually connected before the command returns 0.

```bash
jetstan-agent status
jetstan-agent logs
```

Reboot the Pi whenever you like afterward — the service comes back on its
own, no manual commands required.

### Prefer to configure credentials by hand instead?

Omit all three `--mqtt-*` flags:

```bash
sudo jetstan-agent-install
sudo nano /etc/jetstan/agent.env   # fill in MQTT_HOST / MQTT_USERNAME / MQTT_PASSWORD
sudo systemctl restart jetstan-agent
```

On a brand-new install with no credentials yet, the service failing to
connect until you fill them in is expected, not an error.

### Troubleshooting

- **`sudo: jetstan-agent-install: command not found`** after
  `pipx install --global` — you're on a pipx older than 1.4 (check
  `pipx --version`; `sudo apt install -y pipx` on current Raspberry Pi OS
  is well past this). Upgrade pipx, or fall back to
  `sudo pipx install jetstan-agent` and manually symlink the three
  scripts pipx prints the path to (under `/root/.local/bin/`) into
  `/usr/local/bin/`.
- **A fresh `pip3 install jetstan-agent` fails with an
  `externally-managed-environment` error** — expected on current Debian;
  this is exactly why the instructions above use `pipx`, not `pip3`,
  directly.

## Everyday operations

```bash
jetstan-agent status     # systemctl status, under the hood
jetstan-agent start
jetstan-agent stop
jetstan-agent restart
jetstan-agent logs        # journalctl -u jetstan-agent -f
jetstan-agent logs -n 200 --no-follow
```

(`jetstan-agent run` also exists — that's the foreground agent process
itself, which is what the systemd unit's `ExecStart=` invokes. You
normally never call it directly.)

## How the OTA update loop works

There is no separate download/artifact server — the existing PyPI
distribution channel *is* the update mechanism. The agent is purely
event-driven: it never polls PyPI on its own, it only acts when it
receives a specific MQTT message.

**On your dev machine**, `jetstan-cli`'s `jetstan_deploy.py` does the
whole publish side in one command:

```bash
cd jetstan-cli && pip install -e ".[dev,deploy]"    # once, installs build + twine
python -m jetstan_cli.jetstan_deploy
```

It: reads the version from the root `pyproject.toml` → refuses to
proceed if that version is already published → runs the test suite →
rebuilds `dist/` from a clean slate → uploads to PyPI via `twine` (your
credentials, your prompt, same as running twine by hand) → waits
(default 45s, see `--wait-seconds`) for PyPI's index to actually
propagate → publishes `{"type": "ota.available", "version": "..."}` to
your configured MQTT topic, using the same `MQTTConfig.from_env()` the
agent itself uses (so it reads from your local `.env` — see the root
[`README.md`](../README.md)).

Run `python -m jetstan_cli.jetstan_deploy --dry-run` to build and test
without publishing anything, or `--yes` to skip the confirmation prompt.
This performs a real, public PyPI upload and pushes an update to every
Pi listening on the topic — run it yourself, deliberately, not as part
of an unattended script.

**On each Pi**, once that message arrives:

1. The running agent receives it, runs
   `pip install --upgrade jetstan-agent==1.4.0` inside its own venv, and
   then deliberately exits.
2. systemd's `Restart=always` immediately relaunches the service — now
   running the newly installed version.

If the pip upgrade fails (bad version, network blip), the agent logs the
error and keeps running the current version — nothing is torn down.
If the message names a version that's already running, it's a no-op
(safe against duplicate/retained MQTT deliveries). If the upgrade
*succeeds* but the new version turns out to be broken, see "A bad release
can't take a device permanently offline" below — that's handled too.

## Uninstalling

```bash
sudo jetstan-agent-uninstall                          # stops/disables the service, keeps config & venv
sudo jetstan-agent-uninstall --purge-config            # also deletes /etc/jetstan (credentials)
sudo jetstan-agent-uninstall --purge-state             # also deletes /opt/jetstan and the jetstan user
sudo jetstan-agent-uninstall --purge-config --purge-state   # full removal
```

## Reliability details

- `Restart=always` + `RestartSec=5`: any crash or clean exit restarts
  the service after 5s.
- `WatchdogSec=60`: the agent pings systemd every ~30s while its MQTT
  loop is alive; if it ever hangs (not just crashes), systemd restarts
  it too.
- `StartLimitBurst=8` / `StartLimitIntervalSec=120`: if something is
  genuinely broken and it crash-loops more than 8 times in 2 minutes,
  systemd stops trying rather than looping forever. Set deliberately
  above the automatic-rollback threshold below, with margin — see there
  for why the exact numbers matter.
- The service runs as an unprivileged `jetstan` user with heavy systemd
  sandboxing (`ProtectSystem=strict`, no new privileges, no device/kernel
  access, etc.) — the only writable path is its own venv, which is what
  lets self-upgrade work without weakening anything else.

### A bad release can't take a device permanently offline

`Restart=always` on its own has a gap: if a *newly installed* release
crashes on startup, systemd just keeps relaunching the same broken code,
burns through its restart budget in under a minute, then gives up
entirely — the unit sits dead, unreachable by design (SSH is your only
way back in at that point). That defeats the actual point of having an
OTA channel.

`ExecStart=` doesn't invoke the agent directly — it invokes
`/opt/jetstan/supervisor.py`, a small file `jetstan-agent-install` writes
once, deliberately *outside* `/opt/jetstan/venv`, the only path
`pip install --upgrade jetstan-agent` ever touches. That's what lets it
survive a release that breaks at import time, which a fix living inside
the upgraded package itself couldn't. It launches the real agent as a
child process; if the *same* version fails 3 times in a row, it
automatically `pip install`s back to the previously-running version
instead of retrying the broken one a 4th time, and restarts into it. A
version is only considered "safe" once it successfully connects to
MQTT — a version that connects fine and crashes hours later for an
unrelated reason is never rolled back; only a version that never proved
itself healthy in the first place is a rollback candidate. If the
rollback target *also* keeps failing, the supervisor stops intervening
and lets it settle into systemd's normal failed state — an honest,
loudly-logged signal for a human to SSH in, not an infinite ping-pong
between two bad versions or a silent brick either way.

### Every OTA push only ever touches this package's own code

`pip install --upgrade jetstan-agent` runs with `--no-deps`, and every
dependency in `pyproject.toml` is pinned to an exact version, not a
range. Without both of these, a routine push could silently pull in a
new transitive dependency release (e.g. `paho-mqtt`) that was never
tested against this codebase — on a push that didn't even touch
`jetstan-agent`'s own code. Bumping a dependency is now a deliberate,
tested part of a normal release (bump the pin,
`jetstan-cli`'s `jetstan_deploy.py` test gate covers it) rather than something
that can change underneath a live fleet on its own schedule.

### Every device gets its own MQTT identity by default

Two devices sharing one MQTT `client_id` isn't a hypothetical — it's
already happened once during this project's own testing: the broker
disconnects whichever connection is older the instant a second one
claims the same ID. The default `client_id` is now derived per-machine
(from `/etc/machine-id`), not a fixed string, so this can't happen from
two otherwise-identical installs. Set `JETSTAN_MQTT_CLIENT_ID` explicitly
in `agent.env` only if you need a specific, human-meaningful ID (e.g.
matching a fleet inventory system) — the default is safe to leave alone.

## Arduino firmware updates (optional)

The installer always adds the `jetstan` service account to the `dialout`
group and the systemd unit always leaves real `/dev` visible
(`PrivateDevices` is not set) — both needed for USB-serial access, and
both dormant/unused unless you set `ARDUINO_ENABLED=true` in
`agent.env`. See [README.md](../README.md#arduino-firmware-updates) for the
full variable list and how the automatic compile-and-flash pipeline works.

### Zero-touch: `jetstan-agent-install` provisions the whole toolchain

`avrdude` and `arduino-cli` are provisioned **automatically, by the
installer itself**, as part of the same one `sudo jetstan-agent-install`
run described at the top of this file — no separate manual step, no
`curl | sh`. Specifically (see `jetstan_agent/arduino_provision.py`):

- **avrdude** is installed via `apt-get install avrdude` (root, one-time).
- **arduino-cli** is installed by downloading a *pinned* release archive
  directly from its GitHub Releases page, together with that release's
  own published `_checksums.txt`, verifying the archive's SHA-256
  against it, and only then extracting the single binary to
  `/usr/local/bin/arduino-cli` — never a piped-to-shell installer script.
  A checksum mismatch is a hard failure, not a warning.

Both steps are **best-effort during install**: a failure (typically, no
network reachable at install time) is logged clearly but does not fail
the overall `jetstan-agent-install` run — the always-needed OTA/MQTT
agent still installs and starts. If you enable `ARDUINO_ENABLED=true`
later without these having succeeded, the runtime's own
`validate_environment` step (see `software/deployment/bootstrap.py`)
is the real, hard gate — it fails loudly, with a specific reason, and
blocks firmware compile/flash/telemetry until it's resolved.

To retry provisioning without a full reinstall:

```bash
sudo python3 -c "from jetstan_agent import arduino_provision as p; p.ensure_avrdude_installed(); p.ensure_arduino_cli_installed()"
```

(using the venv's own interpreter, e.g. `/opt/jetstan/venv/bin/python3`,
so the installed `jetstan_agent` package is importable.)

**AVR core install location** — the `jetstan` service account has no
home directory (`useradd --no-create-home`), and `arduino-cli` (a Go
binary) resolves several of its own paths (config file, build cache)
straight from `$HOME`/XDG variables, which has nowhere valid to go for a
homeless account. `jetstan-agent-install` already creates
`/var/lib/jetstan` (via systemd's `StateDirectory=jetstan`); the
runtime's own `bootstrap_environment` step points `arduino-cli` at
`/var/lib/jetstan/arduino-cli/` for all of this automatically — no
manual `mkdir`/`chown`/config-file authoring needed (this replaces what
used to be a manual step here).

**You do not need to run `arduino-cli core install arduino:avr`
yourself.** Once `arduino-cli` and `avrdude` are present (installed
automatically above, or manually — see the fallback below), the
deployment DAG's `bootstrap_environment` task (see
`software/deployment/bootstrap.py` and `arduino_updates/provisioner.py`)
installs and verifies the `arduino:avr` core itself, automatically, on
every startup — as the unprivileged `jetstan` user, no root involved.
It's idempotent: already-correct is a fast no-op, and it never
reinstalls on every OTA push, only when the core is genuinely missing or
(if `ARDUINO_CORE_VERSION` is set in `agent.env`) at the wrong version.
The network-dependent steps (fetching the index, downloading the core)
are retried a bounded number of times with backoff — a transient outage
recovers on its own; a host with no network at all fails the
`bootstrap_environment` DAG task loudly rather than hanging.

**What stays manual by design, even with automatic provisioning**: the
*decision* to trust and run `apt`/a downloaded binary at all happens only
during the root-privileged, human-invoked `jetstan-agent-install` run —
never from the OTA-triggered, unprivileged `jetstan` process, which has
no path to root and never attempts either install itself. This is the
same security boundary as before; only *where* the automation lives
changed (installer, not "a step INSTALL.md tells you to run by hand").

### Manual fallback

If automatic provisioning fails (no network at install time, an
unsupported architecture, or you simply prefer to control it yourself):

```bash
sudo apt install avrdude
curl -fsSL https://raw.githubusercontent.com/arduino/arduino-cli/master/install.sh | sudo BINDIR=/usr/local/bin sh
```

Everything downstream of that (AVR core provisioning, compiling,
flashing) still happens automatically exactly as described above — this
fallback only replaces the two binaries the installer otherwise fetches
for you.

## Known limitation

The installer uses `useradd`/`groupadd`/`userdel` (standard on Debian-based
systems, including Raspberry Pi OS). It hasn't been adapted for Yocto/busybox
-based images, which use different user-management tools.
