Metadata-Version: 2.4
Name: splent_feature_autodeploy
Version: 0.1.2
License-Expression: LGPL-3.0-or-later
Requires-Python: >=3.13
Description-Content-Type: text/markdown
License-File: COPYING
License-File: COPYING.LESSER
Dynamic: license-file

# splent_feature_autodeploy

Release-gated continuous delivery for a SPLENT product. A product that
selects this feature runs a small agent next to its web container in
production. The agent follows a registry image channel, for example
`git.splent.io/splent-io/<product>:stable`, and when the channel points to a
new image it redeploys the product. The database is backed up first, the
new version has to pass its health check, and an automatic rollback of both the container and the database when
the new version does not come up.

The channel only moves when someone runs `splent product:release`, so nothing
reaches production that nobody released. The agent never builds anything and
never follows a branch.

## How a check goes

Every `AUTODEPLOY_INTERVAL` seconds, and whenever an administrator asks,
the agent goes through these steps.

1. Pull the channel image. If its id is the one the web container runs,
   there is nothing to do.
2. Skip an image that failed before, until the channel points elsewhere.
   Honour the pause flag and the update window (`AUTODEPLOY_WINDOW`). A
   check somebody asks for (Check now) deploys outside the window too.
3. Back up the database. `mariadb-dump --single-transaction --routines
   --triggers` runs as root inside the database container, with the
   credentials that container already holds, and the gzipped dump is kept
   only if `mariadb-dump` exited cleanly and the dump ends with its
   `Dump completed` trailer. No complete backup, no deploy.
4. Record what the web container runs (`splent version --json`).
5. Recreate the web container on the new image with everything it was
   configured with, the way Watchtower does it. That is its name, its
   environment (the container's own values on top of the new image's), its
   labels, mounts and binds, port bindings, restart policy, healthcheck, and
   its networks with their aliases. Cmd, Entrypoint and WorkingDir follow the new image when the old
   container only inherited them, and are kept when it chose them (the
   compose file's `command:`). The old container is renamed
   `<name>-previous` and stopped, not removed.
6. Wait for the healthcheck, up to `AUTODEPLOY_HEALTH_TIMEOUT` seconds. The
   entrypoint runs every feature's migrations before gunicorn starts, so this
   is generous.
7. Once it is healthy, remove the previous container and write the history
   entry (from and to image, product version, the features whose version
   changed, duration).
8. If it turns unhealthy, exits, restarts or runs out of time, keep the new
   container's last 80 log lines, remove it, drop and recreate the database and import the
   backup (so tables the new migrations created do not linger), put the
   previous container back under its name, wait for it to be healthy, and
   refuse that image from then on. The history entry says `rolled_back` and
   carries the log lines.

Every step is written to `/data/state.json` before it is taken. An agent
restarted in the middle of a deploy (a reboot, a redeploy of the agent
itself) finishes it if the new container turns healthy, and rolls it back
otherwise.

| State                | Meaning                                                      |
|----------------------|--------------------------------------------------------------|
| `idle`               | The web container runs the image the channel points to       |
| `waiting_for_window` | A new image is ready and waits for the update window         |
| `deploying`          | Backup, recreate or health wait in progress                  |
| `rolling_back`       | The new image failed and the previous version is coming back |
| `paused`             | An administrator paused automatic deploys                    |
| `held`               | The only image available failed before; waiting for the next release |
| `error`              | Something needs a person, the detail says what               |
| `not_configured`     | A required setting is missing, the detail lists it           |

## Enable it in a product

1. Add the feature to the product line as an optional feature, from a
   detached workspace.

   ```bash
   splent spl:add-feature <spl> splent_feature_autodeploy
   ```

   In the UVL model it reads as below, with the constraint that keeps the
   admin panel it lives in.

   ```
   optional
       autodeploy {org 'splent-io', package 'splent_feature_autodeploy'}
   constraints
       autodeploy => admin
   ```

2. Attach it to the product and release the product, so the channel exists.

   ```bash
   splent product:select <product>
   splent feature:attach splent-io/splent_feature_autodeploy v0.1.0
   splent product:release
   ```

   A product release pushes `<registry>/<org>/<product>:<version>` and moves
   `:stable` to it.

3. On the server, set `SPLENT_IMAGE` in `docker/.env.deploy` to the channel
   the server follows, then build and deploy. The web service of the
   product's `docker-compose.prod.yml` must take its image from that
   variable, as in `image: ${SPLENT_IMAGE:-<product>:latest}`.

   ```bash
   SPLENT_IMAGE=git.splent.io/splent-io/<product>:stable
   ```

   ```bash
   splent product:build --no-image
   splent product:deploy
   ```

   `product:build` adds the agent's service to `docker-compose.deploy.yml`
   and copies `docker/` to `docker/features/splent_feature_autodeploy/`.
   `product:deploy` offers a random value for `AUTODEPLOY_SECRET`, builds the
   agent's image and starts it.

4. For a private registry set `AUTODEPLOY_REGISTRY_USER` and
   `AUTODEPLOY_REGISTRY_TOKEN` (a token that can read packages).

From then on, `splent product:release` is the whole deploy.

## Configuration

Declared in `[tool.splent.config]`, so they appear in `.env.deploy.example`
with the product's names filled in.

| Variable                     | Default                             | Meaning |
|------------------------------|-------------------------------------|---------|
| `AUTODEPLOY_IMAGE`           | empty                               | Image to follow. Empty means `SPLENT_IMAGE` |
| `AUTODEPLOY_INTERVAL`        | `300`                               | Seconds between checks (at least 30) |
| `AUTODEPLOY_WINDOW`          | empty                               | UTC span for deploys, like `02:00-05:00` or `22:00-04:00`. Empty means any time |
| `AUTODEPLOY_HEALTH_TIMEOUT`  | `600`                               | Seconds the new container gets to turn healthy |
| `AUTODEPLOY_KEEP_BACKUPS`    | `5`                                 | Backups kept, newest first |
| `AUTODEPLOY_WEB_CONTAINER`   | `<product>_web_deploy`              | The container that is replaced |
| `AUTODEPLOY_DB_CONTAINER`    | `<product>_db_deploy`               | The MariaDB container that is backed up |
| `AUTODEPLOY_AGENT_CONTAINER` | `<product>_autodeploy`              | The agent's container name |
| `AUTODEPLOY_URL`             | `http://<product>_autodeploy:8080`  | Where the web side finds the agent |
| `AUTODEPLOY_REGISTRY_USER`   | empty                               | Registry account, for a private channel |
| `AUTODEPLOY_REGISTRY_TOKEN`  | empty                               | Registry token, for a private channel |
| `AUTODEPLOY_SECRET`          | `<SET>`                             | Shared by the web container and the agent |

## The Deployments page

`/admin/deployments`, in the admin panel's sidebar, for administrators only
(the admin feature's `ADMIN_ROLES`). It shows the channel, the running image
and product version, what the agent is doing and why, the last check, the
update window, whether updates are paused, and the latest deploys with the
features each one moved. A rolled back deploy shows its log lines in a
collapsible block, and a banner says so until the next deploy. Check now,
Pause and Resume are CSRF protected forms.

When the agent is missing (development), stopped or refuses the secret the
page explains what is going on. The site itself is never affected.

## Commands

```bash
splent feature:autodeploy status [--json]
splent feature:autodeploy history [--limit N] [--json]
splent feature:autodeploy check
splent feature:autodeploy pause
splent feature:autodeploy resume
```

They talk to the agent's API, so they need the product network. On a
production server run them in the web container.

```bash
docker exec <product>_web_deploy splent feature:autodeploy status
```

## The agent's API

Port 8080 on the product network, never published on the host. Every request
carries `Authorization: Bearer <AUTODEPLOY_SECRET>`, compared in constant
time; anything else is a 401, whatever the path.

| Method | Path               | Does |
|--------|--------------------|------|
| GET    | `/status`          | State, configuration, running versions, backups |
| GET    | `/history?limit=N` | The newest N deploys, newest first (1 to 200) |
| POST   | `/check`           | Look for a new image now |
| POST   | `/pause`           | Stop deploying; a deploy in progress finishes |
| POST   | `/resume`          | Deploy again, and check right away |

There is no rollback endpoint on purpose. A rollback restores the backup taken
before the deploy, so run later by hand it would throw away everything written
since.

## Security

- The agent mounts `/var/run/docker.sock`. Whoever controls that socket is
  root on the host, so the agent is as sensitive as the host itself. It runs
  only in production, from the image `product:deploy` builds out of this
  feature's `docker/` directory, and does only what this README describes.
- It publishes no port. Only containers on the product network reach the API,
  and every call needs the shared secret. The secret never appears in the
  status, the history or the logs.
- The web side calls the agent without any proxy, so an `HTTP_PROXY` in the
  environment never sees the secret.
- The database credentials stay in the database container. The dump and the
  restore read them from that container's environment and pass them through
  `MYSQL_PWD`, so they are not on any command line and are never printed.
- Registry credentials are only sent to the registry, with the pull.

## Backups

One per deploy, in the agent's `autodeploy_data` volume under
`/data/backups/`, named `<UTC time>-<short id of the image being replaced>.sql.gz`,
so each file says which code its schema belongs to. The newest
`AUTODEPLOY_KEEP_BACKUPS` are kept. They live on the same host as the
database, so they protect against a bad release, not against losing the
server. Copy them elsewhere if you need that.

## Restoring a backup by hand

Restoring a backup loses everything written after it was taken. Pause the
agent first, so it does not deploy in the middle.

```bash
docker exec <product>_web_deploy splent feature:autodeploy pause
docker cp <product>_autodeploy:/data/backups/<file>.sql.gz .
docker stop <product>_web_deploy
docker exec <product>_db_deploy bash -c 'MYSQL_PWD="$MARIADB_ROOT_PASSWORD" mariadb -uroot -e "DROP DATABASE IF EXISTS \`$MARIADB_DATABASE\`; CREATE DATABASE \`$MARIADB_DATABASE\`"'
gunzip -c <file>.sql.gz | docker exec -i <product>_db_deploy bash -c 'MYSQL_PWD="$MARIADB_ROOT_PASSWORD" mariadb -uroot "$MARIADB_DATABASE"'
```

Then run a web container on the image named in the file name, so code and
schema match, and resume the agent.

To let the agent try an image it refused, pause it, delete the image id from
`bad_images` in `/data/state.json` in the agent's volume, and resume.

## What the agent does not do

- It replaces the web container only. A release that adds a service (a
  feature that ships its own container), or needs a new setting without a
  default in `.env.deploy`, has to go through `splent product:deploy`; pause
  the agent while you do it. Left to the agent such a release fails its
  health check and is rolled back, which is safe but not what you want.
- It does not update itself. A new version of this feature reaches the
  server with `splent product:deploy`, which rebuilds the agent's image.
- It replaces one web container, not a set of replicas.

## Develop

The agent is plain Python 3.13 with docker-py, in `docker/autodeploy_agent/`.
It gets the Docker client handed in, so its tests run against a fake Engine.

```bash
splent feature:test splent_feature_autodeploy
```

From a bare checkout, with a Python that has the framework, Flask-Babel,
Flask-Login and Flask-WTF installed, the same tests run without a product.

```bash
python -m pytest src/splent_io/splent_feature_autodeploy/tests
```
