Metadata-Version: 2.5
Name: lts-backup-cli
Version: 3.2.1
Summary: LTS backup utility: a central backup server pulls Apache vhosts, MariaDB databases, folders and Docker hosts over SSH into local restic repositories, with discovery and scheduled rotation.
Author-email: LTS <joaomartins@lts.pt>
License: BSD-3-Clause
License-File: LICENSE
Keywords: apache,backup,cli,docker,mariadb,mysql,restic,restore,rsync,systemd
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: System Administrators
Classifier: License :: OSI Approved :: BSD License
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: System :: Archiving :: Backup
Requires-Python: >=3.9
Requires-Dist: click<9,>=8.1
Requires-Dist: questionary<3,>=2.0
Requires-Dist: tomli-w>=1.0
Requires-Dist: tomli>=2.0; python_version < '3.11'
Provides-Extra: dev
Requires-Dist: behave>=1.2.6; extra == 'dev'
Requires-Dist: build>=1.0; extra == 'dev'
Requires-Dist: mypy>=1.8; extra == 'dev'
Requires-Dist: pytest-cov>=4; extra == 'dev'
Requires-Dist: pytest>=7; extra == 'dev'
Requires-Dist: ruff>=0.5; extra == 'dev'
Requires-Dist: twine>=5; extra == 'dev'
Description-Content-Type: text/markdown

# lts-backup-cli — `ltsbu`

`ltsbu` runs on **one backup server** and **pulls** every **server** into its own set of
[restic](https://restic.net) repositories, **incrementally on a schedule, with automatic rotation**:

- **one repo per Apache/httpd vhost**: its DocumentRoot
- **one database repo per server**: any number of MariaDB databases, each in its own snapshot
- **one repo per extra folder**: anything else on the server, such as `/etc/apache2`
- **one `docker` repo per compose project** (e.g. each `*.lab.example.com` site) with its volumes,
  binds, compose folder and a dump of its MariaDB container, **plus a host repo `docker`** with every
  container definition (e.g. Bitbucket Pipelines runners) and the volumes no project owns

`ltsbu discover <host>` logs in to a server over SSH, finds its vhosts, databases and Docker host,
lets you pick what to back up and installs the systemd timers. Run it again whenever you want to
change anything.

```
┌──────────── backup server (ltsbu, restic, timers) ────────────┐        ┌──────── server ─────────┐
│ 1. rsync            -> <mirror_root>/<server>/<repo>/  (NVMe) │  ssh   │ sshd · rsync            │
│ 2. restic backup    -> <repo_root>/<server>/<repo>/    (RAID) │ ◀───── │ DocumentRoots, folders  │
│ 3. ssh … dump | restic backup --stdin                         │        │ docker volumes, binds   │
│ 4. restic forget (+ prune on the weekly run)                  │        │ MariaDB / DB containers │
└───────────────────────────────────────────────────────────────┘        └─────────────────────────┘
```

- **Central pull.** restic, the repositories, their passwords and the timers live on the backup
  server only. A backed-up server needs nothing but an SSH login and `rsync` (plus the MariaDB client
  for a database repo, or `docker` for a docker repo). It never sees a repository or a restic password.
  If the backup server is itself the server, use the host `localhost`.
- **Incremental, every 2 hours, plus a weekly run** at a staggered night slot. rsync only moves what
  changed into a local mirror, and restic only stores new chunks. After each run ltsbu rotates
  (`restic forget`), and the weekly run also prunes. Two backups never run at the same time.
- **Databases are streamed** (`ssh … mariadb-dump | restic backup --stdin`): no dump file is ever
  written to disk, on either side.
- **Verified before use.** The MariaDB admin login is checked during `discover` and saved on the
  server (`0600`). `ltsbu doctor` re-checks it with every database, container, path, repo and timer.
- **Secrets never appear in a process list or a log.**

> **Upgrading from 2.x?** Version 3 is a clean break: v2 pushed from each server to SFTP storage,
> v3 pulls. ltsbu refuses a v1/v2 `config.toml` with a hint: move it aside and run
> `ltsbu discover <host>` on the backup server. Old SFTP repositories are not touched and remain
> readable with plain restic. Servers no longer need restic, the storage SSH block or their
> `<repo>.pass` copies, which you can remove.

---

## Contents

1. [Requirements](#requirements)
2. [Installation](#installation)
3. [Quick start](#quick-start)
4. [Commands](#commands): `discover` · `doctor` · `list` · `backup` · `snapshot` · `restore` · `forget` · `credentials`
5. [Common options](#common-options)
6. [How backups work](#how-backups-work)
7. [Docker hosts](#docker-hosts)
8. [Schedule and rotation](#schedule-and-rotation)
9. [Files and configuration](#files-and-configuration)
10. [Managed SSH config block](#managed-ssh-config-block)
11. [Restoring](#restoring)
12. [Moving to another backup server / disaster recovery](#moving-to-another-backup-server--disaster-recovery)
13. [Troubleshooting](#troubleshooting)
14. [Development](#development)
15. [License](#license)

---

## Requirements

| Where | Component | Version | Notes |
|---|---|---|---|
| backup server | OS | Rocky Linux 8.10, Ubuntu 22.04, Debian 13 | with systemd (timers) |
| backup server | Python | ≥ 3.9 | Rocky 8: `python39` package |
| backup server | restic | ≥ 0.12 | Rocky 8: official release binary (not in EPEL 8); Ubuntu/Debian: `apt` |
| backup server | rsync, OpenSSH client | any | key-based login to each server |
| backup server | disks | | `repo_root` (default `/srv/restic`, ideally RAID) and `mirror_root` (default `/srv/ltsbu/mirror`, room for a full copy of every server's files) |
| server | OpenSSH server, rsync | any | the SSH user must be able to read every backed-up path (Docker data: `root`) |
| server | MariaDB client | 10.4+ | **only for a database repo**: `mariadb`/`mysql` and `mariadb-dump`/`mysqldump` |
| server | docker CLI | any | **only for a docker repo**; the dump runs *inside* each container (`mariadb-dump`/`mysqldump` there) |

## Installation

On the **backup server**:

```bash
# Rocky Linux 8.10: restic is not packaged in EPEL 8, install the official static binary
sudo dnf install -y python39 python39-pip openssh-clients rsync
curl -fsSL https://github.com/restic/restic/releases/download/v0.16.4/restic_0.16.4_linux_amd64.bz2 \
  | bunzip2 | sudo tee /usr/local/bin/restic >/dev/null && sudo chmod 755 /usr/local/bin/restic
python3.9 -m pip install --user lts-backup-cli

# Ubuntu 22.04 / Debian 13
sudo apt install -y pipx restic rsync openssh-client
pipx install lts-backup-cli

ltsbu --version          # make sure ~/.local/bin is on PATH
```

On every **server**: `rsync` (usually already there), and the MariaDB client for database repos.

## Quick start

1. **SSH key** from the backup server to the server (key-only; for a Docker host log in as root and
   restrict the key, e.g. `from="<backup server IP>",no-port-forwarding,no-agent-forwarding,no-X11-forwarding`):

   ```bash
   ssh-keygen -t ed25519 -f ~/.ssh/id_ltsbu -N '' && ssh-copy-id -i ~/.ssh/id_ltsbu.pub root@web1.example.com
   ```

2. **Discover the server** (interactive):

   ```bash
   ltsbu discover web1.example.com
   ```

   The wizard asks for, or detects:
   - the SSH login. A host already defined in your `~/.ssh/config` is used as is.
   - where the restic repositories and the rsync mirror live **on this backup server**
   - the **MariaDB admin ("sa") login**, if no working one is saved yet. It is verified and stored
     `0600` on the server.
   - **vhosts** to back up, one repo each (multi-select, all checked by default)
   - **databases** for the server's `db` repo (multi-select)
   - extra **folders**, one repo each
   - the **Docker host**, if `docker` answers: paths, excludes and MariaDB containers to dump
   - optionally a custom schedule and retention per repo (default: every 2h + weekly, rotated)
   - a summary, then: initialise the restic repos and install the timers

3. **Check everything:** `ltsbu doctor`

4. **Done.** Backups now run on their own. To run one by hand and look at it:

   ```bash
   ltsbu backup --server web1 --all
   ltsbu snapshot --server web1 --repo db
   ```

> **Keep the restic passwords safe.** Without them the repos can't be decrypted. They live only in
> `~/.config/ltsbu/servers/<server>/<repo>.pass` on the backup server. Keep an offline copy with
> `ltsbu credentials export` (see [below](#moving-to-another-backup-server--disaster-recovery)).

## Commands

Every command accepts the [common options](#common-options), before or after the command name.

### `ltsbu discover HOST`

Creates or edits the backups of one server. `HOST` is a hostname or IP address, or a `Host` alias
from your `~/.ssh/config` (then ltsbu writes no SSH block for it), or `localhost`.

Running it again for the same server is **edit mode**:
- current choices are the defaults
- the vhosts' DocumentRoots are refreshed
- new repos get a password and a staggered schedule
- the timers are refreshed

Repos you uncheck are only removed after an explicit confirmation (`--remove` when non-interactive).
Removing a repo stops its timers and drops it from the config, **but never deletes restic data**.

For automation, pass the answers as options with `--non-interactive`:

```bash
export LTSBU_DB_PASSWORD='…'            # used with --db-user; never on the command line
ltsbu discover web1.example.com --non-interactive \
  --ssh-user root --identity ~/.ssh/id_ltsbu \
  --db-user root --vhosts all --databases shop_db,blog_db --folders "/etc/apache2"

ltsbu discover lab.example.com --non-interactive \
  --ssh-user root --identity ~/.ssh/id_ltsbu --no-db-login --vhosts none --docker all
```

| Option | Default |
|---|---|
| `--server LABEL` | derived from the host (`web1` for `web1.example.com`) |
| `--ssh-user`, `--ssh-port`, `--identity` | *(required for a new server unless HOST is an ssh alias)*, `22`, none |
| `--repo-root`, `--mirror-root` | `/srv/restic`, `/srv/ltsbu/mirror` (on this backup server) |
| `--db-user` / `--no-db-login` | password from `$LTSBU_DB_PASSWORD` / skip databases |
| `--vhosts`, `--databases` | `all`, `none` or a comma-separated list; new server: all; edit: unchanged |
| `--folders "/a /b"` | `~/public_html` of the SSH user on a new server, if it exists and no vhost already covers it; otherwise none (`-` = none) |
| `--docker all\|none`, `--docker-containers` | non-interactive: `none`; `all` = one repo per compose project plus `docker`; containers: `all`, `none` or a list (default: every running container with a `/var/lib/mysql` mount) |
| `--every-hours N`, `--weekly-day DAY`, `--weekly-time HH:MM` | `2`, staggered night slot |
| `--keep-hourly/-daily/-weekly/-monthly N` | `12 / 7 / 4 / 6` |
| `--remove` | off: unchecked repos are kept |
| `--no-restic-init`, `--no-timers` | off: repos are initialised and timers installed |

### `ltsbu doctor`

Runs health checks and exits with **3** if any check fails. Remote checks are skipped for a server
that ltsbu can't log in to.

| Check | What it verifies |
|---|---|
| local binaries | `ssh`, `restic` and `rsync` exist here (`systemctl` missing is only a warning) |
| config lint / ssh config lint | values are valid, local secrets are `0600`, managed SSH blocks are present and balanced |
| dns / tcp | each server's name resolves and its SSH port answers |
| ssh login | non-interactive key login to the server (`BatchMode=yes`) |
| remote binaries | `rsync` (file repos), `docker` (docker repos), the MariaDB client (database repos) |
| paths | each backed-up path exists and is **readable by the SSH user** (Docker data needs root) |
| MariaDB | the saved login file is `0600`, the login works, and **every configured database exists** |
| containers | each container to dump is running and has `mariadb-dump`/`mysqldump` inside |
| restic repository | every local repo opens with its password |
| timers | both timers of every repo are installed and enabled; user timers warn if linger is off |
| last run | the last interval and weekly run of every repo succeeded (else: which `journalctl -u` to read) |

### `ltsbu list`

```
SERVER  REPO              KIND    WHAT                                 SCHEDULE                    REPOSITORY
lab     docker            docker  /srv/docker/containers, …            every 2h, weekly Sun 02:30  /srv/restic/lab/docker
lab     kiosk             docker  /srv/certs/…, /srv/docker/volumes/…  every 2h, weekly Sun 03:00  /srv/restic/lab/kiosk
web1    db                db      shop_db, blog_db                     every 2h, weekly Sat 01:00  /srv/restic/web1/db
web1    shop-example-com  vhost   /var/www/shop                        every 2h, weekly Thu 02:00  /srv/restic/web1/shop-example-com
```

### `ltsbu backup`

Pulls and backs up repos, then rotates them. This is what the timers run.

```bash
ltsbu backup --server web1 --repo shop-example-com
ltsbu backup --repo db                 # --server is optional when the repo name is unique
ltsbu backup --server web1 --all       # every repo of web1; --all alone: every server
ltsbu backup --repo db --weekly        # tag "weekly" and prune after forget
ltsbu backup --repo shop-example-com --include-git
```

In a database repo each database is dumped separately, and in a docker repo each container. If one
fails, the others still run and the command exits 1, naming what failed.

### `ltsbu snapshot`

Lists a repo's snapshots (`--json` for machine output), selected by the `server:<s>` and
`repo:<r>` tags, never by hostname. The DATABASE column shows the database or container of a dump.

### `ltsbu restore [SNAPSHOT]`

Restores **to the server**. `SNAPSHOT` is `latest` (the default) or an ID or ID prefix.

| Repo kind | Options | Action |
|---|---|---|
| vhost / folder | `--target DIR` | restore under `DIR` on the server; `--target /` restores in place (asks first) |
| db | `--database NAME [--into OTHER]` | stream the dump into `NAME`, or into `OTHER` which is created if missing (asks first) |
| docker | `--target DIR` | the files, under `DIR` on the server (`/` = in place, asks first) |
| docker | `--container NAME` | stream that container's dump back into it (all its databases; asks first) |
| docker | `--image REF` | stream a saved image into `docker load` on the server (asks first) |

Any other combination is an error. Files are restored into a private (`0700`) staging directory under
`mirror_root`, copied to the server with `rsync` (which never deletes anything there), and the staging
copy is removed. Confirmation is skipped with `--yes` and refused without it when there is no terminal.

### `ltsbu forget`

Rotation runs automatically after every backup. Use `forget` to apply a different policy once.

```bash
ltsbu forget --repo shop-example-com --yes
ltsbu forget --repo db --keep-daily 30 --prune --yes   # any --keep-* replaces the whole policy
```

An empty policy (everything 0) is refused, because it would delete every snapshot.

### `ltsbu credentials export [FILE]` / `import FILE`

Moves servers from one backup server to another: config, restic passwords, the SSH keys and the
**sites' database users** (user, host, `mysql_native_password` hash and database-level grants, read
from each database in a db repo). `--server` limits the export to one server. `--no-db-users` leaves
the users out.

`import` validates everything **before** writing anything. It:
- writes the secrets `0600` to this box's locations
- adds the SSH blocks
- recreates the database users on the server, building the SQL from validated fields and sending it on stdin

A server that already exists here needs `--force`, and then **keeps its local connection settings**
(SSH login, `repo_root`, `mirror_root`; see [disaster recovery](#moving-to-another-backup-server--disaster-recovery)).

> **The bundle is the key to your backups.** It can decrypt every snapshot and delete the repos.
> Pipe it over SSH (`ltsbu credentials export | ssh nas2 ltsbu credentials import -`), or
> `shred -u` every copy.

## Common options

| Option | Meaning |
|---|---|
| `--server LABEL`, `-s` | server to act on; optional when only one matches |
| `--repo NAME`, `-r`, `--label` | repo on that server; optional when only one matches |
| `--dry-run` | print the commands that would change something instead of running them; read-only commands still run (a new server's `discover` logs in through a temporary copy of `~/.ssh/config` with the block it would add) |
| `--verbose`, `-v` | log every external command (secrets are never on a command line) |
| `--config FILE` | alternative config file (default `~/.config/ltsbu/config.toml`); put it before the command |

Exit codes: `0` success, `1` error, `2` invalid usage, `3` doctor found failures.

## How backups work

- **Files** (vhost, folder and docker repos), all on the backup server:

  ```
  rsync -aHAXx --numeric-ids --protect-args --relative --delete --delete-excluded \
        --exclude=… -- ltsbu-<server>:<path>… <mirror_root>/<server>/<repo>/
  restic -r <repo_root>/<server>/<repo> backup --host <server> --tag … -- <mirror_root>/<server>/<repo>
  ```

  The mirror keeps each path's absolute layout (`…/<repo>/var/www/shop`). rsync's `--delete` only
  ever applies to the local mirror, never to the server. Files that vanish while rsync runs (exit 24)
  are reported, not fatal. Nested `.git` directories are excluded unless the repo has
  `exclude_git = false` (docker repos keep them).
- **Databases** (db repo), one database at a time, the pipe running on the backup server:

  ```
  bash -o pipefail -c "ssh ltsbu-<server> -- 'mariadb-dump --defaults-extra-file=<0600 my.cnf> --single-transaction --quick --routines --triggers <db>' | restic backup --stdin --stdin-filename <db>.sql --tag db:<db> …"
  ```

  The fixed file name `<db>.sql` makes each database its own restic group, so incremental dedup, parent
  detection and rotation all work per database. The dump has no `CREATE DATABASE`, so it can be loaded
  under any name. The MariaDB password stays in the server's `my.cnf`; it never crosses the network.
- **Tags:** `ltsbu`, `server:<server>`, `repo:<repo>`, `db:<db>` / `container:<name>` on dumps,
  `files` on a docker repo's file snapshot, and `weekly` on weekly runs.
- **One at a time:** a lock (`~/.cache/ltsbu/backup.lock`) makes overlapping timer runs queue
  instead of loading the servers in parallel.
- **Stale locks:** each backup first runs `restic unlock`, which only removes locks of restic
  processes that died (a lock left behind would otherwise make the rotation fail).
- Every argument sent over SSH is shell-quoted. The only secret ever written to a server (the MariaDB
  login) travels **on stdin** and is written `0600` with `umask 077`.

## Docker hosts

A Docker host, for example a Portainer installation with its stacks, a reverse proxy and Bitbucket
Pipelines runners, is backed up as **one repo per compose project** (the `com.docker.compose.project`
label, so one repo per site) **plus the host repo `docker`**. Every repo is of kind `docker`.
`discover` reads `docker info` and `docker inspect` of the running containers and proposes:

| Repo | Proposed | Why |
|---|---|---|
| `<project>` | the project's named volumes (`<DockerRootDir>/volumes/<name>`) | its data, e.g. `kiosk_app`, `portainer_data` (Portainer's database and its stacks' compose files) |
| | its compose folder and bind-mount sources | `docker-compose.yml`, `.env`, certificates, deploy keys, nginx config, … A bind shared by several sites is copied into each, so a site restores on its own. `/`, `/tmp`, the docker socket and the Docker root itself are skipped |
| | dump its running containers with a `/var/lib/mysql` mount | `docker exec <c> mariadb-dump --all-databases --single-transaction --routines --events --triggers` as the container's root (socket login, no password) |
| | exclude `*-json.log` and the dumped containers' `/var/lib/mysql` volume | the dump is the consistent copy; a live datadir copy may not start |
| `docker` | `<DockerRootDir>/containers` | every container's definition (`config.v2.json`, `hostconfig.json`): containers started with `docker run`, such as Bitbucket runners and their OAuth settings, can be recreated from it |
| | `<DockerRootDir>/volumes`, **excluding the volumes a project repo already copies** | volumes no running project owns are still kept, and nothing is copied twice |
| | bind sources of containers without a project | e.g. a runner's working folder |

Containers without a project, or whose project name is not a valid repo label (or is `docker` /
`db`), belong to the host repo. On a re-run of `discover`, a new project is added by default; a
project that no longer runs keeps its repo until you uncheck it and confirm the removal.

Overlay layers are **not** backed up, and images are pulled again, **except** the images listed in
the repo's `images` key (set it by hand in `config.toml`, e.g. for images built on the host that
exist in no registry). On the **weekly run** each is streamed `docker save <image> | restic backup
--stdin` (`<name>.tar`, tag `image:<image>`; restic dedups unchanged layers) and `doctor` checks it
exists. Bring one back with `ltsbu restore --repo docker --image <image>`.
Other live state, such as Redis, is copied crash-consistently. rsync uses `-x`, so it
does not cross into other filesystems mounted below a path.

Reading the Docker root needs root on the server: log in as `root` with a key restricted to the backup
server. `ltsbu doctor` fails a path the SSH user cannot read.

**Recovery of one site:** `ltsbu restore --repo <project> --target / --yes` (its volumes, compose
folder and binds), `docker compose up -d` in its folder, then
`ltsbu restore --repo <project> --container <c> --yes` to load its database.

**Recovery of a whole Docker host**, roughly: reinstall Docker, `ltsbu restore --repo docker --target / --yes`
(container definitions, orphan volumes, runner folders), restore every site as above (Portainer's
`portainer` repo first, so it redeploys its stacks), then recreate `docker run` containers such as
the runners from their restored `config.v2.json`.

## Schedule and rotation

Each repo gets two systemd timers on the backup server:

| Timer | Default | Runs |
|---|---|---|
| `ltsbu-<server>-<repo>.timer` | `OnCalendar=*-*-* 00/2:MM:00`, where `MM` is a per-repo minute | `backup`, then `forget` |
| `ltsbu-<server>-<repo>-weekly.timer` | one night slot between 01:00 and 04:30, a different day/time for each repo of every server | `backup --weekly`, then `forget --prune` |

All timers have `Persistent=true`, so a run missed while the box was off happens at boot, and
`RandomizedDelaySec=5min`. The default keep policy is `hourly 12, daily 7, weekly 4, monthly 6`. It
uses restic's `--keep-*` options and can be changed per repo in `discover`. While a policy is not yet
full, restic ≥ 0.14 also keeps the oldest snapshot (restic 0.12 does not), so young repos may show
one snapshot more than you expect.

As a normal user, units go to `~/.config/systemd/user` and need `loginctl enable-linger $USER` to
run while you are logged out (doctor warns). As root they go to `/etc/systemd/system`.

## Files and configuration

| Where | Path | Mode | Contents |
|---|---|---|---|
| backup server | `~/.config/ltsbu/config.toml` | 0600 | servers and repos (no secrets) |
| backup server | `~/.config/ltsbu/servers/<server>/<repo>.pass` | 0600 | restic password (the only copy) |
| backup server | `~/.config/systemd/user/ltsbu-*.{service,timer}` | | timers |
| backup server | `<repo_root>/<server>/<repo>/` | | restic repository |
| backup server | `<mirror_root>/<server>/<repo>/` | 0700 | rsync mirror: a **plain copy** of the server's files, secrets included; keep the box root-only. After each pull, anything a repo path no longer covers is removed from it |
| server | `~/.config/ltsbu/my.cnf` | 0600 | MariaDB admin login (`[client]`, `host=localhost`) |

Example `config.toml`:

```toml
version = 3

[servers.lab]
ssh_host = "192.168.69.99"
ssh_user = "root"
ssh_port = 22
ssh_identity = "/root/.ssh/id_ltsbu"
secrets_dir = "/root/.config/ltsbu"
repo_root = "/srv/restic"
mirror_root = "/srv/ltsbu/mirror"
db_login = false

[servers.lab.repos.docker]
kind = "docker"
password_file = "/root/.config/ltsbu/servers/lab/docker.pass"
images = ["lts/electro-site:php7.2-20260918", "lts/v2-dev:22.04"]  # weekly docker save
paths = ["/srv/bitbucket-runner/logiscreen", "/srv/docker/containers", "/srv/docker/volumes"]
excludes = ["*-json.log", "/srv/docker/volumes/kiosk_app", "/srv/docker/volumes/kiosk_mariadb_data"]
exclude_git = false

[servers.lab.repos.docker.schedule]
every_hours = 2
minute = 17
weekly_day = "Sun"
weekly_time = "02:30"

[servers.lab.repos.docker.keep]
hourly = 12
daily = 7
weekly = 4
monthly = 6

[servers.lab.repos.kiosk]  # compose project "kiosk" = kiosk.lab.mylogitools.pt
kind = "docker"
password_file = "/root/.config/ltsbu/servers/lab/kiosk.pass"
paths = ["/srv/certs/lab.mylogitools.pt", "/srv/docker/volumes/kiosk_app", "/srv/docker/volumes/kiosk_mariadb_data"]
containers = ["kiosk-site"]
excludes = ["*-json.log", "/srv/docker/volumes/kiosk_mariadb_data/_data"]
exclude_git = false
# (schedule and keep tables as above)
```

The MariaDB admin login needs `SELECT`, `LOCK TABLES`, `SHOW VIEW` and `TRIGGER` for backups. Restores
also need `CREATE`, `DROP`, `INSERT`, `ALTER` and `INDEX`, and `credentials import` needs `CREATE USER`
and `GRANT OPTION`. On a Virtualmin server, use `root`.

## Managed SSH config block

ltsbu edits the backup server's `~/.ssh/config`, and only between its tag comments: a block
`ltsbu:<server>` with `Host ltsbu-<server>` for the login to the server (ssh, rsync and the dump
pipes all use it). None is written when `HOST` is your own alias. Nothing is written to the
server's SSH configuration.

```sshconfig
# >>> ltsbu:lab >>>
# Managed by lts-backup-cli. Edits inside this block are overwritten.
Host ltsbu-lab
    HostName 192.168.69.99
    User root
    Port 22
    IdentityFile /root/.ssh/id_ltsbu
    IdentitiesOnly yes
    StrictHostKeyChecking accept-new
    ServerAliveInterval 60
    ServerAliveCountMax 240
# <<< ltsbu:lab <<<
```

Each edit is atomic and first saves a timestamped backup, `config.ltsbu-<timestamp>.bak`.
`StrictHostKeyChecking accept-new` trusts a host's key on first contact and rejects a changed key
afterwards, so unattended runs work on fresh machines.

## Restoring

```bash
ltsbu snapshot --repo shop-example-com                                   # what's there?
ltsbu restore --repo shop-example-com --target /tmp/shop-restore          # files -> server:/tmp/shop-restore/var/www/shop
ltsbu restore 4f2a9c1e --repo shop-example-com --target / --yes           # in place, from a specific snapshot
ltsbu restore --repo db --database shop_db --into shop_check --yes        # inspect a copy
ltsbu restore --repo db --database shop_db --yes                          # put it back
ltsbu restore --repo docker --container electro-site --yes                # a container's databases
```

Plain restic works too, on the backup server:
`restic -r /srv/restic/web1/db --password-file ~/.config/ltsbu/servers/web1/db.pass snapshots`.

restic restores numeric uid/gid. If the site's unix user has a different uid on a rebuilt server,
fix ownership with `chown -R`.

## Moving to another backup server / disaster recovery

**New backup server, same servers:**

```bash
ltsbu credentials export | ssh nas2 ltsbu credentials import -
# copy or rsync <repo_root> to nas2, then:
ssh nas2 'ltsbu discover web1.example.com && ltsbu doctor'   # re-installs the timers there
```

Disable the timers on the old backup server afterwards (`systemctl --user disable --now 'ltsbu-*'`),
so the two boxes never run the same backups.

**Rebuilt or replacement server** (same label, new machine):

1. `ltsbu discover new-web1.example.com --server web1`. This re-points `web1` and keeps the repos
   and their passwords.
2. `ltsbu credentials import --force bundle.toml` (from an earlier export), if the sites' database
   users must be recreated. The local connection settings from step 1 are kept.
3. `ltsbu restore --repo db --database shop_db --yes`, and `ltsbu restore --repo <vhost> --target / --yes`.

## Troubleshooting

| Symptom | Fix |
|---|---|
| `cannot log in to ltsbu-<server>` | `ssh-copy-id` your key to the server, then `ssh ltsbu-<server> true` |
| doctor: **path FAIL** `not readable by …` | log in as a user that can read it; Docker data needs `root` |
| doctor: **remote binary rsync FAIL** | install `rsync` on the server |
| doctor: **container X FAIL** | start the container, or remove it from the docker repo with `discover` |
| doctor: **mariadb login FAIL** | re-run `ltsbu discover <host>` and enter the admin login again |
| doctor: **database X FAIL** `database not found` | the database was dropped or renamed: re-run `discover` and update the selection |
| doctor: **restic repository FAIL** | `repo_root` not mounted, or the repo was never initialised: re-run `discover` |
| doctor: **timers FAIL** | re-run `ltsbu discover <host>` (timers are refreshed); check `systemctl --user list-timers 'ltsbu-*'` |
| doctor: **systemd linger WARN** | `loginctl enable-linger $USER` |
| `v1/v2 config detected` | ltsbu 3 is a clean break: move the old config aside and run `ltsbu discover <host>` |
| backup: `rsync … exited 23` | some files could not be read (permissions): see the message, fix access or add an exclude |
| `Access denied … LOCK TABLES` during a backup | grant the admin login `LOCK TABLES` and `SHOW VIEW` |
| `refusing without --yes (no terminal)` | add `--yes` in scripts |
| Ubuntu: a restic flag is unsupported | Ubuntu's restic 0.12 is old: `sudo restic self-update` |

`--verbose` shows every command ltsbu runs.

---

## Development

```bash
python3 -m venv .venv && . .venv/bin/activate
pip install -e '.[dev]'

ruff check . && ruff format --check . && mypy src   # lint + types
pytest -q --cov=lts_backup_cli                      # unit
behave --tags=~@e2e                                 # BDD acceptance (no network)

# Smoke: the built wheel in a clean venv
python -m build --wheel --outdir /tmp/dist . && python -m venv /tmp/smoke \
  && /tmp/smoke/bin/pip install /tmp/dist/*.whl \
  && PATH=/tmp/smoke/bin:$PATH PYTHON=/tmp/smoke/bin/python sh tests/smoke/smoke.sh

# E2E: real SSH, rsync, restic and MariaDB 10.4 on both supported distros (smoke runs first)
docker compose -f tests/docker/docker-compose.yml run --rm --build rocky8
docker compose -f tests/docker/docker-compose.yml run --rm --build ubuntu2204
docker compose -f tests/docker/docker-compose.yml down && docker run --rm --entrypoint rm -v "$PWD/tests/docker:/d" mariadb:10.4 -rf /d/.e2e
```

Design principles, the testing strategy and the Bitbucket CI/CD pipeline are described in
[CLAUDE.md](CLAUDE.md). The requirements are in [docs/SPEC3.md](docs/SPEC3.md) (v3), with the
superseded [docs/SPEC2.md](docs/SPEC2.md) (v2) and [docs/SPEC.md](docs/SPEC.md) (v1). Pushing a `vX.Y.Z` tag that matches `__version__` publishes the
package to PyPI.

## License

BSD 3-Clause. Copyright (c) 2026, joaomartins@logitools.pt and LTS, Lda. See [LICENSE](LICENSE).
