Metadata-Version: 2.4
Name: django-dbs
Version: 0.2.0
Summary: Redundant, self-healing, encrypted single-file backup for Django projects.
Author: bn7ya
License: MIT
Project-URL: Homepage, https://github.com/bn7ya/dbs
Project-URL: Repository, https://github.com/bn7ya/dbs
Project-URL: Issues, https://github.com/bn7ya/dbs/issues
Project-URL: Changelog, https://github.com/bn7ya/dbs/blob/main/CHANGELOG.md
Keywords: django,backup,restore,encryption,reed-solomon,redundancy
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Framework :: Django
Classifier: Framework :: Django :: 4.2
Classifier: Framework :: Django :: 5.0
Classifier: Framework :: Django :: 5.1
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: System :: Archiving :: Backup
Classifier: Topic :: System :: Systems Administration
Classifier: Topic :: Security :: Cryptography
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: Django<6,>=4.2
Requires-Dist: cryptography>=41
Requires-Dist: argon2-cffi>=21.3
Requires-Dist: reedsolo>=1.7
Provides-Extra: ssh
Requires-Dist: paramiko>=3; extra == "ssh"
Provides-Extra: client
Requires-Dist: paramiko>=3; extra == "client"
Requires-Dist: tomli>=1.1.0; python_version < "3.11" and extra == "client"
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == "dev"
Requires-Dist: pytest-django>=4; extra == "dev"
Requires-Dist: pytest-cov>=4; extra == "dev"
Requires-Dist: ruff>=0.4; extra == "dev"
Requires-Dist: paramiko>=3; extra == "dev"
Requires-Dist: tomli>=1.1.0; python_version < "3.11" and extra == "dev"
Dynamic: license-file

# DBS — Django Backup Solution

> 🚧 **Under development.** DBS is in active development and its API may change.
>
> Created by **Sudum Technology — Research and Development sector**.
> Our very tiny contribution to this world.

A backup library you drop into a Django project's source. It reads your models,
relations and files and writes **one encrypted file** that is *redundant* and
*self-healing*: every backup stores **two copies** of the data plus
**Reed-Solomon parity**, so silent corruption — the kind a non-ECC RAM stick
produces — is detected and repaired on restore instead of quietly poisoning
your data.

```
io  ──►  security (Argon2id + AES-256-GCM)  ──►  integrity (2 copies · per-block
hashes · Reed-Solomon)  ──►  data (models · relations · files)
```

## Features

* **Two self-healing copies in one file** plus Reed-Solomon parity for bit-rot.
* **Per-field mapping**: value, embedded file, or file-path.
* **Passphrase-derived key, never stored** (Argon2id).
* **Per-file and per-block integrity hashes.**
* **Scheduled backups** on the server, with retention.
* **A `dbs-client` command** that pulls backups off your server over SSH.
* CLI · admin-UI download · SFTP.

## Install

```bash
pip install django-dbs              # on the server: the library + management commands
pip install "django-dbs[client]"    # on your machine: the dbs-client command
```

Add the app on the server:

```python
INSTALLED_APPS = [..., "dbs"]
```

The `[client]` extra adds `paramiko` (SSH/SFTP) and, on Python 3.9/3.10, a TOML
parser. `[ssh]` remains available as the server-side SFTP extra.

> The `[client]` extra still installs Django, because DBS depends on it. You do
> not need a Django project on the machine running `dbs-client` — the client
> never reads Django settings.

---

# On the server

## 1. (Optional) declare what to back up

By default DBS auto-discovers every model, treats `FileField`/`ImageField` as
files, and preserves relations. Register a model only to override the defaults —
in a `dbs.py` module inside your app (auto-discovered like `admin.py`):

```python
# myapp/dbs.py
from dbs import backup_registry, FieldType, ModelBackup
from .models import Invoice

@backup_registry.register(Invoice)
class InvoiceBackup(ModelBackup):
    overrides = {
        "scanned_pdf_path": FieldType.FILE_PATH,  # CharField holding a path -> embed the file
        "render_cache": FieldType.EXCLUDE,        # don't back this column up
    }
    file_roots = ["/srv/myapp/uploads"]           # extra non-model file trees
```

`FieldType` values: `VALUE` (default), `FILE` (embed a FileField's bytes),
`FILE_PATH` (a string column whose path's file is embedded), `EXCLUDE`.

## 2. Back up / restore / validate from the CLI

```bash
python manage.py dbs_backup   backup.dbs            # prompts for a passphrase
python manage.py dbs_validate backup.dbs            # structural check, no passphrase
python manage.py dbs_validate backup.dbs -p secret  # + verify decryption
python manage.py dbs_restore  backup.dbs            # restore rows + files
```

The passphrase comes from `--passphrase`, then `$DBS_PASSPHRASE`, then a prompt.
`--passphrase-stdin` reads it from the first line of standard input instead,
which keeps it out of the process's command line:

```bash
printf '%s\n' "$SECRET" | python manage.py dbs_backup backup.dbs --passphrase-stdin
```

## 3. Automatic backups on a schedule

`dbs_schedule` runs backups on a repeating interval, prunes old ones, and can
push each backup to another host in the same cycle.

```bash
export DBS_PASSPHRASE='…'
python manage.py dbs_schedule --interval 6h --output-dir /var/backups/myproject --keep 14
```

| Flag | Meaning |
|---|---|
| `--interval` | How often to run: `90s`, `30m`, `6h`, `1d`. Defaults to `DBS_SCHEDULE_INTERVAL`, else 24h. |
| `--output-dir` | Where backups are written. Defaults to `DBS_BACKUP_DIR`. |
| `--prefix` | Filename prefix, so several projects can share a directory. |
| `--keep` | How many local backups to retain. |
| `--push` | Name of a `DBS_SSH_TARGETS` entry to also upload to. |
| `--keep-remote` | How many pushed backups to retain. |
| `--once` | Run one cycle and exit — use this from cron, or to test the setup. |
| `--database` `--no-compress` `--no-verify` `--block-size` `--kdf-time` `--kdf-memory` | As for `dbs_backup`. |

It reads the passphrase from `$DBS_PASSPHRASE` and **never prompts**; if the
variable is missing it exits immediately with an explanation rather than hanging
on an invisible prompt. A cycle that fails is logged and the schedule continues.

Backups are named `prefix-YYYYMMDD-HHMMSSZ.dbs` in UTC, so sorting by name is
sorting by time. Retention only ever considers files matching that convention
and the configured prefix — anything else in the directory is left alone, and
`--keep` must be at least 1.

**systemd:**

```ini
[Unit]
Description=DBS scheduled backups
After=network-online.target

[Service]
User=deploy
WorkingDirectory=/srv/myproject
Environment=DBS_PASSPHRASE=…
ExecStart=/srv/myproject/.venv/bin/python manage.py dbs_schedule --interval 6h \
          --output-dir /var/backups/myproject --keep 14
Restart=always

[Install]
WantedBy=multi-user.target
```

`SIGTERM` stops the loop straight away rather than after the current interval,
so `systemctl restart` and container shutdowns are immediate. Run **one
scheduler per output directory** — two loops sharing a directory will race on
retention. Prefer `Environment=` in a `chmod 600` unit file, or a systemd
credential, over a passphrase in a shell profile.

**cron / Docker:** use `--once` from cron for a single cycle, or run the loop as
the container's main process.

## 4. Download / upload from the admin UI

```python
# urls.py
urlpatterns += [path("dbs/", include("dbs.contrib.urls"))]
```

Superusers can then visit `/dbs/backup/` to download an encrypted backup and
`/dbs/restore/` to upload one. The passphrase is entered in the form and never
stored server-side.

## 5. Ship a backup to another server (SFTP)

```python
# settings.py — profiles reference a key by path; no secrets embedded
DBS_SSH_TARGETS = {
    "offsite": {
        "host": "backups.example.com", "username": "deploy",
        "key_filename": "/home/deploy/.ssh/id_ed25519",
        "remote_dir": "/var/backups/myproject",
        "known_hosts": "/home/deploy/.ssh/known_hosts",
    }
}
```

```python
from dbs import create_backup
from dbs.transports import SSHTarget, push_backup

data = create_backup("my passphrase")
push_backup(data, "backup.dbs", SSHTarget.from_settings("offsite"))
```

---

# On your machine: `dbs-client`

`dbs-client` connects to a server where django-dbs is installed, asks it to make
a fresh backup, and downloads the file — all over one encrypted SSH connection.
It can also list, pull, push, prune, and run on a schedule of its own.

```bash
pip install "django-dbs[client]"
dbs-client init                 # writes dbs-client.toml, mode 600
$EDITOR dbs-client.toml
dbs-client test-connection
dbs-client backup
```

## Configuration

`dbs-client` reads the first config it finds: `--config PATH`, then
`$DBS_CLIENT_CONFIG`, then `./dbs-client.toml`, then `~/.config/dbs/client.toml`.
`dbs-client init --print` writes an annotated template to standard output.

```toml
[defaults]
server = "production"
dest = "~/dbs-backups"
keep = 14
keep_remote = 7
interval = "6h"

[servers.production]
host = "app.example.com"
username = "deploy"
port = 22
key_filename = "~/.ssh/production.pem"
known_hosts = "~/.ssh/known_hosts"

project_dir = "/srv/myproject"
python = "/srv/myproject/.venv/bin/python"
manage = "manage.py"
django_settings_module = "myproject.settings.production"
remote_dir = "/var/backups/myproject"

passphrase_env = "DBS_PASSPHRASE"
prefix = "production"
keep = 30
```

Anything under `[defaults]` applies to every server unless that server overrides
it. Pick a server with `--server NAME`, before or after the subcommand.

| Key | Meaning |
|---|---|
| `host` · `username` · `port` | Where to connect. Required: `host`, `username`. |
| `key_filename` | `.pem` or OpenSSH private key. `~` is expanded. |
| `key_passphrase` · `key_passphrase_env` | For an encrypted key file. |
| `password` · `password_env` | Password authentication. |
| `use_agent` | Use ssh-agent (default `true`). |
| `known_hosts` · `auto_add_host_key` | Host key verification; see below. |
| `connect_timeout` | Seconds to wait for the SSH handshake. |
| `remote_dir` | Where backups live on the server. Created if missing. |
| `project_dir` · `python` · `manage` · `django_settings_module` | How to run `manage.py` on the server. |
| `env` | Extra environment variables for the remote command. |
| `passphrase` · `passphrase_env` | The **backup encryption** passphrase. |
| `passphrase_transport` | `stdin` (default) or `env`; see below. |
| `database` | Database alias to back up. |
| `dest` · `prefix` · `keep` · `keep_remote` · `interval` | Local defaults for this server. |
| `exec_timeout` | Seconds to allow the remote backup to run. |

A key that isn't recognised is an error, not a warning — a typo like
`known_host` would otherwise silently disable host key checking.

If the file holds a literal `password`, `passphrase` or `key_passphrase` and is
readable by other users, `dbs-client` refuses to run and tells you to
`chmod 600` it. A config writable by other users is refused regardless, because
it decides which command runs on your server.

## Commands

| Command | What it does |
|---|---|
| `init` | Write a starter config (mode 600, never overwrites). `--print` dumps it to stdout. |
| `test-connection` | Check auth, host key policy, remote directory, the server's DBS version, and that `manage.py dbs_backup` runs. |
| `list` | List remote backups with size and time. `--json`, `--all`, `--prefix`. |
| `backup` | Make a backup on the server and download it. |
| `pull NAME` / `pull --latest` | Download a backup that already exists. |
| `push PATH` | Upload a local backup file to the server. |
| `prune` | Apply retention locally, remotely, or both. `--dry-run` shows what would go. |
| `schedule` | Repeat `backup` on an interval. `--once` runs a single cycle. |
| `validate PATH` | Check a local backup file. `--structure-only` skips decryption. |

```bash
dbs-client test-connection
# [ok] ssh authentication      deploy@app.example.com:22
# [ok] host key policy         reject-unknown
# [ok] remote shell            Linux 6.1.0
# [ok] remote directory        /var/backups/myproject
# [ok] django-dbs on server    0.2.0
# [ok] manage.py dbs_backup    ready
# [ok] backups present         7

dbs-client backup --keep 14 --keep-remote 7
dbs-client pull --latest
dbs-client prune --keep 7 --local-only --dry-run
dbs-client schedule --interval 6h --keep 30
```

`backup` triggers `manage.py dbs_backup` on the server, downloads the result to
`dest`, validates it locally, then applies retention. `--no-fetch` leaves it on
the server, `--delete-remote` removes the remote copy once it is safely
downloaded, and `--no-validate` skips the local check. Downloads stream to a
`.part` file and are renamed into place only once complete, so an interrupted
transfer can never be mistaken for a good backup.

`schedule` uses the same loop as the server-side command: `SIGTERM` and `Ctrl-C`
stop it promptly, and a failing cycle is logged without ending the schedule. It
resolves the passphrase once at start-up and never prompts.

`validate` needs no Django project and no settings — it works anywhere the file
is.

## SSH authentication

| Method | Configure |
|---|---|
| `.pem` or OpenSSH key | `key_filename = "~/.ssh/production.pem"` |
| Encrypted key file | the above plus `key_passphrase_env = "…"` (or `key_passphrase`) |
| Password | `password_env = "…"` (or `password`) |
| ssh-agent | name neither a key nor a password; `use_agent` defaults to `true` |

Any secret can be given directly or read from an environment variable by adding
`_env` to the key name, which keeps it out of the file entirely.

Host keys are verified and **unknown hosts are rejected by default**. Point
`known_hosts` at a file (`~/.ssh/known_hosts` is typical) or leave it unset to
use the system host keys. `auto_add_host_key = true` trusts whatever key the
server presents on first contact — `test-connection` always prints which policy
is in effect so this doesn't get left on by accident.

## Passphrases in unattended runs

The backup passphrase never appears in a command line — not locally, and not on
the server. There is no `--passphrase` flag on `dbs-client` at all, because
process arguments are readable by every user on the machine via `/proc`.

* **`passphrase_transport = "stdin"` (default, needs django-dbs ≥ 0.2.0 on the
  server).** The client runs `manage.py dbs_backup --passphrase-stdin` and writes
  the passphrase down the encrypted SSH channel's standard input. It is in
  neither the remote process's arguments nor its environment, and never touches
  the server's disk.
* **`passphrase_transport = "env"` (for servers still on 0.1.x).** The passphrase
  is still sent over stdin, but a shell reads it into `DBS_PASSPHRASE` for the
  backup process. It is then visible in that process's environment to root and
  to the same user — the same exposure as a systemd `Environment=` line, and
  better than a command-line argument.

Locally the passphrase is resolved from `passphrase_env`, then a literal
`passphrase`, then `$DBS_PASSPHRASE`, then an interactive prompt.

---

## Python API

```python
from dbs import create_backup, restore_backup, validate_backup

blob = create_backup("passphrase", output="backup.dbs")
report = validate_backup(blob, "passphrase")   # report.ok / report.summary()
result = restore_backup(blob, "passphrase")    # result.healed is True if it repaired corruption
```

```python
from dbs.transports import SSHTarget, open_session

with open_session(SSHTarget.from_settings("offsite")) as session:
    session.push("backup.dbs", "backup.dbs")
    for item in session.details():
        print(item.name, item.size, item.modified)
    session.delete("old-backup.dbs")
```

`open_session` reuses one connection for a whole trigger-fetch-prune cycle. The
module-level `push_backup`, `pull_backup`, `pull_backup_to`, `list_backups`,
`list_backup_details`, `delete_backup` and `check_connection` helpers open and
close a connection each.

```python
from dbs.client import load_client_config, resolve_client_passphrase, backup_and_fetch

profile = load_client_config().server("production")
result = backup_and_fetch(profile, resolve_client_passphrase(profile), keep=14)
print(result.local_path, result.size)
```

## How it heals

On write, the encrypted stream is split into blocks; each block gets a BLAKE2b
hash and a layer of Reed-Solomon parity, and the whole stream is stored **twice**
(plus the header and manifest are stored twice). On read, each block is taken
from whichever copy verifies; sparse bit-flips are corrected in place by
Reed-Solomon even when *both* copies are hit. Every recovered block is checked
against its stored hash, so a mis-correction can never slip through — and a
freshly written backup is re-read and verified end-to-end (**verify-after-write**)
before the command reports success.

What it can recover from: whole-block loss in one copy, and sparse byte errors
(up to the parity budget, ~8 bytes per 255-byte codeword by default) in both
copies. What it cannot: a block destroyed *beyond* the parity budget in **both**
copies — DBS then refuses to restore and reports exactly which blocks failed,
rather than producing silently wrong data.

## Security model

* **Argon2id** derives a key from your passphrase (memory-hard ⇒ brute-force
  resistant). Raise `KDFParams` cost for more resistance.
* **Envelope encryption**: a random data key encrypts the payload with
  **AES-256-GCM**; that data key is wrapped by the passphrase-derived key. The
  file stores only the salt, Argon2 parameters and the wrapped key — **never the
  passphrase and never the raw data key**.
* A wrong passphrase fails the GCM tag check and is reported as such; it can
  never yield partial/garbage data.
* Transfers run over SSH/SFTP with host key verification on by default.
* Passphrases are never passed as command-line arguments; see *Passphrases in
  unattended runs* above.

## Restore semantics

* The database load runs in a **single transaction**: if anything fails midway,
  every row is rolled back and the database is left untouched.
* Backup collection also runs in a transaction, so a backup is a consistent
  snapshot even while the application keeps writing.
* Writing a `.dbs` file is **atomic and durable** (temp file + fsync + rename),
  and verify-after-write re-reads the bytes that actually landed on disk. Uploads
  and downloads land the same way, via a `.part` file that is renamed on success.
* File restores are **confined**: `FILE_PATH` and file-root entries are only
  written when the target resolves under a directory listed in
  `DBS_RESTORE_ROOTS` (or, if unset, `DBS_FILE_ROOTS`). Anything else raises
  `RestoreError`. Configure the allow-list before restoring backups that embed
  external files.
* Restoring merges by primary key into the target database; rows created after
  the backup was taken are not deleted.
* `dbs-client` never restores into a server's database. Use `push` to place a
  file there, then run `dbs_restore` on the server deliberately.

## Observability

DBS logs to the `dbs` logger: backup/restore start and finish, skipped source
files, SFTP transfers, scheduled cycles and retention, and — most importantly —
a `WARNING` whenever silent corruption was detected and healed. Route that
logger to your monitoring so a degrading disk is noticed before both copies are
damaged:

```python
LOGGING = {
    "version": 1,
    "loggers": {"dbs": {"handlers": ["console"], "level": "INFO"}},
}
```

`dbs-client --verbose` prints the same log to stderr. Files that could not be
read during a backup are also recorded in the manifest
(`validate_backup(...).stats["skipped_files"]`).

## Settings reference

| Setting | Purpose |
|---|---|
| `DBS_EXCLUDE_MODELS` | `["app.Model", ...]` to skip (defaults skip contenttypes, permissions, admin log, sessions). |
| `DBS_FILE_ROOTS` | Extra directories embedded in every backup. |
| `DBS_RESTORE_ROOTS` | Directories file restores may write into (falls back to `DBS_FILE_ROOTS`). |
| `DBS_SSH_TARGETS` | Named SFTP connection profiles. |
| `DBS_BACKUP_DIR` | Default output directory for `dbs_schedule`. |
| `DBS_BACKUP_PREFIX` | Default backup filename prefix. |
| `DBS_SCHEDULE_INTERVAL` | Default interval for `dbs_schedule` (default `24h`). |
| `DBS_SCHEDULE_KEEP` | Default local retention count (default 7). |
| `DBS_SCHEDULE_PUSH_TARGET` | Default `DBS_SSH_TARGETS` entry to push to. |
| `DBS_SCHEDULE_KEEP_REMOTE` | Default retention count for pushed backups. |
| `DBS_MAX_UPLOAD_BYTES` | Admin restore upload size cap (default 1 GiB). |
| `DBS_MAX_PAYLOAD_BYTES` | Decompressed payload size cap on restore (default 4 GiB). |
| `DBS_KDF_TIME_COST` / `DBS_KDF_MEMORY_COST` / `DBS_KDF_PARALLELISM` | Default Argon2id cost parameters for new backups. |

## Development

```bash
pip install -e ".[dev]"
pytest
```

## License

MIT
