Metadata-Version: 2.4
Name: hop3-testing
Version: 0.8.0
Summary: Testing utilities for Hop3
Author: Stefane Fermigier
Author-email: Stefane Fermigier <sf@abilian.com>
License-Expression: Apache-2.0
Requires-Dist: hop3-cli
Requires-Dist: hop3-installer
Requires-Dist: pytest>=8.0.0
Requires-Dist: docker>=7.0.0
Requires-Dist: tomli>=2.0.0 ; python_full_version < '3.11'
Requires-Dist: devtools>=0.12.2
Requires-Dist: python-dotenv>=1.0.1
Requires-Dist: httpcore>=1.0.6
Requires-Dist: httpx>=0.27.2
Requires-Dist: paramiko>=3.4.0,<4
Requires-Dist: click>=8.0.0
Requires-Dist: sqlalchemy>=2.0.0
Requires-Dist: pyjwt>=2.8.0
Requires-Dist: hcloud>=2.0
Requires-Dist: jinja2>=3.0
Requires-Dist: rich>=13.0
Requires-Dist: attrs>=23.0
Requires-Dist: termcolor>=2.0
Requires-Dist: tomlkit>=0.12
Requires-Python: >=3.10, <4
Description-Content-Type: text/markdown

# hop3-testing

Testing framework for Hop3 deployment validation.

## Overview

hop3-testing provides utilities and fixtures for testing Hop3 deployments. It supports running tests against Docker containers or remote servers, with a catalog of test applications covering various languages and frameworks.

## Features

- **Multiple targets**: Test against Docker containers or remote SSH servers
- **App catalog**: Pre-built test applications for various languages
- **Deployment sessions**: Automated deploy/verify/cleanup workflow
- **State checks**: after the validations, an app fails if it wrote into its `src/` while running (every deploy replaces `src/`, so that data is lost at the next one; the failure lists the directories to declare as `[[volumes]]`). The unchanged source is then deployed again and every validation runs a second time
- **pytest fixtures**: Integration with pytest for E2E testing
- **Category filtering**: Run tests by language or framework

## Installation

```bash
pip install hop3-testing
```

## Quick Start

```bash
# List available tests
hop3-test list

# Run system tests on Docker
hop3-test run --docker

# Test specific apps
hop3-test apps 010-flask-pip-wsgi

# Test against remote server
hop3-test apps --host server.example.com

# Run CI tests
hop3-test ci
```

## Commands

| Command | Description |
|---------|-------------|
| `hop3-test run` | Deploy Hop3 and run the catalog (`system` is a deprecated alias) |
| `hop3-test run --reuse` | Test against an existing deployment (skip deploy) |
| `hop3-test list` | List available tests (`--show NAME` for one test's details) |
| `hop3-test run --provider hetzner --images ...` | E2E across cloud OS images (Hetzner) |
| `hop3-test upgrade-chain` | Install a baseline release on a fresh box, then upgrade in-place through a version chain |
| `hop3-test why <run-id>` | Replay a saved diagnostic bundle for a failed run |

Test profile (fast/CI/full) is selected with `--mode` on `run`, not a separate
subcommand: `hop3-test run --docker --mode ci`.

### Upgrade chain

`hop3-test upgrade-chain` validates that a running server survives a *chain* of in-place upgrades. Each hop is a git ref, installed by **that version's own** `hop3-deploy-server` (checked out into a worktree and run via `uv run`), on a **fresh** box; every hop after the first is an in-place update, and each is asserted to come back healthy with a readable schema.

The first hop also seeds the state an upgrade must carry: an app (`010-flask-pip-wsgi`), deployed once and never again, and a plain account. After every hop the app must still serve its page, the admin must still list the accounts, and the account must do exactly what it could on the first hop (get a token, read its app's environment, list the accounts), probed by doing each. That is how the chain checks a migration such as ADR 061's roles, which must turn every non-admin into a deployer. Before a release, run it from the last release: `--chain <last-tag>,local`.

```bash
# Fresh Docker container: 0.6.2 → current tree
hop3-test upgrade-chain --docker

# Fresh Hetzner VPS (needs HETZNER_API_TOKEN + HETZNER_SERVER_ID)
hop3-test upgrade-chain --provider hetzner --image ubuntu-24.04

# Custom chain (release tags + `local` for the current tree)
hop3-test upgrade-chain --docker --chain 0.6.2,local
```

Cheapest smoke — exercises the whole mechanism (fresh install → in-place
upgrade → assertions) with no old-version/worktree variable:

```bash
hop3-test upgrade-chain --docker --chain local,local
```

`--host <server>` is accepted but warns: it targets an existing server, not the clean slate an upgrade chain assumes. `0.6.0` is not a viable baseline (its `hop3-rootd` can't start) and is excluded from the default chain.

### Common Options

| Option | Description |
|--------|-------------|
| `-v, --verbose` | Verbose output |
| `--fail-fast` | Stop on first failure |
| `--keep` | Keep apps deployed after testing |
| `--no-redeploy` | Deploy each app once, skipping the second deploy and validation pass |
| `--docker` | Use Docker target |
| `--host HOST` | Remote server hostname |
| `--screenshots` | Photograph each catalog recipe signed in, into its `screenshots/` (a server only) |
| `--screenshots-to DIR` | Also copy every image into `DIR` (implies `--screenshots`) |

### Screenshots, and the sign-ins only a browser can check

`--screenshots` drives a real browser to each passed catalog recipe before it is destroyed: its sign-in page, then its first page once signed in with the app's admin credential. The images land in the recipe's `screenshots/` in the catalog checkout, which the catalog publishes. The browser reaches the app over HTTPS at its test hostname, resolved to the server, so it needs a server target; on Docker the flag is refused. Playwright is loaded only for this:

```bash
uv run --with playwright playwright install chromium     # once
uv run --with playwright hop3-test run --host <server> --recipes published --screenshots
```

Some apps build their sign-in form in JavaScript, so their `check.py` verifies everything else and leaves the sign-in to a browser (`SIGN-IN VERIFIED BY BROWSER`). With `--screenshots`, a browser that cannot sign in fails the test. Without it, the test is reported as **not verified**: listed by name in the summary, stored with the status `unverified`, and not counted as a failure, since nothing found the app broken. A release run uses `--screenshots`, so those apps count only once a browser has signed in.

### Features follow the apps

The server is installed with what the selected apps declare: their addons (`[[addons]]`) and the runtime their builder needs (`nix` for a Nix-built app, `docker` for a Docker-built one). `--with` adds to that. Nothing is installed by default. The Docker target cannot run Docker, so Docker-built apps are left out there, each with the reason; test them against a server with `--host`.

### A server deploy that warns fails

Every ⚠ line the installer or the deployer prints fails the server deploy, unless it matches `TOLERATED_DEPLOY_WARNINGS` in `targets/helpers.py`, where each entry says why continuing is safe. A warning-and-continue is the failure a review misses and a red run does not: two deployer bugs and an install failing on every Ubuntu 24.04 server hid behind warnings every log carried. Fix the warning, or add it with its justification. Old releases' deployers, driven by `upgrade-chain`, are not gated. See [`notes/installer-fail-loud-audit.md`](../../notes/installer-fail-loud-audit.md).

## Architecture

```
hop3-testing/
├── src/hop3_testing/
│   ├── cli/              # CLI commands
│   │   └── commands/     # Click command implementations
│   ├── catalog/          # Test discovery
│   │   ├── scanner.py    # Discovers test.toml files
│   │   └── models.py     # TestDefinition, Category
│   ├── apps/
│   │   ├── catalog.py    # AppSource dataclass
│   │   └── deployment.py # DeploymentSession
│   ├── targets/
│   │   ├── base.py       # DeploymentTarget ABC
│   │   ├── docker.py     # DockerTarget
│   │   └── remote.py     # RemoteTarget
│   └── results/          # Result storage and reporting
└── tests/
```

## Test Categories

| Category | Languages/Frameworks |
|----------|---------------------|
| `python` | Flask, FastAPI, Django |
| `nodejs` | Express, Fastify |
| `ruby` | Sinatra, Rails |
| `go` | Fiber, Gin |
| `rust` | Actix-web, Axum |
| `static` | HTML, Hugo, Jekyll |

## Cloud Runs & the Image Sweep

Run full E2E tests across cloud OS images with `hop3-test run --provider hetzner
--images ...`. Each image is a full `hop3-test run --provider hetzner` (provision a
fresh box → deploy → test → persist), so a cloud run shares `run`'s lexicon:
positional app names, `--from`, `--branch`, `--with`. Requires `HETZNER_API_TOKEN`
and `HETZNER_SERVER_ID` (a dedicated throwaway box). A rebuild wipes the box, so
it refuses any server that does not carry the Hetzner label
`hop3-throwaway=true`; set it once, on that server only:
`hcloud server add-label $HETZNER_SERVER_ID hop3-throwaway=true`.

```bash
# List available images
hop3-test run --list-images

# Single distribution (a sweep-of-one)
hop3-test run --provider hetzner --image ubuntu-24.04 apps/test-apps-procfile

# Across multiple distributions
hop3-test run --provider hetzner --images ubuntu-24.04,debian-13,fedora-42

# From PyPI instead of local code
hop3-test run --provider hetzner --from pypi --images all
```

### Cloud Run Options (`run --provider hetzner`)

| Option | Description |
|--------|-------------|
| `--image IMAGE` | Single OS image — a sweep-of-one (e.g. ubuntu-24.04) |
| `--images LIST` | Comma-separated images or `all` |
| `--list-images` | List available OS images |
| `--from {local,git,pypi}` | Install source (same as `run`; default: local) |
| `--branch BRANCH` | Git branch (with `--from git`; default: devel) |
| `--with FEATURES` | Extra server features (repeatable or comma-separated) |
| `-x, --fail-fast` | Stop on the first failing image |
| `--continue-on-failure` | Don't stop on first failure (multi-distro) |

## Development

```bash
# Run tests
uv run pytest tests/ -v

# Lint and format
uv run ruff check src/
uv run ruff format src/
```

## Documentation

- [Testing Strategy](../../docs/src/dev/testing-strategy.md)

## Related Packages

- [hop3-server](../hop3-server/) - The server being tested
- [hop3-cli](../hop3-cli/) - CLI used for deployments in tests

## License

Apache-2.0 - Copyright (c) 2024-2026, Abilian SAS
