Metadata-Version: 2.4
Name: horde-image-utilities
Version: 0.2.1
Summary: Modular image utility microservice with optional heavy backends
License-File: LICENSE
License-File: NOTICE
Requires-Python: >=3.12
Requires-Dist: pillow>=11.0.0
Requires-Dist: pydantic-settings>=2.13.1
Requires-Dist: pydantic>=2.13.0
Provides-Extra: annotators
Requires-Dist: addict; extra == 'annotators'
Requires-Dist: einops; extra == 'annotators'
Requires-Dist: huggingface-hub; extra == 'annotators'
Requires-Dist: matplotlib; extra == 'annotators'
Requires-Dist: numpy; extra == 'annotators'
Requires-Dist: opencv-contrib-python-headless; extra == 'annotators'
Requires-Dist: pydot; extra == 'annotators'
Requires-Dist: pyyaml; extra == 'annotators'
Requires-Dist: scikit-image; extra == 'annotators'
Requires-Dist: scipy; extra == 'annotators'
Requires-Dist: timm; extra == 'annotators'
Requires-Dist: torch; extra == 'annotators'
Requires-Dist: torchvision; extra == 'annotators'
Requires-Dist: transformers; extra == 'annotators'
Requires-Dist: yapf; extra == 'annotators'
Provides-Extra: mediapipe
Requires-Dist: mediapipe; extra == 'mediapipe'
Provides-Extra: rembg-cpu
Requires-Dist: rembg[cpu]; extra == 'rembg-cpu'
Provides-Extra: rembg-cuda
Requires-Dist: onnxruntime-gpu; extra == 'rembg-cuda'
Requires-Dist: rembg[gpu]; extra == 'rembg-cuda'
Provides-Extra: rembg-rocm
Requires-Dist: onnxruntime-rocm; (sys_platform == 'linux') and extra == 'rembg-rocm'
Requires-Dist: rembg[rocm]; (sys_platform == 'linux') and extra == 'rembg-rocm'
Provides-Extra: server
Requires-Dist: fastapi[standard]>=0.135.3; extra == 'server'
Description-Content-Type: text/markdown

# horde-image-utilities

Isolated heavy image capabilities for the AI Horde worker ecosystem.

Some image utilities a worker needs (background removal via rembg/onnxruntime, and later controlnet
annotators and layer diffusion) carry heavy, conflict-prone native dependencies. Pulling those into the
worker's main environment bloats it and risks dependency clashes. This package keeps them out of the
way: it runs each capability as a small FastAPI service in its own virtual environment, and the worker
talks to it over loopback HTTP using a dependency-free client.

It ships four cooperating pieces:

- **Service**: a FastAPI application exposing capability endpoints (for example `POST /rembg/remove-background`) plus an always-on operational surface under `/ops`.
- **Client**: a standard-library-only HTTP client (`HordeImageUtilitiesClient`) that the lean consumer uses to call the service. Its only runtime dependency is Pillow, already a base dependency.
- **Launcher**: `CapabilityServerProcess` starts, health-checks, and tears down the service subprocess, optionally under a different interpreter so the heavy dependencies live in a separate venv.
- **Provisioning**: `build_provision_commands` produces the `uv` commands that create a capability venv and install this package with the right extras.

## Two-mode model

- **Lean consumer environment**: installs the base package only (Pillow, pydantic). It imports `horde_image_utilities.client` and `horde_image_utilities.launcher` to drive a capability process. No FastAPI, rembg, torch, or onnxruntime enter this environment.
- **Capability environment**: a separate venv that installs the `server` extra plus exactly one backend extra. It runs the actual model inference. Keeping this environment separate is what makes the dependency isolation real.

## Install matrix

The base install has no heavy backends. Capabilities are opt-in extras:

| Extra | Provides |
| --- | --- |
| `server` | FastAPI transport for running the service (`fastapi[standard]`). |
| `rembg_cpu` | Background removal on CPU (`rembg[cpu]`). |
| `rembg_cuda` | Background removal on NVIDIA CUDA (`rembg[gpu]` + `onnxruntime-gpu`). |
| `rembg_rocm` | Background removal on AMD ROCm, Linux only (`rembg[rocm]` + `onnxruntime-rocm`). |
| `annotators` | ControlNet preprocessors (canny, hed, depth, normal, openpose, seg, ...) via torch. A single extra serves every accelerator backend; the torch wheel index is the installer's concern. |
| `mediapipe` | MediaPipe face-mesh detector (`mediapipe_face`). |

The three `rembg_*` extras are mutually exclusive: exactly one accelerator backend per environment. This
is enforced by a `[tool.uv].conflicts` declaration, so `uv` refuses to install more than one. The
`annotators` extra has no accelerator split (no detector in scope uses onnxruntime), so no conflict
entry is needed.

> **Do not co-install `mediapipe` with `annotators` in the same environment.** `mediapipe` pulls in
> `opencv-contrib-python` (the GUI wheel) while `annotators` pulls in `opencv-contrib-python-headless`.
> Two opencv wheels sharing the `cv2` namespace break `cv2` (it imports as an empty namespace). Until
> this is resolved via a dependency override at deployment-pin compile time, keep `mediapipe` in its own
> environment. The `mediapipe_face` detector is future work and is not required by any worker capability.

Provision a capability venv (CUDA background removal shown):

```bash
uv venv .venv-rembg
uv pip install --python .venv-rembg "horde-image-utilities[server,rembg_cuda]"
```

## Quickstart

Run the service:

```bash
# In the capability environment
python -m horde_image_utilities
# or the console script:
horde-image-utilities
```

Call it from the lean consumer environment:

```python
from PIL import Image

from horde_image_utilities.client import HordeImageUtilitiesClient

client = HordeImageUtilitiesClient("http://127.0.0.1:7860")
if client.health():
    result = client.remove_background(Image.open("input.png"))
    result.save("output.png")
```

Or let the launcher own the subprocess lifecycle (point `python_executable` at the capability venv):

```python
from horde_image_utilities.launcher import CapabilityServerProcess

with CapabilityServerProcess(python_executable="/path/to/.venv-rembg/bin/python") as server:
    memory_report = server.client.get_memory_report()
```

## Configuration

Settings are read from the environment with the `HIU_` prefix (see
`horde_image_utilities.config.ServiceSettings`):

| Variable | Default | Purpose |
| --- | --- | --- |
| `HIU_HOST` | `127.0.0.1` | Bind host for the service. |
| `HIU_PORT` | `7860` | Bind port for the service. |
| `HIU_MAX_REQUEST_SIZE` | `67108864` | Maximum accepted request size in bytes. |
| `HIU_CONTROLNET_MAX_CONCURRENT` | `1` | Concurrent controlnet processing slots. |
| `HIU_CONTROLNET_QUEUE_TIMEOUT` | `30.0` | Seconds to wait for a controlnet slot. |
| `HIU_ISOLATE_MODEL_CACHE` | `true` | Store model files under `AIWORKER_CACHE_HOME` when set. |
| `HIU_ALLOW_DOWNLOADS` | `false` | Allow runtime model downloads when missing from cache. |
| `HIU_ANNOTATOR_MODEL_DIR` | unset | Base directory holding annotator checkpoints. Defaults to the `annotators/` cache directory below. |
| `HIU_VENDOR_DIR` | unset | Base directory for the cloned upstream annotator node. Defaults to a `vendor/` sibling of the cache directories below. |

`HIU_ALLOW_DOWNLOADS` defaults to `false`: the service will not fetch missing models at runtime unless
you opt in. Pre-download models, or set it to `true`.

## Cache layout

When `HIU_ISOLATE_MODEL_CACHE` is enabled and `AIWORKER_CACHE_HOME` is set, model files are stored under
an isolated, per-capability directory:

```
$AIWORKER_CACHE_HOME/horde/image-utilities/rembg/                     # rembg ONNX models
$AIWORKER_CACHE_HOME/horde/image-utilities/controlnet_aux/            # controlnet_aux base
$AIWORKER_CACHE_HOME/horde/image-utilities/controlnet_aux/annotators/ # annotator checkpoints (default)
$AIWORKER_CACHE_HOME/horde/image-utilities/vendor/                    # cloned upstream annotator node (default)
```

If `AIWORKER_CACHE_HOME` is not set, or isolation is disabled, each backend falls back to its own
default (`U2NET_HOME` / `~/.u2net` for rembg, the Hugging Face cache for controlnet). The service logs a
warning when conflicting cache environment variables are detected.

### Annotator checkpoints

The annotators capability resolves weight files under `HIU_ANNOTATOR_MODEL_DIR` (defaulting to the
`annotators/` directory shown above), laid out as `<repo_id>/<filename>` exactly as the worker
pre-populates them (for example `lllyasviel/Annotators/ControlNetHED.pth`). Missing weights return HTTP
409 rather than triggering a download (downloads are off unless `HIU_ALLOW_DOWNLOADS=true`).

The transformers-hub control types are the exception: their weights load through the transformers
library from the standard Hugging Face hub cache, so the deployment must point `HF_HOME` (or
`HUGGINGFACE_HUB_CACHE`) at a cache pre-populated with the relevant model. These are `normal` and
`midas_depth` (`Intel/dpt-hybrid-midas`), `zoe_depth` (`Intel/zoedepth-nyu-kitti`), `depth_anything`
(`LiheYoung/depth-anything-large-hf`), and `oneformer_ade20k` / `oneformer_coco`
(`shi-labs/oneformer_*_swin_large`).

Implemented control types span the classic set and the extended surface the AI-Horde annotation form
and worker pre-annotation use:

- Classic: `canny`, `scribble`, `hed`, `fakescribbles`, `mlsd`, `depth` (LeReS), `normal` (MiDaS),
  `openpose`, `seg` (UniFormer).
- Weightless algorithmic: `binary`, `standard_lineart`, `scribble_xdog`, `pyracanny`, `color`,
  `shuffle`, `recolor_luminance`, `recolor_intensity`, `tile`, `tile_ttplanet_guided`,
  `tile_ttplanet_simple`.
- Annotator-layout weighted: `lineart`, `lineart_anime`, `lineart_anime_denoise`,
  `pidinet`, `scribble_pidinet`, `teed`, `normal_bae`, `depth_anything_v2`.
- Transformers-hub weighted: `midas_depth`, `zoe_depth`, `depth_anything`, `oneformer_ade20k`,
  `oneformer_coco`.

Per-detector parameters match the defaults hordelib passes through the `comfyui_controlnet_aux` AIO
preprocessor (only the resolution knob is set; every other widget stays at its node default), so
pre-annotation here is interchangeable with hordelib's in-graph annotation. Notably `lineart`
runs with coarse disabled, mirroring hordelib, which drives it through the
`LineArtPreprocessor` node without setting the coarse widget.

The `seg` runtime comes from the clone's bundled MMSegmentation/MMCV subtree, which runs on CPU without
compiled ops given the pure-python MMCV config packages the `annotators` extra pulls in. The
`mediapipe_face` control type is registered but not implemented and returns HTTP 501.

At startup a best-effort prefetch (enabled only when `HIU_ALLOW_DOWNLOADS=true`) warms the classic
control types' weights before the extended set, so a fresh install can serve the legacy surface quickly
while the extended weights trickle in.

### Annotator detector source (clone at a pin)

The annotator detector algorithms are not vendored into this repository. On first use (and, best-effort,
at service startup) the capability clones [`comfyui_controlnet_aux`](https://github.com/Fannovel16/comfyui_controlnet_aux)
at a commit pinned in `horde_image_utilities/annotators/upstream_manifest.json`, prepends the clone's
`src` to `sys.path`, and imports `custom_controlnet_aux` directly (the clone is not pip-installed). This
mirrors how hordelib pins ComfyUI: the acquisition is idempotent and self-healing, performing no network
access once the clone is already at the pin. Two runtime monkeypatches redirect the upstream weight
loading through this package's resolver (so pre-populated checkpoints are used and missing files return
HTTP 409) and make the LeReS detector load only its depth model, never the pix2pix "boost" model. See
the repository `NOTICE` for the attribution and the exact patch sites.

Pre-bake the clone for an offline or image-baking step (git only; needs neither torch nor the annotator
extra):

```bash
python -m horde_image_utilities.annotators
```

## The `/ops` surface

The ops endpoints mount in every install, including lean ones with no heavy backend. They assume a
trusted loopback caller (the worker that launched the process) and carry no authentication.

| Endpoint | Purpose |
| --- | --- |
| `GET /ops/memory` | Report process resident-set size, torch CUDA figures (null without torch), the onnxruntime session count, and loaded model names. |
| `POST /ops/release-cache` | Release framework caches (torch's CUDA allocator when present) and return a fresh memory report. |
| `POST /ops/unload` | Drop cached sessions for one capability (`{"capability": "background_removal"}`) or all (`{"capability": null}`). |
| `POST /ops/shutdown` | Schedule a graceful process exit after the response flushes. |

The client exposes typed methods for each: `get_memory_report()`, `release_cache()`,
`unload(capability=None)`, and `shutdown()`.

A `GET /health` liveness endpoint is always available regardless of which backends are installed.

## Development

```bash
uv sync
uv run ruff format . && uv run ruff check . --fix
uv run pytest
```

Tests that require a rembg backend are skipped automatically when it is not installed. The
package version is derived from the release tag via `hatch-vcs`; there is no version string to bump in
source.
