Metadata-Version: 2.5
Name: arrakis-monitor
Version: 0.2.0
Summary: HTTP health sidecar for Arrakis Flight servers
Project-URL: Homepage, https://git.ligo.org/ngdd/arrakis-monitor
Project-URL: Documentation, https://docs.ligo.org/ngdd/arrakis-monitor
Project-URL: Repository, https://git.ligo.org/ngdd/arrakis-monitor.git
Project-URL: Issues, https://git.ligo.org/ngdd/arrakis-monitor/issues
Author-email: Olivia Godwin <olivia.godwin@ligo.org>
Maintainer-email: Olivia Godwin <olivia.godwin@ligo.org>
License-Expression: GPL-3.0-or-later
License-File: LICENSE
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: Scientific/Engineering :: Astronomy
Classifier: Topic :: Scientific/Engineering :: Physics
Classifier: Topic :: System :: Monitoring
Requires-Python: >=3.11
Requires-Dist: arrakis-admin>=0.3
Requires-Dist: arrakis>=0.15
Requires-Dist: confluent-kafka>=2.2
Requires-Dist: gpstime
Requires-Dist: starlette
Requires-Dist: uvicorn[standard]
Provides-Extra: dev
Requires-Dist: httpx; extra == 'dev'
Requires-Dist: markdown-callouts>=0.2; extra == 'dev'
Requires-Dist: markdown-exec>=0.5; extra == 'dev'
Requires-Dist: mkdocs-coverage>=0.2; extra == 'dev'
Requires-Dist: mkdocs-gen-files>=0.3; extra == 'dev'
Requires-Dist: mkdocs-literate-nav>=0.4; extra == 'dev'
Requires-Dist: mkdocs-material-igwn; extra == 'dev'
Requires-Dist: mkdocs-section-index>=0.3; extra == 'dev'
Requires-Dist: mkdocs>=1.3; extra == 'dev'
Requires-Dist: mkdocstrings[python]; extra == 'dev'
Requires-Dist: mypy; extra == 'dev'
Requires-Dist: mypy-extensions; extra == 'dev'
Requires-Dist: pip; extra == 'dev'
Requires-Dist: pytest; extra == 'dev'
Requires-Dist: pytest-asyncio; extra == 'dev'
Requires-Dist: pytest-cov; extra == 'dev'
Requires-Dist: pytest-freezer; extra == 'dev'
Requires-Dist: ruff; extra == 'dev'
Requires-Dist: toml>=0.10; extra == 'dev'
Provides-Extra: docs
Requires-Dist: markdown-callouts>=0.2; extra == 'docs'
Requires-Dist: markdown-exec>=0.5; extra == 'docs'
Requires-Dist: mkdocs-coverage>=0.2; extra == 'docs'
Requires-Dist: mkdocs-gen-files>=0.3; extra == 'docs'
Requires-Dist: mkdocs-literate-nav>=0.4; extra == 'docs'
Requires-Dist: mkdocs-material-igwn; extra == 'docs'
Requires-Dist: mkdocs-section-index>=0.3; extra == 'docs'
Requires-Dist: mkdocs>=1.3; extra == 'docs'
Requires-Dist: mkdocstrings[python]; extra == 'docs'
Requires-Dist: toml>=0.10; extra == 'docs'
Provides-Extra: lint
Requires-Dist: mypy; extra == 'lint'
Requires-Dist: mypy-extensions; extra == 'lint'
Requires-Dist: pip; extra == 'lint'
Requires-Dist: ruff; extra == 'lint'
Provides-Extra: test
Requires-Dist: httpx; extra == 'test'
Requires-Dist: pytest; extra == 'test'
Requires-Dist: pytest-asyncio; extra == 'test'
Requires-Dist: pytest-cov; extra == 'test'
Requires-Dist: pytest-freezer; extra == 'test'
Description-Content-Type: text/markdown

# arrakis-monitor

HTTP health sidecar for [Arrakis](https://git.ligo.org/ngdd/arrakis-server)
Flight servers.

## Overview

`arrakis-monitor` runs alongside each Arrakis server and exposes its health
status over HTTP, allowing standard monitoring tools (Nagios/Icinga via NRPE +
`check_http`) to check gRPC Flight servers without needing a gRPC-native plugin.

## Installation

```bash
pip install arrakis-monitor
```

## Usage

```bash
# basic health sidecar
arrakis-monitor --server grpc://localhost:31206

# with scope map domain validation
arrakis-monitor --server grpc://localhost:31206 --expect-domains H1,L1,V1

# with publisher liveliness/latency checks (online servers)
arrakis-monitor --server grpc://localhost:31206 --check-publishers

# custom bind address
arrakis-monitor --server grpc://localhost:31206 --host 127.0.0.1 --port 9090
```

## Endpoints

| Endpoint | Description |
|---|---|
| `GET /` | Show server configuration |
| `GET /health` | Run a health check (calls `server_info()`) |
| `GET /scope-map` | Run a scope map check (calls `scope_map()`) |
| `GET /publishers` | Probe every publisher in the scope map (`--check-publishers`) |
| `GET /publishers/{publisher}` | Probe a single publisher (`--check-publishers`) |

Health and scope-map endpoints return JSON with an HTTP status code:

- **200** — OK or WARNING
- **503** — CRITICAL (server unreachable or scope map empty)
- **500** — UNKNOWN (unexpected error)

### Response format

Both check endpoints include a `latency_s` field measuring the gRPC call
round-trip time in seconds:

```json
{
  "status": "ok",
  "message": "Server 0.12.0 (kafka)",
  "version": "0.12.0",
  "backend": "kafka",
  "domains": ["H1", "L1"],
  "capabilities": ["find", "count", "describe", "stream"],
  "latency_s": 0.042
}
```

## Publisher checks

On online (Kafka-backed) servers, `--check-publishers` enables
per-publisher liveliness and latency monitoring. For each publisher
tagged in the server's scope map, the sidecar consumes the publisher's
metadata topic from the local broker to select a surrogate channel,
then probes the data topic for that channel's partition, waiting for a
fresh message. The block-time latency of the freshest record batch is
compared against the `max_latency` advertised in the channel metadata:

- **OK** — fresh data arrived within the latency budget
- **WARNING** — latency within `--publisher-warn-margin` (default 1 s)
  of `max_latency`
- **CRITICAL** — latency at/past `max_latency`, or no message within
  `max_latency` plus one block stride (reported with the age of the
  last retained message)

`/publishers` probes every publisher concurrently and reports the
worst status; `/publishers/{publisher}` probes one, for fine-grained
per-publisher alerting. Publishers registered in the scope map ahead
of going live would keep the aggregate check CRITICAL, so list them
with `--ignore-publisher` — `/publishers` skips them (reporting them
under `ignored`) while `/publishers/{publisher}` still probes them on
request. Surrogate resolution is cached per publisher
(`--metadata-ttl`, default 600 s), and check results are cached
briefly (`--probe-cache-ttl`, default 60 s) so aggregate and
per-publisher checks polling around the same time share one probe —
a shared result carries its age in a `cached_age_s` field.

Options:

| Flag | Default | Description |
|---|---|---|
| `--check-publishers` | off | Enable publisher checks |
| `--bootstrap-servers` | `localhost:9092` | Kafka bootstrap server(s) |
| `--publisher-warn-margin` | `1` | Warn within this many seconds of `max_latency` |
| `--publisher-max-latency` | `30` | Fallback budget when metadata has no `max_latency` |
| `--probe-timeout` | `30` | Cap on waiting for a data message per probe |
| `--metadata-ttl` | `600` | Seconds to cache surrogate resolution |
| `--probe-cache-ttl` | `60` | Seconds to serve cached probe results (0 disables) |
| `--surrogate PUB=CHANNEL` | auto | Explicit surrogate channel (repeatable) |
| `--ignore-publisher PUB` | none | Skip a publisher in `/publishers`, e.g. one registered ahead of going live (repeatable) |

```json
{
  "status": "ok",
  "message": "Publisher 'H1_lldd' live (latency 3.2s)",
  "publisher": "H1_lldd",
  "channel": "H1:AOS-ETMX_BAFFLEPD_1_DC_POWER",
  "replay_id": null,
  "max_latency_s": 12.0,
  "warn_latency_s": 11.0,
  "latency_s": 3.2,
  "transport_latency_s": 0.05,
  "block_time_gps": 1400000000.0
}
```

## NRPE Integration

Each Arrakis host runs the sidecar locally, and NRPE executes `check_http`
against the sidecar's port. The `-w` and `-c` flags set warning and
critical thresholds on the HTTP response time (in seconds). When the
Arrakis hosts are not reachable from the monitoring server, the commands
can live on an NRPE gateway host on the same network instead, targeting
each host by name:

```ini
# health check (warn at 2s, critical at 5s)
command[check_arrakis]=/usr/lib64/nagios/plugins/check_http -H <host> -p 8080 -u /health -w 2 -c 5

# scope map validation (info servers only)
command[check_arrakis_scope_map]=/usr/lib64/nagios/plugins/check_http -H <host> -p 8080 -u /scope-map -w 2 -c 5

# publisher liveliness/latency, every publisher at once (online servers
# with --check-publishers); probes wait for live data, so allow more
# response time than the other checks
command[check_arrakis_publishers]=/usr/lib64/nagios/plugins/check_http -H <host> -p 8080 -u /publishers -w 10 -c 30 -t 40

# per-publisher liveliness/latency, one command per publisher
command[check_arrakis_publisher_<publisher>]=/usr/lib64/nagios/plugins/check_http -H <host> -p 8080 -u /publishers/<publisher> -w 10 -c 30 -t 40
```

Publisher probes can wait up to `--probe-timeout` (30 s) for live data,
so the NRPE client needs a matching timeout in place of its 10 s
default (`check_nrpe -t 45`, or `vars.nrpe_timeout = 45` on the Icinga
service).

## Resources

- [Documentation](https://docs.ligo.org/ngdd/arrakis-monitor)
- [Source Code](https://git.ligo.org/ngdd/arrakis-monitor)
- [Issue Tracker](https://git.ligo.org/ngdd/arrakis-monitor/issues)
