Metadata-Version: 2.5
Name: arrakis-monitor
Version: 0.1.1
Summary: HTTP health sidecar for Arrakis Flight servers
Project-URL: Homepage, https://git.ligo.org/ngdd/arrakis-monitor
Project-URL: Documentation, https://docs.ligo.org/ngdd/arrakis-monitor
Project-URL: Repository, https://git.ligo.org/ngdd/arrakis-monitor.git
Project-URL: Issues, https://git.ligo.org/ngdd/arrakis-monitor/issues
Author-email: Olivia Godwin <olivia.godwin@ligo.org>
Maintainer-email: Olivia Godwin <olivia.godwin@ligo.org>
License-Expression: GPL-3.0-or-later
License-File: LICENSE
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: Scientific/Engineering :: Astronomy
Classifier: Topic :: Scientific/Engineering :: Physics
Classifier: Topic :: System :: Monitoring
Requires-Python: >=3.11
Requires-Dist: arrakis-admin>=0.2
Requires-Dist: arrakis>=0.15
Requires-Dist: confluent-kafka>=2.2
Requires-Dist: gpstime
Requires-Dist: starlette
Requires-Dist: uvicorn[standard]
Provides-Extra: dev
Requires-Dist: httpx; extra == 'dev'
Requires-Dist: markdown-callouts>=0.2; extra == 'dev'
Requires-Dist: markdown-exec>=0.5; extra == 'dev'
Requires-Dist: mkdocs-coverage>=0.2; extra == 'dev'
Requires-Dist: mkdocs-gen-files>=0.3; extra == 'dev'
Requires-Dist: mkdocs-literate-nav>=0.4; extra == 'dev'
Requires-Dist: mkdocs-material-igwn; extra == 'dev'
Requires-Dist: mkdocs-section-index>=0.3; extra == 'dev'
Requires-Dist: mkdocs>=1.3; extra == 'dev'
Requires-Dist: mkdocstrings[python]; extra == 'dev'
Requires-Dist: mypy; extra == 'dev'
Requires-Dist: mypy-extensions; extra == 'dev'
Requires-Dist: pip; extra == 'dev'
Requires-Dist: pytest; extra == 'dev'
Requires-Dist: pytest-asyncio; extra == 'dev'
Requires-Dist: pytest-cov; extra == 'dev'
Requires-Dist: pytest-freezer; extra == 'dev'
Requires-Dist: ruff; extra == 'dev'
Requires-Dist: toml>=0.10; extra == 'dev'
Provides-Extra: docs
Requires-Dist: markdown-callouts>=0.2; extra == 'docs'
Requires-Dist: markdown-exec>=0.5; extra == 'docs'
Requires-Dist: mkdocs-coverage>=0.2; extra == 'docs'
Requires-Dist: mkdocs-gen-files>=0.3; extra == 'docs'
Requires-Dist: mkdocs-literate-nav>=0.4; extra == 'docs'
Requires-Dist: mkdocs-material-igwn; extra == 'docs'
Requires-Dist: mkdocs-section-index>=0.3; extra == 'docs'
Requires-Dist: mkdocs>=1.3; extra == 'docs'
Requires-Dist: mkdocstrings[python]; extra == 'docs'
Requires-Dist: toml>=0.10; extra == 'docs'
Provides-Extra: lint
Requires-Dist: mypy; extra == 'lint'
Requires-Dist: mypy-extensions; extra == 'lint'
Requires-Dist: pip; extra == 'lint'
Requires-Dist: ruff; extra == 'lint'
Provides-Extra: test
Requires-Dist: httpx; extra == 'test'
Requires-Dist: pytest; extra == 'test'
Requires-Dist: pytest-asyncio; extra == 'test'
Requires-Dist: pytest-cov; extra == 'test'
Requires-Dist: pytest-freezer; extra == 'test'
Description-Content-Type: text/markdown

# arrakis-monitor

HTTP health sidecar for [Arrakis](https://git.ligo.org/ngdd/arrakis-server)
Flight servers.

## Overview

`arrakis-monitor` runs alongside each Arrakis server and exposes its health
status over HTTP, allowing standard monitoring tools (Nagios/Icinga via NRPE +
`check_http`) to check gRPC Flight servers without needing a gRPC-native plugin.

## Installation

```bash
pip install arrakis-monitor
```

## Usage

```bash
# basic health sidecar
arrakis-monitor --server grpc://localhost:31206

# with scope map domain validation
arrakis-monitor --server grpc://localhost:31206 --expect-domains H1,L1,V1

# with publisher liveliness/latency checks (online servers)
arrakis-monitor --server grpc://localhost:31206 --check-publishers

# custom bind address
arrakis-monitor --server grpc://localhost:31206 --host 127.0.0.1 --port 9090
```

## Endpoints

| Endpoint | Description |
|---|---|
| `GET /` | Show server configuration |
| `GET /health` | Run a health check (calls `server_info()`) |
| `GET /scope-map` | Run a scope map check (calls `scope_map()`) |
| `GET /publishers` | Probe every publisher in the scope map (`--check-publishers`) |
| `GET /publishers/{publisher}` | Probe a single publisher (`--check-publishers`) |

Health and scope-map endpoints return JSON with an HTTP status code:

- **200** — OK or WARNING
- **503** — CRITICAL (server unreachable or scope map empty)
- **500** — UNKNOWN (unexpected error)

### Response format

Both check endpoints include a `latency_s` field measuring the gRPC call
round-trip time in seconds:

```json
{
  "status": "ok",
  "message": "Server 0.12.0 (kafka)",
  "version": "0.12.0",
  "backend": "kafka",
  "domains": ["H1", "L1"],
  "capabilities": ["find", "count", "describe", "stream"],
  "latency_s": 0.042
}
```

## Publisher checks

On online (Kafka-backed) servers, `--check-publishers` enables
per-publisher liveliness and latency monitoring. For each publisher
tagged in the server's scope map, the sidecar consumes the publisher's
metadata topic from the local broker to select a surrogate channel,
then probes the data topic for that channel's partition, waiting for a
fresh message. The block-time latency of the freshest record batch is
compared against the `max_latency` advertised in the channel metadata:

- **OK** — fresh data arrived within the latency budget
- **WARNING** — latency within `--publisher-warn-margin` (default 1 s)
  of `max_latency`
- **CRITICAL** — latency at/past `max_latency`, or no message within
  `max_latency` plus one block stride (reported with the age of the
  last retained message)

`/publishers` probes every publisher concurrently and reports the
worst status; `/publishers/{publisher}` probes one, for fine-grained
per-publisher alerting. Surrogate resolution is cached per publisher
(`--metadata-ttl`, default 600 s), and check results are cached
briefly (`--probe-cache-ttl`, default 60 s) so aggregate and
per-publisher checks polling around the same time share one probe —
a shared result carries its age in a `cached_age_s` field.

Options:

| Flag | Default | Description |
|---|---|---|
| `--check-publishers` | off | Enable publisher checks |
| `--bootstrap-servers` | `localhost:9092` | Kafka bootstrap server(s) |
| `--publisher-warn-margin` | `1` | Warn within this many seconds of `max_latency` |
| `--publisher-max-latency` | `30` | Fallback budget when metadata has no `max_latency` |
| `--probe-timeout` | `30` | Cap on waiting for a data message per probe |
| `--metadata-ttl` | `600` | Seconds to cache surrogate resolution |
| `--probe-cache-ttl` | `60` | Seconds to serve cached probe results (0 disables) |
| `--surrogate PUB=CHANNEL` | auto | Explicit surrogate channel (repeatable) |

```json
{
  "status": "ok",
  "message": "Publisher 'H1_lldd' live (latency 3.2s)",
  "publisher": "H1_lldd",
  "channel": "H1:AOS-ETMX_BAFFLEPD_1_DC_POWER",
  "replay_id": null,
  "max_latency_s": 12.0,
  "warn_latency_s": 11.0,
  "latency_s": 3.2,
  "transport_latency_s": 0.05,
  "block_time_gps": 1400000000.0
}
```

## NRPE Integration

Each Arrakis host runs the sidecar locally, and NRPE executes `check_http`
against `localhost:8080`. The `-w` and `-c` flags set warning and critical
thresholds on the HTTP response time (in seconds):

```ini
# health check (warn at 2s, critical at 5s)
command[check_arrakis]=/usr/lib/nagios/plugins/check_http -H localhost -p 8080 -u /health -w 2 -c 5

# scope map validation (info servers only)
command[check_arrakis_scope_map]=/usr/lib/nagios/plugins/check_http -H localhost -p 8080 -u /scope-map -w 2 -c 5

# publisher liveliness/latency (online servers with --check-publishers);
# probes wait for live data, so allow more response time than the other checks
command[check_arrakis_publishers]=/usr/lib/nagios/plugins/check_http -H localhost -p 8080 -u /publishers -w 10 -c 30 -t 40
```

For per-publisher alerting, point `check_http` at
`/publishers/<publisher-id>` with one command per publisher.

See `examples/nrpe/` for per-host configuration examples for all sites.

## Resources

- [Documentation](https://docs.ligo.org/ngdd/arrakis-monitor)
- [Source Code](https://git.ligo.org/ngdd/arrakis-monitor)
- [Issue Tracker](https://git.ligo.org/ngdd/arrakis-monitor/issues)
