Metadata-Version: 2.4
Name: arrakis-lldd-connector
Version: 0.7.1
Summary: A service built on SGN connecting data between LLDD, Arrakis and frame files
Project-URL: Homepage, https://git.ligo.org/ngdd/arrakis-lldd-connector
Project-URL: Issue Tracker, https://git.ligo.org/ngdd/arrakis-lldd-connector/issues
Project-URL: Source Code, https://git.ligo.org/ngdd/arrakis-lldd-connector.git
Author-email: Olivia Godwin <olivia.godwin@ligo.org>
Maintainer-email: Olivia Godwin <olivia.godwin@ligo.org>
License-Expression: GPL-3.0-or-later
License-File: LICENSE
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)
Classifier: Natural Language :: English
Classifier: Operating System :: POSIX
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Scientific/Engineering :: Astronomy
Classifier: Topic :: Scientific/Engineering :: Physics
Requires-Python: >=3.11
Requires-Dist: arrakis>=0.17
Requires-Dist: gpstime
Requires-Dist: igwn-lldd-common>=0.4.1
Requires-Dist: sgn-arrakis>=0.8
Requires-Dist: sgn-gwframe>=0.8
Requires-Dist: sgn-ts>=0.14
Requires-Dist: sgn>=0.12
Requires-Dist: sgnmon>=0.1
Provides-Extra: dev
Requires-Dist: markdown-callouts>=0.2; extra == 'dev'
Requires-Dist: markdown-exec>=0.5; extra == 'dev'
Requires-Dist: mkdocs-coverage>=0.2; extra == 'dev'
Requires-Dist: mkdocs-gen-files>=0.3; extra == 'dev'
Requires-Dist: mkdocs-literate-nav>=0.4; extra == 'dev'
Requires-Dist: mkdocs-material-igwn; extra == 'dev'
Requires-Dist: mkdocs-section-index>=0.3; extra == 'dev'
Requires-Dist: mkdocs>=1.3; extra == 'dev'
Requires-Dist: mkdocstrings[python]; extra == 'dev'
Requires-Dist: mypy; extra == 'dev'
Requires-Dist: mypy-extensions; extra == 'dev'
Requires-Dist: pip; extra == 'dev'
Requires-Dist: pytest; extra == 'dev'
Requires-Dist: pytest-arrakis>=0.4.1; extra == 'dev'
Requires-Dist: pytest-cov; extra == 'dev'
Requires-Dist: ruff; extra == 'dev'
Requires-Dist: toml>=0.10; extra == 'dev'
Provides-Extra: docs
Requires-Dist: markdown-callouts>=0.2; extra == 'docs'
Requires-Dist: markdown-exec>=0.5; extra == 'docs'
Requires-Dist: mkdocs-coverage>=0.2; extra == 'docs'
Requires-Dist: mkdocs-gen-files>=0.3; extra == 'docs'
Requires-Dist: mkdocs-literate-nav>=0.4; extra == 'docs'
Requires-Dist: mkdocs-material-igwn; extra == 'docs'
Requires-Dist: mkdocs-section-index>=0.3; extra == 'docs'
Requires-Dist: mkdocs>=1.3; extra == 'docs'
Requires-Dist: mkdocstrings[python]; extra == 'docs'
Requires-Dist: toml>=0.10; extra == 'docs'
Provides-Extra: lint
Requires-Dist: mypy; extra == 'lint'
Requires-Dist: mypy-extensions; extra == 'lint'
Requires-Dist: pip; extra == 'lint'
Requires-Dist: ruff; extra == 'lint'
Provides-Extra: test
Requires-Dist: pytest; extra == 'test'
Requires-Dist: pytest-arrakis>=0.4.1; extra == 'test'
Requires-Dist: pytest-cov; extra == 'test'
Description-Content-Type: text/markdown

<h1 align="center">arrakis-lldd-connector</h1>

<p align="center">A service built on <a href="https://git.ligo.org/greg/sgn">SGN</a>, connecting data between LLDD, Arrakis, and frame files.</p>

<p align="center">
  <a href="https://git.ligo.org/ngdd/arrakis-lldd-connector/-/pipelines/latest">
    <img alt="ci" src="https://git.ligo.org/ngdd/arrakis-lldd-connector/badges/main/pipeline.svg" />
  </a>
  <a href="https://git.ligo.org/ngdd/arrakis-lldd-connector/-/pipelines/latest">
    <img alt="coverage" src="https://git.ligo.org/ngdd/arrakis-lldd-connector/badges/main/coverage.svg" />
  </a>
  <a href="https://pypi.org/project/arrakis-lldd-connector/">
    <img alt="pypi version" src="https://img.shields.io/pypi/v/arrakis-lldd-connector.svg" />
  </a>
</p>

---

Stream gravitational-wave detector timeseries between the low-latency data
distribution system (LLDD), the [Arrakis](https://docs.ligo.org/ngdd/arrakis-python)
timeseries service, and GWF frame files. Pick a `--source` and a `--sink` and
the connector builds and runs the [SGN](https://git.ligo.org/greg/sgn)
pipeline between them.

## Resources

* [Source Code](https://git.ligo.org/ngdd/arrakis-lldd-connector)
* [Issue Tracker](https://git.ligo.org/ngdd/arrakis-lldd-connector/-/issues)

## Installation

```
pip install arrakis-lldd-connector
```

Or from source:

```
git clone git@git.ligo.org:ngdd/arrakis-lldd-connector.git
cd arrakis-lldd-connector
pip install .
```

## Features

* Sources: LLDD (Kafka frame streams), Arrakis, GWF frame file directories,
  and synthetic test signals
* Sinks: Arrakis, LLDD, GWF frame files, and a null sink for testing
* Any source can be paired with any sink
* Live streams are resilient to interruptions: sources reconnect with capped
  exponential backoff and bridge outages with gap buffers
* Bounded (`--start`/`--end`) or continuous live operation
* Arrakis replay namespace support for both streaming and publishing
  (`--replay-id`)
* Frame writing options: file retention (`--max-files`, `--retention-time`),
  and skipping files that contain no real data (`--skip-all-gap`)
* Live monitoring of the running pipeline (`--monitor`): an HTML dashboard,
  Prometheus metrics, and a health endpoint

## Quickstart

The CLI shows contextual options: pass `--source` and/or `--sink` with
`--help` to see the options relevant to that combination, e.g.

``` bash
arrakis-lldd-connector --source frames --sink arrakis --help
```

### Publish live DAQ data from LLDD to Arrakis

``` bash
arrakis-lldd-connector --source lldd --sink arrakis \
    --ifo L1 --topic Live_LLO_Data \
    --bootstrap-servers kafka1:9092,kafka2:9092 \
    --arrakis-url grpc://arrakis-online1:31206 \
    --publisher-id L1-lldd
```

The channels to publish are discovered from the publisher's registration on
the Arrakis server.

### Write an Arrakis stream to frame files

The Arrakis source reads the server URL from the `ARRAKIS_SERVER`
environment variable:

``` bash
export ARRAKIS_SERVER=grpc://arrakis-online1:31206
arrakis-lldd-connector --source arrakis --sink frames \
    --channels L1:GDS-CALIB_STRAIN L1:GDS-CALIB_STATE_VECTOR \
    --frame-duration 64 --output-dir /data/frames \
    --skip-all-gap
```

Without `--start` and `--end` this streams live data continuously; press
`Ctrl+C` to stop.

### Publish frame files to LLDD

Watch a directory for new frame files and publish them to Kafka:

``` bash
arrakis-lldd-connector --source frames --sink lldd \
    --channels H1:GDS-CALIB_STRAIN \
    --watch-dir /data/live/H1 \
    --ifo H1 --topic Live_LHO_Data \
    --bootstrap-servers kafka1:9092
```

### Test a pipeline with synthetic data

``` bash
arrakis-lldd-connector --source fake --sink null \
    --channels H1:TEST --rate 2048 --signal-type sin \
    --start 0 --end 10
```

## Monitoring

Serve live monitoring for the running pipeline with
[sgnmon](https://greg.docs.ligo.org/sgnmon):

``` bash
arrakis-lldd-connector --source arrakis --sink frames \
    --channels L1:GDS-CALIB_STRAIN \
    --frame-duration 64 --output-dir /data/frames \
    --monitor
```

This starts a background web server (default port 9090; `--monitor-port`
changes it and implies `--monitor`, with `0` picking a free port) exposing:

* `/` — a live dashboard drawing the pipeline graph with per-channel rates,
  latencies, and gap fractions
* `/metrics` — Prometheus metrics for scraping
* `/health` — a JSON health report (HTTP 503 when unhealthy), usable directly
  by container orchestration or `sgnmon check`
* `/status` — a JSON snapshot of all probes

Rather than observing every pad, the connector taps each link once at the
receiving element, plus post-alignment consumption (`adp`) to distinguish
"data arrived" from "data processed", and the source's output pads so input
rate and latency are visible per channel at the origin. The data-freshness
health check tolerates three missed output strides (`--frame-duration` for
the frames sink, `--delta-t` for the LLDD sink) before failing, so slow
output cadences do not false-alarm.

## Usage with Docker

The container image is hosted on `containers.ligo.org`. Pull the latest version with:

```bash
podman pull docker://containers.ligo.org/ngdd/arrakis-lldd-connector:latest
```

Run with podman (or docker):

```bash
podman run --rm --net=host \
    docker://containers.ligo.org/ngdd/arrakis-lldd-connector:latest \
    --source lldd --sink arrakis \
    --ifo L1 \
    --arrakis-url grpc://arrakis-online1:31206 \
    --publisher-id L1-lldd \
    --bootstrap-servers kafka1:9092,kafka2:9092 \
    --topic Live_LLO_Data
```
