Metadata-Version: 2.5
Name: dq-genesis
Version: 0.2.0
Summary: Pipeline construction and execution for the SGN DQ module
Project-URL: Homepage, https://git.ligo.org/detchar/sgn-dq/dq-genesis
Project-URL: Documentation, https://detchar.docs.ligo.org/sgn-dq/dq-genesis
Project-URL: Repository, https://git.ligo.org/detchar/sgn-dq/dq-genesis.git
Project-URL: Issues, https://git.ligo.org/detchar/sgn-dq/dq-genesis/issues
Author-email: Derek Davis <derek.davis@ligo.org>, Olivia Godwin <olivia.godwin@ligo.org>, Zach Yarbrough <zach.yarbrough@ligo.org>
Maintainer-email: Derek Davis <derek.davis@ligo.org>, Olivia Godwin <olivia.godwin@ligo.org>, Zach Yarbrough <zach.yarbrough@ligo.org>
License-Expression: GPL-3.0-or-later
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Astronomy
Classifier: Topic :: Scientific/Engineering :: Physics
Requires-Python: >=3.11
Requires-Dist: arrakis>=0.21.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: scipy>=1.10
Requires-Dist: sgn-arrakis>=0.11.0
Requires-Dist: sgn-dq>=0.3.1
Requires-Dist: sgn-gwframe
Requires-Dist: sgn-ligo
Requires-Dist: sgn-ts>=0.14.4
Requires-Dist: sgn>=0.12.2
Requires-Dist: sgnmon>=0.2.1
Provides-Extra: dev
Requires-Dist: markdown-callouts>=0.2; extra == 'dev'
Requires-Dist: markdown-exec>=0.5; extra == 'dev'
Requires-Dist: mkdocs-coverage>=0.2; extra == 'dev'
Requires-Dist: mkdocs-gen-files>=0.3; extra == 'dev'
Requires-Dist: mkdocs-literate-nav>=0.4; extra == 'dev'
Requires-Dist: mkdocs-material-igwn; extra == 'dev'
Requires-Dist: mkdocs-section-index>=0.3; extra == 'dev'
Requires-Dist: mkdocs>=1.3; extra == 'dev'
Requires-Dist: mkdocstrings[python]; extra == 'dev'
Requires-Dist: mypy; extra == 'dev'
Requires-Dist: mypy-extensions; extra == 'dev'
Requires-Dist: pip; extra == 'dev'
Requires-Dist: pytest; extra == 'dev'
Requires-Dist: pytest-cov; extra == 'dev'
Requires-Dist: pytest-freezer; extra == 'dev'
Requires-Dist: ruff; extra == 'dev'
Requires-Dist: toml>=0.10; extra == 'dev'
Provides-Extra: docs
Requires-Dist: markdown-callouts>=0.2; extra == 'docs'
Requires-Dist: markdown-exec>=0.5; extra == 'docs'
Requires-Dist: mkdocs-coverage>=0.2; extra == 'docs'
Requires-Dist: mkdocs-gen-files>=0.3; extra == 'docs'
Requires-Dist: mkdocs-literate-nav>=0.4; extra == 'docs'
Requires-Dist: mkdocs-material-igwn; extra == 'docs'
Requires-Dist: mkdocs-section-index>=0.3; extra == 'docs'
Requires-Dist: mkdocs>=1.3; extra == 'docs'
Requires-Dist: mkdocstrings[python]; extra == 'docs'
Requires-Dist: toml>=0.10; extra == 'docs'
Provides-Extra: lint
Requires-Dist: mypy; extra == 'lint'
Requires-Dist: mypy-extensions; extra == 'lint'
Requires-Dist: pip; extra == 'lint'
Requires-Dist: ruff; extra == 'lint'
Provides-Extra: test
Requires-Dist: pytest; extra == 'test'
Requires-Dist: pytest-cov; extra == 'test'
Requires-Dist: pytest-freezer; extra == 'test'
Description-Content-Type: text/markdown

<h1 align="center">dq-genesis</h1>

<p align="center">
  <a href="https://git.ligo.org/detchar/sgn-dq/dq-genesis/-/pipelines/latest">
    <img alt="ci" src="https://git.ligo.org/detchar/sgn-dq/dq-genesis/badges/main/pipeline.svg" />
  </a>
  <a href="https://git.ligo.org/detchar/sgn-dq/dq-genesis/-/pipelines/latest">
    <img alt="coverage" src="https://git.ligo.org/detchar/sgn-dq/dq-genesis/badges/main/coverage.svg" />
  </a>
  <a href="https://detchar.docs.ligo.org/sgn-dq/dq-genesis">
    <img alt="documentation" src="https://img.shields.io/badge/docs-mkdocs%20material-blue.svg?style=flat" />
  </a>
  <a href="https://pypi.org/project/dq-genesis/">
    <img alt="pypi version" src="https://img.shields.io/pypi/v/dq-genesis.svg" />
  </a>
</p>

Pipeline construction and execution for the creation of the data quality vector.

`dq-genesis` parses a YAML DQ flag config, assembles an `sgn.Pipeline` from the
evaluators and transforms provided by `sgn-dq`, and runs it against frame data
(offline frame cache), shared-memory frames (online), or an Arrakis stream. The
output is a uint32 `DQG-DQ_VECTOR` channel written to a GWF frame stream.

## Install

```bash
pip install dq-genesis                   # from PyPI
pip install -e /path/to/dq-genesis       # editable install from a source checkout
```

`dq-genesis` depends on `sgn-dq` (evaluators, `Not`, `Parity`), `sgn-ts`,
`sgn-ligo`, and `sgn`.

## Quick start

```bash
# Validate a config
dq-genesis --flags configs/dqmodule_h1.yaml --dry-run

# Offline run against a frame cache
dq-genesis --flags configs/dqmodule_h1.yaml \
  --gps-start 1234567890 --gps-end 1234568000 \
  --frame-cache h1.cache --output-dir /tmp/out

# Bounded replay from Arrakis (omit --gps-start/--gps-end for a live stream)
dq-genesis --flags configs/dqmodule_h1.yaml \
  --arrakis --gps-start 1234567890 --gps-end 1234568000 \
  --output-dir /tmp/out
```

See `src/dqgenesis/cli.py --help` for the full argument surface.

## Output destinations

The assembled DQ vector can be fanned out to one or more sinks. Select any
combination of the `--publish-*` flags; if none is given it defaults to disk,
preserving the original frame-writing behaviour.

- `--publish-disk` — write the DQ vector to GWF frame files. Control the
  destination with `--output-dir` (and `--write-path`, `--frame-duration`).
- `--publish-arrakis` — publish the DQ vector channel back to an Arrakis
  server. Requires `--publisher-id`; pass `--replay-id` to publish into a
  replay namespace.

```bash
# Save to GWF frames (default)
dq-genesis --flags configs/dqmodule_h1.yaml \
  --gps-start 1234567890 --gps-end 1234568000 \
  --frame-cache h1.cache --publish-disk --output-dir /tmp/out

# Publish back into Arrakis
dq-genesis --flags configs/dqmodule_h1.yaml \
  --arrakis --gps-start 1234567890 --gps-end 1234568000 \
  --publish-arrakis --publisher-id <publisher_id>

# Both at once — the DQ vector is fanned out to disk and Arrakis
dq-genesis --flags configs/dqmodule_h1.yaml \
  --arrakis --gps-start 1234567890 --gps-end 1234568000 \
  --publish-disk --output-dir /tmp/out \
  --publish-arrakis --publisher-id <publisher_id>
```

The output channel is named `<IFO>:DQG-DQ_VECTOR` by default; override it with
`--output-channel`.

## Monitoring

Pass `--monitor` to serve live monitoring for the running pipeline via
[sgnmon](https://greg.docs.ligo.org/sgnmon):

```bash
dq-genesis --flags configs/dqmodule_h1.yaml --arrakis --monitor
```

This starts a background web server (default port 9090; `--monitor-port`
changes it and implies `--monitor`, with `0` picking a free port) exposing:

- `/` -- a live dashboard drawing the pipeline graph with per-element rates,
  latencies, and gap fractions
- `/metrics` -- Prometheus metrics for scraping, including the DQ vector
  itself: `dqgenesis_flag_active` and `dqgenesis_flag_active_fraction` (each
  flag's state in the latest sample and the fraction of the latest frame it
  was set, labelled `bit` and `flag`), `dqgenesis_flag_transitions_total`
  (how often each flag has flipped), `dqgenesis_dq_vector` (the latest value)
  and `dqgenesis_dq_vector_gps_seconds`, `dqgenesis_dq_vector_samples_total`,
  and `dqgenesis_parity_violations_total`. All are labelled `ifo`.
- `/health` -- a JSON health report (HTTP 503 when unhealthy), usable
  directly by container orchestration or `sgnmon check`; the DQ vector's
  own checks are also served one at a time at `/health/dq_vector_fresh` and
  `/health/dq_parity`, for monitors that only see status codes
- `/readyz` and `/healthz` -- readiness (the pipeline is running and every
  check passes) and liveness (the run loop strided recently) from the
  pipeline's own lifecycle
- `/status` -- a JSON snapshot of all probes

The data-freshness check on every pad tolerates three missed
`--frame-duration`s (30 s at least). `dq_vector_fresh` fails when no DQ
vector sample has been produced for five frame durations (5 minutes at
least), measured in wall-clock time from startup or the last sample: inputs
can keep flowing, so every pad looks healthy, while the bit vector stalls
waiting on one evaluator. `dq_parity` fails once any output sample has failed
odd parity (it always passes when the config has no parity bit).

Under `--monitor` the DQ vector is also fed to a small observer sink,
`dq_monitor`, next to the frame and Arrakis sinks; without it the pipeline
graph is exactly as before.

The lifecycle also reaches systemd: under a `Type=notify` unit (or a Podman
quadlet with `Notify=true`) the process sends `READY=1`, `WATCHDOG=1` as it
progresses and `STOPPING=1` on exit, so `WatchdogSec=` restarts a hung
process; the unit's timeout must exceed the liveness threshold above.
