Metadata-Version: 2.5
Name: wdflow
Version: 1.3.0
Summary: WDF (Wavelet Detection Filter): real-time trigger generation for transient signals, with the downstream analysis that turns triggers into candidate events
Project-URL: Homepage, https://elenacuoco.github.io/wdflow/
Project-URL: Source, https://github.com/elenacuoco/wdflow
Project-URL: Documentation, https://wdflow.readthedocs.io
Project-URL: Issues, https://github.com/elenacuoco/wdflow/issues
Project-URL: Changelog, https://github.com/elenacuoco/wdflow/blob/master/CHANGELOG.md
Author-email: Elena Cuoco <elena.cuoco@unibo.it>
License: GPL-3.0-or-later
License-File: LICENSE
Keywords: gravitational-waves,time-series,transients,triggers,wavelet
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Astronomy
Classifier: Topic :: Scientific/Engineering :: Physics
Requires-Python: >=3.10
Requires-Dist: h5py
Requires-Dist: matplotlib>=3.7
Requires-Dist: numpy
Requires-Dist: pandas>=2.0
Requires-Dist: pyarrow
Requires-Dist: scikit-learn>=1.2
Requires-Dist: scipy
Provides-Extra: all
Requires-Dist: coloredlogs; extra == 'all'
Requires-Dist: gwpy>=3.0; extra == 'all'
Requires-Dist: ipykernel; extra == 'all'
Requires-Dist: jupyter; extra == 'all'
Requires-Dist: myst-nb>=1; extra == 'all'
Requires-Dist: nbclient; extra == 'all'
Requires-Dist: py4tsa>=3.2; extra == 'all'
Requires-Dist: pycbc; extra == 'all'
Requires-Dist: pytest>=7; extra == 'all'
Requires-Dist: sphinx-rtd-theme>=2; extra == 'all'
Requires-Dist: sphinx>=7; extra == 'all'
Requires-Dist: torch-geometric>=2.5; extra == 'all'
Requires-Dist: torch>=2.1; extra == 'all'
Provides-Extra: data
Requires-Dist: gwpy>=3.0; extra == 'data'
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == 'dev'
Provides-Extra: docs
Requires-Dist: myst-nb>=1; extra == 'docs'
Requires-Dist: sphinx-rtd-theme>=2; extra == 'docs'
Requires-Dist: sphinx>=7; extra == 'docs'
Provides-Extra: gnn
Requires-Dist: torch-geometric>=2.5; extra == 'gnn'
Requires-Dist: torch>=2.1; extra == 'gnn'
Provides-Extra: mock
Requires-Dist: pycbc; extra == 'mock'
Provides-Extra: pipeline
Requires-Dist: coloredlogs; extra == 'pipeline'
Requires-Dist: py4tsa>=3.2; extra == 'pipeline'
Provides-Extra: tutorials
Requires-Dist: ipykernel; extra == 'tutorials'
Requires-Dist: jupyter; extra == 'tutorials'
Requires-Dist: nbclient; extra == 'tutorials'
Description-Content-Type: text/markdown

# wdflow

[![Documentation](https://readthedocs.org/projects/wdflow/badge/?version=latest)](https://wdflow.readthedocs.io/en/latest/)
[![DOI](https://img.shields.io/badge/DOI-10.5281%2Fzenodo.22025594-blue.svg)](https://doi.org/10.5281/zenodo.22025594)

A maintained, standalone package for the WDF (Wavelet Detection Filter) trigger-generation
pipeline for transient time-series signals, built on the C++ core
[p4TSA](https://github.com/elenacuoco/p4TSA) (exposed to Python as `py4tsa`), plus the downstream
trigger analysis that turns raw per-window triggers into candidate events.

**It runs in real time**, at a latency that is fixed and known before the filter
runs, and its per-window arithmetic -- an autoregressive filter of known order,
an orthonormal transform of a power-of-two window, a threshold and a sum of
squares, with no iteration to convergence, no dynamic memory and no
data-dependent branching -- is simple enough to put on an FPGA. No template and
no signal model: every trigger carries the coefficients that produced it, so
what a transient is can be decided afterwards.

## Layout

- `wdf.config`, `wdf.processes`, `wdf.observers`, `wdf.structures` -- trigger generation. Needs
  the compiled `py4tsa`/p4TSA core (`pip install -e ".[pipeline]"`).
- `wdf.analysis` -- clustering, multi-detector coincidence (classical + GNN), background/
  false-alarm-probability, ROC analysis, sky localisation, and the submission writer that
  puts the surviving candidates in a challenge's own columns. Operates on plain pandas
  DataFrames / saved trigger files, no `py4tsa` dependency, so it works standalone
  (`pip install -e .`, no extras needed).
- `wdf.mock` -- the simulated two-detector data set: coloured Gaussian noise, compact-binary
  injections projected through the antenna responses, single-detector glitch morphologies, and
  waveforms read from a catalogue when a class has no closed form. Paired foreground and
  background frames with a truth table (`pip install -e ".[mock]"`).

## Relationship to the legacy `wdf` package

`wdflow` carries forward the parts of the retired `wdf` package
that are actually used by the real trigger-search pipeline (`wdfUnitDSWorker` and everything it
calls) -- same file/class/module names and shapes, no gratuitous renaming -- plus the
`wdf.analysis` layer (formerly a separate `wdfLib` package), merged in here as one namespace to
install and import from.

Changes from the legacy `wdf` package:

- **AR-whitening lookahead is fixed-size, independent of `par.len`.** `DoubleWhitening`'s backward
  pass needs a lookahead buffer of real future data to settle before producing a good estimate for
  the current chunk. This lookahead (`WhiteningExtraSize`, default 20 resampled-rate seconds) is
  now a fixed size, decoupled from `par.len` (the streaming chunk size, an I/O batching/throughput
  knob). Set `parameters.WhiteningExtraSize = 0` to reproduce the legacy behavior.

- **The detection loop stops `par.len` seconds before the requested segment end** (unchanged from
  the legacy package): it checks the previous read's start before issuing the next one, so its
  last read can extend up to `par.len` past the checked bound, and `FrameIChannel` does not always
  raise cleanly when asked to read past the last available frame data. This margin keeps the last
  read within legitimately available data. A larger `par.len` therefore analyzes up to `par.len`
  fewer seconds at a segment's tail.

- **Per-trigger statistics remain somewhat sensitive to `par.len`** even with the fixed whitening
  lookahead above: `WaveletThreshold`'s `dohonojohnston` mode recomputes its detection threshold
  fresh per window from that window's own coefficient median, so triggers whose statistic sits
  close to that threshold can flip in or out of detection depending on floating-point roundoff
  accumulated differently across different chunk sizes. This is an inherent sensitivity of any
  hard/soft-threshold statistic to input near its decision boundary, not a data-corruption bug --
  triggers well away from threshold are unaffected.

- **`EnWDF` is the detection statistic, everywhere.** It is the norm of a window's surviving
  wavelet coefficients on the noise scale, `||c||/sigma`, computed in p4TSA and carried on every
  trigger. The candidate bases are orthonormal, so by Parseval this is also `||x_hat||/sigma`, the
  matched-filter signal-to-noise ratio of the reconstructed transient. Clustering, coincidence,
  GNN scoring and the false-alarm-probability estimate all rank on it, per window, per event and
  per network -- foreground and background have to be ranked on the same quantity for a
  probability to mean anything.

  `snrPeak` and `snrMean` sit alongside it as amplitudes on the same scale, `max|x|/sigma` and the
  r.m.s. of `x` over its own support divided by `sigma`. `EnWDF` grows as the square root of the
  number of samples a transient occupies, as an energy statistic must; the other two do not.

- **Every frequency a trigger reports is a moment over the wavelet tiles**, not a periodogram.
  `freqMean` is the energy-weighted band, taken in the log-frequency the dyadic tiling is uniform
  in; `freqMin`/`freqMax` are the support the surviving tiles cover, which is what the band
  overlap tests read; `freqQ05`/`freqQ95` are the band the energy occupies, which follows the
  signal rather than its faintest coefficient. There is no single peak frequency: a band an octave
  wide is what the transform can resolve, and naming one frequency inside it would state more than
  was measured.

- **Events are clustered on the wavegram; the energy that measures is not the statistic that
  selects.** `wdf.analysis.clustering.wavegram_events` percolates over the time-frequency tiles of
  the surviving coefficients to decide which windows belong to the same transient. The event's
  energy is then `||x_hat||/sigma` over the stitched reconstruction of those windows, which counts
  each sample once and is what the event is worth. What ranks it for detection is its loudest
  block: a hard threshold admits every tile at a floor of `2 ln N` in normalised energy, so a sum
  over tiles accumulates that floor in the noise as well as in the signal, while a maximum over
  blocks the search has already scored has a background that is a subset of the ungrouped one ---
  the grouping reduces the trials factor without being able to lose a candidate. `TriggerClusterer`
  (DBSCAN or a greedy merge on the per-window scalar summaries) is kept as a cross-check.

- **A pair is admitted on the events' extents and ranked on their instants.** The two events'
  stretches of time have to meet once one may shift by the light travel time plus their own
  spreads: a transient longer than one window is assembled as several events and two detectors
  need not keep the same one, so gating on an instant would take evidence away from a candidate
  the detectors did assemble. The difference of the two events' instants --- the centre of the
  tile carrying each one's largest coefficient, the order
  `wdf.analysis.robust_events.INSTANT_COLUMNS` names --- then ranks the survivors. It is a
  property of one event, so a time slide carries it with the event and a background of millions
  of accidental pairs costs nothing per pair, which is what such a background requires. What
  that difference resolves is bounded by the tiling: a tile's length is one over the upper edge
  of its own band, so where an event's loudest coefficient sits low the tile is longer than the
  light travel time itself, and `tSpread` is what declares that. It is a bound on what a pair's
  timing is worth, not a bound on the search: nothing before it reads a time at all.

- **Trigger output is Parquet, not CSV** (`wdf.observers.SingleEventPrintFileObserver`), written
  incrementally in row-group batches (`flush_every`, default 500 triggers) and finalized by a
  `close()` call at the end of `segmentProcess`. `wdf.analysis.io`'s loaders accept both
  `*.parquet` (default) and `*.csv` (for older runs), dispatching on file extension.

- **The downstream analysis layer (formerly the separate `wdfLib` package) is merged in** as the
  `wdf.analysis` subpackage: clustering, multi-detector coincidence (classical + GNN),
  background/false-alarm-probability, and ROC analysis. It has no `py4tsa` dependency and operates
  on plain pandas DataFrames / saved trigger files, so it works standalone.

Left behind deliberately (not used by `wdfUnitDSWorker`'s pipeline, not ported or audited):
`AdaptiveWhitening`, `createsegmentsMinMax`, `CreateSegments`, `DownSamplingLF`,
`DownSampling`, `StateVectorSegments`, `wdf_reconstruct`, `wdfUnitBPDSWorker`, `wdfUnitWorker`
(the last two share the pre-fix `ExtraSize=0` whitening issue -- worth the same fix if/when
ported), `structures.ClusteredEvent` (an empty data holder), `structures.segment`, `utility.*`.

## Tutorials

`tutorials/` holds four runnable notebooks. Everything in them is built in the notebook itself --
no data set to download, no frame file to point at:

1. `01_the_statistic_and_the_parameters` -- one window of whitened data, the transform,
   thresholding, basis competition, and what the estimated parameters mean. Needs `py4tsa`.
2. `02_the_wavegram_and_long_signals` -- reading parameters off the time-frequency tiles, and
   recovering a signal that spans several analysis windows. Needs `py4tsa`.
3. `03_coincidence_and_significance` -- coincidence, time-slide background, false-alarm
   probability and ROC, using only `wdf.analysis`. **No `py4tsa` required.**
4. `04_reconstruction_and_phase` -- the event's own coefficients inverted and stitched across
   windows, compared against the waveform that was injected, and the phase read sample by
   sample. Needs `py4tsa`.

## Install

### Requirements

Python 3.10 or newer. `wdflow` is not on an index: it installs from a checkout
of this repository, and everything it depends on comes from PyPI.

```bash
git clone https://github.com/elenacuoco/wdflow
cd wdflow
pip install -e ".[all]"
```

**Always installed** — the analysis layer runs on these alone:

| Package | Used for |
|---------|----------|
| `numpy`, `scipy` | arrays, signal processing |
| `pandas >= 2.0` | trigger and event tables |
| `pyarrow` | reading and writing the Parquet trigger files |
| `h5py` | the AR and lattice-filter coefficient files |
| `scikit-learn >= 1.2` | DBSCAN, in `TriggerClusterer` |
| `matplotlib >= 3.7` | the report figures |

**Optional groups**, each installed with `pip install -e ".[name]"`:

| Group | Packages | Needed for |
|-------|----------|------------|
| `gnn` | `torch >= 2.1`, `torch_geometric >= 2.5` | `wdf.analysis.gnn` — the learned cross-detector coincidence |
| `data` | `gwpy >= 3.0` | fetching public strain, e.g. from GWOSC |
| `pipeline` | `py4tsa >= 3.2`, `coloredlogs` | `wdf.processes`, `wdf.observers` — trigger generation, and any wavelet transform |
| `mock` | `pycbc` | `wdf.mock` — generating the simulated data sets: CBC injections, glitch morphologies, catalogue waveforms |
| `tutorials` | `jupyter`, `nbclient`, `ipykernel` | running `tutorials/` |
| `docs` | `sphinx >= 7`, `sphinx-rtd-theme >= 2`, `myst-nb >= 1` | building the documentation |
| `dev` | `pytest >= 7` | the test suite |
| `all` | all of the above | |

**Not in any group:**

| Requirement | Needed for | Where it comes from |
|-------------|------------|---------------------|
| a GWF backend for gwpy | reading frame files through gwpy | `pip install lalsuite`, or conda's `python-ldas-tools-framecpp` |

### The compiled core

Trigger generation needs [p4TSA](https://github.com/elenacuoco/p4TSA), imported
as `py4tsa`, which the `pipeline` group installs from PyPI. The module used to
be called `pytsa`, a name that on PyPI belongs to an unrelated project, so
change any `import pytsa` to `import py4tsa`.

The wheels are for Linux x86_64, CPython 3.10 to 3.13, and carry GSL, FFTW3
and FrameL inside them. Elsewhere pip builds p4TSA from source, which needs
GSL, FFTW3, FrameL, the Boost.uBLAS headers and the Cereal headers, all on
conda-forge.

If `import py4tsa` behaves oddly, check what you actually have:

```bash
pip show py4tsa
python -c "import py4tsa; print(py4tsa.__file__)"   # must be a compiled .so, not a .py
```

### The legacy `wdf` package must not be installed alongside

`wdflow`'s importable package is `wdf`, the same top-level name the retired
`wdf` package used. Whichever of the two sits directly in `site-packages` wins,
and it shadows even an editable install of the other, silently:

```bash
python -c "import wdf; print(wdf.__file__)"   # must point at your wdflow checkout
pip uninstall wdf                             # if it points into site-packages
```

## What comes next

Parameter estimation by normalising flow, on the coefficient matrix an event
already carries. The coefficients a trigger keeps are a fixed-length description
of the transient on the time-frequency plane, already on the noise scale, so a
flow can be conditioned on them directly without a spectrogram in between. That
is where the question of what a transient *is* belongs -- after the selection,
never inside it -- and it is the reason the search stores the coefficients
rather than a summary of them.

## Contributing

Changes reach `master` through pull requests only, and a pull request merges
only once CI is green. See [`CONTRIBUTING.md`](https://github.com/elenacuoco/wdflow/blob/master/CONTRIBUTING.md).

## How to cite

**Use of this code in published work requires citation of the following.**

*The Wavelet Detection Filter:*

- E. Cuoco, *The Wavelet Detection Filter: A Real Time Unmodeled Pipeline for
  Gravitational Wave Transients, Ranking Coincidences with a Graph Neural
  Network*, arXiv:2609.12797 (2026).
  [10.48550/arXiv.2609.12797](https://doi.org/10.48550/arXiv.2609.12797)

*WDFX:*

- E. Cuoco, M. Razzano, A. Utina, *Wavelet-based classification of transient
  signals for gravitational wave detectors*, 26th European Signal Processing
  Conference (EUSIPCO), 2648–2652 (2018).
  [10.23919/EUSIPCO.2018.8553393](https://doi.org/10.23919/EUSIPCO.2018.8553393)

*Time-domain whitening, which the conditioning stage implements:*

- E. Cuoco *et al.*, *On-line power spectra identification and whitening for the
  noise in interferometric gravitational wave detectors*, Class. Quantum Grav.
  **18**, 1727 (2001).
  [10.1088/0264-9381/18/9/309](https://doi.org/10.1088/0264-9381/18/9/309)
- E. Cuoco *et al.*, *Noise parametric identification and
  whitening for LIGO 40-m interferometer data*, Phys. Rev. D **64**, 122002
  (2001).
  [10.1103/PhysRevD.64.122002](https://doi.org/10.1103/PhysRevD.64.122002)

`CITATION.cff` in this repository carries the same list in machine-readable
form; GitHub's *Cite this repository* button reads it.

## Use of generative AI

The Wavelet Detection Filter, the `p4TSA` core it runs on and the design of this
pipeline are the author's own work. Claude (Opus 5, Anthropic), used through
Claude Code, wrote parts of the implementation, its tests and its documentation,
and produced the logo and stylesheet of the project page. Every such
contribution was reviewed and is covered by the test suite that runs in CI: the
golden-output fixture pins trigger generation end to end, so a generated change
that moves the numerics fails the build rather than passing silently.
Responsibility for the method and for everything published here rests with the
author.

## Status

`tests/` includes a golden-output regression fixture pinning trigger generation end to end on a
small synthetic frame (the legacy `wdf` package has none), plus the `wdf.analysis` test suite
(synthetic trigger data, no WDF run required):

```bash
pytest tests                    # needs py4tsa for the golden fixture
pytest tests/test_clustering.py tests/test_coincidence.py tests/test_significance.py   # no py4tsa
```

`tests/test_gnn.py` needs the `gnn` extra and `tests/test_mock_dataset.py` needs `pycbc`; both are
skipped by deselecting them if those are not installed.
