Metadata-Version: 2.4
Name: microstructure-tpw
Version: 0.4.0
Summary: Ocean microstructure turbulence processing: TKE dissipation and chi from Rockland Scientific instruments
Author-email: Pat Welch <pat@mousebrains.com>
License-Expression: GPL-3.0-or-later
Project-URL: Homepage, https://github.com/mousebrains/turbulence
Project-URL: Repository, https://github.com/mousebrains/turbulence
Project-URL: Documentation, https://github.com/mousebrains/turbulence/tree/main/docs
Project-URL: Changelog, https://github.com/mousebrains/turbulence/blob/main/CHANGELOG.md
Project-URL: Issues, https://github.com/mousebrains/turbulence/issues
Keywords: oceanography,turbulence,microprofiler,rockland
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: Topic :: Scientific/Engineering :: Physics
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Requires-Python: >=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy<3,>=2.0
Requires-Dist: netCDF4<2,>=1.6
Requires-Dist: scipy<2,>=1.12
Requires-Dist: xarray>=2024.1
Requires-Dist: gsw<4,>=3.6
Requires-Dist: ruamel.yaml>=0.18
Requires-Dist: matplotlib<4,>=3.8
Requires-Dist: cmocean<5,>=3.0
Provides-Extra: cal
Requires-Dist: pypdf>=4; extra == "cal"
Provides-Extra: dbd
Requires-Dist: xarray-dbd>=0.1; extra == "dbd"
Requires-Dist: lz4>=4; extra == "dbd"
Provides-Extra: completion
Requires-Dist: argcomplete>=3; extra == "completion"
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Requires-Dist: pytest-cov; extra == "dev"
Requires-Dist: pytest-xdist; extra == "dev"
Requires-Dist: ruff; extra == "dev"
Requires-Dist: mypy; extra == "dev"
Requires-Dist: pre-commit; extra == "dev"
Requires-Dist: pypdf>=4; extra == "dev"
Requires-Dist: argcomplete>=3; extra == "dev"
Dynamic: license-file

# microstructure-tpw

[![CI](https://github.com/mousebrains/turbulence/actions/workflows/ci.yml/badge.svg)](https://github.com/mousebrains/turbulence/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/microstructure-tpw.svg)](https://pypi.org/project/microstructure-tpw/)
[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.21366142.svg)](https://doi.org/10.5281/zenodo.21366142)
[![Python 3.12+](https://img.shields.io/badge/python-3.12%2B-blue.svg)](https://www.python.org/downloads/)
[![License: GPL v3](https://img.shields.io/badge/License-GPLv3-blue.svg)](https://www.gnu.org/licenses/gpl-3.0)
[![codecov](https://codecov.io/gh/mousebrains/turbulence/graph/badge.svg?token=RwbKxeE7rA)](https://codecov.io/gh/mousebrains/turbulence)
[![Code style: ruff](https://img.shields.io/badge/code%20style-ruff-000000.svg)](https://github.com/astral-sh/ruff)
[![type checked: mypy](https://img.shields.io/badge/type%20checked-mypy-blue.svg)](https://mypy-lang.org/)

Python tools for reading Rockland Scientific microprofiler data and computing turbulent dissipation rates from VMP (Vertical Microstructure Profiler) and MicroRider instruments.

## Overview

`microstructure-tpw` provides a complete processing pipeline for ocean turbulence measurements from Rockland Scientific instruments equipped with shear probes and fast thermistors (FP07). The package reads proprietary `.p` binary data files, converts channels to physical units, detects profiles, and computes both the rate of dissipation of turbulent kinetic energy (epsilon) and the rate of dissipation of thermal variance (chi), following the methods described in the Rockland Scientific ODAS MATLAB Library and associated Technical Notes.

- **Epsilon (TKE dissipation rate)** from shear probe spectra ([detailed mathematics](docs/epsilon_mathematics.md)), including Goodman coherent noise removal, Nasmyth spectrum fitting, and Macoun & Lueck wavenumber correction.

- **Chi (thermal variance dissipation rate)** from FP07 thermistor spectra ([detailed mathematics](docs/chi_mathematics.md)), including Batchelor/Kraichnan spectrum models, FP07 transfer function correction, and spectral fitting (iterative Peterson & Fer 2014, the default, or MLE).

The package is organized into six subpackages under `odas_tpw`:

- **rsi** — Rockland Scientific instrument I/O, NetCDF conversion, profiles, epsilon/chi orchestration
- **chi** — Chi (thermal variance dissipation) calculation
- **scor160** — ATOMIX shear-probe benchmark processing and shared physics modules
- **perturb** — Full campaign processing pipeline (trim, merge, calibrate, compute, bin)
- **processing** — Instrument-agnostic profile processing (top trim, bottom-crash, CT alignment, multi-probe epsilon combining)

## Installation

```bash
pip install -e ".[dev]"    # editable install with dev dependencies
pip install .              # standard install
```

See [docs/installation.md](docs/installation.md) for more options.

## Quick Start

Several CLIs ship with the package (`rsi-tpw`, `perturb`, `perturb-plot`, `perturb-diag`, `scor160-tpw`); the two primary ones, for different workflows:

- **`rsi-tpw`** — single-file or short batch processing on the command line.
- **`perturb`** — full campaign pipeline driven by a YAML config file
  (trim → merge → profiles → epsilon → chi → bin → combo), with parallel
  workers and stage-aware logging. Use this for cruise-scale runs.

The two CLIs use different spectral defaults (`rsi-tpw`: `fft_length=1024`
with a 4096-sample dissipation window; `perturb`: duration-based
`fft_sec=1.0` — one second, resolved per instrument sampling rate,
= 512 samples on a 512-Hz VMP-250 — for both epsilon and chi), so their
outputs differ in vertical resolution and noise behavior.

### `rsi-tpw` (single files / short batches)

```bash
# Full pipeline: .p files → epsilon → chi
rsi-tpw pipeline VMP/*.p -o results/

# Or run individual stages
rsi-tpw eps VMP/*.p -o epsilon/                       # writes into epsilon/eps_00/
rsi-tpw chi VMP/*.p --epsilon-dir epsilon/ -o chi/    # finds eps_00/ automatically

# Pre-deployment bench test (dummy probes, instrument at rest): raw-count
# figures + spectra + an auto-evaluated Rockland checklist (see docs/rsi-tpw/bench.md)
rsi-tpw bench VMP/*.p -o bench/
```

> **Note:** `eps` writes into a hash-tracked subdirectory (`epsilon/eps_00/`,
> see [output directories](docs/rsi-tpw/output_directories.md)). The chi
> command searches the given `--epsilon-dir` and its `eps_*` subdirectories
> (most recently modified first), and matches both single-profile
> (`{stem}_eps.nc`) and per-profile (`{stem}_prof001_eps.nc`, ...) file
> names, concatenating the latter along time. Only when no matching epsilon
> files exist anywhere does chi fall back to Method 2 for that file, with a
> console warning.

```python
from odas_tpw.rsi.pipeline import run_pipeline
from pathlib import Path

# Full pipeline: .p → profiles → epsilon → chi → binning → combine
run_pipeline([Path("VMP/file.p")], Path("results/"))

# Or use the modular API
from odas_tpw.rsi.dissipation import compute_diss_file
from odas_tpw.rsi.chi_io import compute_chi_file

compute_diss_file("VMP/file.p", "epsilon/")
compute_chi_file("VMP/file.p", "chi/")
```

> **Note:** `get_diss()` and `get_chi()` still work for backward compatibility
> but are deprecated in favor of `run_pipeline()` or the modular
> `compute_diss_file()` / `compute_chi_file()` functions.

### `perturb` (campaign-scale batch pipeline)

```bash
# Write a template config you can edit (cruise paths, GPS file,
# instrument-specific overrides, parallelism, etc.).
perturb init my-cruise.yaml

# Run the full pipeline — outputs land in <output_root>/{profiles_NN,
# diss_NN, chi_NN, ctd_NN, *_binned_NN, combo}, with a per-run log under
# <output_root>/logs/.
perturb run -c my-cruise.yaml

# Or run individual stages against the same config:
perturb trim -c my-cruise.yaml
perturb profiles -c my-cruise.yaml
perturb diss     -c my-cruise.yaml
perturb chi      -c my-cruise.yaml
perturb bin      -c my-cruise.yaml
perturb combo    -c my-cruise.yaml
```

See [docs/perturb/pipeline.md](docs/perturb/pipeline.md) for the full stage list and [docs/perturb/configuration.md](docs/perturb/configuration.md) for the config schema.

## Documentation

### rsi-tpw (science library)

| Document | Description |
|----------|-------------|
| [CLI Reference](docs/rsi-tpw/cli.md) | All `rsi-tpw` subcommands and flags |
| [ODAS `.p` File Format](docs/rsi-tpw/odas_file_format.md) | TN-051 compliance: versions, bad buffers, sampling-rate and timestamp notes |
| [Configuration](docs/rsi-tpw/configuration.md) | YAML config file format and all parameter defaults |
| [Pipeline](docs/rsi-tpw/pipeline.md) | Processing stages and data flow |
| [Python API](docs/rsi-tpw/python_api.md) | Using microstructure-tpw from Python code |
| [Output Directories](docs/rsi-tpw/output_directories.md) | Sequential hash-tracked output scheme |
| [Vectorization](docs/rsi-tpw/vectorization.md) | Vectorized compute internals |
| [Calibration Provenance](docs/rsi-tpw/calibration_provenance.md) | Plausible-range checks on calibration coefficients, and the `cal_*` attrs on L1 variables |
| [Bench Test](docs/rsi-tpw/bench.md) | Pre-deployment bench check (`bench`): quick_bench figures + auto checklist |
| [Deployment Procedures](docs/deployment_procedures.md) | Field procedures that keep the data analysable: pre/post bench runs, probe/channel swaps, calibration bracketing, config hygiene |
| [Tab-completion](docs/rsi-tpw/completion.md) | Optional shell `<TAB>` completion for the CLIs (argcomplete) |

### perturb (batch pipeline)

| Document | Description |
|----------|-------------|
| [Pipeline](docs/perturb/pipeline.md) | Batch processing stages and data flow |
| [CLI Reference](docs/perturb/cli.md) | All `perturb` subcommands and flags |
| [Configuration](docs/perturb/configuration.md) | YAML config file format for perturb |
| [Parallel Scaling](docs/perturb/parallel.md) | Benchmark results for multi-core scaling |
| [Modules](docs/perturb/modules.md) | Module-level reference |
| [pyturb Comparison](docs/perturb/pyturb_comparison.md) | Comparison with oceancascades/pyturb |

### scor160 (ATOMIX benchmark)

| Document | Description |
|----------|-------------|
| [CLI Reference](docs/scor160/cli.md) | All `scor160-tpw` subcommands and flags |

### Shared

| Document | Description |
|----------|-------------|
| [Epsilon Mathematics](docs/epsilon_mathematics.md) | TKE dissipation algorithm details |
| [Chi Mathematics](docs/chi_mathematics.md) | Thermal dissipation algorithm details |
| [Mixing Efficiency](docs/mixing_efficiency.md) | Mathematics of the dissipation ratio |
| [Best Practices](docs/best_practices.md) | Guidance for comparing turbulence measurements |
| [Bibliography](docs/bibliography.md) | Consolidated references |
| [Installation](docs/installation.md) | Installation options |
| [Changelog](CHANGELOG.md) | Version history |

## References

This package is a Python implementation derived from the
[Rockland Scientific ODAS MATLAB Library](https://rocklandscientific.com/support/software/) (v4.5.1)
and associated
[Technical Notes](https://rocklandscientific.com/support/technical-notes/)
(TN-028, TN-051, TN-061).

See [docs/bibliography.md](docs/bibliography.md) for the full list of references.

## Testing

```bash
python -m pytest                          # run all tests
python -m pytest tests/test_epsilon.py    # epsilon pipeline tests only
```

## Development

This project was developed in collaboration with [Claude Code](https://claude.ai/code).

## Validation and known limitations

**Epsilon** is validated against the [ATOMIX shear-probe
benchmark](https://doi.org/10.1038/s41597-024-03323-y) (Fer et al. 2024) —
1416 spectra from six benchmark records spanning 10⁻¹⁰ to 10⁻³ W/kg, with a
log10 bias below 0.09, log10 RMSD of 0.008–0.096, and 100% of estimates within
half a decade of the published reference.

**Chi** has no equivalent benchmark and its absolute level is *not* anchored:
report it as a relative field with a systematic of order **1.5×**. Note also
that a chi `fom` near 1.0 does not mean the spectral fit is good — it is an
integrated ratio, and cancellation across the band hides a factor-of-two
structured residual.

Before trusting an absolute level on your own data, audit the calibration
constants (`rsi-tpw sensors --cal-dir`, `--diff-gain`): they, not the
estimator, are usually the dominant uncertainty.

Full detail, and suggested wording for a manuscript, in
[docs/validation_and_limitations.md](docs/validation_and_limitations.md).

## Citation

If you use this software in published work, please cite it. Citation metadata
lives in [`CITATION.cff`](CITATION.cff) — GitHub renders a **"Cite this
repository"** button from it (BibTeX/APA export). Each tagged release is
archived on Zenodo with a DOI.

Cite the **version DOI** of the release that produced your numbers — the
concept DOI resolves to whatever release is newest, which is not what you ran.
Version DOIs are listed in `CITATION.cff`; v0.3.0 is
[`10.5281/zenodo.21366143`](https://doi.org/10.5281/zenodo.21366143). The
**concept (all-versions) DOI**
[`10.5281/zenodo.21366142`](https://doi.org/10.5281/zenodo.21366142) is the
right one to cite when referring to the software in general rather than to a
specific result.

## License

GPLv3 — see [LICENSE](LICENSE) for details.
