Metadata-Version: 2.5
Name: watch-duck
Version: 1.2.1
Summary: Personal python toolbox to track progress of ecflow experiments
Project-URL: Issues, https://github.com/aFarchi/watch-duck/issues
Project-URL: Source, https://github.com/aFarchi/watch-duck
Author-email: Alban Farchi <alban.farchi@ecmwf.int>
License-File: LICENSE
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: Scientific/Engineering
Requires-Python: >=3.11
Requires-Dist: bottleneck
Requires-Dist: cfgrib
Requires-Dist: dask
Requires-Dist: h5netcdf
Requires-Dist: numpy
Requires-Dist: omegaconf
Requires-Dist: pandas
Requires-Dist: rich
Requires-Dist: rich-click
Requires-Dist: xarray
Requires-Dist: zarr
Description-Content-Type: text/markdown

# watch-duck

[![PyPI version](https://badge.fury.io/py/watch-duck.svg)](https://badge.fury.io/py/watch-duck)
[![Conda Version](https://img.shields.io/conda/vn/conda-forge/watch-duck.svg)](https://anaconda.org/conda-forge/watch-duck)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

Toolbox to monitor the progress of ecflow experiments.

## Installation

Install using `pip`:

    $ pip install watch-duck

or using `conda`:

    $ conda install -c conda-forge watch-duck

For some functionalities (see below), you need to install `sitesctl` 
from [here](https://confluence.ecmwf.int/display/UDOC/SitesCTL%3A+Manage+Website+Content).

## Usage

### Configuration

When running, `watch-duck` reads the config file `~/.config/watch-duck.toml`.
The file is organised in sections, each one being associated to a specific command,
except for the main section, which is used in all commands to provide the working
directory:
```toml
[main]
wdir = '/path/to/the/working/directory'
```
In the following, the path to the working directory will simply be called `/wdir`.

### Global options

The following options go before the command name:
- `--debug`: enable debug logging for all commands, e.g. `watch-duck --debug parse`;
- `--version/-v`: show the version and exit;
- `-h/--help`: show the help message and exit.

### Commands

#### Get status

On the HPC, use the following command to get the state of a given ecflow suite
in a log file:
```sh
ecflow_client --host=<host> --get_state=<suite> "/wdir/log/<suite>_$(date +%Y_%m_%d_%H_%M_%S).log"
```
where `<suite>` should be replaced by the suite (eg. `daaf`) that you want
to monitor and `<host>` by the host of the given suite (eg. `ecflow-pifs-rd-f-1`).

You need to run this command at least once per hour to get a proper report.
Note that if you do so using `hpc-cron`, you would need to run
```sh
source /etc/profile
module load ecflow
```
before you are able to use `ecflow_client`.

#### Parse

On the HPC, use the following command to parse the log files stored in `/wdir/log/`:
```sh
watch-duck parse
```
The progress and state of each experiment will be stored in dedicated zarr archives:
`/wdir/progress/<exp>.zarr` and `/wdir/state/<exp>.zarr`.

For this command, you need to provide the following config:
```toml
[parse]
chunk_size_state = 128
chunk_size_progress = 128
suites = [
    'daaf',
]
exclude_experiments = [
]
exclude_experiment_types = [
    'fc50',
    'unknown',
]
```
where you can specify:
- the zarr chunk size (over time) of the progress and state archives;
- the list of suites that you want to monitor;
- the list of experiments to ignore;
- the list of experiment types to ignore.

Ideally, you should call this command each time you produced new log files, e.g.
using `hpc-cron`.

#### Report

On the HPC, use the following command to produce a short report about
the active experiments:
```sh
watch-duck report
```
This will read the progress archive of each active experiment and write a short
report of the progress over the last 10 days (subsampled to a 1-hour frequency)
in `/wdir/report.h5`.

For this command, you don't need to provide further config.

Ideally, you should call this command each time you updated the progress archive,
e.g. using `hpc-cron`.

#### Upload

On the HPC, use the following command to upload the latest report to the
specified website:
```sh
watch-duck upload
```
For this command, you need `sitesctl` and you need to provide the following config:
```toml
[site]
space = '<space>'
name = '<name>'
```
where `<space>` and `<name>` correspond to the space and name of the website.

Ideally, you should call this command each time you updated the report,
e.g. using `hpc-cron`.

#### Download

On any device, use the following command to download the latest report from
the specified website:
```sh
watch-duck download
```
For this command, you need `sitesctl` and you need to provide the following config:
```toml
[site]
space = '<space>'
name = '<name>'
```
where `<space>` and `<name>` correspond to the space and name of the website.

Ideally, you should call this command before showing a summary.

#### Summary

On any device, use the following command to show a summary of the latest report:
```sh
watch-duck summary
```
Use the help option to show all available options (in particular how to show
the progress only for a given suite or experiment type).

For this command, you need to provide the following config:
```toml
[summary]
delta_t = '48h'
exclude_aborted = true
exclude_suspended = true
vref_fc = 25
vref_lw = 4
vref_elda = 4
```
where you can specify:
- some parameters to compute the instantaneous speed of the experiments 
(averaging period, and whether to exclude times where the experiment was aborted or suspended);
- the reference speed for each experiment type (used to set the upper limit of the 
colour scale).

#### Finished

On any device, use the following command to show the list of experiments
that recently finished (from the latest report):
```sh
watch-duck finished
```
For this command, you don't need to provide further config.

#### Show

The `show` command is simply a concatenation of the `download` (optional),
`summary`, and `finished` commands. Use the `--download/-d` option to download
the latest report first:
```sh
watch-duck show -d
```

For this command, you need to provide the `[summary]` config (see above), and
also the `[site]` config and `sitesctl` (see `download`) if you use `-d`.

#### Iver

On the HPC, use the following command to compute the IVER scores of compatible
experiments (types fc, lw and elda):
```sh
watch-duck iver <profile>
```
where `<profile>` is the name of the IVER profile to run. Following IVER's
behaviour, the profile's working directory (called `/iver-wdir` in the following)
is found in the `~/.iver.<profile>` file.
Within this directory, `watch-duck iver` will look for a yaml file named 
`/iver-wdir/watch_duck.yaml`. This yaml file contains all the parameters used
to configure IVER for that specific profile:
```yaml
date_start: 2025-01-01
date_end: 2025-12-27
date_freq: 288h
version: 3.17
obstat: false
tech: false
exp_template: <ID>
experiments:
  exp_id: exp_type
  ...
```
where you can specify:
- the IVER version;
- the start and end date of the forecasts;
- the frequency between forecasts;
- whether to include obstat and a tech report;
- a template experiment ID (will only be used in the following section);
- a list of all experiments IDs and their corresponding type.

For each experiment within this list, `watch-duck iver` will use IVER to (1)
either download the latest forecasts if the experiment is still ongoing or (2)
compute the IVER scores if the experiment is finished and the scores have not been
computed yet.

You can bypass the list of experiments and use IVER on a specific
experiment by providing its ID via the `--experiment` option and its type via
the `--experiment-type` option. You can also force IVER to re-download the forecasts
from scratch using the `--clean` option. These additional options are summarised
in the help option of the `watch-duck iver` command.

Ideally, you should call this command on a regular basis, typically once a day,
e.g. using `hpc-cron`.

#### Iver without MARS

Alternatively, you can set up your experiment so that it directly outputs the
forecast fields on disk instead of archiving them on MARS. In that case,
the IVER scores can easily be computed in two steps once the experiment is finished:
1. convert the grib files into NetCDF (as used by IVER);
2. run IVER on the gathered NetCDF files.

Step 1 can be done on the HPC using the following command:
```sh
watch-duck grib-to-nc <profile>
```
where `<profile>` is the name of the IVER profile.
The list of experiments matching that profile is found in the 
`watch_duck.yaml` file of that profile (see previous section).
For each experiment, if the NetCDF file (containing the forecast fields)
is required and if the experiment is finished (i.e. if all the grib files
are available), the grib files will be read and converted into two
NetCDF files (one for surface and one for upper air).
Each NetCDF file is written separately, and right after one is written the grib
files that were read for it are deleted from `<tmpdir>/<exp>/`. Once both NetCDF
files exist, the `<tmpdir>/<exp>/` folder is removed if it is empty (otherwise,
a warning is logged).

For this command, you need to provide the following config:
```toml
[forecasts]
tmpdir = '/path/to/forecasts'
```
where `tmpdir` is the temporary directory in which the forecast fields will appear.

Note that IVER expects a non-standard encoding of dimensions, that
is difficult to reproduce in python. Therefore, for simplicity the encoding
of the dimensions of the NetCDF files is inferred from a template experiment,
whose ID is specified in the `watch_duck.yaml` file of that profile
(see previous section). For this experiment, the IVER has to be produced using 
the "standard" process (i.e. with the `watch-duck iver` command). 

Then, step 2 can be done on the HPC using the following command:
```sh
watch-duck iver-no-mars <profile>
```
where `<profile>` is the name of the IVER profile.
This command will compute the IVER scores of the profile's experiments
for which the NetCDF files are already available.

#### Clean

On the HPC, use the following command to clean old files:
```sh
watch-duck clean
```
This removes the files older than 30 days among the archived log files
(`/wdir/log/arxiv/*.log`), the archived active-experiment files
(`/wdir/active/arxiv/*.txt`) and the slurm outputs (`/wdir/slurm/*.out`).
IVER's temporary files are not handled here: they are cleaned automatically at
the end of `watch-duck iver`.

For this command, you don't need to provide further config.

Ideally, you should call this command every hour, e.g. using `hpc-cron`.

### Summary

On the HPC, every hour you should call:
- `ecflow_client` to get the state of the suites;
- `watch-duck parse`;
- `watch-duck report`;
- `watch-duck clean`;
- `watch-duck upload`;

and every day you should call:
- `watch-duck iver <profile>` for each profile where forecasts are archived in MARS;
- `watch-duck grib-to-nc <profile>` and `watch-duck iver-no-mars <profile>`
for each profile where forecasts are stored on disk.

NB: if you are only interested in the following suites: daaf, dae, dav, nemc,
all the hourly HPC commands above (`ecflow_client`, `parse`, `report`, `clean`
and `upload`) are already run for them by daaf, so you can skip this first step.
Just make sure to set your `/wdir` to daaf's `/wdir` if you are on the HPC, or
to use daaf's `iver` site if you are on any other device.

Then, on any device, you can call at any time:
- `watch-duck download` (you don't need this if you are on the HPC);
- `watch-duck summary`;
- `watch-duck finished`;

or simply `watch-duck show -d`.

## Version update checklist

- format and lint the code
- update the README.md
- add the version in the CHANGELOG.md
- bump version number in pyproject.toml
- commit and push
- build and publish (`hatch build` & `hatch publish`); hatch is intentionally
kept outside the project environment and is installed with `pixi global install hatch`
- add the tag release on github
