Metadata-Version: 2.3
Name: xdiffly
Version: 0.4.2
Summary: X-Diff: a general-purpose tool for exploring differences between datasets (netCDF today, more targets to come)
Author: Antonio Mariani
Author-email: Antonio Mariani <antonio.mariani@cmcc.it>
Requires-Dist: xarray>=2024.6.0,<2027.0.0
Requires-Dist: pandas>=2.2.2,<4.0.0
Requires-Dist: numpy>=1.23.2,<3.0.0
Requires-Dist: netcdf4>=1.7.1,<2.0.0
Requires-Dist: rich>=13.7.1,<14.0.0
Requires-Dist: click>=8.3.1,<9.0.0
Requires-Dist: xdiffly[dask] ; extra == 'all'
Requires-Dist: xdiffly[plot] ; extra == 'all'
Requires-Dist: dask>=2024.6.0,<2027.0.0 ; extra == 'dask'
Requires-Dist: distributed>=2024.6.0,<2027.0.0 ; extra == 'dask'
Requires-Dist: bokeh>=3.1.0,<4.0.0 ; extra == 'dask'
Requires-Dist: matplotlib>=3.8.0,<4.0.0 ; extra == 'plot'
Requires-Dist: holoviews>=1.19.0,<2.0.0 ; extra == 'plot'
Requires-Dist: panel>=1.9.0,<2.0.0 ; extra == 'plot'
Requires-Dist: bokeh>=3.8.0,<4.0.0 ; extra == 'plot'
Requires-Dist: datashader>=0.16.0,<0.19.0 ; extra == 'plot'
Requires-Dist: numba>=0.60.0,<1.0.0 ; extra == 'plot'
Requires-Python: >=3.11, <3.15
Project-URL: Homepage, https://github.com/anto6715/X-Diff
Project-URL: Repository, https://github.com/anto6715/X-Diff
Project-URL: Issues, https://github.com/anto6715/X-Diff/issues
Project-URL: Changelog, https://github.com/anto6715/X-Diff/blob/master/CHANGES.md
Provides-Extra: all
Provides-Extra: dask
Provides-Extra: plot
Description-Content-Type: text/markdown

# X-Diff

`xdiff` is the CLI for **X-Diff**, a general diff tool name where `X` can stand for different comparison targets.
Today, X-Diff supports detailed comparison of netCDF files and helps users identify differences between datasets stored
in netCDF format.

![Python](https://img.shields.io/badge/Python-3.11--3.14-blue.svg)
[![Tests](https://github.com/anto6715/X-Diff/actions/workflows/tests.yml/badge.svg?branch=master)](https://github.com/anto6715/X-Diff/actions/workflows/tests.yml)
[![Coverage](https://codecov.io/gh/anto6715/X-Diff/graph/badge.svg?branch=master)](https://codecov.io/gh/anto6715/X-Diff)

![Output](https://github.com/anto6715/X-Diff/raw/master/docs/output.png)

## Installation

### Install uv

Follow the official installer:

```shell
curl -LsSf https://astral.sh/uv/install.sh | sh
```

### Install in a local virtual environment (recommended for development)

`xdiff` currently supports Python 3.11 through 3.14. Create the project-local environment and install dependencies from `uv.lock` with:

```shell
uv sync --python 3.14
```

Run the CLI through uv:

```shell
uv run xdiff --help
```

### Install globally with uv tool

The package is published on PyPI as `xdiffly`; it installs the `xdiff` command.

```shell
uv tool install --python 3.14 xdiffly
```

`uv tool install` installs `xdiffly` in uv's global tool environment (similar to `pipx`), not inside this repository's `.venv`. After installation, run it as `xdiff`.

The base install runs serially and is intentionally lightweight. Optional extras add the heavier backends:

| Extra | Adds |
|---|---|
| `dask` | Dask-backed parallel execution (`-w`, `--dask-scheduler`, `--dask-scheduler-file`) |
| `plot` | The `plot` subcommand (matplotlib for static images; holoviews/panel/bokeh/datashader for the live server) |
| `all` | Both of the above |

To install everything:

```shell
uv tool install --python 3.14 "xdiffly[all]"
```

Or pick one, e.g. `"xdiffly[dask]"` or `"xdiffly[plot]"`.

> **Ask for every extra in one command.** Each `uv tool install` replaces the tool's environment, so installing `"xdiffly[plot]"` after `"xdiffly[dask]"` removes Dask. Use `"xdiffly[all]"` or `"xdiffly[dask,plot]"` instead. (`pip install` adds to what is already installed, so it does not have this problem.)

## Usage

From a source checkout, prefix commands with `uv run`. If you installed with `uv tool install`, use `xdiff` directly.

```shell
uv run xdiff [OPTIONS] COMMAND [ARGS]...

  Explore differences between datasets.

Options:
  --version   Show the version and exit.
  -h, --help  Show this message and exit.

Commands:
  dirs   Compare two directories of datasets.
  files  Compare two dataset files directly, even if their filenames differ.
  plot   Plot where two netCDF files differ (static image or live server).

```

### Reading the result

Each compared variable gets one row with the minimum and maximum difference (reference minus comparison), the maximum relative error, and whether the two missing-value masks are equal. A variable **passes only when it is identical on both sides**: same shape and data type, both differences and the relative error `0`, and equal masks. There is no tolerance, so a difference of `1e-12` is reported as a failure. A file pair passes when at least one variable was compared and all of them passed; a file with no counterpart, a file that cannot be read, and a pair with no comparable variables are failures.

By default every data variable and every dimension coordinate **of the reference file** is compared, under the same name in the comparison file. String variables are skipped. Not compared: variables and files that exist only on the comparison side, auxiliary coordinates (for example 2-D `nav_lon`/`nav_lat`), and attributes. Two inputs that differ only in those still pass.

Exit codes, so `xdiff` can be used directly in scripts and CI jobs:

| Code | Meaning |
|---|---|
| `0` | Every compared pair passed. |
| `1` | A pair failed, nothing was compared (for example no file matched the filter), or the run stopped on an error. |
| `2` | Invalid command-line usage (unknown option, missing path, conflicting options). |

### Select Variables

It is possible to choose which parameter to compare:

```shell
uv run xdiff dirs folder1 folder2 -v votemper -v vosaline
```

![Variables](https://github.com/anto6715/X-Diff/raw/master/docs/variables.png)

To compare variables that are named differently between the two inputs, use `REF=CMP`:

```shell
uv run xdiff files reference.nc comparison.nc -v thetao=votemper -v lon=longitude
```

### Restrict to a lon/lat box

When two inputs share the same grid but cover different extents (e.g. a global file vs a regional subset), crop both to a common `--bbox LON_MIN LON_MAX LAT_MIN LAT_MAX` before comparing. Both 1-D (rectilinear) and 2-D (curvilinear, e.g. NEMO `nav_lon`/`nav_lat`) coordinates are supported:

```shell
uv run xdiff files global.nc regional.nc -v thetao --bbox -6 36 30 46
```

Inputs on *different* grids or resolutions need regridding first — that is out of scope for `--bbox`.

### Plot where two files differ

`xdiff plot` turns the comparison from *numbers* into a *picture* of **where** two files differ. The difference is the focus — drawn on a diverging colormap centered at 0, so red/blue shows the sign of the disagreement. It reuses the comparison options — `-v` (including `REF=CMP`), `--bbox`, and `--last-time-step` — so you plot exactly what you would compare. Requires the [`plot` extra](#install-globally-with-uv-tool).

Both modes draw a plain lon/lat map (no projection or coastlines): land is the data's own NaN mask, painted grey. There are two modes, selected by the presence of `-o`:

**Static image** — render the **difference** map (one full-size figure per variable) to a file and exit, for reports and scripting. The map is smoothly shaded and drawn with a latitude-corrected aspect so the domain is not distorted. The extension picks the format (`.png`, `.pdf`, `.svg`); with multiple variables the label is inserted into the filename (`diff.png` → `diff_thetao.png`, …):

```shell
uv run xdiff plot reference.nc comparison.nc -v thetao -o diff.png
```

**Live interactive server** — omit `-o` to start a local server, open the browser, and block until Ctrl-C. A sidebar drives everything: pick any **variable** in the file, step through **time/depth** levels, adjust the **colour limit** and **colormap** live (zoom preserved), toggle **smooth ↔ blocks** rendering, and optionally overlay a web-map **basemap** (Carto, OSM, Esri, …; needs internet). The difference is shown large and **datashaded**, so it **re-aggregates server-side as you zoom** (scroll to zoom) and scales to large grids; the min/max readout shows the true magnitude and hovering reads off values. The reference and comparison maps sit in a collapsed card at the bottom. Nothing is written to disk; when `xdiff` exits, the server stops.

```shell
uv run xdiff plot reference.nc comparison.nc -v thetao
```

The server binds `localhost` only. On a remote/HPC login node, forward the port over SSH and open the URL locally:

```shell
# on your laptop
ssh -L 5006:localhost:5006 user@login-node
# then, in that session
xdiff plot reference.nc comparison.nc -v thetao --no-open
# finally, open http://localhost:5006 in your local browser
```

Use `--port N` if `5006` is taken (a busy port fails immediately with a clear message — it is never silently moved, which would break the tunnel). `--no-open` skips launching a browser and just prints the URL, for headless sessions.

### Filter files

By default **xdiff** iterates over all files in **folder1** and expects to find them in **folder2**. Using filters,
it is possible to select only a subset of input files. For example:

```shell
uv run xdiff dirs folder1 folder2 -f "*_grid_T.nc"
```

### Compare files with different filenames

It is possible to compare two files with different filenames directly:

```shell
uv run xdiff files a/my-simu_19820101_grid_T.nc b/another-exp_19820101_grid_T.nc
```

For directory comparisons, files with different names can still be matched if they share a common substring.
For example, given:

- `a/my-simu_19820101_grid_T.nc`
- `b/another-exp_19820101_grid_T.nc`

Pass the common part as a regex pattern:

```shell
uv run xdiff dirs folder1 folder2 --common-pattern "\d{8}"
```

The pattern is matched against both filenames using `re.findall`. Two files are considered a pair when the
pattern produces the same match in both names — in this case the shared date `19820101`.

### Dask file-level execution

`xdiff` defaults to serial execution. Dask-backed file-level execution is opt-in and requires the optional `dask` extra (`uv tool install "xdiffly[dask]"`, or `uv sync --extra dask` from a source checkout). See [docs/dask.md](docs/dask.md) for local-cluster and external-scheduler examples.

## Testing

GitHub Actions runs the test suite on every pull request and on pushes to `master`. Coverage is uploaded from CI to Codecov, which powers the README coverage badge.

To run the same checks locally, install the project and dev dependencies with a single command:

```shell
uv sync --group dev
```

Then run the suite:

```shell
uv run pytest --cov --cov-report=term-missing --cov-report=xml
```

## Changelog

This repository uses `towncrier` for release notes. Every pull request except release PRs must include a changelog entry under `changes.d/` (CI checks it), for example:

```text
changes.d/123.bugfix.md
changes.d/124.doc.md
changes.d/+internal-cleanup.misc.md
```

Use the pull request number as the filename prefix when you want Towncrier to render a linked PR reference. With the current configuration, `changes.d/123.bugfix.md` will render as `[#123]` in `CHANGES.md`. Use `+` instead of a number when there is no associated PR to link.

Create a changelog entry with the Towncrier CLI:

```shell
uv run towncrier create 123.bugfix.md --content "Improved CLI filtering so directory comparisons skip unrelated files more reliably."
```

Create an orphan entry when there is no associated PR:

```shell
uv run towncrier create +internal-cleanup.misc.md --content "Cleaned up internal comparison helpers and simplified related tests."
```

If you omit `--content`, `towncrier create` will open your editor so you can write the entry interactively.

Validate or preview changelog entries locally with:

```shell
uv run towncrier build --draft --version X.Y.Z
```

To mirror the CI-style branch check after committing or staging your changelog entry:

```shell
git fetch origin master:refs/remotes/origin/master
uv run towncrier check --compare-with origin/master --staged
```

In normal feature work, contributors create entries with `towncrier create` and optionally preview them with `towncrier build --draft`. The final `towncrier build --yes` is done by the release workflow.

### Releasing

There is no `develop` branch: feature and fix branches are cut from `master` and merged back into it. To release what is on `master`:

1. In the repository's **Actions** tab, run the **Release** workflow ([`.github/workflows/release.yml`](.github/workflows/release.yml)) from `master` with the version, written as `X.Y.Z`.
2. The workflow creates `release/X.Y.Z` with one commit (the version in `pyproject.toml`, the regenerated `uv.lock`, and `CHANGES.md` built from `changes.d/`), starts the test suite on that commit, and opens a pull request into `master`.
3. Merge the pull request once its checks are green.
4. Publish a GitHub Release tagged `X.Y.Z` (no `v` prefix). That uploads the package to PyPI.

## Author

- Antonio Mariani (antonio.mariani@cmcc.it)

## Contributing

Contributions are welcome! Please open an issue or submit a pull request for any improvements or bug fixes.

## Contact

For any questions or suggestions, please open an issue on the project's GitHub repository.
