Metadata-Version: 2.5
Name: ua-metrics
Version: 1.0.1
Summary: Standard and uncertainty-adjusted regression metrics with Gaussian, Student-t, and lognormal uncertainty models.
Project-URL: Homepage, https://github.com/Moha-Abbas/ua-metrics
Project-URL: Repository, https://github.com/Moha-Abbas/ua-metrics
Project-URL: Issues, https://github.com/Moha-Abbas/ua-metrics/issues
Author: Mohammad Abbas
License: MIT
License-File: LICENSE
Keywords: coefficient-of-variation,evaluation,metrics,regression,uncertainty
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering
Requires-Python: >=3.10
Requires-Dist: numpy>=1.24
Requires-Dist: scipy>=1.10
Provides-Extra: dev
Requires-Dist: build>=1.2; extra == 'dev'
Requires-Dist: pytest-cov>=5.0; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.5; extra == 'dev'
Requires-Dist: twine>=5.0; extra == 'dev'
Description-Content-Type: text/markdown

﻿# ua-metrics

[![PyPI version](https://img.shields.io/pypi/v/ua-metrics.svg)](https://pypi.org/project/ua-metrics/)
[![Python versions](https://img.shields.io/pypi/pyversions/ua-metrics.svg)](https://pypi.org/project/ua-metrics/)
[![License](https://img.shields.io/pypi/l/ua-metrics.svg)](LICENSE)
[![CI](https://github.com/Moha-Abbas/ua-metrics/actions/workflows/ci.yml/badge.svg)](https://github.com/Moha-Abbas/ua-metrics/actions/workflows/ci.yml)

`ua-metrics` is a Python package for standard and uncertainty-adjusted regression metrics,
accompanying the paper [*"Uncertainty-aware metrics for evaluating machine learning regression
models in materials testing"*](https://doi.org/10.1016/j.measurement.2026.122931).

The package provides three uncertainty-model modules:
- `ua_metrics.gaussian`
- `ua_metrics.student_t`
- `ua_metrics.lognormal`

## Why uncertainty-aware metrics?

Classical regression metrics (MAE, RMSE, MAPE, R²) all implicitly treat the target values
(`y_obs`) as exact, deterministic ground truth. That assumption does not hold in materials
testing, or in most physical measurement settings: every measurement carries some uncertainty
from instrument precision, environmental conditions, or sample variability, often already
quantified by a testing standard (e.g. a reported coefficient of variation).

Ignoring that uncertainty causes two problems:

- **Learning ceiling.** When a dataset contains a certain level of measurement uncertainty,
  models can reach a learning ceiling: the point beyond which further training does not
  produce real improvement in learning the true underlying signal, but instead reflects
  overcorrection within the uncertainty region. Classical metrics do not indicate when this
  irreducible limit has been reached.
- **Unstable rankings.** When comparing models with similar performance, small changes in the
  measured dataset can change which model performs better under classical metrics, leading to
  erroneous model ranking and potentially wrong conclusions.

`ua-metrics` implements uncertainty-adjusted versions of the standard metrics that subtract out
the expected contribution of a known measurement uncertainty (Gaussian, Student-t, or
lognormal), giving estimates that stay closer to a model's true performance. See the
accompanying paper for the full derivation and validation.

## Features

- Standard regression metrics.
- Uncertainty-adjusted regression metrics.
- Gaussian, Student-t, and lognormal uncertainty models.
- Heteroscedastic and homoscedastic uncertainty handling.
- A consistent module-based API.

## Installation

```bash
pip install ua-metrics
```

## Quick start

```python
from ua_metrics import gaussian as gau
from ua_metrics import lognormal as logn
from ua_metrics import student_t as t
```

Default behavior (heteroscedastic uncertainty, mean scaling) and its equivalent explicit form:

```python
v1 = gau.mae_ua([100, 110, 95], [102, 108, 97], 5.0)

v2 = gau.rmse_ua(
    [100, 110, 95],
    [102, 108, 97],
    5.0,
    mode="hetero",
    scale="mean",
)
```

Homoscedastic uncertainty with a constant absolute standard deviation:

```python
v3 = gau.mae_ua([100, 110, 95], [102, 108, 97], 1.0, mode="homo")
```

Student-t and lognormal uncertainty models:

```python
v4 = t.mae_ua([100, 110, 95], [102, 108, 97], 7.5, df=3.0)
v5 = logn.mae_ua([100, 110, 95], [102, 108, 97], 12.0)
```

## Package structure

### Standard metrics

```python
from ua_metrics import mae, median_absolute_error, mse, rmse
from ua_metrics import mape, smape, r2_score, adjusted_r2_score
```

### Uncertainty-adjusted metrics

```python
from ua_metrics import gaussian as gau
from ua_metrics import student_t as t
from ua_metrics import lognormal as logn
```

Each uncertainty module provides:
- `mae_ua`
- `median_absolute_error_ua`
- `mse_ua`
- `rmse_ua`
- `mape_ua`
- `smape_ua`
- `r2_score_ua`
- `adjusted_r2_score_ua`

## Interface

All uncertainty-adjusted metrics use the same public interface:

```python
metric_ua(y_obs, y_pred, value, *, mode="hetero", scale="mean", ...)
```

### Argument semantics

- `value` with `mode="hetero"` is interpreted as CV percent of uncertainty.
- `value` with `mode="homo"` is interpreted as a constant absolute uncertainty standard deviation.

### Default behavior

The default configuration is:
- `mode="hetero"`
- `scale="mean"`

Under this default, `value=5.0` means the uncertainty standard deviation is 5% of the observation-wise mean scale.

## Scale definitions

Supported `scale` values are:
- `"mean"`, defined as `0.5 * (y_obs + y_pred)`
- `"y_obs"`
- `"y_pred"`

## Notes

- Values greater than 100 are allowed in heteroscedastic mode.
- Homoscedastic mode expects a scalar constant absolute uncertainty value.

## Citation

If you use `ua-metrics` in your work, please cite the accompanying paper, published in
*Measurement*, Volume 290, Part C, Article 122931 (2026),
[doi:10.1016/j.measurement.2026.122931](https://doi.org/10.1016/j.measurement.2026.122931),
and/or the software itself. Machine-readable citation metadata is kept up to date in
[`CITATION.cff`](CITATION.cff).

```bibtex
@article{abbas_ua_metrics_paper,
  title   = {Uncertainty-aware metrics for evaluating machine learning regression models in materials testing},
  author  = {Abbas, Mohammad and Zaumanis, Martins},
  journal = {Measurement},
  volume  = {290},
  number  = {Part C},
  pages   = {122931},
  year    = {2026},
  issn    = {0263-2241},
  doi     = {10.1016/j.measurement.2026.122931}
}

@software{abbas_ua_metrics_software,
  title   = {ua-metrics},
  author  = {Abbas, Mohammad},
  year    = {2026},
  version = {1.0.1},
  url     = {https://github.com/Moha-Abbas/ua-metrics}
}
```

## License

MIT. See [LICENSE](LICENSE).
