Metadata-Version: 2.4
Name: PMRIVariants
Version: 2.2.5
Summary: PMRI variants and PMRI-SRI for predictive reliability analysis with exports and plots
Author: Dr Md Galal Uddin
License: MIT
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.23
Requires-Dist: pandas>=1.5
Requires-Dist: scipy>=1.10
Requires-Dist: scikit-learn>=1.2
Requires-Dist: matplotlib>=3.7
Requires-Dist: openpyxl>=3.1
Dynamic: license-file

# PMRIVariants

**PMRIVariants** is a Python package for computing the **Predictive Model Reliability Index (PMRI)** family of metrics for predictive, environmental, hydrological, remote-sensing, geospatial, and water-quality models.

The package is designed to move beyond conventional average-error statistics and provide a more structured evaluation of **predictive reliability**, including agreement, stability, uncertainty behavior, regime sensitivity, and spatial trustworthiness.

---

## Project description

PMRIVariants was developed to support researchers, analysts, engineers, and practitioners who need more than conventional accuracy metrics when assessing predictive models. In many scientific and operational applications, a model may achieve a strong average fit while still being unreliable across decision boundaries, unstable across subsets, poorly calibrated in uncertainty, weak under regime shifts, or spatially inconsistent.

This package addresses that gap by providing a family of reliability-oriented metrics under the **PMRI framework**. It supports compact screening metrics, broader operational reliability assessment, regime-aware data-driven reliability analysis, and spatial reliability mapping for georeferenced predictions. The package is especially suitable for environmental, hydrological, water-quality, geospatial, and remote-sensing applications, but it can also be used more broadly for machine learning and predictive modelling problems.

In addition to PMRI-family metrics, the package supports common statistical performance metrics, multiple export options, and plotting utilities that make it suitable for both research workflows and publication-ready evaluation.

---

## Author

**Dr. Md Galal Uddin**  
Postdoctoral Researcher, Civil Engineering  
University of Galway, Ireland  
Adjunct Research Faculty, Munster Technological University, Cork, Ireland  
Email: `jalaluddinbd1987@gmail.com`

---

## Package variants

The package currently includes the following PMRI variants:

- **PMRI-Lite**
- **PMRI-Core**
- **PMRI+**
- **PMRI+ Data-Driven**
- **PMRI-SRI** (*Spatial Reliability Index*)

---

## Table of contents

1. [Concept and motivation](#1-concept-and-motivation)  
2. [Key updates](#2-key-updates)  
3. [Mathematical overview](#3-mathematical-overview)  
4. [Installation](#4-installation)  
5. [Package structure](#5-package-structure)  
6. [Required inputs by variant](#6-required-inputs-by-variant)  
7. [Supported input file formats](#7-supported-input-file-formats)  
8. [Common evaluation metrics included](#8-common-evaluation-metrics-included)  
9. [Example usage](#9-example-usage)  
10. [Example for standard metrics](#10-example-for-standard-metrics)  
11. [Plotting support](#11-plotting-support)  
12. [Example input tables](#12-example-input-tables)  
13. [Main API](#13-main-api)  
14. [Reliability classes](#14-reliability-classes)  
15. [Recommended applications](#15-recommended-applications)  
16. [Scientific note](#16-scientific-note)  
17. [License](#17-license)  

---

## 1. Concept and motivation

Traditional predictive performance metrics such as **R²**, **RMSE**, **MSE**, **MAE**, and related statistics are useful for quantifying average model accuracy. However, they do not fully determine whether a model is sufficiently **reliable** for scientific interpretation, decision-support, operational deployment, regulatory assessment, or spatial applications.

A model may appear accurate on average while still being:

- unstable across subsets or ordered blocks of data,
- inconsistent near important decision thresholds,
- poorly calibrated with respect to uncertainty,
- weak in preserving the observed distribution,
- unable to represent extremes,
- or spatially unreliable across different locations.

The **PMRI framework** was developed to address these limitations by quantifying predictive reliability in a hierarchical and extensible manner.

### PMRI-Lite

A minimal reliability score based only on monotonic agreement between observations and predictions. It is designed for quick screening when only actual and predicted values are available.

### PMRI-Core

A compact reliability score that combines:

- **rank consistency**,
- **decision agreement**, and
- **stability across ordered data blocks**.

PMRI-Core is designed to work **without requiring an explicit time variable**. If a time column is not available, the native row order of the dataset can be used.

### PMRI+

A broader operational reliability score that extends PMRI-Core with:

- uncertainty coverage,
- distributional similarity,
- variance equivalence,
- extreme-value fidelity,
- and equivalence testing.

PMRI+ is intended for more serious scientific or operational evaluation where reliability must be assessed across multiple dimensions.

### PMRI+ Data-Driven

A regime-conditioned extension of PMRI+ that automatically clusters the available **numeric predictor space**. It does **not require fixed variables** such as temperature, flow, or turbidity. Any meaningful numeric predictors can be used. This makes the method flexible for machine learning, environmental, and remote-sensing applications where predictor sets vary between datasets.

### PMRI-SRI

A spatially explicit reliability index for georeferenced datasets that combines local error, local bias, and local interval coverage to generate **spatial trust maps**. This is particularly useful when global metrics appear strong but local failures remain hidden.

---

## 2. Key updates

Recent updates include:

- `compute_pmri_sri()` now supports:
  - `export_excel`
  - `export_csv`
  - `export_json`
  - `export_txt`
  - `make_plots`
  - `plot_dir`

- `compute_pmri_plus_data_driven()` no longer requires `feature_cols`.
  - If omitted, the function automatically detects numeric predictor columns, excluding the actual and predicted columns.

- Common predictive metrics are now included:
  - **R2**
  - **RMSE**
  - **MSE**
  - **MAE**
  - **PABE**
  - **MAPE**
  - **MBE**
  - **Bias**
  - **Pearson r**
  - **Spearman rho**

- Exportable outputs now support:
  - summary tables,
  - detailed metric tables,
  - regime-specific outputs,
  - spatial reliability outputs,
  - text summaries,
  - JSON exports,
  - and diagnostic plots.

These updates make the package more suitable for both **publication-ready analysis** and **routine model evaluation workflows**.

---

## 3. Mathematical overview

### 3.1 PMRI-Lite

PMRI-Lite is defined as:

\[
PMRI_{Lite} = \frac{\rho_s + 1}{2}
\]

where:

- \(\rho_s\) = Spearman rank correlation coefficient.

This rescales Spearman correlation from \([-1,1]\) to \([0,1]\), where higher values indicate stronger monotonic agreement.

---

### 3.2 PMRI-Core

\[
PMRI_{Core} = \frac{CAR + Stability + Rank_{01}}{3}
\]

where:

- \(CAR\) = compliance or decision agreement rate,
- \(Stability\) = consistency across ordered data blocks,
- \(Rank_{01}\) = rescaled monotonic agreement.

PMRI-Core summarizes whether a model is not only accurate in rank ordering but also stable and decision-consistent across the dataset.

---

### 3.3 PMRI+

\[
PMRI_{+} = mean(PICP,\ CAR,\ Stability,\ DistEq,\ VarEq,\ Tail,\ Rank_{01},\ Equivalence)
\]

where:

- \(PICP\) = prediction interval coverage probability,
- \(CAR\) = classification or decision agreement,
- \(Stability\) = block-wise stability,
- \(DistEq\) = distributional equivalence,
- \(VarEq\) = variance equivalence,
- \(Tail\) = extreme-value fidelity,
- \(Rank_{01}\) = rank-based agreement,
- \(Equivalence\) = equivalence-test outcome or score.

This broader score attempts to capture operational reliability rather than only average predictive error.

---

### 3.4 PMRI+ Data-Driven

\[
PMRI_{+}^{DD} = \sum_{k=1}^{K} w_k \, PMRI_{+,k}
\]

where:

- \(K\) = number of regimes or clusters,
- \(w_k\) = weight of cluster \(k\),
- \(PMRI_{+,k}\) = PMRI+ score computed within cluster \(k\).

This formulation allows reliability to be evaluated under **different data regimes**, which is particularly useful when model behavior changes across environmental states, spatial domains, or predictor conditions.

---

### 3.5 PMRI-SRI

\[
PMRI_{SRI}(s)=w_1(1-\tilde e(s)) + w_2(1-\widetilde{|b|}(s)) + w_3PICP(s)
\]

where:

- \(s\) = spatial location,
- \(\tilde e(s)\) = normalized local error,
- \(\widetilde{|b|}(s)\) = normalized absolute local bias,
- \(PICP(s)\) = local interval coverage,
- \(w_1, w_2, w_3\) = component weights.

PMRI-SRI is useful for identifying **locally reliable** and **locally unreliable** model zones and for generating spatial trust diagnostics.

---

## 4. Installation

Install directly from PyPI:

```bash
pip install PMRIVariants
```

You may also install from local source:

```bash
pip install .
```

To upgrade an existing installation:

```bash
pip install --upgrade PMRIVariants
```

---

## 5. Package structure

```text
pmri_variants_package_v2/
├── pyproject.toml
├── README.md
├── LICENSE
├── src/
│   └── pmri_variants/
│       ├── __init__.py
│       ├── core.py
│       ├── sri.py
│       ├── analyzer.py
│       └── utils.py
├── examples/
│   ├── example_pmri_lite.csv
│   ├── example_pmri_core.csv
│   ├── example_pmri_plus.csv
│   ├── example_pmri_dd.csv
│   └── example_pmri_sri.csv
└── tests/
    └── test_smoke.py
```

### Description of main modules

- `core.py` — core PMRI variants and shared metric logic  
- `sri.py` — spatial reliability index routines  
- `analyzer.py` — object-oriented analysis interface  
- `utils.py` — helper functions for metrics, exports, and utilities  

---

## 6. Required inputs by variant

| Variant | Required inputs | Notes |
|---|---|---|
| PMRI-Lite | `Actual`, `Predicted` | Minimal version |
| PMRI-Core | `Actual`, `Predicted` | Uses dataset order if no explicit time is provided |
| PMRI+ | `Actual`, `Predicted` | Uses empirical prediction interval if no interval is supplied |
| PMRI+ Data-Driven | `Actual`, `Predicted`, numeric predictor columns | No fixed predictor names required |
| PMRI-SRI | `Actual`, `Predicted`, `Latitude`, `Longitude` | Spatially explicit reliability |

### Column naming guidance

The package is flexible, but users should explicitly define column names when calling the functions. Typical names include:

- `Actual`
- `Predicted`
- `Latitude`
- `Longitude`

For data-driven variants, additional numeric predictor columns may include any application-specific variables.

---

## 7. Supported input file formats

The package can be used with data loaded from:

- **CSV**
- **Excel** (`.xlsx`)
- **TXT** / delimited text files
- preloaded `pandas.DataFrame`

### Example: CSV

```python
import pandas as pd
df = pd.read_csv("PMRI.csv")
```

### Example: Excel

```python
import pandas as pd
df = pd.read_excel("PMRI.xlsx")
```

### Example: TXT

```python
import pandas as pd
df = pd.read_csv("PMRI.txt", sep="\t")
```

### Example: existing DataFrame

```python
# df already exists in memory
result = some_function(df)
```

---

## 8. Common evaluation metrics included

In addition to PMRI-family scores, the package can compute or export the following standard performance metrics:

| Metric | Meaning |
|---|---|
| **R2** | Coefficient of determination |
| **RMSE** | Root mean squared error |
| **MSE** | Mean squared error |
| **MAE** | Mean absolute error |
| **PABE** | Percentage absolute bias error |
| **MAPE** | Mean absolute percentage error |
| **MBE** | Mean bias error |
| **Bias** | Average signed error |
| **Pearson r** | Linear correlation |
| **Spearman rho** | Rank correlation |

These metrics are useful for comparing predictive performance alongside reliability.

### Brief interpretation

- **R2** indicates the proportion of variance explained.
- **RMSE** emphasizes larger errors more strongly.
- **MSE** is the squared-error equivalent of RMSE without the square root.
- **MAE** measures average absolute error magnitude.
- **PABE** quantifies percentage-based absolute bias behavior.
- **MAPE** gives relative percentage error magnitude.
- **MBE** indicates directional bias in predictions.
- **Bias** summarizes the signed average deviation.
- **Pearson r** captures linear association.
- **Spearman rho** captures monotonic association.

In practice, these should be reported alongside PMRI outputs to show both accuracy and reliability.

---

## 9. Example usage

### 9.1 PMRI-Lite

```python
import pandas as pd
from pmri_variants import compute_pmri_lite

df = pd.read_csv("examples/example_pmri_lite.csv")
result = compute_pmri_lite(df["Actual"], df["Predicted"])

print(result)
```

### 9.2 PMRI-Core

PMRI-Core does **not require a time column**.

```python
import pandas as pd
from pmri_variants import compute_pmri_core

df = pd.read_csv("examples/example_pmri_core.csv")
result = compute_pmri_core(df["Actual"], df["Predicted"])

print(result)
```

### 9.3 PMRI+

```python
import pandas as pd
from pmri_variants import compute_pmri_plus

df = pd.read_csv("examples/example_pmri_plus.csv")
result = compute_pmri_plus(df["Actual"], df["Predicted"])

print(result)
```

### 9.4 PMRI+ Data-Driven with explicit predictors

```python
import pandas as pd
from pmri_variants import compute_pmri_plus_data_driven

df = pd.read_csv("examples/example_pmri_dd.csv")
result = compute_pmri_plus_data_driven(
    df,
    actual_col="Actual",
    predicted_col="Predicted",
    feature_cols=["Feature_1", "Feature_2", "Feature_3"],
    n_clusters=3,
)

print(result["pmri_plus_dd"])
print(result["regimes"])
```

### 9.5 PMRI+ Data-Driven with automatic predictor detection

```python
import pandas as pd
from pmri_variants import compute_pmri_plus_data_driven

df = pd.read_csv("/content/PMRI.csv")
result = compute_pmri_plus_data_driven(
    df,
    actual_col="Actual",
    predicted_col="Predicted",
    n_clusters=3,
    export_excel="pmri_dd_results.xlsx",
    export_csv="pmri_dd_results.csv",
    export_json="pmri_dd_results.json",
    export_txt="pmri_dd_results.txt",
    make_plots=True,
    plot_dir="plots_dd"
)

print(result["pmri_plus_dd"])
```

### 9.6 PMRI-SRI basic use

```python
import pandas as pd
from pmri_variants import compute_pmri_sri

df = pd.read_csv("examples/example_pmri_sri.csv")
result = compute_pmri_sri(
    df,
    actual_col="Actual",
    predicted_col="Predicted",
    lat_col="Latitude",
    lon_col="Longitude",
)

print(result["sri_mean"])
```

### 9.7 PMRI-SRI with export options and plots

```python
import pandas as pd
from pmri_variants import compute_pmri_sri

df = pd.read_csv("/content/PMRI.csv")
result = compute_pmri_sri(
    df,
    actual_col="Actual",
    predicted_col="Predicted",
    lat_col="Latitude",
    lon_col="Longitude",
    export_excel="pmri_sri_results.xlsx",
    export_csv="pmri_sri_results.csv",
    export_json="pmri_sri_results.json",
    export_txt="pmri_sri_results.txt",
    make_plots=True,
    plot_dir="plots_sri"
)

print(result["sri_mean"])
```

### 9.8 Object-oriented interface

```python
import pandas as pd
from pmri_variants import PMRIAnalyzer

df = pd.read_csv("examples/example_pmri_plus.csv")

analyzer = PMRIAnalyzer(
    actual_col="Actual",
    predicted_col="Predicted",
)

result = analyzer.compute_all(df)
print(result.summary)
```

---

## 10. Example for standard metrics

The following illustrates how conventional metrics may be evaluated together with PMRI outputs.

```python
import pandas as pd
from pmri_variants import compute_pmri_sri

df = pd.read_csv("/content/PMRI.csv")

result = compute_pmri_sri(
    df,
    actual_col="Actual",
    predicted_col="Predicted",
    lat_col="Latitude",
    lon_col="Longitude",
    export_excel="pmri_metrics_results.xlsx",
    make_plots=True,
    plot_dir="pmri_plots"
)

print("SRI Mean:", result["sri_mean"])

# Depending on implementation, metrics may be available under:
# result["metrics"] or similar structured output
if "metrics" in result:
    print(result["metrics"])
```

### Typical exported metrics may include

- `R2`
- `RMSE`
- `MSE`
- `MAE`
- `PABE`
- `MAPE`
- `MBE`
- `Bias`
- `Pearson r`
- `Spearman rho`

### Example interpretation

A model might show:

- strong **R2**,
- moderate **RMSE**,
- low **Bias**,
- and high **Pearson r**,

but still exhibit weak **PMRI+** if its uncertainty coverage, stability, or regime consistency is poor. This is exactly why the PMRI framework is useful: it captures reliability characteristics that average metrics alone can miss.

---

## 11. Plotting support

When `make_plots=True`, the package can export relevant diagnostic and comparison plots depending on the selected variant and implementation. These may include:

- observed vs predicted scatter plots,
- residual plots,
- error distribution histograms,
- block stability plots,
- regime-wise comparison plots,
- spatial reliability maps,
- local bias maps,
- local coverage maps,
- performance summary charts.

### Example

```python
result = compute_pmri_sri(
    df,
    actual_col="Actual",
    predicted_col="Predicted",
    lat_col="Latitude",
    lon_col="Longitude",
    make_plots=True,
    plot_dir="plots_sri"
)
```

Generated plots are typically stored in the folder specified by `plot_dir`.

### Suggested use of plots

Plots are especially useful for:

- visually checking systematic bias,
- identifying clusters or regimes with different reliability,
- locating spatial hotspots of weak predictive trust,
- supporting journal manuscripts and technical reports,
- and communicating model behavior to non-technical stakeholders.

---

## 12. Example input tables

### PMRI-Lite

| Actual | Predicted |
|---:|---:|
| 72.0 | 70.8 |
| 75.0 | 73.6 |
| 81.0 | 82.1 |

### PMRI-Core

| Actual | Predicted |
|---:|---:|
| 71.2 | 70.9 |
| 72.0 | 71.8 |
| 73.4 | 73.2 |

### PMRI+

| Actual | Predicted |
|---:|---:|
| 7.90 | 7.95 |
| 8.05 | 8.02 |
| 8.10 | 8.08 |

### PMRI+ Data-Driven

| Actual | Predicted | Feature_1 | Feature_2 | Feature_3 |
|---:|---:|---:|---:|---:|
| 69.5 | 68.9 | 14.2 | 21.0 | 2.1 |
| 71.0 | 71.4 | 15.1 | 25.4 | 2.8 |

### PMRI-SRI

| Latitude | Longitude | Actual | Predicted |
|---:|---:|---:|---:|
| 53.27 | -9.05 | 74.0 | 73.2 |
| 53.28 | -9.01 | 76.1 | 75.8 |

---

## 13. Main API

### Functional API

- `compute_pmri_lite(actual, predicted)`
- `compute_pmri_core(actual, predicted, threshold=None)`
- `compute_pmri_plus(actual, predicted, threshold=None, delta=None)`
- `compute_pmri_plus_data_driven(df, actual_col, predicted_col, feature_cols=None, n_clusters=3, export_excel=None, export_csv=None, export_json=None, export_txt=None, make_plots=False, plot_dir=None)`
- `compute_pmri_sri(df, actual_col, predicted_col, lat_col, lon_col, export_excel=None, export_csv=None, export_json=None, export_txt=None, make_plots=False, plot_dir=None)`

### Object-oriented API

- `PMRIAnalyzer`

### Notes on API style

The functional API is convenient for scripted workflows, while the object-oriented API is helpful for more structured and repeatable analysis pipelines.

---

## 14. Reliability classes

The following classes may be used for interpretation:

| Score | Class |
|---:|---|
| > 0.80 | Operational |
| 0.60–0.80 | Decision-support |
| < 0.60 | Exploratory |

These thresholds help distinguish whether a model is suitable for:

- exploratory research,
- scientific decision-support,
- regulatory communication,
- or near-operational deployment.

---

## 15. Recommended applications

PMRIVariants can be applied in:

- water-quality modelling,
- hydrological prediction,
- environmental forecasting,
- remote sensing model validation,
- geospatial model comparison,
- ecological and biogeochemical modelling,
- AI/ML predictive reliability assessment,
- spatial decision-support systems.

### Typical use cases

- evaluating machine learning predictions for water-quality variables,
- comparing multiple predictive models beyond RMSE and R²,
- assessing whether a model is trustworthy across different regimes,
- diagnosing local spatial failures in otherwise strong models,
- generating publication-ready reliability evidence.

---

## 16. Scientific note

PMRI is intended to complement rather than replace conventional predictive metrics. In scientific studies, it is recommended to report:

- accuracy metrics such as **R²**, **RMSE**, **MAE**, and **MSE**,
- alongside reliability metrics such as **PMRI-Core**, **PMRI+**, or **PMRI-SRI**.

This combined reporting provides a more robust basis for:

- model comparison,
- interpretation,
- operational readiness assessment,
- and communication with decision-makers.

For serious applications, users are encouraged to report both:

1. **predictive accuracy**, and  
2. **predictive reliability**.

This helps distinguish between models that are merely accurate on average and models that are consistently dependable under real analytical conditions.

---

## 17. License

This project is licensed under the **MIT License**.
