Metadata-Version: 2.4
Name: rankfit
Version: 0.1.0
Summary: Detect local risk-ranking failures that AUC can miss.
Author-email: Emma Chung <hsiaoyin.chung@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/EMMA-CHUNG/rankfit
Project-URL: Repository, https://github.com/EMMA-CHUNG/rankfit
Project-URL: Issues, https://github.com/EMMA-CHUNG/rankfit/issues
Project-URL: Documentation, https://github.com/EMMA-CHUNG/rankfit#readme
Keywords: machine learning,model evaluation,ranking,AUC,decile analysis,monotonicity,kendall tau
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.21
Requires-Dist: pandas>=1.3
Requires-Dist: scipy>=1.7
Requires-Dist: matplotlib>=3.4
Requires-Dist: scikit-learn>=1.0
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0; extra == "dev"
Requires-Dist: black>=23.0; extra == "dev"
Requires-Dist: ruff>=0.1; extra == "dev"
Requires-Dist: twine>=4.0; extra == "dev"
Requires-Dist: build>=1.0; extra == "dev"
Dynamic: license-file

# RankFit: Detecting Local Risk-Ranking Failures That AUC Can Miss

[![PyPI version](https://badge.fury.io/py/rankfit.svg)](https://pypi.org/project/rankfit/)
[![Python](https://img.shields.io/pypi/pyversions/rankfit.svg)](https://pypi.org/project/rankfit/)
[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE)

RankFit is a model-agnostic diagnostic for binary classifiers. It checks whether
observed event rates decrease consistently as predicted scores move from the
highest-ranked segment to the lowest-ranked segment.

Lift charts have traditionally been inspected by eye. That is manageable for a
few models, but automated training and tuning workflows can produce hundreds of
candidates. RankFit turns the visual monotonicity check into a repeatable screen
that helps teams compare models and identify the charts that need attention.

ROC-AUC summarizes global discrimination. It can dilute a local inversion inside
the portion of a ranked list that drives an operational decision. RankFit makes
those local failures visible. It complements rather than replaces AUC,
Precision@k, Recall@k, Lift@k, calibration, and application-specific utility
metrics.

## Installation

```bash
pip install rankfit
```

## Quick start

```python
import numpy as np

from rankfit import RankFitAnalyzer

rng = np.random.default_rng(7)
n = 20_000
scores = rng.random(n)
order = np.argsort(-scores)
decile_size = n // 10

# The second-ranked decile has a higher true event rate than the first.
true_rates = [
    0.40,
    0.65,
    0.30,
    0.12,
    0.05,
    0.02,
    0.01,
    0.005,
    0.003,
    0.002,
]

y_true = np.zeros(n, dtype=int)
for i, rate in enumerate(true_rates):
    idx = order[i * decile_size : (i + 1) * decile_size]
    y_true[idx] = (rng.random(len(idx)) < rate).astype(int)

analyzer = RankFitAnalyzer(n_bins=10)
results = analyzer.calculate_metrics(scores, y_true)

print(analyzer.summary(results))
fig = analyzer.plot_analysis(results, title="Patient outreach ranking")
```

Use an independent validation or test sample. Evaluating a fitted model on its
training data can make every performance diagnostic look more favorable than it
will be in production.

## Focus on the region where action is taken

If the team can act on only the top 10%, configure that capacity directly:

```python
analyzer = RankFitAnalyzer(n_bins=20, action_fraction=0.10)
results = analyzer.calculate_metrics(scores, y_true)

print(results["lift_at_action"])
print(results["action_rankfit_v"])
```

Precision, recall, and lift are calculated on the exact highest-scored 10%.
The focused RankFit diagnostics cover the score bins touched by that action
region and the first adjacent bin below the cutoff. This makes a failure at the
selection boundary visible without replacing the operational metric.

When a cutoff falls inside a tied score group, RankFit warns that the exact
top-fraction result depends on the tie-breaking policy.

## Compare many candidate models

```python
comparison = analyzer.compare_models(
    {
        "logistic": logistic_scores,
        "random_forest": forest_scores,
        "gradient_boosting": boosting_scores,
    },
    y_true,
)
print(comparison)
```

The comparison is ordered first by lift at the action fraction, then by focused
RankFit-V and focused RankFit-T. This keeps operational yield primary and uses
RankFit as a transparent diagnostic tie-breaker instead of blending unrelated
measures into an arbitrary weighted score.

## What the metrics mean

### RankFit-V

RankFit-V penalizes adjacent increases in observed event rates. Bins are ordered
from the highest predicted score to the lowest, so an increase from bin `i` to
bin `i + 1` is a local ranking inversion.

```text
RankFit-V = 1 - sum(penalized adjacent increases) / observed event-rate range
```

The score is clipped to `[0, 1]`. Higher is better. With significance filtering
enabled, locally significant inversions are penalized. If a separate global
adverse-trend test confirms that the complete ranking runs in the wrong
direction, all observed inversion magnitudes are penalized without being
relabeled as individually significant.

`rankfit_v_raw` reports the same magnitude calculation over every observed
inversion, without significance filtering. It is still affected by sampling
noise and should not be compared across datasets with very different event
counts per bin.

### RankFit-T

RankFit-T rescales Kendall's tau to `[0, 1]`:

- `1.0`: perfectly decreasing event rates
- `0.5`: no consistent directional trend
- `0.0`: perfectly increasing event rates

RankFit-T describes the global trend across bins. RankFit-V focuses on adjacent
local inversions. The two metrics answer different questions and should be read
together.

## Statistical filtering

By default, RankFit uses a one-sided Fisher exact test for every planned
adjacent-bin comparison. Benjamini-Hochberg adjustment is applied across the
complete family of `B - 1` tests before observed inversions are flagged.

```python
analyzer = RankFitAnalyzer(
    n_bins=10,
    use_significance=True,
    alpha=0.05,
    fdr=True,
)
```

Benjamini-Hochberg FDR guarantees require independence or an appropriate form
of positive dependence. Adjacent-bin tests share bins, so applications requiring
formal guarantees under arbitrary dependence should consider a more conservative
procedure or application-specific simulation.

Failure to confirm an inversion is not proof that the ordering is correct. Sparse
events or very small bins can make adjacent comparisons unstable or insensitive.
RankFit warns when bin counts warrant extra caution.

## Result fields

`calculate_metrics()` returns:

| Field | Meaning |
| --- | --- |
| `rankfit_v` | Evidence-aware adjacent-inversion score |
| `rankfit_v_raw` | Magnitude score over all observed inversions |
| `rankfit_t` | Rescaled Kendall trend score |
| `violations` | Locally reported violations; BH-significant by default |
| `raw_violations` | Every observed adjacent increase |
| `penalty_violations` | Inversions whose magnitudes contribute to RankFit-V |
| `violation_details` | Raw and adjusted p-values for observed inversions |
| `global_inversion` | Whether the overall ranking significantly trends adversely |
| `adverse_trend_p` | One-sided global adverse-trend p-value |
| `bin_stats` | Counts, positives, event rates, and mean scores by bin |
| `auc` | ROC-AUC when both outcome classes are present |
| `n_bins_effective` | Number of populated bins actually produced |
| `precision_at_action` | Positive rate within the exact action fraction |
| `recall_at_action` | Share of all positives captured in the action fraction |
| `lift_at_action` | Precision at the action fraction divided by base rate |
| `action_rankfit_v` | Focused violation score through the cutoff boundary |
| `action_rankfit_t` | Focused trend score through the cutoff boundary |
| `action_violations` | Penalized inversions affecting the action region |

## Choosing granularity

Deciles are the default. Fewer bins provide more observations per comparison but
can smooth over localized problems. More bins provide finer resolution but noisier
event-rate estimates.

```python
comparison, fig = analyzer.compare_granularities(
    scores,
    y_true,
    bins=[5, 10, 20],
)
```

Avoid selecting the most favorable granularity after inspecting the labels. For
confirmatory use, specify the binning plan and test family in advance.

## Interpretation

The labels printed by `summary()` are descriptive aids, not universal approval
thresholds. Appropriate tolerances depend on prevalence, sample size, intervention
capacity, error costs, and the portion of the ranking used operationally.

RankFit should be interpreted alongside:

- ROC-AUC or PR-AUC
- Precision@k, Recall@k, or Lift@k
- Calibration and reliability plots
- Event counts and confidence intervals by bin
- Application-specific utility, fairness, and stability checks

Evaluate models not only by summary metrics and lift charts, but also by how
reliably they rank the cases where action is taken.

## Limitations

- Results depend on the chosen binning granularity.
- Observed bin rates are noisy when events are sparse.
- RankFit-V normalizes by the observed event-rate range, which can amplify noise
  when that range is very small.
- RankFit does not establish causal effects or determine the correct action cutoff.
- A synthetic example demonstrates a possible failure mode, not its prevalence in
  production systems.
- Repeated testing across many datasets, subgroups, or granularities creates a
  larger multiplicity problem than the within-analysis adjustment addresses.

## Development

```bash
python -m pip install -e ".[dev]"
python -m pytest
python -m ruff check rankfit tests
python -m black --check rankfit tests
python -m build
python -m twine check dist/*
```

## License

MIT License. See [LICENSE](LICENSE).
