Metadata-Version: 2.4
Name: metricsifter
Version: 0.2.1
Summary: Feature Reduction for Multivariate Time Series Data
Author: yuuki
License-Expression: BSD-3-Clause
Project-URL: Homepage, https://github.com/ai4sre/metricsifter
Project-URL: Bug Tracker, https://github.com/ai4sre/metricsifter/issues
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Operating System :: OS Independent
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy
Requires-Dist: pandas
Requires-Dist: scipy
Requires-Dist: joblib
Requires-Dist: ruptures
Requires-Dist: statsmodels
Provides-Extra: viz
Requires-Dist: matplotlib>=3.7; extra == "viz"
Provides-Extra: dev
Requires-Dist: black; extra == "dev"
Requires-Dist: ruff; extra == "dev"
Requires-Dist: coverage; extra == "dev"
Requires-Dist: packaging; extra == "dev"
Requires-Dist: pytest; extra == "dev"
Requires-Dist: setuptools; extra == "dev"
Requires-Dist: matplotlib>=3.7; extra == "dev"
Requires-Dist: scikit-learn>=1.5; extra == "dev"
Requires-Dist: tomli; python_version < "3.11" and extra == "dev"
Provides-Extra: experiments
Requires-Dist: numpy<2; python_version < "3.12" and extra == "experiments"
Requires-Dist: pandas-stubs; python_version < "3.12" and extra == "experiments"
Requires-Dist: scikit-learn<1.2,>=0.24; python_version < "3.12" and extra == "experiments"
Requires-Dist: hdbscan; python_version < "3.12" and extra == "experiments"
Requires-Dist: networkx; python_version < "3.12" and extra == "experiments"
Requires-Dist: threadpoolctl; python_version < "3.12" and extra == "experiments"
Dynamic: license-file

# MetricSifter: Feature Reduction of Multivariate Time Series Data for Efficient Fault Localization in Cloud Applications
![CI workflow](https://github.com/ai4sre/metricsifter/actions/workflows/ci.yaml/badge.svg)

This repository contains code and datasets used in the experiments described in our paper [1].

- [1]: Yuuki Tsubouchi, Hirofumi Tsuruta, ["MetricSifter: Feature Reduction of Multivariate Time Series Data for Efficient Fault Localization in Cloud Applications"](https://doi.org/10.1109/ACCESS.2024.3374334), IEEE Access (ACCESS) , Vol. 12, pp. 37398-37417, March 2024.

## Introduction

MetricSifter is a feature reduction framework designed to accurately identify anomalous metrics caused by faults for enhancing fault localization. Our key insight is that the change point times inside the failure duration are close to each other for the failure-related metrics. MetricSifter detects change points per metric, localizes the time frame with the highest change point density, and excludes metrics with no change points in that time frame. The offline change point detection is implemented by [ruptures](https://github.com/deepcharles/ruptures), and the segmentation of the detected change points is based on kernel density estimation (KDE).

## Installation

### Prerequisites

If you want to use `uv` (recommended for faster installation), install it first:

```bash
# Install uv
curl -LsSf https://astral.sh/uv/install.sh | sh

# Or using pip
pip install uv
```

### From PyPI

You can install `metricsifter` package from PyPI:

```bash
# Using pip
pip install metricsifter

# Using uv (recommended for faster installation)
uv pip install metricsifter
```

### For Development

**Note**: The core package supports Python 3.10-3.14.

```bash
# Clone the repository
git clone https://github.com/ai4sre/metricsifter.git
cd metricsifter

# Install the current checkout and development dependencies with uv (recommended)
uv sync --extra dev

# Or using pip
pip install -e ".[dev]"
```

**For running experiments** (use a separate Python 3.10 environment):

The pinned PyRCA revision requires scikit-learn 0.x or 1.0-1.1, whereas the development
environment tests against current scikit-learn releases. The two extras are therefore
intentionally incompatible and must not be installed into the same environment. From the
repository root, create a separate Python 3.10 environment, install the current checkout
with the experiment dependencies, then install the pinned PyRCA revision:

```bash
# Keep experiment dependencies separate from the standard .venv
uv venv --python 3.10 .venv-experiments
uv pip install --python .venv-experiments/bin/python -e ".[experiments]"

# Install the pinned sfr-pyrca revision
uv pip install --python .venv-experiments/bin/python \
  "sfr-pyrca @ git+https://github.com/salesforce/PyRCA@d85512b"
```

## Getting Started

```python
import numpy as np
import pandas as pd

from metricsifter import Sifter

## Create synthetic time series data:
## - 3 failure-related metrics with a level shift around t=60
## - 1 unrelated metric with a level shift at a different time (t=20)
## - 6 flat (no-change) metrics
rng = np.random.default_rng(0)
length = 80
data = {}
for i in range(3):
    data[f"failure_related_{i}"] = np.concatenate(
        [rng.normal(0, 0.1, 60), rng.normal(5, 0.1, 20)]
    )
data["unrelated"] = np.concatenate([rng.normal(0, 0.1, 20), rng.normal(3, 0.1, 60)])
for i in range(6):
    data[f"flat_{i}"] = np.full(length, float(i))
data = pd.DataFrame(data)

## Remove the metrics unrelated to the failure
sifter = Sifter(penalty_adjust=2.0, n_jobs=1)
sifted_data = sifter.run(data=data)
print("(#removed metrics) / (#total metrics):", len(set(data.columns) - set(sifted_data.columns)), "/", len(data.columns))
print("remained metrics:", list(sifted_data.columns))
```

The example of original synthetic data and its sifted data is shown in the following figure.

### Before
<img src="./docs/images/original_time_series.png" width="600" height="480">

### After
<img src="./docs/images/sifted_time_series.png" width="600" height="360">


## Diagnostic Report

`Sifter.run()` returns only the filtered metrics. When you need to know **why** each
metric was kept or dropped (for debugging, calibration, or handing the result to an
LLM agent), use `Sifter.sift()`, which returns a `SiftResult`:

```python
from metricsifter import Sifter

result = Sifter(penalty_adjust=2.0, n_jobs=1).sift(data=data)

# The filtered DataFrame (same as run())
result.data

# Why each metric was dropped (three mutually-exclusive reasons)
result.filtered_no_change          # removed by the no-variation filter
result.filtered_no_change_points   # no change point detected
result.filtered_out_of_segment     # change point outside the densest segment
result.selected_metrics            # metrics that were kept

# Per-metric change points (row positions) and every candidate segment with its score
result.metric_to_change_points     # {"failure_related_0": [60], ...}
result.segments                    # list of SegmentInfo (label, metrics, index range, score)
result.selected_segment            # the chosen densest segment

# JSON serialization for LLM agents / MCP tools (the DataFrame is not included)
print(result.to_json(indent=2))
```

If the input DataFrame has a `DatetimeIndex`, change points and segments are additionally
expressed as wall-clock timestamps (`result.metric_to_change_times`,
`SegmentInfo.start_time` / `end_time`). Irregular (non-uniform) sampling is supported,
since positions are converted to times purely by index lookup. `run()` and
`run_with_selected_segment()` are unchanged and fully backward compatible.

## Algorithm Tuning

Several optional knobs let you adapt the pipeline to your data. All are backward
compatible: the defaults reproduce the original behavior exactly.

**Fully automatic tuning (`penalty_adjust="auto"` / `bandwidth="auto"`).** Both
main hyperparameters can be chosen from the data by stability selection:

- `penalty_adjust="auto"` sweeps the penalty multiplier over a geometric grid
  (one change-point `fit` per metric, then cheap re-`predict` per candidate) and
  picks the midpoint of the widest *plateau* -- the range of multipliers over
  which the detected change points barely move. A stable plateau sits away from
  both the over-segmentation regime (small multipliers) and the
  missed-detection regime (large multipliers). Deterministic, no randomness.
- `bandwidth="auto"` bootstrap-resamples the metrics (change points stay fixed,
  so only the cheap KDE segmentation reruns) and picks the bandwidth whose
  final `selected_metrics` is the most reproducible across resamples. Only
  bandwidths that still split the data into at least two segments compete, so
  the degenerate "one giant segment" solution can never win. Seed it with
  `random_state` for reproducibility.

The chosen values and their diagnostics are reported in
`SiftResult.penalty_tuning` / `SiftResult.bandwidth_tuning` (also included in
`to_json()` and the CLI `--report`).

```python
result = Sifter(penalty_adjust="auto", bandwidth="auto", random_state=0, n_jobs=1).sift(data)
print(result.penalty_tuning.resolved, result.penalty_tuning.reason)    # e.g. 2.0 plateau
print(result.bandwidth_tuning.resolved, result.bandwidth_tuning.reason)  # e.g. 1.5 stability
```

**Robust penalty (`sigma_estimator`).** The change-point penalty scales with an
estimate of the noise scale `sigma`. The default `"std"` uses the global standard
deviation, which a strong trend, a large level shift, or outliers can inflate --
making the penalty too strict and causing **missed** change points. Two robust
alternatives fix this:

- `"mad"` -- Median Absolute Deviation (`1.4826 * median(|x - median(x)|)`), robust
  to a minority of spikes/outliers. Prefer it for spiky metrics.
- `"diff_std"` -- standard deviation of the first difference divided by `sqrt(2)`,
  which cancels any trend or level shift. Prefer it for trending metrics.

```python
# Estimate the noise floor from the first difference (trend-independent)
sifter = Sifter(sigma_estimator="diff_std", n_jobs=1)
sifted = sifter.run(data)
```

**KDE bandwidth auto-estimation (`bandwidth`).** Instead of the fixed default of
`2.5`, pass `"scott"` or `"silverman"` to derive the bandwidth from the
change-point distribution (via statsmodels). A float is still accepted; an invalid
string raises `ValueError`.

```python
sifter = Sifter(bandwidth="scott", n_jobs=1)
sifted = sifter.run(data)
```

**Custom segment-selection strategy (`segment_selection_method`).** Besides the
built-in `"max"` / `"weighted_max"`, pass any `Callable[[SegmentCandidate], float]`;
the segment with the highest score is selected. `SegmentCandidate` exposes
`label`, `metrics`, `change_points`, and `metric_to_cps`.

```python
from metricsifter import SegmentCandidate

def widest_segment(candidate: SegmentCandidate) -> float:
    if not candidate.change_points:
        return 0.0
    return float(max(candidate.change_points) - min(candidate.change_points))

result = Sifter(segment_selection_method=widest_segment, n_jobs=1).sift(data)
```

**Evaluating a selection (`evaluate_selection`).** A dependency-free helper to
score the kept metrics against a known ground truth -- handy for tuning the knobs
above or guarding against regressions in CI. Ratios with a zero denominator are
defined as `0.0`.

```python
from metricsifter import evaluate_selection

result = Sifter(n_jobs=1).sift(data)
metrics = evaluate_selection(
    selected=result.selected_metrics,
    ground_truth={"failure_related_0", "failure_related_1", "failure_related_2"},
    all_metrics=set(data.columns),  # optional: enables reduction_ratio
)
print(metrics.precision, metrics.recall, metrics.f1, metrics.reduction_ratio)
```

## Visualization

Plotting helpers live in `metricsifter.plot` and depend on **matplotlib**, which is an
optional extra (install with `pip install 'metricsifter[viz]'`). Importing the module
without matplotlib raises a clear error pointing at that command; the core install stays
matplotlib-free.

```python
from metricsifter import Sifter
from metricsifter import plot

result = Sifter(penalty_adjust=2.0, n_jobs=1).sift(data=data)

# Before/after time series on stacked panels, with change-point markers and the
# selected-segment band. Returns a matplotlib Figure.
fig = plot.plot_sifted_metrics(result, original_data=data)
fig.savefig("sifted.png")

# Change-point lag plot with segment boundaries and the internal KDE density curve.
# Returns a matplotlib Axes.
ax = plot.plot_change_point_density(result, time_series_length=len(data), kde_bandwidth=2.5)
```

## scikit-learn Pipeline

`SifterTransformer` exposes the sift as a scikit-learn-style transformer **without adding a
dependency on scikit-learn** (the estimator API is duck-typed). `fit` runs the sift and
remembers the selected columns; `transform` returns those columns from any DataFrame with a
matching schema (a missing column raises a clear `ValueError`).

```python
from metricsifter import SifterTransformer

tr = SifterTransformer(penalty_adjust=2.0, n_jobs=1)
reduced = tr.fit_transform(data)     # -> DataFrame of the selected metrics
tr.selected_metrics_                  # columns chosen at fit time
tr.result_                            # the full SiftResult from the fit

# When scikit-learn is installed, it drops into a Pipeline and survives clone():
from sklearn.pipeline import Pipeline
pipe = Pipeline([("sift", SifterTransformer(penalty_adjust=2.0, n_jobs=1))])
reduced = pipe.fit_transform(data)
```

## Prometheus

`metricsifter.adapters.prometheus` converts a parsed Prometheus `query_range` response
(`resultType == "matrix"`) into a wide DataFrame ready for `sift()`. It performs no HTTP:
fetch the payload yourself and pass the parsed dict. Series with mismatched timestamps are
outer-joined (missing samples become `NaN`, handled by the sift NaN support), and each column
keeps a reverse mapping back to its original Prometheus labels.

```python
from metricsifter.adapters import prometheus

# `response` is the parsed JSON of GET /api/v1/query_range
df = prometheus.from_query_range(response)     # DatetimeIndex (UTC), one column per series
labels = prometheus.to_metric_labels(df, df.columns[0])  # {"__name__": "...", "job": "...", ...}

from metricsifter import Sifter
result = Sifter(n_jobs=1).sift(df)
```

## CLI

Installing the package provides a `metricsifter` command (stdlib `argparse` only):

```bash
# Read a CSV of time series, write the sifted metrics, and dump a diagnostic report.
metricsifter run input.csv --output sifted.csv --report report.json \
    --penalty-adjust 2.0 --bandwidth 2.5 --search-method pelt --n-jobs 1

# With no --output, the sifted CSV is written to stdout.
# By default every column is treated as a metric; pass --index-col when the
# CSV has a time/index column.
metricsifter run input.csv --index-col 0 --parse-dates

# Auto-tune both hyperparameters by stability selection (see Algorithm Tuning);
# the chosen values land in the --report JSON under penalty_tuning / bandwidth_tuning.
metricsifter run input.csv --penalty-adjust auto --bandwidth auto --random-state 0 --report report.json
```

Exit codes: `0` on success, `2` on input errors (missing/empty/unparseable CSV, or bad
arguments).

## Agent Integration

[agent-metricsifter](https://github.com/ai4sre/agent-metricsifter) provides Claude Code Agent Skills that combine MetricSifter with [mcp-grafana](https://github.com/grafana/mcp-grafana) for interactive incident investigation. It enables automated Prometheus metrics filtering, Grafana dashboard creation, and human-in-the-loop parameter calibration.

## For Developers

### Setup Development Environment

```bash
# Using uv (recommended)
uv sync --extra dev

# Or using pip
pip install -e ".[dev]"

# Experiments use a separate Python 3.10 environment; see experiments/README.md.
uv venv --python 3.10 .venv-experiments
uv pip install --python .venv-experiments/bin/python -r experiments/requirements.txt
```

### Run Tests

```bash
# The standard suite does not require the optional PyRCA Git dependency.
uv run pytest -s -vv tests
```

CI runs on every push and pull request. It executes the complete `tests` suite on
Python 3.10, 3.11, 3.12, 3.13, and 3.14. A separate quality job verifies the lock file,
Ruff, and Black, and the package job builds a wheel and tests its import and CLI from a
fresh environment outside the source checkout.

### Code Quality

```bash
# Verify the same quality gates as CI
uv lock --check
uv run ruff check .
uv run black --check .

# Apply the formatter locally
uv run black .

# Build and inspect the distributions before release
uv build
```

### Code Complexity

CI measures Cognitive and Cyclomatic Complexity with
[cccc](https://github.com/moznion/cccc) on pull requests and pushes to `main`
(`.github/workflows/complexity.yml`) and tracks them with
[octocov](https://github.com/k1LoW/octocov) as custom metrics: the number of
functions and the total, maximum, median and 90th percentile of each score. A
pull request gets a comment comparing them with `main`, each push to `main`
stores a report in the `octocov` branch, and the job summary ranks the 20 most
complex functions. Nothing fails on a score: the numbers point at hotspots and
show trends, such as a component taking on too much, instead of being a target.
`cccc.toml` lists what is not analyzed: the tests and `experiments/`. To check
locally, install cccc (a [release
binary](https://github.com/moznion/cccc/releases) or `cargo install cccc-cli`)
and run `cccc --table --top-cognitive 20 .` from the repository root, which
ranks the most complex functions.

### Publishing to PyPI

This package uses GitHub Actions to publish to PyPI when a `v*` tag is pushed. The
release workflow publishes only after its tests, Ruff, Black, lock-file check,
tag/version check, build, and fresh-environment wheel smoke test have succeeded.

#### Publishing Process

1. **Update and lock the version**
   ```bash
   # Edit the project version in pyproject.toml to X.Y.Z, then refresh the lock.
   uv lock
   ```

2. **Run the publication gates locally**
   ```bash
   uv run pytest -s -vv tests
   uv run ruff check .
   uv run black --check .
   uv lock --check
   uv build

   # Install only the built wheel into a fresh environment and test it outside the checkout.
   WHEEL_PATH="$PWD/dist/metricsifter-X.Y.Z-py3-none-any.whl"
   SMOKE_DIR="$(mktemp -d)"
   uv venv --python 3.11 "$SMOKE_DIR/.venv"
   uv pip install --python "$SMOKE_DIR/.venv/bin/python" "$WHEEL_PATH"
   (
     cd "$SMOKE_DIR"
     "$SMOKE_DIR/.venv/bin/python" -c \
       'import metricsifter; assert metricsifter.__version__ == "X.Y.Z"'
     "$SMOKE_DIR/.venv/bin/metricsifter" --help
   )
   ```

3. **Commit and tag the release**
   ```bash
   git add pyproject.toml uv.lock
   git commit -m "Bump version to X.Y.Z"
   git tag vX.Y.Z
   git push origin main
   git push origin vX.Y.Z
   ```

   The tag without its leading `v` must exactly match `project.version`; otherwise the
   workflow stops before publication.

4. **Automatic publication**

   The workflow validates and builds the distribution once, smoke-tests that wheel,
   and publishes the validated artifacts to PyPI through Trusted Publishing.

#### Setup Requirements

For the workflow to work, configure Trusted Publishing in PyPI:

1. Go to [PyPI](https://pypi.org/)
2. Create/login to your account
3. Go to your account settings → Publishing
4. Add a new Trusted Publisher with:
   - **PyPI project name**: `metricsifter`
   - **Owner**: `ai4sre`
   - **Repository name**: `metricsifter`
   - **Workflow name**: `publish.yaml`
   - **Environment name**: `metricsifter_pypi`

Note: Trusted Publishing uses OpenID Connect (OIDC) and doesn't require manual API tokens.

#### Local Build Testing

To test the build locally before publishing:

```bash
# Build the package
uv build

# The built files will be in the dist/ directory:
# - metricsifter-X.Y.Z.tar.gz (source distribution)
# - metricsifter-X.Y.Z-py3-none-any.whl (wheel)
```

## License

[BSD-3-Clause](LICENSE)
