Metadata-Version: 2.4
Name: feature_pruning
Version: 0.1.0
Summary: High-performance PySpark library for scalable feature reduction and Information Value (IV) pruning in Credit Risk Scorecards
Author-email: Vrukshya <vrukshyaai@gmail.com>
Maintainer-email: Vrukshya <vrukshyaai@gmail.com>
License: MIT License
        
        Copyright (c) 2026 Vrukshya Org
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
Project-URL: Homepage, https://github.com/vrukshya/feature_pruning
Project-URL: Documentation, https://github.com/vrukshya/feature_pruning#readme
Project-URL: Repository, https://github.com/vrukshya/feature_pruning
Project-URL: Bug Tracker, https://github.com/vrukshya/feature_pruning/issues
Keywords: credit-risk,credit-scoring,scorecard,feature-reduction,feature-pruning,feature-selection,information-value,woe,pyspark,fintech,machine-learning
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Financial and Insurance Industry
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Office/Business :: Financial
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE.txt
Requires-Dist: numpy>=1.20.0
Requires-Dist: pandas>=1.3.0
Requires-Dist: pyspark>=3.1.0
Provides-Extra: dev
Requires-Dist: pytest>=7.0.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
Requires-Dist: black>=23.0.0; extra == "dev"
Requires-Dist: flake8>=6.0.0; extra == "dev"
Requires-Dist: mypy>=1.0.0; extra == "dev"
Dynamic: license-file

# feature_pruning

[![Python Version](https://img.shields.io/badge/python-3.8%20%7C%203.9%20%7C%203.10%20%7C%203.11%20%7C%203.12-blue.svg)](https://pypi.org/project/feature-pruning/)
[![PySpark](https://img.shields.io/badge/PySpark-%3E%3D3.1.0-orange.svg)](https://spark.apache.org/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)

**High-Performance Distributed Feature Reduction Engine for Credit Risk Scorecards.**

`feature_pruning` is an enterprise-grade Python library built natively on Apache Spark for selecting and pruning variables in credit risk scorecard development (Probability of Default / Basel II/III / IFRS 9 / Retail Scorecards).

It automates the transition from thousands of raw credit bureau, transaction, and demographic attributes down to an optimal, highly predictive, non-collinear feature set ready for Weight of Evidence (WoE) binning and Logistic Regression.

---

## The Credit Risk Scorecard Challenge

Building regulatory-compliant credit scorecards presents unique data engineering and modeling hurdles:

- **Audit & Governance (SR 11-7 / Basel / IFRS 9)**: Model risk management (MRM) and regulatory auditors require an explicit justification for every discarded or retained variable.
- **Extreme Multicollinearity**: Credit bureau tables often contain dozens of collinear metrics (e.g., `num_inquiries_3m`, `num_inquiries_6m`, `num_inquiries_12m`). In standard logistic regression scorecards, collinearity causes unstable coefficients and counter-intuitive sign reversals.
- **Predictive Quality**: Features must meet minimum **Information Value (IV)** standards while preserving mandatory business or regulatory key indicators.
- **Big Data Scale**: Modern credit datasets often span millions of accounts and thousands of features. Single-machine libraries (pandas/scikit-learn) crash with `OutOfMemory` errors when computing quantile cuts and pairwise correlations.

`feature_pruning` solves these challenges by running distributed quantile binning, IV evaluation, and correlation pruning entirely within **Apache Spark**.

---

## Pipeline Architecture

```
                           Raw Spark DataFrame (Millions of Rows, 1000s of Features)
                                                   │
                                                   ▼
┌────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Stage 1: Feature Classification                                                                        │
│ • Detects Numeric vs. Categorical vs. Datetime columns                                                 │
│ • Flags & filters high-cardinality strings (> max_cat_levels)                                          │
│ • Protects mandatory features and excludes requested columns                                          │
└──────────────────────────────────────────────────┬─────────────────────────────────────────────────────┘
                                                   │
                                                   ▼
┌────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Stage 2: Distributed Precision Downcasting                                                             │
│ • Downcasts DoubleType & DecimalType to 32-bit FloatType                                              │
│ • Cuts executor memory consumption by ~50% during matrix aggregation                                  │
└──────────────────────────────────────────────────┬─────────────────────────────────────────────────────┘
                                                   │
                                                   ▼
┌────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Stage 3: High-Throughput Information Value (IV) Computation                                            │
│ • Computes continuous quantiles and distributed binning via Spark mapInPandas                         │
│ • Aggregates Goods/Bads and calculates IV for numeric and categorical variables                        │
│ • Filters out variables with IV < iv_threshold                                                         │
└──────────────────────────────────────────────────┬─────────────────────────────────────────────────────┘
                                                   │
                                                   ▼
┌────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Stage 4: IV-Prioritized Correlation Pruning                                                            │
│ • Computes Pearson correlation matrix via PySpark VectorAssembler + Correlation                        │
│ • Between collinear pairs (r >= corr_threshold), retains the feature with higher IV                    │
│ • Dynamic threshold relaxation ensures minimum required feature count is met                          │
└──────────────────────────────────────────────────┬─────────────────────────────────────────────────────┘
                                                   │
                                                   ▼
┌────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Stage 5: Full Audit Reporting & Data Delivery                                                          │
│ • Generates comprehensive audit DataFrame (is_selected, iv, correlated_with, exclusion_reason)         │
│ • Outputs pruned Spark DataFrame ready for Weight of Evidence (WoE) binning                           │
└────────────────────────────────────────────────────────────────────────────────────────────────────────┘
```

---

## Key Features

- ⚡ **Native PySpark Scalability**: Distributed quantile histograms and correlation matrices executed directly on cluster workers via Spark ML and `mapInPandas`.
- 📊 **IV-Driven Pruning**: When two features are collinear, the pipeline drops the weaker predictor and retains the variable with higher Information Value.
- 🛡️ **Mandatory Variable Protection**: Ensure business-critical variables (e.g., debt-to-income, credit bureau score) are never removed, regardless of their statistical properties.
- 📋 **Regulatory Audit Trail**: Automatically produces a full governance table explaining why every single column was accepted or eliminated (e.g., `low_iv (0.012)`, `correlated (r=0.962)`, `high_cardinality (>200)`, or `datetime_column`).
- 🔄 **Dynamic Threshold Relaxation**: Automatically adjusts correlation thresholds if filtering becomes overly aggressive, keeping feature counts within target bounds (`expected_final_min`, `expected_final_max`).

---

## Installation

### From PyPI
```bash
pip install feature-pruning
```

### For Local Development
```bash
git clone https://github.com/vrukshya/feature_pruning.git
cd feature_pruning
pip install -e ".[dev]"
```

### In Databricks / Cloud Notebooks
In your Databricks notebook cell:
```python
%pip install feature-pruning
```
Or add `feature-pruning` to your Databricks cluster libraries.

---

## Quickstart

```python
from pyspark.sql import SparkSession
from feature_pruning import FeatureSelectionPipeline

# 1. Initialize Spark session (or use active session in Databricks/EMR)
spark = SparkSession.builder.appName("CreditRiskScorecard").getOrCreate()

# 2. Load credit training data
df = spark.table("risk_catalog.credit_data.application_train")

# 3. Configure the feature selection pipeline
pipeline = FeatureSelectionPipeline(
    df=df,
    target_col="default_flag",              # 0 = Good loan, 1 = Default / Bad loan
    mandatory_features=["bureau_score", "dti_ratio"],  # Keep regardless of correlation
    exclude_features=["application_id", "ssn_hash"],   # Exclude identifiers
    iv_threshold=0.03,                      # Industry baseline: IV >= 0.03
    corr_threshold=0.95,                    # Multicollinearity cutoff
    expected_final_min=20,
    expected_final_max=150,
    verbose=True,
)

# 4. Execute pipeline
selected_spark_df, audit_report = pipeline.run()

# 5. Review results
print("Selected features count:", len(pipeline.get_selected_columns()))
print("Selected columns:", pipeline.get_selected_columns())

# 6. Inspect audit report
print(audit_report.head(20))
```

---

## Inspection & Diagnostic Methods

After calling `pipeline.run()`, several diagnostic helpers allow inspection of the feature reduction decisions:

```python
# 1. Retrieve list of final selected feature names
selected_features = pipeline.get_selected_columns()

# 2. Inspect Information Value ranking for all evaluated features
iv_summary = pipeline.get_iv_summary()
print(iv_summary.head(10))

# 3. Inspect which features were pruned due to correlation and their collinear counterpart
correlation_drops = pipeline.get_correlation_drops()
print(correlation_drops.head(10))

# 4. Full audit report with exact exclusion reasons
print(pipeline.feature_report_)
```

### Sample Audit Report Output

| feature | dtype | iv | correlated_with | correlation_value | is_selected | exclusion_reason |
|---|---|---|---|---|---|---|
| `bureau_score` | numeric | 0.4521 | None | None | True | |
| `utilization_rate`| numeric | 0.3180 | None | None | True | |
| `num_inquiries_6m` | numeric | 0.1420 | None | None | True | |
| `num_inquiries_3m` | numeric | 0.1210 | `num_inquiries_6m`| 0.965 | False | correlated (r=0.965) |
| `employer_name` | categorical | None | None | None | False | high_cardinality (>200) |
| `postal_code_raw`| numeric | 0.0120 | None | None | False | low_iv (0.0120) |
| `application_date`| datetime | None | None | None | False | datetime_column |

---

## API Reference

### `FeatureSelectionPipeline`

```python
FeatureSelectionPipeline(
    df: SparkDataFrame,
    target_col: str,
    mandatory_features: Optional[List[str]] = None,
    exclude_features: Optional[List[str]] = None,
    iv_threshold: float = 0.03,
    iv_n_bins: int = 100,
    iv_sample_rows: int = 1_000_000,
    corr_threshold: float = 0.95,
    corr_sample_rows: int = 500_000,
    max_corr_features: int = 5000,
    max_cat_levels: int = 200,
    expected_final_min: int = 200,
    expected_final_max: int = 1000,
    downcast: bool = True,
    verbose: bool = True,
)
```

| Parameter | Type | Default | Description |
|---|---|---|---|
| `df` | `SparkDataFrame` | *Required* | Input Spark DataFrame containing raw candidate variables and target. |
| `target_col` | `str` | *Required* | Name of the binary target (0 = Good, 1 = Bad). Case-insensitive. |
| `mandatory_features` | `List[str]` | `None` | Columns guaranteed to be retained regardless of IV or correlation. |
| `exclude_features` | `List[str]` | `None` | Columns explicitly excluded from candidate pool (IDs, timestamps). |
| `iv_threshold` | `float` | `0.03` | Minimum Information Value (IV) required for feature inclusion. |
| `iv_n_bins` | `int` | `100` | Quantile bin resolution for continuous variable histogram computation. |
| `iv_sample_rows` | `int` | `1,000,000` | Sample ceiling for computing quantile thresholds on continuous columns. |
| `corr_threshold` | `float` | `0.95` | Pearson correlation ceiling. Collinear variable with lower IV is pruned. |
| `corr_sample_rows` | `int` | `500,000` | Sample ceiling for Pearson correlation matrix calculation. |
| `max_corr_features` | `int` | `5000` | Upper limit of top IV features fed into correlation assembler. |
| `max_cat_levels` | `int` | `200` | Maximum unique levels allowed before categorical is dropped as high-cardinality. |
| `expected_final_min` | `int` | `200` | Minimum retained count. Relaxes `corr_threshold` if pruned too aggressively. |
| `expected_final_max` | `int` | `1000` | Maximum cap on final retained variables. |
| `downcast` | `bool` | `True` | Downcasts Decimal/Double columns to Float32 to optimize Spark worker memory. |
| `verbose` | `bool` | `True` | Prints stage progress and execution timing logs. |

---

## Credit Risk Scorecard Rules of Thumb

### Information Value (IV) Benchmarks

In credit risk modeling (Siddiqi, 2005), Information Value serves as the primary metric for filtering out uninformative signals:

| Information Value (IV) | Predictive Power | Action in Scorecard Development |
|---|---|---|
| **< 0.02** | Unpredictive | **Drop**: Adds noise and degrees of freedom without signal. |
| **0.02 – 0.10** | Weak Predictor | **Evaluate**: May be retained if part of a key credit policy dimension. |
| **0.10 – 0.30** | Medium Predictor | **Keep**: Core candidate for scorecard inclusion. |
| **0.30 – 0.50** | Strong Predictor | **Keep**: High diagnostic quality feature. |
| **> 0.50** | Suspicious / Too Good | **Investigate**: Often indicative of target leakage or operational bias. |

### Correlation Thresholds

- Standard practice sets the correlation threshold between `0.80` and `0.95`.
- Setting `corr_threshold=0.95` catches near-duplicate metrics (e.g., balance in dollars vs. balance in thousands).
- Setting `corr_threshold=0.85` produces a tighter, more orthogonal set of features that prevents variance inflation in final logistic regression models.

---

## Performance & Spark Optimization

- **Quantile Binning (`mapInPandas`)**: Rather than running expensive full-dataset sorting on each column, `feature_pruning` samples continuous variables to establish robust quantile boundaries, then computes distributed frequency histograms across partitions in a single pass.
- **Spark Storage Management**: Intermediate working DataFrames are cached at `StorageLevel.MEMORY_AND_DISK` and unpersisted automatically at the conclusion of report generation.
- **Vector Correlation**: Correlation is calculated using Spark ML's native distributed linear algebra (`Correlation.corr`), supporting thousands of features simultaneously.

---

## Contributing

Contributions, bug reports, and feature requests are welcome!
Please feel free to submit a pull request or open an issue on GitHub.

1. Fork the Project
2. Create your Feature Branch (`git checkout -b feature/AmazingFeature`)
3. Commit your Changes (`git commit -m 'Add some AmazingFeature'`)
4. Push to the Branch (`git push origin feature/AmazingFeature`)
5. Open a Pull Request

---

## License

Distributed under the MIT License. See [LICENSE.txt](LICENSE.txt) for more details.
