callmefair

CallMeFair: A Comprehensive Framework for Automatic Bias Mitigation in AI Systems

CallMeFair provides tools and techniques to identify, measure, and reduce algorithmic bias in machine learning models through preprocessing, in-processing, and postprocessing methods.

For more information, visit: https://github.com/your-repo/callmefair

Submodules

• callmefair.mitigation
• callmefair.search
• callmefair.util

Classes

BMGridSearch	Grid Search for Bias Mitigation Combinations.
BMInterface	Main interface for managing datasets and bias mitigation operations.
BMManager	Bias Mitigation Manager for applying various fairness techniques.
BMMetrics	Comprehensive fairness metrics evaluation for bias mitigation.
BMType	Enumeration of bias mitigation techniques.
BaseSearch	Base class for bias search functionality in the CallMeFair framework.
BiasSearch	Main class for bias search and evaluation in the CallMeFair framework.
CType	Enumeration of attribute combination operations for bias search.

Functions

calculate_fairness_score(EOD, AOD, SPD, DI, TI)	Calculate a comprehensive fairness score based on multiple fairness metrics.
combine_attributes(df, col1, col2, operation)	Combines two binary columns in a DataFrame using a specified set operation,

Package Contents

class callmefair.BMGridSearch(bmI, model, bm_list, privileged_group, unprivileged_group)

Grid Search for Bias Mitigation Combinations.

This class provides a systematic framework for evaluating different combinations of bias mitigation techniques. It supports preprocessing, in-processing, and postprocessing methods, and can work with various machine learning models.

The grid search evaluates each combination of bias mitigation techniques and logs the results for comparison. It currently supports single sensitive attribute evaluation, with plans for multiple sensitive attributes.

Variables:

• bmI (BMInterface) – Interface for managing binary label datasets

• bmMR (BMManager) – Bias mitigation manager for applying techniques

• model – The machine learning model to evaluate

• bm_list (list[list[BMType]]) – List of bias mitigation combinations to test

• privileged_group (list[dict]) – List of dictionaries defining privileged groups

• unprivileged_group (list[dict]) – List of dictionaries defining unprivileged groups

• is_model_in (bool) – Whether using in-processing bias mitigation

Parameters:

• bmI (callmefair.util.fair_util.BMInterface)

• bm_list (list[list[callmefair.mitigation.fair_bm.BMType]])

• privileged_group (list[dict])

• unprivileged_group (list[dict])

Example

>>> from callmefair.mitigation.fair_grid import BMGridSearch
>>> from callmefair.mitigation.fair_bm import BMType
>>>
>>> # Define combinations to test
>>> combinations = [
>>>     [BMType.preReweighing],
>>>     [BMType.preDisparate, BMType.posCEO],
>>>     [BMType.inAdversarial]
>>> ]
>>>
>>> # Create grid search
>>> grid_search = BMGridSearch(
>>>     bmI=bm_interface,
>>>     model=RandomForestClassifier(),
>>>     bm_list=combinations,
>>>     privileged_group=privileged_groups,
>>>     unprivileged_group=unprivileged_groups
>>> )
>>>
>>> # Run evaluation
>>> grid_search.run_single_sensitive()

Initialize the Bias Mitigation Grid Search.

Parameters:

• bmI (BMInterface) – Interface for managing binary label datasets

• model – The machine learning model to evaluate. Can be None for in-processing techniques that have their own models.

• bm_list (list[list[BMType]]) – List of bias mitigation combinations to test. Each inner list represents one combination of techniques.

• privileged_group (list[dict]) – List of dictionaries defining privileged groups. Each dict should contain protected attribute names and their privileged values.

• unprivileged_group (list[dict]) – List of dictionaries defining unprivileged groups. Each dict should contain protected attribute names and their unprivileged values.

Example

>>> bm_combinations = [
>>>     [BMType.preReweighing],
>>>     [BMType.preDisparate, BMType.posCEO],
>>>     [BMType.inAdversarial]
>>> ]
>>>
>>> grid_search = BMGridSearch(
>>>     bmI=bm_interface,
>>>     model=RandomForestClassifier(),
>>>     bm_list=bm_combinations,
>>>     privileged_group=[{'gender': 1}],
>>>     unprivileged_group=[{'gender': 0}]
>>> )

run_single_sensitive()

Run grid search evaluation for single sensitive attribute.

This method performs a comprehensive evaluation of all bias mitigation combinations in the grid search. It evaluates each combination and logs the results for comparison. Currently supports single sensitive attribute evaluation, with plans for multiple sensitive attributes.

The method: 1. Validates in-processing configurations 2. Evaluates baseline performance (no bias mitigation) 3. Evaluates each bias mitigation combination 4. Logs results to CSV files for analysis

Raises:

ValueError – If in-processing bias mitigation is defined with a classifier model

Return type:

None

Example

>>> # Define bias mitigation combinations
>>> combinations = [
>>>     [BMType.preReweighing],
>>>     [BMType.preDisparate, BMType.posCEO],
>>>     [BMType.inAdversarial]
>>> ]
>>>
>>> # Run grid search
>>> grid_search.run_single_sensitive()
>>> # Results are logged to CSV files

bmI

bmMR

bm_list

is_model_in = False

model

privileged_group

unprivileged_group

class callmefair.BMInterface(df_train, df_val, df_test, label, protected)

Main interface for managing datasets and bias mitigation operations.

This class provides a unified interface for managing binary classification datasets with bias mitigation capabilities. It handles dataset splitting, feature scaling, and provides access to train/validation/test sets in various formats required by different bias mitigation techniques.

The interface supports: - Automatic dataset splitting and scaling - Binary label dataset creation for AIF360 compatibility - Feature scaling with train/validation/test consistency - Dataset restoration and transformation modes

Variables:

• data_sets (list) – List of DataFrames [train, validation, test]

• BM_attr (BMnames) – Bias mitigation attribute configuration

• transform (bool) – Whether to apply feature scaling

• biLData (list) – List of BinaryLabelDataset objects

• trainXY (tuple) – (features, labels) for training data

• valXY (tuple) – (features, labels) for validation data

• testXY (tuple) – (features, labels) for test data

Parameters:

• df_train (pandas.DataFrame)

• df_val (pandas.DataFrame)

• df_test (pandas.DataFrame)

• label (str)

• protected (list)

Example

>>> from callmefair.util.fair_util import BMInterface
>>> import pandas as pd
>>>
>>> # Load your data
>>> train_df = pd.read_csv('train.csv')
>>> val_df = pd.read_csv('val.csv')
>>> test_df = pd.read_csv('test.csv')
>>>
>>> # Initialize interface
>>> bm_interface = BMInterface(
>>>     df_train=train_df,
>>>     df_val=val_df,
>>>     df_test=test_df,
>>>     label='income',
>>>     protected=['gender', 'race']
>>> )
>>>
>>> # Get data in different formats
>>> train_bld = bm_interface.get_train_BLD()
>>> X_train, y_train = bm_interface.get_train_xy()

Initialize the Bias Mitigation Interface.

Parameters:

• df_train (pd.DataFrame) – Training dataset

• df_val (pd.DataFrame) – Validation dataset

• df_test (pd.DataFrame) – Test dataset

• label (str) – Name of the target/label column

• protected (list) – List of protected attribute column names

Example

>>> bm_interface = BMInterface(
>>>     df_train=train_df,
>>>     df_val=val_df,
>>>     df_test=test_df,
>>>     label='income',
>>>     protected=['gender', 'race']
>>> )

get_protected_att()

Get the list of protected attributes.

Returns:

List of protected attribute column names

Return type:

list

Example

>>> protected_attrs = bm_interface.get_protected_att()
>>> print(f"Protected attributes: {protected_attrs}")

get_test_BLD()

Get test dataset as BinaryLabelDataset.

Returns:

Test dataset in AIF360 format

Return type:

BinaryLabelDataset

Example

>>> test_bld = bm_interface.get_test_BLD()
>>> print(f"Test samples: {len(test_bld.features)}")

get_test_xy()

Get test data as (features, labels) tuple.

Returns:

(features, labels) for test data

Return type:

tuple

Example

>>> X_test, y_test = bm_interface.get_test_xy()
>>> print(f"Test features shape: {X_test.shape}")

get_train_BLD()

Get training dataset as BinaryLabelDataset.

Returns:

Training dataset in AIF360 format

Return type:

BinaryLabelDataset

Example

>>> train_bld = bm_interface.get_train_BLD()
>>> print(f"Training samples: {len(train_bld.features)}")

get_train_xy()

Get training data as (features, labels) tuple.

Returns:

(features, labels) for training data

Return type:

tuple

Example

>>> X_train, y_train = bm_interface.get_train_xy()
>>> print(f"Training features shape: {X_train.shape}")
>>> print(f"Training labels shape: {y_train.shape}")

get_val_BLD()

Get validation dataset as BinaryLabelDataset.

Returns:

Validation dataset in AIF360 format

Return type:

BinaryLabelDataset

Example

>>> val_bld = bm_interface.get_val_BLD()
>>> print(f"Validation samples: {len(val_bld.features)}")

get_val_xy()

Get validation data as (features, labels) tuple.

Returns:

(features, labels) for validation data

Return type:

tuple

Example

>>> X_val, y_val = bm_interface.get_val_xy()
>>> print(f"Validation features shape: {X_val.shape}")

pos_bm_set(new_test_BLD)

Set new test dataset after postprocessing bias mitigation.

This method updates the test dataset with the result of postprocessing bias mitigation techniques and reapplies scaling if needed.

Parameters:

new_test_BLD (BinaryLabelDataset) – New test dataset after bias mitigation

Return type:

None

Example

>>> # After applying equalized odds
>>> modified_test_bld = eq_odds.predict(original_test_bld)
>>> bm_interface.pos_bm_set(modified_test_bld)

pre_BM_set(new_train_BLD)

Set new training dataset after preprocessing bias mitigation.

This method updates the training dataset with the result of preprocessing bias mitigation techniques and reapplies scaling if needed.

Parameters:

new_train_BLD (BinaryLabelDataset) – New training dataset after bias mitigation

Return type:

None

Example

>>> # After applying reweighing
>>> modified_train_bld = reweighing.transform(original_train_bld)
>>> bm_interface.pre_BM_set(modified_train_bld)

restore_BLD()

Restore BinaryLabelDataset objects to their original state.

This method regenerates the BinaryLabelDataset objects and reapplies scaling if transform mode is enabled. Useful after bias mitigation operations that modify the datasets.

Example

>>> # After applying bias mitigation
>>> bm_interface.pre_BM_set(modified_train_bld)
>>> # Restore to original state
>>> bm_interface.restore_BLD()

Return type:

None

set_transform()

Enable transform mode for feature scaling.

This method enables the transform mode, which applies StandardScaler to all datasets. This is useful when working with models that require scaled features.

Example

>>> bm_interface.set_transform()
>>> # All subsequent operations will use scaled features

Return type:

None

BM_attr

data_sets

transform = False

class callmefair.BMManager(bmI, privileged_group, unprivileged_group)

Bias Mitigation Manager for applying various fairness techniques.

This class provides a unified interface for applying different bias mitigation techniques to machine learning datasets. It supports preprocessing, in-processing, and postprocessing approaches to reduce algorithmic bias.

The manager works with BinaryLabelDataset objects from the AIF360 library and provides methods for each type of bias mitigation technique.

Variables:

• bmI (BMInterface) – Interface for managing binary label datasets

• privileged_group (list[dict]) – List of dictionaries defining privileged groups

• unprivileged_group (list[dict]) – List of dictionaries defining unprivileged groups

Parameters:

• bmI (callmefair.util.fair_util.BMInterface)

• privileged_group (list[dict])

• unprivileged_group (list[dict])

Example

>>> from callmefair.util.fair_util import BMInterface
>>> from callmefair.mitigation.fair_bm import BMManager
>>>
>>> # Initialize with your data
>>> bm_interface = BMInterface(train_df, val_df, test_df, 'label', ['protected_attr'])
>>> privileged_groups = [{'protected_attr': 1}]
>>> unprivileged_groups = [{'protected_attr': 0}]
>>>
>>> bm_manager = BMManager(bm_interface, privileged_groups, unprivileged_groups)
>>>
>>> # Apply preprocessing bias mitigation
>>> bm_manager.pre_Reweighing()

Initialize the Bias Mitigation Manager.

Parameters:

• bmI (BMInterface) – Interface for managing binary label datasets

• privileged_group (list[dict]) – List of dictionaries defining privileged groups. Each dict should contain protected attribute names and their privileged values.

• unprivileged_group (list[dict]) – List of dictionaries defining unprivileged groups. Each dict should contain protected attribute names and their unprivileged values.

Example

>>> privileged_groups = [{'gender': 1, 'race': 1}]
>>> unprivileged_groups = [{'gender': 0, 'race': 0}]
>>> bm_manager = BMManager(bm_interface, privileged_groups, unprivileged_groups)

in_AD(debias=False)

Create an Adversarial Debiasing in-processing model.

Adversarial Debiasing is an in-processing technique that trains a classifier while simultaneously training an adversary that tries to predict the sensitive attribute from the classifier’s predictions. This encourages the classifier to make predictions that are independent of the sensitive attribute.

Parameters:

debias (bool) – Whether to apply debiasing. If True, the model will be trained to be fair. If False, it will be a standard classifier. Defaults to False.

Returns:

Configured adversarial debiasing model.

Return type:

AdversarialDebiasing

Raises:

ImportError – If TensorFlow is not available.

Example

>>> ad_model = bm_manager.in_AD(debias=True)
>>> # Train the model with your data
>>> ad_model.fit(train_data, train_labels)

in_Meta(sensitive_attribute, tau=0)

Create a MetaFair Classifier in-processing model.

MetaFair is an in-processing technique that uses a meta-learning approach to learn fair classifiers. It optimizes for both accuracy and fairness using a regularization term that penalizes unfairness.

Parameters:

• sensitive_attribute (str) – Name of the sensitive attribute to consider for fairness. This should be one of the protected attributes.

• tau (float) – Fairness parameter that controls the trade-off between accuracy and fairness. Higher values prioritize fairness. Defaults to 0.

Returns:

Configured MetaFair classifier.

Return type:

MetaFairClassifier

Example

>>> meta_model = bm_manager.in_Meta('gender', tau=0.1)
>>> # Train the model with your data
>>> meta_model.fit(train_data, train_labels)

pos_CEO(valid_BLD_pred, test_BLD_pred)

Apply Calibrated Equalized Odds (CEO) postprocessing bias mitigation.

CEO is a postprocessing technique that adjusts model predictions to achieve equalized odds while maintaining calibration. It ensures that the true positive and false positive rates are equal across different groups.

Parameters:

• valid_BLD_pred (BinaryLabelDataset) – Validation dataset with model predictions

• test_BLD_pred (BinaryLabelDataset) – Test dataset with model predictions

Returns:

Postprocessed test predictions with equalized odds

Return type:

BinaryLabelDataset

Example

>>> mitigated_predictions = bm_manager.pos_CEO(valid_pred, test_pred)
>>> # The predictions now satisfy equalized odds

pos_EO(valid_BLD_pred, test_BLD_pred)

Apply Equalized Odds (EO) postprocessing bias mitigation.

EO is a postprocessing technique that adjusts model predictions to achieve equalized odds. It ensures that the true positive and false positive rates are equal across different groups, without considering calibration.

Parameters:

• valid_BLD_pred (BinaryLabelDataset) – Validation dataset with model predictions

• test_BLD_pred (BinaryLabelDataset) – Test dataset with model predictions

Returns:

Postprocessed test predictions with equalized odds

Return type:

BinaryLabelDataset

Example

>>> mitigated_predictions = bm_manager.pos_EO(valid_pred, test_pred)
>>> # The predictions now satisfy equalized odds

pos_ROC(valid_BLD_pred, test_BLD_pred)

Apply Reject Option Classification (ROC) postprocessing bias mitigation.

ROC is a postprocessing technique that rejects predictions for instances where the model is uncertain, particularly when this uncertainty is correlated with the sensitive attribute. This helps reduce bias by abstaining from making predictions on potentially unfair cases.

Parameters:

• valid_BLD_pred (BinaryLabelDataset) – Validation dataset with model predictions

• test_BLD_pred (BinaryLabelDataset) – Test dataset with model predictions

Returns:

Postprocessed test predictions with reject option

Return type:

BinaryLabelDataset

Example

>>> mitigated_predictions = bm_manager.pos_ROC(valid_pred, test_pred)
>>> # Some predictions may be rejected to improve fairness

pre_DR(sensitive_attribute)

Apply Disparate Impact Remover (DIR) preprocessing bias mitigation.

DIR is a preprocessing technique that repairs the training data to remove disparate impact while preserving the utility of the data. It works by learning a repair transformation that minimizes the difference in conditional distributions between groups.

Parameters:

sensitive_attribute (str) – Name of the sensitive attribute to repair. This should be one of the protected attributes in your dataset.

Return type:

None

Example

>>> bm_manager.pre_DR('gender')
>>> # The training dataset is now repaired for the 'gender' attribute

pre_LFR()

Apply Learning Fair Representations (LFR) preprocessing bias mitigation.

LFR is a preprocessing technique that learns a fair representation of the data that removes information about the sensitive attributes while preserving the utility for the prediction task. It uses an adversarial approach to learn representations that are fair across different groups.

Example

>>> bm_manager.pre_LFR()
>>> # The training dataset now has fair representations

Return type:

None

pre_Reweighing()

Apply Reweighing preprocessing bias mitigation.

Reweighing is a preprocessing technique that assigns different weights to instances based on their group membership to achieve statistical parity. This method modifies the training dataset by reweighting instances.

The method fits the reweighing algorithm on the training data and transforms it to create a fairer training dataset.

Example

>>> bm_manager.pre_Reweighing()
>>> # The training dataset is now reweighted for fairness

Return type:

None

bmI

privileged_group

unprivileged_group

class callmefair.BMMetrics(bmI, class_array, pred_val, pred_test, privileged_group, unprivileged_group)

Comprehensive fairness metrics evaluation for bias mitigation.

This class provides comprehensive evaluation of both classification performance and fairness metrics for machine learning models. It supports both standard models and in-processing bias mitigation techniques.

The class evaluates: - Classification metrics: accuracy, precision, recall, F1, MCC - Fairness metrics: EOD, AOD, SPD, DI, TI - Optimal threshold selection based on balanced accuracy - Comprehensive fairness scoring

Variables:

• bmI (BMInterface) – Interface for managing datasets

• pos_idx (int) – Index of positive class

• in_mode (bool) – Whether using in-processing bias mitigation

• pred_test – Test predictions (numpy array or BinaryLabelDataset)

• pred_val – Validation predictions (numpy array or BinaryLabelDataset)

• privileged_group (list[dict]) – List of privileged group definitions

• unprivileged_group (list[dict]) – List of unprivileged group definitions

• balanced_acc (np.ndarray) – Balanced accuracy scores for different thresholds

• class_threshold (np.ndarray) – Threshold values for evaluation

• best_class_thresh (float) – Optimal threshold based on validation performance

• cmetrics (ClassificationMetric) – AIF360 classification metrics object

Parameters:

• bmI (BMInterface)

• class_array (numpy.ndarray)

• pred_val (numpy.ndarray | aif360.datasets.BinaryLabelDataset)

• pred_test (numpy.ndarray | aif360.datasets.BinaryLabelDataset)

• privileged_group (list[dict])

• unprivileged_group (list[dict])

Example

>>> from callmefair.util.fair_util import BMMetrics
>>> import numpy as np
>>>
>>> # Create metrics evaluator
>>> metrics = BMMetrics(
>>>     bmI=bm_interface,
>>>     class_array=np.array([0, 1]),
>>>     pred_val=val_predictions,
>>>     pred_test=test_predictions,
>>>     privileged_group=privileged_groups,
>>>     unprivileged_group=unprivileged_groups
>>> )
>>>
>>> # Get comprehensive report
>>> report = metrics.get_report()
>>> print(f"Accuracy: {report['acc']:.4f}")
>>> print(f"Fairness score: {report['spd']:.4f}")

Initialize the Bias Mitigation Metrics evaluator.

Parameters:

• bmI (BMInterface) – Interface for managing datasets

• class_array (np.ndarray) – Array of class labels [0, 1]

• pred_val (numpy.ndarray | aif360.datasets.BinaryLabelDataset) – Validation predictions (numpy array or BinaryLabelDataset)

• pred_test (numpy.ndarray | aif360.datasets.BinaryLabelDataset) – Test predictions (numpy array or BinaryLabelDataset)

• privileged_group (list[dict]) – List of privileged group definitions

• unprivileged_group (list[dict]) – List of unprivileged group definitions

Example

>>> metrics = BMMetrics(
>>>     bmI=bm_interface,
>>>     class_array=np.array([0, 1]),
>>>     pred_val=val_pred,
>>>     pred_test=test_pred,
>>>     privileged_group=[{'gender': 1}],
>>>     unprivileged_group=[{'gender': 0}]
>>> )

get_groups()

Get privileged and unprivileged group definitions.

Returns:

(privileged_group, unprivileged_group)

• privileged_group (list[dict]): List of privileged group definitions

• unprivileged_group (list[dict]): List of unprivileged group definitions

Return type:

tuple

Example

>>> privileged, unprivileged = metrics.get_groups()
>>> print(f"Privileged groups: {privileged}")
>>> print(f"Unprivileged groups: {unprivileged}")

get_pred_test()

Get test predictions.

Returns:

Test predictions (numpy array or BinaryLabelDataset)

Example

>>> test_pred = metrics.get_pred_test()
>>> print(f"Test predictions shape: {test_pred.shape}")

get_report()

Get comprehensive performance and fairness report.

Returns:

Dictionary containing all classification and fairness metrics

• Classification metrics: balanced_acc, acc, precision, recall, f1, mcc

• Fairness metrics: eq_opp_diff, avg_odd_diff, spd, disparate_impact, theil_idx

Return type:

dict

Example

>>> report = metrics.get_report()
>>> print(f"Accuracy: {report['acc']:.4f}")
>>> print(f"Statistical Parity Difference: {report['spd']:.4f}")
>>> print(f"Equal Opportunity Difference: {report['eq_opp_diff']:.4f}")

get_score()

Get comprehensive fairness score using the calculate_fairness_score function.

Returns:

Fairness score evaluation with overall score and detailed breakdown

• ’raw_score’ (float): Unnormalized fairness score

• ’overall_score’ (float): Normalized fairness score (0-1, lower is better)

• ’metric_evaluations’ (dict): Boolean evaluation of each metric

• ’deviations’ (dict): Normalized deviations from optimal values

• ’is_fair’ (bool): Whether all metrics are within acceptable ranges

Return type:

dict

Example

>>> score_dict = metrics.get_score()
>>> print(f"Overall fairness score: {score_dict['overall_score']:.3f}")
>>> print(f"Is fair: {score_dict['is_fair']}")
>>>
>>> # Check individual metric evaluations
>>> for metric, is_acceptable in score_dict['metric_evaluations'].items():
>>>     print(f"{metric}: {'✓' if is_acceptable else '✗'}")

set_new_pred(pred_val, pred_test)

Update predictions and recalculate metrics.

This method allows updating the predictions and recalculating all metrics without recreating the entire BMMetrics object.

Parameters:

• pred_val (np.ndarray) – New validation predictions

• pred_test (np.ndarray) – New test predictions

Return type:

None

Example

>>> # After training a new model
>>> new_val_pred = model.predict_proba(X_val)
>>> new_test_pred = model.predict_proba(X_test)
>>> metrics.set_new_pred(new_val_pred, new_test_pred)

set_pos_pred(test_BLD_pred)

Set postprocessed predictions and update metrics.

This method is used for postprocessing bias mitigation techniques that modify the test predictions directly.

Parameters:

test_BLD_pred (BinaryLabelDataset) – Postprocessed test predictions

Return type:

None

Example

>>> # After applying equalized odds
>>> postprocessed_pred = eq_odds.predict(test_bld)
>>> metrics.set_pos_pred(postprocessed_pred)

balanced_acc

bmI

class_threshold

in_mode = False

pos_idx

pred_test

pred_val

privileged_group

unprivileged_group

class callmefair.BMType(*args, **kwds)

Bases: enum.Enum

Enumeration of bias mitigation techniques.

This enum categorizes bias mitigation methods into three main groups: - Preprocessing: Applied to training data before model training - In-processing: Applied during model training - Postprocessing: Applied to model predictions after training

Variables:

• preReweighing – Reweighing preprocessing technique

• preDisparate – Disparate Impact Remover preprocessing technique

• preLFR – Learning Fair Representations preprocessing technique

• inAdversarial – Adversarial Debiasing in-processing technique

• inMeta – MetaFair Classifier in-processing technique

• posCalibrated – Calibrated Equalized Odds postprocessing technique

• posEqqOds – Equalized Odds postprocessing technique

• posROC – Reject Option Classification postprocessing technique

inAdversarial = 4

inMeta = 5

property is_in: bool

Check if the bias mitigation technique is an in-processing method.

Returns:

True if the technique is in-processing, False otherwise.

Return type:

bool

property is_pos: bool

Check if the bias mitigation technique is a postprocessing method.

Returns:

True if the technique is postprocessing, False otherwise.

Return type:

bool

property is_pre: bool

Check if the bias mitigation technique is a preprocessing method.

Returns:

True if the technique is preprocessing, False otherwise.

Return type:

bool

posCalibrated = 6

posEqqOds = 7

posROC = 8

preDisparate = 2

preLFR = 3

preReweighing = 1

class callmefair.BaseSearch(df, label_name)

Base class for bias search functionality in the CallMeFair framework.

This class provides the core functionality for evaluating bias in machine learning models with respect to sensitive attributes. It handles dataset preparation, model training, and fairness metric calculation using AIF360.

The class supports: - Individual attribute bias evaluation - Multiple ML model types - Imbalanced dataset handling with NearMiss - Parallel processing for efficient evaluation - Comprehensive fairness metrics calculation

Variables:

• df (pd.DataFrame) – Input dataset with features and target

• label_name (str) – Name of the target variable

• scaler (StandardScaler) – Feature scaler for model training

Parameters:

• df (pandas.DataFrame)

• label_name (str)

Example

>>> searcher = BaseSearch(df, 'target')
>>> results = searcher.evaluate_attribute('gender', iterate=10, model_name='lr')

Initialize the BaseSearch object.

Parameters:

• df (pd.DataFrame) – Input dataset containing features and target variable

• label_name (str) – Name of the target variable column

evaluate_attribute(attribute, treat_umbalance=False, iterate=10, model_name='lr', df_new=None)

Evaluate bias for a specific attribute across multiple iterations.

This method performs comprehensive bias evaluation by: - Running multiple iterations for statistical robustness - Training models with optional class balancing - Using parallel processing for efficiency - Aggregating results across iterations

Parameters:

• attribute (str) – Name of the sensitive attribute to evaluate

• treat_umbalance (bool) – Whether to apply NearMiss undersampling

• iterate (int) – Number of iterations for robust evaluation

• model_name (str) – Type of model to use (‘lr’, ‘cat’, ‘xgb’, ‘mlp’)

• df_new (pd.DataFrame, optional) – Alternative dataset to use

Returns:

Dictionary containing averaged fairness scores for the attribute

Return type:

dict

Example

>>> results = searcher.evaluate_attribute('gender', iterate=5, model_name='lr')
>>> print(results['gender_raw'], results['gender_overall'])

df

label_name

scaler

class callmefair.BiasSearch(df, label_name, attribute_names)

Bases: callmefair.search._search_base.BaseSearch

Main class for bias search and evaluation in the CallMeFair framework.

This class extends BaseSearch to provide comprehensive bias evaluation across individual attributes and their combinations. It supports both individual attribute analysis and complex attribute combination evaluation using different set operations.

The class provides methods for: - Individual attribute bias evaluation - 2-way and 3-way attribute combinations - Set operation comparison (union, intersection, differences) - Pretty table output for results - Comprehensive bias summarization

Variables:

• df (pd.DataFrame) – Input dataset with features and target

• label_name (str) – Name of the target variable

• attribute_names (list[str]) – List of sensitive attributes to evaluate

Parameters:

• df (pandas.DataFrame)

• label_name (str)

• attribute_names (list[str])

Example

>>> searcher = BiasSearch(df, 'target', ['gender', 'race', 'age'])
>>> table, printable = searcher.evaluate_average()
>>> print(printable)

Initialize the BiasSearch object.

Parameters:

• df (pd.DataFrame) – Input dataset containing features and target variable

• label_name (str) – Name of the target variable column

• attribute_names (list[str]) – List of sensitive attributes to evaluate

evaluate_average(treat_umbalance=False, iterate=10, model_name='lr')

Evaluate bias for all individual attributes and return averaged results.

This method evaluates bias for each individual sensitive attribute and returns both raw and normalized fairness scores. The results are formatted into a pretty table for easy visualization.

Parameters:

• treat_umbalance (bool) – Whether to apply NearMiss undersampling

• iterate (int) – Number of iterations for robust evaluation

• model_name (str) – Type of model to use (‘lr’, ‘cat’, ‘xgb’, ‘mlp’)

Returns:

(table_data, pretty_table) - Raw data and formatted table

Return type:

tuple

Example

>>> table, printable = searcher.evaluate_average(iterate=5)
>>> print(printable)

evaluate_combination_average(col_1, col_2, treat_umbalance=False, iterate=10, model_name='lr')

Evaluate bias for all set operations between two attributes.

This method compares all possible set operations (union, intersection, differences, symmetric difference) between two attributes and evaluates bias for each combination. This helps understand how different ways of combining attributes affect bias.

Parameters:

• col_1 (str) – Name of the first attribute

• col_2 (str) – Name of the second attribute

• treat_umbalance (bool) – Whether to apply NearMiss undersampling

• iterate (int) – Number of iterations for robust evaluation

• model_name (str) – Type of model to use (‘lr’, ‘cat’, ‘xgb’, ‘mlp’)

Returns:

(table_data, pretty_table) - Raw data and formatted table

Return type:

tuple

Example

>>> table, printable = searcher.evaluate_combination_average('gender', 'race')
>>> print(printable)

evaluate_combinations(treat_umbalance=False, iterate=10, model_name='lr')

Evaluate bias for all 2-way and 3-way attribute combinations.

This method creates combinations of sensitive attributes using intersection operations and evaluates bias for each combination. It generates both 2-way combinations (e.g., gender_race) and 3-way combinations (e.g., gender_race_age).

Parameters:

• treat_umbalance (bool) – Whether to apply NearMiss undersampling

• iterate (int) – Number of iterations for robust evaluation

• model_name (str) – Type of model to use (‘lr’, ‘cat’, ‘xgb’, ‘mlp’)

Returns:

(table_data, pretty_table) - Raw data and formatted table

Return type:

tuple

Example

>>> table, printable = searcher.evaluate_combinations()
>>> print(printable)

attribute_names

class callmefair.CType(*args, **kwds)

Bases: enum.Enum

Enumeration of attribute combination operations for bias search.

This enum defines the set operations that can be performed when combining two binary sensitive attributes to create composite protected groups.

Variables:

• union – Logical OR operation (either attribute is 1)

• intersection – Logical AND operation (both attributes are 1)

• difference_1_minus_2 – Set difference (attribute1=1 AND attribute2=0)

• difference_2_minus_1 – Set difference (attribute2=1 AND attribute1=0)

• symmetric_difference – XOR operation (exactly one attribute is 1)

__str__()

Return string representation of the operation type.

difference_1_minus_2 = 3

difference_2_minus_1 = 4

intersection = 2

symmetric_difference = 5

union = 1

callmefair.calculate_fairness_score(EOD, AOD, SPD, DI, TI)

Calculate a comprehensive fairness score based on multiple fairness metrics.

This function aggregates five key fairness metrics into a single normalized score that represents the overall fairness of a machine learning model. The function evaluates each metric against predefined acceptable ranges and calculates deviations from optimal values to produce a unified fairness assessment.

The function uses a weighted scoring system where: - Each metric contributes up to 0.2 for being outside acceptable ranges - Each metric contributes up to 0.16 based on deviation from optimal values - The final score is normalized to the 0-1 range where 0 = perfect fairness

Parameters:

• EOD (float) – Equal Opportunity Difference - measures difference in true positive rates between groups. Optimal value: 0.0, Acceptable range: (-0.1, 0.1)

• AOD (float) – Average Odds Difference - measures difference in average of true positive and false positive rates between groups. Optimal value: 0.0, Acceptable range: (-0.1, 0.1)

• SPD (float) – Statistical Parity Difference - measures difference in positive prediction rates between groups. Optimal value: 0.0, Acceptable range: (-0.1, 0.1)

• DI (float) – Disparate Impact - ratio of positive prediction rates between groups. Optimal value: 1.0, Acceptable range: (0.8, 1.2)

• TI (float) – Theil Index - measures inequality in prediction distributions. Optimal value: 0.0, Acceptable range: (0.0, 0.25)

Returns:

Dictionary containing fairness evaluation results with keys:

• ’raw_score’ (float): Unnormalized fairness score

• ’overall_score’ (float): Normalized fairness score (0-1, lower is better)

• ’metric_evaluations’ (dict): Boolean evaluation of each metric

• ’deviations’ (dict): Normalized deviations from optimal values

• ’is_fair’ (bool): Whether all metrics are within acceptable ranges

Return type:

dict

Example

>>> # Perfect fairness
>>> result = calculate_fairness_score(0.0, 0.0, 0.0, 1.0, 0.0)
>>> print(f"Score: {result['overall_score']}")  # 0.0 (perfect)
>>> print(f"Is fair: {result['is_fair']}")  # True
>>>
>>> # Moderate unfairness
>>> result = calculate_fairness_score(0.15, 0.12, 0.18, 0.7, 0.3)
>>> print(f"Score: {result['overall_score']}")  # ~0.6-0.8
>>> print(f"Is fair: {result['is_fair']}")  # False
>>>
>>> # Check individual metric evaluations
>>> for metric, is_acceptable in result['metric_evaluations'].items():
>>>     print(f"{metric}: {'✓' if is_acceptable else '✗'}")

callmefair.combine_attributes(df, col1, col2, operation)

Combines two binary columns in a DataFrame using a specified set operation, replacing the original columns with a single combined column.

This function creates composite protected groups by combining two binary sensitive attributes using set operations. The resulting combined attribute can be used for more sophisticated bias analysis.

Parameters:

• df (pd.DataFrame) – Input DataFrame containing the binary columns

• col1 (str) – Name of the first binary column (e.g., ‘gender’)

• col2 (str) – Name of the second binary column (e.g., ‘race’)

• operation (CType) – Set operation to apply (‘union’, ‘intersection’, ‘difference_1_minus_2’, ‘difference_2_minus_1’, ‘symmetric_difference’)

Returns:

New DataFrame with original columns replaced by combined column

Return type:

pd.DataFrame

Raises:

ValueError – If columns are not binary (contain values other than 0 or 1)

Example

>>> df = pd.DataFrame({'gender': [1, 0, 1, 0], 'race': [1, 1, 0, 0]})
>>> result = combine_attributes(df, 'gender', 'race', CType.intersection)
>>> print(result.columns)
['gender_race']