PRACTICAL 3: DATA CLEANING ON DATASET (LOAN CREDIT SCORE)

Dataset taken: Loan_Credit_Score (loan_credit_dataset_240_rows_with_outliers.csv)

STEPS:
1. Install the libraries:  pip install pandas numpy matplotlib
2. Load the loan credit dataset and view head(), info() and describe().
3. Check the missing values, duplicate rows and data types.
4. Draw a boxplot of each numeric column (before removing outliers).
5. Remove the outliers of each column using the IQR method.
6. Draw the boxplots again (after removing outliers).

CODE:
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
df=pd.read_csv('/content/sample_data/loan_credit_dataset_240_rows_with_outliers.csv')
print(df.head())
print(df.info())
print(df.describe())
print(df.isnull().sum())
print("Duplicate rows:",df.duplicated().sum())
print(df.dtypes)
print("Before removing outliers:")
columns = [
    "AnnualIncome",
    "CreditScore",
    "LoanAmount",
    "DebtToIncomeRatio",
    "EmploymentYears"
]
for col in columns:
    plt.figure(figsize=(6,4))
    plt.boxplot(df[col])
    plt.title(f"Box Plot of {col}")
    plt.ylabel(col)
    plt.grid(True)
    plt.show()
print("identifying and removing outliers")
def remove_outliers(column):
    Q1=column.quantile(0.25)
    Q3=column.quantile(0.75)
    IQR=Q3-Q1
    lower=Q1-1.5*IQR
    upper=Q3+1.5*IQR
    print(f"\nColumn statistics:Q1={Q1},Q3={Q3},IQR={IQR}")
    print(f"Outlier bounds: lower={lower}, upper={upper}")
    return column.apply(lambda x: np.nan if (x < lower or x > upper) else x)
for col in columns:
    df[col] = remove_outliers(df[col])
print("After removing outliers:")
for col in columns:
    plt.figure(figsize=(6,4))
    plt.boxplot(df[col].dropna())
    plt.title(f"Box Plot of {col}")
    plt.ylabel(col)
    plt.grid(True)
    plt.show()

EXPLANATION:
- The first part inspects the dataset: shape of data, statistics, missing values, duplicates and data types.
- Five numeric columns (income, credit score, loan amount, debt-to-income ratio, employment years) are checked for outliers.
- Boxplots drawn before cleaning show the extreme values in each column.
- remove_outliers() uses the IQR rule (Q1-1.5*IQR, Q3+1.5*IQR) and replaces the outliers with NaN.
- The boxplots after cleaning show the same columns without the extreme values.

OUTPUT:
Before removing outliers: box plots with outlier points.
After removing outliers: box plots with the outliers removed.
