PRACTICAL 7: COMPARING CLASSIFICATION MODELS

Dataset: pulsar_stars.csv (target_class = 0 / 1)

STEPS (same for every model):
1. Install the libraries:  pip install pandas numpy scikit-learn matplotlib
2. Load pulsar_stars.csv and separate the features X and the target y (target_class).
3. Scale the features (StandardScaler) and split the data into training and testing sets.
4. Train the model on the training set and predict on the test set.
5. Calculate the metrics (accuracy, confusion matrix, precision, sensitivity, specificity, AUC).
6. Draw the plots and compare the results of all six models.


1. LINEAR REGRESSION

CODE:
import pandas as pd
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, r2_score, accuracy_score
import matplotlib.pyplot as plt
import numpy as np
file_path = "C:/Users/Admin/Downloads/pulsar_stars.csv"
data = pd.read_csv(file_path)
data.columns = data.columns.str.strip()
X = data[['Skewness_DM_SNR_curve']]
y = data['target_class']
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)
X_train, X_test, y_train, y_test = train_test_split(X_scaled, y, test_size=0.2, random_state=42)
lr_model = LinearRegression()
lr_model.fit(X_train, y_train)
y_pred = lr_model.predict(X_test)
y_pred_binary = [1 if pred >= 0.5 else 0 for pred in y_pred]
mse = mean_squared_error(y_test, y_pred)
r2 = r2_score(y_test, y_pred)
accuracy = accuracy_score(y_test, y_pred_binary)
plt.figure(figsize=(10, 6))
plt.scatter(y_test, y_pred, color='blue', alpha=0.6, edgecolor='k')
plt.plot([y_test.min(), y_test.max()], [y_test.min(), y_test.max()], color='red', lw=2)
plt.xlabel('Actual Values')
plt.ylabel('Predicted Values')
plt.title('Actual vs Predicted Values - Linear Regression')
plt.show()
residuals = y_test - y_pred
plt.figure(figsize=(10, 6))
plt.scatter(y_pred, residuals, color='green', alpha=0.6, edgecolor='k')
plt.axhline(0, color='red', linestyle='--', lw=2)
plt.xlabel('Predicted Values')
plt.ylabel('Residuals')
plt.title('Residuals Plot - Linear Regression')
plt.show()
print("Mean Squared Error:", mse)
print("R-squared:", r2)
print("Accuracy:", accuracy)

EXPLANATION:
- Linear regression is used with one feature (Skewness_DM_SNR_curve) to predict the class value.
- The predicted values are converted to 0 / 1 using the 0.5 threshold, so accuracy can be calculated.
- Mean squared error and R-squared tell how close the predictions are to the actual values.
- The Actual vs Predicted plot and the Residuals plot show the errors of the model.
- Linear regression is not really meant for classification, so it is the weakest model here.


2. DECISION TREE

CODE:
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier, plot_tree
from sklearn.metrics import (accuracy_score, confusion_matrix,
precision_score,
recall_score, roc_curve, auc)
import matplotlib.pyplot as plt
import numpy as np
filepath = "C:/Users/Admin/Downloads/pulsar_stars.csv"
data = pd.read_csv(filepath)
data['target_class'] = data['target_class'].apply(lambda x: 1 if x >= 0.5 else 0)
X = data.drop(columns=['target_class'])
y = data['target_class']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2,
random_state=42)
clf = DecisionTreeClassifier(random_state=42)
clf.fit(X_train, y_train)
y_pred = clf.predict(X_test)
y_pred_prob = clf.predict_proba(X_test)[:, 1]
accuracy = accuracy_score(y_test, y_pred)
conf_matrix = confusion_matrix(y_test, y_pred)
precision = precision_score(y_test, y_pred)
sensitivity = recall_score(y_test, y_pred)
TN = conf_matrix[0, 0]
FP = conf_matrix[0, 1]
specificity = TN / (TN + FP)
standard_error = np.sqrt((accuracy * (1 - accuracy)) / len(y_test))
fpr, tpr, _ = roc_curve(y_test, y_pred_prob)
roc_auc = auc(fpr, tpr)
plt.figure()
plt.plot(fpr, tpr, color='darkorange', lw=2, label='ROC curve (area = %0.2f)'% roc_auc)
plt.plot([0, 1], [0, 1], color='navy', lw=2, linestyle='--')
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('Receiver Operating Characteristic')
plt.legend(loc="lower right")
plt.show()
plt.figure(figsize=(12, 8))
plot_tree(clf, feature_names=X.columns, class_names=['0', '1'], filled=True,
max_depth=3)
plt.title("Decision Tree (Max Depth = 3)")
plt.show()
print("Predictions:", y_pred)
print("Accuracy:", accuracy)
print("Confusion Matrix:\n", conf_matrix)
print("Standard Error:", standard_error)
print("Precision:", precision)
print("Sensitivity (Recall):", sensitivity)
print("Specificity:", specificity)
print("AUC:", roc_auc)

EXPLANATION:
- The decision tree splits the data again and again on feature conditions until the classes are separated.
- All eight features are used and the data is split 80% train / 20% test.
- The confusion matrix gives precision, sensitivity (recall), specificity and the standard error of the accuracy.
- The ROC curve and AUC value show how well the tree separates pulsar and non-pulsar stars.
- plot_tree() draws the first 3 levels of the tree.


3. NAIVE BAYES

CODE:
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.naive_bayes import GaussianNB
from sklearn.metrics import (
    accuracy_score,
    confusion_matrix,
    precision_score,
    recall_score,
    roc_curve,
    auc
)
import matplotlib.pyplot as plt
import numpy as np
file_path = "C:/Users/Admin/Downloads/pulsar_stars.csv"
data = pd.read_csv(file_path)
data['target_class'] = data['target_class'].apply(lambda x: 1 if x >= 0.5 else 0)
X = data.drop(columns=['target_class'])
y = data['target_class']
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)
X_train, X_test, y_train, y_test = train_test_split(X_scaled, y, test_size=0.2, random_state=42)
nb_model = GaussianNB()
nb_model.fit(X_train, y_train)
y_pred = nb_model.predict(X_test)
y_pred_prob = nb_model.predict_proba(X_test)[:, 1]
accuracy = accuracy_score(y_test, y_pred)
conf_matrix = confusion_matrix(y_test, y_pred)
precision = precision_score(y_test, y_pred)
sensitivity = recall_score(y_test, y_pred)
TN = conf_matrix[0, 0]
FP = conf_matrix[0, 1]
specificity = TN / (TN + FP)
standard_error = np.sqrt((accuracy * (1 - accuracy)) / len(y_test))
fpr, tpr, _ = roc_curve(y_test, y_pred_prob)
roc_auc = auc(fpr, tpr)
plt.figure()
plt.plot(fpr, tpr, color='darkorange', lw=2, label='ROC curve (area = %0.2f)' % roc_auc)
plt.plot([0, 1], [0, 1], color='navy', lw=2, linestyle='--')
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('Receiver Operating Characteristic - Naive Bayes')
plt.legend(loc="lower right")
plt.show()
print("Predictions:", y_pred)
print("Accuracy:", accuracy)
print("Confusion Matrix:\n", conf_matrix)
print("Standard Error:", standard_error)
print("Precision:", precision)
print("Sensitivity (Recall):", sensitivity)
print("Specificity:", specificity)
print("AUC:", roc_auc)

EXPLANATION:
- Gaussian Naive Bayes uses Bayes' theorem and assumes the features are independent and follow a normal distribution.
- The features are standardized first and the data is split 80% train / 20% test.
- It is very fast because it only calculates the mean and variance of each feature for each class.
- predict_proba() gives the class probabilities, which are used to draw the ROC curve.
- Accuracy, precision, sensitivity, specificity, standard error and AUC are printed for comparison.


4. PCA (PRINCIPAL COMPONENT ANALYSIS) WITH LOGISTIC REGRESSION

CODE:
import pandas as pd
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score
import matplotlib.pyplot as plt
import numpy as np
file_path = "C:/Users/Admin/Downloads/pulsar_stars.csv"
data = pd.read_csv(file_path)
data['target_class'] = data['target_class'].apply(lambda x: 1 if x >= 0.5 else 0)
X = data.drop(columns=['target_class'])
y = data['target_class']
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)
n_components = 7
pca = PCA(n_components=n_components)
X_pca = pca.fit_transform(X_scaled)
explained_variance_ratio = pca.explained_variance_ratio_
total_variance_explained = np.sum(explained_variance_ratio)
print("Explained Variance Ratio:", explained_variance_ratio)
print("Total Variance Explained:", total_variance_explained)
plt.figure(figsize=(8, 5))
plt.bar(range(1, n_components + 1), explained_variance_ratio, alpha=0.7, align='center', label='Individual Explained Variance')
plt.step(range(1, n_components + 1), np.cumsum(explained_variance_ratio), where='mid', label='Cumulative Explained Variance')
plt.xlabel('Principal Component Index')
plt.ylabel('Explained Variance Ratio')
plt.title('PCA Explained Variance')
plt.legend(loc='best')
plt.show()
X_train, X_test, y_train, y_test = train_test_split(X_pca, y, test_size=0.2, random_state=42)
model = LogisticRegression()
model.fit(X_train, y_train)
y_pred = model.predict(X_test)
accuracy = accuracy_score(y_test, y_pred)
print("Accuracy:", accuracy)

EXPLANATION:
- PCA reduces the 8 features to 7 new uncorrelated features called principal components.
- The explained variance ratio tells how much of the information each component keeps.
- The bar and step plot show individual and cumulative explained variance.
- Logistic regression is trained on the reduced features and its accuracy is printed.
- This checks whether the model still works well after reducing the dimensions.


5. SVM (SUPPORT VECTOR MACHINE)

CODE:
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
from sklearn.metrics import classification_report, confusion_matrix, roc_auc_score
file_path = "C:/Users/Admin/Downloads/pulsar_stars.csv"
data = pd.read_csv(file_path)
data.columns = data.columns.str.strip()
X = data.drop(columns=['target_class'])
y = data['target_class']
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=42, stratify=y
)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)
svm_model = SVC(
    kernel='rbf',
    class_weight='balanced',
    probability=True,
    random_state=42
)
svm_model.fit(X_train_scaled, y_train)
y_pred = svm_model.predict(X_test_scaled)
y_prob = svm_model.predict_proba(X_test_scaled)[:, 1]
print("--- Confusion Matrix ---")
print(confusion_matrix(y_test, y_pred))
print("\n--- Classification Report ---")
print(classification_report(y_test, y_pred))
print(f"ROC-AUC Score: {roc_auc_score(y_test, y_prob):.4f}")

EXPLANATION:
- SVM finds the best boundary (hyperplane) that separates the two classes with the widest margin.
- The RBF kernel lets it draw a curved boundary, and class_weight='balanced' gives more importance to the rare pulsar class.
- The data is split 70% / 30% with stratify=y so both sets keep the same class ratio.
- The scaler is fitted only on the training data and then applied to the test data.
- Confusion matrix, classification report and ROC-AUC score show how well the rare class is found.


6. RANDOM FOREST

CODE:
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import classification_report, confusion_matrix, roc_auc_score
df = pd.read_csv('C:/Users/Admin/Downloads/pulsar_stars.csv')
df.columns = df.columns.str.strip()
X = df.drop(columns=['target_class'])
y = df['target_class']
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=42, stratify=y
)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)
rf_model = RandomForestClassifier(
    n_estimators=100,
    class_weight='balanced',
    random_state=42,
    n_jobs=-1
)
rf_model.fit(X_train_scaled, y_train)
y_pred = rf_model.predict(X_test_scaled)
y_prob = rf_model.predict_proba(X_test_scaled)[:, 1]
print("--- Confusion Matrix ---")
print(confusion_matrix(y_test, y_pred))
print("\n--- Classification Report ---")
print(classification_report(y_test, y_pred))
print(f"ROC-AUC Score: {roc_auc_score(y_test, y_prob):.4f}")
importances = pd.Series(rf_model.feature_importances_, index=X.columns)
print("\n--- Feature Importances ---")
print(importances.sort_values(ascending=False))

EXPLANATION:
- Random forest builds 100 decision trees on random samples of the data and the final class is decided by their majority vote.
- Voting over many trees reduces overfitting compared with a single decision tree.
- class_weight='balanced' penalizes mistakes on the rare pulsar class more.
- The same confusion matrix, classification report and ROC-AUC score are printed for comparison.
- feature_importances_ shows which features contributed most to the prediction.
