PRACTICAL 13: CLASSIFIERS EXPLANATION (PCA, SVM, RANDOM FOREST)

(The code for these three is given in Practical 7: 4. PCA, 5. SVM and 6. Random Forest.)


1. PCA (PRINCIPAL COMPONENT ANALYSIS)

STEPS:
1. Standardize the data (mean 0, standard deviation 1).
2. Calculate the covariance matrix of the features.
3. Find the eigenvectors (directions) and eigenvalues (importance) of the matrix.
4. Sort them and keep the top components (usually 80% to 95% of the variance), then project the data on them.

EXPLANATION:
- PCA is an unsupervised dimensionality reduction method: it reduces many features into a few new ones called principal components.
- The principal components are combinations of the original features, and PC1 holds the maximum variance.
- Each next component is perpendicular (uncorrelated) to the earlier ones and holds the next highest variance.
- Standardization is needed first, otherwise a feature with large numbers (like salary) would dominate.
- Dropping the last components loses little information but makes the data simpler.


2. SVM (SUPPORT VECTOR MACHINE)

STEPS:
1. Plot the data points of the two classes.
2. Find the hyperplane (decision boundary) that separates the classes.
3. Choose the hyperplane with the maximum margin, with the support vectors (nearest points) on its edges.
4. If the data is not linearly separable, apply a kernel (linear, polynomial or RBF) to separate it in a higher dimension.

EXPLANATION:
- SVM is a supervised classifier that finds the widest "street" between the two classes.
- Only the support vectors (the points closest to the boundary) decide the position of the boundary.
- Parameter C controls the balance: high C allows fewer mistakes (risk of overfitting), low C gives a wider margin.
- The kernel trick lets SVM draw curved boundaries without actually moving the data to a higher dimension.
- The RBF kernel is the common choice for complex, non-linear data.


3. RANDOM FOREST

STEPS:
1. Bootstrapping: take a random sample (with replacement) of the rows for each tree.
2. Feature randomness: at each split, let the tree choose from a random subset of the features only.
3. Train every decision tree fully on its own sample.
4. Aggregation: every tree votes and the majority class is the final answer (average for regression).

EXPLANATION:
- Random forest is an ensemble method that combines many decision trees ("wisdom of the crowd").
- Different rows and different features make every tree different, so their errors cancel out.
- A single deep tree overfits easily, but the forest generalizes better.
- For classification the majority vote is used, for regression the average of the tree outputs.
- It also gives feature importance, which shows the most useful features.
