Principal Component Analysis (PCA)

Definition of PCA

Principal Component Analysis (PCA) is a statistical technique used for dimensionality reduction. It transforms a high-dimensional dataset into a lower-dimensional subspace while preserving as much variance (information) as possible. PCA identifies new orthogonal axes called principal components.

---

Objectives of PCA

• Reduce dimensionality of data.
• Identify directions (principal components) that capture maximum variance.
• Remove redundancy and noise from data.

---

Step-by-Step Procedure to Perform PCA

Step 0: Mean Centering the Data

Given a dataset with $n$ observations and $d$ features, represented as matrix $X$:

$$X=\left[\begin{array}{cccc}{x}_{11}& x_{12}& \dots & x_{1d}\\ x_{21}& x_{22}& \dots & x_{2d}\\ ⋮& ⋮& \ddots & ⋮\\ x_{n1}& x_{n2}& \dots & x_{nd}\end{array}\right]$$

Compute the mean vector $\mu$:

$${\mu }_j=\frac{1}{n}\sum _{i=1}^n{x}_{ij}$$

Center the data by subtracting the mean from each feature:

$$X_{\text{centered}}=X-\mu$$

---

Step-by-Step PCA Calculation

Step 1: Compute Covariance Matrix

The covariance matrix $C$ measures how variables vary together:

$$C=\frac{1}{n-1}{X}_{\text{centered}}^T{X}_{\text{centered}}$$

$C$ is a symmetric matrix of dimension $d\times d$.

---

Step 2: Eigenvalue Decomposition

Solve the eigenvalue problem for covariance matrix $C$:

$$Cv=\lambda v$$

• $v$: eigenvector (principal component direction)
• $\lambda$: eigenvalue (variance along that direction)

Eigenvectors are orthogonal and normalized:

$$V^{T}V=I$$

---

Step 3: Sort Eigenvalues and Eigenvectors

Sort eigenvalues in descending order:

$${\lambda }_1\ge {\lambda }_2\ge ...\ge {\lambda }_d$$

Select top $k$ eigenvectors corresponding to largest eigenvalues. These form the projection matrix $V_k=[v_1,v_2,...,v_k]$.

---

Step 4: Project Data onto Principal Components

Transform original data $X_{\text{centered}}$ into new subspace using projection matrix $V_k$:

$$Y=X_{\text{centered}}{V}_k$$

• $Y$: projected data in lower-dimensional space ($n\times k$).

---

Explained Variance:

The proportion of variance explained by each principal component is given by:

$$\frac{{\lambda }_i}{{\sum }_{j=1}^d{\lambda }_j}$$

Choosing principal components involves selecting first few components whose cumulative explained variance exceeds a threshold (e.g., 95%).

---

Reconstruction and Reconstruction Error:

Reconstruct original data from reduced data:

Reconstructed data:

$$X_{\text{reconstructed}}=Y{V}_k^T+\mu$$

Reconstruction error (minimized by PCA):

$$\epsilon =||X-X_{\text{reconstructed}}|{|}^2$$

where,$X_{\text{reconstructed}}=Y{V}_k^T+\mu$