{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Week 8: PCA Interactive Demonstrations\n", "## MPS311/439 Machine Learning - Dr. Wei Xing\n", "\n", "This notebook contains 5 interactive demonstrations to help you understand Principal Component Analysis.\n", "\n", "**Instructions**:\n", "- Run all cells in order\n", "- Play with sliders and buttons to see how PCA works\n", "- Observe what happens when you change parameters\n", "\n", "---" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "βœ… All libraries imported successfully!\n", "πŸ“Š Ready for interactive PCA demonstrations\n" ] } ], "source": [ "# Import all required libraries\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", "from matplotlib.patches import FancyArrowPatch, Ellipse\n", "from sklearn.decomposition import PCA\n", "from sklearn.preprocessing import StandardScaler\n", "from sklearn.datasets import load_digits\n", "from ipywidgets import interact, FloatSlider, IntSlider, Checkbox, Button, Dropdown, VBox, HBox, Output\n", "import ipywidgets as widgets\n", "from IPython.display import display, clear_output\n", "\n", "# Set random seed for reproducibility\n", "np.random.seed(42)\n", "\n", "# Set matplotlib style\n", "plt.rcParams['figure.figsize'] = (10, 6)\n", "plt.rcParams['font.size'] = 10\n", "\n", "print(\"βœ… All libraries imported successfully!\")\n", "print(\"πŸ“Š Ready for interactive PCA demonstrations\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## Demo 1: Interactive 2D PCA - Understanding Variance Directions\n", "\n", "**Goal**: See how principal components align with data spread\n", "\n", "**What to observe**:\n", "- PC1 (red arrow) always points in the direction of maximum variance\n", "- PC2 (blue arrow) is always perpendicular to PC1\n", "- Arrow lengths represent eigenvalues (variance in each direction)\n", "\n", "**πŸ’‘ Try this**:\n", "1. Set covariance to 0 β†’ PCs align with X/Y axes\n", "2. Increase covariance to 0.9 β†’ PC1 points diagonally\n", "3. Make variances equal with high covariance β†’ see dramatic PC1, tiny PC2\n", "\n", "---" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "95318544bab04d57a5614187b9852dad", "version_major": 2, "version_minor": 0 }, "text/plain": [ "interactive(children=(FloatSlider(value=2.0, continuous_update=False, description='Var X:', max=5.0, min=0.5),…" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Demo 1: Interactive 2D PCA with covariance control\n", "\n", "def demo1_pca_2d(var_x=2.0, var_y=1.0, covariance=0.7, n_samples=300):\n", " \"\"\"\n", " Visualize how PCA finds principal components in 2D data.\n", " \"\"\"\n", " # Generate correlated 2D data\n", " mean = [0, 0]\n", " cov = [[var_x, covariance * np.sqrt(var_x * var_y)],\n", " [covariance * np.sqrt(var_x * var_y), var_y]]\n", " \n", " data = np.random.multivariate_normal(mean, cov, n_samples)\n", " \n", " # Perform PCA\n", " pca = PCA(n_components=2)\n", " pca.fit(data)\n", " \n", " # Get principal components and eigenvalues\n", " pc1 = pca.components_[0]\n", " pc2 = pca.components_[1]\n", " eigenval1 = pca.explained_variance_[0]\n", " eigenval2 = pca.explained_variance_[1]\n", " \n", " # Create figure\n", " fig, ax = plt.subplots(figsize=(10, 8))\n", " \n", " # Plot data points\n", " ax.scatter(data[:, 0], data[:, 1], alpha=0.4, s=30, color='steelblue', label='Data points')\n", " \n", " # Plot principal components as arrows (scaled by sqrt of eigenvalues)\n", " scale = 2.5\n", " arrow1 = FancyArrowPatch((0, 0), (pc1[0] * np.sqrt(eigenval1) * scale, pc1[1] * np.sqrt(eigenval1) * scale),\n", " color='red', linewidth=3, arrowstyle='->', mutation_scale=20,\n", " label=f'PC1 (λ₁={eigenval1:.2f})')\n", " arrow2 = FancyArrowPatch((0, 0), (pc2[0] * np.sqrt(eigenval2) * scale, pc2[1] * np.sqrt(eigenval2) * scale),\n", " color='blue', linewidth=3, arrowstyle='->', mutation_scale=20,\n", " label=f'PC2 (Ξ»β‚‚={eigenval2:.2f})')\n", " ax.add_patch(arrow1)\n", " ax.add_patch(arrow2)\n", " \n", " # Add grid lines at origin\n", " ax.axhline(y=0, color='k', linewidth=0.5, alpha=0.3)\n", " ax.axvline(x=0, color='k', linewidth=0.5, alpha=0.3)\n", " \n", " # Set axis properties\n", " max_range = max(np.abs(data).max(), 5)\n", " ax.set_xlim(-max_range, max_range)\n", " ax.set_ylim(-max_range, max_range)\n", " ax.set_xlabel('Feature 1', fontsize=12)\n", " ax.set_ylabel('Feature 2', fontsize=12)\n", " ax.set_title('PCA: Finding Directions of Maximum Variance', fontsize=14, fontweight='bold')\n", " ax.legend(loc='upper right', fontsize=10)\n", " ax.grid(True, alpha=0.3)\n", " ax.set_aspect('equal')\n", " \n", " # Add text box with variance explained\n", " var_ratio1 = pca.explained_variance_ratio_[0]\n", " var_ratio2 = pca.explained_variance_ratio_[1]\n", " textstr = f'PC1 explains {var_ratio1:.1%} of variance\\nPC2 explains {var_ratio2:.1%} of variance'\n", " props = dict(boxstyle='round', facecolor='wheat', alpha=0.8)\n", " ax.text(0.02, 0.98, textstr, transform=ax.transAxes, fontsize=11,\n", " verticalalignment='top', bbox=props)\n", " \n", " plt.tight_layout()\n", " plt.show()\n", "\n", "# Create interactive widgets\n", "interact(demo1_pca_2d,\n", " var_x=FloatSlider(value=2.0, min=0.5, max=5.0, step=0.1, description='Var X:', continuous_update=False),\n", " var_y=FloatSlider(value=1.0, min=0.5, max=5.0, step=0.1, description='Var Y:', continuous_update=False),\n", " covariance=FloatSlider(value=0.7, min=-0.95, max=0.95, step=0.05, description='Covariance:', continuous_update=False),\n", " n_samples=IntSlider(value=300, min=100, max=500, step=50, description='N samples:', continuous_update=False));" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## Demo 2: Component Selection - How Many Components?\n", "\n", "**Goal**: Learn how to choose the number of components\n", "\n", "**What to observe**:\n", "- First few components capture most variance\n", "- Cumulative variance shows total information kept\n", "- 90-95% variance is often sufficient\n", "\n", "**πŸ’‘ Try this**:\n", "1. Move slider to see how many components needed for 90% variance\n", "2. Look for the \"elbow\" in the variance plot\n", "3. Notice diminishing returns after ~20 components\n", "\n", "---" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "38ba1dc50e1243d1a0309c8888a499aa", "version_major": 2, "version_minor": 0 }, "text/plain": [ "interactive(children=(IntSlider(value=20, continuous_update=False, description='Components:', max=64, min=1), …" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Demo 2: Component selection with scree plot\n", "\n", "# Load digits dataset (this loads once)\n", "digits = load_digits()\n", "X_digits = digits.data\n", "\n", "# Fit PCA with all components\n", "pca_full = PCA()\n", "pca_full.fit(X_digits)\n", "cumulative_variance = np.cumsum(pca_full.explained_variance_ratio_)\n", "\n", "def demo2_component_selection(n_components=20, show_type='Both', show_thresholds=True):\n", " \"\"\"\n", " Visualize variance explained by different numbers of components.\n", " \"\"\"\n", " n_total = len(pca_full.explained_variance_ratio_)\n", " \n", " if show_type == 'Both':\n", " fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(14, 5))\n", " else:\n", " fig, ax1 = plt.subplots(figsize=(10, 6))\n", " \n", " # Plot 1: Individual variance (bar chart)\n", " if show_type in ['Individual Variance', 'Both']:\n", " n_show = min(30, n_total)\n", " colors = ['red' if i < n_components else 'steelblue' for i in range(n_show)]\n", " \n", " if show_type == 'Both':\n", " ax = ax1\n", " else:\n", " ax = ax1\n", " \n", " ax.bar(range(1, n_show + 1), pca_full.explained_variance_ratio_[:n_show], \n", " color=colors, alpha=0.7, edgecolor='black')\n", " ax.set_xlabel('Principal Component', fontsize=12)\n", " ax.set_ylabel('Explained Variance Ratio', fontsize=12)\n", " ax.set_title('Variance Explained by Each Component', fontsize=13, fontweight='bold')\n", " ax.grid(True, alpha=0.3, axis='y')\n", " \n", " # Add legend\n", " from matplotlib.patches import Patch\n", " legend_elements = [Patch(facecolor='red', label='Selected'),\n", " Patch(facecolor='steelblue', label='Not selected')]\n", " ax.legend(handles=legend_elements, loc='upper right')\n", " \n", " # Plot 2: Cumulative variance (line plot)\n", " if show_type in ['Cumulative Variance', 'Both']:\n", " if show_type == 'Both':\n", " ax = ax2\n", " else:\n", " ax = ax1\n", " \n", " ax.plot(range(1, n_total + 1), cumulative_variance, 'b-', linewidth=2.5, label='Cumulative variance')\n", " \n", " # Highlight selected number of components\n", " ax.plot(n_components, cumulative_variance[n_components - 1], 'ro', markersize=10, \n", " label=f'k={n_components}')\n", " \n", " # Add threshold lines\n", " if show_thresholds:\n", " ax.axhline(y=0.90, color='green', linestyle='--', linewidth=1.5, alpha=0.7, label='90% threshold')\n", " ax.axhline(y=0.95, color='orange', linestyle='--', linewidth=1.5, alpha=0.7, label='95% threshold')\n", " \n", " # Find where we cross thresholds\n", " idx_90 = np.argmax(cumulative_variance >= 0.90) + 1\n", " idx_95 = np.argmax(cumulative_variance >= 0.95) + 1\n", " \n", " ax.plot(idx_90, 0.90, 'go', markersize=8)\n", " ax.plot(idx_95, 0.95, 'o', color='orange', markersize=8)\n", " \n", " ax.set_xlabel('Number of Components', fontsize=12)\n", " ax.set_ylabel('Cumulative Explained Variance', fontsize=12)\n", " ax.set_title('Cumulative Variance Explained', fontsize=13, fontweight='bold')\n", " ax.grid(True, alpha=0.3)\n", " ax.legend(fontsize=9)\n", " ax.set_xlim(0, n_total + 1)\n", " ax.set_ylim(0, 1.05)\n", " \n", " # Add info box\n", " var_kept = cumulative_variance[n_components - 1]\n", " reduction = (1 - n_components / 64) * 100\n", " \n", " info_text = f\"Components: {n_components}/64\\n\"\n", " info_text += f\"Variance kept: {var_kept:.1%}\\n\"\n", " info_text += f\"Reduction: {reduction:.0f}%\"\n", " \n", " fig.text(0.5, 0.02, info_text, ha='center', fontsize=11,\n", " bbox=dict(boxstyle='round', facecolor='lightblue', alpha=0.8))\n", " \n", " plt.tight_layout()\n", " plt.subplots_adjust(bottom=0.15)\n", " plt.show()\n", "\n", "# Create interactive widgets\n", "interact(demo2_component_selection,\n", " n_components=IntSlider(value=20, min=1, max=64, step=1, description='Components:', continuous_update=False),\n", " show_type=Dropdown(options=['Individual Variance', 'Cumulative Variance', 'Both'], \n", " value='Both', description='Show:'),\n", " show_thresholds=Checkbox(value=True, description='Show 90%/95% lines'));" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## Demo 3: Reconstruction Quality - Information Loss vs. Compression\n", "\n", "**Goal**: See how much information is lost with different numbers of components\n", "\n", "**What to observe**:\n", "- With few components (k=1-5): Very blurry, basic shape only\n", "- With moderate components (k=10-20): Good reconstruction\n", "- With many components (k=30+): Almost identical to original\n", "- Difference map shows what's lost (often just noise/fine details)\n", "\n", "**πŸ’‘ Try this**:\n", "1. Start at k=1 and slowly increase to see improvement\n", "2. Enable difference map to see what information is discarded\n", "3. Find the sweet spot where reconstruction is \"good enough\"\n", "\n", "---" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "08410221d1ca42149f58c621d177f3b5", "version_major": 2, "version_minor": 0 }, "text/plain": [ "interactive(children=(IntSlider(value=10, continuous_update=False, description='Components:', max=64, min=1), …" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Demo 3: Reconstruction quality visualization\n", "\n", "def demo3_reconstruction(n_components=10, digit_idx=0, show_difference=False):\n", " \"\"\"\n", " Visualize reconstruction quality with different numbers of components.\n", " \"\"\"\n", " # Apply PCA with specified number of components\n", " pca = PCA(n_components=n_components)\n", " X_reduced = pca.fit_transform(X_digits)\n", " X_reconstructed = pca.inverse_transform(X_reduced)\n", " \n", " # Get original and reconstructed images\n", " original = X_digits[digit_idx].reshape(8, 8)\n", " reconstructed = X_reconstructed[digit_idx].reshape(8, 8)\n", " difference = original - reconstructed\n", " \n", " # Calculate metrics\n", " mse = np.mean((original - reconstructed) ** 2)\n", " variance_kept = pca.explained_variance_ratio_.sum()\n", " \n", " # Create figure\n", " if show_difference:\n", " fig, axes = plt.subplots(1, 3, figsize=(12, 4))\n", " else:\n", " fig, axes = plt.subplots(1, 2, figsize=(10, 4))\n", " \n", " # Original image\n", " axes[0].imshow(original, cmap='gray')\n", " axes[0].set_title('Original Digit', fontsize=13, fontweight='bold')\n", " axes[0].axis('off')\n", " \n", " # Reconstructed image\n", " axes[1].imshow(reconstructed, cmap='gray')\n", " axes[1].set_title(f'Reconstructed ({n_components} components)', fontsize=13, fontweight='bold')\n", " axes[1].axis('off')\n", " \n", " # Difference map (if enabled)\n", " if show_difference:\n", " im = axes[2].imshow(np.abs(difference), cmap='hot')\n", " axes[2].set_title('Difference (What Was Lost)', fontsize=13, fontweight='bold')\n", " axes[2].axis('off')\n", " plt.colorbar(im, ax=axes[2], fraction=0.046)\n", " \n", " # Add info text\n", " info_text = f\"Components: {n_components}/64\\n\"\n", " info_text += f\"Variance kept: {variance_kept:.1%}\\n\"\n", " info_text += f\"Reconstruction error (MSE): {mse:.4f}\\n\"\n", " info_text += f\"True label: {digits.target[digit_idx]}\"\n", " \n", " fig.text(0.5, 0.02, info_text, ha='center', fontsize=11,\n", " bbox=dict(boxstyle='round', facecolor='lightgreen', alpha=0.8))\n", " \n", " plt.tight_layout()\n", " plt.subplots_adjust(bottom=0.2)\n", " plt.show()\n", "\n", "# Create interactive widgets\n", "interact(demo3_reconstruction,\n", " n_components=IntSlider(value=10, min=1, max=64, step=1, description='Components:', continuous_update=False),\n", " digit_idx=IntSlider(value=0, min=0, max=len(X_digits)-1, step=1, description='Image #:', continuous_update=False),\n", " show_difference=Checkbox(value=False, description='Show difference map'));" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## Demo 4: 2D/3D Visualization - Seeing High-Dimensional Data\n", "\n", "**Goal**: Use PCA to visualize 64-dimensional data in 2D/3D\n", "\n", "**What to observe**:\n", "- Without colors: Can you see natural clustering?\n", "- With colors: Different digits cluster together!\n", "- PC1 and PC2 capture meaningful structure (even with ~28% variance)\n", "- 3D adds a bit more separation\n", "\n", "**πŸ’‘ Try this**:\n", "1. Start with \"Color by labels\" OFF - do you see structure?\n", "2. Turn colors ON - look how digits naturally separate!\n", "3. Try 3D mode for extra perspective\n", "\n", "---" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "654278b6093943a18f4d2356c2540284", "version_major": 2, "version_minor": 0 }, "text/plain": [ "interactive(children=(Dropdown(description='View:', options=('2D', '3D'), value='2D'), Checkbox(value=True, de…" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Demo 4: 2D/3D projection playground\n", "\n", "def demo4_projection(projection_type='2D', color_by_label=True):\n", " \"\"\"\n", " Visualize digits dataset in 2D or 3D PCA space.\n", " \"\"\"\n", " if projection_type == '2D':\n", " # 2D projection\n", " pca = PCA(n_components=2)\n", " X_proj = pca.fit_transform(X_digits)\n", " \n", " fig, ax = plt.subplots(figsize=(10, 8))\n", " \n", " if color_by_label:\n", " scatter = ax.scatter(X_proj[:, 0], X_proj[:, 1], \n", " c=digits.target, cmap='tab10',\n", " alpha=0.6, s=40, edgecolors='k', linewidth=0.3)\n", " cbar = plt.colorbar(scatter, ax=ax, ticks=range(10))\n", " cbar.set_label('Digit Class', fontsize=11)\n", " else:\n", " ax.scatter(X_proj[:, 0], X_proj[:, 1],\n", " alpha=0.5, s=40, color='steelblue', edgecolors='k', linewidth=0.3)\n", " \n", " var_pc1 = pca.explained_variance_ratio_[0]\n", " var_pc2 = pca.explained_variance_ratio_[1]\n", " total_var = var_pc1 + var_pc2\n", " \n", " ax.set_xlabel(f'PC1 ({var_pc1:.1%} variance)', fontsize=12)\n", " ax.set_ylabel(f'PC2 ({var_pc2:.1%} variance)', fontsize=12)\n", " ax.set_title(f'Digits in 2D PCA Space (Total: {total_var:.1%} variance)', \n", " fontsize=14, fontweight='bold')\n", " ax.grid(True, alpha=0.3)\n", " \n", " else:\n", " # 3D projection\n", " from mpl_toolkits.mplot3d import Axes3D\n", " \n", " pca = PCA(n_components=3)\n", " X_proj = pca.fit_transform(X_digits)\n", " \n", " fig = plt.figure(figsize=(10, 8))\n", " ax = fig.add_subplot(111, projection='3d')\n", " \n", " if color_by_label:\n", " scatter = ax.scatter(X_proj[:, 0], X_proj[:, 1], X_proj[:, 2],\n", " c=digits.target, cmap='tab10',\n", " alpha=0.6, s=30, edgecolors='k', linewidth=0.2)\n", " cbar = plt.colorbar(scatter, ax=ax, ticks=range(10), pad=0.1, shrink=0.8)\n", " cbar.set_label('Digit Class', fontsize=10)\n", " else:\n", " ax.scatter(X_proj[:, 0], X_proj[:, 1], X_proj[:, 2],\n", " alpha=0.5, s=30, color='steelblue', edgecolors='k', linewidth=0.2)\n", " \n", " var_pc1 = pca.explained_variance_ratio_[0]\n", " var_pc2 = pca.explained_variance_ratio_[1]\n", " var_pc3 = pca.explained_variance_ratio_[2]\n", " total_var = var_pc1 + var_pc2 + var_pc3\n", " \n", " ax.set_xlabel(f'PC1 ({var_pc1:.1%})', fontsize=11)\n", " ax.set_ylabel(f'PC2 ({var_pc2:.1%})', fontsize=11)\n", " ax.set_zlabel(f'PC3 ({var_pc3:.1%})', fontsize=11)\n", " ax.set_title(f'Digits in 3D PCA Space (Total: {total_var:.1%} variance)',\n", " fontsize=13, fontweight='bold')\n", " ax.grid(True, alpha=0.3)\n", " \n", " plt.tight_layout()\n", " plt.show()\n", "\n", "# Create interactive widgets\n", "interact(demo4_projection,\n", " projection_type=Dropdown(options=['2D', '3D'], value='2D', description='View:'),\n", " color_by_label=Checkbox(value=True, description='Color by labels'));" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## Demo 5: Standardization Impact - Why It Matters!\n", "\n", "**Goal**: Understand why feature scaling is critical for PCA\n", "\n", "**What to observe**:\n", "- Without standardization + high scale: PC1 dominated by large-scale feature\n", "- With standardization: Both features get fair weight\n", "- The \"⚠️ Warning\" shows when PCA is biased\n", "\n", "**πŸ’‘ Try this**:\n", "1. Disable standardization, set scale factor to 1000\n", " - See PC1 β‰ˆ 100% variance (all from Feature 2!)\n", "2. Enable standardization with same settings\n", " - See balanced variance between PCs\n", "3. Try different scale factors to see the effect\n", "\n", "---" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "d91bd37a19dd478f92694121912bea5b", "version_major": 2, "version_minor": 0 }, "text/plain": [ "interactive(children=(IntSlider(value=100, continuous_update=False, description='Scale Factor:', max=1000, min…" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Demo 5: Standardization impact demonstration\n", "\n", "def demo5_standardization(scale_factor=100, apply_standardization=False, n_samples=300):\n", " \"\"\"\n", " Show the impact of standardization on PCA.\n", " \"\"\"\n", " # Generate data with two features on different scales\n", " np.random.seed(42)\n", " feature1 = np.random.randn(n_samples) * 1.0 # Small variance\n", " feature2 = np.random.randn(n_samples) * scale_factor # Large variance\n", " \n", " # Add some correlation\n", " feature2 = feature2 + feature1 * (scale_factor * 0.3)\n", " \n", " X = np.column_stack([feature1, feature2])\n", " \n", " # Create figure with two panels\n", " fig = plt.figure(figsize=(14, 10))\n", " \n", " # Panel 1: Without standardization\n", " ax1 = plt.subplot(2, 2, 1)\n", " pca_raw = PCA(n_components=2)\n", " X_pca_raw = pca_raw.fit_transform(X)\n", " \n", " ax1.scatter(X[:, 0], X[:, 1], alpha=0.4, s=30, color='steelblue')\n", " \n", " # Draw PCs\n", " for i, (comp, var) in enumerate(zip(pca_raw.components_, pca_raw.explained_variance_)):\n", " color = 'red' if i == 0 else 'blue'\n", " scale = np.sqrt(var) * 0.3\n", " ax1.arrow(X[:, 0].mean(), X[:, 1].mean(), \n", " comp[0]*scale, comp[1]*scale,\n", " head_width=scale*0.1, head_length=scale*0.15, \n", " fc=color, ec=color, linewidth=2, alpha=0.7,\n", " label=f'PC{i+1}')\n", " \n", " ax1.set_xlabel('Feature 1 (scale=1)', fontsize=11)\n", " ax1.set_ylabel(f'Feature 2 (scale={scale_factor})', fontsize=11)\n", " ax1.set_title('Raw Data (No Standardization)', fontsize=12, fontweight='bold')\n", " ax1.legend()\n", " ax1.grid(True, alpha=0.3)\n", " \n", " # Variance plot for raw\n", " ax2 = plt.subplot(2, 2, 2)\n", " ax2.bar([1, 2], pca_raw.explained_variance_ratio_, color=['red', 'blue'], alpha=0.7, edgecolor='black')\n", " ax2.set_xlabel('Component', fontsize=11)\n", " ax2.set_ylabel('Explained Variance Ratio', fontsize=11)\n", " ax2.set_title('Variance Without Standardization', fontsize=12, fontweight='bold')\n", " ax2.set_xticks([1, 2])\n", " ax2.set_xticklabels(['PC1', 'PC2'])\n", " ax2.grid(True, alpha=0.3, axis='y')\n", " ax2.set_ylim(0, 1.1)\n", " \n", " # Add percentage labels\n", " for i, v in enumerate(pca_raw.explained_variance_ratio_):\n", " ax2.text(i+1, v + 0.02, f'{v:.1%}', ha='center', fontsize=10, fontweight='bold')\n", " \n", " # Panel 2: With standardization\n", " ax3 = plt.subplot(2, 2, 3)\n", " scaler = StandardScaler()\n", " X_scaled = scaler.fit_transform(X)\n", " \n", " pca_scaled = PCA(n_components=2)\n", " X_pca_scaled = pca_scaled.fit_transform(X_scaled)\n", " \n", " ax3.scatter(X_scaled[:, 0], X_scaled[:, 1], alpha=0.4, s=30, color='steelblue')\n", " \n", " # Draw PCs\n", " for i, (comp, var) in enumerate(zip(pca_scaled.components_, pca_scaled.explained_variance_)):\n", " color = 'red' if i == 0 else 'blue'\n", " scale = np.sqrt(var) * 1.5\n", " ax3.arrow(0, 0, comp[0]*scale, comp[1]*scale,\n", " head_width=0.15, head_length=0.2, \n", " fc=color, ec=color, linewidth=2, alpha=0.7,\n", " label=f'PC{i+1}')\n", " \n", " ax3.set_xlabel('Feature 1 (standardized)', fontsize=11)\n", " ax3.set_ylabel('Feature 2 (standardized)', fontsize=11)\n", " ax3.set_title('Standardized Data', fontsize=12, fontweight='bold')\n", " ax3.legend()\n", " ax3.grid(True, alpha=0.3)\n", " ax3.set_aspect('equal')\n", " \n", " # Variance plot for standardized\n", " ax4 = plt.subplot(2, 2, 4)\n", " ax4.bar([1, 2], pca_scaled.explained_variance_ratio_, color=['red', 'blue'], alpha=0.7, edgecolor='black')\n", " ax4.set_xlabel('Component', fontsize=11)\n", " ax4.set_ylabel('Explained Variance Ratio', fontsize=11)\n", " ax4.set_title('Variance With Standardization', fontsize=12, fontweight='bold')\n", " ax4.set_xticks([1, 2])\n", " ax4.set_xticklabels(['PC1', 'PC2'])\n", " ax4.grid(True, alpha=0.3, axis='y')\n", " ax4.set_ylim(0, 1.1)\n", " \n", " # Add percentage labels\n", " for i, v in enumerate(pca_scaled.explained_variance_ratio_):\n", " ax4.text(i+1, v + 0.02, f'{v:.1%}', ha='center', fontsize=10, fontweight='bold')\n", " \n", " # Add warning if not standardized and scale is high\n", " if not apply_standardization and scale_factor > 50:\n", " fig.text(0.5, 0.95, '⚠️ WARNING: PC1 dominated by large-scale feature!', \n", " ha='center', fontsize=13, fontweight='bold', color='red',\n", " bbox=dict(boxstyle='round', facecolor='yellow', alpha=0.8))\n", " \n", " # Add info box\n", " info_text = \"Comparison:\\n\"\n", " info_text += f\"Raw PCA - PC1: {pca_raw.explained_variance_ratio_[0]:.1%}, PC2: {pca_raw.explained_variance_ratio_[1]:.1%}\\n\"\n", " info_text += f\"Standardized PCA - PC1: {pca_scaled.explained_variance_ratio_[0]:.1%}, PC2: {pca_scaled.explained_variance_ratio_[1]:.1%}\"\n", " \n", " fig.text(0.5, 0.02, info_text, ha='center', fontsize=10,\n", " bbox=dict(boxstyle='round', facecolor='lightblue', alpha=0.8))\n", " \n", " plt.tight_layout()\n", " plt.subplots_adjust(top=0.92, bottom=0.12)\n", " plt.show()\n", " \n", " # Print recommendation\n", " if apply_standardization:\n", " print(\"βœ… Using standardization - both features get fair weight!\")\n", " else:\n", " if scale_factor > 50:\n", " print(\"❌ Without standardization, PC1 is biased toward the large-scale feature.\")\n", " print(\" Try enabling standardization to see the difference!\")\n", " else:\n", " print(\"ℹ️ Features have similar scales, standardization has less impact.\")\n", "\n", "# Create interactive widgets\n", "interact(demo5_standardization,\n", " scale_factor=IntSlider(value=100, min=1, max=1000, step=50, description='Scale Factor:', continuous_update=False),\n", " apply_standardization=Checkbox(value=False, description='Apply standardization'),\n", " n_samples=IntSlider(value=300, min=100, max=500, step=50, description='N samples:', continuous_update=False));" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## Summary: What We Explored\n", "\n", "Through these 5 interactive demonstrations, you've learned:\n", "\n", "1. **Demo 1**: How PCA finds directions of maximum variance\n", " - PC1 always aligns with maximum spread\n", " - PC2 is perpendicular with next-most variance\n", " - Eigenvalues tell us importance of each direction\n", "\n", "2. **Demo 2**: How to choose the number of components\n", " - 90-95% variance threshold is common\n", " - Look for \"elbow\" in scree plot\n", " - Often only need 20-30% of components\n", "\n", "3. **Demo 3**: What information is lost/preserved\n", " - Few components: Basic structure only\n", " - Moderate components: Good reconstruction\n", " - Many components: Almost perfect\n", " - Lost information is often just noise!\n", "\n", "4. **Demo 4**: How PCA enables visualization\n", " - Can see 64D data in 2D/3D\n", " - Natural clusters emerge without labels\n", " - Structure revealed even with limited variance\n", "\n", "5. **Demo 5**: Why standardization is critical\n", " - Different feature scales bias PCA\n", " - Always standardize when units differ\n", " - Gives each feature fair weight\n", "\n", "**Key Takeaway**: PCA is a powerful tool for understanding high-dimensional data, but use it wisely!\n", "\n", "---\n", "\n", "### Practice Exercise\n", "\n", "Try applying PCA to your own data:\n", "1. Load your dataset\n", "2. Standardize if features have different scales\n", "3. Apply PCA and examine explained variance\n", "4. Choose appropriate number of components\n", "5. Visualize in 2D or reconstruct data\n", "\n", "Good luck with your PCA journey! πŸŽ‰" ] } ], "metadata": { "kernelspec": { "display_name": "expTorch", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.10" } }, "nbformat": 4, "nbformat_minor": 4 }