{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Week 5: Linear Discriminant Analysis - Interactive Demonstrations\n", "\n", "## MPS311/439 Machine Learning\n", "\n", "This notebook contains three interactive demonstrations to help you understand:\n", "1. How LDA finds the optimal projection direction\n", "2. How LDA compares to Logistic Regression\n", "3. When to use LDA vs QDA\n", "\n", "**Instructions**: Run each cell and use the sliders to explore different scenarios!" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Import required libraries\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", "from sklearn.discriminant_analysis import LinearDiscriminantAnalysis, QuadraticDiscriminantAnalysis\n", "from sklearn.linear_model import LogisticRegression\n", "from sklearn.datasets import make_classification\n", "from sklearn.model_selection import train_test_split\n", "from sklearn.metrics import accuracy_score\n", "import ipywidgets as widgets\n", "from ipywidgets import interact, interactive, fixed\n", "from IPython.display import display\n", "import warnings\n", "warnings.filterwarnings('ignore')\n", "\n", "# Set random seed for reproducibility\n", "np.random.seed(42)\n", "\n", "# Set plot style\n", "plt.style.use('seaborn-v0_8-darkgrid')\n", "%matplotlib inline\n", "\n", "print(\"āœ… All libraries imported successfully!\")\n", "print(\"\\nšŸ“Š Ready for interactive demonstrations!\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "\n", "## Demo 1: LDA Projection Visualization\n", "\n", "### Understanding How LDA Finds the Optimal Projection\n", "\n", "This demo shows:\n", "- Original 2D data with two classes\n", "- The projection direction (arrow) found by LDA\n", "- The decision boundary (perpendicular to projection)\n", "- Projected 1D values as histograms\n", "\n", "**Play with the sliders to see how different data characteristics affect LDA!**" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def plot_lda_projection(n_samples=200, class_sep=1.0, cluster_std=1.0, random_state=42):\n", " \"\"\"\n", " Interactive visualization of LDA projection\n", " \n", " Parameters:\n", " - n_samples: Number of samples per class\n", " - class_sep: Separation between class centers (higher = easier to separate)\n", " - cluster_std: Standard deviation within each class (higher = more spread)\n", " - random_state: Random seed for reproducibility\n", " \"\"\"\n", " \n", " # Generate synthetic 2D data\n", " X, y = make_classification(\n", " n_samples=n_samples*2,\n", " n_features=2,\n", " n_redundant=0,\n", " n_informative=2,\n", " n_clusters_per_class=1,\n", " class_sep=class_sep,\n", " flip_y=0,\n", " random_state=random_state,\n", " shuffle=True\n", " )\n", " \n", " # Scale the cluster spread\n", " X = X * cluster_std\n", " \n", " # Fit LDA\n", " lda = LinearDiscriminantAnalysis()\n", " lda.fit(X, y)\n", " \n", " # Get projection direction (normalized)\n", " w = lda.coef_[0]\n", " w_normalized = w / np.linalg.norm(w)\n", " \n", " # Project data onto LDA direction\n", " X_projected = X @ w.T\n", " \n", " # Create figure with subplots\n", " fig = plt.figure(figsize=(16, 6))\n", " \n", " # Subplot 1: Original 2D data with projection direction\n", " ax1 = plt.subplot(1, 3, 1)\n", " \n", " # Plot data points\n", " scatter1 = ax1.scatter(X[y==0, 0], X[y==0, 1], c='blue', label='Class 0', \n", " alpha=0.6, edgecolors='k', s=50)\n", " scatter2 = ax1.scatter(X[y==1, 0], X[y==1, 1], c='red', label='Class 1', \n", " alpha=0.6, edgecolors='k', s=50)\n", " \n", " # Plot projection direction as arrow\n", " center = X.mean(axis=0)\n", " arrow_scale = 3\n", " ax1.arrow(center[0], center[1], \n", " w_normalized[0]*arrow_scale, w_normalized[1]*arrow_scale,\n", " head_width=0.3, head_length=0.3, fc='green', ec='green', \n", " linewidth=3, label='Projection direction w')\n", " \n", " # Plot decision boundary (perpendicular to w)\n", " x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1\n", " y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1\n", " \n", " xx, yy = np.meshgrid(np.linspace(x_min, x_max, 200),\n", " np.linspace(y_min, y_max, 200))\n", " Z = lda.predict(np.c_[xx.ravel(), yy.ravel()])\n", " Z = Z.reshape(xx.shape)\n", " \n", " ax1.contour(xx, yy, Z, levels=[0.5], colors='black', linewidths=2, \n", " linestyles='--', label='Decision boundary')\n", " ax1.contourf(xx, yy, Z, alpha=0.2, levels=[0, 0.5, 1], colors=['blue', 'red'])\n", " \n", " ax1.set_xlabel('Feature 1', fontsize=12)\n", " ax1.set_ylabel('Feature 2', fontsize=12)\n", " ax1.set_title('Original 2D Data with LDA Projection', fontsize=14, fontweight='bold')\n", " ax1.legend(loc='best')\n", " ax1.grid(True, alpha=0.3)\n", " ax1.set_xlim(x_min, x_max)\n", " ax1.set_ylim(y_min, y_max)\n", " \n", " # Subplot 2: Projection onto line\n", " ax2 = plt.subplot(1, 3, 2)\n", " \n", " # Project points onto the projection direction line\n", " projections_on_line = np.outer(X @ w.T, w_normalized)\n", " \n", " # Plot the projection line\n", " line_extent = max(np.abs(X @ w.T)) * 1.2\n", " line_points = np.array([-line_extent, line_extent])\n", " ax2.plot(center[0] + line_points * w_normalized[0], \n", " center[1] + line_points * w_normalized[1], \n", " 'g-', linewidth=3, label='Projection line')\n", " \n", " # Plot original points (faded)\n", " ax2.scatter(X[y==0, 0], X[y==0, 1], c='blue', alpha=0.2, s=30)\n", " ax2.scatter(X[y==1, 0], X[y==1, 1], c='red', alpha=0.2, s=30)\n", " \n", " # Plot projected points on the line\n", " ax2.scatter(center[0] + projections_on_line[y==0, 0], \n", " center[1] + projections_on_line[y==0, 1], \n", " c='blue', s=50, edgecolors='k', label='Class 0 projected')\n", " ax2.scatter(center[0] + projections_on_line[y==1, 0], \n", " center[1] + projections_on_line[y==1, 1], \n", " c='red', s=50, edgecolors='k', label='Class 1 projected')\n", " \n", " # Draw lines from points to their projections\n", " for i in range(0, len(X), max(1, len(X)//20)): # Draw only subset for clarity\n", " ax2.plot([X[i, 0], center[0] + projections_on_line[i, 0]], \n", " [X[i, 1], center[1] + projections_on_line[i, 1]], \n", " 'gray', alpha=0.3, linewidth=0.5)\n", " \n", " ax2.set_xlabel('Feature 1', fontsize=12)\n", " ax2.set_ylabel('Feature 2', fontsize=12)\n", " ax2.set_title('Data Projected onto LDA Direction', fontsize=14, fontweight='bold')\n", " ax2.legend(loc='best')\n", " ax2.grid(True, alpha=0.3)\n", " ax2.set_xlim(x_min, x_max)\n", " ax2.set_ylim(y_min, y_max)\n", " \n", " # Subplot 3: 1D projected values (histogram)\n", " ax3 = plt.subplot(1, 3, 3)\n", " \n", " # Plot histograms of projected values\n", " ax3.hist(X_projected[y==0], bins=20, color='blue', alpha=0.6, \n", " label='Class 0', edgecolor='black')\n", " ax3.hist(X_projected[y==1], bins=20, color='red', alpha=0.6, \n", " label='Class 1', edgecolor='black')\n", " \n", " # Mark class means\n", " mean0 = X_projected[y==0].mean()\n", " mean1 = X_projected[y==1].mean()\n", " ax3.axvline(mean0, color='blue', linewidth=3, linestyle='--', label=f'Mean 0: {mean0:.2f}')\n", " ax3.axvline(mean1, color='red', linewidth=3, linestyle='--', label=f'Mean 1: {mean1:.2f}')\n", " \n", " # Mark decision threshold\n", " threshold = (mean0 + mean1) / 2\n", " ax3.axvline(threshold, color='black', linewidth=2, linestyle=':', \n", " label=f'Threshold: {threshold:.2f}')\n", " \n", " ax3.set_xlabel('Projected Value (z = w^T x)', fontsize=12)\n", " ax3.set_ylabel('Frequency', fontsize=12)\n", " ax3.set_title('1D Projected Values Distribution', fontsize=14, fontweight='bold')\n", " ax3.legend(loc='best')\n", " ax3.grid(True, alpha=0.3)\n", " \n", " # Calculate and display metrics\n", " separation = abs(mean1 - mean0)\n", " std0 = X_projected[y==0].std()\n", " std1 = X_projected[y==1].std()\n", " fisher_ratio = separation / (std0 + std1)\n", " \n", " # Add text box with metrics\n", " textstr = f'Separation: {separation:.2f}\\nWithin-class std: {(std0+std1)/2:.2f}\\nFisher ratio: {fisher_ratio:.2f}'\n", " props = dict(boxstyle='round', facecolor='wheat', alpha=0.8)\n", " ax3.text(0.05, 0.95, textstr, transform=ax3.transAxes, fontsize=10,\n", " verticalalignment='top', bbox=props)\n", " \n", " plt.tight_layout()\n", " plt.show()\n", " \n", " # Print interpretation\n", " print(f\"\\nšŸ“Š Interpretation:\")\n", " print(f\" ā€¢ Projection direction w = [{w[0]:.3f}, {w[1]:.3f}]\")\n", " print(f\" ā€¢ Between-class separation: {separation:.3f}\")\n", " print(f\" ā€¢ Within-class spread: {(std0+std1)/2:.3f}\")\n", " print(f\" ā€¢ Fisher's ratio (separation/spread): {fisher_ratio:.3f}\")\n", " print(f\"\\nšŸ’” Higher Fisher ratio = Better class separation!\")\n", "\n", "# Create interactive widget\n", "interactive_plot = interactive(\n", " plot_lda_projection,\n", " n_samples=widgets.IntSlider(min=50, max=300, step=50, value=150, \n", " description='Samples/class:', style={'description_width': '150px'}),\n", " class_sep=widgets.FloatSlider(min=0.5, max=3.0, step=0.5, value=1.5, \n", " description='Class separation:', style={'description_width': '150px'}),\n", " cluster_std=widgets.FloatSlider(min=0.5, max=2.0, step=0.25, value=1.0, \n", " description='Cluster spread:', style={'description_width': '150px'}),\n", " random_state=widgets.IntSlider(min=0, max=100, step=1, value=42, \n", " description='Random seed:', style={'description_width': '150px'})\n", ")\n", "\n", "display(interactive_plot)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### šŸŽÆ Key Observations from Demo 1:\n", "\n", "1. **Projection Direction (Green Arrow)**: This is the direction $\\mathbf{w}$ that LDA finds\n", "2. **Decision Boundary (Dashed Line)**: Perpendicular to the projection direction\n", "3. **1D Histograms**: Shows how classes separate after projection\n", "4. **Fisher's Ratio**: Higher values mean better separation!\n", "\n", "**Try this**:\n", "- Increase \"Class separation\" ā†’ See Fisher's ratio increase\n", "- Increase \"Cluster spread\" ā†’ See more overlap, Fisher's ratio decreases\n", "- Change \"Random seed\" ā†’ See different random datasets\n", "\n", "---" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Demo 2: LDA vs Logistic Regression\n", "\n", "### Comparing Two Linear Classifiers\n", "\n", "Both LDA and Logistic Regression create linear decision boundaries, but they find them differently:\n", "- **LDA**: Finds optimal projection direction (assumes Gaussian classes)\n", "- **Logistic Regression**: Directly optimizes decision boundary\n", "\n", "**Explore how they compare under different conditions!**" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def compare_lda_logreg(n_samples=200, class_sep=1.5, cluster_std=1.0, \n", " noise_level=0.0, random_state=42):\n", " \"\"\"\n", " Compare LDA and Logistic Regression decision boundaries\n", " \n", " Parameters:\n", " - n_samples: Number of samples per class\n", " - class_sep: Separation between classes\n", " - cluster_std: Within-class standard deviation\n", " - noise_level: Amount of label noise (0-0.3)\n", " - random_state: Random seed\n", " \"\"\"\n", " \n", " # Generate data\n", " X, y = make_classification(\n", " n_samples=n_samples*2,\n", " n_features=2,\n", " n_redundant=0,\n", " n_informative=2,\n", " n_clusters_per_class=1,\n", " class_sep=class_sep,\n", " flip_y=noise_level,\n", " random_state=random_state,\n", " shuffle=True\n", " )\n", " X = X * cluster_std\n", " \n", " # Split data\n", " X_train, X_test, y_train, y_test = train_test_split(\n", " X, y, test_size=0.3, random_state=random_state\n", " )\n", " \n", " # Fit models\n", " lda = LinearDiscriminantAnalysis()\n", " logreg = LogisticRegression(max_iter=1000)\n", " \n", " lda.fit(X_train, y_train)\n", " logreg.fit(X_train, y_train)\n", " \n", " # Make predictions\n", " lda_pred = lda.predict(X_test)\n", " logreg_pred = logreg.predict(X_test)\n", " \n", " # Calculate accuracies\n", " lda_acc = accuracy_score(y_test, lda_pred)\n", " logreg_acc = accuracy_score(y_test, logreg_pred)\n", " \n", " # Create mesh for decision boundary\n", " x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1\n", " y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1\n", " xx, yy = np.meshgrid(np.linspace(x_min, x_max, 200),\n", " np.linspace(y_min, y_max, 200))\n", " \n", " # Create figure\n", " fig, axes = plt.subplots(1, 3, figsize=(18, 5))\n", " \n", " # Plot 1: LDA\n", " ax1 = axes[0]\n", " Z_lda = lda.predict(np.c_[xx.ravel(), yy.ravel()]).reshape(xx.shape)\n", " ax1.contourf(xx, yy, Z_lda, alpha=0.3, levels=[0, 0.5, 1], colors=['blue', 'red'])\n", " ax1.contour(xx, yy, Z_lda, levels=[0.5], colors='black', linewidths=3, linestyles='-')\n", " \n", " ax1.scatter(X_train[y_train==0, 0], X_train[y_train==0, 1], \n", " c='blue', marker='o', s=50, alpha=0.6, edgecolors='k', label='Train Class 0')\n", " ax1.scatter(X_train[y_train==1, 0], X_train[y_train==1, 1], \n", " c='red', marker='o', s=50, alpha=0.6, edgecolors='k', label='Train Class 1')\n", " ax1.scatter(X_test[y_test==0, 0], X_test[y_test==0, 1], \n", " c='blue', marker='s', s=80, edgecolors='yellow', linewidths=2, label='Test Class 0')\n", " ax1.scatter(X_test[y_test==1, 0], X_test[y_test==1, 1], \n", " c='red', marker='s', s=80, edgecolors='yellow', linewidths=2, label='Test Class 1')\n", " \n", " ax1.set_xlabel('Feature 1', fontsize=12)\n", " ax1.set_ylabel('Feature 2', fontsize=12)\n", " ax1.set_title(f'LDA\\nAccuracy: {lda_acc:.3f}', fontsize=14, fontweight='bold')\n", " ax1.legend(loc='best', fontsize=8)\n", " ax1.grid(True, alpha=0.3)\n", " ax1.set_xlim(x_min, x_max)\n", " ax1.set_ylim(y_min, y_max)\n", " \n", " # Plot 2: Logistic Regression\n", " ax2 = axes[1]\n", " Z_logreg = logreg.predict(np.c_[xx.ravel(), yy.ravel()]).reshape(xx.shape)\n", " ax2.contourf(xx, yy, Z_logreg, alpha=0.3, levels=[0, 0.5, 1], colors=['blue', 'red'])\n", " ax2.contour(xx, yy, Z_logreg, levels=[0.5], colors='black', linewidths=3, linestyles='-')\n", " \n", " ax2.scatter(X_train[y_train==0, 0], X_train[y_train==0, 1], \n", " c='blue', marker='o', s=50, alpha=0.6, edgecolors='k', label='Train Class 0')\n", " ax2.scatter(X_train[y_train==1, 0], X_train[y_train==1, 1], \n", " c='red', marker='o', s=50, alpha=0.6, edgecolors='k', label='Train Class 1')\n", " ax2.scatter(X_test[y_test==0, 0], X_test[y_test==0, 1], \n", " c='blue', marker='s', s=80, edgecolors='yellow', linewidths=2, label='Test Class 0')\n", " ax2.scatter(X_test[y_test==1, 0], X_test[y_test==1, 1], \n", " c='red', marker='s', s=80, edgecolors='yellow', linewidths=2, label='Test Class 1')\n", " \n", " ax2.set_xlabel('Feature 1', fontsize=12)\n", " ax2.set_ylabel('Feature 2', fontsize=12)\n", " ax2.set_title(f'Logistic Regression\\nAccuracy: {logreg_acc:.3f}', fontsize=14, fontweight='bold')\n", " ax2.legend(loc='best', fontsize=8)\n", " ax2.grid(True, alpha=0.3)\n", " ax2.set_xlim(x_min, x_max)\n", " ax2.set_ylim(y_min, y_max)\n", " \n", " # Plot 3: Overlay both boundaries\n", " ax3 = axes[2]\n", " \n", " ax3.scatter(X_train[y_train==0, 0], X_train[y_train==0, 1], \n", " c='blue', marker='o', s=50, alpha=0.6, edgecolors='k', label='Train Class 0')\n", " ax3.scatter(X_train[y_train==1, 0], X_train[y_train==1, 1], \n", " c='red', marker='o', s=50, alpha=0.6, edgecolors='k', label='Train Class 1')\n", " \n", " # Draw both boundaries\n", " ax3.contour(xx, yy, Z_lda, levels=[0.5], colors='blue', linewidths=3, \n", " linestyles='-', label='LDA boundary')\n", " ax3.contour(xx, yy, Z_logreg, levels=[0.5], colors='green', linewidths=3, \n", " linestyles='--', label='LogReg boundary')\n", " \n", " ax3.set_xlabel('Feature 1', fontsize=12)\n", " ax3.set_ylabel('Feature 2', fontsize=12)\n", " ax3.set_title('Both Boundaries Overlaid', fontsize=14, fontweight='bold')\n", " ax3.legend(loc='best')\n", " ax3.grid(True, alpha=0.3)\n", " ax3.set_xlim(x_min, x_max)\n", " ax3.set_ylim(y_min, y_max)\n", " \n", " plt.tight_layout()\n", " plt.show()\n", " \n", " # Print comparison\n", " print(f\"\\nšŸ“Š Model Comparison:\")\n", " print(f\" ā€¢ LDA Test Accuracy: {lda_acc:.4f}\")\n", " print(f\" ā€¢ Logistic Regression Test Accuracy: {logreg_acc:.4f}\")\n", " print(f\" ā€¢ Difference: {abs(lda_acc - logreg_acc):.4f}\")\n", " \n", " if abs(lda_acc - logreg_acc) < 0.02:\n", " print(f\"\\nšŸ’” Both methods perform similarly - boundaries are nearly identical!\")\n", " elif lda_acc > logreg_acc:\n", " print(f\"\\nšŸ’” LDA performs better - data likely follows Gaussian assumptions!\")\n", " else:\n", " print(f\"\\nšŸ’” LogReg performs better - fewer assumptions help with noisy data!\")\n", "\n", "# Create interactive widget\n", "interactive_plot2 = interactive(\n", " compare_lda_logreg,\n", " n_samples=widgets.IntSlider(min=50, max=300, step=50, value=150, \n", " description='Samples/class:', style={'description_width': '150px'}),\n", " class_sep=widgets.FloatSlider(min=0.5, max=3.0, step=0.5, value=1.5, \n", " description='Class separation:', style={'description_width': '150px'}),\n", " cluster_std=widgets.FloatSlider(min=0.5, max=2.0, step=0.25, value=1.0, \n", " description='Cluster spread:', style={'description_width': '150px'}),\n", " noise_level=widgets.FloatSlider(min=0.0, max=0.3, step=0.05, value=0.0, \n", " description='Label noise:', style={'description_width': '150px'}),\n", " random_state=widgets.IntSlider(min=0, max=100, step=1, value=42, \n", " description='Random seed:', style={'description_width': '150px'})\n", ")\n", "\n", "display(interactive_plot2)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### šŸŽÆ Key Observations from Demo 2:\n", "\n", "1. **Similar Performance**: For well-separated Gaussian data, both methods work similarly\n", "2. **Different Boundaries**: The decision boundaries are usually close but not identical\n", "3. **Noise Sensitivity**: Try increasing \"Label noise\" - see how each method handles it\n", "\n", "**Try this**:\n", "- Set \"Label noise\" = 0.0 ā†’ Both methods agree\n", "- Increase \"Label noise\" ā†’ See which method is more robust\n", "- High \"Cluster spread\" + Low \"Class separation\" ā†’ More interesting differences\n", "\n", "---" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Demo 3: LDA vs QDA vs Logistic Regression\n", "\n", "### When Classes Have Different Spreads\n", "\n", "This demo shows what happens when we **violate LDA's equal covariance assumption**:\n", "- **LDA**: Forces linear boundary (may be suboptimal)\n", "- **QDA**: Allows quadratic (curved) boundary\n", "- **Logistic Regression**: Linear boundary (different from LDA)\n", "\n", "**See when QDA's flexibility wins!**" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def compare_lda_qda_logreg(n_samples=150, mean_separation=3.0, \n", " class0_cov_ratio=1.0, class1_cov_ratio=3.0,\n", " rotation_angle=45, random_state=42):\n", " \"\"\"\n", " Compare LDA, QDA, and Logistic Regression with different covariances\n", " \n", " Parameters:\n", " - n_samples: Number of samples per class\n", " - mean_separation: Distance between class centers\n", " - class0_cov_ratio: Ratio of variances for class 0 (x-var / y-var)\n", " - class1_cov_ratio: Ratio of variances for class 1 (x-var / y-var)\n", " - rotation_angle: Rotation angle for covariances (degrees)\n", " - random_state: Random seed\n", " \"\"\"\n", " \n", " np.random.seed(random_state)\n", " \n", " # Create rotation matrix\n", " theta = np.radians(rotation_angle)\n", " rotation = np.array([[np.cos(theta), -np.sin(theta)],\n", " [np.sin(theta), np.cos(theta)]])\n", " \n", " # Create covariance matrices with different shapes\n", " cov0_base = np.array([[class0_cov_ratio, 0], [0, 1.0]])\n", " cov1_base = np.array([[class1_cov_ratio, 0], [0, 1.0]])\n", " \n", " # Apply rotation\n", " cov0 = rotation @ cov0_base @ rotation.T\n", " cov1 = rotation @ cov1_base @ rotation.T\n", " \n", " # Generate data with different covariances\n", " mean0 = np.array([0, 0])\n", " mean1 = np.array([mean_separation, 0])\n", " \n", " X0 = np.random.multivariate_normal(mean0, cov0, n_samples)\n", " X1 = np.random.multivariate_normal(mean1, cov1, n_samples)\n", " \n", " X = np.vstack([X0, X1])\n", " y = np.hstack([np.zeros(n_samples), np.ones(n_samples)])\n", " \n", " # Shuffle\n", " shuffle_idx = np.random.permutation(len(X))\n", " X = X[shuffle_idx]\n", " y = y[shuffle_idx]\n", " \n", " # Split data\n", " X_train, X_test, y_train, y_test = train_test_split(\n", " X, y, test_size=0.3, random_state=random_state\n", " )\n", " \n", " # Fit models\n", " lda = LinearDiscriminantAnalysis()\n", " qda = QuadraticDiscriminantAnalysis()\n", " logreg = LogisticRegression(max_iter=1000)\n", " \n", " lda.fit(X_train, y_train)\n", " qda.fit(X_train, y_train)\n", " logreg.fit(X_train, y_train)\n", " \n", " # Calculate accuracies\n", " lda_acc = accuracy_score(y_test, lda.predict(X_test))\n", " qda_acc = accuracy_score(y_test, qda.predict(X_test))\n", " logreg_acc = accuracy_score(y_test, logreg.predict(X_test))\n", " \n", " # Create mesh\n", " x_min, x_max = X[:, 0].min() - 2, X[:, 0].max() + 2\n", " y_min, y_max = X[:, 1].min() - 2, X[:, 1].max() + 2\n", " xx, yy = np.meshgrid(np.linspace(x_min, x_max, 300),\n", " np.linspace(y_min, y_max, 300))\n", " \n", " # Create figure\n", " fig, axes = plt.subplots(2, 2, figsize=(16, 14))\n", " \n", " models = [('LDA', lda, lda_acc), ('QDA', qda, qda_acc), \n", " ('Logistic Regression', logreg, logreg_acc)]\n", " axes_flat = [axes[0, 0], axes[0, 1], axes[1, 0]]\n", " \n", " for idx, (name, model, acc) in enumerate(models):\n", " ax = axes_flat[idx]\n", " \n", " # Predict on mesh\n", " Z = model.predict(np.c_[xx.ravel(), yy.ravel()]).reshape(xx.shape)\n", " \n", " # Plot decision regions\n", " ax.contourf(xx, yy, Z, alpha=0.3, levels=[0, 0.5, 1], colors=['blue', 'red'])\n", " \n", " # Plot decision boundary\n", " ax.contour(xx, yy, Z, levels=[0.5], colors='black', linewidths=3)\n", " \n", " # Plot training data\n", " ax.scatter(X_train[y_train==0, 0], X_train[y_train==0, 1], \n", " c='blue', marker='o', s=50, alpha=0.6, edgecolors='k', label='Train Class 0')\n", " ax.scatter(X_train[y_train==1, 0], X_train[y_train==1, 1], \n", " c='red', marker='o', s=50, alpha=0.6, edgecolors='k', label='Train Class 1')\n", " \n", " # Plot test data\n", " ax.scatter(X_test[y_test==0, 0], X_test[y_test==0, 1], \n", " c='blue', marker='s', s=80, edgecolors='yellow', linewidths=2, label='Test Class 0')\n", " ax.scatter(X_test[y_test==1, 0], X_test[y_test==1, 1], \n", " c='red', marker='s', s=80, edgecolors='yellow', linewidths=2, label='Test Class 1')\n", " \n", " ax.set_xlabel('Feature 1', fontsize=12)\n", " ax.set_ylabel('Feature 2', fontsize=12)\n", " ax.set_title(f'{name}\\nTest Accuracy: {acc:.3f}', fontsize=14, fontweight='bold')\n", " ax.legend(loc='best', fontsize=9)\n", " ax.grid(True, alpha=0.3)\n", " ax.set_xlim(x_min, x_max)\n", " ax.set_ylim(y_min, y_max)\n", " \n", " # Fourth subplot: All boundaries together\n", " ax4 = axes[1, 1]\n", " \n", " # Plot data\n", " ax4.scatter(X_train[y_train==0, 0], X_train[y_train==0, 1], \n", " c='blue', marker='o', s=50, alpha=0.6, edgecolors='k', label='Train Class 0')\n", " ax4.scatter(X_train[y_train==1, 0], X_train[y_train==1, 1], \n", " c='red', marker='o', s=50, alpha=0.6, edgecolors='k', label='Train Class 1')\n", " \n", " # Plot all boundaries\n", " Z_lda = lda.predict(np.c_[xx.ravel(), yy.ravel()]).reshape(xx.shape)\n", " Z_qda = qda.predict(np.c_[xx.ravel(), yy.ravel()]).reshape(xx.shape)\n", " Z_logreg = logreg.predict(np.c_[xx.ravel(), yy.ravel()]).reshape(xx.shape)\n", " \n", " ax4.contour(xx, yy, Z_lda, levels=[0.5], colors='blue', linewidths=3, \n", " linestyles='-', label='LDA')\n", " ax4.contour(xx, yy, Z_qda, levels=[0.5], colors='green', linewidths=3, \n", " linestyles='-', label='QDA (curved!)')\n", " ax4.contour(xx, yy, Z_logreg, levels=[0.5], colors='orange', linewidths=3, \n", " linestyles='--', label='LogReg')\n", " \n", " ax4.set_xlabel('Feature 1', fontsize=12)\n", " ax4.set_ylabel('Feature 2', fontsize=12)\n", " ax4.set_title('All Decision Boundaries Compared', fontsize=14, fontweight='bold')\n", " ax4.legend(loc='best', fontsize=10)\n", " ax4.grid(True, alpha=0.3)\n", " ax4.set_xlim(x_min, x_max)\n", " ax4.set_ylim(y_min, y_max)\n", " \n", " plt.tight_layout()\n", " plt.show()\n", " \n", " # Print detailed comparison\n", " print(f\"\\nšŸ“Š Model Performance Comparison:\")\n", " print(f\" ā€¢ LDA Test Accuracy: {lda_acc:.4f}\")\n", " print(f\" ā€¢ QDA Test Accuracy: {qda_acc:.4f}\")\n", " print(f\" ā€¢ Logistic Regression Test Accuracy: {logreg_acc:.4f}\")\n", " \n", " print(f\"\\nšŸ“ Covariance Settings:\")\n", " print(f\" ā€¢ Class 0 covariance ratio: {class0_cov_ratio:.1f}\")\n", " print(f\" ā€¢ Class 1 covariance ratio: {class1_cov_ratio:.1f}\")\n", " print(f\" ā€¢ Difference in spreads: {abs(class1_cov_ratio - class0_cov_ratio):.1f}\")\n", " \n", " best_model = max([(lda_acc, 'LDA'), (qda_acc, 'QDA'), (logreg_acc, 'Logistic Regression')])\n", " print(f\"\\nšŸ† Best performing model: {best_model[1]} ({best_model[0]:.4f})\")\n", " \n", " if abs(class1_cov_ratio - class0_cov_ratio) > 1.5:\n", " if qda_acc > max(lda_acc, logreg_acc) + 0.02:\n", " print(f\"\\nšŸ’” QDA wins! The classes have different covariances, so QDA's curved boundary helps!\")\n", " else:\n", " print(f\"\\nšŸ’” Despite different covariances, the classes may be too separated for flexibility to matter.\")\n", " else:\n", " print(f\"\\nšŸ’” Covariances are similar, so all methods perform comparably.\")\n", "\n", "# Create interactive widget\n", "interactive_plot3 = interactive(\n", " compare_lda_qda_logreg,\n", " n_samples=widgets.IntSlider(min=50, max=250, step=50, value=150, \n", " description='Samples/class:', style={'description_width': '150px'}),\n", " mean_separation=widgets.FloatSlider(min=1.0, max=5.0, step=0.5, value=3.0, \n", " description='Mean separation:', style={'description_width': '150px'}),\n", " class0_cov_ratio=widgets.FloatSlider(min=0.5, max=4.0, step=0.5, value=1.0, \n", " description='Class 0 cov ratio:', style={'description_width': '150px'}),\n", " class1_cov_ratio=widgets.FloatSlider(min=0.5, max=4.0, step=0.5, value=3.0, \n", " description='Class 1 cov ratio:', style={'description_width': '150px'}),\n", " rotation_angle=widgets.IntSlider(min=0, max=90, step=15, value=45, \n", " description='Rotation angle:', style={'description_width': '150px'}),\n", " random_state=widgets.IntSlider(min=0, max=100, step=1, value=42, \n", " description='Random seed:', style={'description_width': '150px'})\n", ")\n", "\n", "display(interactive_plot3)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### šŸŽÆ Key Observations from Demo 3:\n", "\n", "1. **QDA's Curved Boundary**: Notice how QDA creates a curved decision boundary!\n", "2. **When QDA Wins**: QDA performs best when classes have very different spreads\n", "3. **LDA's Limitation**: LDA forces a straight line even when a curve would be better\n", "4. **Trade-off**: QDA needs more data (more parameters to estimate)\n", "\n", "**Try this**:\n", "- Set both \"Class 0 cov ratio\" = 1.0 and \"Class 1 cov ratio\" = 1.0 ā†’ All methods agree\n", "- Set \"Class 1 cov ratio\" = 4.0 (very different from Class 0) ā†’ QDA should win!\n", "- Reduce \"Mean separation\" ā†’ See when flexibility matters most\n", "- Change \"Rotation angle\" ā†’ See covariances oriented differently\n", "\n", "---" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## šŸŽ“ Summary and Key Takeaways\n", "\n", "### What We Learned:\n", "\n", "1. **LDA finds optimal projection**:\n", " - Maximizes between-class separation\n", " - Minimizes within-class spread\n", " - Reduces dimensionality automatically\n", "\n", "2. **LDA vs Logistic Regression**:\n", " - Both create linear boundaries\n", " - LDA assumes Gaussian classes\n", " - Usually similar performance, but can differ\n", "\n", "3. **QDA for flexibility**:\n", " - Allows different covariances per class\n", " - Creates curved (quadratic) boundaries\n", " - Needs more training data\n", " - Best when classes have different shapes\n", "\n", "### Decision Guide:\n", "\n", "```\n", "Are classes well-separated and roughly Gaussian?\n", " ā””ā”€ Yes ā†’ Try LDA first (simple, efficient)\n", " ā””ā”€ No ā†’ Try Logistic Regression\n", "\n", "Do classes have very different spreads/shapes?\n", " ā””ā”€ Yes, and you have lots of data ā†’ Try QDA\n", " ā””ā”€ No or limited data ā†’ Stick with LDA\n", "\n", "Not sure?\n", " ā””ā”€ Try all three and compare using cross-validation!\n", "```\n", "\n", "### For Lab This Friday:\n", "\n", "You'll apply these methods to real datasets and practice:\n", "- Using sklearn's LDA and QDA\n", "- Visualizing decision boundaries\n", "- Comparing model performance\n", "- Choosing the right method for your data\n", "\n", "---\n", "\n", "## šŸ“š Additional Exercises (Optional)\n", "\n", "Try modifying the code to:\n", "1. Add a third class and see how LDA handles multi-class problems\n", "2. Create 3D data and project to 2D\n", "3. Compare computational time: LDA vs QDA vs Logistic Regression\n", "4. Implement a simple LDA from scratch using numpy\n", "5. Visualize the confidence/probability contours instead of just boundaries\n", "\n", "**Questions?** Ask in lab or email Dr. Xing!\n", "\n", "---\n", "\n", "*End of Interactive Demonstrations*" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.0" } }, "nbformat": 4, "nbformat_minor": 4 }