{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Week 6: Decision Trees - Interactive Demonstrations\n", "## MPS311/439 Machine Learning - Dr. Wei Xing\n", "\n", "This notebook contains interactive demonstrations for the Week 6 lecture on Decision Trees.\n", "\n", "**Instructions**: Run all cells sequentially. Interact with sliders and buttons to explore concepts." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "✅ All libraries imported successfully!\n" ] } ], "source": [ "# Install required packages (only needed once in Colab)\n", "!pip install ipywidgets -q\n", "\n", "# Import all required libraries\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", "from matplotlib.colors import ListedColormap\n", "import seaborn as sns\n", "from sklearn.datasets import load_iris, load_wine, make_moons, make_classification\n", "from sklearn.tree import DecisionTreeClassifier, plot_tree\n", "from sklearn.linear_model import LogisticRegression\n", "from sklearn.discriminant_analysis import QuadraticDiscriminantAnalysis\n", "from sklearn.model_selection import train_test_split, GridSearchCV\n", "from sklearn.metrics import accuracy_score\n", "from ipywidgets import interact, interactive, fixed, IntSlider, FloatSlider, Dropdown\n", "import ipywidgets as widgets\n", "from IPython.display import display, clear_output\n", "\n", "# Set random seed for reproducibility\n", "np.random.seed(42)\n", "\n", "# Set style\n", "plt.style.use('seaborn-v0_8-darkgrid')\n", "sns.set_palette(\"husl\")\n", "\n", "print(\"✅ All libraries imported successfully!\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## Demo 1: XOR Problem - When Linear Methods Fail\n", "\n", "**Purpose**: Show students a concrete example where linear methods completely fail.\n", "\n", "**Key Learning**: No straight line can separate the XOR pattern, motivating the need for decision trees." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "6d98779ca41a451ebb540d42f60331b4", "version_major": 2, "version_minor": 0 }, "text/plain": [ "interactive(children=(FloatSlider(value=0.1, description='Noise Level:', max=0.3, min=0.05, step=0.05), IntSli…" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "def plot_xor_problem(noise_level=0.1, n_samples=100):\n", " \"\"\"\n", " Generate and visualize XOR problem with linear classifier failure.\n", " \"\"\"\n", " # Generate XOR dataset\n", " np.random.seed(42)\n", " n_per_cluster = n_samples // 4\n", " \n", " # Four clusters arranged in XOR pattern\n", " cluster1 = np.random.randn(n_per_cluster, 2) * noise_level + [0, 0] # Class 0\n", " cluster2 = np.random.randn(n_per_cluster, 2) * noise_level + [1, 1] # Class 0\n", " cluster3 = np.random.randn(n_per_cluster, 2) * noise_level + [0, 1] # Class 1\n", " cluster4 = np.random.randn(n_per_cluster, 2) * noise_level + [1, 0] # Class 1\n", " \n", " X = np.vstack([cluster1, cluster2, cluster3, cluster4])\n", " y = np.array([0]*n_per_cluster*2 + [1]*n_per_cluster*2)\n", " \n", " # Train logistic regression\n", " lr = LogisticRegression(random_state=42)\n", " lr.fit(X, y)\n", " lr_acc = lr.score(X, y)\n", " \n", " # Train decision tree\n", " dt = DecisionTreeClassifier(max_depth=3, random_state=42)\n", " dt.fit(X, y)\n", " dt_acc = dt.score(X, y)\n", " \n", " # Create mesh for decision boundary\n", " h = 0.02\n", " x_min, x_max = X[:, 0].min() - 0.5, X[:, 0].max() + 0.5\n", " y_min, y_max = X[:, 1].min() - 0.5, X[:, 1].max() + 0.5\n", " xx, yy = np.meshgrid(np.arange(x_min, x_max, h),\n", " np.arange(y_min, y_max, h))\n", " \n", " # Plot\n", " fig, axes = plt.subplots(1, 2, figsize=(14, 5))\n", " \n", " # Logistic Regression\n", " Z_lr = lr.predict(np.c_[xx.ravel(), yy.ravel()])\n", " Z_lr = Z_lr.reshape(xx.shape)\n", " axes[0].contourf(xx, yy, Z_lr, alpha=0.3, cmap='RdYlBu')\n", " axes[0].scatter(X[y==0, 0], X[y==0, 1], c='blue', s=50, edgecolors='black', label='Class 0')\n", " axes[0].scatter(X[y==1, 0], X[y==1, 1], c='red', s=50, edgecolors='black', label='Class 1')\n", " axes[0].set_title(f'Logistic Regression FAILS\\nAccuracy: {lr_acc:.1%}', fontsize=14, fontweight='bold')\n", " axes[0].set_xlabel('Feature 1', fontsize=12)\n", " axes[0].set_ylabel('Feature 2', fontsize=12)\n", " axes[0].legend()\n", " axes[0].grid(True, alpha=0.3)\n", " \n", " # Decision Tree\n", " Z_dt = dt.predict(np.c_[xx.ravel(), yy.ravel()])\n", " Z_dt = Z_dt.reshape(xx.shape)\n", " axes[1].contourf(xx, yy, Z_dt, alpha=0.3, cmap='RdYlBu')\n", " axes[1].scatter(X[y==0, 0], X[y==0, 1], c='blue', s=50, edgecolors='black', label='Class 0')\n", " axes[1].scatter(X[y==1, 0], X[y==1, 1], c='red', s=50, edgecolors='black', label='Class 1')\n", " axes[1].set_title(f'Decision Tree SUCCEEDS\\nAccuracy: {dt_acc:.1%}', fontsize=14, fontweight='bold', color='green')\n", " axes[1].set_xlabel('Feature 1', fontsize=12)\n", " axes[1].set_ylabel('Feature 2', fontsize=12)\n", " axes[1].legend()\n", " axes[1].grid(True, alpha=0.3)\n", " \n", " plt.tight_layout()\n", " plt.show()\n", " \n", " print(f\"\\n🔴 Logistic Regression: {lr_acc:.1%} (barely better than random guessing!)\")\n", " print(f\"🟢 Decision Tree: {dt_acc:.1%} (captures the XOR pattern!)\\n\")\n", " print(\"💡 Key Insight: No straight line can separate XOR pattern. Trees use axis-aligned splits to solve it!\")\n", "\n", "# Interactive widget\n", "interact(plot_xor_problem, \n", " noise_level=FloatSlider(value=0.1, min=0.05, max=0.3, step=0.05, description='Noise Level:'),\n", " n_samples=IntSlider(value=100, min=40, max=200, step=20, description='Samples:'));" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## Demo 2: Decision Tree Structure on Iris Dataset\n", "\n", "**Purpose**: Show students how to visualize and interpret a real decision tree.\n", "\n", "**Key Learning**: Understanding tree structure, node information, and prediction paths." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "836eb84396aa43a19c7783aba5440fa6", "version_major": 2, "version_minor": 0 }, "text/plain": [ "interactive(children=(IntSlider(value=3, description='Max Depth:', max=8, min=1), IntSlider(value=2, descripti…" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "def visualize_iris_tree(max_depth=3, min_samples_split=2, min_samples_leaf=1):\n", " \"\"\"\n", " Train and visualize decision tree on Iris dataset.\n", " \"\"\"\n", " # Load Iris dataset\n", " iris = load_iris()\n", " X, y = iris.data, iris.target\n", " \n", " # Split data\n", " X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)\n", " \n", " # Train tree\n", " clf = DecisionTreeClassifier(\n", " max_depth=max_depth,\n", " min_samples_split=min_samples_split,\n", " min_samples_leaf=min_samples_leaf,\n", " random_state=42\n", " )\n", " clf.fit(X_train, y_train)\n", " \n", " # Calculate accuracies\n", " train_acc = clf.score(X_train, y_train)\n", " test_acc = clf.score(X_test, y_test)\n", " \n", " # Visualize tree\n", " fig, ax = plt.subplots(figsize=(20, 10))\n", " plot_tree(clf, \n", " filled=True, \n", " rounded=True,\n", " feature_names=iris.feature_names,\n", " class_names=iris.target_names,\n", " fontsize=10,\n", " ax=ax)\n", " plt.title(f'Decision Tree (depth={max_depth})\\nTrain Acc: {train_acc:.3f} | Test Acc: {test_acc:.3f}', \n", " fontsize=16, fontweight='bold', pad=20)\n", " plt.tight_layout()\n", " plt.show()\n", " \n", " # Feature importance\n", " importances = clf.feature_importances_\n", " feature_names = iris.feature_names\n", " \n", " print(\"\\n📊 Feature Importance:\")\n", " print(\"─\" * 50)\n", " for name, importance in sorted(zip(feature_names, importances), key=lambda x: x[1], reverse=True):\n", " if importance > 0:\n", " print(f\"{name:30s}: {importance:.3f} {'█' * int(importance * 50)}\")\n", " else:\n", " print(f\"{name:30s}: {importance:.3f} (not used)\")\n", " \n", " print(f\"\\n📈 Number of leaves: {clf.get_n_leaves()}\")\n", " print(f\"📏 Actual depth: {clf.get_depth()}\")\n", " \n", " # Show prediction example\n", " sample_idx = 0\n", " sample = X_test[sample_idx].reshape(1, -1)\n", " pred = clf.predict(sample)[0]\n", " true_label = y_test[sample_idx]\n", " \n", " print(f\"\\n🔍 Example Prediction:\")\n", " print(f\"Sample features: {sample[0]}\")\n", " print(f\"Predicted: {iris.target_names[pred]}\")\n", " print(f\"True label: {iris.target_names[true_label]}\")\n", " print(f\"Correct: {'✅' if pred == true_label else '❌'}\")\n", "\n", "# Interactive widget\n", "interact(visualize_iris_tree,\n", " max_depth=IntSlider(value=3, min=1, max=8, step=1, description='Max Depth:'),\n", " min_samples_split=IntSlider(value=2, min=2, max=20, step=2, description='Min Split:'),\n", " min_samples_leaf=IntSlider(value=1, min=1, max=10, step=1, description='Min Leaf:'));" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## Demo 3: Decision Boundaries in 2D\n", "\n", "**Purpose**: Visualize how decision trees create axis-aligned rectangular regions.\n", "\n", "**Key Learning**: Trees split along axes, creating rectangular decision regions." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "af9447be3bae41dab731248ae5697c9c", "version_major": 2, "version_minor": 0 }, "text/plain": [ "interactive(children=(IntSlider(value=3, description='Max Depth:', max=10, min=1), Dropdown(description='Datas…" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "def plot_2d_decision_boundary(max_depth=3, dataset='moons'):\n", " \"\"\"\n", " Visualize decision boundaries for 2D data.\n", " \"\"\"\n", " # Generate dataset\n", " if dataset == 'moons':\n", " X, y = make_moons(n_samples=300, noise=0.25, random_state=42)\n", " title_dataset = \"Moons Dataset\"\n", " elif dataset == 'circles':\n", " from sklearn.datasets import make_circles\n", " X, y = make_circles(n_samples=300, noise=0.15, factor=0.5, random_state=42)\n", " title_dataset = \"Circles Dataset\"\n", " else: # blobs\n", " from sklearn.datasets import make_blobs\n", " X, y = make_blobs(n_samples=300, centers=2, random_state=42, cluster_std=1.5)\n", " title_dataset = \"Blobs Dataset\"\n", " \n", " # Train models\n", " X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)\n", " \n", " dt = DecisionTreeClassifier(max_depth=max_depth, random_state=42)\n", " dt.fit(X_train, y_train)\n", " dt_acc = dt.score(X_test, y_test)\n", " \n", " lr = LogisticRegression(random_state=42)\n", " lr.fit(X_train, y_train)\n", " lr_acc = lr.score(X_test, y_test)\n", " \n", " # Create mesh\n", " h = 0.02\n", " x_min, x_max = X[:, 0].min() - 0.5, X[:, 0].max() + 0.5\n", " y_min, y_max = X[:, 1].min() - 0.5, X[:, 1].max() + 0.5\n", " xx, yy = np.meshgrid(np.arange(x_min, x_max, h),\n", " np.arange(y_min, y_max, h))\n", " \n", " # Plot\n", " fig, axes = plt.subplots(1, 2, figsize=(14, 5))\n", " \n", " # Logistic Regression\n", " Z_lr = lr.predict(np.c_[xx.ravel(), yy.ravel()])\n", " Z_lr = Z_lr.reshape(xx.shape)\n", " axes[0].contourf(xx, yy, Z_lr, alpha=0.4, cmap='RdYlBu')\n", " axes[0].scatter(X_train[y_train==0, 0], X_train[y_train==0, 1], \n", " c='blue', s=50, edgecolors='black', label='Class 0 (train)', alpha=0.7)\n", " axes[0].scatter(X_train[y_train==1, 0], X_train[y_train==1, 1], \n", " c='red', s=50, edgecolors='black', label='Class 1 (train)', alpha=0.7)\n", " axes[0].scatter(X_test[y_test==0, 0], X_test[y_test==0, 1], \n", " c='blue', s=100, marker='s', edgecolors='black', label='Class 0 (test)', alpha=0.9)\n", " axes[0].scatter(X_test[y_test==1, 0], X_test[y_test==1, 1], \n", " c='red', s=100, marker='s', edgecolors='black', label='Class 1 (test)', alpha=0.9)\n", " axes[0].set_title(f'Logistic Regression\\nTest Acc: {lr_acc:.3f}', fontsize=14, fontweight='bold')\n", " axes[0].set_xlabel('Feature 1', fontsize=12)\n", " axes[0].set_ylabel('Feature 2', fontsize=12)\n", " axes[0].legend(loc='best', fontsize=8)\n", " axes[0].grid(True, alpha=0.3)\n", " \n", " # Decision Tree\n", " Z_dt = dt.predict(np.c_[xx.ravel(), yy.ravel()])\n", " Z_dt = Z_dt.reshape(xx.shape)\n", " axes[1].contourf(xx, yy, Z_dt, alpha=0.4, cmap='RdYlBu')\n", " axes[1].scatter(X_train[y_train==0, 0], X_train[y_train==0, 1], \n", " c='blue', s=50, edgecolors='black', label='Class 0 (train)', alpha=0.7)\n", " axes[1].scatter(X_train[y_train==1, 0], X_train[y_train==1, 1], \n", " c='red', s=50, edgecolors='black', label='Class 1 (train)', alpha=0.7)\n", " axes[1].scatter(X_test[y_test==0, 0], X_test[y_test==0, 1], \n", " c='blue', s=100, marker='s', edgecolors='black', label='Class 0 (test)', alpha=0.9)\n", " axes[1].scatter(X_test[y_test==1, 0], X_test[y_test==1, 1], \n", " c='red', s=100, marker='s', edgecolors='black', label='Class 1 (test)', alpha=0.9)\n", " axes[1].set_title(f'Decision Tree (depth={max_depth})\\nTest Acc: {dt_acc:.3f}', \n", " fontsize=14, fontweight='bold')\n", " axes[1].set_xlabel('Feature 1', fontsize=12)\n", " axes[1].set_ylabel('Feature 2', fontsize=12)\n", " axes[1].legend(loc='best', fontsize=8)\n", " axes[1].grid(True, alpha=0.3)\n", " \n", " plt.suptitle(f'{title_dataset}', fontsize=16, fontweight='bold', y=1.02)\n", " plt.tight_layout()\n", " plt.show()\n", " \n", " print(f\"\\n📊 Results on {title_dataset}:\")\n", " print(f\"Logistic Regression Test Accuracy: {lr_acc:.3f}\")\n", " print(f\"Decision Tree Test Accuracy: {dt_acc:.3f}\")\n", " print(f\"\\n💡 Notice: Tree boundary is made of horizontal/vertical lines (axis-aligned splits)!\")\n", "\n", "# Interactive widget\n", "interact(plot_2d_decision_boundary,\n", " max_depth=IntSlider(value=3, min=1, max=10, step=1, description='Max Depth:'),\n", " dataset=Dropdown(options=['moons', 'circles', 'blobs'], value='moons', description='Dataset:'));" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## Demo 4: The Overfitting Problem (MOST IMPORTANT!)\n", "\n", "**Purpose**: Demonstrate how tree depth affects overfitting - the critical concept!\n", "\n", "**Key Learning**: Deep trees memorize training data, leading to poor generalization." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "b48bc1cbbd284114a25b1ba80f4700bf", "version_major": 2, "version_minor": 0 }, "text/plain": [ "interactive(children=(IntSlider(value=5, description='Max Depth:', max=15, min=1), IntSlider(value=300, descri…" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "def demonstrate_overfitting(max_depth=5, dataset_size=300, noise_level=0.3):\n", " \"\"\"\n", " Interactive demonstration of overfitting with tree depth.\n", " Shows both decision boundaries and accuracy curves.\n", " \"\"\"\n", " # Generate dataset\n", " X, y = make_moons(n_samples=dataset_size, noise=noise_level, random_state=42)\n", " X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)\n", " \n", " # Train tree with specified depth\n", " clf = DecisionTreeClassifier(max_depth=max_depth, random_state=42)\n", " clf.fit(X_train, y_train)\n", " train_acc = clf.score(X_train, y_train)\n", " test_acc = clf.score(X_test, y_test)\n", " \n", " # Train trees with different depths for comparison\n", " depths = range(1, 16)\n", " train_accs = []\n", " test_accs = []\n", " \n", " for d in depths:\n", " clf_temp = DecisionTreeClassifier(max_depth=d, random_state=42)\n", " clf_temp.fit(X_train, y_train)\n", " train_accs.append(clf_temp.score(X_train, y_train))\n", " test_accs.append(clf_temp.score(X_test, y_test))\n", " \n", " # Create mesh for decision boundary\n", " h = 0.02\n", " x_min, x_max = X[:, 0].min() - 0.5, X[:, 0].max() + 0.5\n", " y_min, y_max = X[:, 1].min() - 0.5, X[:, 1].max() + 0.5\n", " xx, yy = np.meshgrid(np.arange(x_min, x_max, h),\n", " np.arange(y_min, y_max, h))\n", " \n", " Z = clf.predict(np.c_[xx.ravel(), yy.ravel()])\n", " Z = Z.reshape(xx.shape)\n", " \n", " # Create figure with 2 subplots\n", " fig = plt.figure(figsize=(16, 6))\n", " \n", " # Left: Decision boundary\n", " ax1 = plt.subplot(1, 2, 1)\n", " ax1.contourf(xx, yy, Z, alpha=0.4, cmap='RdYlBu')\n", " ax1.scatter(X_train[y_train==0, 0], X_train[y_train==0, 1], \n", " c='blue', s=50, edgecolors='black', label='Class 0', alpha=0.7)\n", " ax1.scatter(X_train[y_train==1, 0], X_train[y_train==1, 1], \n", " c='red', s=50, edgecolors='black', label='Class 1', alpha=0.7)\n", " ax1.set_xlabel('Feature 1', fontsize=12)\n", " ax1.set_ylabel('Feature 2', fontsize=12)\n", " ax1.legend()\n", " ax1.grid(True, alpha=0.3)\n", " \n", " # Determine if overfitting\n", " gap = train_acc - test_acc\n", " if gap < 0.05:\n", " status = \"Good Balance ✅\"\n", " color = 'green'\n", " elif gap < 0.15:\n", " status = \"Slight Overfitting ⚠️\"\n", " color = 'orange'\n", " else:\n", " status = \"Severe Overfitting ❌\"\n", " color = 'red'\n", " \n", " ax1.set_title(f'Decision Boundary (depth={max_depth})\\nTrain: {train_acc:.3f} | Test: {test_acc:.3f}\\n{status}', \n", " fontsize=13, fontweight='bold', color=color)\n", " \n", " # Right: Accuracy vs depth\n", " ax2 = plt.subplot(1, 2, 2)\n", " ax2.plot(depths, train_accs, 'b-o', linewidth=2, markersize=8, label='Training Accuracy')\n", " ax2.plot(depths, test_accs, 'r--s', linewidth=2, markersize=8, label='Test Accuracy')\n", " ax2.axvline(x=max_depth, color='green', linestyle=':', linewidth=2, label=f'Current Depth ({max_depth})')\n", " \n", " # Mark optimal depth\n", " optimal_depth = depths[np.argmax(test_accs)]\n", " ax2.axvline(x=optimal_depth, color='purple', linestyle='--', linewidth=2, label=f'Optimal Depth ({optimal_depth})')\n", " \n", " # Shade regions\n", " ax2.axvspan(1, 3, alpha=0.1, color='yellow', label='Underfitting')\n", " ax2.axvspan(optimal_depth-1, optimal_depth+1, alpha=0.1, color='green', label='Sweet Spot')\n", " ax2.axvspan(10, 15, alpha=0.1, color='red', label='Overfitting')\n", " \n", " ax2.set_xlabel('Tree Depth', fontsize=12)\n", " ax2.set_ylabel('Accuracy', fontsize=12)\n", " ax2.set_title('Accuracy vs Tree Depth\\n(Watch the Gap!)', fontsize=13, fontweight='bold')\n", " ax2.legend(loc='best', fontsize=9)\n", " ax2.grid(True, alpha=0.3)\n", " ax2.set_ylim([0.5, 1.05])\n", " ax2.set_xticks(depths)\n", " \n", " plt.tight_layout()\n", " plt.show()\n", " \n", " print(f\"\\n📊 Current Configuration:\")\n", " print(f\" Tree Depth: {max_depth}\")\n", " print(f\" Training Accuracy: {train_acc:.3f}\")\n", " print(f\" Test Accuracy: {test_acc:.3f}\")\n", " print(f\" Train-Test Gap: {gap:.3f}\")\n", " print(f\" Status: {status}\")\n", " print(f\"\\n🎯 Optimal Depth (best test acc): {optimal_depth}\")\n", " print(f\"\\n💡 Key Insights:\")\n", " if max_depth <= 3:\n", " print(\" - Tree is too shallow (underfitting)\")\n", " print(\" - Both train and test accuracy are suboptimal\")\n", " print(\" - Try increasing max_depth!\")\n", " elif max_depth > 10:\n", " print(\" - Tree is too deep (overfitting)\")\n", " print(\" - Perfect training but poor test accuracy\")\n", " print(\" - The model memorized noise in training data\")\n", " print(\" - Try decreasing max_depth!\")\n", " else:\n", " print(\" - Tree depth is in reasonable range\")\n", " print(\" - Monitor the train-test gap\")\n", " print(\" - Adjust based on the gap size\")\n", "\n", "# Interactive widget\n", "interact(demonstrate_overfitting,\n", " max_depth=IntSlider(value=5, min=1, max=15, step=1, description='Max Depth:'),\n", " dataset_size=IntSlider(value=300, min=100, max=500, step=50, description='Dataset Size:'),\n", " noise_level=FloatSlider(value=0.3, min=0.1, max=0.5, step=0.05, description='Noise Level:'));" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## Demo 5: Feature Importance Analysis\n", "\n", "**Purpose**: Show which features drive the tree's decisions.\n", "\n", "**Key Learning**: Trees automatically rank features by importance." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "3f71252def124a8eb9ce0aba2f94926f", "version_major": 2, "version_minor": 0 }, "text/plain": [ "interactive(children=(IntSlider(value=5, description='Max Depth:', max=10, min=1), Dropdown(description='Datas…" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "def analyze_feature_importance(max_depth=5, dataset='iris'):\n", " \"\"\"\n", " Visualize feature importance for different datasets.\n", " \"\"\"\n", " # Load dataset\n", " if dataset == 'iris':\n", " data = load_iris()\n", " title = \"Iris Dataset\"\n", " else: # wine\n", " data = load_wine()\n", " title = \"Wine Dataset\"\n", " \n", " X, y = data.data, data.target\n", " feature_names = data.feature_names\n", " \n", " # Split and train\n", " X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)\n", " \n", " clf = DecisionTreeClassifier(max_depth=max_depth, random_state=42)\n", " clf.fit(X_train, y_train)\n", " \n", " train_acc = clf.score(X_train, y_train)\n", " test_acc = clf.score(X_test, y_test)\n", " \n", " # Get feature importances\n", " importances = clf.feature_importances_\n", " indices = np.argsort(importances)[::-1]\n", " \n", " # Create figure\n", " fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(16, 6))\n", " \n", " # Left: Bar chart\n", " colors = ['green' if imp > 0 else 'lightgray' for imp in importances[indices]]\n", " bars = ax1.barh(range(len(importances)), importances[indices], color=colors, edgecolor='black')\n", " ax1.set_yticks(range(len(importances)))\n", " ax1.set_yticklabels([feature_names[i] for i in indices], fontsize=10)\n", " ax1.set_xlabel('Importance', fontsize=12)\n", " ax1.set_title(f'Feature Importance\\n{title} (depth={max_depth})', fontsize=13, fontweight='bold')\n", " ax1.invert_yaxis()\n", " ax1.grid(True, alpha=0.3, axis='x')\n", " \n", " # Add value labels\n", " for i, (imp, bar) in enumerate(zip(importances[indices], bars)):\n", " if imp > 0:\n", " ax1.text(imp + 0.01, i, f'{imp:.3f}', va='center', fontsize=9, fontweight='bold')\n", " \n", " # Right: Pie chart (only non-zero importances)\n", " non_zero_indices = [i for i in indices if importances[i] > 0]\n", " if len(non_zero_indices) > 0:\n", " non_zero_importances = [importances[i] for i in non_zero_indices]\n", " non_zero_names = [feature_names[i] for i in non_zero_indices]\n", " \n", " # Wrap long feature names\n", " wrapped_names = [name[:20] + '...' if len(name) > 20 else name for name in non_zero_names]\n", " \n", " wedges, texts, autotexts = ax2.pie(non_zero_importances, \n", " labels=wrapped_names,\n", " autopct='%1.1f%%',\n", " startangle=90,\n", " textprops={'fontsize': 10})\n", " ax2.set_title(f'Feature Importance Distribution\\n(Only features used in splits)', \n", " fontsize=13, fontweight='bold')\n", " \n", " # Make percentage text bold\n", " for autotext in autotexts:\n", " autotext.set_color('white')\n", " autotext.set_fontweight('bold')\n", " else:\n", " ax2.text(0.5, 0.5, 'No features used\\n(tree too shallow)', \n", " ha='center', va='center', fontsize=14, transform=ax2.transAxes)\n", " ax2.set_xlim(0, 1)\n", " ax2.set_ylim(0, 1)\n", " \n", " plt.tight_layout()\n", " plt.show()\n", " \n", " # Print detailed statistics\n", " print(f\"\\n📊 {title} Results:\")\n", " print(f\" Training Accuracy: {train_acc:.3f}\")\n", " print(f\" Test Accuracy: {test_acc:.3f}\")\n", " print(f\"\\n🏆 Feature Importance Ranking:\")\n", " print(\" \" + \"─\" * 60)\n", " for rank, idx in enumerate(indices, 1):\n", " imp = importances[idx]\n", " name = feature_names[idx]\n", " if imp > 0:\n", " bar = '█' * int(imp * 50)\n", " print(f\" {rank:2d}. {name:35s}: {imp:.4f} {bar}\")\n", " else:\n", " print(f\" {rank:2d}. {name:35s}: {imp:.4f} (not used)\")\n", " \n", " print(f\"\\n💡 Interpretation:\")\n", " top_feature = feature_names[indices[0]]\n", " top_importance = importances[indices[0]]\n", " if top_importance > 0:\n", " print(f\" - '{top_feature}' is the most important feature ({top_importance:.1%})\")\n", " print(f\" - It contributes most to reducing impurity across all splits\")\n", " \n", " n_used = sum(1 for imp in importances if imp > 0)\n", " n_total = len(importances)\n", " print(f\" - {n_used}/{n_total} features are actually used in this tree\")\n", " print(f\" - {n_total - n_used} features have zero importance (ignored)\")\n", "\n", "# Interactive widget\n", "interact(analyze_feature_importance,\n", " max_depth=IntSlider(value=5, min=1, max=10, step=1, description='Max Depth:'),\n", " dataset=Dropdown(options=['iris', 'wine'], value='iris', description='Dataset:'));" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## Demo 6: Hyperparameter Tuning with Grid Search\n", "\n", "**Purpose**: Show systematic hyperparameter optimization.\n", "\n", "**Key Learning**: Grid search finds the best combination of hyperparameters." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "c08e21497ede4c92bf440ac78d7c1061", "version_major": 2, "version_minor": 0 }, "text/plain": [ "interactive(children=(Checkbox(value=True, description='Show Heatmap'), Output()), _dom_classes=('widget-inter…" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "def demonstrate_grid_search(show_heatmap=True):\n", " \"\"\"\n", " Demonstrate hyperparameter tuning with GridSearchCV.\n", " \"\"\"\n", " print(\"🔍 Starting Grid Search for Hyperparameter Tuning...\\n\")\n", " \n", " # Load dataset\n", " wine = load_wine()\n", " X, y = wine.data, wine.target\n", " X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)\n", " \n", " # Define parameter grid\n", " param_grid = {\n", " 'max_depth': [3, 5, 7, 10],\n", " 'min_samples_split': [2, 10, 20],\n", " 'min_samples_leaf': [1, 5, 10]\n", " }\n", " \n", " print(\"📋 Parameter Grid:\")\n", " for param, values in param_grid.items():\n", " print(f\" {param}: {values}\")\n", " \n", " total_combinations = np.prod([len(v) for v in param_grid.values()])\n", " print(f\"\\n🔢 Total combinations to test: {total_combinations}\")\n", " print(f\" With 5-fold CV: {total_combinations * 5} trees will be trained!\\n\")\n", " \n", " # Baseline model (default parameters)\n", " baseline = DecisionTreeClassifier(random_state=42)\n", " baseline.fit(X_train, y_train)\n", " baseline_score = baseline.score(X_test, y_test)\n", " \n", " # Grid search\n", " print(\"⏳ Running Grid Search (this may take a few seconds)...\")\n", " grid_search = GridSearchCV(\n", " DecisionTreeClassifier(random_state=42),\n", " param_grid,\n", " cv=5,\n", " scoring='accuracy',\n", " n_jobs=-1,\n", " verbose=0\n", " )\n", " grid_search.fit(X_train, y_train)\n", " \n", " # Best model\n", " best_clf = grid_search.best_estimator_\n", " best_score = best_clf.score(X_test, y_test)\n", " \n", " print(\"✅ Grid Search Complete!\\n\")\n", " \n", " # Results\n", " print(\"=\"*70)\n", " print(\"📊 RESULTS\")\n", " print(\"=\"*70)\n", " print(f\"\\n🔵 Baseline Model (default parameters):\")\n", " print(f\" Test Accuracy: {baseline_score:.4f}\")\n", " print(f\"\\n🟢 Best Model (optimized parameters):\")\n", " print(f\" Test Accuracy: {best_score:.4f}\")\n", " print(f\" Best Parameters: {grid_search.best_params_}\")\n", " print(f\" Best CV Score: {grid_search.best_score_:.4f}\")\n", " print(f\"\\n📈 Improvement: {(best_score - baseline_score)*100:.2f} percentage points\")\n", " \n", " if show_heatmap:\n", " # Create heatmap of results\n", " results = grid_search.cv_results_\n", " \n", " # Extract results for heatmap (fix max_depth vs min_samples_split)\n", " scores = results['mean_test_score'].reshape(len(param_grid['max_depth']), \n", " len(param_grid['min_samples_split']),\n", " len(param_grid['min_samples_leaf']))\n", " \n", " # Average over min_samples_leaf for visualization\n", " scores_2d = scores.mean(axis=2)\n", " \n", " # Plot heatmap\n", " fig, ax = plt.subplots(figsize=(10, 6))\n", " im = ax.imshow(scores_2d, cmap='RdYlGn', aspect='auto', vmin=scores_2d.min(), vmax=scores_2d.max())\n", " \n", " # Set ticks\n", " ax.set_xticks(np.arange(len(param_grid['min_samples_split'])))\n", " ax.set_yticks(np.arange(len(param_grid['max_depth'])))\n", " ax.set_xticklabels(param_grid['min_samples_split'])\n", " ax.set_yticklabels(param_grid['max_depth'])\n", " \n", " # Labels\n", " ax.set_xlabel('min_samples_split', fontsize=12, fontweight='bold')\n", " ax.set_ylabel('max_depth', fontsize=12, fontweight='bold')\n", " ax.set_title('Grid Search Results Heatmap\\n(averaged over min_samples_leaf)', \n", " fontsize=14, fontweight='bold', pad=15)\n", " \n", " # Add text annotations\n", " for i in range(len(param_grid['max_depth'])):\n", " for j in range(len(param_grid['min_samples_split'])):\n", " text = ax.text(j, i, f'{scores_2d[i, j]:.3f}',\n", " ha=\"center\", va=\"center\", color=\"black\", fontweight='bold')\n", " \n", " # Colorbar\n", " cbar = plt.colorbar(im, ax=ax)\n", " cbar.set_label('CV Accuracy', fontsize=11, fontweight='bold')\n", " \n", " plt.tight_layout()\n", " plt.show()\n", " \n", " print(\"\\n💡 Heatmap Interpretation:\")\n", " print(\" - Greener = Better accuracy\")\n", " print(\" - Redder = Worse accuracy\")\n", " print(\" - Look for the greenest cell!\")\n", " \n", " # Top 5 configurations\n", " print(\"\\n🏆 Top 5 Configurations:\")\n", " print(\" \" + \"─\" * 65)\n", " results_df = pd.DataFrame(grid_search.cv_results_)\n", " top_5 = results_df.nsmallest(5, 'rank_test_score')[['params', 'mean_test_score', 'std_test_score']]\n", " \n", " for idx, row in top_5.iterrows():\n", " print(f\" {row['params']}\")\n", " print(f\" → CV Score: {row['mean_test_score']:.4f} (±{row['std_test_score']:.4f})\")\n", " print()\n", "\n", "# Add pandas import at the beginning if not already imported\n", "import pandas as pd\n", "\n", "# Interactive widget\n", "interact(demonstrate_grid_search,\n", " show_heatmap=widgets.Checkbox(value=True, description='Show Heatmap'));" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## Demo 7: Comparing Trees with Logistic Regression and QDA\n", "\n", "**Purpose**: Direct comparison with previous methods to solidify understanding.\n", "\n", "**Key Learning**: When to use which method - trees excel at non-linear patterns." ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "9e5db0400b4f4f7cb8b84f9ddefd01b6", "version_major": 2, "version_minor": 0 }, "text/plain": [ "interactive(children=(Dropdown(description='Dataset Type:', index=1, options=('linear', 'nonlinear'), value='n…" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "def compare_all_methods(dataset_type='nonlinear', noise_level=0.25):\n", " \"\"\"\n", " Compare Decision Tree, Logistic Regression, and QDA on different datasets.\n", " \"\"\"\n", " # Generate dataset based on type\n", " if dataset_type == 'linear':\n", " # Linearly separable data\n", " from sklearn.datasets import make_classification\n", " X, y = make_classification(n_samples=300, n_features=2, n_redundant=0, \n", " n_informative=2, n_clusters_per_class=1,\n", " flip_y=noise_level, random_state=42)\n", " title = \"Linear Dataset\"\n", " else: # nonlinear\n", " X, y = make_moons(n_samples=300, noise=noise_level, random_state=42)\n", " title = \"Non-linear Dataset (Moons)\"\n", " \n", " # Split data\n", " X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)\n", " \n", " # Train all models\n", " models = {\n", " 'Logistic Regression': LogisticRegression(random_state=42),\n", " 'QDA': QuadraticDiscriminantAnalysis(),\n", " 'Decision Tree (depth=3)': DecisionTreeClassifier(max_depth=3, random_state=42),\n", " 'Decision Tree (depth=8)': DecisionTreeClassifier(max_depth=8, random_state=42)\n", " }\n", " \n", " # Train and evaluate\n", " results = {}\n", " for name, model in models.items():\n", " model.fit(X_train, y_train)\n", " train_acc = model.score(X_train, y_train)\n", " test_acc = model.score(X_test, y_test)\n", " results[name] = {'model': model, 'train_acc': train_acc, 'test_acc': test_acc}\n", " \n", " # Create mesh\n", " h = 0.02\n", " x_min, x_max = X[:, 0].min() - 0.5, X[:, 0].max() + 0.5\n", " y_min, y_max = X[:, 1].min() - 0.5, X[:, 1].max() + 0.5\n", " xx, yy = np.meshgrid(np.arange(x_min, x_max, h),\n", " np.arange(y_min, y_max, h))\n", " \n", " # Plot all models\n", " fig, axes = plt.subplots(2, 2, figsize=(14, 12))\n", " axes = axes.ravel()\n", " \n", " for idx, (name, result) in enumerate(results.items()):\n", " model = result['model']\n", " train_acc = result['train_acc']\n", " test_acc = result['test_acc']\n", " \n", " # Predict on mesh\n", " Z = model.predict(np.c_[xx.ravel(), yy.ravel()])\n", " Z = Z.reshape(xx.shape)\n", " \n", " # Plot\n", " axes[idx].contourf(xx, yy, Z, alpha=0.4, cmap='RdYlBu')\n", " axes[idx].scatter(X_train[y_train==0, 0], X_train[y_train==0, 1], \n", " c='blue', s=50, edgecolors='black', alpha=0.7, label='Class 0')\n", " axes[idx].scatter(X_train[y_train==1, 0], X_train[y_train==1, 1], \n", " c='red', s=50, edgecolors='black', alpha=0.7, label='Class 1')\n", " \n", " axes[idx].set_xlabel('Feature 1', fontsize=11)\n", " axes[idx].set_ylabel('Feature 2', fontsize=11)\n", " axes[idx].set_title(f'{name}\\nTrain: {train_acc:.3f} | Test: {test_acc:.3f}', \n", " fontsize=12, fontweight='bold')\n", " axes[idx].legend(loc='best')\n", " axes[idx].grid(True, alpha=0.3)\n", " \n", " plt.suptitle(f'Method Comparison on {title}', fontsize=16, fontweight='bold', y=0.995)\n", " plt.tight_layout()\n", " plt.show()\n", " \n", " # Print comparison table\n", " print(f\"\\n{'='*80}\")\n", " print(f\"📊 COMPARISON RESULTS - {title}\")\n", " print(f\"{'='*80}\\n\")\n", " print(f\"{'Method':<30} {'Train Acc':<12} {'Test Acc':<12} {'Gap':<10} {'Status'}\")\n", " print(\"─\" * 80)\n", " \n", " for name, result in results.items():\n", " train_acc = result['train_acc']\n", " test_acc = result['test_acc']\n", " gap = train_acc - test_acc\n", " \n", " if gap < 0.05:\n", " status = \"✅ Good\"\n", " elif gap < 0.15:\n", " status = \"⚠️ Slight Overfit\"\n", " else:\n", " status = \"❌ Overfitting\"\n", " \n", " print(f\"{name:<30} {train_acc:<12.3f} {test_acc:<12.3f} {gap:<10.3f} {status}\")\n", " \n", " # Find best model\n", " best_name = max(results.items(), key=lambda x: x[1]['test_acc'])[0]\n", " best_acc = results[best_name]['test_acc']\n", " \n", " print(\"\\n\" + \"─\" * 80)\n", " print(f\"🏆 Best Model: {best_name} (Test Acc: {best_acc:.3f})\")\n", " print(\"─\" * 80)\n", " \n", " print(\"\\n💡 Key Insights:\")\n", " if dataset_type == 'linear':\n", " print(\" - On linear data, all methods perform similarly\")\n", " print(\" - Simpler models (LR) are preferred for interpretability\")\n", " print(\" - Trees don't add much value here\")\n", " else:\n", " print(\" - On non-linear data, trees excel!\")\n", " print(\" - Linear methods struggle with curved boundaries\")\n", " print(\" - Deeper trees capture more complexity (but watch for overfitting!)\")\n", " print(\" - QDA can handle some non-linearity, but limited to quadratic\")\n", "\n", "# Interactive widget\n", "interact(compare_all_methods,\n", " dataset_type=Dropdown(options=['linear', 'nonlinear'], value='nonlinear', description='Dataset Type:'),\n", " noise_level=FloatSlider(value=0.25, min=0.1, max=0.5, step=0.05, description='Noise Level:'));" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## Bonus Demo: Interactive Tree Playground\n", "\n", "**Purpose**: Let students freely explore all hyperparameters at once!\n", "\n", "**Key Learning**: Build intuition through hands-on experimentation." ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "🎮 INTERACTIVE DECISION TREE PLAYGROUND\n", "Use the sliders below to explore how different hyperparameters affect the tree!\n", "\n" ] }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "5460cbe04898422ebb01275b9f09e45c", "version_major": 2, "version_minor": 0 }, "text/plain": [ "interactive(children=(IntSlider(value=5, description='Max Depth:', max=15, min=1, style=SliderStyle(descriptio…" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "def tree_playground(max_depth=5, min_samples_split=2, min_samples_leaf=1, \n", " dataset='moons', noise=0.25, n_samples=300):\n", " \"\"\"\n", " Complete interactive playground for decision trees.\n", " Students can adjust all parameters and see results immediately.\n", " \"\"\"\n", " # Generate dataset\n", " if dataset == 'moons':\n", " X, y = make_moons(n_samples=n_samples, noise=noise, random_state=42)\n", " elif dataset == 'circles':\n", " from sklearn.datasets import make_circles\n", " X, y = make_circles(n_samples=n_samples, noise=noise, factor=0.5, random_state=42)\n", " else: # xor\n", " np.random.seed(42)\n", " n_per = n_samples // 4\n", " X = np.vstack([\n", " np.random.randn(n_per, 2) * noise + [0, 0],\n", " np.random.randn(n_per, 2) * noise + [1, 1],\n", " np.random.randn(n_per, 2) * noise + [0, 1],\n", " np.random.randn(n_per, 2) * noise + [1, 0]\n", " ])\n", " y = np.array([0]*n_per*2 + [1]*n_per*2)\n", " \n", " # Split and train\n", " X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)\n", " \n", " clf = DecisionTreeClassifier(\n", " max_depth=max_depth,\n", " min_samples_split=min_samples_split,\n", " min_samples_leaf=min_samples_leaf,\n", " random_state=42\n", " )\n", " clf.fit(X_train, y_train)\n", " train_acc = clf.score(X_train, y_train)\n", " test_acc = clf.score(X_test, y_test)\n", " gap = train_acc - test_acc\n", " \n", " # Create mesh\n", " h = 0.02\n", " x_min, x_max = X[:, 0].min() - 0.5, X[:, 0].max() + 0.5\n", " y_min, y_max = X[:, 1].min() - 0.5, X[:, 1].max() + 0.5\n", " xx, yy = np.meshgrid(np.arange(x_min, x_max, h),\n", " np.arange(y_min, y_max, h))\n", " \n", " Z = clf.predict(np.c_[xx.ravel(), yy.ravel()])\n", " Z = Z.reshape(xx.shape)\n", " \n", " # Create figure\n", " fig = plt.figure(figsize=(16, 6))\n", " \n", " # Left: Decision boundary\n", " ax1 = plt.subplot(1, 2, 1)\n", " ax1.contourf(xx, yy, Z, alpha=0.4, cmap='RdYlBu')\n", " ax1.scatter(X_train[y_train==0, 0], X_train[y_train==0, 1], \n", " c='blue', s=50, edgecolors='black', alpha=0.7, label='Class 0 (train)')\n", " ax1.scatter(X_train[y_train==1, 0], X_train[y_train==1, 1], \n", " c='red', s=50, edgecolors='black', alpha=0.7, label='Class 1 (train)')\n", " ax1.scatter(X_test[y_test==0, 0], X_test[y_test==0, 1], \n", " c='blue', s=120, marker='s', edgecolors='black', linewidth=2, alpha=0.9, label='Class 0 (test)')\n", " ax1.scatter(X_test[y_test==1, 0], X_test[y_test==1, 1], \n", " c='red', s=120, marker='s', edgecolors='black', linewidth=2, alpha=0.9, label='Class 1 (test)')\n", " \n", " # Status color\n", " if gap < 0.05:\n", " status = \"Good Balance ✅\"\n", " color = 'green'\n", " elif gap < 0.15:\n", " status = \"Slight Overfitting ⚠️\"\n", " color = 'orange'\n", " else:\n", " status = \"Severe Overfitting ❌\"\n", " color = 'red'\n", " \n", " ax1.set_xlabel('Feature 1', fontsize=12)\n", " ax1.set_ylabel('Feature 2', fontsize=12)\n", " ax1.set_title(f'Decision Boundary\\nTrain: {train_acc:.3f} | Test: {test_acc:.3f} | Gap: {gap:.3f}\\n{status}', \n", " fontsize=12, fontweight='bold', color=color)\n", " ax1.legend(loc='best', fontsize=9)\n", " ax1.grid(True, alpha=0.3)\n", " \n", " # Right: Tree structure (simplified)\n", " ax2 = plt.subplot(1, 2, 2)\n", " \n", " # Get tree statistics\n", " n_nodes = clf.tree_.node_count\n", " n_leaves = clf.get_n_leaves()\n", " actual_depth = clf.get_depth()\n", " \n", " # Display tree info as text\n", " info_text = f\"\"\"\n", "🌳 TREE STATISTICS\n", "{'='*40}\n", "\n", "Structure:\n", " • Total Nodes: {n_nodes}\n", " • Leaf Nodes: {n_leaves}\n", " • Actual Depth: {actual_depth}\n", "\n", "Hyperparameters:\n", " • max_depth: {max_depth}\n", " • min_samples_split: {min_samples_split}\n", " • min_samples_leaf: {min_samples_leaf}\n", "\n", "Performance:\n", " • Training Accuracy: {train_acc:.4f}\n", " • Test Accuracy: {test_acc:.4f}\n", " • Train-Test Gap: {gap:.4f}\n", "\n", "Dataset:\n", " • Type: {dataset.upper()}\n", " • Training Samples: {len(X_train)}\n", " • Test Samples: {len(X_test)}\n", " • Noise Level: {noise:.2f}\n", "\n", "Status: {status}\n", " \"\"\"\n", " \n", " ax2.text(0.1, 0.5, info_text, transform=ax2.transAxes, \n", " fontsize=11, verticalalignment='center',\n", " fontfamily='monospace',\n", " bbox=dict(boxstyle='round', facecolor='wheat', alpha=0.5))\n", " ax2.axis('off')\n", " \n", " plt.tight_layout()\n", " plt.show()\n", " \n", " # Recommendations\n", " print(\"\\n\" + \"=\"*80)\n", " print(\"💡 RECOMMENDATIONS\")\n", " print(\"=\"*80)\n", " \n", " if gap > 0.15:\n", " print(\"\\n⚠️ High overfitting detected! Try these:\")\n", " print(f\" 1. Decrease max_depth (current: {max_depth}) → try {max(1, max_depth-2)}\")\n", " print(f\" 2. Increase min_samples_split (current: {min_samples_split}) → try {min_samples_split + 10}\")\n", " print(f\" 3. Increase min_samples_leaf (current: {min_samples_leaf}) → try {min_samples_leaf + 5}\")\n", " elif gap < 0.02 and train_acc < 0.85:\n", " print(\"\\n📉 Possible underfitting! Try these:\")\n", " print(f\" 1. Increase max_depth (current: {max_depth}) → try {max_depth + 2}\")\n", " print(f\" 2. Decrease min_samples_split (current: {min_samples_split}) → try {max(2, min_samples_split - 5)}\")\n", " print(f\" 3. Decrease min_samples_leaf (current: {min_samples_leaf}) → try {max(1, min_samples_leaf - 2)}\")\n", " else:\n", " print(\"\\n✅ Model looks good! Current hyperparameters are reasonable.\")\n", " print(\" You could still fine-tune further if needed.\")\n", " \n", " print(\"\\n💭 General Tips:\")\n", " print(\" • Start with shallow trees (depth 3-5) and increase if needed\")\n", " print(\" • Watch the train-test gap more than absolute accuracy\")\n", " print(\" • Use cross-validation for more reliable estimates\")\n", " print(\" • More data helps reduce overfitting\")\n", " print(\"=\"*80)\n", "\n", "# Create interactive widget with all controls\n", "print(\"\\n🎮 INTERACTIVE DECISION TREE PLAYGROUND\")\n", "print(\"Use the sliders below to explore how different hyperparameters affect the tree!\\n\")\n", "\n", "interact(tree_playground,\n", " max_depth=IntSlider(value=5, min=1, max=15, step=1, \n", " description='Max Depth:', \n", " style={'description_width': '150px'}),\n", " min_samples_split=IntSlider(value=2, min=2, max=50, step=2, \n", " description='Min Samples Split:', \n", " style={'description_width': '150px'}),\n", " min_samples_leaf=IntSlider(value=1, min=1, max=20, step=1, \n", " description='Min Samples Leaf:', \n", " style={'description_width': '150px'}),\n", " dataset=Dropdown(options=['moons', 'circles', 'xor'], value='moons', \n", " description='Dataset:', \n", " style={'description_width': '150px'}),\n", " noise=FloatSlider(value=0.25, min=0.1, max=0.5, step=0.05, \n", " description='Noise Level:', \n", " style={'description_width': '150px'}),\n", " n_samples=IntSlider(value=300, min=100, max=500, step=50, \n", " description='Sample Size:', \n", " style={'description_width': '150px'}));" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## Summary and Key Takeaways\n", "\n", "### What We Explored:\n", "\n", "1. **Demo 1 - XOR Problem**: Linear methods fail on non-linear patterns, motivating decision trees\n", "\n", "2. **Demo 2 - Tree Structure**: How to read and interpret decision tree visualizations\n", "\n", "3. **Demo 3 - Decision Boundaries**: Trees create axis-aligned rectangular regions\n", "\n", "4. **Demo 4 - Overfitting**: The critical concept - deep trees memorize training data\n", "\n", "5. **Demo 5 - Feature Importance**: Understanding which features drive predictions\n", "\n", "6. **Demo 6 - Grid Search**: Systematic hyperparameter optimization\n", "\n", "7. **Demo 7 - Method Comparison**: When to use trees vs linear methods\n", "\n", "8. **Bonus - Tree Playground**: Free exploration of all parameters\n", "\n", "### Key Insights:\n", "\n", "✅ **Trees excel at non-linear patterns** where linear methods struggle\n", "\n", "✅ **Overfitting is the main challenge** - always monitor train-test gap\n", "\n", "✅ **Start simple** (depth 3-5) and increase complexity only if needed\n", "\n", "✅ **Hyperparameters matter** - max_depth, min_samples_split, min_samples_leaf\n", "\n", "✅ **Interpretability is a superpower** - can explain any prediction\n", "\n", "### Next Steps:\n", "\n", "1. Try these demos on your own datasets\n", "2. Experiment with different hyperparameter combinations\n", "3. Compare trees with other methods you've learned\n", "4. Read lecture notes Section 8-9 for ensemble methods (MPS439)\n", "5. Prepare for Week 8: PCA and unsupervised learning!\n", "\n", "---\n", "\n", "**Remember**: The goal isn't to memorize formulas, but to build intuition through experimentation! 🚀" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## Additional Exploration (Optional)\n", "\n", "Try modifying the code above to:\n", "- Test on your own datasets\n", "- Add more evaluation metrics (precision, recall, F1-score)\n", "- Compare with Random Forests (ensemble method)\n", "- Visualize the effect of class imbalance\n", "- Implement cost-complexity pruning" ] } ], "metadata": { "kernelspec": { "display_name": "expTorch", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.10" } }, "nbformat": 4, "nbformat_minor": 4 }