{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Week 10: Neural Networks - Interactive Demonstrations\n", "\n", "**MPS311/439 Machine Learning - Dr. Wei Xing**\n", "\n", "This notebook contains interactive demonstrations to help you understand neural networks.\n", "\n", "## Setup\n", "\n", "Run this cell first to install and import required libraries." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Install required packages\n", "!pip install -q ipywidgets\n", "\n", "# Import libraries\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", "from matplotlib.colors import ListedColormap\n", "import ipywidgets as widgets\n", "from IPython.display import display, clear_output\n", "import warnings\n", "warnings.filterwarnings('ignore')\n", "\n", "# For neural networks\n", "from sklearn.datasets import make_circles, make_moons, make_blobs\n", "from sklearn.model_selection import train_test_split\n", "from sklearn.preprocessing import StandardScaler\n", "\n", "# Keras/TensorFlow\n", "import tensorflow as tf\n", "from tensorflow import keras\n", "from keras.models import Sequential\n", "from keras.layers import Dense, Dropout, BatchNormalization\n", "from keras.regularizers import l2\n", "from keras.optimizers import SGD, Adam, RMSprop\n", "\n", "# Set random seeds for reproducibility\n", "np.random.seed(42)\n", "tf.random.set_seed(42)\n", "\n", "print(\"āœ“ All libraries imported successfully!\")\n", "print(f\"āœ“ TensorFlow version: {tf.__version__}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "\n", "## Demo 1: XOR Decision Boundary Evolution šŸŽÆ\n", "\n", "**Purpose**: Watch a neural network learn to solve the XOR problem in real-time!\n", "\n", "**What to observe:**\n", "- How the decision boundary starts random and becomes non-linear\n", "- How loss decreases over epochs\n", "- Effect of different architectures (number of hidden units)\n", "- Effect of learning rate on convergence speed\n", "\n", "**Instructions:**\n", "1. Adjust the sliders to change network architecture and training parameters\n", "2. Click \"Train Network\" to watch it learn\n", "3. Try different settings and observe the differences!" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# XOR Decision Boundary Evolution\n", "\n", "class XORVisualizer:\n", " def __init__(self):\n", " # XOR dataset\n", " self.X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])\n", " self.y = np.array([0, 1, 1, 0])\n", " self.model = None\n", " self.history = None\n", " \n", " def create_model(self, hidden_units, learning_rate):\n", " \"\"\"Create a simple neural network for XOR\"\"\"\n", " model = Sequential([\n", " Dense(hidden_units, input_dim=2, activation='relu'),\n", " Dense(1, activation='sigmoid')\n", " ])\n", " optimizer = Adam(learning_rate=learning_rate)\n", " model.compile(optimizer=optimizer, loss='binary_crossentropy', metrics=['accuracy'])\n", " return model\n", " \n", " def plot_decision_boundary(self, ax, model, epoch, loss):\n", " \"\"\"Plot current decision boundary\"\"\"\n", " # Create mesh\n", " h = 0.02\n", " x_min, x_max = -0.5, 1.5\n", " y_min, y_max = -0.5, 1.5\n", " xx, yy = np.meshgrid(np.arange(x_min, x_max, h),\n", " np.arange(y_min, y_max, h))\n", " \n", " # Predict on mesh\n", " Z = model.predict(np.c_[xx.ravel(), yy.ravel()], verbose=0)\n", " Z = Z.reshape(xx.shape)\n", " \n", " # Plot\n", " ax.contourf(xx, yy, Z, levels=20, cmap='RdBu', alpha=0.6)\n", " ax.contour(xx, yy, Z, levels=[0.5], colors='black', linewidths=2, linestyles='--')\n", " \n", " # Plot XOR points\n", " colors = ['blue' if label == 0 else 'red' for label in self.y]\n", " markers = ['o' if label == 0 else 'x' for label in self.y]\n", " for i, (x, y, c, m) in enumerate(zip(self.X[:, 0], self.X[:, 1], colors, markers)):\n", " ax.scatter(x, y, c=c, marker=m, s=200, edgecolors='black', linewidths=2, zorder=3)\n", " \n", " ax.set_xlim(x_min, x_max)\n", " ax.set_ylim(y_min, y_max)\n", " ax.set_xlabel('$x_1$', fontsize=12)\n", " ax.set_ylabel('$x_2$', fontsize=12)\n", " ax.set_title(f'Epoch {epoch} | Loss: {loss:.4f}', fontsize=14, fontweight='bold')\n", " ax.grid(True, alpha=0.3)\n", " \n", " def train_and_visualize(self, hidden_units, learning_rate, epochs, update_freq):\n", " \"\"\"Train model and show evolution\"\"\"\n", " self.model = self.create_model(hidden_units, learning_rate)\n", " \n", " # Prepare figure\n", " fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(14, 5))\n", " \n", " losses = []\n", " \n", " # Training loop with visualization\n", " for epoch in range(0, epochs + 1, update_freq):\n", " if epoch > 0:\n", " self.model.fit(self.X, self.y, epochs=update_freq, verbose=0)\n", " \n", " # Get current loss\n", " loss = self.model.evaluate(self.X, self.y, verbose=0)[0]\n", " losses.append(loss)\n", " \n", " # Clear and update plots\n", " ax1.clear()\n", " ax2.clear()\n", " \n", " # Plot decision boundary\n", " self.plot_decision_boundary(ax1, self.model, epoch, loss)\n", " \n", " # Plot loss curve\n", " ax2.plot(range(0, epoch + 1, update_freq), losses, 'b-', linewidth=2)\n", " ax2.set_xlabel('Epoch', fontsize=12)\n", " ax2.set_ylabel('Loss', fontsize=12)\n", " ax2.set_title('Training Loss', fontsize=14, fontweight='bold')\n", " ax2.grid(True, alpha=0.3)\n", " ax2.set_xlim(0, epochs)\n", " \n", " plt.tight_layout()\n", " display(fig)\n", " clear_output(wait=True)\n", " \n", " plt.close()\n", " \n", " # Final predictions\n", " predictions = self.model.predict(self.X, verbose=0)\n", " print(\"\\n\" + \"=\"*60)\n", " print(\"FINAL RESULTS\")\n", " print(\"=\"*60)\n", " print(f\"Final Loss: {losses[-1]:.4f}\\n\")\n", " print(\"Predictions:\")\n", " for i, (x, true_y, pred_y) in enumerate(zip(self.X, self.y, predictions)):\n", " print(f\" Input {x} ā†’ Predicted: {pred_y[0]:.4f}, True: {true_y}\")\n", "\n", "# Create visualizer\n", "xor_viz = XORVisualizer()\n", "\n", "# Interactive widgets\n", "hidden_units_slider = widgets.IntSlider(\n", " value=4, min=2, max=16, step=1,\n", " description='Hidden Units:',\n", " style={'description_width': '150px'},\n", " layout=widgets.Layout(width='500px')\n", ")\n", "\n", "learning_rate_slider = widgets.FloatLogSlider(\n", " value=0.1, base=10, min=-3, max=0, step=0.1,\n", " description='Learning Rate:',\n", " style={'description_width': '150px'},\n", " layout=widgets.Layout(width='500px'),\n", " readout_format='.4f'\n", ")\n", "\n", "epochs_slider = widgets.IntSlider(\n", " value=500, min=100, max=2000, step=100,\n", " description='Epochs:',\n", " style={'description_width': '150px'},\n", " layout=widgets.Layout(width='500px')\n", ")\n", "\n", "update_freq_slider = widgets.IntSlider(\n", " value=50, min=10, max=200, step=10,\n", " description='Update Every:',\n", " style={'description_width': '150px'},\n", " layout=widgets.Layout(width='500px')\n", ")\n", "\n", "train_button = widgets.Button(\n", " description='šŸš€ Train Network',\n", " button_style='success',\n", " layout=widgets.Layout(width='200px', height='40px')\n", ")\n", "\n", "def on_train_clicked(b):\n", " print(\"Training started... Watch the decision boundary evolve!\\n\")\n", " xor_viz.train_and_visualize(\n", " hidden_units=hidden_units_slider.value,\n", " learning_rate=learning_rate_slider.value,\n", " epochs=epochs_slider.value,\n", " update_freq=update_freq_slider.value\n", " )\n", "\n", "train_button.on_click(on_train_clicked)\n", "\n", "# Display widgets\n", "print(\"šŸŽÆ XOR DECISION BOUNDARY EVOLUTION\")\n", "print(\"=\"*60)\n", "print(\"Adjust parameters and click 'Train Network' to see learning in action!\\n\")\n", "\n", "display(widgets.VBox([\n", " widgets.HTML(\"

Network Architecture

\"),\n", " hidden_units_slider,\n", " widgets.HTML(\"

Training Parameters

\"),\n", " learning_rate_slider,\n", " epochs_slider,\n", " update_freq_slider,\n", " widgets.HTML(\"
\"),\n", " train_button\n", "]))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "\n", "## Demo 2: Architecture Explorer šŸ“Š\n", "\n", "**Purpose**: Experiment with different network architectures and see their effects!\n", "\n", "**What to observe:**\n", "- How network depth (number of layers) affects performance\n", "- How network width (units per layer) affects capacity\n", "- Effect on training time and accuracy\n", "- Overfitting indicators (train-test gap)\n", "\n", "**Instructions:**\n", "1. Choose a dataset (different complexity levels)\n", "2. Adjust architecture parameters\n", "3. Click \"Train Model\" to see performance\n", "4. Click \"Compare\" to add to comparison plot\n", "5. Try multiple configurations and compare!" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Architecture Explorer\n", "\n", "class ArchitectureExplorer:\n", " def __init__(self):\n", " self.comparison_results = []\n", " self.comparison_colors = ['blue', 'red', 'green', 'orange', 'purple', 'brown']\n", " \n", " def generate_dataset(self, dataset_name, n_samples=1000):\n", " \"\"\"Generate different datasets\"\"\"\n", " if dataset_name == 'XOR':\n", " X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])\n", " y = np.array([0, 1, 1, 0])\n", " # Repeat to have more samples\n", " X = np.tile(X, (250, 1)) + np.random.normal(0, 0.1, (1000, 2))\n", " y = np.tile(y, 250)\n", " elif dataset_name == 'Circles':\n", " X, y = make_circles(n_samples=n_samples, noise=0.1, factor=0.5, random_state=42)\n", " elif dataset_name == 'Moons':\n", " X, y = make_moons(n_samples=n_samples, noise=0.15, random_state=42)\n", " else: # Blobs\n", " X, y = make_blobs(n_samples=n_samples, centers=2, n_features=2, random_state=42)\n", " \n", " return X, y\n", " \n", " def build_model(self, n_layers, units_per_layer, activation, input_dim=2):\n", " \"\"\"Build model with specified architecture\"\"\"\n", " model = Sequential()\n", " \n", " # First hidden layer\n", " model.add(Dense(units_per_layer, input_dim=input_dim, activation=activation))\n", " \n", " # Additional hidden layers\n", " for _ in range(n_layers - 1):\n", " model.add(Dense(units_per_layer, activation=activation))\n", " \n", " # Output layer\n", " model.add(Dense(1, activation='sigmoid'))\n", " \n", " return model\n", " \n", " def train_and_evaluate(self, dataset_name, n_layers, units_per_layer, \n", " activation, epochs, optimizer_name):\n", " \"\"\"Train model and return results\"\"\"\n", " # Generate dataset\n", " X, y = self.generate_dataset(dataset_name)\n", " \n", " # Split data\n", " X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n", " \n", " # Scale features\n", " scaler = StandardScaler()\n", " X_train = scaler.fit_transform(X_train)\n", " X_test = scaler.transform(X_test)\n", " \n", " # Build model\n", " model = self.build_model(n_layers, units_per_layer, activation)\n", " \n", " # Select optimizer\n", " if optimizer_name == 'SGD':\n", " optimizer = SGD(learning_rate=0.01)\n", " elif optimizer_name == 'Adam':\n", " optimizer = Adam(learning_rate=0.001)\n", " else: # RMSprop\n", " optimizer = RMSprop(learning_rate=0.001)\n", " \n", " model.compile(optimizer=optimizer, loss='binary_crossentropy', metrics=['accuracy'])\n", " \n", " # Train\n", " import time\n", " start_time = time.time()\n", " history = model.fit(X_train, y_train, epochs=epochs, batch_size=32,\n", " validation_data=(X_test, y_test), verbose=0)\n", " train_time = time.time() - start_time\n", " \n", " # Evaluate\n", " train_loss, train_acc = model.evaluate(X_train, y_train, verbose=0)\n", " test_loss, test_acc = model.evaluate(X_test, y_test, verbose=0)\n", " \n", " # Count parameters\n", " n_params = model.count_params()\n", " \n", " return {\n", " 'model': model,\n", " 'history': history,\n", " 'train_acc': train_acc,\n", " 'test_acc': test_acc,\n", " 'train_loss': train_loss,\n", " 'test_loss': test_loss,\n", " 'train_time': train_time,\n", " 'n_params': n_params,\n", " 'config': f\"{n_layers}L-{units_per_layer}U-{activation}\",\n", " 'scaler': scaler,\n", " 'X_test': X_test,\n", " 'y_test': y_test\n", " }\n", " \n", " def visualize_results(self, results):\n", " \"\"\"Visualize training results\"\"\"\n", " fig = plt.figure(figsize=(16, 10))\n", " \n", " # Create grid\n", " gs = fig.add_gridspec(3, 3, hspace=0.3, wspace=0.3)\n", " \n", " # Training curves\n", " ax1 = fig.add_subplot(gs[0, :])\n", " ax1.plot(results['history'].history['loss'], label='Training Loss', linewidth=2)\n", " ax1.plot(results['history'].history['val_loss'], label='Validation Loss', linewidth=2)\n", " ax1.set_xlabel('Epoch', fontsize=11)\n", " ax1.set_ylabel('Loss', fontsize=11)\n", " ax1.set_title('Training and Validation Loss', fontsize=13, fontweight='bold')\n", " ax1.legend()\n", " ax1.grid(True, alpha=0.3)\n", " \n", " # Accuracy curves\n", " ax2 = fig.add_subplot(gs[1, :])\n", " ax2.plot(results['history'].history['accuracy'], label='Training Accuracy', linewidth=2)\n", " ax2.plot(results['history'].history['val_accuracy'], label='Validation Accuracy', linewidth=2)\n", " ax2.set_xlabel('Epoch', fontsize=11)\n", " ax2.set_ylabel('Accuracy', fontsize=11)\n", " ax2.set_title('Training and Validation Accuracy', fontsize=13, fontweight='bold')\n", " ax2.legend()\n", " ax2.grid(True, alpha=0.3)\n", " \n", " # Metrics\n", " ax3 = fig.add_subplot(gs[2, 0])\n", " metrics = ['Train Acc', 'Test Acc']\n", " values = [results['train_acc'] * 100, results['test_acc'] * 100]\n", " colors = ['lightblue', 'lightcoral']\n", " bars = ax3.bar(metrics, values, color=colors, edgecolor='black', linewidth=1.5)\n", " ax3.set_ylabel('Accuracy (%)', fontsize=11)\n", " ax3.set_title('Final Accuracy', fontsize=12, fontweight='bold')\n", " ax3.set_ylim(0, 105)\n", " for bar, val in zip(bars, values):\n", " height = bar.get_height()\n", " ax3.text(bar.get_x() + bar.get_width()/2., height + 1,\n", " f'{val:.1f}%', ha='center', va='bottom', fontweight='bold')\n", " \n", " # Training time and parameters\n", " ax4 = fig.add_subplot(gs[2, 1])\n", " ax4.axis('off')\n", " info_text = f\"\"\"\n", " Configuration: {results['config']}\n", " \n", " Training Time: {results['train_time']:.2f}s\n", " Parameters: {results['n_params']:,}\n", " \n", " Final Train Loss: {results['train_loss']:.4f}\n", " Final Test Loss: {results['test_loss']:.4f}\n", " \"\"\"\n", " ax4.text(0.1, 0.5, info_text, fontsize=11, verticalalignment='center',\n", " bbox=dict(boxstyle='round', facecolor='wheat', alpha=0.5))\n", " \n", " # Overfitting indicator\n", " ax5 = fig.add_subplot(gs[2, 2])\n", " ax5.axis('off')\n", " gap = results['train_acc'] - results['test_acc']\n", " if gap < 0.05:\n", " status = \"šŸŸ¢ Good Generalization\"\n", " color = 'lightgreen'\n", " elif gap < 0.15:\n", " status = \"šŸŸ” Slight Overfitting\"\n", " color = 'lightyellow'\n", " else:\n", " status = \"šŸ”“ Overfitting!\"\n", " color = 'lightcoral'\n", " \n", " ax5.text(0.5, 0.5, f\"{status}\\n\\nTrain-Test Gap:\\n{gap*100:.1f}%\",\n", " fontsize=13, ha='center', va='center', fontweight='bold',\n", " bbox=dict(boxstyle='round', facecolor=color, alpha=0.8, pad=1))\n", " \n", " plt.tight_layout()\n", " plt.show()\n", " \n", " print(\"\\n\" + \"=\"*70)\n", " print(\"SUMMARY\")\n", " print(\"=\"*70)\n", " print(f\"Configuration: {results['config']}\")\n", " print(f\"Test Accuracy: {results['test_acc']*100:.2f}%\")\n", " print(f\"Training Time: {results['train_time']:.2f}s\")\n", " print(f\"Parameters: {results['n_params']:,}\")\n", " if gap >= 0.15:\n", " print(\"\\nāš ļø WARNING: Model is overfitting! Consider:\")\n", " print(\" - Reducing model complexity\")\n", " print(\" - Adding regularization\")\n", " print(\" - Getting more training data\")\n", " print(\"=\"*70)\n", " \n", " def add_to_comparison(self, results):\n", " \"\"\"Add results to comparison list\"\"\"\n", " self.comparison_results.append(results)\n", " print(f\"\\nāœ“ Added to comparison ({len(self.comparison_results)} models total)\")\n", " \n", " def plot_comparison(self):\n", " \"\"\"Plot comparison of multiple models\"\"\"\n", " if len(self.comparison_results) == 0:\n", " print(\"No models to compare yet! Train and add some models first.\")\n", " return\n", " \n", " fig, (ax1, ax2, ax3) = plt.subplots(1, 3, figsize=(16, 5))\n", " \n", " # Loss comparison\n", " for i, res in enumerate(self.comparison_results):\n", " color = self.comparison_colors[i % len(self.comparison_colors)]\n", " ax1.plot(res['history'].history['val_loss'], \n", " label=res['config'], color=color, linewidth=2)\n", " ax1.set_xlabel('Epoch', fontsize=11)\n", " ax1.set_ylabel('Validation Loss', fontsize=11)\n", " ax1.set_title('Validation Loss Comparison', fontsize=13, fontweight='bold')\n", " ax1.legend()\n", " ax1.grid(True, alpha=0.3)\n", " \n", " # Accuracy comparison\n", " configs = [res['config'] for res in self.comparison_results]\n", " test_accs = [res['test_acc'] * 100 for res in self.comparison_results]\n", " colors = [self.comparison_colors[i % len(self.comparison_colors)] \n", " for i in range(len(configs))]\n", " bars = ax2.bar(range(len(configs)), test_accs, color=colors, \n", " edgecolor='black', linewidth=1.5)\n", " ax2.set_xticks(range(len(configs)))\n", " ax2.set_xticklabels(configs, rotation=45, ha='right')\n", " ax2.set_ylabel('Test Accuracy (%)', fontsize=11)\n", " ax2.set_title('Test Accuracy Comparison', fontsize=13, fontweight='bold')\n", " ax2.set_ylim(0, 105)\n", " for bar, val in zip(bars, test_accs):\n", " height = bar.get_height()\n", " ax2.text(bar.get_x() + bar.get_width()/2., height + 1,\n", " f'{val:.1f}%', ha='center', va='bottom', fontsize=9)\n", " \n", " # Parameters vs Accuracy\n", " params = [res['n_params'] for res in self.comparison_results]\n", " ax3.scatter(params, test_accs, c=colors, s=200, edgecolors='black', linewidths=2)\n", " for i, config in enumerate(configs):\n", " ax3.annotate(config, (params[i], test_accs[i]), \n", " xytext=(5, 5), textcoords='offset points', fontsize=8)\n", " ax3.set_xlabel('Number of Parameters', fontsize=11)\n", " ax3.set_ylabel('Test Accuracy (%)', fontsize=11)\n", " ax3.set_title('Parameters vs Accuracy', fontsize=13, fontweight='bold')\n", " ax3.grid(True, alpha=0.3)\n", " \n", " plt.tight_layout()\n", " plt.show()\n", " \n", " print(\"\\n\" + \"=\"*70)\n", " print(\"COMPARISON SUMMARY\")\n", " print(\"=\"*70)\n", " for i, res in enumerate(self.comparison_results):\n", " print(f\"{i+1}. {res['config']}: {res['test_acc']*100:.2f}% accuracy, \"\n", " f\"{res['n_params']:,} params, {res['train_time']:.2f}s\")\n", " print(\"=\"*70)\n", " \n", " def clear_comparison(self):\n", " \"\"\"Clear comparison results\"\"\"\n", " self.comparison_results = []\n", " print(\"āœ“ Comparison cleared!\")\n", "\n", "# Create explorer\n", "explorer = ArchitectureExplorer()\n", "\n", "# Widgets\n", "dataset_dropdown = widgets.Dropdown(\n", " options=['XOR', 'Circles', 'Moons', 'Blobs'],\n", " value='Moons',\n", " description='Dataset:',\n", " style={'description_width': '120px'},\n", " layout=widgets.Layout(width='350px')\n", ")\n", "\n", "layers_buttons = widgets.ToggleButtons(\n", " options=[1, 2, 3],\n", " value=1,\n", " description='Hidden Layers:',\n", " style={'description_width': '120px'},\n", " button_style='info'\n", ")\n", "\n", "units_slider = widgets.IntSlider(\n", " value=8, min=2, max=64, step=2,\n", " description='Units/Layer:',\n", " style={'description_width': '120px'},\n", " layout=widgets.Layout(width='450px')\n", ")\n", "\n", "activation_dropdown = widgets.Dropdown(\n", " options=['relu', 'sigmoid', 'tanh'],\n", " value='relu',\n", " description='Activation:',\n", " style={'description_width': '120px'},\n", " layout=widgets.Layout(width='350px')\n", ")\n", "\n", "epochs_slider_arch = widgets.IntSlider(\n", " value=100, min=50, max=500, step=50,\n", " description='Epochs:',\n", " style={'description_width': '120px'},\n", " layout=widgets.Layout(width='450px')\n", ")\n", "\n", "optimizer_dropdown = widgets.Dropdown(\n", " options=['Adam', 'SGD', 'RMSprop'],\n", " value='Adam',\n", " description='Optimizer:',\n", " style={'description_width': '120px'},\n", " layout=widgets.Layout(width='350px')\n", ")\n", "\n", "train_button_arch = widgets.Button(\n", " description='šŸš€ Train Model',\n", " button_style='success',\n", " layout=widgets.Layout(width='150px', height='35px')\n", ")\n", "\n", "compare_button = widgets.Button(\n", " description='šŸ“Š Add to Compare',\n", " button_style='info',\n", " layout=widgets.Layout(width='150px', height='35px')\n", ")\n", "\n", "show_comparison_button = widgets.Button(\n", " description='šŸ“ˆ Show Comparison',\n", " button_style='warning',\n", " layout=widgets.Layout(width='150px', height='35px')\n", ")\n", "\n", "clear_button = widgets.Button(\n", " description='šŸ—‘ļø Clear',\n", " button_style='danger',\n", " layout=widgets.Layout(width='100px', height='35px')\n", ")\n", "\n", "current_results = {'data': None}\n", "\n", "def on_train_arch_clicked(b):\n", " print(\"Training model...\\n\")\n", " results = explorer.train_and_evaluate(\n", " dataset_name=dataset_dropdown.value,\n", " n_layers=layers_buttons.value,\n", " units_per_layer=units_slider.value,\n", " activation=activation_dropdown.value,\n", " epochs=epochs_slider_arch.value,\n", " optimizer_name=optimizer_dropdown.value\n", " )\n", " current_results['data'] = results\n", " explorer.visualize_results(results)\n", "\n", "def on_compare_clicked(b):\n", " if current_results['data'] is not None:\n", " explorer.add_to_comparison(current_results['data'])\n", " else:\n", " print(\"Train a model first!\")\n", "\n", "def on_show_comparison_clicked(b):\n", " explorer.plot_comparison()\n", "\n", "def on_clear_clicked(b):\n", " explorer.clear_comparison()\n", "\n", "train_button_arch.on_click(on_train_arch_clicked)\n", "compare_button.on_click(on_compare_clicked)\n", "show_comparison_button.on_click(on_show_comparison_clicked)\n", "clear_button.on_click(on_clear_clicked)\n", "\n", "# Display\n", "print(\"šŸ“Š ARCHITECTURE EXPLORER\")\n", "print(\"=\"*70)\n", "print(\"Experiment with different architectures and compare their performance!\\n\")\n", "\n", "display(widgets.VBox([\n", " widgets.HTML(\"

Dataset Selection

\"),\n", " dataset_dropdown,\n", " widgets.HTML(\"

Network Architecture

\"),\n", " layers_buttons,\n", " units_slider,\n", " activation_dropdown,\n", " widgets.HTML(\"

Training Parameters

\"),\n", " epochs_slider_arch,\n", " optimizer_dropdown,\n", " widgets.HTML(\"
\"),\n", " widgets.HBox([train_button_arch, compare_button, show_comparison_button, clear_button])\n", "]))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "\n", "## Demo 3: Activation Function Comparison šŸ”§\n", "\n", "**Purpose**: Compare different activation functions and understand their characteristics!\n", "\n", "**What to observe:**\n", "- Shape of different activation functions (sigmoid, ReLU, tanh)\n", "- Their derivatives (important for gradient flow)\n", "- How they affect training speed and final performance\n", "- Which works best for different datasets\n", "\n", "**Instructions:**\n", "1. Select a dataset\n", "2. Configure the network\n", "3. Click \"Train All\" to train with all three activations\n", "4. Compare the results!" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Activation Function Comparison\n", "\n", "class ActivationComparison:\n", " def __init__(self):\n", " self.activations = ['sigmoid', 'relu', 'tanh']\n", " self.colors = {'sigmoid': 'blue', 'relu': 'red', 'tanh': 'green'}\n", " \n", " def plot_activation_functions(self):\n", " \"\"\"Plot activation functions and their derivatives\"\"\"\n", " fig, axes = plt.subplots(1, 3, figsize=(15, 4))\n", " x = np.linspace(-5, 5, 200)\n", " \n", " # Sigmoid\n", " sigmoid = 1 / (1 + np.exp(-x))\n", " sigmoid_deriv = sigmoid * (1 - sigmoid)\n", " axes[0].plot(x, sigmoid, 'b-', linewidth=2.5, label='Sigmoid')\n", " axes[0].plot(x, sigmoid_deriv, 'b--', linewidth=2, alpha=0.7, label=\"Sigmoid'\")\n", " axes[0].set_title('Sigmoid', fontsize=13, fontweight='bold')\n", " axes[0].legend()\n", " axes[0].grid(True, alpha=0.3)\n", " axes[0].axhline(y=0, color='k', linewidth=0.5)\n", " axes[0].axvline(x=0, color='k', linewidth=0.5)\n", " \n", " # ReLU\n", " relu = np.maximum(0, x)\n", " relu_deriv = (x > 0).astype(float)\n", " axes[1].plot(x, relu, 'r-', linewidth=2.5, label='ReLU')\n", " axes[1].plot(x, relu_deriv, 'r--', linewidth=2, alpha=0.7, label=\"ReLU'\")\n", " axes[1].set_title('ReLU', fontsize=13, fontweight='bold')\n", " axes[1].legend()\n", " axes[1].grid(True, alpha=0.3)\n", " axes[1].axhline(y=0, color='k', linewidth=0.5)\n", " axes[1].axvline(x=0, color='k', linewidth=0.5)\n", " \n", " # Tanh\n", " tanh = np.tanh(x)\n", " tanh_deriv = 1 - tanh**2\n", " axes[2].plot(x, tanh, 'g-', linewidth=2.5, label='Tanh')\n", " axes[2].plot(x, tanh_deriv, 'g--', linewidth=2, alpha=0.7, label=\"Tanh'\")\n", " axes[2].set_title('Tanh', fontsize=13, fontweight='bold')\n", " axes[2].legend()\n", " axes[2].grid(True, alpha=0.3)\n", " axes[2].axhline(y=0, color='k', linewidth=0.5)\n", " axes[2].axvline(x=0, color='k', linewidth=0.5)\n", " \n", " for ax in axes:\n", " ax.set_xlabel('Input (z)', fontsize=11)\n", " ax.set_ylabel('Output', fontsize=11)\n", " \n", " plt.tight_layout()\n", " plt.show()\n", " \n", " def train_all_activations(self, dataset_name, n_layers, units, epochs):\n", " \"\"\"Train networks with all three activations\"\"\"\n", " results = {}\n", " \n", " # Generate dataset\n", " if dataset_name == 'Circles':\n", " X, y = make_circles(n_samples=1000, noise=0.1, factor=0.5, random_state=42)\n", " elif dataset_name == 'Moons':\n", " X, y = make_moons(n_samples=1000, noise=0.15, random_state=42)\n", " elif dataset_name == 'Spirals':\n", " # Create spiral dataset\n", " n = 500\n", " theta = np.sqrt(np.random.rand(n)) * 2 * np.pi\n", " r_a = 2 * theta + np.pi\n", " data_a = np.array([np.cos(theta) * r_a, np.sin(theta) * r_a]).T\n", " x_a = data_a + np.random.randn(n, 2) * 0.5\n", " \n", " r_b = -2 * theta - np.pi\n", " data_b = np.array([np.cos(theta) * r_b, np.sin(theta) * r_b]).T\n", " x_b = data_b + np.random.randn(n, 2) * 0.5\n", " \n", " X = np.vstack([x_a, x_b])\n", " y = np.hstack([np.zeros(n), np.ones(n)])\n", " else: # XOR\n", " X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])\n", " y = np.array([0, 1, 1, 0])\n", " X = np.tile(X, (250, 1)) + np.random.normal(0, 0.1, (1000, 2))\n", " y = np.tile(y, 250)\n", " \n", " # Split and scale\n", " X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n", " scaler = StandardScaler()\n", " X_train = scaler.fit_transform(X_train)\n", " X_test = scaler.transform(X_test)\n", " \n", " print(\"Training networks with different activations...\\n\")\n", " \n", " for activation in self.activations:\n", " print(f\"Training with {activation}...\")\n", " \n", " # Build model\n", " model = Sequential()\n", " model.add(Dense(units, input_dim=2, activation=activation))\n", " for _ in range(n_layers - 1):\n", " model.add(Dense(units, activation=activation))\n", " model.add(Dense(1, activation='sigmoid'))\n", " \n", " model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])\n", " \n", " # Train\n", " history = model.fit(X_train, y_train, epochs=epochs, batch_size=32,\n", " validation_data=(X_test, y_test), verbose=0)\n", " \n", " # Evaluate\n", " _, test_acc = model.evaluate(X_test, y_test, verbose=0)\n", " \n", " results[activation] = {\n", " 'model': model,\n", " 'history': history,\n", " 'test_acc': test_acc,\n", " 'X_test': X_test,\n", " 'y_test': y_test\n", " }\n", " \n", " return results, scaler\n", " \n", " def visualize_comparison(self, results):\n", " \"\"\"Visualize comparison of all three activations\"\"\"\n", " fig = plt.figure(figsize=(16, 10))\n", " gs = fig.add_gridspec(2, 3, hspace=0.3, wspace=0.3)\n", " \n", " # Training loss comparison\n", " ax1 = fig.add_subplot(gs[0, :])\n", " for activation in self.activations:\n", " ax1.plot(results[activation]['history'].history['loss'],\n", " label=f'{activation.capitalize()}',\n", " color=self.colors[activation], linewidth=2)\n", " ax1.set_xlabel('Epoch', fontsize=12)\n", " ax1.set_ylabel('Training Loss', fontsize=12)\n", " ax1.set_title('Training Loss Comparison', fontsize=14, fontweight='bold')\n", " ax1.legend(fontsize=11)\n", " ax1.grid(True, alpha=0.3)\n", " \n", " # Decision boundaries\n", " for idx, activation in enumerate(self.activations):\n", " ax = fig.add_subplot(gs[1, idx])\n", " \n", " # Create mesh\n", " X_test = results[activation]['X_test']\n", " x_min, x_max = X_test[:, 0].min() - 0.5, X_test[:, 0].max() + 0.5\n", " y_min, y_max = X_test[:, 1].min() - 0.5, X_test[:, 1].max() + 0.5\n", " xx, yy = np.meshgrid(np.linspace(x_min, x_max, 100),\n", " np.linspace(y_min, y_max, 100))\n", " \n", " # Predict\n", " Z = results[activation]['model'].predict(np.c_[xx.ravel(), yy.ravel()], verbose=0)\n", " Z = Z.reshape(xx.shape)\n", " \n", " # Plot\n", " ax.contourf(xx, yy, Z, levels=20, cmap='RdBu', alpha=0.6)\n", " ax.contour(xx, yy, Z, levels=[0.5], colors='black', linewidths=2)\n", " \n", " # Plot points\n", " y_test = results[activation]['y_test']\n", " ax.scatter(X_test[y_test == 0, 0], X_test[y_test == 0, 1],\n", " c='blue', marker='o', s=50, edgecolors='black', linewidths=1)\n", " ax.scatter(X_test[y_test == 1, 0], X_test[y_test == 1, 1],\n", " c='red', marker='x', s=50, linewidths=2)\n", " \n", " test_acc = results[activation]['test_acc']\n", " ax.set_title(f'{activation.capitalize()}\\nAccuracy: {test_acc*100:.1f}%',\n", " fontsize=12, fontweight='bold')\n", " ax.set_xlabel('Feature 1', fontsize=10)\n", " ax.set_ylabel('Feature 2', fontsize=10)\n", " \n", " plt.tight_layout()\n", " plt.show()\n", " \n", " print(\"\\n\" + \"=\"*70)\n", " print(\"COMPARISON SUMMARY\")\n", " print(\"=\"*70)\n", " for activation in self.activations:\n", " acc = results[activation]['test_acc']\n", " final_loss = results[activation]['history'].history['loss'][-1]\n", " print(f\"{activation.capitalize():8s}: Test Acc = {acc*100:5.2f}%, Final Loss = {final_loss:.4f}\")\n", " print(\"=\"*70)\n", " print(\"\\nšŸ’” Observations:\")\n", " print(\" - ReLU often trains fastest (no saturation)\")\n", " print(\" - Sigmoid can suffer from vanishing gradients\")\n", " print(\" - Tanh is zero-centered (sometimes helps)\")\n", " print(\"=\"*70)\n", "\n", "# Create comparison object\n", "activation_comp = ActivationComparison()\n", "\n", "# First, show activation functions\n", "print(\"šŸ”§ ACTIVATION FUNCTION COMPARISON\")\n", "print(\"=\"*70)\n", "print(\"First, let's visualize the activation functions and their derivatives:\\n\")\n", "activation_comp.plot_activation_functions()\n", "\n", "# Widgets\n", "dataset_act = widgets.Dropdown(\n", " options=['XOR', 'Circles', 'Moons', 'Spirals'],\n", " value='Moons',\n", " description='Dataset:',\n", " style={'description_width': '120px'},\n", " layout=widgets.Layout(width='350px')\n", ")\n", "\n", "layers_act = widgets.IntSlider(\n", " value=2, min=1, max=3, step=1,\n", " description='Hidden Layers:',\n", " style={'description_width': '120px'},\n", " layout=widgets.Layout(width='400px')\n", ")\n", "\n", "units_act = widgets.IntSlider(\n", " value=8, min=4, max=32, step=4,\n", " description='Units/Layer:',\n", " style={'description_width': '120px'},\n", " layout=widgets.Layout(width='400px')\n", ")\n", "\n", "epochs_act = widgets.IntSlider(\n", " value=100, min=50, max=300, step=50,\n", " description='Epochs:',\n", " style={'description_width': '120px'},\n", " layout=widgets.Layout(width='400px')\n", ")\n", "\n", "train_all_button = widgets.Button(\n", " description='šŸš€ Train All Activations',\n", " button_style='success',\n", " layout=widgets.Layout(width='200px', height='40px')\n", ")\n", "\n", "def on_train_all_clicked(b):\n", " results, scaler = activation_comp.train_all_activations(\n", " dataset_name=dataset_act.value,\n", " n_layers=layers_act.value,\n", " units=units_act.value,\n", " epochs=epochs_act.value\n", " )\n", " activation_comp.visualize_comparison(results)\n", "\n", "train_all_button.on_click(on_train_all_clicked)\n", "\n", "print(\"\\n\" + \"=\"*70)\n", "print(\"Now configure and train networks with all three activations:\")\n", "print(\"=\"*70 + \"\\n\")\n", "\n", "display(widgets.VBox([\n", " widgets.HTML(\"

Configuration

\"),\n", " dataset_act,\n", " layers_act,\n", " units_act,\n", " epochs_act,\n", " widgets.HTML(\"
\"),\n", " train_all_button\n", "]))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "\n", "## Demo 4: Training Dynamics Visualizer šŸ“ˆ\n", "\n", "**Purpose**: Monitor training in real-time and understand overfitting!\n", "\n", "**What to observe:**\n", "- How training and validation loss evolve\n", "- When overfitting begins (validation loss increases)\n", "- Effect of regularization (L2, Dropout) on overfitting\n", "- Impact of learning rate on convergence\n", "- Weight distribution during training\n", "\n", "**Instructions:**\n", "1. Adjust dataset size and noise\n", "2. Configure regularization (try with and without)\n", "3. Click \"Start Training\" and watch in real-time\n", "4. Observe overfitting indicators!" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Training Dynamics Visualizer\n", "\n", "class TrainingDynamicsVisualizer:\n", " def __init__(self):\n", " self.training_active = False\n", " \n", " def create_model(self, use_l2, l2_lambda, use_dropout, dropout_rate, use_batchnorm):\n", " \"\"\"Create model with optional regularization\"\"\"\n", " model = Sequential()\n", " \n", " # First layer\n", " if use_l2:\n", " model.add(Dense(32, input_dim=2, activation='relu', kernel_regularizer=l2(l2_lambda)))\n", " else:\n", " model.add(Dense(32, input_dim=2, activation='relu'))\n", " \n", " if use_batchnorm:\n", " model.add(BatchNormalization())\n", " \n", " if use_dropout:\n", " model.add(Dropout(dropout_rate))\n", " \n", " # Second layer\n", " if use_l2:\n", " model.add(Dense(16, activation='relu', kernel_regularizer=l2(l2_lambda)))\n", " else:\n", " model.add(Dense(16, activation='relu'))\n", " \n", " if use_batchnorm:\n", " model.add(BatchNormalization())\n", " \n", " if use_dropout:\n", " model.add(Dropout(dropout_rate))\n", " \n", " # Output layer\n", " model.add(Dense(1, activation='sigmoid'))\n", " \n", " return model\n", " \n", " def train_with_visualization(self, train_size, noise_level, learning_rate,\n", " use_l2, l2_lambda, use_dropout, dropout_rate,\n", " use_batchnorm, max_epochs, update_freq):\n", " \"\"\"Train model with real-time visualization\"\"\"\n", " # Generate dataset\n", " X, y = make_moons(n_samples=train_size, noise=noise_level, random_state=42)\n", " X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.3, random_state=42)\n", " \n", " # Scale\n", " scaler = StandardScaler()\n", " X_train = scaler.fit_transform(X_train)\n", " X_val = scaler.transform(X_val)\n", " \n", " # Create model\n", " model = self.create_model(use_l2, l2_lambda, use_dropout, dropout_rate, use_batchnorm)\n", " optimizer = Adam(learning_rate=learning_rate)\n", " model.compile(optimizer=optimizer, loss='binary_crossentropy', metrics=['accuracy'])\n", " \n", " # Prepare figure\n", " fig = plt.figure(figsize=(16, 10))\n", " gs = fig.add_gridspec(3, 2, hspace=0.3, wspace=0.3)\n", " \n", " ax1 = fig.add_subplot(gs[0, :])\n", " ax2 = fig.add_subplot(gs[1, :])\n", " ax3 = fig.add_subplot(gs[2, 0])\n", " ax4 = fig.add_subplot(gs[2, 1])\n", " \n", " train_losses = []\n", " val_losses = []\n", " train_accs = []\n", " val_accs = []\n", " epochs_list = []\n", " \n", " overfitting_detected = False\n", " best_val_loss = float('inf')\n", " patience_counter = 0\n", " \n", " print(\"Training started... Watch the dynamics!\\n\")\n", " \n", " for epoch in range(0, max_epochs + 1, update_freq):\n", " if epoch > 0:\n", " model.fit(X_train, y_train, epochs=update_freq, batch_size=32, verbose=0)\n", " \n", " # Evaluate\n", " train_loss, train_acc = model.evaluate(X_train, y_train, verbose=0)\n", " val_loss, val_acc = model.evaluate(X_val, y_val, verbose=0)\n", " \n", " train_losses.append(train_loss)\n", " val_losses.append(val_loss)\n", " train_accs.append(train_acc)\n", " val_accs.append(val_acc)\n", " epochs_list.append(epoch)\n", " \n", " # Check for overfitting\n", " if val_loss > best_val_loss:\n", " patience_counter += 1\n", " if patience_counter >= 3 and not overfitting_detected:\n", " overfitting_detected = True\n", " else:\n", " best_val_loss = val_loss\n", " patience_counter = 0\n", " \n", " # Clear axes\n", " ax1.clear()\n", " ax2.clear()\n", " ax3.clear()\n", " ax4.clear()\n", " \n", " # Plot 1: Loss curves\n", " ax1.plot(epochs_list, train_losses, 'b-', linewidth=2, label='Training Loss')\n", " ax1.plot(epochs_list, val_losses, 'orange', linewidth=2, label='Validation Loss')\n", " if overfitting_detected and len(epochs_list) > 3:\n", " overfit_idx = len(epochs_list) - patience_counter - 1\n", " ax1.axvspan(epochs_list[overfit_idx], epochs_list[-1], \n", " alpha=0.2, color='red', label='Overfitting Zone')\n", " ax1.set_xlabel('Epoch', fontsize=11)\n", " ax1.set_ylabel('Loss', fontsize=11)\n", " ax1.set_title('Training and Validation Loss', fontsize=13, fontweight='bold')\n", " ax1.legend()\n", " ax1.grid(True, alpha=0.3)\n", " \n", " # Plot 2: Accuracy curves\n", " ax2.plot(epochs_list, train_accs, 'b-', linewidth=2, label='Training Accuracy')\n", " ax2.plot(epochs_list, val_accs, 'orange', linewidth=2, label='Validation Accuracy')\n", " ax2.set_xlabel('Epoch', fontsize=11)\n", " ax2.set_ylabel('Accuracy', fontsize=11)\n", " ax2.set_title('Training and Validation Accuracy', fontsize=13, fontweight='bold')\n", " ax2.legend()\n", " ax2.grid(True, alpha=0.3)\n", " \n", " # Plot 3: Weight distribution\n", " all_weights = []\n", " for layer in model.layers:\n", " if isinstance(layer, Dense):\n", " all_weights.extend(layer.get_weights()[0].flatten())\n", " \n", " ax3.hist(all_weights, bins=50, color='steelblue', edgecolor='black', alpha=0.7)\n", " ax3.set_xlabel('Weight Value', fontsize=11)\n", " ax3.set_ylabel('Count', fontsize=11)\n", " ax3.set_title('Weight Distribution', fontsize=13, fontweight='bold')\n", " ax3.axvline(x=0, color='red', linestyle='--', linewidth=2)\n", " ax3.grid(True, alpha=0.3, axis='y')\n", " \n", " # Plot 4: Status panel\n", " ax4.axis('off')\n", " \n", " # Determine status\n", " if overfitting_detected:\n", " status = \"šŸ”“ Overfitting Detected!\"\n", " status_color = 'lightcoral'\n", " elif val_loss < train_loss * 1.1:\n", " status = \"šŸŸ¢ Training Normally\"\n", " status_color = 'lightgreen'\n", " else:\n", " status = \"šŸŸ” Possible Overfitting\"\n", " status_color = 'lightyellow'\n", " \n", " # Weight health\n", " max_weight = np.max(np.abs(all_weights))\n", " if max_weight > 10:\n", " weight_status = \"šŸ”“ Weights Exploding!\"\n", " elif max_weight < 0.01:\n", " weight_status = \"šŸ”“ Weights Vanishing!\"\n", " else:\n", " weight_status = \"šŸŸ¢ Weights Healthy\"\n", " \n", " status_text = f\"\"\"\n", " EPOCH: {epoch} / {max_epochs}\n", " \n", " Current Learning Rate: {learning_rate:.4f}\n", " \n", " {status}\n", " {weight_status}\n", " \n", " Train Loss: {train_loss:.4f}\n", " Val Loss: {val_loss:.4f}\n", " Gap: {abs(train_loss - val_loss):.4f}\n", " \n", " Train Acc: {train_acc*100:.1f}%\n", " Val Acc: {val_acc*100:.1f}%\n", " \"\"\"\n", " \n", " ax4.text(0.5, 0.5, status_text, fontsize=11, ha='center', va='center',\n", " bbox=dict(boxstyle='round', facecolor=status_color, alpha=0.8, pad=1.5))\n", " \n", " plt.tight_layout()\n", " display(fig)\n", " clear_output(wait=True)\n", " \n", " plt.close()\n", " \n", " # Final summary\n", " print(\"\\n\" + \"=\"*70)\n", " print(\"TRAINING COMPLETE\")\n", " print(\"=\"*70)\n", " print(f\"Final Training Loss: {train_losses[-1]:.4f}\")\n", " print(f\"Final Validation Loss: {val_losses[-1]:.4f}\")\n", " print(f\"Final Training Accuracy: {train_accs[-1]*100:.2f}%\")\n", " print(f\"Final Validation Accuracy: {val_accs[-1]*100:.2f}%\")\n", " print(f\"\\nTrain-Val Gap: {abs(train_accs[-1] - val_accs[-1])*100:.2f}%\")\n", " \n", " if overfitting_detected:\n", " print(\"\\nāš ļø OVERFITTING DETECTED!\")\n", " print(\"\\nSuggestions:\")\n", " if not use_l2:\n", " print(\" - Try enabling L2 regularization\")\n", " if not use_dropout:\n", " print(\" - Try enabling dropout\")\n", " if train_size < 1000:\n", " print(\" - Consider increasing training data size\")\n", " else:\n", " print(\"\\nāœ“ Model is generalizing well!\")\n", " print(\"=\"*70)\n", "\n", "# Create visualizer\n", "dynamics_viz = TrainingDynamicsVisualizer()\n", "\n", "# Widgets\n", "train_size_slider = widgets.IntSlider(\n", " value=500, min=100, max=2000, step=100,\n", " description='Train Size:',\n", " style={'description_width': '150px'},\n", " layout=widgets.Layout(width='450px')\n", ")\n", "\n", "noise_slider = widgets.FloatSlider(\n", " value=0.2, min=0.0, max=0.5, step=0.05,\n", " description='Noise Level:',\n", " style={'description_width': '150px'},\n", " layout=widgets.Layout(width='450px')\n", ")\n", "\n", "lr_slider_dyn = widgets.FloatLogSlider(\n", " value=0.001, base=10, min=-4, max=-1, step=0.1,\n", " description='Learning Rate:',\n", " style={'description_width': '150px'},\n", " layout=widgets.Layout(width='450px'),\n", " readout_format='.4f'\n", ")\n", "\n", "use_l2_check = widgets.Checkbox(\n", " value=False,\n", " description='Use L2 Regularization',\n", " style={'description_width': '200px'}\n", ")\n", "\n", "l2_slider = widgets.FloatSlider(\n", " value=0.01, min=0.0, max=0.1, step=0.01,\n", " description='L2 Lambda:',\n", " style={'description_width': '150px'},\n", " layout=widgets.Layout(width='450px')\n", ")\n", "\n", "use_dropout_check = widgets.Checkbox(\n", " value=False,\n", " description='Use Dropout',\n", " style={'description_width': '200px'}\n", ")\n", "\n", "dropout_slider = widgets.FloatSlider(\n", " value=0.3, min=0.0, max=0.7, step=0.1,\n", " description='Dropout Rate:',\n", " style={'description_width': '150px'},\n", " layout=widgets.Layout(width='450px')\n", ")\n", "\n", "use_batchnorm_check = widgets.Checkbox(\n", " value=False,\n", " description='Use Batch Normalization',\n", " style={'description_width': '200px'}\n", ")\n", "\n", "epochs_slider_dyn = widgets.IntSlider(\n", " value=200, min=50, max=500, step=50,\n", " description='Max Epochs:',\n", " style={'description_width': '150px'},\n", " layout=widgets.Layout(width='450px')\n", ")\n", "\n", "update_freq_dyn = widgets.IntSlider(\n", " value=10, min=5, max=50, step=5,\n", " description='Update Every:',\n", " style={'description_width': '150px'},\n", " layout=widgets.Layout(width='450px')\n", ")\n", "\n", "start_button = widgets.Button(\n", " description='šŸš€ Start Training',\n", " button_style='success',\n", " layout=widgets.Layout(width='200px', height='40px')\n", ")\n", "\n", "def on_start_clicked(b):\n", " dynamics_viz.train_with_visualization(\n", " train_size=train_size_slider.value,\n", " noise_level=noise_slider.value,\n", " learning_rate=lr_slider_dyn.value,\n", " use_l2=use_l2_check.value,\n", " l2_lambda=l2_slider.value,\n", " use_dropout=use_dropout_check.value,\n", " dropout_rate=dropout_slider.value,\n", " use_batchnorm=use_batchnorm_check.value,\n", " max_epochs=epochs_slider_dyn.value,\n", " update_freq=update_freq_dyn.value\n", " )\n", "\n", "start_button.on_click(on_start_clicked)\n", "\n", "print(\"šŸ“ˆ TRAINING DYNAMICS VISUALIZER\")\n", "print(\"=\"*70)\n", "print(\"Watch training in real-time and see overfitting happen!\\n\")\n", "\n", "display(widgets.VBox([\n", " widgets.HTML(\"

Dataset Configuration

\"),\n", " train_size_slider,\n", " noise_slider,\n", " widgets.HTML(\"

Regularization (Try with and without!)

\"),\n", " use_l2_check,\n", " l2_slider,\n", " use_dropout_check,\n", " dropout_slider,\n", " use_batchnorm_check,\n", " widgets.HTML(\"

Training Parameters

\"),\n", " lr_slider_dyn,\n", " epochs_slider_dyn,\n", " update_freq_dyn,\n", " widgets.HTML(\"
\"),\n", " start_button\n", "]))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "\n", "## šŸŽ“ Conclusion\n", "\n", "### What You've Learned Today:\n", "\n", "1. **XOR Problem** - Neural networks can solve non-linearly separable problems\n", "2. **Architecture** - How layers, units, and activations affect performance\n", "3. **Activation Functions** - ReLU vs Sigmoid vs Tanh and their characteristics\n", "4. **Training Dynamics** - How to detect and prevent overfitting\n", "\n", "### Key Takeaways:\n", "\n", "- āœ… **Start simple**: 1 layer, few units, ReLU + Adam\n", "- āœ… **Monitor curves**: Training and validation loss tell you everything\n", "- āœ… **Regularize when needed**: L2, Dropout, BatchNorm help prevent overfitting\n", "- āœ… **Experiment**: Try different configurations and learn from results\n", "\n", "### Practice Suggestions:\n", "\n", "1. Go back to Demo 1 and try to find the minimal architecture that solves XOR\n", "2. Use Demo 2 to compare different architectures on the same dataset\n", "3. In Demo 3, see which activation works best for different datasets\n", "4. Use Demo 4 to understand when regularization is necessary\n", "\n", "### Resources:\n", "\n", "- šŸ“ Lecture notes with full mathematical derivations\n", "- šŸ’» This Colab notebook (save a copy!)\n", "- šŸ“š Keras documentation: https://keras.io/\n", "\n", "---\n", "\n", "**Questions? Observations? Share them with the class!**" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.0" } }, "nbformat": 4, "nbformat_minor": 4 }