{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# MPS311/439 Week 3 Interactive Demos\n",
    "# Feature Engineering & Regularization\n",
    "\n",
    "**Course:** Machine Learning  \n",
    "**Lecturer:** Dr. Wei Xing  \n",
    "**Week:** 3\n",
    "\n",
    "This notebook contains two interactive demonstrations:\n",
    "1. **Demo 1:** Polynomial Features and Overfitting\n",
    "2. **Demo 2:** Regularization (Ridge & Lasso)\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup: Import Libraries\n",
    "\n",
    "Run this cell first to import all necessary libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "✓ All libraries imported successfully!\n",
      "✓ Ready for interactive demos!\n"
     ]
    }
   ],
   "source": [
    "# Import necessary libraries\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.preprocessing import PolynomialFeatures, StandardScaler\n",
    "from sklearn.linear_model import LinearRegression, Ridge, Lasso\n",
    "from sklearn.metrics import r2_score\n",
    "from ipywidgets import interact, IntSlider, FloatLogSlider, RadioButtons\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "# Set random seed for reproducibility\n",
    "np.random.seed(42)\n",
    "\n",
    "print(\"✓ All libraries imported successfully!\")\n",
    "print(\"✓ Ready for interactive demos!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "# Demo 1: Polynomial Features and Overfitting\n",
    "\n",
    "## Objective\n",
    "Discover how polynomial features can fit curved data, but also lead to overfitting.\n",
    "\n",
    "## Instructions\n",
    "1. Run the cell below to generate synthetic house price data\n",
    "2. Run the interactive widget cell\n",
    "3. Move the slider to explore different polynomial degrees\n",
    "4. Watch what happens to training and test R² scores!\n",
    "\n",
    "## What to Look For\n",
    "- **Degree 1:** Straight line, poor fit\n",
    "- **Degree 2:** Perfect! Captures the U-curve\n",
    "- **Degree 10:** Training R² ≈ 1.0, but test R² is terrible (overfitting!)\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "✓ Generated training data: 50 points\n",
      "✓ Generated test data: 30 points\n",
      "✓ True relationship: U-shaped curve (quadratic)\n"
     ]
    }
   ],
   "source": [
    "# Generate synthetic house price data with U-shaped relationship\n",
    "# True relationship: price = 300 - 5*age + 0.5*age² + noise\n",
    "\n",
    "np.random.seed(42)\n",
    "\n",
    "# Training data\n",
    "X_train = np.linspace(0, 10, 50).reshape(-1, 1)\n",
    "y_train = 300 - 5*X_train.ravel() + 0.5*X_train.ravel()**2 + np.random.normal(0, 20, 50) * 0.1\n",
    "\n",
    "# Test data (different from training)\n",
    "X_test = np.linspace(0, 10, 30).reshape(-1, 1)\n",
    "y_test = 300 - 5*X_test.ravel() + 0.5*X_test.ravel()**2 + np.random.normal(0, 20, 30) * 0.1\n",
    "\n",
    "# Ground truth relationship\n",
    "X_truth = np.linspace(0, 10, 200).reshape(-1, 1)\n",
    "y_truth = 300 - 5*X_truth.ravel() + 0.5*X_truth.ravel()**2\n",
    "\n",
    "\n",
    "print(f\"✓ Generated training data: {X_train.shape[0]} points\")\n",
    "print(f\"✓ Generated test data: {X_test.shape[0]} points\")\n",
    "print(f\"✓ True relationship: U-shaped curve (quadratic)\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c2935e9636cf45b2841917a3e1b2a1a2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "interactive(children=(IntSlider(value=1, description='Polynomial Degree:', layout=Layout(width='600px'), max=2…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Demo 1: Interactive Polynomial Degree Explorer\n",
    "\n",
    "def explore_polynomial_degree(degree=1):\n",
    "    \"\"\"\n",
    "    Interactive function to explore polynomial regression with different degrees.\n",
    "    \n",
    "    Parameters:\n",
    "    - degree: Polynomial degree (1 to 15)\n",
    "    \"\"\"\n",
    "    # Create polynomial features\n",
    "    poly = PolynomialFeatures(degree=degree, include_bias=False)\n",
    "    X_train_poly = poly.fit_transform(X_train)\n",
    "    X_test_poly = poly.transform(X_test)\n",
    "    \n",
    "    # Fit linear regression on polynomial features\n",
    "    model = LinearRegression()\n",
    "    model.fit(X_train_poly, y_train)\n",
    "    \n",
    "    # Generate smooth curve for visualization\n",
    "    X_plot = np.linspace(0, 10, 200).reshape(-1, 1)\n",
    "    X_plot_poly = poly.transform(X_plot)\n",
    "    y_plot = model.predict(X_plot_poly)\n",
    "    \n",
    "    # Calculate R² scores\n",
    "    train_r2 = r2_score(y_train, model.predict(X_train_poly))\n",
    "    test_r2 = r2_score(y_test, model.predict(X_test_poly))\n",
    "    \n",
    "    # Determine performance category\n",
    "    if test_r2 > 0.7:\n",
    "        perf_color = '#06A77D'  # Green\n",
    "        perf_text = '✓ Good!'\n",
    "    elif test_r2 > 0.5:\n",
    "        perf_color = '#F77F00'  # Orange\n",
    "        perf_text = '⚠ Okay'\n",
    "    else:\n",
    "        perf_color = '#E63946'  # Red\n",
    "        perf_text = '✗ Overfitting!'\n",
    "    \n",
    "    # Create the plot\n",
    "    fig, ax = plt.subplots(figsize=(14, 7))\n",
    "    \n",
    "    # Plot training and test data\n",
    "    ax.scatter(X_train, y_train, alpha=0.6, s=120, label='Training Data', \n",
    "               color='#2E86AB', edgecolors='black', linewidth=1)\n",
    "    ax.scatter(X_test, y_test, alpha=0.6, s=120, label='Test Data', \n",
    "               color='#F77F00', marker='^', edgecolors='black', linewidth=1)\n",
    "    \n",
    "    # Plot fitted curve\n",
    "    ax.plot(X_plot, y_plot, 'r-', linewidth=3, label=f'Degree {degree} Fit', zorder=5)\n",
    "    \n",
    "    # Plot ground truth relationship\n",
    "    ax.plot(X_truth, y_truth, 'k-', linewidth=3, label='Ground Truth', zorder=10)\n",
    "    \n",
    "    # Styling\n",
    "    ax.set_xlabel('House Age (years)', fontsize=16, fontweight='bold')\n",
    "    ax.set_ylabel('House Price (£1000s)', fontsize=16, fontweight='bold')\n",
    "    ax.set_title(f'Polynomial Degree {degree}', fontsize=20, fontweight='bold', pad=20)\n",
    "    ax.legend(fontsize=14, loc='upper right', framealpha=0.9)\n",
    "    ax.grid(True, alpha=0.3, linestyle='--')\n",
    "    ax.set_xlim(-0.5, 10.5)\n",
    "    \n",
    "    # Add metrics text box\n",
    "    metrics_text = f'Train R² = {train_r2:.3f}\\nTest R² = {test_r2:.3f}\\nFeatures: {X_train_poly.shape[1]}'\n",
    "    props = dict(boxstyle='round', facecolor='wheat', alpha=0.9, edgecolor='black', linewidth=2)\n",
    "    ax.text(0.02, 0.98, metrics_text, transform=ax.transAxes, fontsize=18,\n",
    "            verticalalignment='top', bbox=props, fontweight='bold', family='monospace')\n",
    "    \n",
    "    # Add performance indicator\n",
    "    ax.text(0.98, 0.98, perf_text, transform=ax.transAxes, fontsize=22,\n",
    "            verticalalignment='top', horizontalalignment='right',\n",
    "            color=perf_color, fontweight='bold',\n",
    "            bbox=dict(boxstyle='round', facecolor='white', alpha=0.9, \n",
    "                     edgecolor=perf_color, linewidth=3))\n",
    "    \n",
    "    plt.tight_layout()\n",
    "    plt.show()\n",
    "    \n",
    "    # Print analysis\n",
    "    gap = train_r2 - test_r2\n",
    "    print(f\"\\n{'='*60}\")\n",
    "    print(f\"Analysis for Degree {degree}:\")\n",
    "    print(f\"{'='*60}\")\n",
    "    print(f\"Training R²:     {train_r2:.4f}\")\n",
    "    print(f\"Test R²:         {test_r2:.4f}\")\n",
    "    print(f\"Train-Test Gap:  {gap:.4f}\")\n",
    "    print(f\"Number of features: {X_train_poly.shape[1]}\")\n",
    "    \n",
    "    if gap < 0.1:\n",
    "        print(\"\\n✓ Good generalization! Small train-test gap.\")\n",
    "    elif gap < 0.3:\n",
    "        print(\"\\n⚠ Moderate overfitting. Consider regularization.\")\n",
    "    else:\n",
    "        print(\"\\n✗ Severe overfitting! Model memorizing training data.\")\n",
    "    \n",
    "    if degree == 2:\n",
    "        print(\"\\n🎯 This is the SWEET SPOT for this data!\")\n",
    "    elif degree >= 10:\n",
    "        print(\"\\n⚠️ Very high degree - watch for wild oscillations!\")\n",
    "\n",
    "# Create interactive widget\n",
    "interact(explore_polynomial_degree, \n",
    "         degree=IntSlider(min=1, max=20, step=1, value=1, \n",
    "                         description='Polynomial Degree:', \n",
    "                         style={'description_width': '150px'},\n",
    "                         layout={'width': '600px'}));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "# Demo 2: Regularization (Ridge & Lasso)\n",
    "\n",
    "## Objective\n",
    "Discover how regularization controls overfitting by penalizing large weights.\n",
    "\n",
    "## Instructions\n",
    "1. Run the cell below to prepare high-degree polynomial features\n",
    "2. Run the interactive widget cell\n",
    "3. Start with Ridge, λ=0.01 (almost no regularization)\n",
    "4. Gradually increase λ and watch:\n",
    "   - Coefficients shrinking\n",
    "   - Test R² improving\n",
    "5. Switch to Lasso and see coefficients become exactly ZERO!\n",
    "\n",
    "## What to Look For\n",
    "- **λ = 0:** Overfitting (huge coefficients, poor test R²)\n",
    "- **λ = 10:** Often optimal (controlled coefficients, good test R²)\n",
    "- **λ = 1000:** Underfitting (all coefficients ≈ 0)\n",
    "- **Lasso:** Watch coefficients drop to exactly zero (sparsity!)\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "✓ Created degree 10 polynomial features\n",
      "✓ Number of features: 10\n",
      "✓ Features scaled (mean=0, std=1)\n",
      "\n",
      "Starting state: Degree 10 with no regularization = OVERFITTING\n",
      "Let's fix it with regularization...\n"
     ]
    }
   ],
   "source": [
    "# Prepare data for Demo 2: High-degree polynomial with scaling\n",
    "\n",
    "# Use same training/test data from Demo 1\n",
    "# Create degree 10 polynomial features (to start in overfit state)\n",
    "poly_demo2 = PolynomialFeatures(degree=10, include_bias=False)\n",
    "X_train_poly_demo2 = poly_demo2.fit_transform(X_train)\n",
    "X_test_poly_demo2 = poly_demo2.transform(X_test)\n",
    "\n",
    "# Scale features (CRITICAL for regularization!)\n",
    "scaler = StandardScaler()\n",
    "X_train_scaled = scaler.fit_transform(X_train_poly_demo2)\n",
    "X_test_scaled = scaler.transform(X_test_poly_demo2)\n",
    "\n",
    "print(f\"✓ Created degree 10 polynomial features\")\n",
    "print(f\"✓ Number of features: {X_train_scaled.shape[1]}\")\n",
    "print(f\"✓ Features scaled (mean=0, std=1)\")\n",
    "print(f\"\\nStarting state: Degree 10 with no regularization = OVERFITTING\")\n",
    "print(f\"Let's fix it with regularization...\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9031c78df9c84f5c937538a5937ea98b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "interactive(children=(FloatLogSlider(value=0.01, description='λ (lambda):', layout=Layout(width='600px'), max=…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Demo 2: Interactive Regularization Explorer\n",
    "\n",
    "def explore_regularization(lambda_val=0.01, method='Ridge'):\n",
    "    \"\"\"\n",
    "    Interactive function to explore Ridge and Lasso regularization.\n",
    "    \n",
    "    Parameters:\n",
    "    - lambda_val: Regularization strength (λ)\n",
    "    - method: 'Ridge' or 'Lasso'\n",
    "    \"\"\"\n",
    "    # Choose model based on method\n",
    "    if method == 'Ridge':\n",
    "        model = Ridge(alpha=lambda_val)\n",
    "        color = '#2E86AB'  # Blue\n",
    "        color_bg = '#E3F2FD'\n",
    "    else:  # Lasso\n",
    "        model = Lasso(alpha=lambda_val, max_iter=10000)\n",
    "        color = '#06A77D'  # Green\n",
    "        color_bg = '#E8F5E9'\n",
    "    \n",
    "    # Fit model\n",
    "    model.fit(X_train_scaled, y_train)\n",
    "    \n",
    "    # Predictions\n",
    "    X_plot = np.linspace(0, 10, 200).reshape(-1, 1)\n",
    "    X_plot_poly = poly_demo2.transform(X_plot)\n",
    "    X_plot_scaled = scaler.transform(X_plot_poly)\n",
    "    y_plot = model.predict(X_plot_scaled)\n",
    "    \n",
    "    # Calculate R² scores\n",
    "    train_r2 = r2_score(y_train, model.predict(X_train_scaled))\n",
    "    test_r2 = r2_score(y_test, model.predict(X_test_scaled))\n",
    "    \n",
    "    # Count non-zero coefficients (for Lasso)\n",
    "    non_zero = np.sum(np.abs(model.coef_) > 1e-5)\n",
    "    max_coef = np.max(np.abs(model.coef_))\n",
    "    total_coefs = len(model.coef_)\n",
    "    \n",
    "    # Create subplots\n",
    "    fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(18, 7))\n",
    "    \n",
    "    # ========== LEFT PLOT: Fitted Curve ==========\n",
    "    ax1.scatter(X_train, y_train, alpha=0.6, s=120, label='Training', \n",
    "                color='#2E86AB', edgecolors='black', linewidth=1)\n",
    "    ax1.scatter(X_test, y_test, alpha=0.6, s=120, label='Test', \n",
    "                color='#F77F00', marker='^', edgecolors='black', linewidth=1)\n",
    "    ax1.plot(X_plot, y_plot, '-', linewidth=3, label=f'{method} Fit', color=color)\n",
    "    \n",
    "    ax1.set_xlabel('House Age (years)', fontsize=14, fontweight='bold')\n",
    "    ax1.set_ylabel('House Price (£1000s)', fontsize=14, fontweight='bold')\n",
    "    ax1.set_title(f'{method} Regression (λ = {lambda_val:.4f})', \n",
    "                  fontsize=18, fontweight='bold', pad=15)\n",
    "    ax1.legend(fontsize=12, loc='upper right', framealpha=0.9)\n",
    "    ax1.grid(True, alpha=0.3, linestyle='--')\n",
    "    ax1.set_xlim(-0.5, 10.5)\n",
    "    \n",
    "    # Metrics text box\n",
    "    metrics_text = (f'Train R²: {train_r2:.3f}\\n'\n",
    "                   f'Test R²: {test_r2:.3f}\\n'\n",
    "                   f'Non-zero: {non_zero}/{total_coefs}\\n'\n",
    "                   f'Max |coef|: {max_coef:.1f}')\n",
    "    props = dict(boxstyle='round', facecolor='wheat', alpha=0.9, \n",
    "                edgecolor='black', linewidth=2)\n",
    "    ax1.text(0.02, 0.98, metrics_text, transform=ax1.transAxes, fontsize=14,\n",
    "            verticalalignment='top', bbox=props, fontweight='bold', family='monospace')\n",
    "    \n",
    "    # ========== RIGHT PLOT: Coefficient Values ==========\n",
    "    coef_indices = np.arange(len(model.coef_))\n",
    "    colors_coef = [color if abs(c) > 1e-5 else '#E5E5E5' for c in model.coef_]\n",
    "    \n",
    "    ax2.bar(coef_indices, model.coef_, color=colors_coef, alpha=0.8, edgecolor='black', linewidth=0.5)\n",
    "    ax2.axhline(y=0, color='black', linestyle='-', linewidth=1)\n",
    "    ax2.set_xlabel('Coefficient Index', fontsize=14, fontweight='bold')\n",
    "    ax2.set_ylabel('Coefficient Value', fontsize=14, fontweight='bold')\n",
    "    ax2.set_title(f'Coefficient Values ({method})', fontsize=18, fontweight='bold', pad=15)\n",
    "    ax2.grid(True, alpha=0.3, axis='y', linestyle='--')\n",
    "    \n",
    "    # Add sparsity annotation for Lasso\n",
    "    if method == 'Lasso' and non_zero < total_coefs:\n",
    "        zeros_count = total_coefs - non_zero\n",
    "        sparsity_text = f'{zeros_count} coefficients = 0\\n(Automatic feature selection!)'\n",
    "        ax2.text(0.98, 0.98, sparsity_text, transform=ax2.transAxes, fontsize=14,\n",
    "                verticalalignment='top', horizontalalignment='right',\n",
    "                color='#E63946', fontweight='bold',\n",
    "                bbox=dict(boxstyle='round', facecolor='white', alpha=0.9, \n",
    "                         edgecolor='#E63946', linewidth=2))\n",
    "    \n",
    "    plt.tight_layout()\n",
    "    plt.show()\n",
    "    \n",
    "    # Print detailed analysis\n",
    "    gap = train_r2 - test_r2\n",
    "    print(f\"\\n{'='*70}\")\n",
    "    print(f\"{method} Regression with λ = {lambda_val:.4f}\")\n",
    "    print(f\"{'='*70}\")\n",
    "    print(f\"Training R²:        {train_r2:.4f}\")\n",
    "    print(f\"Test R²:            {test_r2:.4f}\")\n",
    "    print(f\"Train-Test Gap:     {gap:.4f}\")\n",
    "    print(f\"Non-zero coeffs:    {non_zero} / {total_coefs}\")\n",
    "    print(f\"Max |coefficient|:  {max_coef:.2f}\")\n",
    "    \n",
    "    # Provide guidance\n",
    "    if lambda_val < 0.01:\n",
    "        print(\"\\n⚠️ Very low λ - likely overfitting!\")\n",
    "        print(\"   Try increasing λ to control the coefficients.\")\n",
    "    elif lambda_val > 100:\n",
    "        print(\"\\n⚠️ Very high λ - likely underfitting!\")\n",
    "        print(\"   Try decreasing λ to allow more flexibility.\")\n",
    "    elif test_r2 > 0.7 and gap < 0.15:\n",
    "        print(\"\\n✓ Good balance! Test performance is strong and gap is small.\")\n",
    "    \n",
    "    if method == 'Lasso' and non_zero < total_coefs * 0.5:\n",
    "        print(f\"\\n🎯 Lasso achieved {(1-non_zero/total_coefs)*100:.0f}% sparsity!\")\n",
    "        print(f\"   Only {non_zero} features are actually being used.\")\n",
    "\n",
    "# Create interactive widget\n",
    "interact(explore_regularization,\n",
    "         lambda_val=FloatLogSlider(\n",
    "             value=0.01,\n",
    "             base=10,\n",
    "             min=-3,  # 10^-3 = 0.001\n",
    "             max=3,   # 10^3 = 1000\n",
    "             step=0.1,\n",
    "             description='λ (lambda):',\n",
    "             style={'description_width': '150px'},\n",
    "             layout={'width': '600px'},\n",
    "             readout_format='.4f'\n",
    "         ),\n",
    "         method=RadioButtons(\n",
    "             options=['Ridge', 'Lasso'],\n",
    "             description='Method:',\n",
    "             style={'description_width': '150px'}\n",
    "         ));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "# Summary & Key Takeaways\n",
    "\n",
    "## Demo 1: Polynomial Features\n",
    "- ✓ **Degree 2** captured the U-shaped curve perfectly\n",
    "- ⚠️ **Degree 10+** caused severe overfitting (Train R² → 1.0, Test R² → poor)\n",
    "- 📊 **Train-test gap** is the signature of overfitting\n",
    "\n",
    "## Demo 2: Regularization\n",
    "- ✓ **Increasing λ** shrinks coefficients and improves test performance\n",
    "- 🎯 **Ridge** shrinks all coefficients smoothly (never exactly zero)\n",
    "- ⚡ **Lasso** creates exact zeros (automatic feature selection)\n",
    "- ⚖️ **Optimal λ** balances underfitting and overfitting\n",
    "\n",
    "## The Big Picture\n",
    "```\n",
    "Feature Engineering → Power to fit complex patterns\n",
    "     +\n",
    "Regularization → Control to prevent overfitting\n",
    "     =\n",
    "Powerful & Reliable Models!\n",
    "```\n",
    "\n",
    "## Next Steps\n",
    "1. Experiment with different λ values\n",
    "2. Try switching between Ridge and Lasso\n",
    "3. In the lab: Implement this with real datasets using sklearn\n",
    "4. Learn cross-validation to automatically find optimal λ\n",
    "\n",
    "---\n",
    "\n",
    "**Questions?** Discuss with your neighbor or ask during office hours!\n",
    "\n",
    "**Office Hours:** Tuesday 12:00-1:00 PM, Hicks Building I22  \n",
    "**Email:** w.xing@sheffield.ac.uk\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Optional: Quick Experiments\n",
    "\n",
    "Try these experiments to deepen your understanding!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Experiment 1: Compare Ridge and Lasso at same λ\n",
    "# Uncomment and run:\n",
    "\n",
    "# lambda_test = 1.0\n",
    "# print(\"Comparing Ridge vs Lasso at λ = 1.0\\n\")\n",
    "# print(\"RIDGE:\")\n",
    "# explore_regularization(lambda_test, 'Ridge')\n",
    "# print(\"\\n\" + \"=\"*70 + \"\\n\")\n",
    "# print(\"LASSO:\")\n",
    "# explore_regularization(lambda_test, 'Lasso')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Experiment 2: Find the optimal polynomial degree yourself\n",
    "# Try degrees 1-5 and record test R² for each\n",
    "# Which degree gives the best test performance?\n",
    "\n",
    "# Your code here..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Experiment 3: Explore extreme λ values\n",
    "# What happens at λ = 0.0001 (almost no regularization)?\n",
    "# What happens at λ = 10000 (extreme regularization)?\n",
    "\n",
    "# Your code here..."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "expTorch",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}