Week 4: Linear Classification with Logistic Regression

MPS311/439 Machine Learning - Dr. Wei Xing

Welcome Back!

Last week, we explored how to extend linear regression through feature engineering and regularization. We learned how to create polynomial features to capture non-linear patterns and how Ridge and Lasso regression can prevent overfitting by controlling model complexity. These techniques gave us powerful tools for predicting continuous outcomes like house prices or temperatures.

But what happens when we want to predict categories instead of numbers? What if we want to know whether an email is spam or not? Whether a patient has a disease? Whether a student will pass or fail? These are classification problems, and they require a different approach.

This week, we'll discover how to adapt our linear models for classification tasks. We'll learn about logistic regression, understand why it uses something called the sigmoid function, and explore how to properly evaluate classification models. By the end, you'll be able to build models that make yes/no decisions with confidence scores!

1. The Classification Challenge

1.1 What is Classification?

Classification is the task of predicting which discrete category (or class) an observation belongs to, based on its features. Unlike regression where we predict continuous values, classification predicts discrete labels.

Real-world examples:

In this lecture, we'll focus on binary classification where there are exactly two classes, typically labeled as 0 and 1 (or negative and positive).

1.2 Why Can't We Just Use Linear Regression?

You might wonder: "Why can't we just use what we learned last week? Can't we treat the classes as numbers (0 and 1) and use linear regression?"

Great question! Let's see what happens when we try. Imagine we're predicting whether students will pass or fail based on hours studied.

Why Linear Regression Fails

The fundamental problems:

  1. Impossible predictions: Linear regression can predict values less than 0 or greater than 1. But probabilities must be between 0 and 1! What does it mean to have a "-0.2 probability" of passing?

  2. Sensitivity to outliers: Notice the orange star in the figure? That's a student who studied only 1 hour but passed (maybe they're naturally talented, or got lucky). This single outlier drastically affects the regression line, making predictions worse for everyone else.

  3. No probability interpretation: Even if predictions happen to fall between 0 and 1, linear regression doesn't give us proper probabilities. The output doesn't have the mathematical properties that probabilities require.

  4. Wrong loss function: We'll see later that squared error (which we used for regression) isn't the right way to measure classification errors.

1.3 What We Need Instead

For classification, we need a model that:

Logistic regression gives us all of these! Despite its name containing "regression", logistic regression is actually a classification algorithm. The name comes from the fact that it uses regression techniques internally, but its output is a classification decision.

2. From Regression to Classification: The Sigmoid Function

2.1 The Core Idea

Here's the key insight: We can still use a linear combination of features (just like linear regression), but we'll pass it through a special function that squashes any real number into the range (0, 1).

The two-step process:

  1. Linear combination: Compute z=w0+w1x1+w2x2+…+wpxpz = w_0 + w_1x_1 + w_2x_2 + \ldots + w_px_p (same as before!)
  2. Squashing: Apply a function that maps z→z \rightarrow probability between 0 and 1

This special function is called the sigmoid function (also known as the logistic function).

2.2 Meet the Sigmoid Function

The sigmoid function is defined as:

σ(z)=11+e−z\sigma(z) = \frac{1}{1 + e^{-z}}

Let's visualize what this function does:

The Sigmoid Function

Why sigmoid is perfect for our needs:

Key properties to remember:

2.3 The Logistic Regression Model

Now we can define our complete model:

Step 1: Compute the linear combination

z=w0+w1x1+w2x2+…+wpxpz = w_0 + w_1x_1 + w_2x_2 + \ldots + w_px_p

Or in vector notation: z=wTxz = \mathbf{w}^T \mathbf{x}

Step 2: Apply sigmoid to get probability

P(y=1∣x)=σ(z)=11+e−zP(y=1|\mathbf{x}) = \sigma(z) = \frac{1}{1 + e^{-z}}

Interpretation of weights:

Example: Imagine predicting disease risk:

2.4 Making Decisions: The Decision Boundary

Once we have P(y=1∣x)P(y=1|\mathbf{x}), how do we actually make a prediction? We need a decision rule:

Decision Rule:

Since σ(0)=0.5\sigma(0) = 0.5, this is equivalent to:

The decision boundary is the line (or hyperplane) where wTx=0\mathbf{w}^T \mathbf{x} = 0. This is still a linear boundary, just like in linear regression!

Let's see this in action with two features:

Decision Boundary in 2D

In this figure:

Key insight: Even though we're using sigmoid to get probabilities, the decision boundary itself is still linear. This means logistic regression works best when the two classes can be separated (at least approximately) by a straight line or hyperplane.

3. Training Logistic Regression

Now we know what logistic regression predicts, but how do we find the best weights w\mathbf{w}? We need to define what "best" means by choosing an appropriate loss function.

3.1 Why Not Squared Error?

In linear regression, we used the squared error: L=(y−y^)2L = (y - \hat{y})^2

For classification, this seems natural at first:

But there's a critical problem: When we use squared error with the sigmoid function, the resulting optimization landscape is non-convex. This means:

We need a loss function that creates a convex optimization problem, guaranteeing that gradient descent will find the best solution.

3.2 Cross-Entropy Loss: The Right Loss for Classification

The cross-entropy loss (also called log loss) is specifically designed for probability predictions:

For a single sample:

L(y,y^)=−[ylog⁡(y^)+(1−y)log⁡(1−y^)]L(y, \hat{y}) = -\left[y \log(\hat{y}) + (1-y) \log(1-\hat{y})\right]

Where:

Let's understand this intuitively:

Case 1: When true label y=1y = 1

Case 2: When true label y=0y = 0

Beautiful properties:

3.3 Using Sklearn's LogisticRegression

Fortunately, we don't need to implement all the optimization mathematics ourselves. Sklearn provides a clean interface:

from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

# Split data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Create and train model
model = LogisticRegression()
model.fit(X_train, y_train)

# Get class predictions (0 or 1)
y_pred = model.predict(X_test)

# Get probability predictions
y_prob = model.predict_proba(X_test)
# Returns array like: [[P(class 0), P(class 1)] for each sample]

# Access learned parameters
print("Intercept:", model.intercept_)
print("Coefficients:", model.coef_)

Key methods:

3.4 A Simple Example

Let's see logistic regression in action with minimal code:

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import make_classification

# Generate simple binary classification data
X, y = make_classification(n_samples=100, n_features=2, n_redundant=0, 
                           n_clusters_per_class=1, random_state=42)

# Train logistic regression
model = LogisticRegression()
model.fit(X, y)

# Make predictions
y_pred = model.predict(X)
y_prob = model.predict_proba(X)[:, 1]  # Get P(y=1) for each sample

# Show a few examples
for i in range(5):
    print(f"Sample {i}: True={y[i]}, Predicted={y_pred[i]}, P(y=1)={y_prob[i]:.3f}")

Output might look like:

Sample 0: True=1, Predicted=1, P(y=1)=0.912
Sample 1: True=0, Predicted=0, P(y=1)=0.143
Sample 2: True=1, Predicted=1, P(y=1)=0.876
Sample 3: True=0, Predicted=0, P(y=1)=0.089
Sample 4: True=1, Predicted=0, P(y=1)=0.487

Notice sample 4: The model predicted class 0, but the true label was 1, and the probability was 0.487 (very close to the decision threshold). This is an uncertain prediction where the model could easily be wrong.

4. Evaluating Classification Models

Training a model is only half the battle. How do we know if it's actually good? For regression, we used metrics like MSE or R². For classification, we need different tools.

4.1 The Problem with Accuracy

The most obvious metric is accuracy: what percentage of predictions are correct?

Accuracy=Number of Correct PredictionsTotal Predictions\text{Accuracy} = \frac{\text{Number of Correct Predictions}}{\text{Total Predictions}}

This seems reasonable, but it can be very misleading!

Example - Email Spam Detection:
Imagine 95% of emails are legitimate (not spam), and only 5% are spam.

Consider a "naive model" that simply predicts "not spam" for every single email:

This is the problem of imbalanced classes: when one class is much more common than the other, accuracy doesn't tell the full story.

When accuracy IS useful:

When accuracy is MISLEADING:

4.2 The Confusion Matrix: Foundation of Classification Metrics

To truly understand how our classifier performs, we need to break down its predictions into four categories:

Confusion Matrix

The four outcomes:

  1. True Positive (TP): Model predicted 1, and the true label was 1 ✓

    • Correctly identified positive cases
    • Example: Correctly diagnosed a disease

  2. True Negative (TN): Model predicted 0, and the true label was 0 ✓

    • Correctly identified negative cases
    • Example: Correctly identified a healthy patient

  3. False Positive (FP): Model predicted 1, but true label was 0 ✗

    • Incorrectly identified as positive (Type I Error)
    • Example: False alarm - diagnosed disease when patient is healthy

  4. False Negative (FN): Model predicted 0, but true label was 1 ✗

    • Missed a positive case (Type II Error)
    • Example: Failed to diagnose disease in sick patient

Different scenarios care about different errors:

Creating a confusion matrix in Python:

from sklearn.metrics import confusion_matrix

y_true = [1, 0, 1, 1, 0, 1, 0, 0]
y_pred = [1, 0, 1, 0, 0, 1, 0, 1]

cm = confusion_matrix(y_true, y_pred)
print(cm)
# Output:
# [[3 1]   <- Row 0: True negatives and false positives
#  [1 3]]  <- Row 1: False negatives and true positives

4.3 Precision, Recall, and F1-Score

From the confusion matrix, we can compute metrics that focus on different aspects of performance:

Precision: "When I predict positive, how often am I right?"

Precision=TPTP+FP\text{Precision} = \frac{TP}{TP + FP}

High precision means: Few false alarms

Example: In email spam filtering:

Recall (Sensitivity): "Of all actual positives, how many did I catch?"

Recall=TPTP+FN\text{Recall} = \frac{TP}{TP + FN}

High recall means: Few missed detections

Example: In disease screening:

F1-Score: The Harmonic Mean

F1=2×Precision×RecallPrecision+RecallF1 = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}

The F1-score balances precision and recall:

Let's visualize how different classifiers perform on these metrics:

Metrics Comparison

Understanding the different classifier types:

Computing metrics in Python:

from sklearn.metrics import precision_score, recall_score, f1_score, classification_report

y_true = [1, 0, 1, 1, 0, 1, 0, 0]
y_pred = [1, 0, 1, 0, 0, 1, 0, 1]

precision = precision_score(y_true, y_pred)
recall = recall_score(y_true, y_pred)
f1 = f1_score(y_true, y_pred)

print(f"Precision: {precision:.3f}")
print(f"Recall: {recall:.3f}")
print(f"F1-Score: {f1:.3f}")

# Or get everything at once:
print(classification_report(y_true, y_pred))

4.4 The Precision-Recall Tradeoff

Here's a fundamental insight: You usually can't maximize both precision and recall simultaneously. There's a tradeoff.

Remember that by default, logistic regression predicts class 1 when P(y=1) > 0.5. But we can change this threshold!

Increasing the threshold (e.g., to 0.7):

Decreasing the threshold (e.g., to 0.3):

Practical example - Medical diagnosis:

Adjusting the threshold in Python:

# Get probability predictions
y_prob = model.predict_proba(X_test)[:, 1]

# Use custom threshold
threshold = 0.7
y_pred_custom = (y_prob > threshold).astype(int)

# Compare with default threshold (0.5)
y_pred_default = model.predict(X_test)

4.5 ROC Curve and AUC: The Complete Picture

Instead of choosing a single threshold, we can evaluate model performance across all possible thresholds using the ROC curve (Receiver Operating Characteristic).

The ROC curve plots:

Each point on the curve represents a different classification threshold.

ROC Curve

Interpreting the ROC curve:

What AUC really means:
AUC represents the probability that the model ranks a random positive sample higher than a random negative sample. In other words, if you pick a positive example and a negative example at random, AUC is the probability that your model assigns a higher score to the positive one.

Creating ROC curves in Python:

from sklearn.metrics import roc_curve, roc_auc_score
import matplotlib.pyplot as plt

# Get probability predictions
y_prob = model.predict_proba(X_test)[:, 1]

# Compute ROC curve
fpr, tpr, thresholds = roc_curve(y_test, y_prob)
auc_score = roc_auc_score(y_test, y_prob)

# Plot
plt.plot(fpr, tpr, label=f'ROC Curve (AUC = {auc_score:.2f})')
plt.plot([0, 1], [0, 1], 'k--', label='Random Guess')
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('ROC Curve')
plt.legend()
plt.show()

print(f"AUC Score: {auc_score:.3f}")

4.6 Choosing the Right Metric for Your Problem

With so many metrics, how do you choose? Here's a decision guide:

Scenario Recommended Metric Reasoning
Balanced classes + equal error costs Accuracy Simple and interpretable
Imbalanced classes F1-Score, Precision, or Recall Accuracy is misleading
False positives are very costly Precision Minimize false alarms
False negatives are very costly Recall Don't miss positive cases
Need to compare models overall AUC Threshold-independent
Need to choose operating point ROC Curve Shows all precision-recall tradeoffs
Real costs for FP and FN Cost-sensitive evaluation Incorporate business costs

Examples:

5. Practical Implementation: A Complete Example

Let's bring everything together with a complete worked example. We'll use a real dataset to predict heart disease risk.

5.1 Loading and Exploring Data 

import numpy as np
import pandas as pd
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix, classification_report, roc_auc_score

# Load breast cancer dataset (binary classification)
data = load_breast_cancer()
X = data.data
y = data.target  # 0 = malignant, 1 = benign

print(f"Dataset shape: {X.shape}")
print(f"Features: {data.feature_names[:5]}...")  # Show first 5 features
print(f"Class distribution: {np.bincount(y)}")  # Count of each class

# Split into train and test sets
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=42, stratify=y
)

Output:

Dataset shape: (569, 30)
Features: ['mean radius' 'mean texture' 'mean perimeter' 'mean area' 'mean smoothness']...
Class distribution: [212 357]

5.2 Training the Model 

# Create and train logistic regression model
model = LogisticRegression(max_iter=10000, random_state=42)
model.fit(X_train, y_train)

# Make predictions
y_pred = model.predict(X_test)
y_prob = model.predict_proba(X_test)[:, 1]

print("Training complete!")
print(f"Training accuracy: {model.score(X_train, y_train):.3f}")
print(f"Testing accuracy: {model.score(X_test, y_test):.3f}")

5.3 Evaluating Performance 

# Confusion matrix
cm = confusion_matrix(y_test, y_pred)
print("\nConfusion Matrix:")
print(cm)

# Detailed classification report
print("\nClassification Report:")
print(classification_report(y_test, y_pred, target_names=['Malignant', 'Benign']))

# AUC score
auc = roc_auc_score(y_test, y_prob)
print(f"\nAUC Score: {auc:.3f}")

Output:

Confusion Matrix:
[[ 59   4]
 [  3 105]]

Classification Report:
              precision    recall  f1-score   support
   Malignant       0.95      0.94      0.94        63
      Benign       0.96      0.97      0.97       108
    accuracy                           0.96       171

AUC Score: 0.991

Interpretation:

5.4 Interpreting Model Coefficients 

# Get feature names and coefficients
feature_names = data.feature_names
coefficients = model.coef_[0]

# Create dataframe for better visualization
coef_df = pd.DataFrame({
    'Feature': feature_names,
    'Coefficient': coefficients
}).sort_values('Coefficient', key=abs, ascending=False)

print("\nTop 5 Most Important Features:")
print(coef_df.head())

Example output:

                    Feature  Coefficient
worst perimeter             2.156
worst concave points        1.842
mean concave points         1.234
worst radius                0.987
worst texture              -0.654

Interpretation:

Important note: To properly interpret coefficients, features should be standardized (same scale). Otherwise, features with larger ranges will appear to have smaller coefficients just due to their scale.

5.5 Probability Predictions vs Hard Classifications 

# Show some examples with probabilities
print("\nSample Predictions:")
print("True | Pred | P(Benign) | Confidence")
print("-" * 40)
for i in range(10):
    true_label = "Benign" if y_test[i] == 1 else "Malignant"
    pred_label = "Benign" if y_pred[i] == 1 else "Malignant"
    prob = y_prob[i]
    confidence = max(prob, 1 - prob)
    
    print(f"{true_label:10} | {pred_label:10} | {prob:.3f} | {confidence:.3f}")

Example output:

True       | Pred       | P(Benign) | Confidence
----------------------------------------
Benign     | Benign     | 0.982      | 0.982
Malignant  | Malignant  | 0.045      | 0.955
Benign     | Benign     | 0.917      | 0.917
Benign     | Benign     | 0.523      | 0.523  <- Low confidence!
Malignant  | Malignant  | 0.112      | 0.888

Key observations:

5.6 When Logistic Regression Works Well

Logistic regression is a great choice when:

✅ Classes are approximately linearly separable

✅ You need interpretable results

✅ You want probability estimates

✅ As a baseline model

✅ With high-dimensional data

5.7 When Logistic Regression Struggles

Logistic regression has limitations:

❌ Non-linear decision boundaries

❌ Feature engineering burden

❌ Multicollinearity

Let's visualize when logistic regression succeeds vs fails:

Linear vs Nonlinear Data

Left panel: Linearly separable data

Right panel: Non-linearly separable data (XOR pattern)

Solutions for non-linear problems:

  1. Feature engineering: Create polynomial features, like we did in Week 3

    from sklearn.preprocessing import PolynomialFeatures
poly = PolynomialFeatures(degree=2, include_bias=False)
X_poly = poly.fit_transform(X)

  2. Use non-linear models: Decision trees (Week 6), neural networks (Weeks 10-11)

  3. Kernel methods: Transform features into higher dimensions (advanced topic)

6. Looking Ahead

6.1 The Linear Boundary Limitation

As we've seen, logistic regression always produces linear decision boundaries. This is both a strength (simple, interpretable) and a limitation (can't capture complex patterns).

What we mean by "linear":

Real-world data often has non-linear patterns:

6.2 Extension to Multi-Class Classification

So far, we've focused on binary classification (2 classes). But what about predicting among multiple categories?

Examples:

Two main approaches:

  1. One-vs-Rest (OvR):

    • Train KK binary classifiers (one per class)
    • Classifier kk predicts: "Is this class kk or not?"
    • Final prediction: Choose class with highest probability

  2. Softmax Regression (Multinomial Logistic Regression):

    • Direct generalization of logistic regression
    • Uses softmax function instead of sigmoid
    • Outputs probability distribution over all KK classes
    • Probabilities sum to 1.0

Good news: Sklearn handles multi-class automatically!

# Works the same way for 2, 3, or more classes!
model = LogisticRegression(multi_class='auto')
model.fit(X_train, y_train)
y_pred = model.predict(X_test)  # Works for K classes
y_prob = model.predict_proba(X_test)  # Returns K probabilities per sample

6.3 Next Week: Linear and Quadratic Discriminant Analysis

Next week, we'll learn about LDA (Linear Discriminant Analysis) and QDA (Quadratic Discriminant Analysis). These are alternative approaches to classification that:

Linear Discriminant Analysis (LDA):

Quadratic Discriminant Analysis (QDA):

Key comparison:

Method Boundary Shape Approach Best When
Logistic Regression Linear Discriminative (models P(y∣x)P(y|\mathbf{x})) Large datasets, need probabilities
LDA Linear Generative (models P(x∣y)P(\mathbf{x}|y)) Small datasets, Gaussian features
QDA Quadratic/Curved Generative (models P(x∣y)P(\mathbf{x}|y)) Non-linear patterns, enough data

6.4 Key Takeaways

Let's recap what you've learned this week:

  1. Classification predicts categories, not continuous values

    • Binary classification: Two classes (0 and 1)
    • Different from regression in fundamental ways

  2. Logistic regression uses sigmoid to get probabilities

    • Linear combination: z=wTxz = \mathbf{w}^T \mathbf{x}
    • Sigmoid transformation: P(y=1)=11+e−zP(y=1) = \frac{1}{1 + e^{-z}}
    • Decision boundary at P(y=1)=0.5P(y=1) = 0.5, i.e., where z=0z = 0

  3. Cross-entropy loss is designed for classification

    • Penalizes confident wrong predictions heavily
    • Creates convex optimization landscape
    • Better than squared error for classification

  4. Multiple metrics are needed to evaluate classifiers

    • Accuracy can be misleading with imbalanced data
    • Confusion matrix shows all types of errors
    • Precision, recall, F1-score focus on different aspects
    • ROC curve and AUC evaluate across all thresholds

  5. Linear decision boundaries are both strength and limitation

    • Strength: Simple, fast, interpretable
    • Limitation: Can't capture non-linear patterns
    • Solutions: Feature engineering or non-linear models

Congratulations! You now understand one of the most widely-used machine learning algorithms. Logistic regression is everywhere - from medical diagnosis to credit scoring to online advertising. Its simplicity and interpretability make it a go-to choice for many real-world applications.

7. Advanced Section (MPS439 Students Only)

This section is for MPS439 students. MPS311 students: feel free to read if curious, but this material is not required for your coursework.

7.1 Mathematical Derivation of Cross-Entropy Loss

Where does the cross-entropy loss actually come from? It's not arbitrary - it emerges naturally from maximum likelihood estimation.

7.1.1 The Maximum Likelihood Principle

Goal: Find parameters w\mathbf{w} that make our observed data most likely.

For a single training example (x,y)(\mathbf{x}, y):

We can write both cases compactly as:

P(y∣x,w)=[σ(wTx)]y×[1−σ(wTx)](1−y)P(y|\mathbf{x}, \mathbf{w}) = \left[\sigma(\mathbf{w}^T \mathbf{x})\right]^y \times \left[1 - \sigma(\mathbf{w}^T \mathbf{x})\right]^{(1-y)}

Why this works:

7.1.2 From Single Sample to Dataset

For nn independent samples, the likelihood of the entire dataset is:

L(w)=∏i=1nP(yi∣xi,w)\mathcal{L}(\mathbf{w}) = \prod_{i=1}^{n} P(y_i|\mathbf{x}_i, \mathbf{w})

L(w)=∏i=1n[σ(wTxi)]yi×[1−σ(wTxi)](1−yi)\mathcal{L}(\mathbf{w}) = \prod_{i=1}^{n} \left[\sigma(\mathbf{w}^T \mathbf{x}_i)\right]^{y_i} \times \left[1 - \sigma(\mathbf{w}^T \mathbf{x}_i)\right]^{(1-y_i)}

7.1.3 Log-Likelihood Transformation

Products are hard to optimize. Taking the logarithm converts products to sums:

log⁡L(w)=∑i=1n{yilog⁡[σ(wTxi)]+(1−yi)log⁡[1−σ(wTxi)]}\log \mathcal{L}(\mathbf{w}) = \sum_{i=1}^{n} \left\{y_i \log\left[\sigma(\mathbf{w}^T \mathbf{x}_i)\right] + (1-y_i) \log\left[1 - \sigma(\mathbf{w}^T \mathbf{x}_i)\right]\right\}

Why logarithm?

7.1.4 From Maximization to Minimization

Machine learning convention: minimize loss rather than maximize likelihood.

Negative log-likelihood per sample:

Loss=−1nlog⁡L(w)\text{Loss} = -\frac{1}{n} \log \mathcal{L}(\mathbf{w})

=−1n∑i=1n{yilog⁡(y^i)+(1−yi)log⁡(1−y^i)}= -\frac{1}{n} \sum_{i=1}^{n} \left\{y_i \log(\hat{y}_i) + (1-y_i) \log(1-\hat{y}_i)\right\}

Where y^i=σ(wTxi)\hat{y}_i = \sigma(\mathbf{w}^T \mathbf{x}_i)

This is exactly the cross-entropy loss!

Key insight: Minimizing cross-entropy is equivalent to maximizing likelihood of correct labels

7.2 Why Cross-Entropy Creates a Convex Problem

With linear regression, we had a closed-form solution (normal equation). With logistic regression, we must use iterative optimization.

The good news: The optimization landscape is convex, meaning:

Mathematical insight (intuition, not rigorous proof):

This is why we use cross-entropy instead of squared error - it gives us the mathematical guarantee of finding the best solution!

7.3 Implementing Gradient Descent from Scratch

Let's implement logistic regression ourselves to see what's happening under the hood.

7.3.1 Computing the Gradient

Through calculus (chain rule through sigmoid and log), we can show that:

Gradient of loss with respect to weights:

∂L∂w=1nXT(y^−y)\frac{\partial L}{\partial \mathbf{w}} = \frac{1}{n} \mathbf{X}^T (\hat{\mathbf{y}} - \mathbf{y})

Where:

Beautiful result: This has exactly the same form as linear regression! The only difference is that y^\hat{\mathbf{y}} uses sigmoid instead of being directly equal to wTx\mathbf{w}^T \mathbf{x}.

Derivation sketch (for the curious):

∂L∂w=∂∂w[−1n∑iyilog⁡(y^i)+(1−yi)log⁡(1−y^i)]\frac{\partial L}{\partial \mathbf{w}} = \frac{\partial}{\partial \mathbf{w}} \left[-\frac{1}{n} \sum_i y_i \log(\hat{y}_i) + (1-y_i) \log(1-\hat{y}_i)\right]

By chain rule:

=1n∑i[∂L∂y^i×∂y^i∂w]= \frac{1}{n} \sum_i \left[\frac{\partial L}{\partial \hat{y}_i} \times \frac{\partial \hat{y}_i}{\partial \mathbf{w}}\right]

Where:

∂L∂y^i=−yiy^i+1−yi1−y^i\frac{\partial L}{\partial \hat{y}_i} = -\frac{y_i}{\hat{y}_i} + \frac{1-y_i}{1-\hat{y}_i}

And (key sigmoid derivative property):

∂y^i∂w=y^i(1−y^i)⋅xi\frac{\partial \hat{y}_i}{\partial \mathbf{w}} = \hat{y}_i(1-\hat{y}_i) \cdot \mathbf{x}_i

Combining (algebra omitted):

=1n∑i(y^i−yi)xi=1nXT(y^−y)= \frac{1}{n} \sum_i (\hat{y}_i - y_i) \mathbf{x}_i = \frac{1}{n} \mathbf{X}^T (\hat{\mathbf{y}} - \mathbf{y})

7.3.2 The Gradient Descent Algorithm

Pseudocode:

  1. Initialize weights w\mathbf{w} randomly (or to zeros)
  2. For each iteration t=1,2,…,Tt = 1, 2, \ldots, T:
    • a. Compute predictions: y^=σ(Xw)\hat{\mathbf{y}} = \sigma(\mathbf{X}\mathbf{w})
    • b. Compute gradient: g=1nXT(y^−y)\mathbf{g} = \frac{1}{n} \mathbf{X}^T (\hat{\mathbf{y}} - \mathbf{y})
    • c. Update weights: w←w−αg\mathbf{w} \leftarrow \mathbf{w} - \alpha \mathbf{g}
    • d. (Optional) Compute loss to monitor convergence
  3. Return final weights w\mathbf{w}

Where α\alpha is the learning rate (step size).

7.3.3 Python Implementation

Here's a minimal implementation (~30 lines):

import numpy as np

def sigmoid(z):
    """Sigmoid function with clipping to avoid overflow"""
    return 1 / (1 + np.exp(-np.clip(z, -500, 500)))

def compute_loss(X, y, w):
    """Cross-entropy loss"""
    n = len(y)
    predictions = sigmoid(X @ w)
    # Clip to avoid log(0)
    predictions = np.clip(predictions, 1e-10, 1 - 1e-10)
    loss = -np.mean(y * np.log(predictions) + (1 - y) * np.log(1 - predictions))
    return loss

def gradient_descent_logistic(X, y, learning_rate=0.1, max_iters=1000):
    """Train logistic regression using gradient descent"""
    n, p = X.shape
    
    # Add intercept term
    X_with_intercept = np.c_[np.ones(n), X]
    
    # Initialize weights
    w = np.zeros(p + 1)
    
    # Store losses for visualization
    losses = []
    
    # Gradient descent loop
    for iteration in range(max_iters):
        # Forward pass: compute predictions
        predictions = sigmoid(X_with_intercept @ w)
        
        # Compute gradient
        gradient = X_with_intercept.T @ (predictions - y) / n
        
        # Update weights
        w = w - learning_rate * gradient
        
        # Track loss
        loss = compute_loss(X_with_intercept, y, w)
        losses.append(loss)
        
        # Optional: print progress
        if iteration % 100 == 0:
            print(f"Iteration {iteration}, Loss: {loss:.4f}")
    
    return w, losses

# Example usage
from sklearn.datasets import make_classification

# Generate data
X, y = make_classification(n_samples=200, n_features=2, n_redundant=0,
                           n_clusters_per_class=1, random_state=42)

# Train our implementation
weights, losses = gradient_descent_logistic(X, y, learning_rate=0.1, max_iters=500)

print(f"\nFinal weights: {weights}")
print(f"Final loss: {losses[-1]:.4f}")

7.3.4 Visualizing Convergence

Let's see how loss decreases over iterations with different learning rates:

Gradient Descent Convergence

Observations:

  1. Good learning rate (α=0.1\alpha=0.1, blue solid line):

    • Smooth, rapid convergence
    • Reaches global minimum
    • This is the "just right" setting

  2. Too low learning rate (α=0.01\alpha=0.01, green dotted line):

    • Very slow convergence
    • Takes many more iterations to reach minimum
    • Safe but inefficient

  3. Too high learning rate (α=1.0\alpha=1.0, red dashed line):

    • Oscillates or diverges
    • May never converge
    • Overshoots the minimum repeatedly

Choosing learning rate:

7.3.5 Comparing with Sklearn 

from sklearn.linear_model import LogisticRegression

# Our implementation
w_ours, _ = gradient_descent_logistic(X, y, learning_rate=0.1, max_iters=1000)

# Sklearn's implementation
model_sklearn = LogisticRegression(max_iter=1000)
model_sklearn.fit(X, y)
w_sklearn = np.concatenate([[model_sklearn.intercept_[0]], model_sklearn.coef_[0]])

print("Our weights:    ", w_ours)
print("Sklearn weights:", w_sklearn)
print("Difference:     ", np.abs(w_ours - w_sklearn))

Why sklearn is better:

But understanding the implementation helps you know what's happening inside the black box!

7.4 Connection to Neural Networks

Logistic regression is actually a single-layer neural network!

The components:

Deep learning (Weeks 10-11) extends this by:

Gradient descent still works because:

Understanding logistic regression is your first step toward understanding deep learning!

7.5 Advanced Topics: Regularization Revisited

Just like with linear regression (Week 3), we can add regularization to logistic regression:

L2 Regularization (Ridge):

Loss=CrossEntropy+λ∥w∥22\text{Loss} = \text{CrossEntropy} + \lambda \|\mathbf{w}\|_2^2

L1 Regularization (Lasso):

Loss=CrossEntropy+λ∥w∥1\text{Loss} = \text{CrossEntropy} + \lambda \|\mathbf{w}\|_1

In sklearn:

# L2 regularization (default)
model_ridge = LogisticRegression(penalty='l2', C=1.0)

# L1 regularization
model_lasso = LogisticRegression(penalty='l1', solver='liblinear', C=1.0)

# Note: C is inverse of regularization strength
# Smaller C = more regularization

Where C=1λC = \frac{1}{\lambda} (inverse of regularization strength)

When to use:

Summary: Learning Outcomes Checklist

Core Learning Outcomes (All Students) ✓

By now, you should be able to:

Advanced Learning Outcomes (MPS439 Students) ✓

Additionally, you should be able to:

What's Next?

This week's lab: You'll practice applying logistic regression to real datasets, tuning thresholds, and comparing different evaluation metrics.

Next week: Linear and Quadratic Discriminant Analysis (LDA/QDA) - alternative approaches to classification that model class distributions and can handle curved decision boundaries.

Looking ahead: Decision Trees (Week 6) will give us truly non-linear decision boundaries without requiring manual feature engineering!

Additional Resources

For deeper understanding:

Practice datasets:

Good luck with your studies! Remember to experiment with the code and visualize your results. Machine learning is best learned by doing!

Dr. Wei Xing