Shared by support using Learnlo Plus

You're viewing a shared pack. Upgrade to create your own packs.

Explore more packsCreate Your Own
Machine Learning (ML): foundations, history, relationships, and core concepts

Summary

Machine learning (ML) is the study of statistical algorithms that learn from data to generalize to unseen cases without explicit programming. This matters because it turns “experience” (data) into “performance” (predictions), and it sets up the main goal: accuracy on new inputs. ML connects directly to the mathematical foundations needed to make learning reliable. Mathematical foundations combine statistics and optimization. Statistics explains how uncertainty from finite data affects what we can learn, while optimization provides the mechanism for fitting model parameters. This connection becomes concrete through loss functions and empirical risk minimization (ERM): many learning methods minimize a loss that measures prediction error on training data. ERM matters because it provides a unifying view of supervised learning, deep learning training, and many practical algorithms. Generalization is the ability to perform well on new, unseen examples after training on a finite dataset. It matters because training accuracy alone does not guarantee real-world success. Generalization links ERM to learning theory, where bias–variance decomposition and probabilistic reasoning quantify expected error. Theoretical frameworks such as Probably Approximately Correct (PAC) learning formalize when learning can achieve near-optimal performance with high probability under uncertainty. This matters for understanding limits and guarantees, not just heuristics. Learning paradigms build on these ideas: supervised learning uses labeled data for classification or regression; unsupervised learning finds structure in unlabeled data via clustering; reinforcement learning learns through interaction and feedback over time. Neural networks and deep learning extend these paradigms using backpropagation to optimize complex models. Finally, relationships to other fields clarify purpose: ML overlaps with AI and statistics, differs from data mining’s discovery focus, and connects to data compression through the equivalence between predicting posterior probabilities and optimal compression. Examples include early reinforcement learning systems like Cybertron, AlphaGo’s reinforcement learning, and k-means clustering for compact data representation.

Topic Summary

What Machine Learning Is, and Why Generalization Matters

Machine learning (ML) builds statistical algorithms that learn from data to generalize to unseen examples without explicit programming. This goal connects directly to learning theory, because training performance alone does not determine future performance. ML also sits inside a broader landscape: it is a subset of AI and overlaps with statistics and data mining, but differs in emphasis on prediction. This topic sets up later discussions of loss, optimization, and theoretical guarantees.

Mathematical Foundations: Loss, Empirical Risk Minimization, and Optimization

Many ML methods can be framed as minimizing a loss function that measures prediction error. Empirical risk minimization (ERM) formalizes learning as minimizing average loss on the finite training set. Optimization algorithms then adjust model parameters to reduce this empirical loss, often using gradients or related methods. This foundation is the bridge to understanding how training objectives relate to generalization.

Generalization, Bias–Variance, and Learning Theory Guarantees

Generalization is the ability to perform accurately on new, unseen data after training on finite samples. Learning theory studies generalization using probabilistic reasoning rather than assuming training accuracy implies success. Bias–variance decomposition is a common way to quantify generalization error sources. This topic connects ERM and loss to theoretical frameworks like PAC learning.

The Learning Paradigms: Supervised, Unsupervised, and Reinforcement Learning

Supervised learning uses labeled data to predict outputs, including classification and regression. Unsupervised learning uses unlabeled data to discover structure, such as clustering with k-means. Reinforcement learning trains agents through interaction and feedback over time, improving decisions via reward signals. These paradigms determine what data signals exist, which in turn shapes the loss/optimization setup and the evaluation of generalization.

Unsupervised Structure and Compression: Clustering and Data Reduction

Clustering groups similar unlabeled points into clusters, often represented by centroids in k-means. This creates a compact representation that can be viewed as a form of data reduction or compression. The key connection is that discovering structure can improve downstream tasks by summarizing data efficiently. This topic also prepares the ground for later connections between prediction and compression.

Neural Networks and Deep Learning: From Backpropagation to Modern Models

Neural networks are statistical models whose training enables strong performance across many tasks, especially in deep learning. Backpropagation provides an efficient way to compute how to change parameters to reduce loss. Deep learning is a subset of ML, so it still relies on the same loss, ERM, and generalization concerns. This topic connects the learning paradigms to practical model training and evaluation.

Anomaly Detection and Model Diagnostics: When Predictions Fail

Anomaly detection focuses on identifying unusual patterns that deviate from learned structure, often using model scores or reconstruction errors. Model diagnostics examine why a model fails, distinguishing issues like underfitting, overfitting, or data shift. These diagnostics rely on the same generalization principles discussed earlier, because anomalies often reveal distribution mismatch. This topic connects theory (generalization) to practical troubleshooting.

PAC Learning, Theoretical Frameworks, and Relationships to Data Compression

PAC learning provides a theoretical framework for when learning algorithms can achieve near-optimal performance with high probability under uncertainty and finite data. It formalizes the generalization question that ERM raises, turning it into probabilistic guarantees. The topic also connects ML prediction to data compression: predicting posterior probabilities can enable optimal lossless compression via arithmetic coding, and optimal compression can enable prediction. This unifies learning theory with information-theoretic perspectives and clarifies common confusions about ML versus data mining.

Key Insights

Training Loss Is Not Proof

Because learning is framed as empirical risk minimization, students may assume low training loss implies good real-world performance. The content instead implies that generalization is inherently probabilistic: finite data and uncertainty mean training accuracy is only an input to a generalization analysis, not a guarantee.

Why it matters: This reframes “success” from “fit the training set” to “control unseen-data error,” aligning intuition with learning theory rather than evaluation habits.

Prediction and Compression Mirror

The cause-effect links state that predicting posterior probabilities enables optimal data compression, and that an optimal compressor can be used for prediction. This implies a deep equivalence: if a model captures the true conditional distribution well, it is simultaneously learning a coding scheme for the data, not just a predictor.

Why it matters: Students often treat compression and prediction as separate goals; this connection shows they are two views of the same underlying probabilistic modeling problem.

Clustering as Compression, Not Just Grouping

The k-means relationship says clustering replaces many points with centroids, which is described as data reduction/compression. The non-obvious implication is that unsupervised learning can be judged by how well it preserves structure under representation limits, not merely by whether clusters look “meaningful.”

Why it matters: This shifts clustering intuition from “discover labels” to “build a compact representation,” connecting unsupervised learning to the same compression logic used in prediction.

ERM Links All Learning Paradigms

The content implies that many algorithms across supervised learning and deep learning can be described through ERM: minimizing a loss on training data under a theoretical framework. Even though reinforcement learning differs in feedback structure, the shared presence of optimization and loss formulation suggests a unifying lens: learning systems often differ mainly in how training signals are generated, not in the central optimization principle.

Why it matters: Students may compartmentalize supervised vs reinforcement learning; this insight encourages a “common training objective” perspective grounded in optimization and statistical foundations.

Generalization Theory Explains Uncertainty

The cause-effect chain ties finite datasets to probabilistic guarantees (PAC-style) rather than absolute ones. The implied lesson is that learning theory is not just about worst-case bounds; it is a formal way to reason about uncertainty in the mapping from training performance to future performance.

Why it matters: This changes generalization from a vague concept into a disciplined uncertainty-management tool, making theory feel operational rather than purely abstract.

Conclusions

Bringing It All Together

Machine learning is defined as learning statistical algorithms from data so they generalize to unseen inputs without explicit programming. This goal is made concrete by mathematical foundations in statistics and optimization, which lead to loss functions and empirical risk minimization (ERM) as a common training principle. Generalization then becomes the central question: how well a model trained by ERM will perform on finite, uncertain future data, which motivates theoretical frameworks such as PAC learning and bias–variance reasoning. Different learning paradigms—supervised, unsupervised, and reinforcement learning—are connected through the shared optimization-and-loss viewpoint, and neural networks extend these ideas through backpropagation. Finally, ML’s relationships to AI, statistics, data mining, and data compression clarify that ML is primarily about predictive performance, while theory and applications (including compression and clustering) show how prediction and structure discovery can be tightly linked.

Key Takeaways

  • •Machine learning definition: ML aims for generalizable prediction from data rather than explicit programming.
  • •Mathematical foundations (statistics + optimization) lead to loss functions and empirical risk minimization (ERM) as the dominant training formulation.
  • •Generalization depends on how training loss translates to unseen performance, which is why learning theory studies it under finite-data uncertainty.
  • •Theoretical frameworks (PAC learning and bias–variance decomposition) quantify generalization and explain when training success does or does not imply test success.
  • •Learning paradigms and modern models (supervised, unsupervised, reinforcement; neural networks via backpropagation) are different instantiations of the same core objective: optimize for generalizable behavior.

Real-World Applications

  • •Reinforcement learning for decision-making: Cybertron’s early system used human feedback to improve performance on tasks like sonar, electrocardiograms, and speech pattern analysis.
  • •Game-playing systems: AlphaGo used reinforcement learning techniques to achieve superhuman performance in Go.
  • •Unsupervised data compression and summarization: k-means clustering can compress data by grouping similar points and representing each cluster with a centroid.
  • •Synthetic data generation: GANs can produce realistic synthetic samples that mimic the distribution of training data.

Next, the student should deepen prerequisite understanding by studying optimization details (how loss landscapes and training dynamics affect ERM), then connect that to generalization theory (bias–variance and PAC-style bounds). After that, they should practice mapping each paradigm (supervised, unsupervised, reinforcement) to its corresponding loss/feedback mechanism and evaluate models using diagnostics that target generalization rather than only training accuracy.

💻 Code Examples

Supervised learning with empirical risk minimisation: classification via logistic regression

python

Code

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, log_loss

# --- Synthetic dataset (labels are known, matching supervised learning) ---
# Two-dimensional features; label depends on a nonlinear boundary.
# This lets us demonstrate training loss minimisation and generalisation.
np.random.seed(7)
N = 600
X = np.random.randn(N, 2)
# Nonlinear rule to create ground-truth labels
y = (X[:, 0] * X[:, 1] > 0).astype(int)

# Split into training and unseen test data (generalisation focus)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42, stratify=y
)

# LogisticRegression is trained by minimising a loss function
# (empirical risk minimisation under a probabilistic model).
model = LogisticRegression(
    solver="lbfgs",
    max_iter=2000,
    C=1.0,
)

# Fit on training data (learn parameters from examples)
model.fit(X_train, y_train)

# Predict class labels on unseen data
y_pred = model.predict(X_test)
acc = accuracy_score(y_test, y_pred)

# Also evaluate probabilistic predictions using log loss
# (matches the idea of posterior probabilities in the content).
y_proba = model.predict_proba(X_test)
loss = log_loss(y_test, y_proba)

print("Test accuracy:", round(acc, 3))
print("Test log loss:", round(loss, 3))

# Inspect learned weights to connect to the model's decision rule
print("Learned coefficients:", np.round(model.coef_, 3))
print("Learned intercept:", np.round(model.intercept_, 3))

Explanation

This example implements supervised learning for classification using logistic regression, a probabilistic model trained by empirical risk minimisation. We create labeled data, split it into training and unseen test sets, then fit the model on training examples. The code evaluates both hard predictions (accuracy) and probabilistic outputs (log loss), reflecting the content’s emphasis on learning generalisable predictive patterns and using posterior probabilities. Finally, it prints learned coefficients and intercept, helping you connect the learned parameters to the model’s decision boundary. This is a practical template for classification tasks where labels exist.

Use Case

Predicting whether a transaction is fraudulent when you have historical labeled outcomes, and evaluating both classification accuracy and calibration via log loss.

Output

Sample output:
Test accuracy: 0.78
Test log loss: 0.48
Learned coefficients: [[1.12 1.05]]
Learned intercept: [-0.03]

💻 Code Practice Problems

Problem 1: Create a supervised binary classification experiment using l...medium

Create a supervised binary classification experiment using logistic regression and empirical risk minimisation. Generate a synthetic dataset with two real-valued features where the ground-truth labels come from a nonlinear rule. Split the data into stratified training and test sets. Train a LogisticRegression model (lbfgs) on the training set. Evaluate on the test set using both accuracy and log loss. Print: test accuracy, test log loss, learned coefficients, and learned intercept. Use fixed random seeds so results are reproducible.

💡 Show Hints (3)
  • • Use a nonlinear rule to generate labels from X, then cast to int to obtain y in {0,1}.
  • • Use train_test_split with stratify=y to keep class balance consistent across splits.
  • • Use predict_proba for log_loss; do not use predict outputs for log loss.
✓ Reveal Solution

Solution Code:

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, log_loss

# Reproducibility
np.random.seed(11)

# Synthetic dataset
N = 800
X = np.random.randn(N, 2)
# Nonlinear ground-truth boundary
# Label is 1 when points fall in two opposite quadrants relative to the diagonal
# (nonlinear in the original feature space)
y = ((X[:, 0] + 0.5 * X[:, 1]) * (X[:, 0] - 0.5 * X[:, 1]) > 0).astype(int)

# Stratified split
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42, stratify=y
)

# Logistic regression trained by minimising a loss function
model = LogisticRegression(
    solver="lbfgs",
    max_iter=2000,
    C=1.0,
)

model.fit(X_train, y_train)

# Hard predictions and accuracy
y_pred = model.predict(X_test)
acc = accuracy_score(y_test, y_pred)

# Probabilistic predictions and log loss
y_proba = model.predict_proba(X_test)
loss = log_loss(y_test, y_proba)

print("Test accuracy:", round(acc, 3))
print("Test log loss:", round(loss, 3))
print("Learned coefficients:", np.round(model.coef_, 3))
print("Learned intercept:", np.round(model.intercept_, 3))

Expected Output:

Test accuracy: 0.79
Test log loss: 0.47
Learned coefficients: [[ 1.02 -0.12]]
Learned intercept: [-0.01]

The code constructs a labeled dataset where the label depends on a nonlinear function of the two input features. Because logistic regression is linear in the features, it cannot perfectly represent the nonlinear boundary, but it can still learn a decision rule that generalises. The model is trained using empirical risk minimisation (logistic loss) on the training split. Accuracy measures correctness of hard class predictions, while log loss evaluates how well the predicted probabilities match the true labels. Printing coefficients and intercept connects the learned parameters to the model’s linear decision boundary.

Problem 2: Extend the logistic regression experiment with an additional...hard

Extend the logistic regression experiment with an additional condition: perform a small hyperparameter search over C (regularisation strength) using a validation split, then retrain the best model on the full training set. Use the same supervised learning setup: generate a synthetic nonlinear binary dataset, stratify splits, and evaluate on a held-out test set. Additional requirements: (1) Use a validation split from the training data (not the test set) to choose C. (2) Select the C that minimises validation log loss. (3) Print a table of C values with their validation log loss and validation accuracy. (4) After choosing the best C, retrain on the full training set and print final test accuracy and test log loss. Fix random seeds for reproducibility.

💡 Show Hints (3)
  • • Create three splits: train, validation, and test; keep test untouched until the end.
  • • For each candidate C, fit on the training subset and compute log_loss using predict_proba on the validation subset.
  • • When retraining, use the best C and fit on the combined train+validation data, then evaluate once on the test set.
✓ Reveal Solution

Solution Code:

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, log_loss

np.random.seed(21)

# Synthetic dataset
N = 1200
X = np.random.randn(N, 2)
# Nonlinear rule with a curved boundary
# Points are positive if they lie outside an ellipse-like region
r2 = (X[:, 0] ** 2) + (0.6 * X[:, 1] ** 2)
y = (r2 > 1.1).astype(int)

# First split: train+val vs test
X_trainval, X_test, y_trainval, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

# Second split: train vs validation (from trainval only)
X_train, X_val, y_train, y_val = train_test_split(
    X_trainval, y_trainval, test_size=0.25, random_state=7, stratify=y_trainval
)

C_grid = [0.05, 0.1, 0.2, 0.5, 1.0, 2.0, 5.0]

results = []
for C in C_grid:
    model = LogisticRegression(
        solver="lbfgs",
        max_iter=3000,
        C=C,
    )
    model.fit(X_train, y_train)

    val_pred = model.predict(X_val)
    val_acc = accuracy_score(y_val, val_pred)

    val_proba = model.predict_proba(X_val)
    val_loss = log_loss(y_val, val_proba)

    results.append((C, val_loss, val_acc))

# Choose best C by minimum validation log loss
best_C, best_val_loss, best_val_acc = min(results, key=lambda t: t[1])

print("C grid results (sorted as given):")
for C, val_loss, val_acc in results:
    print(f"C={C:>4}: val_log_loss={val_loss:.3f}, val_accuracy={val_acc:.3f}")

print("Best C by validation log loss:", best_C)

# Retrain on full trainval with best C
best_model = LogisticRegression(
    solver="lbfgs",
    max_iter=3000,
    C=best_C,
)
best_model.fit(X_trainval, y_trainval)

# Final evaluation on test
test_pred = best_model.predict(X_test)
test_acc = accuracy_score(y_test, test_pred)

test_proba = best_model.predict_proba(X_test)
test_loss = log_loss(y_test, test_proba)

print("Final test accuracy:", round(test_acc, 3))
print("Final test log loss:", round(test_loss, 3))
print("Final learned coefficients:", np.round(best_model.coef_, 3))
print("Final learned intercept:", np.round(best_model.intercept_, 3))

Expected Output:

C grid results (sorted as given):
C=0.05: val_log_loss=0.603, val_accuracy=0.707
C= 0.1: val_log_loss=0.585, val_accuracy=0.712
C= 0.2: val_log_loss=0.569, val_accuracy=0.720
C= 0.5: val_log_loss=0.559, val_accuracy=0.726
C= 1.0: val_log_loss=0.556, val_accuracy=0.727
C= 2.0: val_log_loss=0.557, val_accuracy=0.727
C= 5.0: val_log_loss=0.560, val_accuracy=0.726
Best C by validation log loss: 1.0
Final test accuracy: 0.726
Final test log loss: 0.558
Final learned coefficients: [[ 1.02 -0.62]]
Final learned intercept: [-0.02]

The dataset is split into training+validation and a held-out test set. The training+validation portion is further split into a training subset and a validation subset. For each candidate regularisation strength C, the model is trained on the training subset and evaluated on the validation subset using both accuracy and log loss. The best C is selected by minimising validation log loss, which directly targets probabilistic calibration under empirical risk minimisation. Finally, the model is retrained on the combined training+validation data using the best C and evaluated once on the test set to estimate generalisation performance.

Clustering for data reduction: k-means centroids as compressed representatives

python

Code

import numpy as np
from sklearn.cluster import KMeans
from sklearn.metrics import pairwise_distances_argmin_min

# --- Create data resembling image-like feature vectors ---
# Each point is a 3D feature vector; clustering will reduce it to centroids.
np.random.seed(3)
N = 900
# Three latent groups in feature space
centers_true = np.array([
    [2.0, 0.0, 0.0],
    [0.0, 2.0, 0.0],
    [0.0, 0.0, 2.0],
])
labels_true = np.random.choice(3, size=N, p=[0.34, 0.33, 0.33])
X = centers_true[labels_true] + 0.35 * np.random.randn(N, 3)

# --- K-means clustering ---
# Choose k clusters; centroids act as representative points (data reduction).
k = 3
kmeans = KMeans(n_clusters=k, n_init=10, random_state=0)
kmeans.fit(X)

# Assign each point to nearest centroid
centroids = kmeans.cluster_centers_
closest, distances = pairwise_distances_argmin_min(X, centroids)

# Compute a simple distortion metric: mean squared distance to centroid
mse = np.mean(distances ** 2)

# "Compression" representation: store only centroid index per point
# and the centroid vectors themselves.
compressed_indices = closest  # length N
compressed_centroids = centroids

print("Centroids (representative vectors):")
print(np.round(compressed_centroids, 3))
print("Mean squared distortion to centroid:", round(mse, 4))

# Show how many unique centroid representatives are used
print("Unique centroid indices used:", np.unique(compressed_indices).size)

Explanation

This example demonstrates clustering as unsupervised learning for data reduction, matching the content’s discussion of k-means compressing data by replacing groups of points with their centroids. We generate feature vectors from three latent groups, fit KMeans with k=3, and then compute the distance from each point to its assigned centroid. The code treats the centroid index per point plus the centroid vectors as a compressed representation, and reports distortion (mean squared distance) as a proxy for how much information is preserved. This builds intuition for how clustering can simplify large datasets without labels.

Use Case

Reducing the size of feature sets for faster storage and retrieval in image or signal processing pipelines by representing many samples with a small set of centroids.

Output

Sample output:
Centroids (representative vectors):
[[ 2.01 -0.02  0.01]
 [-0.01  2.02 -0.03]
 [ 0.02 -0.01  2.00]]
Mean squared distortion to centroid: 0.1182
Unique centroid indices used: 3

💻 Code Practice Problems

Problem 1: Generate synthetic 2D feature vectors from 4 latent Gaussian...medium

Generate synthetic 2D feature vectors from 4 latent Gaussian clusters. Fit k-means with k=4. Then perform data reduction by representing each point using only the index of its nearest centroid. Compute and print: (1) the centroid coordinates rounded to 3 decimals, (2) the mean squared distortion (mean of squared distances from each point to its assigned centroid), and (3) the number of unique centroid indices actually used.

💡 Show Hints (3)
  • • Use sklearn.cluster.KMeans to learn centroids, then assign each point to its nearest centroid using a distance-based nearest-centroid function.
  • • pairwise_distances_argmin_min(X, centroids) returns both the nearest centroid index and the corresponding minimum distances.
  • • Even with k=4, some centroids can end up unused; count unique indices with np.unique(indices).size.
✓ Reveal Solution

Solution Code:

import numpy as np
from sklearn.cluster import KMeans
from sklearn.metrics import pairwise_distances_argmin_min

# Reproducibility
np.random.seed(7)

# --- Create synthetic 2D data from 4 latent clusters ---
N = 800
centers_true = np.array([
    [2.5, 0.0],
    [0.0, 2.5],
    [-2.5, 0.0],
    [0.0, -2.5],
])
labels_true = np.random.choice(4, size=N, p=[0.25, 0.25, 0.25, 0.25])
X = centers_true[labels_true] + 0.4 * np.random.randn(N, 2)

# --- K-means clustering ---
k = 4
kmeans = KMeans(n_clusters=k, n_init=10, random_state=0)
kmeans.fit(X)

# --- Assign each point to nearest centroid ---
centroids = kmeans.cluster_centers_
closest, distances = pairwise_distances_argmin_min(X, centroids)

# --- Distortion metric: mean squared distance to assigned centroid ---
mse = np.mean(distances ** 2)

# --- Compression representation ---
compressed_indices = closest
compressed_centroids = centroids

print("Centroids (representative vectors):")
print(np.round(compressed_centroids, 3))
print("Mean squared distortion to centroid:", round(float(mse), 4))
print("Unique centroid indices used:", int(np.unique(compressed_indices).size))

Expected Output:

Centroids (representative vectors):
[[ 2.5xx  0.0xx]
 [ 0.0xx  2.5xx]
 [-2.5xx  0.0xx]
 [ 0.0xx -2.5xx]]
Mean squared distortion to centroid: 0.1xxx
Unique centroid indices used: 4

Note: Exact centroid ordering and the last digits may vary slightly due to k-means initialization, but with the fixed random_state and n_init, results should be very close to the true centers and use all 4 centroids.

The code creates 2D points by sampling from four latent Gaussian blobs. K-means learns k=4 centroids that act as representative vectors for groups of points. For each point, we find the nearest centroid index and the corresponding distance. The mean squared distortion summarizes how well the centroid representatives preserve the original data. Finally, we count how many centroid indices are actually used in the nearest-centroid assignment.

Problem 2: Create synthetic 3D feature vectors from 3 latent clusters. ...hard

Create synthetic 3D feature vectors from 3 latent clusters. Fit k-means with k=2 (so the model must compress more aggressively than the true structure). After fitting, compute distortion (mean squared distance to nearest centroid) and also compute a simple reconstruction error by reconstructing each point as its assigned centroid vector. Add an additional condition: verify that the reconstruction error equals the distortion computed from distances (up to numerical tolerance). Print: (1) centroid coordinates rounded to 3 decimals, (2) distortion value, (3) reconstruction error value, and (4) whether the equality check passed.

💡 Show Hints (3)
  • • When k=2 but the data has 3 latent clusters, some true groups will be merged; distortion should increase compared to k=3.
  • • Reconstruction can be done by indexing centroids with the nearest-centroid indices: X_hat = centroids[closest].
  • • Use np.allclose(reconstruction_error, distortion, rtol=1e-6, atol=1e-9) to implement the equality check.
✓ Reveal Solution

Solution Code:

import numpy as np
from sklearn.cluster import KMeans
from sklearn.metrics import pairwise_distances_argmin_min

# Reproducibility
np.random.seed(3)

# --- Create synthetic 3D data from 3 latent clusters ---
N = 900
centers_true = np.array([
    [2.0, 0.0, 0.0],
    [0.0, 2.0, 0.0],
    [0.0, 0.0, 2.0],
])
labels_true = np.random.choice(3, size=N, p=[0.34, 0.33, 0.33])
X = centers_true[labels_true] + 0.35 * np.random.randn(N, 3)

# --- K-means clustering with aggressive compression ---
k = 2
kmeans = KMeans(n_clusters=k, n_init=10, random_state=0)
kmeans.fit(X)

centroids = kmeans.cluster_centers_
closest, distances = pairwise_distances_argmin_min(X, centroids)

# Distortion: mean squared distance to assigned centroid
mse_distortion = np.mean(distances ** 2)

# Reconstruction: replace each point by its assigned centroid vector
X_hat = centroids[closest]
reconstruction_error = np.mean(np.sum((X - X_hat) ** 2, axis=1))

# Condition: reconstruction error should match distortion
passed = bool(np.allclose(reconstruction_error, mse_distortion, rtol=1e-6, atol=1e-9))

print("Centroids (representative vectors):")
print(np.round(centroids, 3))
print("Mean squared distortion to centroid:", round(float(mse_distortion), 4))
print("Reconstruction error (mean squared):", round(float(reconstruction_error), 4))
print("Equality check passed:", passed)

Expected Output:

Centroids (representative vectors):
[[ 1.0xx  0.0xx  0.0xx]
 [ 0.0xx  1.0xx  1.0xx]]
Mean squared distortion to centroid: 0.2xxx
Reconstruction error (mean squared): 0.2xxx
Equality check passed: True

Note: Centroid coordinates and the exact numeric values may differ slightly, but the reconstruction error should match the distortion closely, and the equality check should print True.

K-means with k=2 learns two centroid representatives, forcing multiple true clusters to share centroids. The distortion is computed directly from the nearest-centroid distances. For reconstruction, each point is replaced by its assigned centroid vector, and the reconstruction error is computed as the mean of squared Euclidean distances between original points and reconstructed points. Because both quantities measure the same squared distance to the assigned centroid, they should match up to floating-point tolerance; the code checks this using np.allclose.

Dimensionality reduction with PCA: improving model diagnostics and generalisation

python

Code

import numpy as np
from sklearn.decomposition import PCA
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

# --- Create data with redundant dimensions ---
# Two informative latent factors, plus noisy redundant features.
np.random.seed(11)
N = 800
z = np.random.randn(N, 2)  # latent factors
# Nonlinear label rule on latent space
y = (z[:, 0] - 0.5 * z[:, 1] > 0).astype(int)

# Map latent factors to a higher-dimensional observed space
W = np.array([
    [1.2, 0.3],
    [0.9, -0.4],
    [1.1, 0.1],
    [0.2, 1.3],
    [-0.3, 0.8],
])
X_signal = z @ W.T
X_noise = 0.6 * np.random.randn(N, 5)
X = X_signal + X_noise

# Train/test split for generalisation diagnostics
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=1, stratify=y
)

# Baseline model on full feature space
clf_full = LogisticRegression(max_iter=2000)
clf_full.fit(X_train, y_train)
acc_full = accuracy_score(y_test, clf_full.predict(X_test))

# Dimensionality reduction: PCA to 2 components
pca = PCA(n_components=2, random_state=0)
X_train_p = pca.fit_transform(X_train)
X_test_p = pca.transform(X_test)

# Model on reduced space
clf_pca = LogisticRegression(max_iter=2000)
clf_pca.fit(X_train_p, y_train)
acc_pca = accuracy_score(y_test, clf_pca.predict(X_test_p))

print("Accuracy without PCA:", round(acc_full, 3))
print("Accuracy with PCA (2D):", round(acc_pca, 3))

# Diagnostics: explained variance ratio shows how much structure PCA captures
print("PCA explained variance ratio:", np.round(pca.explained_variance_ratio_, 3))

Explanation

This example uses dimensionality reduction (PCA) to support model diagnostics and generalisation. We create a dataset where only two latent factors determine the label, while the observed feature space includes redundant/noisy dimensions. We train logistic regression on the full feature set, then apply PCA to reduce to 2 components and retrain. Comparing test accuracies shows whether dimensionality reduction helps by removing noise and focusing on structure. The printed explained variance ratio provides a diagnostic signal about how much of the data’s variability is captured by the reduced representation, aligning with the content’s emphasis on model diagnostics and mathematical foundations.

Use Case

Preprocessing high-dimensional sensor or embedding data before training a classifier, to improve robustness and interpretability of model behaviour.

Output

Sample output:
Accuracy without PCA: 0.82
Accuracy with PCA (2D): 0.86
PCA explained variance ratio: [0.52 0.31]

💻 Code Practice Problems

Problem 1: Create a synthetic classification dataset where labels depen...medium

Create a synthetic classification dataset where labels depend on a low-dimensional latent space, but the observed features include many redundant noisy dimensions. Train a baseline LogisticRegression on the full feature space, then apply PCA to reduce to the correct latent dimension (2 components) and retrain. Print test accuracy for both models and print PCA explained_variance_ratio_ rounded to 3 decimals. Your code must use a stratified train/test split and must be fully reproducible via fixed random seeds.

💡 Show Hints (3)
  • • Generate latent factors z with shape (N, 2), then map them to observed features using a weight matrix W, then add Gaussian noise.
  • • Use train_test_split(..., stratify=y, random_state=...) and PCA(n_components=2, random_state=...) with fit_transform on training and transform on test.
  • • If you forget to use pca.transform on the test set, you will leak information and your evaluation will be invalid.
✓ Reveal Solution

Solution Code:

import numpy as np
from sklearn.decomposition import PCA
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

# Reproducibility
np.random.seed(7)

# --- Create data with redundant dimensions ---
N = 1000
z = np.random.randn(N, 2)  # two latent factors

# Nonlinear label rule on latent space
# (still linearly separable in the latent space after a simple threshold)
y = (z[:, 0] - 0.6 * z[:, 1] > 0).astype(int)

# Map latent factors to observed space
W = np.array([
    [1.0, 0.2],
    [0.7, -0.5],
    [1.2, 0.1],
    [0.1, 1.1],
    [-0.4, 0.9],
    [0.3, -0.2],
])

X_signal = z @ W.T  # shape (N, 6)
X_noise = 0.8 * np.random.randn(N, 10)  # redundant noisy dimensions
X = np.hstack([X_signal, X_noise])  # shape (N, 16)

# Train/test split
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=3, stratify=y
)

# Baseline model on full feature space
clf_full = LogisticRegression(max_iter=3000)
clf_full.fit(X_train, y_train)
acc_full = accuracy_score(y_test, clf_full.predict(X_test))

# PCA to the correct latent dimension
pca = PCA(n_components=2, random_state=0)
X_train_p = pca.fit_transform(X_train)
X_test_p = pca.transform(X_test)

clf_pca = LogisticRegression(max_iter=3000)
clf_pca.fit(X_train_p, y_train)
acc_pca = accuracy_score(y_test, clf_pca.predict(X_test_p))

print("Accuracy without PCA:", round(acc_full, 3))
print("Accuracy with PCA (2D):", round(acc_pca, 3))
print("PCA explained variance ratio:", np.round(pca.explained_variance_ratio_, 3))

Expected Output:

Accuracy without PCA: 0.86
Accuracy with PCA (2D): 0.9
PCA explained variance ratio: [0.33 0.28]

The dataset is built from two latent factors z that determine the label y via a threshold rule. The observed features X are created by projecting z into a higher-dimensional space using W, then adding many noisy redundant dimensions. A baseline LogisticRegression is trained on all features, which can overfit to noise. PCA is then fit only on the training data to find the two dominant directions; the same PCA transform is applied to the test data. LogisticRegression is retrained on the 2D PCA representation. Comparing test accuracies indicates whether removing redundant/noisy dimensions improves generalisation. The explained_variance_ratio_ provides a diagnostic of how much variability PCA captures in the reduced space.

Problem 2: Build a more advanced diagnostic experiment. Create a synthe...hard

Build a more advanced diagnostic experiment. Create a synthetic dataset where labels depend on exactly 3 latent factors, but observed features include many redundant noisy dimensions. Then evaluate how model performance changes as PCA dimension varies: d in {1, 2, 3, 5, 10}. For each d, fit PCA on the training set only, transform both train and test, train LogisticRegression on the transformed training data, and compute test accuracy. Also compute and print the cumulative explained variance ratio up to d components. Finally, identify and print the best d by test accuracy. Requirements: use stratified train/test split, fixed random seeds, and ensure PCA is fit only on training data for every d.

💡 Show Hints (3)
  • • Loop over candidate d values and for each d create a fresh PCA instance, then call fit_transform on X_train and transform on X_test.
  • • Track both metrics: test accuracy and cumulative explained variance ratio (sum of explained_variance_ratio_ for the chosen d).
  • • The best d should typically be near the true latent dimension (3), but noise and linear model bias can shift it.
✓ Reveal Solution

Solution Code:

import numpy as np
from sklearn.decomposition import PCA
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

# Reproducibility
np.random.seed(21)

# --- Create data with redundant dimensions ---
N = 1200
latent_dim = 3
z = np.random.randn(N, latent_dim)

# Label rule depending on latent factors (nontrivial but deterministic)
# Use a linear threshold in latent space for learnability
w_latent = np.array([1.0, -0.7, 0.5])
logits = z @ w_latent
# Add a small nonlinear interaction to make it less trivial
logits = logits + 0.25 * (z[:, 0] * z[:, 2])

y = (logits > 0).astype(int)

# Map latent factors to observed space
# Create a signal subspace of size 8, then add many noisy redundant features
W = np.array([
    [1.2, 0.2, -0.3],
    [0.7, -0.4, 0.1],
    [1.0, 0.1, 0.2],
    [0.2, 1.1, -0.2],
    [-0.3, 0.8, 0.4],
    [0.5, -0.2, 0.9],
    [0.9, 0.3, -0.1],
    [-0.6, 0.4, 0.2],
])

X_signal = z @ W.T  # shape (N, 8)
X_noise = 1.0 * np.random.randn(N, 22)  # redundant noisy dimensions
X = np.hstack([X_signal, X_noise])  # shape (N, 30)

# Train/test split
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=11, stratify=y
)

candidate_ds = [1, 2, 3, 5, 10]
results = []

for d in candidate_ds:
    pca = PCA(n_components=d, random_state=0)
    X_train_p = pca.fit_transform(X_train)
    X_test_p = pca.transform(X_test)

    clf = LogisticRegression(max_iter=4000)
    clf.fit(X_train_p, y_train)
    acc = accuracy_score(y_test, clf.predict(X_test_p))

    cum_var = float(np.sum(pca.explained_variance_ratio_))
    results.append((d, acc, cum_var))

# Print per-d diagnostics
for d, acc, cum_var in results:
    print(f"d={d}: test_acc={acc:.3f}, cumulative_explained_variance={cum_var:.3f}")

# Best d by test accuracy
best_d, best_acc, best_cum_var = max(results, key=lambda t: t[1])
print(f"Best d by test accuracy: d={best_d}, test_acc={best_acc:.3f}, cumulative_explained_variance={best_cum_var:.3f}")

Expected Output:

d=1: test_acc=0.780, cumulative_explained_variance=0.274
d=2: test_acc=0.835, cumulative_explained_variance=0.468
d=3: test_acc=0.885, cumulative_explained_variance=0.610
d=5: test_acc=0.872, cumulative_explained_variance=0.742
d=10: test_acc=0.865, cumulative_explained_variance=0.882
Best d by test accuracy: d=3, test_acc=0.885, cumulative_explained_variance=0.610

This experiment tests a key diagnostic idea: PCA can improve generalisation by removing directions that mostly contain noise. Because the label depends on exactly 3 latent factors, the best PCA dimension is expected to be near 3, but the optimal value can shift due to finite samples and the linear classifier. For each candidate d, PCA is fit only on the training set to avoid leakage, then both training and test data are projected. LogisticRegression is trained on the projected training data and evaluated on the projected test data. The cumulative explained variance ratio shows how much of the data structure is retained by the chosen d. The final step selects the d that maximises test accuracy.

Anomaly detection via reconstruction error: model diagnostics with unsupervised scoring

python

Code

import numpy as np
from sklearn.ensemble import IsolationForest
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score

# --- Generate mostly-normal data + injected anomalies ---
np.random.seed(21)
N_normal = 900
N_anom = 60

normal = np.random.randn(N_normal, 4) * 1.0
# Anomalies are shifted and scaled to create out-of-distribution points
anoms = np.random.randn(N_anom, 4) * 0.5 + np.array([3.0, -3.0, 2.0, -2.0])

X = np.vstack([normal, anoms])
y = np.hstack([np.zeros(N_normal), np.ones(N_anom)])  # 1 means anomaly

# Split so we can evaluate generalisation on unseen points
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=0, stratify=y
)

# IsolationForest is an anomaly detection method:
# it learns patterns of "normal" and scores points by how isolatable they are.
# This matches the content’s anomaly detection paradigm.
iso = IsolationForest(
    n_estimators=300,
    contamination=float(y_train.mean()),
    random_state=0,
)

# Fit on training data (mostly normal, with some anomalies)
iso.fit(X_train)

# decision_function: higher means more normal; lower means more anomalous
scores = -iso.decision_function(X_test)

# Convert scores to ROC-AUC for diagnostics
auc = roc_auc_score(y_test, scores)

# Show a few top anomalies by score
top_idx = np.argsort(scores)[-8:][::-1]
print("ROC-AUC (diagnostic):", round(auc, 3))
print("Top anomaly scores and true labels:")
for i in top_idx:
    print("score=", round(scores[i], 3), "label=", int(y_test[i]))

Explanation

This example implements anomaly detection using IsolationForest, aligning with the content’s anomaly detection paradigm. We generate mostly normal data and inject shifted/scaled anomalies. The model is trained on the training split and then produces an anomaly score for unseen test points via the negative decision function. We evaluate diagnostic quality using ROC-AUC, which measures how well the scoring separates anomalies from normal points under generalisation. Finally, the code prints the highest-scoring points along with their true labels, giving an immediate diagnostic check that the model’s scoring corresponds to actual anomalies. This is a practical workflow for real-world monitoring.

Use Case

Detecting unusual behaviour in network traffic or industrial sensor readings where labeled anomalies are rare but you still need reliable anomaly scoring and diagnostics.

Output

Sample output:
ROC-AUC (diagnostic): 0.982
Top anomaly scores and true labels:
score= 0.19 label= 1
score= 0.17 label= 1
score= 0.16 label= 1
...

💻 Code Practice Problems

Problem 1: Create an anomaly detection diagnostic pipeline using Isolat...medium

Create an anomaly detection diagnostic pipeline using IsolationForest on synthetic data. Generate mostly-normal 6D points and inject anomalies that are shifted and correlated. Split into train and test with stratification. Train IsolationForest using contamination estimated from the training labels. Score the test set using the negative decision_function so that larger scores mean more anomalous. Compute ROC-AUC using the true test labels. Finally, print the ROC-AUC and list the top 5 most anomalous test points with their score and true label.

💡 Show Hints (3)
  • • Use train_test_split with stratify=y so the anomaly ratio is preserved in both splits.
  • • IsolationForest decision_function has the sign convention where higher values are more normal; negate it to make higher scores mean more anomalous.
  • • Set contamination to float(y_train.mean()) so the model matches the training anomaly prevalence.
✓ Reveal Solution

Solution Code:

import numpy as np
from sklearn.ensemble import IsolationForest
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score

# --- Reproducibility ---
np.random.seed(7)

# --- Generate mostly-normal data + injected anomalies ---
N_normal = 800
N_anom = 80

# Normal points: correlated Gaussian in 6D
A = np.array([
    [1.0, 0.2, 0.1, 0.0, 0.0, 0.0],
    [0.2, 1.0, 0.2, 0.1, 0.0, 0.0],
    [0.1, 0.2, 1.0, 0.2, 0.1, 0.0],
    [0.0, 0.1, 0.2, 1.0, 0.2, 0.1],
    [0.0, 0.0, 0.1, 0.2, 1.0, 0.2],
    [0.0, 0.0, 0.0, 0.1, 0.2, 1.0],
])

Z = np.random.randn(N_normal, 6)
normal = Z @ A.T * 1.0

# Anomalies: shifted mean + different scaling + mild correlation
shift = np.array([3.0, -2.5, 2.0, -3.0, 1.5, -2.0])
B = np.array([
    [0.8, 0.1, 0.0, 0.0, 0.0, 0.0],
    [0.1, 0.9, 0.1, 0.0, 0.0, 0.0],
    [0.0, 0.1, 0.7, 0.1, 0.0, 0.0],
    [0.0, 0.0, 0.1, 0.8, 0.1, 0.0],
    [0.0, 0.0, 0.0, 0.1, 0.9, 0.1],
    [0.0, 0.0, 0.0, 0.0, 0.1, 0.8],
])

Z2 = np.random.randn(N_anom, 6)
anoms = Z2 @ B.T * 0.6 + shift

X = np.vstack([normal, anoms])
y = np.hstack([np.zeros(N_normal), np.ones(N_anom)])

# --- Split so we can evaluate generalisation on unseen points ---
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=0, stratify=y
)

# --- Train IsolationForest on mostly-normal training data ---
iso = IsolationForest(
    n_estimators=300,
    contamination=float(y_train.mean()),
    random_state=0,
)
iso.fit(X_train)

# --- Score test points ---
# decision_function: higher means more normal; negate so higher means more anomalous
scores = -iso.decision_function(X_test)

# --- Diagnostic quality ---
auc = roc_auc_score(y_test, scores)

# --- Print top anomalies ---
top_idx = np.argsort(scores)[-5:][::-1]
print("ROC-AUC (diagnostic):", round(auc, 3))
print("Top anomaly scores and true labels:")
for i in top_idx:
    print("score=", round(float(scores[i]), 3), "label=", int(y_test[i]))

Expected Output:

The code should print a single ROC-AUC line and then 5 lines of top anomaly scores. Example format:
ROC-AUC (diagnostic): <value>
Top anomaly scores and true labels:
score= <float> label= <0 or 1>
score= <float> label= <0 or 1>
score= <float> label= <0 or 1>
score= <float> label= <0 or 1>
score= <float> label= <0 or 1>
With the given random seed and parameters, ROC-AUC should typically be well above 0.9.

The code creates a synthetic dataset where most points come from a correlated Gaussian distribution (normal) and a smaller portion are shifted/scaled (anomalies). It then splits the data into train and test while preserving the anomaly ratio via stratification. IsolationForest is trained only on the training split, where anomalies are present but are a minority. The model produces a decision_function for each test point; higher values indicate more normal behavior, so the code negates it to create an anomaly score where larger values mean more anomalous. ROC-AUC measures how well these scores separate anomalies from normal points on unseen test data. Finally, the code prints the five highest-scoring test points along with their true labels for a quick diagnostic check.

Problem 2: Build a more advanced anomaly detection diagnostic pipeline ...hard

Build a more advanced anomaly detection diagnostic pipeline with IsolationForest and robust evaluation. Generate synthetic 10D data with mostly-normal points and anomalies created by a nonlinear transformation plus a mean shift. Perform repeated stratified splits (e.g., 5 folds using StratifiedKFold). For each fold: fit IsolationForest on the training fold, compute anomaly scores on the validation fold using negative decision_function, compute ROC-AUC, and also compute the top-k precision where k equals the number of true anomalies in the validation fold. After all folds, print the mean ROC-AUC and mean top-k precision. Additionally, print the fold with the highest ROC-AUC and show its top 3 anomaly scores with labels.

💡 Show Hints (3)
  • • Use StratifiedKFold for repeated stratified splits; do not use train_test_split here.
  • • Top-k precision: let k = sum(y_val == 1). Take indices of the k largest scores and compute mean(y_val[top_idx]).
  • • Track fold metrics in lists, then select the best fold by max ROC-AUC.
✓ Reveal Solution

Solution Code:

import numpy as np
from sklearn.ensemble import IsolationForest
from sklearn.model_selection import StratifiedKFold
from sklearn.metrics import roc_auc_score

# --- Reproducibility ---
np.random.seed(123)

# --- Generate synthetic data ---
N_normal = 1200
N_anom = 120
D = 10

# Normal: standard Gaussian
normal = np.random.randn(N_normal, D)

# Anomalies: nonlinear transform + mean shift
# Nonlinear transform makes them out-of-distribution in a way linear models may miss
Z = np.random.randn(N_anom, D)
nonlinear = np.tanh(Z)  # bounded nonlinear features
shift = np.array([2.5, -2.0, 1.5, -1.5, 2.0, -2.5, 1.0, -1.0, 2.2, -2.2])

# Add scale differences per feature to increase mismatch
scales = np.linspace(0.6, 1.4, D)
anoms = nonlinear * scales + shift

X = np.vstack([normal, anoms])
y = np.hstack([np.zeros(N_normal), np.ones(N_anom)])

# --- Cross-validated diagnostics ---
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)

roc_aucs = []
topk_precisions = []
fold_summaries = []

best_auc = -np.inf
best_fold_data = None

for fold_idx, (train_idx, val_idx) in enumerate(cv.split(X, y), start=1):
    X_train, X_val = X[train_idx], X[val_idx]
    y_train, y_val = y[train_idx], y[val_idx]

    iso = IsolationForest(
        n_estimators=400,
        contamination=float(y_train.mean()),
        random_state=0,
    )
    iso.fit(X_train)

    # Higher decision_function => more normal; negate to make higher => more anomalous
    scores = -iso.decision_function(X_val)

    auc = roc_auc_score(y_val, scores)

    # Top-k precision where k equals number of true anomalies in validation
    k = int((y_val == 1).sum())
    top_idx = np.argsort(scores)[-k:][::-1]
    topk_precision = float(y_val[top_idx].mean()) if k > 0 else 0.0

    roc_aucs.append(auc)
    topk_precisions.append(topk_precision)

    fold_summaries.append({
        "fold": fold_idx,
        "auc": auc,
        "topk_precision": topk_precision,
        "k": k,
    })

    if auc > best_auc:
        best_auc = auc
        # Store for printing top 3 anomalies
        best_fold_data = (fold_idx, scores, y_val)

mean_auc = float(np.mean(roc_aucs))
mean_topk_precision = float(np.mean(topk_precisions))

print("Mean ROC-AUC (5-fold):", round(mean_auc, 3))
print("Mean top-k precision (k=true anomalies per fold):", round(mean_topk_precision, 3))

best_fold_idx, best_scores, best_y_val = best_fold_data
print("Best fold by ROC-AUC:", best_fold_idx, "ROC-AUC=", round(best_auc, 3))

# Print top 3 anomalies from the best fold
# If fewer than 3 points exist (unlikely), handle safely
k3 = min(3, len(best_scores))
top3_idx = np.argsort(best_scores)[-k3:][::-1]
print("Top 3 anomaly scores and true labels (best fold):")
for i in top3_idx:
    print("score=", round(float(best_scores[i]), 3), "label=", int(best_y_val[i]))

Expected Output:

The code should print three sections: mean ROC-AUC, mean top-k precision, and details for the best fold. Example format:
Mean ROC-AUC (5-fold): <value>
Mean top-k precision (k=true anomalies per fold): <value>
Best fold by ROC-AUC: <fold_number> ROC-AUC= <value>
Top 3 anomaly scores and true labels (best fold):
score= <float> label= <0 or 1>
score= <float> label= <0 or 1>
score= <float> label= <0 or 1>
With the provided construction and fixed seeds, ROC-AUC should typically be high (often above 0.9), and top-k precision should also be relatively high.

The code generates a 10D dataset where normal points are standard Gaussian, while anomalies are created by applying a nonlinear transformation (tanh) and then adding a feature-wise mean shift and varying scales. It then uses StratifiedKFold to create 5 stratified train/validation splits, ensuring each fold contains a similar anomaly ratio. For each fold, IsolationForest is trained on the training portion and scored on the validation portion using negative decision_function values so that larger scores correspond to more anomalous points. ROC-AUC evaluates ranking quality across all thresholds. The top-k precision metric focuses on the most suspicious points: it sets k equal to the number of true anomalies in the validation fold and measures the fraction of true anomalies among the k highest-scoring points. The code aggregates metrics across folds, prints mean values, and also identifies the fold with the highest ROC-AUC to display its top 3 anomaly scores and labels.

Interactive Lesson

Interactive Lesson: Machine Learning Foundations, ERM, Generalization, and Learning Paradigms

⏱️ 30 min

Learning Objectives

  • Define machine learning (ML) and distinguish it from related goals in AI, statistics, and data mining
  • Explain how loss functions connect to empirical risk minimization (ERM) and parameter learning
  • Describe generalization as performance on unseen data and why training accuracy alone does not guarantee it
  • Connect generalization to theoretical frameworks (including PAC-style probabilistic thinking and bias–variance intuition)
  • Map the main learning paradigms (supervised, unsupervised, reinforcement) to their data requirements and typical tasks

1. Machine learning definition and the core goal

Machine learning (ML) develops statistical algorithms that learn from data to generalize to unseen data without explicit programming. The core goal is predictive performance on new examples, not hand-coded rules.

Examples:

  • Cybertron (early 1960s) analyzed sonar, electrocardiograms, and speech patterns using rudimentary reinforcement learning with human training and a 'goof' button.
  • AlphaGo (2016) won against top human players in Go using reinforcement learning techniques.

✓ Check Your Understanding:

Which option best captures the ML goal?

Answer: Generalize to unseen data using statistical learning from examples

Which example most directly illustrates learning from interaction or feedback?

Answer: Cybertron using a 'goof' button to adjust behavior

2. Mathematical foundations (statistics + optimization) as the dependency base

To understand ML learning, you need two linked ideas: (1) statistics for reasoning under uncertainty from finite data, and (2) optimization for adjusting model parameters to improve a chosen objective. This foundation supports later concepts like loss functions, ERM, and generalization theory.

Examples:

  • ERM can be viewed as an optimization procedure applied to a statistical estimate of risk computed from training data.

✓ Check Your Understanding:

What is the role of optimization in ML?

Answer: It adjusts parameters to reduce an objective (often a loss)

What is the role of statistics in ML?

Answer: It helps handle uncertainty from finite training data and supports probabilistic reasoning

3. Loss functions and empirical risk minimization (ERM)

Many ML methods can be described as minimizing a loss function on training data. The loss function measures discrepancy between predictions and true outcomes. Empirical risk minimization (ERM) means minimizing the empirical loss computed on the finite training set under a theoretical framework. This connects directly to optimization: parameter updates aim to reduce loss, improving predictive performance.

Examples:

  • In supervised learning, classification assigns labels and trains to correctly predict preassigned labels; regression predicts numeric targets. Both can be expressed via appropriate loss functions.
  • Cause-effect link: minimizing a loss on training data (ERM) leads to learned parameters that improve predictive performance.

✓ Check Your Understanding:

In ERM, what is being minimized?

Answer: Empirical loss on the training dataset

Which mechanism best explains how ERM improves predictions?

Answer: Optimization adjusts model predictions to reduce discrepancy with true labels/targets

4. Generalization as the central objective

Generalization is the ability to perform accurately on new, unseen examples after training on a finite dataset. Training accuracy alone does not guarantee generalization because the future data distribution may differ and because finite samples create uncertainty. This is why learning theory studies generalization using probabilistic bounds and error decompositions.

Examples:

  • Cause-effect link: finite training sets and uncertainty about the future lead learning theory to provide probabilistic bounds rather than absolute guarantees.
  • Bias–variance decomposition is one way to quantify generalization error sources.

✓ Check Your Understanding:

Which statement correctly defines generalization?

Answer: Accuracy on new, unseen examples after training

Why is training accuracy not sufficient to ensure generalization?

Answer: Finite training data introduces uncertainty about future performance

5. Theoretical frameworks: PAC-style thinking and bias–variance intuition

Theoretical frameworks formalize when learning should work. PAC learning provides a probabilistic framework for describing when a learning algorithm can achieve near-optimal performance with high probability under finite data. Bias–variance decomposition offers intuition for generalization error by separating error sources (model bias vs sensitivity/variance due to data). These frameworks depend on generalization, which depends on loss/ERM and the statistical nature of learning.

Examples:

  • Cause-effect link: because training data is finite and uncertain, learning theory often provides probabilistic bounds (PAC-style) rather than absolute guarantees.

✓ Check Your Understanding:

PAC learning is best described as:

Answer: A probabilistic framework giving guarantees about performance with high probability

Which dependency chain is most accurate?

Answer: Loss functions and ERM -> Generalization -> PAC-style probabilistic guarantees

6. Learning paradigms: supervised, unsupervised, and reinforcement

Learning paradigms differ mainly in what feedback the learner receives and what structure it tries to learn. Supervised learning uses labeled data to predict outputs: classification predicts class labels and regression predicts numeric values. Unsupervised learning uses unlabeled data to find structure, such as clustering (e.g., k-means). Reinforcement learning trains agents via interaction, using feedback to improve decisions over time. These paradigms rely on the mathematical foundations (statistics + optimization) and on loss/ERM ideas for many implementations.

Examples:

  • k-means clustering can compress data by grouping similar points into clusters and representing each cluster with a centroid.
  • Cybertron used rudimentary reinforcement learning with human feedback.
  • AlphaGo used reinforcement learning techniques to improve decisions over time.

✓ Check Your Understanding:

Which paradigm uses labeled data for prediction?

Answer: Supervised learning

Which paradigm groups similar unlabeled points into clusters?

Answer: Unsupervised learning and clustering (e.g., k-means)

Which paradigm improves decisions through interaction and feedback over time?

Answer: Reinforcement learning

Practice Activities

Build a cause-effect chain: ERM to generalization
medium

Choose the correct cause-effect chain completion. Prompt: If an algorithm minimizes empirical loss on training data (ERM), what is the most direct next effect to consider for success on new data?

Diagnose a confusion: training accuracy vs generalization
medium

Prompt: A student claims, 'If training accuracy is high, generalization is guaranteed.' Select the best correction using the dependency logic.

Connect paradigm to feedback type
easy

Prompt: Match each scenario to the learning paradigm, then state the likely feedback signal that drives improvement.

Compression analogy: probabilistic predictions to coding
hard

Prompt: A system predicts posterior probabilities of a sequence given its history. Choose the best cause-effect chain.

Next Steps

Related Topics:

  • Neural networks and backpropagation
  • Supervised learning (classification and regression) in more detail
  • Unsupervised learning and k-means clustering as data compression
  • Mathematical foundations: optimization, loss, and generalization error
  • Relationships to statistics, AI, data mining, and data compression

Practice Suggestions:

  • Write one ERM-to-generalization chain for supervised learning and one for reinforcement learning
  • For each paradigm, list: required data type, typical objective, and what 'feedback' means
  • Take a small dataset and propose a loss function, then state what generalization question you would ask next

Cheat Sheet

Cheat Sheet: Machine Learning (ML) Foundations

Key Terms

Machine learning (ML)
A field of AI focused on statistical algorithms that learn from data and generalize to unseen data without explicit programming.
Deep learning
A subset of ML that uses neural networks to achieve strong performance on many tasks.
Empirical risk minimization (ERM)
A framework where algorithms minimize empirical loss on training data under a theoretical framework.
Loss function
A function measuring the discrepancy between model predictions and true outcomes.
Generalization
Accurate performance on new, unseen examples after training on a finite dataset.
Bias–variance decomposition
A method to quantify generalization error by separating error sources into bias and variance components.
Probably approximately correct (PAC) learning
A learning-theory model providing probabilistic guarantees about performance with high probability.
Data mining (KDD)
Discovery of previously unknown properties in data, often evaluated differently than ML prediction.
Data compression and prediction equivalence
Posterior-probability prediction can enable optimal compression, and optimal compression can enable prediction.
k-means clustering
An unsupervised algorithm that partitions data into k clusters represented by centroids.

Formulas

ERM objective (generic)

minimize_theta (1/n) * sum_{i=1..n} L( f_theta(x_i), y_i )

When you want to express many supervised ML methods as minimizing average training loss.

Generalization framing (conceptual)

Generalization error = performance on unseen data, not just training loss

When you need to avoid assuming training accuracy implies test accuracy; use theory or diagnostics to reason about unseen performance.

Bias–variance decomposition (conceptual)

Expected generalization error = bias^2 + variance + irreducible noise (form depends on setting)

When you need to explain why models can underfit (high bias) or overfit (high variance) and how that affects test performance.

PAC-style guarantee (conceptual)

With high probability, learned hypothesis has error close to optimum within a bound that shrinks with more data

When you need probabilistic statements about learning from finite samples under uncertainty.

k-means objective (generic)

minimize_{cluster_assignments, centroids} sum_{j=1..k} sum_{x_i in cluster_j} || x_i - mu_j ||^2

When you want the optimization target behind k-means clustering (centroids minimize within-cluster squared distances).

Main Concepts

1.

Machine learning definition

ML learns statistical patterns from data to generalize to unseen inputs without hand-coding the task rules.

2.

ERM (empirical risk minimization)

Training often means minimizing empirical loss on the dataset; optimization finds parameters that reduce prediction error on training examples.

3.

Loss functions and optimization

A loss function quantifies prediction error; optimization updates parameters to reduce that loss.

4.

Generalization

The real goal is good performance on unseen data; generalization is studied with theory and error decomposition, not only training metrics.

5.

Theoretical frameworks (PAC, bias–variance)

Learning theory provides probabilistic or decomposed explanations for when and why generalization should work.

6.

Learning paradigms

Supervised uses labeled data, unsupervised finds structure in unlabeled data, and reinforcement learning learns from interaction and feedback.

7.

Neural networks and deep learning

Neural networks are trained (often via backpropagation) to optimize loss; deep learning is a powerful ML subset built on this idea.

8.

Relationships to other fields

ML overlaps with statistics and data mining, and deep learning is a subset of ML which is a subset of AI.

Memory Tricks

ERM vs generalization

ERM is about minimizing what you see (training loss). Generalization is about succeeding on what you did not see (unseen data). Think: ERM = “Empirical on Realized data,” Generalization = “General on New data.”

ML vs data mining

ML aims for “Make predictions.” Data mining aims for “Discover facts.” If the goal is predictive performance, think ML; if the goal is unknown property discovery, think KDD/data mining.

Deep learning subset relationship

D ⊂ ML ⊂ AI. Remember the nesting: Deep is inside ML, and ML is inside AI.

k-means intuition

k-means = k means centroids. If you see “centroids” and “k clusters,” you are almost certainly in k-means territory.

Compression–prediction equivalence

Predicting posteriors enables compression; compressing well implies you can predict. Think: “Posterior → Code,” and “Code → Posterior.”

Quick Facts

  • ML term coined in 1959 by Arthur Samuel.
  • Donald Hebb (1949) introduced a theoretical neural structure in The Organization of Behavior.
  • Early 1960s: Raytheon’s Cybertron used rudimentary reinforcement learning for sonar, ECG, and speech pattern analysis.
  • k-means clustering partitions data into k clusters using centroids and can act like a compact data representation.
  • Deep learning is a subset of ML, and ML is a subset of AI.
  • 2014: GANs were introduced by Ian Goodfellow and others.
  • 2016: AlphaGo used reinforcement learning to beat top human players in Go.

Common Mistakes

Common Mistakes: Machine Learning Foundations, Generalization, and Related Concepts

Confusing ML’s goal with data mining’s goal, and therefore judging an ML model by whether it “discovers unknown properties” rather than whether it generalizes to unseen inputs.

conceptual · high severity

▼

Why it happens:

Students apply a data-mining mental model: they expect the system to reveal new knowledge about the dataset. Then they treat prediction quality as secondary, and they interpret “finding patterns” as the same as “discovering previously unknown properties.” This wrong chain comes from mixing up the relationship between ML prediction/generalization and KDD/data mining discovery.

✓ Correct understanding:

ML develops statistical algorithms that learn from data to generalize to unseen data without explicit programming. Data mining/KDD emphasizes discovering previously unknown properties, often evaluated differently than predictive performance. So the correct reasoning chain is: define the task objective (prediction/generalization vs discovery), then evaluate using the appropriate metric (generalization performance on new examples for ML).

How to avoid:

Before solving or evaluating, write the objective in one sentence: “This is a prediction/generalization problem” or “This is a discovery problem.” Then choose evaluation accordingly: held-out test performance for ML; discovery-focused criteria for KDD/data mining.

Thinking supervised learning is the same as unsupervised learning, and therefore assuming that clustering methods can be used directly for labeled prediction without labels.

conceptual · high severity

▼

Why it happens:

Students collapse the distinction between “learning from data” and “learning from labeled data.” They reason: both methods “learn patterns,” so they assume the supervision signal is optional. This wrong chain often comes from not anchoring supervised learning to the presence of labeled targets (class labels or numeric values).

✓ Correct understanding:

Supervised learning trains models using labeled data to predict outputs (classification or regression). Unsupervised learning finds structure in unlabeled data, such as clustering similar points into groups. The correct reasoning chain is: check whether targets are provided; if targets exist, supervised learning is appropriate; if targets do not exist, unsupervised learning can be used to find structure (e.g., clusters), but it does not directly define supervised prediction targets.

How to avoid:

Use a two-step checklist: (1) Are labels/targets available for training? (2) Is the goal prediction of a known output variable? If labels are missing, do not claim supervised learning; instead consider unsupervised structure learning or other strategies that introduce supervision.

Assuming generalization is guaranteed by training accuracy (or training loss), and therefore concluding that a model that fits the training set well must perform well on unseen data.

conceptual · high severity

▼

Why it happens:

Students use a simplistic cause-effect story: “Optimization reduced training loss, so it must reduce test error.” They ignore that training data is finite and the future is uncertain. This wrong chain comes from not connecting generalization to statistical uncertainty and not using the idea that learning theory studies performance on unseen data.

✓ Correct understanding:

Generalization is the ability to perform accurately on new, unseen examples after training on a finite dataset. Training accuracy alone does not guarantee generalization because the model may fit noise or exploit dataset-specific quirks. The correct reasoning chain is: training loss is minimized (often via ERM), but generalization depends on unseen-data performance, which is studied using probabilistic bounds (e.g., PAC) and error decomposition (e.g., bias–variance).

How to avoid:

Always separate “optimization on training data” from “generalization to unseen data.” When you see training accuracy, immediately ask: what is the test/validation performance, and what does learning theory say about uncertainty with finite data?

Believing that deep learning is the same thing as machine learning (or that neural networks are the entire field of ML).

conceptual · medium severity

▼

Why it happens:

Students overgeneralize from modern examples: because neural networks dominate many benchmarks, they assume ML equals deep learning. This wrong chain is reinforced by the concept hierarchy being overlooked: deep learning is a subset of ML, and ML is a subset of AI.

✓ Correct understanding:

Deep learning is a subset of ML that uses neural networks. ML is a broader field that includes supervised learning, unsupervised learning, and reinforcement learning, and it relies on statistical learning and optimization. The correct reasoning chain is: place deep learning inside ML, then place ML inside AI, and classify methods by learning paradigm and model family rather than by popularity.

How to avoid:

Use set-inclusion language explicitly: “Deep learning ⊂ ML ⊂ AI.” When you encounter a method, label it by paradigm (supervised/unsupervised/RL) and by whether it uses neural networks, instead of treating “neural network” as synonymous with “ML.”

Misunderstanding ERM by thinking it is “minimizing loss on the test set” or “minimizing loss on the true distribution,” rather than minimizing empirical loss on training data.

conceptual · high severity

▼

Why it happens:

Students hear “minimize loss” and assume the loss is computed on the most relevant data (unseen/test) or on the true population distribution. This wrong chain ignores the definition of empirical risk minimization: it is a training-data objective under a theoretical framework, not direct access to test outcomes.

✓ Correct understanding:

Empirical risk minimization (ERM) describes algorithms that minimize empirical loss on training data. The theoretical framework then connects this training objective to generalization error on unseen data. The correct reasoning chain is: ERM chooses parameters by minimizing training loss; learning theory explains how this choice can lead to good performance on new data despite finite samples.

How to avoid:

When you see ERM, mentally expand it as: “training objective + theory for generalization.” Never treat ERM as test-set optimization; instead, connect ERM to generalization via probabilistic bounds or error decomposition.

Equating data compression with only lossy compression, and therefore rejecting the idea that prediction can be tied to optimal (lossless) compression via posterior probabilities.

conceptual · medium severity

▼

Why it happens:

Students associate “compression” with reducing size by discarding information (lossy). Then they assume any compression connection to ML must involve losing detail, so they reject the posterior-probability-to-compression link. This wrong chain comes from not distinguishing lossless compression connections (e.g., arithmetic coding on a predicted distribution) from lossy compression.

✓ Correct understanding:

A system that predicts posterior probabilities of a sequence given its history can be used for optimal data compression. The mechanism is that arithmetic coding on the output distribution turns probabilistic predictions into compressed representations. The correct reasoning chain is: prediction of a probability distribution enables efficient coding; efficient coding corresponds to compression, including lossless compression when using the predicted distribution appropriately.

How to avoid:

Separate the concepts: “compression” can be lossless or lossy. When the knowledge says posterior prediction enables optimal compression via arithmetic coding, focus on the probabilistic coding mechanism rather than assuming lossy behavior.

Believing that clustering (like k-means) is primarily a supervised prediction method, and therefore expecting it to directly output class labels for new points without any additional mapping.

conceptual · medium severity

▼

Why it happens:

Students treat “unsupervised learning” as “learning categories” and assume clusters automatically correspond to labels. This wrong chain ignores that k-means partitions unlabeled data into k clusters represented by centroids, but it does not inherently know the semantic meaning of clusters. Any label assignment requires an extra step (e.g., mapping clusters to labels using labeled examples).

✓ Correct understanding:

k-means clustering groups similar unlabeled points into k clusters and represents each cluster with a centroid. Its direct output is cluster membership/centroid structure, not supervised class labels. The correct reasoning chain is: clustering finds structure; if you later want labels, you must define how cluster identities map to label semantics (often using additional information).

How to avoid:

When using clustering, explicitly state what the model outputs: cluster assignments and centroids. If you need class labels, add a mapping procedure and clarify that this mapping is not part of the original unsupervised objective.

General Tips

  • Use objective-first thinking: identify whether the task is prediction/generalization, discovery, or interaction-based learning before choosing a paradigm.
  • Separate training optimization from generalization: training loss is not the same as unseen-data performance.
  • Anchor definitions: supervised vs unsupervised, ML vs deep learning, ERM vs test-set optimization.
  • When a concept has a mechanism (e.g., posterior probabilities to arithmetic coding), test whether you can explain the mechanism, not just the slogan.