Skip to content

About

Complete implementation of Support Vector Machines from scratch including Hard Margin, Soft Margin, Kernel SVM, and Multi-Class SVM with detailed explanations and visualizations

Resources

Stars

1 star

Watchers

0 watching

Forks

Latest commit

 

History

1 Commit

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 

Repository files navigation

Support Vector Machines (SVM) - Deep Dive Explanation

Table of Contents

  1. Introduction & Intuition
  2. Hard Margin SVM
  3. Soft Margin SVM
  4. Kernel SVM
  5. Multi-Class SVM
  6. Step-by-Step Flow Examples
  7. Internal Changes During Training

Introduction & Intuition

What is SVM?

Support Vector Machine is a supervised learning algorithm that finds the optimal hyperplane to separate different classes in your data. Think of it as finding the best "decision boundary" that maximizes the margin between classes.

The Core Idea

Imagine you have two groups of points (red and blue) on a 2D plane. You could draw many lines to separate them, but which line is best?

Dataset

SVM says: The best line is the one that has the maximum distance (margin) from the nearest points of both classes.

Key Terms

  • Hyperplane: The decision boundary (a line in 2D, a plane in 3D, etc.)
  • Support Vectors: The data points closest to the hyperplane that "support" or define it
  • Margin: The distance between the hyperplane and the nearest data points
  • Weight Vector (w): Defines the orientation of the hyperplane
  • Bias (b): Shifts the hyperplane position

Hard Margin SVM

Mathematical Foundation

The hyperplane equation is: $$w \cdot x - b = 0$$

Where:

  • $w$ = weight vector (normal to the hyperplane)
  • $x$ = data point
  • $b$ = bias term

For a point to be correctly classified: $$y_i(w \cdot x_i - b) \geq 1$$

Where $y_i \in {-1, 1}$ is the class label.

The Optimization Problem

Goal: Maximize the margin = $\frac{2}{||w||}$

This is equivalent to: Minimize $\frac{1}{2}||w||^2$

Subject to: $y_i(w \cdot x_i - b) \geq 1$ for all training points

Implementation Breakdown

class HardMarginSVM:
    def __init__(self, learning_rate=0.001, lambda_param=0.01, n_iterations=1000):
        self.lr = learning_rate           # How fast we update weights
        self.lambda_param = lambda_param   # Regularization strength
        self.n_iterations = n_iterations   # Number of training iterations
        self.w = None                      # Weight vector (to be learned)
        self.b = None                      # Bias term (to be learned)

The Training Process (fit method)

def fit(self, X, y):
    n_samples, n_features = X.shape
    y_ = np.where(y <= 0, -1, 1)  # Convert labels to -1 and 1

    self.w = np.zeros(n_features)  # Start with zero weights
    self.b = 0                      # Start with zero bias

    for _ in range(self.n_iterations):
        for idx, x_i in enumerate(X):
            # Check if point is correctly classified with margin >= 1
            condition = y_[idx] * (np.dot(x_i, self.w) - self.b) >= 1

            if condition:
                # Point is on correct side with sufficient margin
                # Only update weights to keep them small (regularization)
                self.w -= self.lr * (2 * self.lambda_param * self.w)
            else:
                # Point violates margin constraint
                # Update both weights and bias
                self.w -= self.lr * (2 * self.lambda_param * self.w - np.dot(x_i, y_[idx]))
                self.b -= self.lr * y_[idx]

What Happens Internally: Step-by-Step Example

Let's trace through one iteration with a concrete example:

Initial State:

  • $w = [0.5, 0.3]$
  • $b = 0.1$
  • Learning rate = 0.001
  • Lambda = 0.01

Training Point:

  • $x_i = [2.0, 3.0]$
  • $y_i = 1$ (positive class)

Step 1: Calculate the decision value $$w \cdot x_i - b = [0.5, 0.3] \cdot [2.0, 3.0] - 0.1$$ $$= 0.5 \times 2.0 + 0.3 \times 3.0 - 0.1 = 1.0 + 0.9 - 0.1 = 1.8$$

Step 2: Check the margin condition $$y_i \times (w \cdot x_i - b) = 1 \times 1.8 = 1.8$$

Since $1.8 \geq 1$, the point is correctly classified with good margin.

Step 3: Update weights (regularization only) $$w_{new} = w - lr \times (2 \times \lambda \times w)$$ $$w_{new} = [0.5, 0.3] - 0.001 \times (2 \times 0.01 \times [0.5, 0.3])$$ $$w_{new} = [0.5, 0.3] - 0.001 \times [0.01, 0.006]$$ $$w_{new} = [0.5, 0.3] - [0.00001, 0.000006]$$ $$w_{new} = [0.49999, 0.299994]$$

If the point violated the margin:

Say $x_i = [-1.0, 0.5]$, $y_i = -1$

$$w \cdot x_i - b = [0.5, 0.3] \cdot [-1.0, 0.5] - 0.1 = -0.5 + 0.15 - 0.1 = -0.45$$ $$y_i \times (w \cdot x_i - b) = -1 \times (-0.45) = 0.45 &lt; 1$$ ❌

Update both weights and bias: $$w_{new} = w - lr \times (2 \times \lambda \times w - x_i \times y_i)$$ $$w_{new} = [0.5, 0.3] - 0.001 \times (2 \times 0.01 \times [0.5, 0.3] - [-1.0, 0.5] \times (-1))$$ $$w_{new} = [0.5, 0.3] - 0.001 \times ([0.01, 0.006] - [1.0, -0.5])$$ $$w_{new} = [0.5, 0.3] - 0.001 \times [-0.99, 0.506]$$ $$w_{new} = [0.5, 0.3] + [0.00099, -0.000506]$$ $$w_{new} = [0.50099, 0.299494]$$

$$b_{new} = b - lr \times y_i = 0.1 - 0.001 \times (-1) = 0.101$$

Hard Margin Result

The solid black line is the decision boundary, and the dashed lines show the margin.


Soft Margin SVM

Why Soft Margin?

Hard margin SVM assumes data is perfectly linearly separable. In reality, data often has:

  • Noise
  • Outliers
  • Overlapping classes

Soft margin SVM allows some points to be inside the margin or even misclassified, but penalizes them.

The Mathematics

We introduce slack variables $\xi_i \geq 0$:

Minimize: $\frac{1}{2}||w||^2 + C\sum_{i=1}^{n}\xi_i$

Subject to: $y_i(w \cdot x_i - b) \geq 1 - \xi_i$

Where:

  • $C$ is the penalty parameter (how much we care about misclassifications)
  • $\xi_i$ measures how much point $i$ violates the margin

The Loss Function

We use the Hinge Loss: $$L_{hinge} = \max(0, 1 - y_i(w \cdot x_i - b))$$

Total Loss: $L = \lambda||w||^2 + \frac{1}{n}\sum_{i=1}^{n}\max(0, 1 - y_i(w \cdot x_i - b))$

Note: $\lambda = \frac{1}{2C}$ (inverse relationship)

Implementation Breakdown

class SoftMarginSVM:
    def __init__(self, learning_rate=0.001, lambda_param=0.01, n_iterations=1000):
        self.lr = learning_rate
        self.lambda_param = lambda_param
        self.n_iterations = n_iterations
        self.w = None
        self.b = None
        self.losses = []  # Track loss over time

    def fit(self, X, y):
        n_samples, n_features = X.shape
        y_ = np.where(y <= 0, -1, 1)

        self.w = np.zeros(n_features)
        self.b = 0

        for iteration in range(self.n_iterations):
            epoch_loss = 0

            for idx, x_i in enumerate(X):
                condition = y_[idx] * (np.dot(x_i, self.w) - self.b) >= 1

                if condition:
                    # No hinge loss, only regularization
                    self.w -= self.lr * (2 * self.lambda_param * self.w)
                    loss = self.lambda_param * np.dot(self.w, self.w)
                else:
                    # Has hinge loss, update both w and b
                    self.w -= self.lr * (2 * self.lambda_param * self.w - np.dot(x_i, y_[idx]))
                    self.b -= self.lr * y_[idx]
                    loss = self.lambda_param * np.dot(self.w, self.w) + \
                           max(0, 1 - y_[idx] * (np.dot(x_i, self.w) - self.b))

                epoch_loss += loss

            self.losses.append(epoch_loss / n_samples)

Internal Changes Example

Scenario: Training on noisy data with outliers

Epoch 0 (Initial):

  • $w = [0, 0]$
  • $b = 0$
  • Total Loss = High (many misclassifications)

Point 1: $x = [2.5, 1.8]$, $y = 1$ (correctly on positive side) $$w \cdot x - b = 0 - 0 = 0$$ $$y(w \cdot x - b) = 1 \times 0 = 0 &lt; 1$$ ❌

  • Hinge Loss = $\max(0, 1 - 0) = 1$
  • Update: $w$ increases toward $x$ direction

Point 2: $x = [-1.2, -0.8]$, $y = -1$ (correctly on negative side) $$w \cdot x - b = [0.002, 0.001] \cdot [-1.2, -0.8] - 0.001 = -0.0034$$ $$y(w \cdot x - b) = -1 \times (-0.0034) = 0.0034 &lt; 1$$ ❌

  • Hinge Loss = $\max(0, 1 - 0.0034) = 0.9966$
  • Update: $w$ adjusts, $b$ adjusts

Outlier Point: $x = [0.5, 0.5]$, $y = -1$ (but close to positive region)

  • This point will consistently have high hinge loss
  • But the algorithm tolerates it (doesn't try too hard to fix)
  • The loss is bounded by the C parameter

After 1000 iterations:

  • $w$ converges to optimal direction
  • $b$ converges to optimal offset
  • Loss decreases steadily

Soft Margin Result Loss Over Time

Notice how the decision boundary is more robust to outliers compared to hard margin.


Kernel SVM

The Problem with Linear Boundaries

Some datasets are not linearly separable in their original space:

Non-Linear Data

No straight line can separate these concentric circles!

The Kernel Trick

Key Insight: Transform data to a higher-dimensional space where it becomes linearly separable.

Example: Transform 2D data to 3D

  • Original: $(x_1, x_2)$
  • Transformed: $(x_1, x_2, x_1^2 + x_2^2)$

Now the circles become separable by a plane in 3D!

But Wait... 🤔

Computing in high-dimensional spaces is expensive.

The Trick: We never explicitly compute the transformation! Instead, we use kernel functions that compute dot products in the transformed space directly.

Kernel Functions

1. Linear Kernel (no transformation)

$$K(x_i, x_j) = x_i \cdot x_j$$

2. RBF (Radial Basis Function) Kernel

$$K(x_i, x_j) = e^{-\gamma ||x_i - x_j||^2}$$

Where:

  • $\gamma$ controls the "influence radius" of each point
  • Higher $\gamma$ = more local decision boundaries (can overfit)
  • Lower $\gamma$ = smoother decision boundaries

Implementation Breakdown

class KernelSVM:
    def __init__(self, kernel='rbf', C=1.0, gamma=0.5, n_iterations=1000):
        self.kernel = kernel        # Which kernel to use
        self.C = C                  # Regularization parameter
        self.gamma = gamma          # RBF kernel parameter
        self.n_iterations = n_iterations
        self.alpha = None           # Lagrange multipliers (dual formulation)
        self.b = 0
        self.X_train = None         # Need to store training data!
        self.y_train = None

    def rbf_kernel(self, X1, X2):
        # Compute squared distances efficiently
        sq_dists = np.sum(X1**2, axis=1).reshape(-1, 1) + \
                   np.sum(X2**2, axis=1) - 2 * np.dot(X1, X2.T)
        return np.exp(-self.gamma * sq_dists)

    def linear_kernel(self, X1, X2):
        return np.dot(X1, X2.T)

    def compute_kernel(self, X1, X2):
        if self.kernel == 'rbf':
            return self.rbf_kernel(X1, X2)
        elif self.kernel == 'linear':
            return self.linear_kernel(X1, X2)

The Dual Formulation

In kernel SVM, we work with alpha values instead of weights directly.

Prediction becomes: $$f(x) = \sum_{i=1}^{n} \alpha_i y_i K(x, x_i) - b$$

Where:

  • $\alpha_i$ are learned coefficients
  • Only support vectors have $\alpha_i &gt; 0$
  • Other points have $\alpha_i = 0$

Training Process

def fit(self, X, y):
    n_samples = X.shape[0]
    self.X_train = X
    self.y_train = np.where(y <= 0, -1, 1)

    self.alpha = np.zeros(n_samples)  # Start with zero alphas
    K = self.compute_kernel(X, X)     # Compute kernel matrix

    for iteration in range(self.n_iterations):
        for i in range(n_samples):
            # Predict using current alphas
            prediction = np.sum(self.alpha * self.y_train * K[i, :]) - self.b

            # Check margin violation
            if self.y_train[i] * prediction < 1:
                # Update alpha for this point
                self.alpha[i] += 0.01 * (1 - self.y_train[i] * prediction)
                # Clip alpha to [0, C]
                self.alpha[i] = min(self.C, max(0, self.alpha[i]))

    # Compute bias using support vectors
    support_vectors = self.alpha > 1e-5
    self.b = np.mean(self.y_train[support_vectors] - np.sum(
        self.alpha[support_vectors] * self.y_train[support_vectors] *
        K[support_vectors][:, support_vectors], axis=1))

RBF Kernel Step-by-Step Example

Training Data:

  • Point A: $x_A = [1.0, 1.0]$, $y_A = 1$
  • Point B: $x_B = [-1.0, -1.0]$, $y_B = -1$
  • Point C: $x_C = [0.0, 0.0]$, $y_C = 1$

Parameters:

  • $\gamma = 1.0$
  • $C = 1.0$

Step 1: Compute Kernel Matrix

$K_{AB} = e^{-\gamma ||x_A - x_B||^2}$

$$||x_A - x_B||^2 = (1-(-1))^2 + (1-(-1))^2 = 4 + 4 = 8$$ $$K_{AB} = e^{-1.0 \times 8} = e^{-8} \approx 0.000335$$

$K_{AA} = e^{-\gamma ||x_A - x_A||^2} = e^0 = 1$ (point to itself)

$K_{AC} = e^{-\gamma ||x_A - x_C||^2}$ $$||x_A - x_C||^2 = (1-0)^2 + (1-0)^2 = 2$$ $$K_{AC} = e^{-1.0 \times 2} = e^{-2} \approx 0.135$$

Kernel Matrix K:

     A      B      C
A [1.000  0.000  0.135]
B [0.000  1.000  0.135]
C [0.135  0.135  1.000]

Step 2: Initialize

  • $\alpha = [0, 0, 0]$
  • $b = 0$

Step 3: First Iteration - Point A

Prediction for point A: $$pred_A = \sum \alpha_i y_i K(x_A, x_i) - b$$ $$pred_A = 0 \times 1 \times 1.0 + 0 \times (-1) \times 0.0 + 0 \times 1 \times 0.135 - 0 = 0$$

Margin check: $$y_A \times pred_A = 1 \times 0 = 0 &lt; 1$$ ❌

Update alpha: $$\alpha_A = \alpha_A + 0.01 \times (1 - y_A \times pred_A)$$ $$\alpha_A = 0 + 0.01 \times (1 - 0) = 0.01$$

Step 4: After many iterations

  • $\alpha_A \approx 0.5$ (support vector)
  • $\alpha_B \approx 0.5$ (support vector)
  • $\alpha_C \approx 0.2$ (support vector, closer to boundary)

Making a Prediction for new point $x_{new} = [0.5, 0.5]$:

Compute kernels: $$K(x_{new}, x_A) = e^{-1.0 \times ((0.5-1)^2 + (0.5-1)^2)} = e^{-0.5} \approx 0.606$$ $$K(x_{new}, x_B) = e^{-1.0 \times ((0.5-(-1))^2 + (0.5-(-1))^2)} = e^{-4.5} \approx 0.011$$ $$K(x_{new}, x_C) = e^{-1.0 \times ((0.5-0)^2 + (0.5-0)^2)} = e^{-0.5} \approx 0.606$$

Prediction: $$f(x_{new}) = 0.5 \times 1 \times 0.606 + 0.5 \times (-1) \times 0.011 + 0.2 \times 1 \times 0.606 - b$$ $$f(x_{new}) = 0.303 - 0.0055 + 0.1212 - b \approx 0.42 - b$$

If $f(x_{new}) &gt; 0$, predict class 1; otherwise, class -1.

RBF Kernel Result

Notice how the RBF kernel creates curved decision boundaries that perfectly separate the concentric circles!


Parameter Effects

C Parameter (Regularization)

$C = \frac{1}{2\lambda}$

  • Small C (high $\lambda$): More regularization, wider margin, more tolerance for errors
  • Large C (low $\lambda$): Less regularization, narrower margin, less tolerance for errors

C Parameter Effects

C = 0.01: Very smooth boundary, allows many margin violations C = 1.0: Balanced approach C = 100.0: Very strict, fits training data very closely (risk of overfitting)

Gamma Parameter (RBF Kernel)

  • Small $\gamma$: Each point has far-reaching influence, smoother boundaries
  • Large $\gamma$: Each point has local influence only, more complex boundaries

Gamma Parameter Effects

Gamma = 0.1: Very smooth, almost linear Gamma = 1.0: Moderately complex Gamma = 10.0: Very complex, risk of overfitting


Multi-Class SVM

One-vs-Rest Strategy

SVM is inherently a binary classifier. For multi-class problems, we use the One-vs-Rest (OvR) approach:

  1. Train N classifiers (one for each class)
  2. Classifier $i$ learns: "Class $i$" vs "All other classes"
  3. At prediction time: Run all classifiers, pick the one with highest confidence

Implementation

class MultiClassSVM:
    def __init__(self, n_classes, learning_rate=0.001, lambda_param=0.01, n_iterations=1000):
        self.n_classes = n_classes
        self.classifiers = []

        # Create one binary classifier for each class
        for _ in range(n_classes):
            self.classifiers.append(SoftMarginSVM(learning_rate, lambda_param, n_iterations))

    def fit(self, X, y):
        for class_idx in range(self.n_classes):
            # Create binary labels: +1 for this class, -1 for others
            y_binary = np.where(y == class_idx, 1, -1)
            self.classifiers[class_idx].fit(X, y_binary)

    def predict(self, X):
        predictions = np.zeros((X.shape[0], self.n_classes))

        # Get decision values from each classifier
        for class_idx in range(self.n_classes):
            decision_values = np.dot(X, self.classifiers[class_idx].w) - \
                            self.classifiers[class_idx].b
            predictions[:, class_idx] = decision_values

        # Pick class with highest decision value
        return np.argmax(predictions, axis=1)

Example Flow

3-Class Problem: Red, Blue, Green

Classifier 1 (Red vs Others):

  • Red points: label = +1
  • Blue & Green points: label = -1
  • Learns boundary separating Red from (Blue + Green)

Classifier 2 (Blue vs Others):

  • Blue points: label = +1
  • Red & Green points: label = -1
  • Learns boundary separating Blue from (Red + Green)

Classifier 3 (Green vs Others):

  • Green points: label = +1
  • Red & Blue points: label = -1
  • Learns boundary separating Green from (Red + Blue)

Making a Prediction for point $x = [1.5, 2.0]$:

$$f_1(x) = w_1 \cdot x - b_1 = 0.8$$ (distance to Red boundary) $$f_2(x) = w_2 \cdot x - b_2 = 1.5$$ (distance to Blue boundary) ← Highest $$f_3(x) = w_3 \cdot x - b_3 = -0.3$$ (distance to Green boundary)

Prediction: Blue (class 2)

Multi-Class Result

The visualization shows how three binary classifiers create decision regions for three classes.


Step-by-Step Flow Examples

Complete Training Flow

Let's trace through a complete training cycle with a tiny dataset.

Dataset:

Point 1: x = [1, 1], y = 1
Point 2: x = [2, 2], y = 1
Point 3: x = [-1, -1], y = -1
Point 4: x = [-2, -2], y = -1

Hyperparameters:

  • Learning rate = 0.1
  • Lambda = 0.01
  • Iterations = 3 (for demonstration)

Iteration 1

Initial State:

w = [0, 0]
b = 0

Point 1: x = [1, 1], y = 1

  1. Compute decision value: $$w \cdot x - b = [0, 0] \cdot [1, 1] - 0 = 0$$

  2. Check condition: $$y(w \cdot x - b) = 1 \times 0 = 0 &lt; 1$$ ❌ (Margin violated)

  3. Update:

    w_new = w - lr * (2 * lambda * w - x * y)
    w_new = [0, 0] - 0.1 * (2 * 0.01 * [0, 0] - [1, 1] * 1)
    w_new = [0, 0] - 0.1 * (0 - [1, 1])
    w_new = [0, 0] + 0.1 * [1, 1]
    w_new = [0.1, 0.1]
    
    b_new = b - lr * y
    b_new = 0 - 0.1 * 1
    b_new = -0.1
    

After Point 1:

w = [0.1, 0.1]
b = -0.1

Point 2: x = [2, 2], y = 1

  1. Compute decision value: $$w \cdot x - b = [0.1, 0.1] \cdot [2, 2] - (-0.1) = 0.2 + 0.2 + 0.1 = 0.5$$

  2. Check condition: $$y(w \cdot x - b) = 1 \times 0.5 = 0.5 &lt; 1$$ ❌

  3. Update:

    w_new = [0.1, 0.1] - 0.1 * (2 * 0.01 * [0.1, 0.1] - [2, 2] * 1)
    w_new = [0.1, 0.1] - 0.1 * ([0.002, 0.002] - [2, 2])
    w_new = [0.1, 0.1] - 0.1 * [-1.998, -1.998]
    w_new = [0.1, 0.1] + [0.1998, 0.1998]
    w_new = [0.2998, 0.2998] ≈ [0.3, 0.3]
    
    b_new = -0.1 - 0.1 * 1 = -0.2
    

After Point 2:

w = [0.3, 0.3]
b = -0.2

Point 3: x = [-1, -1], y = -1

  1. Compute decision value: $$w \cdot x - b = [0.3, 0.3] \cdot [-1, -1] - (-0.2) = -0.3 - 0.3 + 0.2 = -0.4$$

  2. Check condition: $$y(w \cdot x - b) = -1 \times (-0.4) = 0.4 &lt; 1$$ ❌

  3. Update:

    w_new = [0.3, 0.3] - 0.1 * (2 * 0.01 * [0.3, 0.3] - [-1, -1] * (-1))
    w_new = [0.3, 0.3] - 0.1 * ([0.006, 0.006] - [1, 1])
    w_new = [0.3, 0.3] - 0.1 * [-0.994, -0.994]
    w_new = [0.3, 0.3] + [0.0994, 0.0994]
    w_new = [0.3994, 0.3994] ≈ [0.4, 0.4]
    
    b_new = -0.2 - 0.1 * (-1) = -0.2 + 0.1 = -0.1
    

After Point 3:

w = [0.4, 0.4]
b = -0.1

Point 4: x = [-2, -2], y = -1

  1. Compute decision value: $$w \cdot x - b = [0.4, 0.4] \cdot [-2, -2] - (-0.1) = -0.8 - 0.8 + 0.1 = -1.5$$

  2. Check condition: $$y(w \cdot x - b) = -1 \times (-1.5) = 1.5 \geq 1$$ ✅ (Good margin!)

  3. Update (regularization only):

    w_new = [0.4, 0.4] - 0.1 * (2 * 0.01 * [0.4, 0.4])
    w_new = [0.4, 0.4] - 0.1 * [0.008, 0.008]
    w_new = [0.4, 0.4] - [0.0008, 0.0008]
    w_new = [0.3992, 0.3992]
    
    b stays the same = -0.1
    

After Point 4 (End of Iteration 1):

w = [0.3992, 0.3992]
b = -0.1

Iteration 2 & 3

The same process repeats. After each full iteration through all points:

  • Weights gradually point in the optimal direction
  • Bias gradually shifts to the optimal position
  • Margin violations decrease
  • Loss decreases

Final State (after convergence):

w ≈ [0.707, 0.707]  (normalized direction)
b ≈ 0
Hyperplane: 0.707*x1 + 0.707*x2 = 0  (45° line through origin)

Internal Changes During Training

Visualization of What's Happening

Imagine we're training on the dataset shown earlier. Here's what happens internally:

Before Training (Iteration 0)

Hyperplane: Not defined (random)
Margin: None
Support Vectors: None identified
Loss: Maximum

All points are likely misclassified or have insufficient margin.


Early Training (Iterations 1-100)

Hyperplane: Roughly oriented, but not optimal
Margin: Narrow, many violations
Support Vectors: Many points acting as support vectors
Loss: Decreasing rapidly

What's happening:

  • $w$ is adjusting rapidly toward the optimal direction
  • $b$ is shifting to center the hyperplane
  • Many points violate the margin, causing frequent weight updates
  • The decision boundary is "unstable" - changes significantly each iteration

Mid Training (Iterations 100-500)

Hyperplane: Close to optimal orientation
Margin: Widening, fewer violations
Support Vectors: Narrowing down to key points
Loss: Decreasing slowly

What's happening:

  • $w$ is fine-tuning its direction
  • Most points are correctly classified with good margin
  • Only a few points (near the boundary) still cause updates
  • The decision boundary is "stabilizing"

Late Training (Iterations 500-1000)

Hyperplane: Near-optimal
Margin: Maximum achievable width
Support Vectors: Only the closest points to boundary
Loss: Converged (minimal changes)

What's happening:

  • $w$ makes tiny adjustments (mostly regularization)
  • Only support vectors cause meaningful updates
  • The decision boundary is "stable" - barely changes
  • The algorithm has found the best balance between margin width and error tolerance

After Training (Converged)

Hyperplane: Optimal
Margin: Maximum
Support Vectors: Identified (alpha_i > 0)
Loss: Minimized

Final Decision Boundary:

  • Maximizes distance to nearest points of each class
  • Generalizes well to new data
  • Is defined entirely by support vectors (other points can be removed without affecting the boundary)

Loss Landscape Evolution

Think of the loss function as a landscape with hills and valleys. Training is like rolling a ball down to the lowest valley.

Start (high loss): Ball at top of hill

  • Large gradients
  • Big weight updates
  • Rapid descent

Middle (medium loss): Ball on slope

  • Medium gradients
  • Moderate updates
  • Steady descent

End (low loss): Ball in valley

  • Small gradients
  • Tiny updates
  • Convergence

Loss Over Time

The loss curve shows this progression clearly - steep drop initially, then gradual convergence.


Key Takeaways

When to Use Each SVM Type

  1. Hard Margin SVM

    • ✅ Data is perfectly linearly separable
    • ✅ No noise or outliers
    • ❌ Real-world data (almost never)
  2. Soft Margin SVM

    • ✅ Data is mostly linearly separable with some noise
    • ✅ Presence of outliers
    • ✅ Default choice for linear problems
  3. RBF Kernel SVM

    • ✅ Non-linear decision boundaries needed
    • ✅ Complex patterns
    • ⚠️ Need to tune gamma carefully
  4. Multi-Class SVM

    • ✅ More than 2 classes
    • ✅ Combines any of the above approaches

The Training Process in One Sentence

For each training point, check if it violates the margin; if yes, push the hyperplane away from it; if no, just keep the hyperplane simple (regularization).

The Prediction Process in One Sentence

Compute which side of the hyperplane the point falls on; positive side = class +1, negative side = class -1.

Why SVMs are Powerful

  1. Maximum Margin: More robust to new data than just finding "any" separating boundary
  2. Kernel Trick: Can handle non-linear problems efficiently
  3. Support Vectors: Only care about points near the boundary (efficient)
  4. Regularization: Built-in protection against overfitting
  5. Theoretical Foundation: Strong mathematical guarantees

Conclusion

You've implemented a complete SVM library from scratch with:

  • Hard and soft margin linear SVMs
  • Kernel SVMs with RBF
  • Multi-class classification
  • Parameter tuning visualizations

The key insight is: SVMs find the hyperplane that maximizes the margin between classes, and they do this by iteratively adjusting weights based on which points violate the margin constraint.

Understanding the flow means understanding this cycle:

  1. Check each point's margin
  2. Update weights if margin is violated
  3. Regularize to keep the model simple
  4. Repeat until convergence

The internal changes are the gradual evolution of $w$ and $b$ from random values to optimal values that define the maximum-margin hyperplane.

Now you truly understand not just what your code does, but why and how it works! 🎉

About

Complete implementation of Support Vector Machines from scratch including Hard Margin, Soft Margin, Kernel SVM, and Multi-Class SVM with detailed explanations and visualizations

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages