- Introduction & Intuition
- Hard Margin SVM
- Soft Margin SVM
- Kernel SVM
- Multi-Class SVM
- Step-by-Step Flow Examples
- Internal Changes During Training
Support Vector Machine is a supervised learning algorithm that finds the optimal hyperplane to separate different classes in your data. Think of it as finding the best "decision boundary" that maximizes the margin between classes.
Imagine you have two groups of points (red and blue) on a 2D plane. You could draw many lines to separate them, but which line is best?
SVM says: The best line is the one that has the maximum distance (margin) from the nearest points of both classes.
- Hyperplane: The decision boundary (a line in 2D, a plane in 3D, etc.)
- Support Vectors: The data points closest to the hyperplane that "support" or define it
- Margin: The distance between the hyperplane and the nearest data points
- Weight Vector (w): Defines the orientation of the hyperplane
- Bias (b): Shifts the hyperplane position
The hyperplane equation is:
Where:
-
$w$ = weight vector (normal to the hyperplane) -
$x$ = data point -
$b$ = bias term
For a point to be correctly classified:
Where
Goal: Maximize the margin =
This is equivalent to: Minimize
Subject to:
class HardMarginSVM:
def __init__(self, learning_rate=0.001, lambda_param=0.01, n_iterations=1000):
self.lr = learning_rate # How fast we update weights
self.lambda_param = lambda_param # Regularization strength
self.n_iterations = n_iterations # Number of training iterations
self.w = None # Weight vector (to be learned)
self.b = None # Bias term (to be learned)def fit(self, X, y):
n_samples, n_features = X.shape
y_ = np.where(y <= 0, -1, 1) # Convert labels to -1 and 1
self.w = np.zeros(n_features) # Start with zero weights
self.b = 0 # Start with zero bias
for _ in range(self.n_iterations):
for idx, x_i in enumerate(X):
# Check if point is correctly classified with margin >= 1
condition = y_[idx] * (np.dot(x_i, self.w) - self.b) >= 1
if condition:
# Point is on correct side with sufficient margin
# Only update weights to keep them small (regularization)
self.w -= self.lr * (2 * self.lambda_param * self.w)
else:
# Point violates margin constraint
# Update both weights and bias
self.w -= self.lr * (2 * self.lambda_param * self.w - np.dot(x_i, y_[idx]))
self.b -= self.lr * y_[idx]Let's trace through one iteration with a concrete example:
Initial State:
$w = [0.5, 0.3]$ $b = 0.1$ - Learning rate = 0.001
- Lambda = 0.01
Training Point:
$x_i = [2.0, 3.0]$ -
$y_i = 1$ (positive class)
Step 1: Calculate the decision value
Step 2: Check the margin condition
Since
Step 3: Update weights (regularization only)
If the point violated the margin:
Say
Update both weights and bias:
The solid black line is the decision boundary, and the dashed lines show the margin.
Hard margin SVM assumes data is perfectly linearly separable. In reality, data often has:
- Noise
- Outliers
- Overlapping classes
Soft margin SVM allows some points to be inside the margin or even misclassified, but penalizes them.
We introduce slack variables
Minimize:
Subject to:
Where:
-
$C$ is the penalty parameter (how much we care about misclassifications) -
$\xi_i$ measures how much point$i$ violates the margin
We use the Hinge Loss:
Total Loss:
Note:
class SoftMarginSVM:
def __init__(self, learning_rate=0.001, lambda_param=0.01, n_iterations=1000):
self.lr = learning_rate
self.lambda_param = lambda_param
self.n_iterations = n_iterations
self.w = None
self.b = None
self.losses = [] # Track loss over time
def fit(self, X, y):
n_samples, n_features = X.shape
y_ = np.where(y <= 0, -1, 1)
self.w = np.zeros(n_features)
self.b = 0
for iteration in range(self.n_iterations):
epoch_loss = 0
for idx, x_i in enumerate(X):
condition = y_[idx] * (np.dot(x_i, self.w) - self.b) >= 1
if condition:
# No hinge loss, only regularization
self.w -= self.lr * (2 * self.lambda_param * self.w)
loss = self.lambda_param * np.dot(self.w, self.w)
else:
# Has hinge loss, update both w and b
self.w -= self.lr * (2 * self.lambda_param * self.w - np.dot(x_i, y_[idx]))
self.b -= self.lr * y_[idx]
loss = self.lambda_param * np.dot(self.w, self.w) + \
max(0, 1 - y_[idx] * (np.dot(x_i, self.w) - self.b))
epoch_loss += loss
self.losses.append(epoch_loss / n_samples)Scenario: Training on noisy data with outliers
Epoch 0 (Initial):
$w = [0, 0]$ $b = 0$ - Total Loss = High (many misclassifications)
Point 1:
-
Hinge Loss =
$\max(0, 1 - 0) = 1$ - Update:
$w$ increases toward$x$ direction
Point 2:
-
Hinge Loss =
$\max(0, 1 - 0.0034) = 0.9966$ - Update:
$w$ adjusts,$b$ adjusts
Outlier Point:
- This point will consistently have high hinge loss
- But the algorithm tolerates it (doesn't try too hard to fix)
- The loss is bounded by the C parameter
After 1000 iterations:
-
$w$ converges to optimal direction -
$b$ converges to optimal offset - Loss decreases steadily
Notice how the decision boundary is more robust to outliers compared to hard margin.
Some datasets are not linearly separable in their original space:
No straight line can separate these concentric circles!
Key Insight: Transform data to a higher-dimensional space where it becomes linearly separable.
Example: Transform 2D data to 3D
- Original:
$(x_1, x_2)$ - Transformed:
$(x_1, x_2, x_1^2 + x_2^2)$
Now the circles become separable by a plane in 3D!
Computing in high-dimensional spaces is expensive.
The Trick: We never explicitly compute the transformation! Instead, we use kernel functions that compute dot products in the transformed space directly.
Where:
-
$\gamma$ controls the "influence radius" of each point - Higher
$\gamma$ = more local decision boundaries (can overfit) - Lower
$\gamma$ = smoother decision boundaries
class KernelSVM:
def __init__(self, kernel='rbf', C=1.0, gamma=0.5, n_iterations=1000):
self.kernel = kernel # Which kernel to use
self.C = C # Regularization parameter
self.gamma = gamma # RBF kernel parameter
self.n_iterations = n_iterations
self.alpha = None # Lagrange multipliers (dual formulation)
self.b = 0
self.X_train = None # Need to store training data!
self.y_train = None
def rbf_kernel(self, X1, X2):
# Compute squared distances efficiently
sq_dists = np.sum(X1**2, axis=1).reshape(-1, 1) + \
np.sum(X2**2, axis=1) - 2 * np.dot(X1, X2.T)
return np.exp(-self.gamma * sq_dists)
def linear_kernel(self, X1, X2):
return np.dot(X1, X2.T)
def compute_kernel(self, X1, X2):
if self.kernel == 'rbf':
return self.rbf_kernel(X1, X2)
elif self.kernel == 'linear':
return self.linear_kernel(X1, X2)In kernel SVM, we work with alpha values instead of weights directly.
Prediction becomes:
Where:
-
$\alpha_i$ are learned coefficients - Only support vectors have
$\alpha_i > 0$ - Other points have
$\alpha_i = 0$
def fit(self, X, y):
n_samples = X.shape[0]
self.X_train = X
self.y_train = np.where(y <= 0, -1, 1)
self.alpha = np.zeros(n_samples) # Start with zero alphas
K = self.compute_kernel(X, X) # Compute kernel matrix
for iteration in range(self.n_iterations):
for i in range(n_samples):
# Predict using current alphas
prediction = np.sum(self.alpha * self.y_train * K[i, :]) - self.b
# Check margin violation
if self.y_train[i] * prediction < 1:
# Update alpha for this point
self.alpha[i] += 0.01 * (1 - self.y_train[i] * prediction)
# Clip alpha to [0, C]
self.alpha[i] = min(self.C, max(0, self.alpha[i]))
# Compute bias using support vectors
support_vectors = self.alpha > 1e-5
self.b = np.mean(self.y_train[support_vectors] - np.sum(
self.alpha[support_vectors] * self.y_train[support_vectors] *
K[support_vectors][:, support_vectors], axis=1))Training Data:
- Point A:
$x_A = [1.0, 1.0]$ ,$y_A = 1$ - Point B:
$x_B = [-1.0, -1.0]$ ,$y_B = -1$ - Point C:
$x_C = [0.0, 0.0]$ ,$y_C = 1$
Parameters:
$\gamma = 1.0$ $C = 1.0$
Step 1: Compute Kernel Matrix
Kernel Matrix K:
A B C
A [1.000 0.000 0.135]
B [0.000 1.000 0.135]
C [0.135 0.135 1.000]
Step 2: Initialize
$\alpha = [0, 0, 0]$ $b = 0$
Step 3: First Iteration - Point A
Prediction for point A:
Margin check:
Update alpha:
Step 4: After many iterations
-
$\alpha_A \approx 0.5$ (support vector) -
$\alpha_B \approx 0.5$ (support vector) -
$\alpha_C \approx 0.2$ (support vector, closer to boundary)
Making a Prediction for new point
Compute kernels:
Prediction:
If
Notice how the RBF kernel creates curved decision boundaries that perfectly separate the concentric circles!
-
Small C (high
$\lambda$ ): More regularization, wider margin, more tolerance for errors -
Large C (low
$\lambda$ ): Less regularization, narrower margin, less tolerance for errors
C = 0.01: Very smooth boundary, allows many margin violations C = 1.0: Balanced approach C = 100.0: Very strict, fits training data very closely (risk of overfitting)
-
Small
$\gamma$ : Each point has far-reaching influence, smoother boundaries -
Large
$\gamma$ : Each point has local influence only, more complex boundaries
Gamma = 0.1: Very smooth, almost linear Gamma = 1.0: Moderately complex Gamma = 10.0: Very complex, risk of overfitting
SVM is inherently a binary classifier. For multi-class problems, we use the One-vs-Rest (OvR) approach:
- Train N classifiers (one for each class)
- Classifier
$i$ learns: "Class$i$ " vs "All other classes" - At prediction time: Run all classifiers, pick the one with highest confidence
class MultiClassSVM:
def __init__(self, n_classes, learning_rate=0.001, lambda_param=0.01, n_iterations=1000):
self.n_classes = n_classes
self.classifiers = []
# Create one binary classifier for each class
for _ in range(n_classes):
self.classifiers.append(SoftMarginSVM(learning_rate, lambda_param, n_iterations))
def fit(self, X, y):
for class_idx in range(self.n_classes):
# Create binary labels: +1 for this class, -1 for others
y_binary = np.where(y == class_idx, 1, -1)
self.classifiers[class_idx].fit(X, y_binary)
def predict(self, X):
predictions = np.zeros((X.shape[0], self.n_classes))
# Get decision values from each classifier
for class_idx in range(self.n_classes):
decision_values = np.dot(X, self.classifiers[class_idx].w) - \
self.classifiers[class_idx].b
predictions[:, class_idx] = decision_values
# Pick class with highest decision value
return np.argmax(predictions, axis=1)3-Class Problem: Red, Blue, Green
Classifier 1 (Red vs Others):
- Red points: label = +1
- Blue & Green points: label = -1
- Learns boundary separating Red from (Blue + Green)
Classifier 2 (Blue vs Others):
- Blue points: label = +1
- Red & Green points: label = -1
- Learns boundary separating Blue from (Red + Green)
Classifier 3 (Green vs Others):
- Green points: label = +1
- Red & Blue points: label = -1
- Learns boundary separating Green from (Red + Blue)
Making a Prediction for point
Prediction: Blue (class 2)
The visualization shows how three binary classifiers create decision regions for three classes.
Let's trace through a complete training cycle with a tiny dataset.
Dataset:
Point 1: x = [1, 1], y = 1
Point 2: x = [2, 2], y = 1
Point 3: x = [-1, -1], y = -1
Point 4: x = [-2, -2], y = -1
Hyperparameters:
- Learning rate = 0.1
- Lambda = 0.01
- Iterations = 3 (for demonstration)
Initial State:
w = [0, 0]
b = 0
Point 1: x = [1, 1], y = 1
-
Compute decision value:
$$w \cdot x - b = [0, 0] \cdot [1, 1] - 0 = 0$$ -
Check condition:
$$y(w \cdot x - b) = 1 \times 0 = 0 < 1$$ ❌ (Margin violated) -
Update:
w_new = w - lr * (2 * lambda * w - x * y) w_new = [0, 0] - 0.1 * (2 * 0.01 * [0, 0] - [1, 1] * 1) w_new = [0, 0] - 0.1 * (0 - [1, 1]) w_new = [0, 0] + 0.1 * [1, 1] w_new = [0.1, 0.1] b_new = b - lr * y b_new = 0 - 0.1 * 1 b_new = -0.1
After Point 1:
w = [0.1, 0.1]
b = -0.1
Point 2: x = [2, 2], y = 1
-
Compute decision value:
$$w \cdot x - b = [0.1, 0.1] \cdot [2, 2] - (-0.1) = 0.2 + 0.2 + 0.1 = 0.5$$ -
Check condition:
$$y(w \cdot x - b) = 1 \times 0.5 = 0.5 < 1$$ ❌ -
Update:
w_new = [0.1, 0.1] - 0.1 * (2 * 0.01 * [0.1, 0.1] - [2, 2] * 1) w_new = [0.1, 0.1] - 0.1 * ([0.002, 0.002] - [2, 2]) w_new = [0.1, 0.1] - 0.1 * [-1.998, -1.998] w_new = [0.1, 0.1] + [0.1998, 0.1998] w_new = [0.2998, 0.2998] ≈ [0.3, 0.3] b_new = -0.1 - 0.1 * 1 = -0.2
After Point 2:
w = [0.3, 0.3]
b = -0.2
Point 3: x = [-1, -1], y = -1
-
Compute decision value:
$$w \cdot x - b = [0.3, 0.3] \cdot [-1, -1] - (-0.2) = -0.3 - 0.3 + 0.2 = -0.4$$ -
Check condition:
$$y(w \cdot x - b) = -1 \times (-0.4) = 0.4 < 1$$ ❌ -
Update:
w_new = [0.3, 0.3] - 0.1 * (2 * 0.01 * [0.3, 0.3] - [-1, -1] * (-1)) w_new = [0.3, 0.3] - 0.1 * ([0.006, 0.006] - [1, 1]) w_new = [0.3, 0.3] - 0.1 * [-0.994, -0.994] w_new = [0.3, 0.3] + [0.0994, 0.0994] w_new = [0.3994, 0.3994] ≈ [0.4, 0.4] b_new = -0.2 - 0.1 * (-1) = -0.2 + 0.1 = -0.1
After Point 3:
w = [0.4, 0.4]
b = -0.1
Point 4: x = [-2, -2], y = -1
-
Compute decision value:
$$w \cdot x - b = [0.4, 0.4] \cdot [-2, -2] - (-0.1) = -0.8 - 0.8 + 0.1 = -1.5$$ -
Check condition:
$$y(w \cdot x - b) = -1 \times (-1.5) = 1.5 \geq 1$$ ✅ (Good margin!) -
Update (regularization only):
w_new = [0.4, 0.4] - 0.1 * (2 * 0.01 * [0.4, 0.4]) w_new = [0.4, 0.4] - 0.1 * [0.008, 0.008] w_new = [0.4, 0.4] - [0.0008, 0.0008] w_new = [0.3992, 0.3992] b stays the same = -0.1
After Point 4 (End of Iteration 1):
w = [0.3992, 0.3992]
b = -0.1
The same process repeats. After each full iteration through all points:
- Weights gradually point in the optimal direction
- Bias gradually shifts to the optimal position
- Margin violations decrease
- Loss decreases
Final State (after convergence):
w ≈ [0.707, 0.707] (normalized direction)
b ≈ 0
Hyperplane: 0.707*x1 + 0.707*x2 = 0 (45° line through origin)
Imagine we're training on the dataset shown earlier. Here's what happens internally:
Hyperplane: Not defined (random)
Margin: None
Support Vectors: None identified
Loss: Maximum
All points are likely misclassified or have insufficient margin.
Hyperplane: Roughly oriented, but not optimal
Margin: Narrow, many violations
Support Vectors: Many points acting as support vectors
Loss: Decreasing rapidly
What's happening:
-
$w$ is adjusting rapidly toward the optimal direction -
$b$ is shifting to center the hyperplane - Many points violate the margin, causing frequent weight updates
- The decision boundary is "unstable" - changes significantly each iteration
Hyperplane: Close to optimal orientation
Margin: Widening, fewer violations
Support Vectors: Narrowing down to key points
Loss: Decreasing slowly
What's happening:
-
$w$ is fine-tuning its direction - Most points are correctly classified with good margin
- Only a few points (near the boundary) still cause updates
- The decision boundary is "stabilizing"
Hyperplane: Near-optimal
Margin: Maximum achievable width
Support Vectors: Only the closest points to boundary
Loss: Converged (minimal changes)
What's happening:
-
$w$ makes tiny adjustments (mostly regularization) - Only support vectors cause meaningful updates
- The decision boundary is "stable" - barely changes
- The algorithm has found the best balance between margin width and error tolerance
Hyperplane: Optimal
Margin: Maximum
Support Vectors: Identified (alpha_i > 0)
Loss: Minimized
Final Decision Boundary:
- Maximizes distance to nearest points of each class
- Generalizes well to new data
- Is defined entirely by support vectors (other points can be removed without affecting the boundary)
Think of the loss function as a landscape with hills and valleys. Training is like rolling a ball down to the lowest valley.
Start (high loss): Ball at top of hill
- Large gradients
- Big weight updates
- Rapid descent
Middle (medium loss): Ball on slope
- Medium gradients
- Moderate updates
- Steady descent
End (low loss): Ball in valley
- Small gradients
- Tiny updates
- Convergence
The loss curve shows this progression clearly - steep drop initially, then gradual convergence.
-
Hard Margin SVM
- ✅ Data is perfectly linearly separable
- ✅ No noise or outliers
- ❌ Real-world data (almost never)
-
Soft Margin SVM
- ✅ Data is mostly linearly separable with some noise
- ✅ Presence of outliers
- ✅ Default choice for linear problems
-
RBF Kernel SVM
- ✅ Non-linear decision boundaries needed
- ✅ Complex patterns
⚠️ Need to tune gamma carefully
-
Multi-Class SVM
- ✅ More than 2 classes
- ✅ Combines any of the above approaches
For each training point, check if it violates the margin; if yes, push the hyperplane away from it; if no, just keep the hyperplane simple (regularization).
Compute which side of the hyperplane the point falls on; positive side = class +1, negative side = class -1.
- Maximum Margin: More robust to new data than just finding "any" separating boundary
- Kernel Trick: Can handle non-linear problems efficiently
- Support Vectors: Only care about points near the boundary (efficient)
- Regularization: Built-in protection against overfitting
- Theoretical Foundation: Strong mathematical guarantees
You've implemented a complete SVM library from scratch with:
- Hard and soft margin linear SVMs
- Kernel SVMs with RBF
- Multi-class classification
- Parameter tuning visualizations
The key insight is: SVMs find the hyperplane that maximizes the margin between classes, and they do this by iteratively adjusting weights based on which points violate the margin constraint.
Understanding the flow means understanding this cycle:
- Check each point's margin
- Update weights if margin is violated
- Regularize to keep the model simple
- Repeat until convergence
The internal changes are the gradual evolution of
Now you truly understand not just what your code does, but why and how it works! 🎉








