204.6.5 The Non-Linear Decision Boundary
Link to the previous post : https://statinfer.com/204-6-4-building-svm-model-in-python/
The Non-Linear Decision Boundary
- In the above examples we can clearly see the decision boundary is linear.
- SVM works well when the data points are linearly separable.
- If the decision boundary is non-liner then SVM may struggle to classify.
- Observe the below examples, the classes are not linearly separable.
- SVM has no direct theory to set the non-liner decision boundary models.
Mapping to Higher Dimensional Space
- The original maximum-margin hyperplane algorithm proposed by Vapnik in 1963 constructed a linear classifier.
- To fit a non liner boundary classier, we can create new variables(dimensions) in the data and see whether the decision boundary is linear.
- In 1992, Bernhard E. Boser, Isabelle M. Guyon and Vladimir N. Vapnik suggested a way to create nonlinear classifiers by applying the kernel trick.
- In the below example, A single linear classifier is not sufficient.
- Lets create a new variable x2=(x1)2. In the higher dimensional space.
- We can clearly see a possibility of single linear decision boundary.
- This is called kernel trick.
- We used a function ϕ(x)=(x,(x2)) to transform the data x into a higher dimensional space.
- In the higher dimensional space, we could easily fit a liner decision boundary.
- This function ϕ(x) is known as kernel function and this process is known as kernel trick in SVM.
- Kernel trick solves the non-linear decision boundary problem much like the hidden layers in neural networks.
- Kernel trick is simply increasing the number of dimensions. It is to make the non-linear decision boundary in lower dimensional space as a linear decision boundary, in higher dimensional space.
- In simple words, Kernel trick makes the non-linear decision boundary to linear (in higher dimensional space).