204.7.7 Boosting

Link to the previous post : https://statinfer.com/204-7-6-practice-random-forest/

In this post, we will cover how boosting work and the type of boosting algorithms.

Boosting

Boosting is one more famous ensemble method.
Boosting uses a slightly different techniques to that of bagging.
Boosting is a well proven theory that works really well on many of the machine learning problems like speech recognition.
If bagging is wisdom of crowds then boosting is wisdom of crowds where each individual is given some weight based on their expertise.
Boosting in general decreases the bias error and builds strong predictive models.
Boosting is an iterative technique. We adjust the weight of the observation based on the previous classification.
If an observation was classified incorrectly, it tries to increase the weight of this observation and vice versa.

Final Classifier $C = \sum α i c i$

Below is the training data and their classes We need to take a note of record numbers, they will help us in weighted sampling later.

Take the dataset.
Build a classifier $C_{m}$ and find the error.
Calculate error rate of the classifier
- Error rate of $ϵ_{m} = \sum w_{i} I (y_{i} \neq C_{m} (x)) / \sum w_{i}$ = Sum of misclassification weight / sum of sample weights
Calculate an intermediate factor called a. It analogous to accuracy rate of the model. It will be later used in weight updating. It is derived from error.
- $\alpha _m = log(1- \epsilon _m)/\epsilon _m$
Update weights of each record in the sample using the a factor. The indicator function will make sure that the misclassifications are given more weight.
- For i =1,2,… N
  - $W_(i+1) = w_i e^(\alpha _m I(y_i\neq C_m (x)))$
  - Re-normalize so that sum of weights is 1.
Repeat this model building and weight update process until we have no misclassification.
Final collation is done by voting from all the modes. While taking the votes, each model is weighted by the accuracy factor α
- $C = sign(\sum \alpha _i C_i(x))$

Ada boosting
- Adaptive Boosting -Till now we discussed Ada boosting technique. Here we give high weight to misclassified records.
Gradient Boosting
- Similar, to Ada boosting algorithm.
- The approach is same but there are slight modifications during re-weighted sampling.
- We update the weights based on misclassification rate and gradient.
- Gradient boosting serves better for some class of problems like regression.

The next post is a practice session on boosting.

Link to the next post : https://statinfer.com/204-7-8-practice-boosting/

17th March 2017