“Boosting improves the power of machine learning models to improve the accuracy of their prediction.” This is a simple and powerful explanation. In this article, I will explain how a boosting algorithm works. It is a superpower which converts a weak Machine learning model to strong learning model.
Let’s understand this definition in detail by solving a problem of spam email identification:
How would you classify an email as SPAM or not? Like everyone else, our initial approach would be to identify ‘spam’ and ‘not spam’ emails using following criteria. If:
- If the email contains only an image? - SPAM
- Email has only Hyperlink(s) - SPAM
- Email body has sentence like “You won a Lottery of $ xxxxxx”, It’s a SPAM
- Email from known source, Not a SPAM
- Email from our official domain “mycompany.com” , Not a SPAM
- How caps lock was used in the email? All CAPS SUBJECT - SPAM
Above, we’ve defined multiple policies to categorise an e-mail into ‘spam’ or ‘not spam’. However in my opinion these rules are not robust enough to correctly classify an email.
Individually, these policies are not powerful enough to classify an e mail into ‘spam’ or ‘no longer spam’. Therefore, these rules are referred to as vulnerable learner. In order to convert a weak learner to strong learner, we’ll combine the prediction of each weak learner using methods like:
Using Weighted Average and considering prediction with higher votes
For example: Above, we have defined 6 weak learners. Out of these, 4 are voted as ‘SPAM’ and 2 are voted as ‘Not a SPAM’. In this case, by default, we’ll consider an email as SPAM because we have higher(4) vote for ‘SPAM’.
The fundamental principle behind the boosting algorithm is to create multiple weak learners and combine their predictions to form a strong rule. Here’s how the algorithm works:
Step 1: The base algorithm reads the data and assigns equal weight to each observation of the sample
Step 2: False predictions made by the weak learner are known. In the next iteration, on those incorrect predictions, these false predictions are assigned to the next weak learner with a higher weight.
Step 3: Repeat step 2 until the algorithm can correctly classify the output.
Therefore, the main aim of Boosting is to focus more on miss-classified predictions.
Powerful boosting algorithm include:
- Gradient Boosting Machine
- Extreme Gradient Boosting Machine
I will be covering each method in detail in my future Articles. Hope you enjoyed reading this. Hit Like if you gained a new understanding and liked reading.
If you found this Article interesting, why not review the other Articles in our archive.