Category Artificial Intelligence

Boosting is uniquely sensitive to label noise. The same reweighting mechanism that makes AdaBoost powerful on clean data — assigning higher weights to misclassified examples — causes catastrophic weight concentration on mislabelled examples when noise is present. A single incorrectly…

Most classification problems ask “which one class does this example belong to?” Multi-label classification asks “which subset of labels does this example belong to?” — a fundamentally different task. A news article can be tagged sports, finance, and politics simultaneously.…

Binary boosting — AdaBoost, gradient boosting, XGBoost — is well understood, but most real classification tasks involve more than two classes: digit recognition, product categorisation, disease type diagnosis. Extending boosting to K > 2 classes requires a different loss function…

LightGBM (Light Gradient Boosting Machine) is Microsoft’s open-source gradient boosting framework, designed to train on large datasets orders of magnitude faster than scikit-learn’s GradientBoosting while matching or exceeding its accuracy. It achieves this through two key algorithmic innovations — histogram-based…

XGBoost (eXtreme Gradient Boosting) extends the gradient boosting framework with three engineering advances that make it practical on real business data: second-order gradient statistics for more accurate leaf weights, built-in L1/L2 regularisation to prevent overfitting, and column (feature) subsampling to…

Gradient Boosting Machines (GBMs) generalise the AdaBoost idea from a discrete reweighting algorithm into a continuous optimisation framework that can minimise any differentiable loss function. Instead of reweighting examples, each new tree fits the residuals — the negative gradients of…

The sample reweighting step is AdaBoost’s most distinctive mechanism. After each boosting round, misclassified training examples receive higher weights and correctly classified examples receive lower weights, so the next learner must focus on the current ensemble’s hardest failures. This article…

AdaBoost (Adaptive Boosting) was the first practical implementation of the boosting idea. It builds a strong classifier from an ensemble of decision stumps by sequentially reweighting training examples: correctly classified examples lose weight, misclassified examples gain weight, forcing each successive…

Boosting is the idea that many weak learners — models that are only slightly better than random guessing — can be combined sequentially to form a strong learner. Each model in the sequence focuses on the examples the previous model…