Home

Bagging and boosting are the two fundamental ensemble families, but they solve different problems. Bagging reduces variance by averaging many high-variance models trained on different bootstrap samples. Boosting reduces bias by sequentially training models to correct each other’s mistakes. Knowing…

Random Forest’s built-in feature importances are one of its most widely used outputs — and one of its most widely misunderstood. The default importance score, Mean Decrease in Impurity (MDI), is fast and free but biased toward high-cardinality and correlated…

Random Forest works well out of the box, but “well” is not the same as “optimal.” The default hyperparameters were chosen to work reasonably across hundreds of benchmark datasets, not to maximise performance on your specific problem. The gap between…

A Random Forest is not simply a collection of trees trained on the same data — it is a carefully engineered diversity machine. Its accuracy advantage over a single tree comes almost entirely from the diversity of its constituent trees:…

The Random Subspace Method (RSM) is a diversity-generating technique that creates an ensemble by training each base learner on a different random subset of features — not a different sample of rows. Where bagging varies the training examples, RSM varies…

Random Forest extends bagging with one additional idea: at every split in every tree, only a random subset of features is considered as candidates. This feature subsampling decorrelates the trees far more than bootstrap sampling alone, driving the inter-tree correlation…

Bagging reduces variance by averaging predictions from many bootstrap-trained models. But variance reduction only improves generalisation if the base learner actually has high variance to begin with. A logistic regression model trained on two different 80% subsets of your data…

A single decision tree trained on your full dataset is highly sensitive to the training data: change a few rows and the tree changes completely. Bagging — Bootstrap Aggregating — fixes this by training many trees on independently sampled versions…

Training a 1,000-round AdaBoost model to zero training error sounds like a recipe for catastrophic overfitting. Classical statistical learning theory — which ties generalisation error to model complexity — would predict a massive generalisation gap. But empirically, AdaBoost trained to…