{
 "nbformat": 4,
 "nbformat_minor": 5,
 "metadata": {"kernelspec": {"display_name": "Python 3", "language": "python", "name": "python3"}, "language_info": {"name": "python", "version": "3.10.0"}},
 "cells": [
  {"cell_type":"markdown","id":"c01","metadata":{},"source":["# Feature Importance in Random Forest: What It Means and What It Misses\n\nCompares MDI (built-in) vs Permutation Importance on clean data, high-cardinality injected features, and correlated feature pairs. Measures top-k precision against known ground truth on a controlled dataset."]},
  {"cell_type":"code","execution_count":null,"id":"c02","metadata":{},"outputs":[],"source":["import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport warnings; warnings.filterwarnings('ignore')\nnp.random.seed(42)\nimport sklearn\nprint(f'sklearn {sklearn.__version__}')"]},
  {"cell_type":"markdown","id":"c03","metadata":{},"source":["## 1. Load Data"]},
  {"cell_type":"code","execution_count":null,"id":"c04","metadata":{},"outputs":[],"source":["# https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_breast_cancer.html\n# https://scikit-learn.org/stable/modules/generated/sklearn.datasets.make_classification.html\nfrom sklearn.datasets import load_breast_cancer, make_classification\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.ensemble import RandomForestClassifier\nfrom sklearn.inspection import permutation_importance\nfrom sklearn.metrics import accuracy_score\n\nbc = load_breast_cancer()\nX, y = bc.data, bc.target\nfeature_names = bc.feature_names\nX_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, random_state=42, stratify=y)\n\n# Controlled: 20 features, 8 informative, 12 noise — ground truth known\nXc, yc = make_classification(n_samples=2000, n_features=20, n_informative=8,\n                              n_redundant=0, n_repeated=0, n_clusters_per_class=1,\n                              random_state=42)\nXc_tr, Xc_te, yc_tr, yc_te = train_test_split(Xc, yc, test_size=0.25, random_state=42, stratify=yc)\nprint(f'Breast Cancer: {X_tr.shape}, Controlled: {Xc_tr.shape}')"]},
  {"cell_type":"markdown","id":"c05","metadata":{},"source":["## 2. Baseline — MDI vs Permutation Importance (Breast Cancer)"]},
  {"cell_type":"code","execution_count":null,"id":"c06","metadata":{},"outputs":[],"source":["rf = RandomForestClassifier(n_estimators=100, random_state=42, n_jobs=-1)\nrf.fit(X_tr, y_tr)\n\nmdi = rf.feature_importances_\npi_res = permutation_importance(rf, X_te, y_te, n_repeats=10, random_state=42, n_jobs=-1)\npi_mean = pi_res.importances_mean\npi_std  = pi_res.importances_std\n\nfrom scipy.stats import spearmanr\nrho, _ = spearmanr(mdi, pi_mean)\nprint(f'Test accuracy: {accuracy_score(y_te, rf.predict(X_te)):.4f}')\nprint(f'MDI vs PI Spearman rank correlation: {rho:.4f}')"]},
  {"cell_type":"markdown","id":"c07","metadata":{},"source":["## 3. Side-by-Side Bar Chart — MDI vs PI"]},
  {"cell_type":"code","execution_count":null,"id":"c08","metadata":{},"outputs":[],"source":["order_mdi = np.argsort(mdi)[::-1][:15]\norder_pi  = np.argsort(pi_mean)[::-1][:15]\n\nfig, (ax1, ax2) = plt.subplots(1, 2, figsize=(16, 6))\n\n# MDI\nax1.barh(range(15), mdi[order_mdi], color='#6366f1', alpha=0.85)\nax1.set_yticks(range(15))\nax1.set_yticklabels([feature_names[i] for i in order_mdi], fontsize=8)\nax1.invert_yaxis()\nax1.set_title('MDI (Mean Decrease in Impurity)\\nTop 15 Features')\nax1.set_xlabel('Importance')\n\n# Permutation importance with error bars\nax2.barh(range(15), pi_mean[order_pi], xerr=pi_std[order_pi],\n          color='#22c55e', alpha=0.85, error_kw={'linewidth':1.2})\nax2.set_yticks(range(15))\nax2.set_yticklabels([feature_names[i] for i in order_pi], fontsize=8)\nax2.invert_yaxis()\nax2.set_title('Permutation Importance (test set)\\nTop 15 Features ± std')\nax2.set_xlabel('Mean Accuracy Drop')\n\nplt.suptitle('MDI vs Permutation Importance — Breast Cancer', fontsize=13)\nplt.tight_layout(); plt.show()"]},
  {"cell_type":"markdown","id":"c09","metadata":{},"source":["## 4. High-Cardinality Bias — ID Column Experiment"]},
  {"cell_type":"code","execution_count":null,"id":"c10","metadata":{},"outputs":[],"source":["# Add a random unique-integer 'ID' column — no predictive value\nrng = np.random.RandomState(99)\nX_id_tr = np.hstack([X_tr, rng.randint(0, 100000, (X_tr.shape[0], 1))])\nX_id_te = np.hstack([X_te, rng.randint(0, 100000, (X_te.shape[0], 1))])\nnames_id = list(feature_names) + ['RANDOM_ID']\n\nrf_id = RandomForestClassifier(n_estimators=100, random_state=42, n_jobs=-1)\nrf_id.fit(X_id_tr, y_tr)\n\nmdi_id = rf_id.feature_importances_\npi_id  = permutation_importance(rf_id, X_id_te, y_te, n_repeats=10, random_state=42, n_jobs=-1)\n\nmdi_rank_id = np.argsort(mdi_id)[::-1]\npi_rank_id  = np.argsort(pi_id.importances_mean)[::-1]\nprint(f'RANDOM_ID MDI rank:  {np.where(mdi_rank_id==30)[0][0]+1} / 31')\nprint(f'RANDOM_ID PI rank:   {np.where(pi_rank_id==30)[0][0]+1} / 31')\nprint(f'RANDOM_ID MDI score: {mdi_id[30]:.5f}')\nprint(f'RANDOM_ID PI score:  {pi_id.importances_mean[30]:.5f}')"]},
  {"cell_type":"markdown","id":"c11","metadata":{},"source":["## 5. Correlation Bias — Duplicate Feature Experiment"]},
  {"cell_type":"code","execution_count":null,"id":"c12","metadata":{},"outputs":[],"source":["# Duplicate the most important feature\ntop_feat = np.argmax(rf.feature_importances_)\nX_dup_tr = np.hstack([X_tr, X_tr[:, [top_feat]]])\nX_dup_te = np.hstack([X_te, X_te[:, [top_feat]]])\n\nrf_dup = RandomForestClassifier(n_estimators=100, random_state=42, n_jobs=-1)\nrf_dup.fit(X_dup_tr, y_tr)\n\nmdi_dup  = rf_dup.feature_importances_\npi_dup   = permutation_importance(rf_dup, X_dup_te, y_te, n_repeats=10, random_state=42, n_jobs=-1)\n\nprint(f'Original top feature MDI (solo):     {rf.feature_importances_[top_feat]:.5f}')\nprint(f'Original copy MDI (with duplicate):  {mdi_dup[top_feat]:.5f}')\nprint(f'Duplicate copy MDI:                  {mdi_dup[-1]:.5f}')\nprint(f'Sum (should ≈ solo):                 {mdi_dup[top_feat]+mdi_dup[-1]:.5f}')\nprint()\nprint(f'Original copy PI (with duplicate):   {pi_dup.importances_mean[top_feat]:.5f}')\nprint(f'Duplicate copy PI:                   {pi_dup.importances_mean[-1]:.5f}')"]},
  {"cell_type":"markdown","id":"c13","metadata":{},"source":["## 6. Controlled Dataset — Top-k Precision (Ground Truth Known)"]},
  {"cell_type":"code","execution_count":null,"id":"c14","metadata":{},"outputs":[],"source":["# First 8 features are informative (by make_classification convention)\ntrue_informative = set(range(8))\n\nrfc = RandomForestClassifier(n_estimators=100, random_state=42, n_jobs=-1)\nrfc.fit(Xc_tr, yc_tr)\n\nmdi_c = rfc.feature_importances_\npi_c  = permutation_importance(rfc, Xc_te, yc_te, n_repeats=10, random_state=42, n_jobs=-1)\n\ndef topk_precision(importances, k, true_set):\n    top_k = set(np.argsort(importances)[::-1][:k])\n    return len(top_k & true_set) / k\n\nprint('Top-k precision (fraction of truly informative features in top-k):')\nprint(f'  k  | MDI   | PI')\nfor k in [4, 6, 8, 10, 12]:\n    p_mdi = topk_precision(mdi_c, k, true_informative)\n    p_pi  = topk_precision(pi_c.importances_mean, k, true_informative)\n    print(f'  {k:2d} | {p_mdi:.3f} | {p_pi:.3f}')"]},
  {"cell_type":"markdown","id":"c15","metadata":{},"source":["## 7. Visualise MDI vs PI on Controlled Dataset"]},
  {"cell_type":"code","execution_count":null,"id":"c16","metadata":{},"outputs":[],"source":["fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(14, 5))\ncolors_ctrl = ['#22c55e' if i < 8 else '#ef4444' for i in range(20)]\n\norder_c_mdi = np.argsort(mdi_c)[::-1]\ncolors_ord_mdi = [colors_ctrl[i] for i in order_c_mdi]\nax1.bar(range(20), mdi_c[order_c_mdi], color=colors_ord_mdi, alpha=0.85, edgecolor='k')\nax1.set_title('MDI — Controlled dataset\\n(green=informative, red=noise)')\nax1.set_xlabel('Feature rank by MDI'); ax1.set_ylabel('MDI score')\n\norder_c_pi = np.argsort(pi_c.importances_mean)[::-1]\ncolors_ord_pi = [colors_ctrl[i] for i in order_c_pi]\nax2.bar(range(20), pi_c.importances_mean[order_c_pi], color=colors_ord_pi, alpha=0.85, edgecolor='k',\n         yerr=pi_c.importances_std[order_c_pi], error_kw={'linewidth':1})\nax2.set_title('Permutation Importance — Controlled dataset\\n(green=informative, red=noise)')\nax2.set_xlabel('Feature rank by PI'); ax2.set_ylabel('Mean accuracy drop')\n\nplt.suptitle('Green = truly informative features; Red = noise features', fontsize=12)\nplt.tight_layout(); plt.show()"]},
  {"cell_type":"markdown","id":"c17","metadata":{},"source":["## 8. Discussion\n\n1. **MDI and PI agree on clean data.** When features have similar cardinality and low correlation, Spearman rank correlation ≈ 0.85+. Discrepancies appear where MDI's known biases operate.\n\n2. **MDI severely inflates high-cardinality features.** The RANDOM_ID column ranks #1 by MDI despite having zero predictive value. PI correctly assigns it near-zero importance. Never use MDI on datasets with ID-like or high-cardinality columns.\n\n3. **MDI splits importance between correlated duplicates.** When a feature is duplicated, MDI splits its importance score approximately in half per copy. PI behaves differently: dropping either copy hurts the model (because the other copy contains the same information), so PI reports both as important — a different but arguably correct view.\n\n4. **PI error bars distinguish reliable from unreliable rankings.** Features with high PI std relative to mean are not reliably important. Report std alongside mean for any feature-selection decision.\n\n5. **Neither method is causal.** Both measure association, not causation. A correlated proxy feature can rank as important even if it has no direct causal relationship to the target."]},
  {"cell_type":"markdown","id":"c18","metadata":{},"source":["## 9. Next Steps\n\n- **Article 23: Bagging vs Boosting** — when to choose variance-reduction vs bias-reduction ensembles\n- **Part 4: Combination Methods** — stacking and voting, where feature importances across base learners can be compared\n- **SHAP values** — per-sample importance attribution that avoids MDI bias without PI's correlation-splitting"]}
 ]
}