{ "deckName": "Exam SRM — Decision Trees & Ensembles", "examCode": "Exam SRM", "cards": [ { "front": "Describe **recursive binary splitting** and why it is a *greedy*, *top-down* algorithm.", "back": "Starting from the full predictor space, the algorithm scans every predictor $X_j$ and every cutpoint $s$ and chooses the single split into half-planes $\\{X_j10$ the **smaller** $5$-leaf tree has the lower cost-complexity score (pruning is preferred); for $\\alpha<10$ the larger $8$-leaf tree wins.", "tag": "Pruning" }, { "front": "Two subtrees are candidates at $\\alpha=4$: Tree X has RSS $=200$, $|T|=10$; Tree Y has RSS $=240$, $|T|=4$. Which does cost-complexity pruning select?", "back": "Cost-complexity score $=\\text{RSS}+\\alpha|T|$.\nTree X: $200+4(10)=200+40=240$.\nTree Y: $240+4(4)=240+16=256$.\nTree X's score $240<256$, so **Tree X** is selected at $\\alpha=4$.", "tag": "Pruning" }, { "front": "List the **advantages** and **disadvantages** of a single decision tree versus linear/regression models.", "back": "**Advantages:** very easy to explain and visualize; can be displayed graphically; handle qualitative predictors without dummy coding; mirror human decision-making; naturally capture interactions and non-linear effects.\n**Disadvantages:** generally **lower predictive accuracy** than the best alternatives; **non-robust** — a small data change can produce a very different tree (high variance). Linear regression is preferable when the true relationship is approximately linear.\nEnsembles (bagging, random forests, boosting) recover much of the lost accuracy at the cost of interpretability.", "tag": "Regression trees" }, { "front": "Define **bagging** (bootstrap aggregation) and explain why it reduces variance.", "back": "Bagging fits the same model (a deep, unpruned tree) on $B$ **bootstrap samples** of the training data and aggregates: for regression it **averages** the $B$ predictions $\\hat f_{\\text{bag}}(x)=\\frac{1}{B}\\sum_{b=1}^{B}\\hat f^{*b}(x)$; for classification it takes a **majority vote**.\nAveraging a set of independent quantities each with variance $\\sigma^2$ gives variance $\\sigma^2/B$, so combining many high-variance, low-bias trees cuts variance while leaving bias roughly unchanged. The trees are grown deep and **not pruned**.", "tag": "Bagging & random forests" }, { "front": "What is the **out-of-bag (OOB) error**, and why does it approximate cross-validation \"for free\"?", "back": "Each bagged tree uses a bootstrap sample, so on average about $\\tfrac{1}{3}$ of observations are left out (out-of-bag) for that tree. To get the OOB prediction for observation $i$, average (or vote) only over the trees in which $i$ was **OOB**; the OOB error is the resulting prediction error across all $i$.\nBecause each observation is predicted only by trees that never saw it, the OOB error is a valid estimate of test error — essentially leave-one-out/CV-like accuracy with no extra refitting.", "tag": "Bagging & random forests" }, { "front": "Why is roughly $\\tfrac{2}{3}$ of the data \"in-bag\" and $\\tfrac{1}{3}$ \"out-of-bag\" for each bootstrap tree? Show the limiting probability.", "back": "A bootstrap sample draws $n$ observations with replacement. The probability a specific observation is **not** drawn in one pick is $1-\\frac1n$, so over $n$ picks it is $\\left(1-\\frac1n\\right)^{n}$.\nAs $n\\to\\infty$, $\\left(1-\\frac1n\\right)^{n}\\to e^{-1}\\approx0.368$.\nThus about $36.8\\%$ of observations are **out-of-bag** and about $63.2\\%$ are **in-bag** for any given tree.", "tag": "Bagging & random forests" }, { "front": "With $n=10$, compute the probability that a specific observation is **out-of-bag** for a given bootstrap sample.", "back": "$P(\\text{not drawn in one pick})=1-\\frac{1}{10}=0.9$.\nOver $n=10$ independent picks with replacement: $P(\\text{OOB})=(0.9)^{10}$.\n$(0.9)^{10}\\approx0.3487$.\nThis is already close to the large-sample limit $e^{-1}\\approx0.3679$, confirming roughly a third of observations sit out of each tree.", "tag": "Bagging & random forests" }, { "front": "How does a **random forest** differ from bagging, and what does the choice $m\\approx\\sqrt{p}$ accomplish?", "back": "A random forest is bagging plus an extra randomization: at **each split**, only a fresh random subset of $m$ of the $p$ predictors is considered as split candidates (typically $m\\approx\\sqrt{p}$ for classification, $m\\approx p/3$ for regression).\nRestricting the candidates **decorrelates** the trees — without it, one dominant predictor would be the top split in almost every tree, making the trees highly correlated and limiting variance reduction. Decorrelated trees average to a lower-variance ensemble than bagging.", "tag": "Bagging & random forests" }, { "front": "Why does averaging *correlated* trees reduce variance less, and how does the formula $\\rho\\sigma^2+\\frac{1-\\rho}{B}\\sigma^2$ make the random-forest motivation precise?", "back": "If $B$ trees each have variance $\\sigma^2$ and pairwise correlation $\\rho$, the variance of their average is\n$\\rho\\sigma^{2}+\\frac{1-\\rho}{B}\\sigma^{2}$.\nAs $B\\to\\infty$ the second term vanishes but the first, $\\rho\\sigma^2$, remains — so high correlation $\\rho$ caps the achievable variance reduction. Random forests **lower $\\rho$** by limiting each split to $m0.10$), so $X_1$ is the **more important** predictor.", "tag": "Bagging & random forests" }, { "front": "Describe **boosting** for regression trees and contrast its core idea with bagging.", "back": "Boosting grows trees **sequentially**: each new tree is fit to the **current residuals** of the model rather than to the response $Y$, then a shrunken version is added to the running fit. Trees are typically **small** (few splits), so the model learns slowly and improves where it currently fits poorly.\nUnlike bagging — independent trees on bootstrap samples, combined by averaging to cut *variance* — boosting builds dependent trees on the full data to slowly reduce *bias*. No bootstrapping is used.", "tag": "Boosting" }, { "front": "Write the boosting algorithm for regression trees in terms of $\\hat f$, the residuals, and the shrinkage $\\lambda$.", "back": "1. Set $\\hat f(x)=0$ and residuals $r_i=y_i$ for all $i$.\n2. For $b=1,\\dots,B$: fit a tree $\\hat f^{b}$ with $d$ splits to the data $(X,r)$; update\n$\\hat f(x)\\leftarrow \\hat f(x)+\\lambda\\,\\hat f^{b}(x)$ and $r_i\\leftarrow r_i-\\lambda\\,\\hat f^{b}(x_i)$.\n3. Output $\\hat f(x)=\\sum_{b=1}^{B}\\lambda\\,\\hat f^{b}(x)$.\nThe shrinkage $\\lambda$ (small, e.g. $0.01$–$0.1$) controls the learning rate.", "tag": "Boosting" }, { "front": "Name the three **tuning parameters** of boosting and the danger of getting each wrong.", "back": "**$B$ — number of trees:** boosting *can* overfit if $B$ is too large (unlike bagging/RF, where large $B$ is safe); choose $B$ by cross-validation.\n**$\\lambda$ — shrinkage / learning rate:** small $\\lambda$ (e.g. $0.001$–$0.1$) learns slowly and usually generalizes better, but needs a larger $B$ to reach good fit.\n**$d$ — interaction depth (splits per tree):** controls complexity; $d=1$ gives a \"stump\" (additive model, no interactions), larger $d$ captures higher-order interactions.", "tag": "Boosting" }, { "front": "Why are **small trees** (low $d$, often stumps with $d=1$) effective for boosting, and what does $d$ represent?", "back": "Because boosting fits trees sequentially to residuals, each tree only needs to nudge the fit a little, so a small tree suffices and the ensemble **learns slowly** — which guards against overfitting.\n$d$ is the **interaction depth**: a tree with $d$ splits can involve at most $d$ variables, so $d$ bounds the order of interactions the model can represent. $d=1$ (a stump) yields a purely **additive** model; raising $d$ permits $2$-way, $3$-way, … interactions.", "tag": "Boosting" }, { "front": "Boosting with $\\lambda=0.1$ starts at $\\hat f=0$. The first tree predicts $+8$ for an observation whose true $y=20$. Give the updated prediction and the new residual.", "back": "Update the model by the shrunken tree prediction:\n$\\hat f(x)\\leftarrow 0+\\lambda(8)=0.1(8)=0.8$.\nNew residual $=y-\\hat f(x)=20-0.8=19.2$ (equivalently $r\\leftarrow 20-\\lambda\\hat f^1=20-0.8=19.2$).\nThe model has moved only slightly toward $20$ because of the small learning rate — subsequent trees keep chipping away at the residual.", "tag": "Boosting" }, { "front": "Continuing the boosting example ($\\lambda=0.1$, current $\\hat f=0.8$, residual $19.2$, true $y=20$), the second tree fit to residuals predicts $+15$. Update $\\hat f$ and the residual.", "back": "Add the shrunken second-tree prediction:\n$\\hat f(x)\\leftarrow 0.8+\\lambda(15)=0.8+0.1(15)=0.8+1.5=2.3$.\nNew residual $=20-2.3=17.7$ (i.e. $r\\leftarrow 19.2-0.1(15)=19.2-1.5=17.7$).\nThe fit creeps toward the target; many such small additive steps are what make boosting accurate.", "tag": "Boosting" }, { "front": "Summarize the **bias–variance and interpretability trade-off** across a single tree, bagging, random forests, and boosting.", "back": "**Single tree:** most interpretable; low bias but **high variance** (overfits, non-robust).\n**Bagging:** averages many deep trees to cut **variance**; loses easy interpretability but gains the OOB error estimate.\n**Random forest:** bagging plus $m