Bagging Predictors

Summary

Bagging ("bootstrap aggregating") builds several versions of a predictor by training each on a bootstrap resample of the training set and then combines them, by voting for classification or averaging for regression. The paper argues that the gain comes from reducing variance: learners whose output changes a lot with small perturbations of the data, such as decision trees or subset selection in regression, benefit most, whereas stable learners such as nearest neighbours gain little or nothing. Experiments on real and simulated datasets show substantial error reductions for unstable methods. The idea became a foundation of later ensemble methods such as random forests.

How Mova uses it

• math/src/bin/mathsolve.rs (--bag) — for the arithmetic word-problem solver, two extra beam-search perceptron models are trained with reversed and with shuffled training order (instead of bootstrap resamples), and their top-K candidate solutions are pooled for the reranking judge.
• --bag-judge — a variant where the judge is trained on the merged candidates of the three models with a "votes" feature (how many models proposed the answer).

Effectiveness in Mova

Measured on SVAMP on the idea test bench (architecture notes, table of ideas from papers):

• --bag: the correct answer appears among the candidates more often (258 → 265), but accuracy is unchanged (57,3%).
• --bag-judge: 57,3% → 56,0%, worse. Models with the same features make the same mistakes, so agreement says little; an ensemble needs genuinely different models (tree, world model, DAG), not one perceptron trained in different orders. Not adopted by default.

👨‍🔬💥

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.1007/bf00058655

@inproceedings{breiman1996bagging,
  title = {{Bagging Predictors}},
  author = {Breiman, Leo},
  booktitle = {Machine Learning},
  year = {1996},
  doi = {10.1007/bf00058655},
  url = {https://doi.org/10.1007/bf00058655}
}

References

16 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Belsley, D., Kuh, E.. & Welsch, R. (1980). "Regression Diagnostics", John Wiley and Sons. find
2. Breiman, L. (1994). Heuristics of instability in model selection, Technical Report, Statistics Department, University of California at Berkeley (to appear, Annals of Statistics). find
3. Breiman, L , Friedman, J., Olshen. R., & Stone, C. (1984). "Classification and Regression Trees", Wadsworth. find
4. Breiman, L. & Friedman, J. (1985). Estimating optimal transformations in multiple regression and correlation (with discussion), Journal of the American Statistical Association, 80, 580-619 . find
5. Breiman, L. & Spector, P (1992). Submodel Selection and Evaluation in Regression - the X-Random Case, International Statistical Review, 3, 291-319 */~ + B untine, W. (1991). "Learning classification trees", Artificial Intelligence Frontiers in Statistics, ed D.J. Hand, Chapman and Hall, London, 182-… find
6. Dietterich, T.G. & Bakiri, G. (1991). Error-correcting output codes: A general method for improving multiclass inductive learning programs, Proceedings of the Ninth National Conference on Artificial Intelligence (AAAI-91 ), Anaheim, CA: AAA1 Press. find
7. Efron, B., & Tibshirani, R. (1993). "An Introduction to the Bootstrap". Chapman and Hall. find
8. Friedman, J. (1991). Multivariate adaptive regression splines (with discussion), Annals of Statistics, 19, 1-141. find
9. Heath, D., KasiC S., & Salzberg, S. (1993). k-dt: a multi-tree learning method. Proceedings of the Second International Workshop on Multi strategy Learning, 1002-1007, Chambery, France, Morgan Kaufman. find
10. Kwok, S., & Carter, C. (1990). Multiple decision trees, Uncertainty in Artificial Intelligence 4, ed. Shachter, R., Levitt, T, Kanal, L., and Lemrner, J., North-Holland, 327-335. find
11. Michie, D., Spiegelhalter, D.J. & Taylor, C.C. (19~4). Machine Learning, Neural and Statistical Classification. Ellis Horwood Limited. find
12. Olshen, R., Gilpin, A., Henning, H., LeWinter, M., Collins, D., & Ross, J. (1985). Twelve-month prognosis following myocardial infarction: Classification trees, logistic regression, and stepwise linear discrimination. Proceedings of the Berkeley conference in honor of Jerzy Neyman and Jack Kiefer, L… find
13. Smith, J., Everhart, J., Dickson, W., Knowler, W., & Johannes, R. (1988). Using the ADAP learning algorithm to forecast the onset of diabetes mellitus. In Proceedings of the Symposium on Computer Applications and Medical Care 261-265. IEEE Computer Society Press. find
14. Sigillito, V. G., Wing, S. P., Hutton, L. V., & Baker, K. B. (1989). Classification of radar returns from the ionosphere using neural networks. Johns Hopkins APL Technical Digest, 10, 262-266. find
15. Wolberg, W. & Mangasarian, O (1990). Multisurface method of pattern separation for medical diagnosis applied to breast cytology, Proceedings of the National Academy of Sciences, U.S.A., Volume 87, December 1990, pp 9193-9196. find
16. Received September 19, 1994 Accepted January 2, 1995 Final Manuscript November 21, 1995 find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/breiman-1996-bagging-predictors/