Curriculum Learning

Summary

The paper brings a pedagogical intuition into machine learning: present training examples in a meaningful order, from easy to hard, instead of at random. The authors formalise this as a sequence of training distributions that start concentrated on simple examples, gradually reweight towards harder ones and end at the target distribution. They interpret a curriculum as a kind of continuation method: the learner first optimises a smoothed, easier version of a non-convex objective and then moves towards the real one, which can steer it into better local minima. Experiments cover a toy convex setting, shape recognition and a neural language model whose vocabulary is grown step by step. Even simple two-stage curricula speed up convergence and improve generalisation, with the clearest gains on test data, so the curriculum also acts somewhat like a regulariser. The work coined the term and gave the theoretical framing for ordering training data in deep learning.

How Mova uses it

• math/src/bin/mathsolve.rs, flag --curriculum of the steps model (word-problem solver as a transition system trained by a structured perceptron, math/src/steps.rs): before normal training the model gets 3 epochs on problems needing one operation, then 3 epochs on problems with at most two, then all problems in the usual order.
• The staging is by number of reduce operations in the gold action sequence, combined with the "worked examples from simple to complex" idea from Sweller.
• It is an optional experiment flag, not part of the default training.

Effectiveness in Mova

Measured on SVAMP: 57.3% → 55.7% with the staged curriculum, i.e. no gain (recorded as "zero"). A naive variant that simply sorts all training problems by difficulty dropped to 50.0%, because it breaks perceptron averaging. The flag is therefore off by default.

👨‍🔬💥

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{bengio2009curriculum,
  title = {{Curriculum Learning}},
  author = {Bengio, Yoshua and Louradour, Jérôme and Collobert, Ronan and Weston, Jason},
  booktitle = {ICML 2009},
  year = {2009},
  url = {https://ronan.collobert.com/pub/2009_curriculum_icml.pdf}
}

References

33 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Allgower, E. L., & Georg, K. (1980). Numerical continuation methods. An introduction. Springer-Verlag. find
2. Bengio, Y. (2009). Learning deep architectures for AI. Foundations & Trends in Mach. Learn., to appear. find
3. Bengio, Y., Ducharme, R., & Vincent, P. (2001). A neural probabilistic language model. Adv. Neural Inf. Proc. Sys. 13 (pp. 932–938). find
4. Bengio, Y., Lamblin, P., Popovici, D., & Larochelle, H. (2007). Greedy layer-wise training of deep networks. Adv. Neural Inf. Proc. Sys. 19 (pp. 153–160). find
5. Cohn, D., Ghahramani, Z., & Jordan, M. (1995). Active learning with statistical models. Adv. Neural Inf. Proc. Sys. 7 (pp. 705–712). find
6. Coleman, T., & Wu, Z. (1994). Parallel continuation-based global optimization for molecular conformation and protein folding (Technical Report). Cornell University, Dept. of Computer Science. find
7. Collobert, R., & Weston, J. (2008). A unified architecture for natural language processing: Deep neural networks with multitask learning. find
8. Int. Conf. Mach. Learn. 2008 (pp. 160–167). find
9. Derényi, I., Geszti, T., & Györgyi, G. (1994). Generalization in the programed teaching of a perceptron. Physical Review E, 50, 3192–3200. find
10. Elman, J. L. (1993). Learning and development in neural networks: The importance of starting small. Cognition, 48, 781–799. find
11. Erhan, D., Manzagol, P.-A., Bengio, Y., Bengio, S., & Vincent, P. (2009). The difficulty of training deep architectures and the effect of unsupervised pre-training. AI & Stat.’2009. find
12. Freund, Y., & Haussler, D. (1994). Unsupervised learning of distributions on binary vectors using two layer networks (Technical Report UCSC-CRL-94-25). University of California, Santa Cruz. find
13. Håstad, J., & Goldmann, M. (1991). On the power of small-depth threshold circuits. Computational Complexity, 1, 113–129. find
14. Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006). A fast learning algorithm for deep belief nets. Neural Computation, 18, 1527–1554. find
15. Hinton, G. E., & Salakhutdinov, R. (2006). Reducing the dimensionality of data with neural networks. Science, 313, 504–507. find
16. Krueger, K. A., & Dayan, P. (2009). Flexible shaping: how learning in small steps helps. Cognition, 110, 380–394. find
17. Larochelle, H., Erhan, D., Courville, A., Bergstra, J., & Bengio, Y. (2007). An empirical evaluation of deep architectures on problems with many factors of variation. Int. Conf. Mach. Learn. (pp. 473–480). find
18. Peterson, G. B. (2004). A day of great illumination: B. F. Skinner’s discovery of shaping. Journal of the Experimental Analysis of Behavior, 82, 317–328. find
19. Ranzato, M., Boureau, Y., & LeCun, Y. (2008). Sparse feature learning for deep belief networks. Adv. Neural Inf. Proc. Sys. 20 (pp. 1185–1192). find
20. Ranzato, M., Poultney, C., Chopra, S., & LeCun, Y. (2007). Efficient learning of sparse representations with an energy-based model. Adv. Neural Inf. Proc. Sys. 19 (pp. 1137–1144). find
21. Rohde, D., & Plaut, D. (1999). Language acquisition in the absence of explicit negative evidence: How important is starting small? Cognition, 72, 67–109. find
22. Salakhutdinov, R., & Hinton, G. (2007). Learning a nonlinear embedding by preserving class neighbourhood structure. AI & Stat.’2007. find
23. Salakhutdinov, R., & Hinton, G. (2008). Using Deep Belief Nets to learn covariance kernels for Gaussian processes. Adv. Neural Inf. Proc. Sys. 20 (pp. 1249– 1256). find
24. Salakhutdinov, R., Mnih, A., & Hinton, G. (2007). Restricted Boltzmann machines for collaborative filtering. find
25. Int. Conf. Mach. Learn. 2007 (pp. 791–798). find
26. Sanger, T. D. (1994). Neural network learning control of robot manipulators using gradually increasing task difficulty. IEEE Trans. on Robotics and Automation, 10. find
27. Schwenk, H., & Gauvain, J.-L. (2002). Connectionist language modeling for large vocabulary continuous speech recognition. International Conference on Acoustics, Speech and Signal Processing (pp. 765– 768). Orlando, Florida. find
28. Skinner, B. F. (1958). Reinforcement today. American Psychologist, 13, 94–99. find
29. Thrun, S. (1996). Explanation-based neural network learning: A lifelong learning approach. Boston, MA: Kluwer Academic Publishers. find
30. Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P.-A. (2008). Extracting and composing robust features with denoising autoencoders. Int. Conf. Mach. Learn. (pp. 1096–1103). find
31. Weston, J., Ratle, F., & Collobert, R. (2008). Deep learning via semi-supervised embedding. find
32. Int. Conf. Mach. Learn. 2008 (pp. 1168–1175). find
33. Wu, Z. (1997). Global continuation for distance geometry problems. SIAM Journal of Optimization, 7, 814–836. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/bengio-2009-curriculum-learning/