TinyStories: How Small Can Language Models Be and Still Speak Coherent English?

What it is about

The paper introduces TinyStories, a synthetic corpus of short stories written by GPT-3.5 and GPT-4 with only words that a 3–4-year-old would know (about 1,500 basic words). Each story is generated from three random words (noun, verb, adjective) and a random set of features such as dialogue or a twist, to force diversity. Small GPT-Neo models trained on it, below 10M parameters or with a single transformer block, write fluent multi-paragraph stories with almost perfect grammar. An evaluation framework asks GPT-4 to grade completions on grammar, creativity, consistency and plot. Grammar appears first in small models; consistency and creativity need larger hidden size; one-layer models fail at following instructions. A 28M-parameter model with eight layers already beats the 1.5B GPT2-XL on some of the paper's factual, reasoning and consistency prompts. Training takes under 30 hours on one V100 GPU. Memorisation is ruled out by n-gram overlap with the training set.

Ideas

I1. A closed-vocabulary parse test: generate sentences only from the child-vocabulary word list, and measure LAS and UAS on that subset with en parse-eval. A closed vocabulary separates syntax from lexical coverage.
I2. Score Mova's school answers on separate axes, like the paper's rubric, so that grammar and consistency gains are reported apart. Keep the counters in world/src/knowledge.rs.
I3. An entity-tracking test: story openings where the continuation must keep names, places and objects. Check in world/src/story.rs that a mention after a gap refers to the same entity. The paper's one-layer failures show where the hard cases are.
I4. Overlap check for any generated text: report the n-gram overlap with the lesson corpus next to each output, as the paper does before claiming diversity.

For Mova

Chapters: 19. The lexicon: vocabulary, multiword expressions and register (the basic vocabulary) and 35. Reference and coreference (entity tracking). Code: en/src/parse.rs, world/src/story.rs, world/src/knowledge.rs. Measure: closed-vocabulary LAS/UAS and entity-tracking accuracy, run twice.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. arXiv 2305.07759

@misc{eldan2023tinystories,
  title = {{TinyStories: How Small Can Language Models Be and Still Speak Coherent English?}},
  author = {Eldan, Ronen and Li, Yuanzhi},
  year = {2023},
  eprint = {2305.07759},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.07759}
}

References

33 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

Common crawl. Accessed: 2019. find
Zeyuan Allen-Zhu and Yuanzhi Li. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. arXiv preprint arXiv:2012.09816, 2020. ↗
Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, March 2021. find
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. find
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712, 2023. ↗
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. What does bert look at? an analysis of bert’s attention. arXiv preprint arXiv:1906.04341, 2019. ↗
John H Flavell, Eleanor R Flavell, Frances L Green, and Louis J Moses. Young children’s understanding of fact beliefs versus value beliefs. Child development, 61(4):915–928, 1990. ↗
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03635, 2018. ↗
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020. ↗
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015. ↗
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022. ↗
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751, 2019. ↗
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Quantized neural networks: Training neural networks with low precision weights and activations. The Journal of Machine Learning Research, 18(1):6869–6898, 2017. ↗
Samy Jelassi, Michael Sander, and Yuanzhi Li. Vision transformers provably learn spatial structure. Advances in Neural Information Processing Systems, 35:37822–37836, 2022. ↗
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551, 2017. ↗
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020. ↗
Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In Thirteenth international conference on the principles of knowledge representation and reasoning, 2012. find
Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky. Visualizing and understanding neural models in nlp. arXiv preprint arXiv:1506.01066, 2015. ↗
Yuchen Li, Yuanzhi Li, and Andrej Risteski. How do transformers learn topic structure: Towards a mechanistic understanding. arXiv preprint arXiv:2303.04245, 2023. ↗
Wick Miller and Susan Ervin. The development of grammar in child language. Monographs of the Society for Research in Child Development, pages 9–34, 1964. ↗
OpenAI. Gpt-4 technical report, 2023. find
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring a broad discourse context. arXiv preprint arXiv:1606.06031, 2016. ↗
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019. find
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485–5551, 2020. ↗
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019. ↗
Michael Santacroce, Zixin Wen, Yelong Shen, and Yuanzhi Li. What matters in the structured pruning of generative language models? arXiv preprint arXiv:2302.03773, 2023. ↗
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615, 2022. ↗
Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. Mobilebert: a compact task-agnostic bert for resource-limited devices. arXiv preprint arXiv:2004.02984, 2020. ↗
Wilson L Taylor. “cloze procedure”: A new tool for measuring readability. Journalism quarterly, 30(4):415–433, 1953. find
Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. arXiv preprint arXiv:1905.09418, 2019. ↗
Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. Ccnet: Extracting high quality monolingual datasets from web crawl data. arXiv preprint arXiv:1911.00359, 2019. ↗
Terry Winograd. Understanding natural language. Cognitive psychology, 3(1):1–191, 1972. ↗
Victor Zhong, Caiming Xiong, and Richard Socher. Seq2sql: Generating structured queries from natural language using reinforcement learning. arXiv preprint arXiv:1709.00103, 2017. ↗

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/eldan-2023-tinystories-small-language-models-still/