A Distributional and Orthographic Aggregation Model for English Derivational Morphology

What it is about

The task is derived word generation: given a root word and a transformation (for example "someone who runs" from "run" by AGENT), produce the derived word ("runner"). The data are 4,222 training, 905 validation and 905 test triples from NomBank, over four transformations (ADVERB, RESULT, AGENT, NOMINAL). Two models are combined. A sequence-to-sequence model over characters captures orthography, and a distributional model learns the transformation as a map in word-embedding space and finds the nearest candidate. An aggregation model learns to choose between them. The aggregation model reaches 78.0% accuracy against 74.2% for the sequence model alone, and it improves on the earlier open-vocabulary system by 6.3 points. Restricting the decoder to known words (a dictionary constraint, with word frequencies from Google Books n-grams) adds 6.2 points to the sequence model (80.4%), and frequency reranking adds a further 0.8. For NOMINAL, which has several competing endings (-ity, -ness, -ence), the aggregation model cuts the error by 49%. The distributional model is most useful for suffixes that are rare in the training data.

Ideas

• I1. Closed-vocabulary check for en/src/morph.rs derivation: generate a derived form and accept it only if it is in the lexicon or in the open text's counts, so that a plausible but non-existent form ("-ity" on a noun that takes "-ness") is pruned. Report accuracy with and without the constraint.
• I2. Frequency reranking from counts: when two derivations are both possible (-ity against -ness), pick the one that is more frequent in Mova's open text, with the counts written into the explanation. This is a count-based rule, not a neural score.
• I3. Suffix-ambiguity table: for each derivational class, list the competing suffixes with their counts, so that a rule for NOMINAL can say "-ity because the base ends in -al and -ity is 3 times more frequent here".
• I4. Negative control: derive from a random root and check that the constrained output drops to the non-word rate; the constraint must not be pass-through.

For Mova

The chapters are 17. Derivation II: suffixes (suffix choice and competition) and 16. Derivation I: prefixes. The code is en/src/morph.rs (derivation), with a count table built from the open corpora in $MOVA_DATA. Measure: accuracy and edit distance on a held-out set of root-transformation pairs, run twice, with the closed-vocabulary and frequency rules switched on and off. The dataset comes from NomBank and must be checked against its licence before it is used for evaluation; it is not to be carried into a build.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/p18-1180

@inproceedings{deutsch2018distributional,
  title = {{A Distributional and Orthographic Aggregation Model for English Derivational Morphology}},
  author = {Deutsch, Daniel and Hewitt, John and Roth, Dan},
  booktitle = {Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2018},
  doi = {10.18653/v1/p18-1180},
  url = {https://doi.org/10.18653/v1/p18-1180}
}

References

24 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. CoRR abs/1409.0473. ↗
2. Ryan Cotterell, Christo Kirov, John Sylak-Glassman, Géraldine Walther, Ekaterina Vylomova, Patrick Xia, Manaal Faruqui, Sandra Kübler, David Yarowsky, Jason Eisner, and Mans Hulden. 2017a. Conll-sigmorphon 2017 shared task: Universal morphological reinflection in 52 languages. In Proceedings of the CoNLL SIGMORPHON 2017 Shared Task: Universal Morphological Reinflection. Association for Computational Linguistics, Vancouver, pages 1–30. http://www.aclweb.org/anthology/K17-2001. ↗
3. Ryan Cotterell, Christo Kirov, John Sylak-Glassman, David Yarowsky, Jason Eisner, and Mans Hulden. 2016. The sigmorphon 2016 shared taskmorpholog-ical reinflection. ACL 2016 page 10. find
4. Ryan Cotterell, Ekaterina Vylomova, Huda Khayrallah, Christo Kirov, and David Yarowsky. 2017b. Paradigm completion for derivational morphology. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Copenhagen, Denmark, pages 714–720. https://www.aclweb.org/anthology/D17-1074. ↗
5. Greg Durrett and John DeNero. 2013. Supervised learning of complete morphological paradigms. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, Atlanta, Georgia, pages 1185–1195. http://www.aclweb.org/anthology/N13-1138. ↗ our card
6. Manaal Faruqui, Yulia Tsvetkov, Graham Neubig, and Chris Dyer. 2016. Morphological inflection generation using character sequence to sequence learning. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, San Diego, California, pages 634–643. http://www.aclweb.org/anthology/N16-1077. ↗
7. Anna Gladkova, Aleksandr Drozd, and Satoshi Mat-suoka. 2016. Analogy-based detection of morphological and semantic relations with word embeddings: what works and what doesn’t. In Proceedings of the NAACL Student Research Workshop. Association for Computational Linguistics, San Diego, California, pages 8–15. http://www.aclweb.org/anthology/N16-2002. ↗
8. Alex Graves and Jürgen Schmidhuber. 2005. Frame-wise phoneme classification with bidirectional lstm and other neural network architectures. Neural Networks 18(5-6):602–610. find
9. Arihant Gupta, Syed Sarfaraz Akhtar, Avijit Vajpayee, Arjit Srivastava, Madan Gopal Jhanwar, and Manish Shrivastava. 2017. Exploiting morphological regularities in distributional word representations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Copenhagen, Denmark, pages 292–297. https://www.aclweb.org/anthology/D17-1028. ↗
10. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation 9(8):1735– 1780. ↗
11. Liang Huang and David Chiang. 2007. Forest rescoring: Faster decoding with integrated language models. In Proceedings of the 45th Annual Meeting of the Association of Computational Cynthia Gan, Dan Garrette, Yangfeng Ji, Lingpeng Kong, Adhiguna Kuncoro, Gaurav Kumar, Chaitanya Malaviya, Paul Michel, Yusuke Oda, Matthew Richardson, Naomi Saphra, Swabha Swayamdipta, and Pengcheng Yin. 2017. Dynet: The dynamic neural network toolkit. arXiv preprint arXiv:1701.03980 . Linguistics. Association for Computational Linguistics, Prague, Czech Republic, pages 144–151. http://www.aclweb.org/anthology/P07-1019. ↗
12. Mans Hulden, Markus Forsberg, and Malin Ahlberg. 2014. Semi-supervised learning of morphological paradigms and lexicons. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics. Association for Computational Linguistics, Gothenburg, Sweden, pages 569–578. http://www.aclweb.org/anthology/E14-1060. ↗ our card
13. Katharina Kann and Hinrich Schütze. 2016. Single-model encoder-decoder with explicit morphological representation for reinflection. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Association for Computational Linguistics, Berlin, Germany, pages 555–560. http://anthology.aclweb.org/P16-2090. ↗
14. Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Lisbon, Portugal, pages 1412– 1421. http://aclweb.org/anthology/D15-1166. ↗
15. Thang Luong, Richard Socher, and Christopher Manning. 2013. Better word representations with recursive neural networks for morphology. In Proceedings of the Seventeenth Conference on Computational Natural Language Learning. Association for Computational Linguistics, Sofia, Bulgaria, pages 104–113. http://www.aclweb.org/anthology/W13-3512. ↗
16. A. Meyers, R. Reeves, C. Macleod, R. Szekely, V. Zielinska, B. Young, and R. Grishman. 2004. The nombank project: An interim report. In A. Meyers, editor, HLT-NAACL 2004 Workshop: Frontiers in Corpus Annotation. Association for Computational Linguistics, Boston, Massachusetts, USA, pages 24– 31. find
17. Jean-Baptiste Michel, Yuan Kui Shen, Aviva Presser Aiden, Adrian Veres, Matthew K. Gray, Joseph P. Pickett, Dale Hoiberg, Dan Clancy, Peter Norvig, Jon Orwant, Steven Pinker, Martin A. Nowak, and Erez Lieberman Aiden. 2011. Quantitative analysis of culture using millions of digitized books. Science 331 6014:176–82. ↗
18. Garrett Nicolai, Colin Cherry, and Grzegorz Kondrak. 2015. Inflection generation as discriminative string transduction. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, Denver, Colorado, pages 922– 931. http://www.aclweb.org/anthology/N15-1093. ↗
19. Pushpendre Rastogi, Ryan Cotterell, and Jason Eisner. 2016. Weighting finite-state transductions with neural context. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, San Diego, California, pages 623–633. http://www.aclweb.org/anthology/N16-1076. ↗
20. D. Roth and W. Yih. 2004. A linear programming formulation for global inference in natural language tasks. In Hwee Tou Ng and Ellen Riloff, editors, Proc. of the Conference on Computational Natural Language Learning (CoNLL). Association for Computational Linguistics, pages 1–8. http://cogcomp.org/papers/RothYi04.pdf. D. ↗
21. Roth and W. Yih. 2007. Global inference for entity and relation identification via a linear programming formulation http://cogcomp.org/papers/RothYi07.pdf. ↗
22. Wolfgang Seeker and Özlem Çetinoglu. 2015. A graph-based lattice dependency parser for joint morphological segmentation and syntactic analysis. Transactions of the Association for Computational Linguistics 3:359–373. ↗ our card
23. Radu Soricut and Franz Och. 2015. Unsupervised morphology induction using word embeddings. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, Denver, Colorado, pages 1627–1637. http://www.aclweb.org/anthology/N15-1186. ↗
24. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In ICLR Workshop Papers. Scottsdale, Arizona. https://arxiv.org/pdf/1301.3781.pdf. Graham Neubig, Chris Dyer, Yoav Goldberg, Austin Matthews, Waleed Ammar, Antonios Anastasopoulos, Miguel Ballesteros, David Chiang, Daniel Clothiaux, Trevor Cohn, Kevin Duh, Manaal Faruqui, ↗

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/deutsch-2018-distributional-orthographic-aggregation-model-english/