RobertNLP at the IWPT 2020 Shared Task: Surprisingly Simple Enhanced UD Parsing for English

What it is about

RobertNLP builds enhanced UD graphs directly, without a basic tree first. A biaffine classifier over fine-tuned RoBERTa embeddings scores every ordered pair of tokens with a label or with "no relation". Labels that carry words (such as obl:instead_of) are replaced by placeholders, for example obl:[case], which cuts the English label set from 399 to 56, and rules put the words back afterwards. A few heuristics add heads to nodes that have none (about 1% of sentences), with a UDify basic parse as the fallback for reachability. The system scores ELAS 88.94 on the English blind test, first among ten systems. The paper also shows that adding training treebanks lowers the score in proportion to their lexical distance from the test data, and that gold tokenization would add about two points (90.80), so segmentation is a large source of loss.

Ideas

• I1. gparse: a delexicalized label set for the enhanced layer (placeholders for case-marked obl and nmod, mark-marked advcl and acl, cc-marked conj), with the relexicalization rules written in Rust, measured by ELAS on the derived layer of EWT dev.
• I2. Before a corpus is added to gparse training, compute the Bhattacharyya distance between its vocabulary and the test data (the paper gives 0.142 for EWT and 0.204 for GUM), and add it only if the distance is small; measure on dev.
• I3. When gparse outputs a graph with unreachable nodes, repair it with the link from the en arc-standard tree, and count repaired sentences per 1k.
• I4. Measure gparse LAS on gold tokenization against en-tokenized input on dev, to size the segmentation loss (about 2 points in the paper) before investing in en/src/tok.rs.

For Mova

The paper's gains come from a pretrained encoder, which Mova does not use (no foreign pretrained weights), so the transferable part is the architecture: the biaffine scorer and the delexicalized labels in gparse, and the validity gates in tools/treeagree. The ideas touch 20. Syntax: the Universal Dependencies frame, 22-phrases.md (multiword case expressions) and 24-coordination-subordination.md. I4 is the cheapest first measurement.

Topics: Universal Dependencies

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.iwpt-1.26

@inproceedings{grunewald2020robertnlp,
  title = {{RobertNLP at the IWPT 2020 Shared Task: Surprisingly Simple Enhanced UD Parsing for English}},
  author = {Grünewald, Stefan and Friedrich, Annemarie},
  booktitle = {Proceedings of the 16th International Conference on Parsing Technologies and the IWPT 2020 Shared Task on Parsing into Enhanced Universal Dependencies},
  year = {2020},
  doi = {10.18653/v1/2020.iwpt-1.26},
  url = {https://doi.org/10.18653/v1/2020.iwpt-1.26}
}

References

20 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Anil Bhattacharya. 1943. On a measure of divergence between two statistical populations defined by their population distributions. In Bulletin of the Calcutta Mathematical Society, volume 35, pages 99–109. find
2. Gosse Bouma, Djamé Seddah, and Daniel Zeman. 2020. Overview of the IWPT 2020 Shared Task on Parsing into Enhanced Universal Dependencies. In Proceedings of the 16th International Conference on Parsing Technologies and the IWPT 2020 Shared Task on Parsing into Enhanced Universal Dependencies, Seattle, US. Association for Computational Linguistics. find our card
3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
4. Timothy Dozat and Christopher D. Manning. 2017. Deep biaffine attention for neural dependency parsing. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. Open-Review.net. find our card
5. Timothy Dozat and Christopher D. Manning. 2018. Simpler but more accurate semantic dependency parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 484–490, Melbourne, Australia. Association for Computational Linguistics. find
6. Nicole Gregoire, Stefan Evert, and Su Nam Kim, editors. 2007. Proceedings of the Workshop on A Broader Perspective on Multiword Expressions. Association for Computational Linguistics, Prague, Czech Republic. find
7. Dan Kondratyuk and Milan Straka. 2019. 75 languages, 1 model: Parsing universal dependencies universally. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2779–2795, Hong Kong, China. Association for Computational Linguistics. find our card
8. Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. 2019a. Linguistic knowledge and transferability of contextual representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1073–1094, Minneapolis, Minnesota. Association for Computational Linguistics. find
9. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
10. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations. find
11. Marie-Catherine de Marneffe, Timothy Dozat, Natalia Silveira, Katri Haverinen, Filip Ginter, Joakim Nivre, and Christopher D. Manning. 2014. Universal Stanford dependencies: A cross-linguistic typology. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 4585–4592, Reykjavik, Iceland. European Language Resources Association (ELRA). find our card
12. Joakim Nivre, Paola Marongiu, Filip Ginter, Jenna Kanerva, Simonetta Montemagni, Sebastian Schuster, and Maria Simi. 2018. Enhancing universal dependency treebanks: A case study. In Proceedings of the Second Workshop on Universal Dependencies (UDW 2018), pages 102–107, Brussels, Belgium. Association for Computational Linguistics. find our card
13. Jenna Nyblom, Samuel Kohonen, Katri Haverinen, Tapio Salakoski, and Filip Ginter. 2013. Predicting conjunct propagation and other extended Stanford dependencies. In Proceedings of the Second International Conference on Dependency Linguistics (DepLing 2013), pages 252–261, Prague, Czech Republic. Charles University in Prague, Matfyzpress, Prague, Czech Republic. find our card
14. Peng Qi, Timothy Dozat, Yuhao Zhang, and Christopher D. Manning. 2018. Universal dependency parsing from scratch. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 160–170, Brussels, Belgium. Association for Computational Linguistics. find our card
15. Sebastian Ruder and Barbara Plank. 2017. Learning to select data for transfer learning with Bayesian optimization. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 372–382, Copenhagen, Denmark. Association for Computational Linguistics. find
16. Agata Savary, Carla Parra Escartín, Francis Bond, Jelena Mitrović, and Verginica Barbu Mititelu, editors. 2019. Proceedings of the Joint Workshop on Multiword Expressions and WordNet (MWE-WN 2019). Association for Computational Linguistics, Florence, Italy. find
17. Agata Savary, Carlos Ramisch, Jena D. Hwang, Nathan Schneider, Melanie Andresen, Sameer Pradhan, and Miriam R. L. Petruck, editors. 2018. Proceedings of the Joint Workshop on Linguistic Annotation, Multiword Expressions and Constructions (LAW-MWE-CxG-2018). Association for Computational Linguistics, Santa Fe, New Mexico, USA. find
18. Sebastian Schuster, Éric Villemonte de La Clergerie, Marie Candito, Benoît Sagot, Christopher Manning, and Djamé Seddah. 2017. Paris and Stanford at EPE 2017: Downstream evaluation of graph-based dependency representations. In Proceedings of the 2017 Shared Task on Extrinsic Parser Evaluation (EPE 2017), pages 47–59. find
19. Sebastian Schuster and Christopher D. Manning. 2016. Enhanced English universal dependencies: An improved representation for natural language understanding tasks. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 2371–2378, Portorož, Slovenia. European Language Resources Association (ELRA). find our card
20. Maria Simi and Simonetta Montemagni. 2018. Bootstrapping enhanced universal dependencies for Italian. In 5th Italian Conference on Computational Linguistics, CLiC-it 2018, volume 2253. CEUR-WS. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/ud-2020-grunewald-robertnlp-iwpt-2020-shared-task/