UDPipe 2.0 Prototype at CoNLL 2018 UD Shared Task

What it is about

The paper describes the UDPipe 2.0 prototype that Straka entered in the CoNLL 2018 shared task (raw text to Universal Dependencies). One neural network predicts UPOS, XPOS and UFeats tags, a lemma rule and the dependency tree, all over shared contextual embeddings from bidirectional RNNs; tokenisation and sentence splitting are reused from UDPipe 1.2. Training uses only the CoNLL-U treebank and pretrained word embeddings, with the embedding source and the tight or loose joint variant chosen per treebank on development data. Of 26 systems the prototype ranked first in MLAS, third in LAS and third in BLEX, and first overall in the extrinsic evaluation EPE 2018. The average model is about 139 MB, roughly ten times UDPipe 1.2. A later refinement, a direct path from character embeddings to the lemma classifier, added 0.68 points of lemma accuracy.

Ideas

• I1. gparse: choose the shared-encoder regime by treebank size (share the contextual encoder for large treebanks, keep separate encoders for small ones) and compare LAS on the EWT dev folds (cv) with the current setup.
• I2. gparse training regularisers from the paper, tested one at a time: label smoothing 0.03 on every softmax head, dropout 0.5 on embeddings and hidden layers (not on recurrent state), and replacing 20 % of trained word embeddings by an unknown-word vector; measure LAS on EWT dev, twice, to confirm determinism.
• I3. en/src/tok.rs: store the exact spacing of the raw text in MISC (SpacesBefore, SpacesAfter, TokenOffset, as UDPipe does) and measure the round-trip rate of raw text through tokenisation and back on EWT dev.

For Mova

The ideas touch gparse (the shared encoder, the training regularisers) and the tokeniser en/src/tok.rs. The chapters that matter are languages/en/07-punctuation-capitals-numbers (contractions and spacing) and languages/en/20-syntax-overview. Measure first I2 on EWT dev, because it changes only training and needs no new data. The round-trip rate in I3 is a cheap check that can run on CPU.

Topics: Universal Dependencies · Treebanks

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/k18-2020

@inproceedings{straka2018udpipe,
  title = {{UDPipe 2.0 Prototype at CoNLL 2018 UD Shared Task}},
  author = {Straka, Milan},
  booktitle = {Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies},
  year = {2018},
  doi = {10.18653/v1/k18-2020},
  url = {https://doi.org/10.18653/v1/k18-2020}
}

References

29 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Daniel Andor, Chris Alberti, David Weiss, Aliaksei Severyn, Alessandro Presta, Kuzman Ganchev, Slav Petrov, and Michael Collins. 2016. Glob-aly normalized transition-based neural networks. In Association for Computational Linguistic. http://arxiv.org/abs/1603.06042. ↗
2. Danqi Chen and Christopher Manning. 2014. A fast and accurate dependency parser using neural networks. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Wang Ling, Tiago Luı́s, Luı́s Marujo, Ramón Fernandez Astudillo, Silvio Amir, Chris Dyer, Alan W. Black, and Isabel T… find
3. Processing (EMNLP). Association for Computational Linguistics, Doha, Qatar, pages 740–750. http://www.aclweb.org/anthology/D14-1082. ↗ our card
4. KyungHyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. On the properties of neural machine translation: Encoder-decoder approaches. CoRR abs/1409.1259. http://arxiv.org/abs/1409.1259. ↗
5. Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings… find
6. Y. J. Chu and T. H. Liu. 1965. On the shortest arborescence of a directed graph. Science Sinica 14. find
7. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. The Journal of Machine Learning Research 12:2493–2537. find
8. Timothy Dozat and Christopher D. Manning. 2016. Deep Biaffine Attention for Neural Dependency Parsing. CoRR abs/1611.01734. http://arxiv.org/abs/1611.01734. ↗ our card
9. Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajič, Christopher Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, Reut Tsarfaty, and Daniel Zeman. 2016. Universal Dependencies v1: A multilingual treebank collection. In Proceedings of the 10th Inter… find our card
10. Timothy Dozat, Peng Qi, and Christopher D. Manning. 2017. Stanford’s Graph-based Neural Dependency Parser at the CoNLL 2017 Shared Task. In Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies. Association for Computational Linguistics, Vancouver, C… find our card
11. Joakim Nivre et al. 2018. Universal dependencies 2.2. LINDAT/CLARIN digital library at the Institute of Formal and Applied Linguistics (ÚFAL), Faculty of Mathematics and Physics, Charles University. http://hdl.handle.net/11234/1-2837. ↗
12. Chris Dyer, Miguel Ballesteros, Wang Ling, Austin Matthews, and Noah A. Smith. 2015. Transition-based dependency parsing with stack long short-term memory. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural… find our card
13. Stephan Oepen, Lilja Øvrelid, Jari Björne, Richard Johansson, Emanuele Lapponi, Filip Ginter, and Erik Velldal. 2017. find
14. The 2017 Shared Task on Extrinsic Parser Evaluation. Towards a reusable community infrastructure. In Proceedings of the 2017 Shared Task on Extrinsic Parser Evaluation at the Fourth International Conference on Dependency Linguistics and the 15th International Conference on Parsing Technologies. Pisa… find
15. Jack Edmonds. 1967. Optimum branchings. Journal of Research of the National Bureau of Standards, B 71:233–240. find
16. Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. CoRR abs/1412.6980. http://arxiv.org/abs/1412.6980. ↗
17. Martin Potthast, Tim Gollub, Francisco Rangel, Paolo Rosso, Efstathios Stamatatos, and Benno Stein. 2014. Improving the reproducibility of PAN’s shared tasks: Plagiarism detection, author identification, and author profiling. In Evangelos Kanoulas, Mihai Lupu, Paul Clough, Mark Sander-son, Mark Hall… — [doi:10.1007/978-3-319-11382-1](https://doi.org/10.1007/978-3-319-11382-1) find
18. Adhiguna Kuncoro, Miguel Ballesteros, Lingpeng Kong, Chris Dyer, Graham Neubig, and Noah A. Smith. 2016. What do recurrent neural network grammars learn about syntax? CoRR abs/1611.05774. http://arxiv.org/abs/1611.05774. ↗
19. Milan Straka, Jan Hajič, and Jana Straková. 2016. UD-Pipe: trainable pipeline for processing CoNLL-U files performing tokenization, morphological analysis, POS tagging and parsing. In Proceedings of the 10th International Conference on Language Alex Graves and Jürgen Schmidhuber. 2005. Frame-wise… find
20. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Comput. 9(8):1735– 1780. https://doi.org/10.1162/neco.1997.9.8.1735. ↗
21. Resources and Evaluation (LREC 2016). European Language Resources Association, Portorož, Slovenia. find
22. Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies. Association for Computational Linguistics, Vancouver, Canada, pages 1–19. Milan Straka and Jana Straková. 2017. Tokenizing, POS Tagging, Lemmatizing and Parsing UD 2.0 with UDPipe. In Proceedings of the CoNLL 2017 Shared Tas… find our card
23. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. CoRR abs/1706.03762. http://arxiv.org/abs/1706.03762. ↗
24. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144 . find
25. Zhilin Yang, Ruslan Salakhutdinov, and William W. Cohen. 2016. Multi-task cross-lingual sequence tagging from scratch. CoRR abs/1603.06270. http://arxiv.org/abs/1603.06270. — [arXiv:1603.06270](https://arxiv.org/abs/1603.06270) ↗
26. Daniel Zeman, Jan Hajič, Martin Popel, Martin Potthast, Milan Straka, Filip Ginter, Joakim Nivre, and Slav Petrov. 2018. find
27. CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies. Association for Computational Linguistics, Brussels, Belgium, pages 1–20. find our card
28. Daniel Zeman, Martin Popel, Milan Straka, Jan Hajič, Joakim Nivre, Filip Ginter, Juhani Luotolahti, Sampo Pyysalo, Slav Petrov, Martin Potthast, Francis Tyers, Elena Badmaeva, Memduh Gökırmak, Anna Nedoluzhko, Silvie Cinková, Jan Hajič jr., Jaroslava Hlaváčová, Václava Kettnerová, Zdeňka U… find
29. CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies. In Proceedings of the CoNLL 2017 find our card

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/ud-2018-straka-udpipe-2-0-prototype-conll/