Minor changes make a difference: a case study on the consistency of UD-based dependency parsers

Summary

The paper asks whether off-the-shelf UD parsers stay consistent when a sentence is changed only trivially. The authors take treebank sentences containing a four-digit number (mostly a year) in English, Swedish, Russian and Ukrainian, substitute 50 random values for that number, and parse every variant with pretrained Stanza models. Whole-sentence parse quality is compared with a tree-kernel similarity, and groups of identical errors are found as maximal cliques. Standard metrics (UAS, LAS, BLEX) barely move, yet a large share of variant bundles (about 44% on average) show errors that differ from one variant to another. Two retraining fixes are tried: augmenting the training data with re-numbered sentences, and replacing numbers with a placeholder token; only augmentation helps noticeably. The authors also point to a likely temporal bias in treebanks and warn applications that rely on UD trees.

How Mova uses it

• How to find annotation errors automatically — listed as method 5 ("robustness to small changes") among automatic ways to find annotation errors: places where the parse flips under a trivial substitution are candidates for errors.
• nummod — only quantity and only before the noun — cited as evidence that numbers are unstable for parsers, as background to the rules en.nominal.nummod-before-noun and en.nominal.nummod-num (numbers after a noun are identifiers, not counts).
• Used as background reading only; Mova has no perturbation-based checker built from this paper.

Effectiveness in Mova

Background reading behind one seed note and one detection idea; no component implements the method. Not measured separately.

👨‍🔬💥

Topics: Dependency parsing · Universal Dependencies · Treebanks

Cite

The paper itself, not this card. BibTeX from the library's record of the work. arXiv 2111.15413

@inproceedings{kalpakchi2021minor,
  title = {{Minor changes make a difference: a case study on the consistency of UD-based dependency parsers}},
  author = {Kalpakchi, Dmytro and Boye, Johan},
  booktitle = {Proceedings of the Fifth Workshop on Universal Dependencies (UDW, SyntaxFest 2021)},
  year = {2021},
  eprint = {2111.15413},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2111.15413}
}

References

11 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Coen Bron and Joep Kerbosch. 1973. Algorithm 457: finding all cliques of an undirected graph. Communications of the ACM, 16(9):575–577. find
2. Aditi Chaudhary, Antonios Anastasopoulos, Adithya Pratapa, David R. Mortensen, Zaid Sheikh, Yulia Tsvetkov, and Graham Neubig. 2020. Automatic extraction of rules governing morphological agreement. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pag… find
3. Danilo Croce, Alessandro Moschitti, and Roberto Basili. 2011. Structured lexical similarity via convolution kernels on dependency trees. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 1034–1046, Edinburgh, Scotland, UK., July. Association for Computa… find
4. Dmytro Kalpakchi and Johan Boye. 2020. UDon2: a library for manipulating Universal Dependencies trees. In Proceedings of the Fourth Workshop on Universal Dependencies (UDW 2020), pages 120–125, Barcelona, Spain (Online), December. Association for Computational Linguistics. find
5. Dmytro Kalpakchi and Johan Boye. 2021. Quinductor: a multilingual data-driven method for generating reading-comprehension questions using universal dependencies. arXiv preprint arXiv:2103.10121. — [arXiv:2103.10121](https://arxiv.org/abs/2103.10121) find
6. Alessandro Moschitti. 2006. Efficient convolution kernels for dependency and constituent syntactic trees. In European Conference on Machine Learning, pages 318–329. Springer. find
7. Siyao Peng and Amir Zeldes. 2018. All roads lead to UD: Converting Stanford and Penn parses to English Universal Dependencies with multilayer annotations. In Proceedings of the Joint Workshop on Linguistic Annotation, Multiword Expressions and Constructions (LAW-MWE-CxG-2018), pages 167–177, Santa F… find our card
8. Martin Popel, Zdeněk Žabokrtský, and Martin Vojtek. 2017. Udapi: Universal API for Universal Dependencies. In Proceedings of the NoDaLiDa 2017 Workshop on Universal Dependencies (UDW 2017), pages 96–101, Gothenburg, Sweden, May. Association for Computational Linguistics. find our card
9. Adithya Pratapa, Antonios Anastasopoulos, Shruti Rijhwani, Aditi Chaudhary, David R Mortensen, Graham Neubig, and Yulia Tsvetkov. 2021. Evaluating the morphosyntactic well-formedness of generated texts. arXiv preprint arXiv:2103.16590. find
10. Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 101–108, Online,… find our card
11. Milan Straka, Jan Hajič, and Jana Straková. 2016. UDPipe: Trainable pipeline for processing CoNLL-U files performing tokenization, morphological analysis, POS tagging and parsing. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 4290–4297,… find our card

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/kalpakchi-2021-minor-changes-make-difference-case/