Validation of Universal Dependencies by regeneration

What it is about

The paper checks UD annotation by regeneration. A UD tree is converted into a constituency-style input for the JSrealB realiser, the realiser produces a sentence, and the result is compared with the original sentence, with differences highlighted in a web tool. Across the 14 English UD 2.8 corpora (32,641 sentences) about 47 percent of sentences are regenerated exactly, ignoring case and spacing, and most differences come from contractions, which the realiser expands. The mismatches found real annotation problems: lemmas such as "best" and "better" that should be "good" and "well" with a degree feature, inconsistent case in nationality adjectives, third-person defaults in features, and single wrong head links that create crossing arcs. Several of these were fixed in later UD releases.

Ideas

• I1. Regeneration check in Rust: run en/src/lin.rs on each bank's gold UD trees with the inflection from en/src/morph.rs, and report the share of exact regenerations per bank; the paper's 47 percent for English is the baseline to beat, not a target.
• I2. Use regeneration mismatches as suspects, not corrections: a token whose regenerated form differs from the gold form is flagged for hand review, in line with the suspect-label mechanism in en/src/ptag.rs; check 100 flags per bank by hand.
• I3. Lemma consistency: compare the en/src/morph.rs lemma with the gold lemma for each (word, UPOS) pair across the English UD banks and list the inconsistent pairs; count them per bank, and send the list to the UD maintainers only with the operator's word.

For Mova

The realiser side is en/src/lin.rs (word-order generator), the lemma side is en/src/morph.rs, and the check belongs in annot or tools/treeagree. The chapters are 12. Pronouns, determiners and numerals (pronoun forms the paper mentions) and 13. Verbs (features). Measure first the exact-regeneration share on EWT dev.

Topics: Universal Dependencies

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{lapalme2021validation,
  title = {{Validation of Universal Dependencies by regeneration}},
  author = {Lapalme, Guy},
  booktitle = {Proceedings of the Fifth Workshop on Universal Dependencies (UDW, SyntaxFest 2021)},
  year = {2021},
  url = {https://aclanthology.org/2021.udw-1.9/}
}

References

9 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Marie-Catherine de Marneffe, Christopher D. Manning, Joakim Nivre, and Daniel Zeman. 2021. Universal Dependencies. Computational Linguistics, 47(2):255–308, 07. find our card
2. Albert Gatt and Ehud Reiter. 2009. SimpleNLG: A realisation engine for practical applications. In Proceedings of the 12th European Workshop on Natural Language Generation (ENLG 2009), pages 90–93, Athens, Greece, March. Association for Computational Linguistics. find
3. Kim Gerdes, Bruno Guillaume, Sylvain Kahane, and Guy Perrier. 2018. SUD or Surface-Syntactic Universal Dependencies: An annotation scheme near-isomorphic to UD. In Universal Dependencies Workshop 2018, Brussels, Belgium, November. find
4. Sylvain Kahane, Alexis Nasr, and Owen Rambow. 1998. Pseudo-projectivity, a polynomially parsable non-projective dependency grammar. In 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics, Volume 1, pages 646–652, Montrea… find
5. Paul Molins and Guy Lapalme. 2015. JSrealB: A bilingual text realizer for web programming. In Proceedings of the 15th European Workshop on Natural Language Generation (ENLG), pages 109–111, Brighton, UK, September. Association for Computational Linguistics. find
6. Guy Perrier. 2021. Étude des dépendances syntaxiques non projectives en français. Revue TAL, 62(1). find
7. Hans van Halteren. 2000. The detection of inconsistency in manually tagged text. In Proceedings of the COLING-2000 Workshop on Linguistically Interpreted Corpora, pages 48–55, Centre Universitaire, Luxembourg, August. International Committee on Computational Linguistics. find
8. Guillaume Wisniewski. 2018. Errator: a tool to help detect annotation errors in the Universal Dependencies project. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, May. European Language Resources Association (ELRA). find our card
9. Fei Xia and Martha Palmer. 2001. Converting dependency structures to phrase structures. In Proceedings of the First International Conference on Human Language Technology Research, pages 1–5. Association for Computational Linguistics. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/ud-2021-lapalme-validation-universal-dependencies-regeneration/