Validation Issues induced by an Automatic Pre-Annotation Mechanism in the Building of Non-projective Dependency Treebanks

What it is about

The CDG Lab builds dependency treebanks with a grammar-based parser. Before parsing, an annotator fills a long selection form, which is slow. The authors replace it with an automatic pre-annotation mechanism that segments the sentence and proposes lexical units, but this creates new validation problems: some false dependencies get marked as correct temporarily so that work can continue elsewhere in the sentence. In a trial on the Sequoia treebank the annotators saved about 25% of their time. The segmentation step alone had 2.85% errors, which fell to 1.62% after a blacklist of 13 cases was added. The paper proposes changes to the tool to make pre-annotation safer.

Ideas

• I1. Pre-annotation audit in annot: after a Haiku silver pass, count the silver arcs accepted without change in review and the arcs that later change; a high accepted-unchanged rate with many later changes marks the temporary-acceptance problem the paper describes, and it should trigger a re-check of those sentences.
• I2. Segmentation error for the learner data: measure the share of sentences where en/src/tok.rs splits or merges a token wrongly on a hand-checked sample of the learner and spoken data, with the paper's 2.85% and 1.62% as reference points.
• I3. Partitive and contraction blacklist: check whether the cases the paper blacklists (de la, du, and similar) are covered by en/src/tok.rs or en/seeds/lexicon, and count the sentences that change when those rules are added (dev set).

For Mova

The pre-annotation step is the annot crate, the gates are in tools/treeagree, and the tokenizer is en/src/tok.rs. The chapter that the ideas touch is 07. Punctuation, capital letters, abbreviations and numbers and, for the audit, 20. Syntax: the Universal Dependencies frame. Measure first: how many silver arcs in one annot batch are accepted without change and later revised.

Topics: Treebanks

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{lacroix2014validation,
  title = {{Validation Issues induced by an Automatic Pre-Annotation Mechanism in the Building of Non-projective Dependency Treebanks}},
  author = {Lacroix, Ophélie and Béchet, Denis},
  booktitle = {Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14)},
  year = {2014},
  url = {https://aclanthology.org/L14-1729/}
}

References

18 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Anne Abeillé, Lionel Clément, and Alexandra Kinyon. 2000. Building a treebank for French. In Proceedings of the Language Resources and Evaluation Conference, LREC 2000, Athens, Greece, May. find
2. Ramadan Alfared, Denis Béchet, and Alexander Dikovsky. 2011. CDG Lab: a toolbox for dependency grammars and dependency treebanks development. In Proceedings of the International Conference on Dependency Linguistics, DEPLING 2011, pages 272–281, Barcelona, Spain, September. find
3. Nart B. Atalay, Kemal Oflazer, Bilge Say, and Informatics Inst. 2003. The annotation process in the Turkish treebank. In Proceedings of the 4th International Workshop on Linguistically Interpreted Corpora, LINC-03, Budapest, Hungary, April. find
4. Denis Béchet, Alexander Dikovsky, and Ophélie Lacroix. 2014. “CDG Lab”: an integrated environment for categorial dependency grammar and dependency treebank development. In Kim Gerdes, Eva Hajičová, and Leo Wanner, editors, Computational Dependency Theory, volume 258 of Frontiers in Artificial In… find
5. Eckhard Bick. 2003. Arboretum, a hybrid treebank for Danish. In Proceedings of the 2nd Workshop on Treebanks and Linguistic Theories. find
6. Igor Boguslavsky, Ivan Chardin, Svetlana Grigorieva, Nikolai Grigoriev, Leonid L. Iomdin, Leonid Kreidlin, and Nadezhda Frid. 2002. Development of a dependency treebank for Russian and its possible applications in NLP. In In Proceedings of the 3rd International Conference on Language Resources and E… find
7. Sabine Brants, Stefanie Dipper, Silvia Hansen, Wolfgang Lezius, and George Smith. 2002. The TIGER treebank. find
8. Marie Candito and Djamé Seddah. 2012. Le corpus Sequoia : annotation syntaxique et exploitation pour l’adaptation d’analyseur par pont lexical (the Sequoia corpus : Syntactic annotation and use for a parser lexical domain adaptation method) [in French]. In Proceedings of the Joint Conference JEP-TA… find
9. Benoı̂t Crabbé and Marie Candito. 2008. Expériences d’analyse syntaxique statistique du français (statistical parsing of French) [in French]. In Proceedings of the Joint Conference JEP-TALN-RECITAL 2008, Avignon, France. find
10. Michael Dekhtyar and Alexander Dikovsky. 2008. Generalized categorial dependency grammars. In Trakhten-brot/Festschrift, LNCS 4800, pages 230–255. Springer. find
11. Pascal Denis and Benoı̂t Sagot. 2009. Coupling an annotated corpus and a morphosyntactic lexicon for state-ofthe-art POS tagging with less human effort. In Proceedings of the Pacific Asia Conference on Language, Information and Computation, PACLIC 2009, Hong Kong, China. find
12. Alexander Dikovsky. 2011. Categorial dependency grammars: from theory to large scale grammars. In Proceedings of the International Conference on Dependency Linguistics, DEPLING 2011, September. find
13. Jan Hajič, Alena Böhmová, Eva Hajičová, and Barbora Vidová-Hladká. 2000. The Prague Dependency Treebank: A Three-Level Annotation Scenario. In A. Abeillé, editor, Treebanks: Building and Using Parsed Corpora, pages 103–127. Amsterdam:Kluwer. find
14. Fred Karlsson, Atro Voutilainen, Juha Heikkila, and Atro Anttila. 1995. Constraint Grammar, A Language-independent System for Parsing Unrestricted Text. Mouton de Gruyter. find
15. Tracy Holloway King, Richard Crouch, Stefan Riezler, Mary Dalrymple, and Ronald M. Kaplan. 2003. The PARC 700 dependency bank. In In Proceedings of the 4th International Workshop on Linguistically Interpreted Corpora (LINC-03), pages 1–8. find
16. Joakim Nivre, Johan Hall, Jens Nilsson, Atanas Chanev, Glsen Eryigit, Sandra Kbler, Svetoslav Marinov, and Erwin Marsi. 2007. MaltParser: A language-independent system for data-driven dependency parsing. Natural Language Engineering, 13:95–135, 6. find
17. Adwait Ratnaparkhi. 1996. A maximum entropy model for part-of-speech tagging. In Proceedings of the 1st Conference on Empirical Methods in Natural Language Processing, EMNLP 1996, Pennsylvania, USA, May. find
18. Benoı̂t Sagot. 2010. The Lefff, a freely available and large-coverage morphological and syntactic lexicon for French. In Proceedings of the Language Resources and Evaluation Conference, LREC 2010. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/bank-2014-lacroix-validation-issues-induced-automatic-pre/