Further Developments in Treebank Error Detection Using Derivation Trees

What it is about

Treebank annotators sometimes bracket the same word sequence in different ways. The paper finds these cases automatically. Each tree is cut into small tree-adjoining units, and for each word sequence (a "nucleus") every instance is compared with the others in the same internal context. Instances whose reduced fragments differ are reported as inconsistent. The paper adds three things to the 2011 system: a rule to remove structure that does not matter for the comparison, an external check that compares the label of the node above the sequence, and grouping of the reports into inconsistency types. On the Penn Arabic Treebank (598,000 tokens) the internal check reports 9,984 nuclei, which fall to 4,272 after removing duplicates and group into 1,911 types. On an OntoNotes English excerpt (524,845 tokens) it reports 3,609 nuclei, or 3,012 without duplicates, in 1,186 types. A manual check of the first 10 types on Arabic found real errors in all 10, covering 266 nuclei. The internal check is in the LDC English treebank pipeline.

Ideas

• I1. A tool in Rust over CoNLL-U: for each word sequence that occurs more than once, compare the reduced UD subtree of each instance and group the differences into types. Run it on Mova's own annotated trees, such as the tools/treeagree output, and check the first ten types by hand.
• I2. Remove duplicates by testing whether the instances of a longer sequence map one-to-one onto those of a shorter one, as the paper does. Report the count before and after, so that the reviewer's load is visible.
• I3. Negative control: corrupt a small share of the POS tags in a copy of the trees. The check should report new inconsistency types at about that rate, and no new types on the clean copy.

For Mova

The chapters are 20. Syntax: the Universal Dependencies frame and 22. Phrases: noun, adjective, adverb and prepositional phrases. The code would be a new check in tools/treeagree or a sibling tool in tools/, reading the trees of en/src/parse.rs. The measures: the number of types found, the hand-checked precision of the first ten types, and the run-twice identity of the report. The paper's TAG machinery is heavy; the grouping by type and the duplicate test are the parts worth copying.

Topics: Treebanks

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{kulick2012further,
  title = {{Further Developments in Treebank Error Detection Using Derivation Trees}},
  author = {Kulick, Seth and Bies, Ann and Mott, Justin},
  booktitle = {Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12)},
  year = {2012},
  url = {https://aclanthology.org/L12-1100/}
}

References

18 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ann Bies, Mark Ferguson, Karen Katz, and Robert MacIntyre. 1995. Bracketing guidelines for Treebank II-style Penn Treebank project. Technical Report MS-CIS-95-06, University of Pennsylvania. find
2. Ann Bies, Justin Mott, and Colin Warner. 2009. English Translation treebank - EATB part3 v2.0. Linguistic Data Consortium LDC2009E55, June. find
3. Ann Bies, Justin Mott, Colin Warner, and Seth Kulick. 2012. Google Web Treebank, v1.0. Linguistic Data Consortium LDC2012R22. find
4. Tim Buckwalter. 2004. Buckwalter Arabic morphological analyzer version 2.0. Linguistic Data Consortium LDC2004L02. find
5. David Chiang. 2003. Statistical parsing with an automatically extracted Tree Adjoining Grammar. In Data Oriented Parsing. CSLI. find
6. Markus Dickinson and Detmar Meurers. 2003a. Detecting errors in part-of-speech annotation. In Proceedings of the 10th Conference of the European Chapter of the Association for Computational Linguistics (EACL-03), pages 107–114, Budapest, Hungary. find
7. Markus Dickinson and Detmar Meurers. 2003b. Detecting inconsistencies in treebanks. In Proceedings of the Second Workshop on Treebanks and Linguistic Theories (TLT 2003), Sweden. Treebanks and Linguistic Theories. find
8. Jason Eisner. 2003. Learning non-isomorphic tree mappings for machine translation. In The Companion Volume to the Proceedings of 41st Annual Meeting of the Association for Computational Linguistics, pages 205– 208, Sapporo, Japan, July. Association for Computational Linguistics. find our card
9. Spence Green and Christopher D. Manning. 2010. Better Arabic parsing: Baselines, evaluations, and analysis. In Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), pages 394– 402, Beijing, China, August. Coling 2010 Organizing Committee. find our card
10. A.K. Joshi and Y. Schabes. 1997. Tree-adjoining grammars. In G. Rozenberg and A. Salomaa, editors, Handbook of Formal Languages, Volume 3: Beyond Words, pages 69–124. Springer, New York. find
11. Yoshihide Kato and Shigeki Matsubara. 2010. Correcting errors in a treebank based on synchronous tree substitution grammar. In Proceedings of the ACL 2010 Conference Short Papers, pages 74–79, Uppsala, Sweden, July. Association for Computational Linguistics. find
12. Seth Kulick, Ann Bies, and Justin Mott. 2011. Using derivation trees for treebank error detection. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 693–698, Portland, Oregon, USA, June. Association for Computational Linguistics. find
13. Mohamed Maamouri, Ann Bies, Seth Kulick, Fatma Gaddeche, Wigdan Mekki, Sondos Krouna, and Basma Bouziri. 2008a. Arabic treebank part 1 - v4.0. Linguistic Data Consortium LDC2008E61, December 4. find
14. Mohamed Maamouri, Ann Bies, Seth Kulick, Fatma Gaddeche, Wigdan Mekki, Sondos Krouna, and Basma Bouziri. 2008b. Arabic treebank part 3 - v3.0. Linguistic Data Consortium LDC2008E22, August 20. find
15. Mohamed Maamouri, Ann Bies, Seth Kulick, Fatma Gaddeche, Wigdan Mekki, Sondos Krouna, and Basma Bouziri. 2009. Arabic treebank part 2- v3.0. Linguistic Data Consortium LDC2008E62, January 20. find
16. Mohamed Maamouri, Ann Bies, Seth Kulick, Sondos Krouna, Fatma Gaddeche, and Wajdi Zaghouani. 2010. Arabic Treebank Part 3 - v3.2. Linguistic Data Consortium LDC2010T08. find
17. Libin Shen, Lucas Champollion, and Aravind Joshi. 2008. LTAG-spinal and the Treebank: A new resource for incremental, dependency and semantic parsing. Language Resources and Evaluation, 42(1):1–19. find
18. Ralph Weischedel, Martha Palmer, Mitchell Marcus, Eduard Hovy, Sameer Pradhan, Lance Ramshaw, Nianwen Xue, Ann Taylor, Jeff Kaufman, Michelle Franchini, Mohammed El-Bachouti, Robert Belvin, and Ann Houston. 2011. OntoNotes 4.0. Linguistic Data Consortium LDC2011T03. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/kulick-2012-further-developments-treebank-error-detection/