First Results in a Study Evaluating Pre-annotation and Correction Propagation for Machine-Assisted Syriac Morphological Analysis

What it is about

The authors ask when machine pre-annotation (the annotator corrects a machine guess instead of starting from nothing) and correction propagation (a correction re-runs the model on the rest of the same sentence) make Syriac morphological annotation faster and more accurate. They use the Syromorph analyzer, trained on subsets of the Peshitta New Testament (about 100,000 tokens), to produce pre-annotations at fixed accuracy levels from 25% to 100%. Nine Syriac experts annotated 30 sentences from the Acts of Judas Thomas in the CCASH web tool, with the conditions assigned by a rotating matrix. The preliminary result: pre-annotations improve accuracy at 60% model accuracy and up, and speed at 80% and up (by about 10 to 20 seconds per word); the extra effect of correction propagation is unclear. The analysis rests on 1,368 word-level data points.

Ideas

• I1. A threshold rule for the assist layer: offer pre-annotations only when the analyzer's measured accuracy on held-out data is above a set level (about 60% for accuracy and 80% for speed, as a starting point). For tools/hbo, this means measuring the segmenter and parser on a held-out set before the output is shown as a pre-fill.
• I2. Correction propagation as a constrained re-parse: after a user fixes one decision (a segment, a POS, a head), re-run the deterministic chain with that decision pinned and show only the decisions that changed. In gparse terms: a re-parse with fixed arcs, not a retrained model.
• I3. The assignment design (each annotator meets each condition about equally often, and the sentence order is permuted so that model quality does not look like it rises over time) is a template for any future Mova study with human annotators.

For Mova

I1 belongs to tools/hbo (a held-out check of the segmenter before any pre-fill). I2 is a feature of gparse on CoNLL-U output. I3 is a method note for a study document, not a rule. Only the threshold figures, not the paper's data, go into a test.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.63317/2gr6g4m4soxd

@inproceedings{felt2012first,
  title = {{First Results in a Study Evaluating Pre-annotation and Correction Propagation for Machine-Assisted Syriac Morphological Analysis}},
  author = {Felt, Paul and Ringger, Eric and Seppi, Kevin and Heal, Kristian and Haertel, Robbie and Lonsdale, Deryle},
  booktitle = {Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12)},
  year = {2012},
  doi = {10.63317/2gr6g4m4soxd},
  url = {https://doi.org/10.63317/2gr6g4m4soxd}
}

References

13 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Baldridge, J., & Osborne, M. (2004). Active Learning and the Total Cost of Annotation. In Proceedings of EMNLP, 2004, pp. 9-16. find
2. Brants, T. & Plaehn, O. (2000). Interactive Corpus Annotation. In Proceedings of LREC 2000. Athens, Greece. ↗
3. Chiou, F., Chiang, D., & Palmer, M. (2001). Facilitating Treebank Annotation with a Statistical Parser. In Proceedings of the Human Language Technology (HLT) Conference. San Diego: ACL. find
4. Felt, P., Merkling, O., Carmen, M., Ringger, E., Lemmon, W., Seppi, K. and Haertel, R. (2010). CCASH: A Web Application Framework for Efficient, Distributed Language Resource Development. In Proceedings of LREC 2010. Malta. ↗ our card
5. Ganchev, K., Pereira, F., & Mandel, M. (2007). Semi-automated Named Entity Annotation. In Proceedings of the Linguistic Annotation Workshop. Prague: ACL, pp. 53-56. ↗
6. Haertel, R., Ringger, E., Seppi, K., Carroll, J., and McClanahan, P. (2008). Assessing the Costs of Sampling Methods in Active Learning for Annotation. In Proceedings of the 46 th Annual Meeting of the ACL-HLT. Columbus, OH: ACL, pp. 65-68. ↗
7. Kiraz, G. (1994). Automatic concordance generation of Syriac texts. R. Lavenant, editor. Rome, Italy: VI Symposium Syriacum 1992, Orientalia Christiana Analecta, pp. 461-475. find
8. Klebanov, B., and Beigman, E. (2009). From Annotator Agreement to Noise Models. Computational Linguistics 35(4), pp. 495-503. ↗
9. Kristjannson, T., Culotta, A., Viola, P., and McCallum, A. M. (2004). Interactive Information Extraction with Constrained Conditional Random Fields. In Nineteenth Conference of the AAAI. San Jose, CA. find
10. Marcus, M., Santorini, B., & Marcinkiewicz, M. (1993). Building a Large Annotated Corpus of English: The Penn Treebank. Computational Linguistics 19(2), pp. 313-330. ↗ our card
11. McClanahan, P., Busby, G., Haertel, R., Heal, K., Lonsdale, D. Seppi, K. and Ringger, E. (2010). A Probabilistic Morphological Analyzer for Syriac. In Proceedings of EMNLP 2010, pp. 810-820. find our card
12. Menke, J. and Martinez, T. R. (2004). Using Permutations Instead of Student's t Distribution for P-values in Paired-difference Algorithm Comparisons. In Proceedings of 2004 IEEE International Joint Conference on Neural Networks, pp. 1331-1335. Tov, E. editor. The Dead Sea Scrolls Electronic Library. find
13. Wright, William. Apocryphal Acts of the Apostles. 2 vols. London: Williams and Norgate, 1871, vol. 1, pp. 171-333; find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/lingbulk-2012-felt-first-results-study-evaluating-pre/