CCASH: A Web Application Framework for Efficient, Distributed Language Resource Development

What it is about

CCASH is a web application framework for corpus annotation, built so that cost-efficient methods such as active learning and automatic pre-labelling can first be tried in user studies and then run on large projects. The browser client is written in GWT, and the server keeps instances and annotations in a database through Hibernate. The instance provider, which selects and pre-labels items, runs as a separate service over XML-RPC and can be written in any language. Every annotation is logged as timed events (requested, presented, completed, paused, resumed), so cost can be measured per item. Two tasks are built in, English POS tagging and NER. The case study describes a user study of 33 graduate students with tag-dictionary coverage levels; the paper describes the design of that study rather than giving its results.

Ideas

• I1. Log a timed event for each tree when it is shown, edited and saved, and derive time per token. This measures what it costs to correct the Hebrew trees from tools/hbo drafts, and it is the first measure to put on the correction workflow for treebanks/hbo/ptnk.
• I2. A balanced design for a pre-labelling study: each annotator sees each of six suggestion levels exactly three times, on the same 18 sentences in a fixed order, with sentence lengths matched across levels. Use this to compare pointed and unpointed pre-labels in tools/hbo.
• I3. Keep the editor and the parser apart, with the parser behind a plain request interface, so that a new segmenter or parser can be swapped in without changing the editor. This is an architecture note for tools/hbo, not a rule.

For Mova

I1 goes into the correction workflow for treebanks/hbo/ptnk. I2 goes into the evaluation plan for tools/hbo. I3 is a design note for tools/hbo. The paper's Java and GWT code is not used.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.63317/36n37t2792qj

@inproceedings{felt2010ccash,
  title = {{CCASH: A Web Application Framework for Efficient, Distributed Language Resource Development}},
  author = {Felt, Paul and Merkling, Owen and Carmen, Marc and Ringger, Eric and Lemmon, Warren and Seppi, Kevin and Haertel, Robbie},
  booktitle = {Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10)},
  year = {2010},
  doi = {10.63317/36n37t2792qj},
  url = {https://doi.org/10.63317/36n37t2792qj}
}

References

18 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Carmen, M., Felt, P., Haertel, R., Lonsdale, D., McClanahan, P., Merkling, O., Ringger, E., Seppi, K. (2010). Tag Dictionaries Accelerate Manual Annotation. In Proceedings of LREC 2010. Malta. ↗
2. Chiou, F.-D., Chiang, D., & Palmer, M. (2001). Facilitating Treebank Annotation with a Statistical Parser. In Proceedings of the Human Language Technology (HLT) Conference. San Diego: ACL. find
3. Culotta, A. & McCallum, A. (2005). Reducing labeling effort for stuctured prediction tasks. In The Twentieth National Conference on Artificial Intelligence (AAAI). Pittsburgh, PA: pp. 746-751. find
4. Cunningham, H., Maynard, D., Kalina, B., & Tablan, V. (2002). GATE: A Framework and Graphical Development Environment for Robust NLP Tools and Applications. In Proceedings of the 40th Anniversary Meeting of the ACL. Philadelphia. find
5. Ganchev, K., Pereira, F., & Mandel, M. (2007). Semi-automated Named Entity Annotation. In Proceedings of the Linguistic Annotation Workshop. Prague: ACL, pp. 53-56. ↗
6. Haertel, R., Ringger, E., Seppi, K., Carroll, J., & McClanahan, P. (2008a). Assessing the Costs of Sampling Methods in Active Learning for Annotation. In Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics on Human Language Technologies. Columbus, Ohio: ACL Short Papers, pp. 65-68. ↗
7. Haertel, R. A., Seppi, K. D., Ringger, E. K., & Carroll, J. (2008b). Return on Investment for Active Learning. In Proceedings of the NIPS Workshop on Cost-Sensitive Learning. ACL Press. find
8. Lowe, J., Jacobson, M., & Michailovsky, B. (2004). Interlinear Text Editor Demonstration and Project Archivage Progress Report. In 4th EMELD Work-shop on Linguistic Databases and Best Practice. Detroit. find
9. Marcus, M., Santorini, B., & Marcinkiewicz, M. (1994). Building a Large Annotated Corpus of English: The Penn Treebank. Computational Linguistics 19(2), pp. 313-330. ↗ our card
10. Morton, T., & Lacivita, J. (2003). Word-Freak: An Open Tool for Linguistic Annotation. In Proceedings of HLT-NAACL, pp. 17-18. find
11. Müller, C., & Strube, M. (2006). Multi-Level Annotation of Linguistic Data with MMAX2. In S. Braun, K. Kohn, & J. Mukherjee (Eds.), Corpus Technology and Language Pedagogy. New Resources, New Tools, New Methods. English Corpus Linguistics Vol. 3. Frankfurt: Peter Lang, pp. 197-214. find
12. Ngai, G., & Yarowsky, D. (2000). Rule Writing or Annotation: Cost-efficient Resource Usage for Base Noun Phrase Chunking. In Proceedings of ACL, pp. 117-125. ↗
13. Ogren, P. V. (2006). Knowtator: A Protégé plug-in for Annotated Corpus Construction. In Proceedings of the 2006 Conference of the NAACL-HLT. New York: Companion Volume, Demonstrations, pp. 273-275. ↗
14. Ringger, E., McClanahan, P., Haertel, R., Busby, G., Carmen, M., Carroll, J., et al. (2007). Active Learning for Part-of-speech Tagging: Accelerating Corpus Annotation. In Proceedings of the Linguistic Annotation Workshop at ACL, pp. 101-108. ↗
15. Ringger, E., Carmen, M., Haertel, R., Seppi, K., Lonsdale D., McClanahan P., Carroll, J., & Ellison, N.. (2008). Assessing the Costs of Machine-assisted Corpus Annotation through a User Study. In Proceedings of LREC 2008. Morocco. ↗
16. Settles, B., Craven, M., & Friedland L. 2008. Active Learning with Real Annotation Costs. In Proceedings of the NIPS Workshop on Cost-Sensitive Learning, pp. 1069-1078. find
17. Settles, B. (2009). Active Learning Literature Survey. University of Wisconsin-Madison. Computer Sciences Technical Report 1648. find
18. Tomanek, K., Wermter, J., & Hahn, U. (2007). Efficient Annotation with the Jena ANnotation Environment (JANE). In Proceedings of the ACL 2007 Linguistic Annotation Workshop–A Merger of NLPXML 2007 and FLAC. Prague. ↗

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/lingbulk-2010-felt-ccash-web-application-framework-efficient/