VARD 2: A tool for dealing with spelling variation in historical corpora

What it is about

Early Modern English has no standard spelling. One text can spell the same word in several ways, and modern taggers, frequency lists and concordances miss the variants. VARD 2 is a pre-processor. It marks each word that is not in a modern lexicon as a possible variant, and lists ranked modern candidates beside it, keeping the original spelling. Candidates come from three sources: a hand-made variant list, a modified SoundEx code, and letter replacement rules (doubled letters, u/v and i/j swaps, a final e). A candidate's confidence is the sum of the weights of the sources that found it, minus a small edit-distance term, and the weights change as each source proves useful. The paper is a tool description with no evaluation of its own; the results it quotes from earlier work show that modernised spelling raised CLAWS tagging accuracy on Shakespeare from 81.94% to 88.88%, and on the Lampeter corpus from 88.46% to 91.24%.

Ideas

• I1. tools/gutenberg and en/src/emode.rs: keep the raw spelling of a token and add its modern candidate, as the paper's XML output does with an attribute, so that en parses the modern form and the output still shows the original. The chapter says Mova reads modernised editions and does not normalise spelling; this step is for raw texts, such as early prints. Measure tag and parse accuracy on a hand-checked sample of raw tale text, with and without the step.
• I2. A transparent candidate list in en/src/emode.rs: for an unknown word, give each candidate the rule that found it (doubling, u/v, final e, phonetic match), a fixed score, and a readable reason, so that explain can say why a form was read as a modern word. Use fixed weights in a fixed order, not weights learned during a run, so that two runs give the same output. Measure top-1 accuracy on 200 hand-labelled variants from one tale, run twice.
• I3. Negative control: run the same step on modern text (EWT test). The paper gives no false-positive rate, so Mova must measure it. A step that changes many modern words is wrong, however well it does on old spelling.

For Mova

The chapter is 09. Early Modern and archaic forms, which says that spelling variants of early prints are not normalised yet. The code is tools/gutenberg, en/src/emode.rs and en/src/tok.rs. The paper has no open licence: take the idea and the measurement design, and write the rules and any variant lists ourselves, from our own texts.

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{baron2008vard,
  title = {{VARD 2: A tool for dealing with spelling variation in historical corpora}},
  author = {Baron, Alistair and Rayson, Paul},
  booktitle = {Proceedings of the Postgraduate Conference in Corpus Linguistics, Aston University, Birmingham, 22 May 2008},
  year = {2008},
  url = {https://eprints.lancs.ac.uk/id/eprint/41666}
}

References

19 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. The VARD 2 tool is designed with EModE spelling variation in mind; however its learning capabilities and flexibility allow the tool to be used with potentially any form of spelling variation. This is an area which requires further investigation. VARD 2 is still under development and various improvements are planned for the near future. Evaluation of the tool’s accuracy (Rayson et al, forthcoming) and its effect on part-of-speech tagging (Rayson et al, 2007) has already taken place. With future improvements to the tool, further evaluation of its accuracy, usability and effect on corpus linguistic tools will be necessary. find
2. References Archer, D., McEnery, T., Rayson, P. and Hardie, A. (2003). Developing an automated semantic analysis system for Early Modern English. In Archer, D, Rayson, P., Wilson, A. and McEnery, T. (eds.). Proceedings of the Corpus Linguistics 2003 conference. UCREL technical paper number 16. UCREL, Lancaster University, pp. 22 - 31. find
3. Culpeper, J. and Kytö, M. (1997). Towards a corpus of dialogues, 1550-1750. In Ramisch, H. and Wynne, K. (eds.). Language in Time and Space. Studies in Honour of Wolfgang Viereck on the Occasion of His 60th Birthday (Zeitschrift für Dialektologie und Linguistik - Beihefte, Heft 97). pp 60-73. Franz Steiner Verlag, Stuttgart. find
4. Garside, R., and Smith, N. (1997). A hybrid grammatical tagger: CLAWS4, in Garside, R., Leech, G., and McEnery, A. (eds.) Corpus Annotation: Linguistic Information from Computer Text Corpora. Longman, London, pp. 102-121. find
5. Görlach, M. (1991). Introduction to Early Modern English, Cambridge University Press, Cambridge. find
6. Kukich, K. (1992). Techniques for Automatically Correcting Words in Text. ACM Computing Surveys, Vol. 24, No. 4, pp. 377-439. find
7. Kytö, M., Rissanen, M. and Wright, S. (1994). Corpora across the Centuries: Proceedings of the First International Colloquium on English Diachronic Corpora, Cambridge, March 1993. Rodopi, Amsterdam. find
8. Kytö, M. and Voutilainen, A. (1995). Applying the Constraint Grammar Parser of English to the Helsinki Corpus. ICAME Journal 19, pp. 23-48. find
9. Leech, G., Rayson, P. and Wilson, A. (2001). Word Frequencies in Written and Spoken English: based on the British National Corpus. Longman, London. find
10. Nevalainen, T. (1997). Ongoing work on the Corpus of Early English Correspondence. In Hickey, R., Kytö, M., Lancashire, I. and Rissanen, M. (eds.) Tracing the Trail of Time: Proceedings from the Second Diachronic Corpora Workshop. Rodopi, Amsterdam. find
11. Rayson, P., Archer, D., Piao, S. L., McEnery, T. (2004). The UCREL Semantic Analysis System. In proceedings of the workshop on Beyond Named Entity Recognition Semantic labelling for NLP tasks in association with 4th International Conference on Language Resources and Evaluation (LREC 2004), 25th May 2004, Lisbon, Portugal, pp. 7-12. find
12. Rayson, P., Archer, D. and Smith, N. (2005). VARD versus Word: A comparison of the UCREL variant detector and modern spell checkers on English historical corpora. In proceedings of the Corpus Linguistics 2005 conference, July 14-17, Birmingham, UK. Proceedings from the Corpus Linguistics Conference Series on-line e-journal, Vol. 1, no. 1, ISSN 1747-9398. find
13. Rayson, P. (2007). Wmatrix: a web-based corpus processing environment, Computing Department, Lancaster University. Available on the internet at http://ucrel.lancs.ac.uk/wmatrix/. ↗
14. Rayson, P., Archer, D., Baron, A., Culpeper, J. and Smith, N. (2007). Tagging the Bard: Evaluating the accuracy of a modern POS tagger on Early Modern English corpora. In proceedings of Corpus Linguistics 2007, July 27-30, University of Birmingham, UK. find
15. Rayson, P., Archer, D., Baron, A. and Smith, N. (forthcoming). Travelling Through Time with Corpus Annotation Software. In Proceedings of Practical Applications in Language and Computers (PALC) 2007, The Department of English Language at Łódź University, Poland, 19th-22nd April 2007. find
16. Richardson, M. (1980). Henry V, the English Chancery, and Chancery English. In Speculum, Vol. 55, No. 4, pp. 726-750. find
17. Scott, M. (2004). WordSmith Tools version 4. Oxford University Press. find
18. Singh, I. (2005). The History of English. Hodder Arnold, London. find
19. Vallins, G. H. and Scragg, D.G. (1954). Spelling. André Deutsch. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/enprag-2008-baron-vard2-tool-dealing-spelling-variation/