A Text Normalisation System for Non-Standard English Words

What it is about

The paper describes a four-stage system (detection, classification, division, expansion) that turns non-standard words such as lbs, 99.3% and 1985 into words that a speech system can read out. A token counts as non-standard only if it fails four tests against a word list: the lower-cased form, the lemma, the non-possessive form, and it is not single punctuation. Alphabetic and numeric tokens are classified with a semi-supervised label-propagation method that uses features from a window of two tokens either side; mixed tokens are split by rules. Abbreviations are expanded from candidates in the word list, filtered by part of speech and by a Lesk-style overlap with the context. On a gold set of 1,000 sentences (21,447 tokens), detection reaches precision 95.1% and recall 97.4%. Expansion accuracy on 291 tokens is 91.4%, against 75.3% for a commercial system and 46% to 60% for the domain-specific models of Sproat et al. (2001).

Ideas

• I1. Make the four detection tests a rule in the school-text preprocessing (en/src/tok.rs). A token goes to the normaliser only when all four tests fail, and precision is preferred: a wrongly flagged word is kept as it is. Measure precision and recall against a small gold set of school lesson sentences, and run it twice.
• I2. The paper's example 16 June (ordinal) against 16 people (cardinal) is a context rule. Write the numeral rule as a small table next to month names, using the same two-token window, and test it on the numerals in the lessons.
• I3. Expand abbreviations by candidate selection: generate candidates from the word list that contain the abbreviation's letters in order, filter by part of speech, and rank them by overlap with the context. Apply it to the science lessons (for example kHz, lbs) in world/src/knowledge.rs and count how many abbreviations get exactly one correct expansion.

For Mova

The relevant chapters are 07. Punctuation, capital letters, abbreviations and numbers (numbers and abbreviations) and 06. Spelling rules: inflection, derivation, compounds and national standards. The code is en/src/tok.rs (tokenizer) and world/src/knowledge.rs (school lessons). The measures: detection precision and recall on a school gold set, and the count of abbreviations with one correct expansion.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/w17-4414

@inproceedings{flint2017text,
  title = {{A Text Normalisation System for Non-Standard English Words}},
  author = {Flint, Emma and Ford, Elliot and Thomas, Olivia and Caines, Andrew and Buttery, Paula},
  booktitle = {Proceedings of the 3rd Workshop on Noisy User-generated Text},
  year = {2017},
  doi = {10.18653/v1/w17-4414},
  url = {https://doi.org/10.18653/v1/w17-4414}
}

References

7 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Vivek Kumar Rangarajan Sridhar. 2015. Unsupervised text normalization using distributed representations of words and phrases. In Proceedings of the 1st Workshop on Vector Space Modeling for Natural Language Processing. find
2. AiTi Aw, Min Zhang, Juan Xiao, and Jian Su. 2006. A phrase-based statistical model for sms text normalization. In Proceedings of the COLING/ACL 2006 Main Conference Poster Session. Association for Computational Linguistics. find
3. Brian Roark and Richard Sproat. 2014. Hippocratic abbreviation expansion. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). find
4. Steven Bird, Edward Loper, and Ewan Klein. 2009. Natural Language Processing with Python. Sebastopol, CA: O’Reilly Media. find
5. Richard Sproat, Alan Black, Stanley Chen, Shankar Kumar, Mari Ostendorf, and Christopher Richards. 2001. Normalization of non-standard words. Computer Speech and Language 15:287–333. find
6. Jacob Eisenstein. 2013. What to do about bad language on the internet. In Proceedings of NAACL-HLT 2013. Association for Computational Linguistics. find
7. Christiane Fellbaum, editor. 1998. WordNet: An Electronic Lexical Database. Cambridge, MA: MIT Press. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/flint-2017-text-normalisation-system-non-standard/