Tokenization as the Initial Phase in NLP

What it is about

The paper argues that tokenization, the first step of NLP, is a real problem with its own definitions. It defines the notions of word and token from two points of view, that of the lexicographer and that of a working implementation. Automatic segmentation of Chinese words serves as the main illustration. For English, the authors develop practical methods to identify compound tokens, such as idioms, phrasal verbs and fixed expressions, so that each of them is read as one unit. The paper gives no evaluation table, so its value for Mova is in the definitions and in the list of compound types.

Ideas

• I1. Keep UD tokens as they are, but mark compound units (phrasal verbs, fixed expressions such as in spite of) as spans from a list in the in-code dictionary, and pass the span to the parser as a feature; measure on EWT test the LAS of fixed, compound:prt and flat arcs with en parse-eval, with and without the span feature. Joining them into single tokens would break UD tokenization, so the span is the safe form of the paper's idea.
• I2. Write the definition of a token used in Mova as a short rule list in 19. The lexicon: vocabulary, multiword expressions and register, and check it against the EWT tokens that the paper's criteria would split or join.
• I3. A count probe: list the candidate compound tokens in EWT test that the dictionary covers, and report how many of them the tokenizer currently splits. This gives the size of the problem before any change.

For Mova

The code is en/src/tok.rs and en/src/dict.rs, with the command en tok-eval. The chapter is 19. The lexicon: vocabulary, multiword expressions and register. The paper's method is for Chinese and for an early lexicon, so only the definitions and the idea of compound tokens carry over. Measure first on EWT test, with the count of joined tokens and the exact-match rate, and check that the result is the same in two runs.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.3115/992424.992434

@inproceedings{webster1992tokenization,
  title = {{Tokenization as the Initial Phase in NLP}},
  author = {Webster, Jonathan J. and Kit, Chunyu},
  booktitle = {COLING 1992 Volume 4: The 14th International Conference on Computational Linguistics},
  year = {1992},
  doi = {10.3115/992424.992434},
  url = {https://doi.org/10.3115/992424.992434}
}

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-1992-webster-tokenization-as-initial-phase-nlp/