A Lexicographer-Friendly Association Score

What it is about

Association scores rank word pairs as collocation candidates, but most of them depend on corpus size and their values have no fixed meaning. Rychlý proposes logDice, a rescaled Dice coefficient: logDice = 14 + log2(2·f_xy / (f_x + f_y)). The maximum is 14, and a value near 0 means fewer than one co-occurrence per 16,000 occurrences of either word. A gain of 1 point means twice as frequent, and 7 points about 100 times. The paper compares T-score, MI, MI3, Dice, minimum sensitivity and MI × log frequency on the verb "break" in the British National Corpus. The Dice lists are the best, and logDice keeps their ranking while giving values that do not depend on corpus size.

Ideas

• I1. en/src/lexgen.rs or the lexicon build: compute logDice for candidate multiword pairs (adjacent and within a 5-token window) from an open corpus, and rank them. Measure precision at the top 100 against the expression base in prag, on a held-out split of it.
• I2. Stability check: compute logDice on random 10%, 50% and 100% subsets of the school reading texts and compare the top 50 lists (overlap and rank change). The paper claims size-independence; Mova should measure it before using a fixed threshold.
• I3. Threshold rule: use a fixed logDice cut-off (for example 0 as "no collocation") as a readable rule in en and report how many EWT verb-object pairs pass, with a short explanation per pair.

For Mova

The touch points are en/src/lexgen.rs and en/src/dict.rs (candidate lists) and the prag expression base for checking. The chapter is 19. The lexicon: vocabulary, multiword expressions and register. The paper's example data is the BNC, which is NC and may only be used for tests; build with open corpora such as EWT (CC BY-SA). Measure first the top-100 precision of logDice against the prag expression base, with two identical runs.

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{rychlu2008lexicographer,
  title = {{A Lexicographer-Friendly Association Score}},
  author = {Rychlý, Pavel},
  booktitle = {RASLAN 2008 (Recent Advances in Slavonic NLP), Masaryk University},
  year = {2008},
  url = {https://nlp.fi.muni.cz/raslan/2008/}
}

References

5 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Smadja, F.: Retrieving Collocations from Text: Xtract. Computational Linguistics 19(1) (1994) 143–177. find
2. Kilgarriff, A., Rychlý, P., Smrž, P., Tugwell, D.: The Sketch Engine. Proceedings of Euralex (2004) 105–116. find
3. Dice, L.: Measures of the Amount of Ecologic Association Between Species. Ecology 26(3) (1945) 297–302. find
4. Evert, S.: The Statistics of Word Cooccurrences: Word Pairs and Collocations. Unpublished Ph.D. dissertation, University of Stuttgart (2004). find
5. Rychlý, P.: Manatee/Bonito – A Modular Corpus Manager. In: P. Sojka, A. Horák (Eds.): RASLAN 2007 Proceedings (2007), pp. 65–70. 2 f xy f x + f y find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/lex-2008-rychly-logdice/