A Gold Standard for Scalar Adjectives

What it is about

The paper builds a gold standard for two things: which adjectives belong on one scale (for example size, or temperature), and in what order. Two Mechanical Turk tasks were run. In the first, 500 workers were shown three prompt words from one of 12 postulated scales and asked to list other adjectives that belonged. Free lists were aggregated with Cultural Consensus Theory (CCT), using a bias term for list length, and the first factor's eigenvalue ratio measured consensus (3:1 is the usual threshold). In the second task, 200 workers dragged the kept words into an order, and CCT again gave the order. The result is 12 ordered scales, with eigenvalue ratios from 29.47 (size) down to 3.52 (speed). Of the prompt words, 65% were judged correct, and 14 of 16 word groups showed a significant link between prompt status and membership. The gold standard was then used to score three automatic ordering methods on sliding windows of three words. The two methods that score highest are Kim and de Marneffe's (mean window accuracy 0.50 for the wider setting) and De Melo and Bansal's; the paper's Table 5 gives per-scale values up to 0.66. For speed and sameness no method was correct on any window.

Ideas

• I1. en/src/dict.rs and world/src/dictionary.rs: store the 12 gold scales as ordered BTreeMap entries with a membership flag, so that word-sense lookups can answer "which is larger, medium or huge" by scale position. Keep the scale file small and build it from the lexicon through build.rs.
• I2. A deterministic consensus check for Mova's own scale lists: for each prompt set, count how often each word is listed, build the informant-by-word matrix, and take the first factor by power iteration. Report the eigenvalue ratio; a scale below 3:1 is marked as not a scale. Run twice and compare.
• I3. Evaluate Mova's scale order with the paper's three-word window test on the 12 scales. A shuffled order must score close to chance (one in three on these windows), as a negative control.
• I4. Store words that most informants reject (the paper lists "fresh", "difficult", "slow" as non-members of some scales) as explicit negative entries, so that the scale builder does not add them back.

For Mova

The chapters are 15. Adjectives and adverbs (scalar adjectives and their order) and 12. Pronouns, determiners and numerals (quantifiers such as "few", "some", "many"). The code is en/src/dict.rs for the lexical scales, world/src/dictionary.rs for word senses, and prag/src/infer.rs where the ordering feeds scalar inference. The measure: window accuracy on the 12 scales, run twice, against a shuffled-order negative control. Evaluation only: the gold standard is CC BY-NC and must not enter a release.

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{wilkinson2016gold,
  title = {{A Gold Standard for Scalar Adjectives}},
  author = {Wilkinson, Bryan and Oates, Tim},
  booktitle = {Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16)},
  year = {2016},
  url = {https://aclanthology.org/L16-1424/}
}

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/wilkinson-2016-gold-standard-scalar-adjectives/