Crowdsourced Hedge Term Disambiguation

What it is about

Hedge annotation is costly when experts do it, so the authors turn it into word sense disambiguation that crowd workers can do. For each of 117 potential hedge terms (relational: think, suppose, appear, in my view…; propositional: about, around, kind of, practically, pretty…) they write one short hedging definition and one non-hedging definition, each with an example. A worker sees a highlighted occurrence and picks the definition that fits. Over 20,683 occurrences in informal forum posts (LDC data), five workers per item agree 92.9% of the time (Fleiss kappa 0.75), and the majority vote matches an expert gold 96.7% of the time. The proportion of hedging uses varies widely by term, and a few terms (hear, read, say) could not be reduced to a binary choice.

Ideas

• I1. Build Mova's hedge lexicon the same way, as a table of term → hedge sense, non-hedge sense and one example of each, stored next to prag/data/cues.tsv. Each pair of examples doubles as a minimal-pair unit test for the disambiguation rules in prag/src/feats.rs (for example «there are about ten» vs «talk about Mark»).
• I2. Use the binary-sense framing for the teacher. Instead of asking Opus "is this sentence hedged?", ask which of the two senses the highlighted term has. This gives a per-occurrence silver for a later wave, which is cheaper to gate (two closed values) and easier to audit.
• I3. The per-term hedge ratio is the number to check before a cue enters the list. Compute it on Mova's own silver and on tale text with the teacher, and keep only cues with a high hedging ratio as unconditional cues; give the rest a tree rule.

For Mova

32. Politeness and hedging §7.2 and §10; prag/data/cues.tsv, prag/src/feats.rs, the silver prompt in prag/src/silver.rs; measure: per-cue precision and hedge macro-F1 in prag eval. The forum corpus itself is LDC data and is not usable by Mova.

Topics: Word sense disambiguation

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/w19-4001

@inproceedings{ulinski2019crowdsourced,
  title = {{Crowdsourced Hedge Term Disambiguation}},
  author = {Ulinski, Morgan and Hirschberg, Julia},
  booktitle = {Proceedings of the 13th Linguistic Annotation Workshop},
  year = {2019},
  doi = {10.18653/v1/w19-4001},
  url = {https://doi.org/10.18653/v1/w19-4001}
}

References

1 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Figure 4: Plot of agreement vs. proportion of hedge usages out of total occurrences for each term.* Cem Akkaya, Alexander Conrad, Janyce Wiebe, and Rada Mihalcea. 2010. Amazon Mechanical Turk for Subjectivity Word Sense Disambiguation. In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk, pages 195–203, Los Angeles. Association for Computational Linguistics. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/ulinski-2019-crowdsourced-hedge-term-disambiguation/