When to generate hedges in peer-tutoring interactions

What it is about

The question is not whether a turn contains a hedge but whether the tutor's next turn should contain one, predicted from the four previous turns. Data are 9,479 turns of face-to-face peer tutoring between teenagers (non-public); about one tutor turn in ten has a hedge. Each past turn is encoded by a sentence embedding, annotated conversational strategies (self-disclosure, praise, teasing, earlier hedges), tutoring strategies (deep or shallow questions, knowledge telling or building), DAMSL dialogue acts from an automatic tagger, rapport ratings, nonverbal behaviour (gaze, smiles, nods) and task context (problem, correctness of the last answer, pre-test scores). Gradient boosting, MLP and LSTM models with class rebalancing all stay low: best F1 0.28 against 0.11 for a dummy baseline. SHAP and an ablation point to the gaze of both partners as the most useful non-text cue.

Ideas

• I1. The measured ceiling (F1 below 0.3 even with rich multimodal features) says that "when to hedge" cannot be learned from a few thousand turns. For Mova's dialogue layer (world/src/talk.rs), hedging should stay a rule with a stated reason rather than a learned decision: hedge a correction of the addressee's wrong answer, a refusal, or an uncertain answer (low margin or model disagreement in the student). Each rule can be tested with a red case.
• I2. The one task feature with a clear rationale, whether the partner's last answer was correct, maps to the student's answer checking. When a family member corrects a student, a softening hedge («I think it's actually…») can be added by rule and logged as such, so a later study can compare learner outcomes with and without it.

For Mova

32. Politeness and hedging §7 and §11, 38. Dialogue, discourse markers and reported speech §11.3 (the family's dialogue layer); world/src/talk.rs; measure: unit tests per hedging rule. The corpus is private and not usable.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.sigdial-1.53 · arXiv 2307.15582

@inproceedings{abulimiti2023when,
  title = {{When to generate hedges in peer-tutoring interactions}},
  author = {Abulimiti, Alafate and Clavel, Chloé and Cassell, Justine},
  booktitle = {Proceedings of the 24th Annual Meeting of the Special Interest Group on Discourse and Dialogue},
  year = {2023},
  doi = {10.18653/v1/2023.sigdial-1.53},
  eprint = {2307.15582},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2023.sigdial-1.53}
}

References

40 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Alafate Abulimiti, Chloé Clavel, and Justine Cassell. 2023. How about kind of generating hedges using end-to-end neural models? In Proceedings of the 61th Annual Meeting of the Association for Computational Linguistics, Toronto, Canada. Association for Computational Linguistics. find
2. Nalini Ambady and Robert Rosenthal. 1993. Half a minute: Predicting teacher evaluations from thin slices of nonverbal behavior and physical attractiveness. Journal of personality and social psychology, 64(3):431. find
3. This study presents an effective approach to predict Dzmitry Bahdanau, Kyung Hyun Cho, and Yoshua Ben-where hedges occur in peer-tutoring interactions gio. 2015. Neural machine translation by jointly using classic ML models. Our results show the learning to align and translate. In 3rd International importance of considering various types of input Conference on Learning Representations, ICLR features, such as turn embeddings, rapport, conver-2015. sational strategies, tutoring strategies, nonverbal Jere Brophy. 1981. Teacher praise: A functional analy-behaviors, and contextual information. Moreover, sis. Review of educational research, 51(1):5–32. Shapley values applied to the predictions of the Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, different models show, for the first time, that the and W Philip Kegelmeyer. 2002. Smote: synthetic gaze of both tutor and tutee may play a critical minority over-sampling technique. Journal of artifi-role in predicting hedges. This observation is sub-cial intelligence research, 16:321–357. stantiated by subsequent ablation studies, where Tianqi Chen and Carlos Guestrin. 2016. Xgboost: A classic classification models, like XGBoost and scalable tree boosting system. In Proceedings of LightGBM, experienced a significant decline in F1 the 22nd acm sigkdd international conference on score when removing nonverbal behavior features. knowledge discovery and data mining, pages 785– 794. For future work, several directions can be pursued. First, the investigation of hedge generation in Valerian J Derlega, Sandra Metts, Sandra Petronio, and the context of expert tutors could provide valuable Stephen T Margulis. 1993. Self-disclosure. Sage insights into how experienced tutors use hedges Publications, Inc. 580 find
4. Bruce Fraser. 2010. Pragmatic competence: The case of hedging. new approaches to hedging. find
5. Lucie Galland, Catherine Pelachaud, and Florian Pe-cune. 2022. Adapting conversational strategies to co-optimize agent’s task performance and user’s engagement. In Proceedings of the 22nd ACM International Conference on Intelligent Virtual Agents, pages 1–3. find
6. Pranav Goel, Yoichi Matsuyama, Michael Madaio, and Justine Cassell. 2019. i think it might help if we multiply, and not add. In Detecting indirectness in conversation. In 9th International Workshop on Spoken Dialogue System Technology, page 27–40. Springer. find
7. Hector Gómez-Gauchía, Belén Díaz-Agudo, and Pedro A González-Calero. 2006. Conversational strategies in cobber: an affective ccbr framework. Journal of Experimental & Theoretical Artificial Intelligence, 18(4):449–469. find
8. Sergiu Hart. 1989. Shapley value. Springer. find
9. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735– 1780. find
10. Dan Jurafsky. 1997. Switchboard swbd-damsl shallow-discourse-function annotation coders manual. www. dcs. shef. ac. uk/nlp/amities/files/bib/ics-tr-97-02. pdf. find
11. Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. 2017. Lightgbm: A highly efficient gradient boosting decision tree. Advances in neural information processing systems, 30. find
12. George Lakoff. 1975. Hedges: A study in meaning criteria and the logic of fuzzy concepts. In Contemporary research in philosophical logic and linguistic semantics, pages 221–271. Springer. find our card
13. Scott M Lundberg and Su-In Lee. 2017. A unified approach to interpreting model predictions. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 4765–4774. Curran Associates, Inc. find
14. Michael Madaio, Justine Cassell, and Amy Ogan. 2017a. The impact of peer tutors’ use of indirect feedback and instructions. Philadelphia, PA: International Society of the Learning Sciences. find
15. Michael Madaio, Justine Cassell, and Amy Ogan. 2017b. “i think you just got mixed up”: confident peer tutors hedge to support partners’ face needs. International Journal of Computer-Supported Collaborative Learning, 12(4):401–421. find
16. Michael Madaio, Rae Lasko, Amy Ogan, and Justine Cassell. 2017c. Using temporal association rule mining to predict dyadic rapport in peer tutoring. International Educational Data Mining Society. find
17. Michael Madaio, Kun Peng, Amy Ogan, and Justine Cassell. 2018. A climate of support: a process-oriented analysis of the impact of rapport on peer tutoring. International Society of the Learning Sciences, Inc.[ISLS]. find
18. Michael A Madaio, Amy Ogan, and Justine Cassell. 2016. The effect of friendship and tutoring roles on reciprocal peer tutoring strategies. In Intelligent Tutoring Systems: 13th International Conference, ITS 2016, Zagreb, Croatia, June 7-10, 2016. Proceedings 13, pages 423–429. Springer. find
19. Navonil Majumder, Pengfei Hong, Shanshan Peng, Jiankun Lu, Deepanway Ghosal, Alexander Gelbukh, Rada Mihalcea, and Soujanya Poria. 2020. Mime: Mimicking emotions for empathetic response generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8968–8979. find
20. Wenqiang Lei, Xiangnan He, Yisong Miao, Qingyun Wu, Richang Hong, Min-Yen Kan, and Tat-Seng Chua. 2020. Estimation-action-reflection: Towards deep interaction between conversational and recommender systems. In Proceedings of the 13th International Conference on Web Search and Data Mining, pages 304–312. find
21. Utku Norman, Tanvi Dinkar, Barbara Bruno, and Chloé Clavel. 2022. Studying alignment in a collaborative learning activity via automatic methods: The link between what we say and do. Dialogue & Discourse, 13(2):1–48. find
22. Qintong Li, Hongshen Chen, Zhaochun Ren, Pengjie Ren, Zhaopeng Tu, and Zhumin Chen. 2020. Empdg: Multi-resolution interactive empathetic dialogue generation. In Proceedings of the 28th International Conference on Computational Linguistics, pages 4454–4466. find
23. Zhaojiang Lin, Andrea Madotto, Jamin Shin, Peng Xu, and Pascale Fung. 2019. Moel: Mixture of empathetic listeners. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 121–132. find
24. OpenAI. 2022. Chatgpt: Optimizing language models for dialogue. find
25. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744. find
26. Natalie Parde and Rodney Nielsen. 2017. Finding patterns in noisy crowds: Regression-based annotation aggregation for crowdsourced data. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1907–1912. find
27. Vipul Raheja and Joel Tetreault. 2019. Dialogue act classification with context-aware self-attention. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3727–3733. find
28. Yann Raphalen, Chloé Clavel, and Justine Cassell. 2022. ”You might think about slightly revising the title”: Identifying hedges in peer-tutoring interactions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2160–2174, Dublin, Ireland. Association for Computational Linguistics. of the 17th annual meeting of the special interest group on discourse and dialogue, pages 404–412. find our card
29. Ran Zhao, Alexandros Papangelis, and Justine Cassell. 2014. Towards a dyadic computational model of rapport management for human-virtual agent interaction. In International conference on intelligent virtual agents, pages 514–527. Springer. find
30. Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2019. Towards empathetic open-domain conversation models: A new benchmark and dataset. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5370–5381. find
31. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992. find
32. Oscar J Romero, Ran Zhao, and Justine Cassell. 2017. Cognitive-inspired conversational-strategy reasoner for socially-aware agents. In IJCAI, pages 3807– 3813. Melbourne, VIC. find
33. John R Searle. 1965. What is a speech act. Perspectives in the philosophy of language: a concise anthology, 2000:253–268. find
34. Jamin Shin, Peng Xu, Andrea Madotto, and Pascale Fung. 2020. Generating empathetic responses by looking ahead the user’s sentiment. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7989– 7993. IEEE. find
35. Helen Spencer-Oatey. 2005. (im)politeness, face and perceptions of rapport: Unpackaging their bases and interrelationships. 1(1):95–119. find
36. Mukund Sundararajan and Amir Najmi. 2020. The many shapley values for model explanation. In International conference on machine learning, pages 9269–9278. PMLR. find
37. Linda Tickle-Degnen and Robert Rosenthal. 1990. The nature of rapport and its nonverbal correlates. Psychological inquiry, 1(4):285–293. find
38. Veronika Vincze. 2014. Uncertainty detection in natural language texts. PhD, University of Szeged, 141. find
39. Timothy Williamson. 2002. Vagueness. Routledge. find
40. Zhou Yu, Ziyu Xu, Alan W Black, and Alexander Rudnicky. 2016. Strategy and policy learning for non-task-oriented conversational systems. In Proceedings Appendix: SHAP Value Graphs The vertical axis indicates the mean contribution of the feature over the model decision. The horizontal axis indicates how the distribution of features influences the model decision. *Figure 2: Feature Importance for Different Models* find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/abulimiti-2023-when-generate-hedges-peer-tutoring/