Is It \textit JUST Semantics? A Case Study of Discourse Particle Understanding in LLMs

What it is about

The English discourse particle "just" has several senses: exclusive ("Betsy just eats chicken nuggets"), temporal ("just arrived"), unelaboratory ("Fido is just a dog"), unexplanatory ("the lights just turn on and off") and emphatic ("just delicious"). The authors test whether instruction-tuned LLMs can tell these senses apart. They use 90 hand-built sentences, 15 per sense, each with one clear sense and no context, and 149 sentences from movie subtitles labelled by semanticists. In the first task, a model picks one of six labels; the log-probabilities of the label continuations are compared. Chance is 0.167 on the hand-built data and 0.403 on subtitles, where the exclusive sense is the majority label. Models beat chance on the hand-built set, except Llama-3.2-1b, and the largest, Llama-3.3-70b, is not much better than mid-size models. Accuracy falls by 0.24 on average on subtitles. Adding context changes it by -0.05 on average. In the second task, models judge whether two sentences use "just" in the same way. Only the adjective and temporal senses are reliably separated. On the four target senses, only the unelaboratory and emphatic senses are separated in a few larger models. Ten models were run at temperature 0.

Ideas

• I1. prag/src/frames.rs or a small decision list in prag: "just" senses from explicit features: a following verb with a temporal adverb (temporal), "just a" (unelaboratory), a following adjective or intensifier (emphatic), "just" plus a sentence-final "why" reading (unexplanatory). Write it as an ordered rule list, so each label names the rule that fired.
• I2. Build a 90-sentence hand-made set for the Mova school (15 per sense, one sense each, no context), and report accuracy against chance 1/6. A shuffled-label control must score at chance.
• I3. Pairwise same-sense test without a model: represent each sentence by its rule features, then check that same-sense pairs score above different-sense pairs, with a rank test rather than a t-test. This tests the rules' internal consistency.
• I4. Context check: add one preceding sentence and require that accuracy does not drop by more than the paper's 0.05 average. A rule that reads the context window is a deterministic way to test this.

For Mova

The chapters are 19. The lexicon: vocabulary, multiword expressions and register (function words and particles), 30. Pragmatics and discourse: overview and 33. Implicature and presupposition (exclusives). The code is prag/src/frames.rs for features, en/src/dict.rs for the entry for "just", and world/src/knowledge.rs for the test items. The measure: accuracy on the 90 hand-built sentences against chance 0.167, run twice, plus the shuffled-label control.

Topics: Discourse analysis

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.findings-acl.1117

@inproceedings{sheffield2025is,
  title = {{Is It textit JUST Semantics? A Case Study of Discourse Particle Understanding in LLMs}},
  author = {Sheffield, William and Misra, Kanishka and Pyatkin, Valentina and Deo, Ashwini and Mahowald, Kyle and Li, Junyi Jessy},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2025},
  year = {2025},
  doi = {10.18653/v1/2025.findings-acl.1117},
  url = {https://doi.org/10.18653/v1/2025.findings-acl.1117}
}

References

30 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. stand the semantics of discourse particles; more work needs to be performed before generalizing our findings to other discourse particles, which we leave for the future. We have used metalinguistic prompting to analyze LLMs’ understanding of just senses. However, this class of methods has been found to underestimate LLMs’ linguistic abilities, especially when compared to using direct sentence log-probabilities (Hu and Levy, 2023). However, it is not obvious how one could analyze the nuanced distinctions in the senses of discourse particles using standard log-probability based approaches (Warstadt et al., 2020; Hu et al., 2020; Misra et al., 2023, i.a.). We therefore leave this direction as an avenue for future work. See Appendix E for details on the results in this section. find
2. Andrea Beltrama. 2018. Metalinguistic" just" and" simply": exploring emphatic exclusives. In Semantics and Linguistic Theory, pages 307–324. find
3. Andrea Beltrama. 2022. Just perfect, simply the best: an analysis of emphatic exclusion. Linguistics and Philosophy, 45(2):321–364. find
4. Anne Beyer, Sharid Loáiciga, and David Schlangen. 2021. Is incoherence surprising? targeted evaluation of coherence prediction from language models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4164–4173, Online. Association for Computational Linguistics. find
5. Andrea Bonomi and Paolo Casalegno. 1993. Only: Association with focus in event semantics. Natural Language Semantics, 2(1):1–45. find
6. Chunkit Chan, Cheng Jiayang, Weiqi Wang, Yuxin Jiang, Tianqing Fang, Xin Liu, and Yangqiu Song. 2024. Exploring the potential of ChatGPT on sentence level relations: A focus on temporal, causal, and discourse relations. In Findings of the Association for Computational Linguistics: EACL 2024, pages 684–721, St. Julian’s, Malta. Association for Computational Linguistics. find
7. Jacob Cohen. 1988. Statistical Power Analysis for the Behavioral Sciences, 2 edition. L. Erlbaum Associates. find
8. Jennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox, and Roger Levy. 2020. A systematic assessment of syntactic generalization in neural language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1725–1744, Online. Association for Computational Linguistics. ↗
9. Jennifer Hu and Roger Levy. 2023. Prompting is not a substitute for probability measurements in large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5040–5060, Singapore. Association for Computational Linguistics. find
10. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv:2310.06825. ↗
11. Najoung Kim, Roma Patel, Adam Poliak, Patrick Xia, Alex Wang, Tom McCoy, Ian Tenney, Alexis Ross, Tal Linzen, Benjamin Van Durme, et al. 2019. Probing what different nlp tasks teach machines about function word comprehension. In Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (* SEM 2019), pages 235–249. find
12. Yan Cong, Emmanuele Chersoni, Yu-Yin Hsu, and Philippe Blache. 2023. Investigating the effect of discourse connectives on transformer surprisal: Language models understand connectives, Even so they are surprised. In Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 222–232, Singapore. Association for Computational Linguistics. find
13. David Lee. 1987. The semantics of just. Journal of pragmatics, 11(3):377–398. find
14. Elizabeth Coppock and David I Beaver. 2014. Principles of the exclusive muddle. Journal of Semantics, 31(3):371–432. find
15. Kanishka Misra. 2022. minicons: Enabling flexible behavioral and representational analyses of transformer language models. arXiv:2203.13112. ↗
16. Ashwini Deo and William Carl Thomas. 2025. Addressing the widest answerable question: English “just” as a domain widening strategy. Journal of Semantics, page ffae015. find
17. Kanishka Misra, Julia Rayz, and Allyson Ettinger. 2023. COMPS: Conceptual minimal pair sentences for testing robust property knowledge and its inheritance in pre-trained language models. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2928– 2949, Dubrovnik, Croatia. Association for Computational Linguistics. find
18. Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, Will Smith, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah A. Smith, and Hannaneh Hajishirzi. 2024. Olmo: Accelerating the science of language models. Preprint. find
19. Patrick Georg Grosz. 2012. On the grammar of optative constructions. find
20. Pierre Lison, Jörg Tiedemann, and Milen Kouylekov. 2018. Opensubtitles2018: Statistical rescoring of sentence alignments in large, noisy parallel corpora. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). find
21. Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, Michal Guerquin, Hamish Ivison, Pang Wei Koh, Jiacheng Liu, Saumya Malik, William Merrill, Lester James V. Miranda, Jacob Morrison, Tyler Murray, Crystal Nam, Valentina Pyatkin, Aman Rangapur, Michael Schmitz, Sam Skjonsberg, David Wadden, Christopher Wilhelm, Michael Wilson, Luke Zettlemoyer, Ali Farhadi, Noah A. Smith, and Hannaneh Hajishirzi. 2024. 2 olmo 2 furious. find
22. Lalchand Pandia, Yan Cong, and Allyson Ettinger. 2021. Pragmatic competence of pre-trained language models through the lens of discourse connectives. In Proceedings of the 25th Conference on Computational Natural Language Learning, pages 367–379, Online. Association for Computational Linguistics. find
23. Alex Warstadt. 2020. "just" don’t ask: Exclusives and potential questions. In Proceedings of Sinn und Bedeutung, volume 24, pages 373–390. find
24. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020. BLiMP: The benchmark of linguistic minimal pairs for English. Transactions of the Association for Computational Linguistics, 8:377– 392. find our card
25. Kangda Wei, Aayush Gautam, and Ruihong Huang. 2024. Are LLMs good annotators for discourse-level event relation extraction? In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 1–19, Miami, Florida, USA. Association for Computational Linguistics. find
26. Bernard L Welch. 1947. The generalization of ‘student’s’problem when several different population varlances are involved. Biometrika, 34(1-2):28–35. find
27. Mia Wiegand. 2018. Exclusive morphosemantics: Just and covert quantification. In Proceedings of the west coast conference on formal linguistics (WCCFL), volume 35, pages 419–429. find
28. Mia Windhearn. 2021. Alternatives, exclusivity and underspecification. find
29. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface’s transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771. ↗
30. Frances Yung, Mansoor Ahmad, Merel Scholman, and Vera Demberg. 2024. Prompting implicit discourse relation annotation. In Proceedings of the 18th Linguistic Annotation Workshop (LAW-XVIII), pages 150–165, St. Julians, Malta. Association for Computational Linguistics. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/sheffield-2025-is-it-textit-just-semantics/