Trustworthy NLP for Low-Resource Languages: Agent-Based Uncertainty Modeling for Hebrew Radiology Report Structuring

What it is about

The paper extracts organ-and-finding labels from 9,683 Hebrew MRE/CTE reports of Crohn's patients, with 512 reports labelled by hand and 462 used for evaluation. Llama 3.1 8B answers yes or no for each organ-finding pair, using six paraphrased prompts (a Bayesian Prompt Ensemble). An agent (Llama 3 70B) reads the six answers and their explanations and assigns one of five confidence levels (Definitely yes, Likely yes, Uncertain, Likely no, Definitely no), mapped to uncertainty 0, 0.5 and 1. Against uniform, linearly optimised and MLP-weighted combinations of the prompts, the agent gives the best F1 (0.397) and Cohen's kappa (0.301) before filtering. Dropping cases with uncertainty of 0.5 or more raises these to F1 0.479 and kappa 0.426, at the cost of 33% of cases excluded. A smaller 8B agent does worse, and the authors note that the 0.0 median for correct answers may be a discretisation effect.

Ideas

• I1. A fixed set of confidence levels for Mova's own decisions: each answer carries a level from the evidence behind it (for example, how many independent rules agree), and a case below the top levels is reported as uncertain rather than answered. Measure, as the paper does, the median level for correct and incorrect answers.
• I2. Paraphrase ensembles for teacher labels: when a large model labels Hebrew training data, ask it the same question with several paraphrases and keep the disagreement count as a quality flag on each label. This is a teacher-side check and never runs at test time.
• I3. Report the coverage against the score for any gate that can abstain (the paper's excluded-share versus F1 table), so the cost of abstaining is visible next to the gain.

For Mova

I1 is a confidence scale for the rule decisions in prag and for the segmenter in tools/hbo. I2 is a note on teacher labelling in docs/, not a crate. I3 is a reporting format for the measurements in 12. Mova and Modern Hebrew: where it stands, what it would need.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.bionlp-1.24

@inproceedings{atya2026trustworthy,
  title = {{Trustworthy NLP for Low-Resource Languages: Agent-Based Uncertainty Modeling for Hebrew Radiology Report Structuring}},
  author = {Atya, Hadas Ben and Gavrielov, Naama and Badash, Zvi and Focht, Gili and Cytter-Kuint, Ruth and Hagopian, Talar and Turner, Dan and Freiman, Moti},
  booktitle = {BioNLP 2026},
  year = {2026},
  doi = {10.18653/v1/2026.bionlp-1.24},
  url = {https://doi.org/10.18653/v1/2026.bionlp-1.24}
}

References

28 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. AI@Meta. 2024. Llama 3 Model Card. David H Bruining, Ellen M Zimmermann, Edward V Loftus Jr, William J Sandborn, Cary G Sauer, Scott A Strong, and Society of Abdominal Radiology Crohn’s Disease-Focused Panel. 2018. Consensus recommendations for evaluation, interpretation, and utilization of computed tomography and magnetic resonance enterography in patients with small bowel crohn’s disease. Radiology, 286(3):776–799. find
2. Felix Busch, Lena Hoffmann, Daniel Pinto Dos Santos, Marcus R Makowski, Luca Saba, Philipp Prucker, Martin Hadamitzky, Nassir Navab, Jakob Nikolas Kather, Daniel Truhn, and 1 others. 2024. Large language models for structured reporting in radiology: past, present, and future. European Radiology, pages 1–14. find
3. Shan Chen, Marco Guevara, Shalini Moningi, Frank Hoebers, Hesham Elhalawani, Benjamin H. Kann, Fallon E. Chipidza, Jonathan Leeman, Hugo J. W. L. Aerts, Timothy Miller, Guergana K. Savova, Jack Gallifant, Leo A. Celi, Raymond H. Mak, Maryam Lustberg, Majid Afshar, and Danielle S. Bitterman. 2024. The effect of using a large language model to respond to patient messages. The Lancet Digital Health, 6(6):e379–e381. find
4. Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep Reinforcement Learning from Human Preferences. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc. find
5. Rushabh Doshi, Kanhai S Amin, Pavan Khosla, Simar S Bajaj, Sophie Chheang, and Howard P Forman. 2024. Quantitative evaluation of large language models to streamline radiology report impressions: A multimodal retrospective analysis. Radiology, 310(3):e231593. find
6. Mira Y Friedman, Maya Leventer-Roberts, Joseph Rosenblum, Nir Zigman, Iris Goren, Vered Mourad, Natan Lederman, Nurit Cohen, Eran Matz, Doron Z Dushnitzky, Nirit Borovsky, Moshe B Hoshen, Gili Focht, Malka Avitzour, Yael Shachar, Yehuda Chow-ers, Rami Eliakim, Shomron Ben-Horin, Shmuel Odes, and 7 others. 2018. Development and validation of novel algorithms to identify patients with inflammatory bowel diseases in israel: an epi-iirn group study. Clinical Epidemiology, pages 671–681. find
7. Melissa A Haendel, Christopher G Chute, and Peter N Robinson. 2018. Classification, ontology, and precision medicine. New England Journal of Medicine, 379(15):1452–1462. find
8. Saeed Hassanpour and Curtis P Langlotz. 2016. Information extraction from multi-institutional radiology reports. Artificial intelligence in medicine, 66:29–39. find
9. L Hazan, G Focht, N Gavrielov, R Reichart, C Friss, R Cytter Kuint, D Turner, and M Freiman. 2024a. P269 harnessing natural language processing for structured information extraction from radiology reports in crohn’s disease: A nationwide study from the epi-iirn. Journal of Crohn’s and Colitis, 18(Supplement_1):i633–i634. find
10. Liam Hazan, Gili Focht, Naama Gavrielov, Roi Reichart, Talar Hagopian, Mary-Louise C Greer, Ruth Cytter Kuint, Dan Turner, and Moti Freiman. 2024b. Leveraging prompt-learning for structured information extraction from crohn’s disease radiology reports in a low-resource language. arXiv preprint arXiv:2405.01682. ↗
11. Yuheng Huang, Jiayang Song, Zhijie Wang, Shengming Zhao, Huaming Chen, Felix Juefei-Xu, and Lei Ma. 2025. Look Before You Leap: An Exploratory Study of Uncertainty Measurement for Large Language Models. IEEE Transactions on Software Engineering, 51(2):413–429. ArXiv:2307.10236 [cs]. ↗
12. Benjamin Kompa, Jasper Snoek, and Andrew L. Beam. 2021. Second opinion needed: communicating uncertainty in medical machine learning. npj Digital Medicine, 4(1):1–6. find
13. Curtis P Langlotz, Bibb Allen, Bradley J Erickson, Jayashree Kalpathy-Cramer, Keith Bigelow, Tessa S Cook, Adam E Flanders, Matthew P Lungren, David S Mendelson, Jeffrey D Rudie, and 1 others. 2019. A roadmap for foundational research on artificial intelligence in medical imaging: from the 2018 nih/rsna/acr/the academy workshop. Radiology, 291(3):781–791. find
14. J Martijn Nobel, Koos van Geel, and Simon GF Robben. 2022. Structured reporting in radiology: a systematic review to explore its potential. European radiology, pages 1–18. find
15. Sebastian Nowak, Benjamin Wulff, Yannik C Layer, Maike Theis, Alexander Isaak, Babak Salam, Wolfgang Block, Daniel Kuetting, Claus C Pieper, Julian A Luetkens, and 1 others. 2025. Privacy-ensuring open-weights large language models are competitive with closed-weights gpt-4o in extracting chest radiography findings from free-text reports. Radiology, 314(1):e240895. find
16. OpenAI. 2024. ChatGPT (GPT-4). find
17. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. arXiv preprint. ArXiv:2203.02155 [cs]. ↗
18. Daniel Reichenpfader, Henning Müller, and Kerstin Denecke. 2024. A scoping review of large language model based approaches for information extraction from radiology reports. NPJ Digital Medicine, 7(1):222. Large language model uncertainty proxies: discrimination and calibration for medical diagnosis and treatment. Journal of the American Medical Informatics Association, 32(1):139–149. find
19. Shaltiel Shmidman, Avi Shmidman, Amir DN Cohen, and Moshe Koppel. 2024. Adapting llms to hebrew: Unveiling dictalm 2.0 with enhanced vocabulary and instruction capabilities. arXiv preprint arXiv:2407.07080. ↗ our card
20. Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. 2022. Learning to summarize from human feedback. arXiv preprint. ArXiv:2009.01325 [cs]. ↗
21. Hao Sun. 2024. Supervised Fine-Tuning as Inverse Reinforcement Learning. arXiv preprint. ArXiv:2403.12017 [cs]. ↗
22. Idit Tessler, Amit Wolfovitz, Eran E. Alon, Nir A. Ge-cel, Nir Livneh, Eyal Zimlichman, and Eyal Klang. 2024. ChatGPT’s adherence to otolaryngology clinical practice guidelines. European archives of oto-rhino-laryngology: official journal of the European Federation of Oto-Rhino-Laryngological Societies (EUFOS): affiliated with the German Society for Oto-Rhino-Laryngology - Head and Neck Surgery, 281(7):3829–3834. find
23. Francesco Tonolini, Jordan Massiah, Nikolaos Aletras, and Gabriella Kazai. 2024. Bayesian prompt ensembles: Model uncertainty estimation for black-box large language models. find
24. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. arXiv preprint. ArXiv:1706.03762 [cs]. ↗
25. Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, and 1 others. 2023. The rise and potential of large language model based agents: A survey. arXiv preprint arXiv:2309.07864. ↗
26. Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. 2024. Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs. arXiv preprint. ArXiv:2306.13063 [cs]. ↗
27. Fang Zeng, Zhiliang Lyu, Quanzheng Li, and Xiang Li. 2024. Enhancing llms for impression generation in radiology reports through a multi-agent system. arXiv preprint arXiv:2412.06828. ↗
28. Thomas Savage, John Wang, Robert Gallo, Abdessalem Boukil, Vishwesh Patel, Seyed Amir Ahmad Safavi-Naini, Ali Soroush, and Jonathan H Chen. 2025. *Figure 4: Distribution of annotated findings.* find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2026-atya-trustworthy-nlp-low-resource-languages/