JudgeMeNot: Personalizing Large Language Models to Emulate Judicial Reasoning in Hebrew

What it is about

The authors personalise LLMs to individual Israeli judges, using the summary judgments of 29 magistrate and district judges (at least 100 each) from the Nevo database of public verdicts. An agentic pipeline (GPT-4.1-mini for extraction, GPT-4o-mini for validation) turns them into 62,051 reasoning sentences with synthetic questions; two annotators found 83% of the extracted statements explicitly reasoned and 91% of the questions aligned. Per-judge QLoRA on Gemma 3 is compared with raw causal-LM training, instruction tuning, a chain of the two (CoLA) and retrieval. CoLA with the 4B model gives the best overlap and semantic scores, and a judge-specific classifier falls to chance (49.8%) on its output, against 84.3% when the negatives are other judges' real sentences. HellaSwag drops only from 66.4% to 64.9%.

Ideas

• I1. Structure before volume: the same verdicts turned into question–reasoning pairs train better than raw text. Mova could mark reasoning-cue sentences in Hebrew court text with a deterministic extractor (summary section, discourse markers) and write them as frames, then measure precision on a hand-labelled sample of about 175 pairs, as the paper did.
• I2. Report two style scores together: the Jensen–Shannon divergence over POS tags is deterministic and needs only a tagger, and the paper shows it can move separately from lexical scores (RAG improved POS while losing reasoning content).
• I3. A next-step test for the event chain: given one to three reasoning sentences, choose the true next sentence among candidates from the same case. No model in the paper did well, so it is a useful hard test for Mova's world event order.

For Mova

I1 is a rule for Hebrew legal register in 8. Lexicon and register. I2 is a measure for any Hebrew generator Mova builds. I3 is a test for the event chain in world. Check the licence and source of the verdicts before any of it is used as data.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1332

@inproceedings{razumenko2026judgemenot,
  title = {{JudgeMeNot: Personalizing Large Language Models to Emulate Judicial Reasoning in Hebrew}},
  author = {Razumenko, Itay and Sturm, Arnon and Grinberg, Nir},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1332},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1332}
}

References

45 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Elliott Ash, Germain Gauthier, and Philine Widmer. 2024. Relatio: Text semantics capture political and economic narratives. Political Analysis, 32(1):115– 132. find
2. J Elliott Casal and Matt Kessler. 2023. Can linguists distinguish between chatgpt/ai and human writing?: A study of research ethics and academic publishing. Research Methods in Applied Linguistics, 2(3):100068. find
3. Shu Chen, Xinyan Guan, Yaojie Lu, Hongyu Lin, Xianpei Han, and Le Sun. 2024. REInstruct: Building instruction data from unlabeled corpus. In Findings of the Association for Computational Linguistics: ACL 2024, pages 6840–6856, Bangkok, Thailand. Association for Computational Linguistics. find
4. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: efficient finetuning of quantized llms. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA. Curran Associates Inc. find
5. Anca Dinu, Andra-Maria Florescu, and Liviu Dinu. 2025. Analyzing large language models’ pastiche ability: a case study on a 20th century Romanian author. In Proceedings of the 5th International Conference on Natural Language Processing for Digital Humanities, pages 20–32, Albuquerque, USA. Association for Computational Linguistics. find
6. Lee Epstein, William M Landes, and Richard A Posner. 2013. The behavior of federal judges: a theoretical and empirical study of rational choice. harvard university press. find
7. Deva Kumar Gajulamandyam, Sainath Veerla, Yasaman Emami, Kichang Lee, Yuanting Li, Jinthy Swetha Mamillapalli, and Simon Shim. 2025. Domain specific finetuning of llms using peft techniques. In 2025 IEEE 15th Annual Computing and Communication Workshop and Conference (CCWC), pages 00484–00490. IEEE. find
8. Gemini 3 pro Google DeepMind. 2025. model card. Model card. https://storage.googleapis.com/ deepmind-media/Model-Cards/ Gemini-3-Pro-Model-Card.pdf. ↗
9. Neel Guha, Julian Nyarko, Daniel Ho, Christopher Ré, Adam Chilton, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel Rockmore, Diego Zambrano, and 1 others. 2023. Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models. Advances in neural information processing systems, 36:44123–44279. find
10. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, and 1 others. 2025a. Deepseek-r1 incentivizes reasoning in llms through reinforcement learning. Nature, 645(8081):633– 638. find
11. Yanzhu Guo, Guokan Shang, and Chloé Clavel. 2025b. Benchmarking linguistic diversity of large language models. Transactions of the Association for Computational Linguistics, 13:1507–1526. find
12. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. International Conference on Machine Learning, pages 2790–2799. find
13. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, and 1 others. 2022. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3. find
14. Jie Huang and Kevin Chen-Chuan Chang. 2023. Towards reasoning in large language models: A survey. In Findings of the Association for Computational Linguistics: ACL 2023, pages 1049–1065, Toronto, Canada. Association for Computational Linguistics. find
15. Moshe Koppel, Jonathan Schler, and Shlomo Argamon. 2009. Computational methods in authorship attribution. Journal of the American Society for information Science and Technology, 60(1):9–26. find
16. Kalpesh Krishna, John Wieting, and Mohit Iyyer. 2020. Reformulating unsupervised style transfer as paraphrase generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 737–762, Online. Association for Computational Linguistics. find our card
17. Cheng Li, Ziang Leng, Chenxi Yan, Junyi Shen, Hao Wang, Weishi Mi, Yaying Fei, Xiaoyang Feng, Song Yan, HaoSheng Wang, and 1 others. 2023. Chatharuhi: Reviving anime character in reality via large language model. arXiv preprint arXiv:2308.09597. ↗
18. Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. Let’s verify step by step. In The Twelfth International Conference on Learning Representations. find
19. Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2024. Let’s verify step by step. In The Twelfth International Conference on Learning Representations. find
20. Xinyue Liu, Harshita Diddee, and Daphne Ippolito. Customizing large language model gen-2024. eration style using parameter-efficient finetuning. In Proceedings of the 17th International Natural Language Generation Conference, pages 412–426, Tokyo, Japan. Association for Computational Linguistics. find
21. Gianluca Moro, Leonardo David Matteo Magnani, and Luca Ragazzi. 2026. Legal lay summarization: exploring methods and data generation with large language models. Artificial Intelligence Review, 59(1):21. find
22. Lev Muchnik, Inbal Yahav, Ariel Nevo, Avichay Chriqui, and Tim Shektov. 2023. The israeli supreme court dataset. https://huggingface.co/datasets/ LevMuchnik/SupremeCourtOfIsrael. Hugging Face dataset. ↗
23. Nihal Nayak, Yiyang Nan, Avi Trost, and Stephen Bach. 2024. Learning to generate instruction tuning datasets for zero-shot task adaptation. In Findings of the Association for Computational Linguistics: ACL 2024, pages 12585–12611, Bangkok, Thailand. Association for Computational Linguistics. find
24. Abhiman Neelakanteswara, Shreyas Chaudhari, and Hamed Zamani. 2024. RAGs to style: Personalizing LLMs with style embeddings. In Proceedings of the 1st Workshop on Personalization of Generative AI Systems (PERSONALIZE 2024), pages 119–123, St. Julians, Malta. Association for Computational Linguistics. find
25. Daisuke Oba, Naoki Yoshinaga, and Masashi Toyoda. 2023. Perplm: Personalized fine-tuning of pre-trained language models via writer-specific intermediate learning and prompts. arXiv. find
26. Jeiyoon Park, Chanjun Park, and Heuiseok Lim. 2025. CharacterGPT: A persona reconstruction framework for role-playing agents. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 3: Industry Track), pages 287–303, Albuquerque, New Mexico. Association for Computational Linguistics. find
27. Joon Sung Park, Carolyn Q Zou, Aaron Shaw, Benjamin Mako Hill, Carrie Cai, Meredith Ringel Morris, Robb Willer, Percy Liang, and Michael S Bernstein. 2024. Generative agent simulations of 1,000 people. arXiv preprint arXiv:2411.10109. ↗
28. R.A. Posner. 2008. How Judges Think. Harvard University Press. find
29. Jaromir Savelka. 2023. Unlocking practical applications in legal domain: Evaluation of gpt for zero-shot semantic annotation of legal texts. In Proceedings of the Nineteenth International Conference on Artificial Intelligence and Law, pages 447–451. find
30. Jaromír Šavelka and Kevin D Ashley. 2018. Segmenting us court decisions into functional and issue specific parts. In Legal knowledge and information systems, pages 111–120. IOS Press. find
31. Jaromir Savelka, Kevin D Ashley, Morgan A Gray, Hannes Westermann, and Huihui Xu. 2023. Can gpt-4 support analysis of textual data in tasks requiring highly specialized domain expertise? arXiv. find
32. Yunfan Shao, Linyang Li, Junqi Dai, and Xipeng Qiu. 2023. Character-LLM: A trainable agent for role-playing. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13153–13187, Singapore. Association for Computational Linguistics. find
33. Shaltiel Shmidman, Avi Shmidman, Amir DN Cohen, and Moshe Koppel. 2025. Dicta-LM 3.0: Advancing The Frontier of Hebrew Sovereign LLMs. Technical report, DICTA, Jerusalem, Israel. find our card
34. Shaltiel Shmidman, Avi Shmidman, and Moshe Koppel. 2023. Dictabert: A state-of-the-art bert suite for modern hebrew. arXiv. find our card
35. Cass R Sunstein, David Schkade, Lisa Ellman, and Andres Sawicki. 2006. Are judges political?: an empirical analysis of the federal judiciary. Brookings Institution Press. find
36. Zhaoxuan Tan, Qingkai Zeng, Yijun Tian, Zheyuan Liu, Bing Yin, and Meng Jiang. 2025. Democratizing large language models via personalized parameter-efficient fine-tuning. Preprint, arXiv:2402.04401. ↗
37. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, and 1 others. 2025. Gemma 3 technical report. arXiv preprint arXiv:2503.19786. ↗
38. Asahi Ushio, Fernando Alva-Manchego, and Jose Camacho-Collados. 2023. An empirical comparison of LM-based question and answer generation methods. In Findings of the Association for Computational Linguistics: ACL 2023, pages 14262– 14272, Toronto, Canada. Association for Computational Linguistics. find
39. Zhengxiang Wang, Nafis Irtiza Tripto, Solha Park, Zhenzhen Li, and Jiawei Zhou. 2025. Catch me if you can? not yet: LLMs still struggle to imitate the implicit writing styles of everyday authors. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 10040–10055, Suzhou, China. Association for Computational Linguistics. find
40. Nirmalie Wiratunga, Ramitha Abeyratne, Lasal Jayawardena, Kyle Martin, Stewart Massie, Ikechukwu Nkisi-Orji, Ruvan Weerasinghe, Anne Liret, and Bruno Fleisch. 2024. Cbr-rag: Case-based reasoning for retrieval augmented generation in llms for legal question answering. In International Conference on Case-Based Reasoning, pages 445–460. Springer. find
41. Wenhan Xia, Chengwei Qin, and Elad Hazan. 2024. Chain of lora: Efficient fine-tuning of language models via residual learning. arXiv preprint arXiv:2401.04151. ↗
42. Yiyan Xu, Jinghao Zhang, Alireza Salemi, Xinting Hu, Wenjie Wang, Fuli Feng, Hamed Zamani, Xiangnan He, and Tat-Seng Chua. 2025. Personalized generation in large model era: A survey. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 24607–24649, Vienna, Austria. Association for Computational Linguistics. find
43. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th annual meeting of the association for computational linguistics, pages 4791–4800. find
44. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675. ↗
45. Zhehao Zhang, Ryan A Rossi, Branislav Kveton, Yijia Shao, Diyi Yang, Hamed Zamani, Franck Dernoncourt, Joe Barrow, Tong Yu, Sungchul Kim, and 1 others. 2024. Personalization of large language models: A survey. arXiv. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/findings-2026-razumenko-judgemenot-personalizing-large-language-models/