Mevaker: Conclusion Extraction and Allocation Resources for the Hebrew Language

What it is about

The authors build Hebrew datasets from 1,109 State Comptroller audit reports (2005–2022). The conclusions are marked by the auditors, so their spans give labels for free. They derive MevakerConcSen (sentence labels), MevakerConcTree (conclusions attached to a heading tree) and MevakerSumm (1,106 reports with abstracts). They train HeConE (one special token per sentence) and HeConEspc (sliding windows of five sentences) on HeRo. Sentence-level F1 is 84.10 and 90.83 respectively. For allocation they train HeCross, a cross-encoder over the heading path and the conclusion. It reaches Kendall 0.0647 against 0.0094 for the best English bi-encoder on a common ranking test. The authors note that window-based training leaks between splits unless shuffling is disabled, and that the minority class is duplicated in training only.

Ideas

• I1. A rule-based conclusion cue: list the Hebrew cue phrases and imperatives that open audit conclusions (recommendations, "the state comptroller found"), write them as a finite-state pattern, and measure its precision and recall against the labelled MevakerConcSen sentences. This is a baseline the neural models must beat.
• I2. A split rule for any test set with overlapping context windows: split by document, never by window, and check the number of shared sentences across the train and test sides. The paper's leakage problem is the reason for this rule.
• I3. Heading allocation as a baseline: attach an unlabelled conclusion to the deepest heading whose lexemes overlap most with the conclusion, in a deterministic score, and measure it on the MevakerConcTree-style structure.

For Mova

I1 is a discourse rule for prag, with the cue list in 8. Lexicon and register (official register). I2 is a rule for every split Mova makes. I3 is a world baseline for document structure.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. arXiv 2403.09719

@misc{shalumov2024mevaker,
  title = {{Mevaker: Conclusion Extraction and Allocation Resources for the Hebrew Language}},
  author = {Shalumov, Vitaly and Haskey, Harel and Solaz, Yuval},
  year = {2024},
  eprint = {2403.09719},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2403.09719}
}

References

13 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Adam Amram, Anat Ben David, and Reut Tsarfaty. 2018. Representations and architectures in neural sentiment analysis for morphologically rich languages: A case study from modern hebrew. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2242– 2252. find our card
2. Dan Bareket and Reut Tsarfaty. 2021. Neural modeling for named entities and morphology (nemoˆ2). Transactions of the Association for Computational Linguistics, 9:909– 928. find
3. Amir Cohen, Hilla Merhav-Fine, Yoav Goldberg, and Reut Tsarfaty. 2023. Heq: a large and diverse hebrew reading comprehension benchmark. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 13693–13705. find our card
4. Maurice G Kendall. 1938. A new measure of rank correlation. Biometrika, 30(1/2):81– 93. find
5. Omri Keren and Omer Levy. 2021. Parashoot: A hebrew question answering dataset. arXiv preprint arXiv:2109.11314. ↗ our card
6. Mahnaz Koupaee and William Yang Wang. 2018. Wikihow: A large scale text summarization dataset. find
7. Michal Lukasik, Boris Dadachev, Gonçalo Simoes, and Kishore Papineni. 2020. Text segmentation by cross segment attention. arXiv preprint arXiv:2004.14535. ↗
8. Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. 2016. Abstractive text summarization using sequence-to-sequence rnns and beyond. arXiv preprint arXiv:1602.06023. ↗
9. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics. find
10. Nils Reimers and Iryna Gurevych. 2020. Making monolingual sentence embeddings multilingual using knowledge distillation. arXiv preprint arXiv:2004.09813. ↗
11. Vitaly Shalumov and Harel Haskey. 2023. Hero: Roberta and longformer hebrew language models. find our card
12. Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2022. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533. ↗
13. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pages 38–45. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/langstream-2024-shalumov-mevaker-conclusion-extraction-allocation-resources/