HeSum: a Novel Dataset for Abstractive Text Summarization in Hebrew

What it is about

The paper builds HeSum, 10,000 Modern Hebrew news articles paired with the editor-written extended subheadings that serve as summaries, taken from three independent news sites and split 8,000 / 1,000 / 1,000. The authors measure how abstractive these summaries are (share of novel n-grams, compression, redundancy) and compare them with CNN/Daily Mail and XSum. Hebrew adds difficulty: the paper reports a high word ambiguity per article (many alternative segmentations) and frequent morphological anaphors and construct-state nouns. Zero-shot GPT-4 and GPT-3.5 are compared with a fine-tuned mLongT5: mLongT5 gets the highest ROUGE, while GPT-4 gets the best BERTScore (77.3 against 57.6 for mLongT5) and the best human coherence and completeness scores on a 1–5 scale. Error analysis (30 summaries per model) finds repetition, copying, wrong disambiguation, gender errors and hallucinated names.

Ideas

• I1. Report the Hebrew segmentation ambiguity the way the paper does: for each article, count the lattice paths that the segmenter produces, and report the average per split. Run the same count on Hebrew news text, so the difficulty of he input is a number, not a feeling.
• I2. Turn the error classes of the paper's Table 4 (construct state after a definite article, feminine inflection on a masculine referent, missing diacritics that change a word's reading) into deterministic checks over the he morphology: agreement of gender and number between a noun and its predicate, and the definite marking before a construct state.
• I3. A held-out check for a future Hebrew summary built from the parse: the share of summary content lemmas that occur in the article's frames (the world of the text). It needs no generation; it is a coverage measure, and the paper's own split is kept held out.

For Mova

I1 goes into 2. Script and spelling (the ambiguity note) and is a measure for 12. Mova and Modern Hebrew: where it stands, what it would need. I2 goes into Nouns, adjectives, pronouns and numerals as test cases. I3 is a measure in 12. Mova and Modern Hebrew: where it stands, what it would need, run on the open HeSum split only as a test set, never as training data.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.loresmt-1.2

@inproceedings{mondshine2024hesum,
  title = {{HeSum: a Novel Dataset for Abstractive Text Summarization in Hebrew}},
  author = {Mondshine, Itai and Paz-Argaman, Tzuf and Mordechai, Asaf Achi and Tsarfaty, Reut},
  booktitle = {Proceedings of the Seventh Workshop on Technologies for Machine Translation of Low-Resource Languages (LoResMT 2024)},
  year = {2024},
  doi = {10.18653/v1/2024.loresmt-1.2},
  url = {https://doi.org/10.18653/v1/2024.loresmt-1.2}
}

References

4 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Kabir Ahuja, Rishav Hada, Millicent Ochieng, Prachi Jain, Harshita Diddee, Samuel Maina, Tanuja Ganu, Sameer Segal, Maxamed Axmed, Kalika Bali, et al. 2023. Mega: Multilingual evaluation of generative ai. arXiv preprint arXiv:2303.12528. ↗
2. Google Translate API. 2023. Google translate api v2 documentation. find
3. Dan Bareket and Reut Tsarfaty. 2021. Neural modeling for named entities and morphology (nemo^2). Transactions of the Association for Computational Linguistics, 9:909–928. find
4. Ethics Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. 2020. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432–7439. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/lingbulk-2024-mondshine-hesum-novel-dataset-abstractive-text/