A Lemma Based Evaluator for Semitic Language Text Summarization Systems

What it is about

The paper asks how to score automatic summaries of Arabic when the same meaning is written in many inflected forms. The authors extend ROUGE, the n-gram recall measure used in summarisation. Two changes are made: each word is first reduced to its lemma by an Arabic lemmatiser, and the n-grams are counted over tokens instead of characters. The method is tested on the Arabic part of the TAC 2011 summarisation data (ten clusters of news articles, with system and human summaries). Lemma-based ROUGE-2 and ROUGE-S scores are higher than native ROUGE for the systems listed, for example CLASSY reaches 0.1529 against 0.1442 for ROUGE-2 F-measure. The authors also argue that lemmas group forms better than roots (too coarse) or stems (too fine), and they report that adding the lemmatiser raises scores for every system.

Ideas

• I1. An evaluator for Mova's own output: compare a generated Hebrew sentence with a reference at lemma level, using the macula dict lemma column, and count matches per token. This is a deterministic metric for paraphrase tests in The lexicon.
• I2. Use their two-step pipeline (lemmatise both texts, then score) as a regression test for tools/hbo: the output of the segmenter and parser should give the same lemma sequence after normalisation, with a measured match rate.
• I3. Report ROUGE-2 and ROUGE-S together, as they do, for any summary or RAG answer that Mova builds from world; the skip-bigram score catches reorderings that bigram overlap misses.

For Mova

I1 is a measure for the answer checks of prag and world (English today, Hebrew later). I2 is a test set and a regression check for tools/hbo and macula compare. I3 is an evaluation rule for any summary feature; it is not a translation of their lemmatiser, which Mova does not need because macula already gives lemmas.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. arXiv 1403.5596

@misc{elshishtawy2014lemma,
  title = {{A Lemma Based Evaluator for Semitic Language Text Summarization Systems}},
  author = {El-Shishtawy, Tarek and El-Ghannam, Fatma},
  year = {2014},
  eprint = {1403.5596},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1403.5596}
}

References

11 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Al-Jlayl, M., Frieder O., 2002 . "On Arabic Search: Improving the Retrieval Effectiveness via Light Stemming Approach". In Proceedings of the 11th ACM International Conference on Information and Knowledge Management, Illinois Institute of Technology, New York: ACM Press, pp.340-347. Native Rouge2 F-Measure 0.1442 0.1099 0.0871 0.1129 0.0711 Native RougeS F-Measure 0.1302 0.1213 0.1123 0.0786 0.0640 We extended ROUGE find
2. Dichy J. “On lemmatization in Arabic. A formal definition of the Arabic entries of multilingual lexical databases”. ACL 39th Annual Meeting. Workshop on Arabic Language Processing; Status and Prospect, Toulouse, 2001, 23-30. find
3. Dichy J, and A.Fargaly, "Roots & Patterns vs. Stems plus Grammar-Lexis Specifications: on what basis should a multilingual lexical database centred on Arabic be built?, Proceedings of the MTSummit, New-Orleans,2003. find
4. El-Ghannam F. and El-Shishtawy T., Multi-Topic Multi-Document Summarizer, International Journal of Computer Science & Information Technology, Vol. 5 Issue 6, Dec 2013. find
5. El-Shishtawy T. and Al-sammak A., Arabic Keyphrase Extraction using Linguistic knowledge and Machine Learning Techniques, Proceedings of the Second International Conference on Arabic Language Resources and Tools, 2009. find
6. El-Shishtawy T. and El-Ghannam F. Keyphrase based Arabic summarizer (KPAS), ieeexplore.ieee.org. The 8th INFOS2012 International Conference on Informatics and Systems, 14-16 May, 2012. find
7. El-Shishtawy T. and El-Ghannam F., An Accurate Arabic Root-Based Lemmatizer for Information Retrieval Purpose, International Journal of Computer Science Issues (IJCSI), Vol. 9 Issue 1, Jan 2012, find
8. El-Shishtawy T., A Hybrid Algorithm for Matching Arabic Names, International Journal of Computational Linguistics Research Volume 4 Number 2 June 2013 find
9. El-Shishtawy T., Linking Databases using Matched Arabic Names, Computational Linguistics and Chinese Language Processing, Vol. 19, No. 1, March 2014. find
10. Lin, C. “Training a Selection Function for Extraction” in Pvoc. of the 8th International Conference on Knowledge Management. Kansas City, MO USA: ACM Press, 1999, find
11. Yingbo L, Bernard M, " VERT: Automatic Evaluation of Video Summaries", Proceedings of the international conference on Multimedia, pp. 851-854, 2010. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/langstream-2014-el-shishtawy-lemma-based-evaluator-semitic-language/