NAS<span style="font-variant: small-caps">ca</span> and NAS<span style="font-variant: small-caps">es</span>: Two Monolingual Pre-Trained Models for Abstractive Summarization in Catalan and Spanish

Most of the models proposed in the literature for abstractive summarization are generally suitable for the English language but not for other languages. Multilingual models were introduced to address that language constraint, but despite their applicability being broader than that of the monolingual...

Descripción completa

Guardado en:

Detalles Bibliográficos
Autores principales:	Vicent Ahuir, Lluís-F. Hurtado, José Ángel González, Encarna Segarra
Formato:	article
Lenguaje:	EN
Publicado:	MDPI AG 2021
Materias:	abstractive summarization monolingual models multilingual models transformer models transfer learning Technology T Engineering (General). Civil engineering (General) TA1-2040 Biology (General) QH301-705.5 Physics QC1-999 Chemistry QD1-999
Acceso en línea:	https://doaj.org/article/369bc8bd33b14dd2bfafc4e2fb9f6daf
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

id	oai:doaj.org-article:369bc8bd33b14dd2bfafc4e2fb9f6daf
record_format	dspace
spelling	oai:doaj.org-article:369bc8bd33b14dd2bfafc4e2fb9f6daf2021-11-11T14:59:36ZNAS<span style="font-variant: small-caps">ca</span> and NAS<span style="font-variant: small-caps">es</span>: Two Monolingual Pre-Trained Models for Abstractive Summarization in Catalan and Spanish10.3390/app112198722076-3417https://doaj.org/article/369bc8bd33b14dd2bfafc4e2fb9f6daf2021-10-01T00:00:00Zhttps://www.mdpi.com/2076-3417/11/21/9872https://doaj.org/toc/2076-3417Most of the models proposed in the literature for abstractive summarization are generally suitable for the English language but not for other languages. Multilingual models were introduced to address that language constraint, but despite their applicability being broader than that of the monolingual models, their performance is typically lower, especially for minority languages like Catalan. In this paper, we present a monolingual model for abstractive summarization of textual content in the Catalan language. The model is a Transformer encoder-decoder which is pretrained and fine-tuned specifically for the Catalan language using a corpus of newspaper articles. In the pretraining phase, we introduced several self-supervised tasks to specialize the model on the summarization task and to increase the abstractivity of the generated summaries. To study the performance of our proposal in languages with higher resources than Catalan, we replicate the model and the experimentation for the Spanish language. The usual evaluation metrics, not only the most used ROUGE measure but also other more semantic ones such as BertScore, do not allow to correctly evaluate the abstractivity of the generated summaries. In this work, we also present a new metric, called <i>content reordering</i>, to evaluate one of the most common characteristics of abstractive summaries, the rearrangement of the original content. We carried out an exhaustive experimentation to compare the performance of the monolingual models proposed in this work with two of the most widely used multilingual models in text summarization, mBART and mT5. The experimentation results support the quality of our monolingual models, especially considering that the multilingual models were pretrained with many more resources than those used in our models. Likewise, it is shown that the pretraining tasks helped to increase the degree of abstractivity of the generated summaries. To our knowledge, this is the first work that explores a monolingual approach for abstractive summarization both in Catalan and Spanish.Vicent AhuirLluís-F. HurtadoJosé Ángel GonzálezEncarna SegarraMDPI AGarticleabstractive summarizationmonolingual modelsmultilingual modelstransformer modelstransfer learningTechnologyTEngineering (General). Civil engineering (General)TA1-2040Biology (General)QH301-705.5PhysicsQC1-999ChemistryQD1-999ENApplied Sciences, Vol 11, Iss 9872, p 9872 (2021)
institution	DOAJ
collection	DOAJ
language	EN
topic	abstractive summarization monolingual models multilingual models transformer models transfer learning Technology T Engineering (General). Civil engineering (General) TA1-2040 Biology (General) QH301-705.5 Physics QC1-999 Chemistry QD1-999
spellingShingle	abstractive summarization monolingual models multilingual models transformer models transfer learning Technology T Engineering (General). Civil engineering (General) TA1-2040 Biology (General) QH301-705.5 Physics QC1-999 Chemistry QD1-999 Vicent Ahuir Lluís-F. Hurtado José Ángel González Encarna Segarra NAS<span style="font-variant: small-caps">ca</span> and NAS<span style="font-variant: small-caps">es</span>: Two Monolingual Pre-Trained Models for Abstractive Summarization in Catalan and Spanish
description	Most of the models proposed in the literature for abstractive summarization are generally suitable for the English language but not for other languages. Multilingual models were introduced to address that language constraint, but despite their applicability being broader than that of the monolingual models, their performance is typically lower, especially for minority languages like Catalan. In this paper, we present a monolingual model for abstractive summarization of textual content in the Catalan language. The model is a Transformer encoder-decoder which is pretrained and fine-tuned specifically for the Catalan language using a corpus of newspaper articles. In the pretraining phase, we introduced several self-supervised tasks to specialize the model on the summarization task and to increase the abstractivity of the generated summaries. To study the performance of our proposal in languages with higher resources than Catalan, we replicate the model and the experimentation for the Spanish language. The usual evaluation metrics, not only the most used ROUGE measure but also other more semantic ones such as BertScore, do not allow to correctly evaluate the abstractivity of the generated summaries. In this work, we also present a new metric, called <i>content reordering</i>, to evaluate one of the most common characteristics of abstractive summaries, the rearrangement of the original content. We carried out an exhaustive experimentation to compare the performance of the monolingual models proposed in this work with two of the most widely used multilingual models in text summarization, mBART and mT5. The experimentation results support the quality of our monolingual models, especially considering that the multilingual models were pretrained with many more resources than those used in our models. Likewise, it is shown that the pretraining tasks helped to increase the degree of abstractivity of the generated summaries. To our knowledge, this is the first work that explores a monolingual approach for abstractive summarization both in Catalan and Spanish.
format	article
author	Vicent Ahuir Lluís-F. Hurtado José Ángel González Encarna Segarra
author_facet	Vicent Ahuir Lluís-F. Hurtado José Ángel González Encarna Segarra
author_sort	Vicent Ahuir
title	NAS<span style="font-variant: small-caps">ca</span> and NAS<span style="font-variant: small-caps">es</span>: Two Monolingual Pre-Trained Models for Abstractive Summarization in Catalan and Spanish
title_short	NAS<span style="font-variant: small-caps">ca</span> and NAS<span style="font-variant: small-caps">es</span>: Two Monolingual Pre-Trained Models for Abstractive Summarization in Catalan and Spanish
title_full	NAS<span style="font-variant: small-caps">ca</span> and NAS<span style="font-variant: small-caps">es</span>: Two Monolingual Pre-Trained Models for Abstractive Summarization in Catalan and Spanish
title_fullStr	NAS<span style="font-variant: small-caps">ca</span> and NAS<span style="font-variant: small-caps">es</span>: Two Monolingual Pre-Trained Models for Abstractive Summarization in Catalan and Spanish
title_full_unstemmed	NAS<span style="font-variant: small-caps">ca</span> and NAS<span style="font-variant: small-caps">es</span>: Two Monolingual Pre-Trained Models for Abstractive Summarization in Catalan and Spanish
title_sort	nas<span style="font-variant: small-caps">ca</span> and nas<span style="font-variant: small-caps">es</span>: two monolingual pre-trained models for abstractive summarization in catalan and spanish
publisher	MDPI AG
publishDate	2021
url	https://doaj.org/article/369bc8bd33b14dd2bfafc4e2fb9f6daf
work_keys_str_mv	AT vicentahuir nasspanstylefontvariantsmallcapscaspanandnasspanstylefontvariantsmallcapsesspantwomonolingualpretrainedmodelsforabstractivesummarizationincatalanandspanish AT lluisfhurtado nasspanstylefontvariantsmallcapscaspanandnasspanstylefontvariantsmallcapsesspantwomonolingualpretrainedmodelsforabstractivesummarizationincatalanandspanish AT joseangelgonzalez nasspanstylefontvariantsmallcapscaspanandnasspanstylefontvariantsmallcapsesspantwomonolingualpretrainedmodelsforabstractivesummarizationincatalanandspanish AT encarnasegarra nasspanstylefontvariantsmallcapscaspanandnasspanstylefontvariantsmallcapsesspantwomonolingualpretrainedmodelsforabstractivesummarizationincatalanandspanish
_version_	1718437876047282176

NAS<span style="font-variant: small-caps">ca</span> and NAS<span style="font-variant: small-caps">es</span>: Two Monolingual Pre-Trained Models for Abstractive Summarization in Catalan and Spanish

Ejemplares similares