Optimizing Small BERTs Trained for German NER

Currently, the most widespread neural network architecture for training language models is the so-called BERT, which led to improvements in various Natural Language Processing (NLP) tasks. In general, the larger the number of parameters in a BERT model, the better the results obtained in these NLP t...

Descripción completa

Guardado en:

Detalles Bibliográficos
Autores principales:	Jochen Zöllner, Konrad Sperfeld, Christoph Wick, Roger Labahn
Formato:	article
Lenguaje:	EN
Publicado:	MDPI AG 2021
Materias:	named entity recognition natural language processing BERT German language pre-training fine-tuning Information technology T58.5-58.64
Acceso en línea:	https://doaj.org/article/c269d264e4014373a18286098ba93af0
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

id	oai:doaj.org-article:c269d264e4014373a18286098ba93af0
record_format	dspace
spelling	oai:doaj.org-article:c269d264e4014373a18286098ba93af02021-11-25T17:58:26ZOptimizing Small BERTs Trained for German NER10.3390/info121104432078-2489https://doaj.org/article/c269d264e4014373a18286098ba93af02021-10-01T00:00:00Zhttps://www.mdpi.com/2078-2489/12/11/443https://doaj.org/toc/2078-2489Currently, the most widespread neural network architecture for training language models is the so-called BERT, which led to improvements in various Natural Language Processing (NLP) tasks. In general, the larger the number of parameters in a BERT model, the better the results obtained in these NLP tasks. Unfortunately, the memory consumption and the training duration drastically increases with the size of these models. In this article, we investigate various training techniques of smaller BERT models: We combine different methods from other BERT variants, such as ALBERT, RoBERTa, and relative positional encoding. In addition, we propose two new fine-tuning modifications leading to better performance: Class-Start-End tagging and a modified form of Linear Chain Conditional Random Fields. Furthermore, we introduce Whole-Word Attention, which reduces BERTs memory usage and leads to a small increase in performance compared to classical Multi-Head-Attention. We evaluate these techniques on five public German Named Entity Recognition (NER) tasks, of which two are introduced by this article.Jochen ZöllnerKonrad SperfeldChristoph WickRoger LabahnMDPI AGarticlenamed entity recognitionnatural language processingBERTGerman languagepre-trainingfine-tuningInformation technologyT58.5-58.64ENInformation, Vol 12, Iss 443, p 443 (2021)
institution	DOAJ
collection	DOAJ
language	EN
topic	named entity recognition natural language processing BERT German language pre-training fine-tuning Information technology T58.5-58.64
spellingShingle	named entity recognition natural language processing BERT German language pre-training fine-tuning Information technology T58.5-58.64 Jochen Zöllner Konrad Sperfeld Christoph Wick Roger Labahn Optimizing Small BERTs Trained for German NER
description	Currently, the most widespread neural network architecture for training language models is the so-called BERT, which led to improvements in various Natural Language Processing (NLP) tasks. In general, the larger the number of parameters in a BERT model, the better the results obtained in these NLP tasks. Unfortunately, the memory consumption and the training duration drastically increases with the size of these models. In this article, we investigate various training techniques of smaller BERT models: We combine different methods from other BERT variants, such as ALBERT, RoBERTa, and relative positional encoding. In addition, we propose two new fine-tuning modifications leading to better performance: Class-Start-End tagging and a modified form of Linear Chain Conditional Random Fields. Furthermore, we introduce Whole-Word Attention, which reduces BERTs memory usage and leads to a small increase in performance compared to classical Multi-Head-Attention. We evaluate these techniques on five public German Named Entity Recognition (NER) tasks, of which two are introduced by this article.
format	article
author	Jochen Zöllner Konrad Sperfeld Christoph Wick Roger Labahn
author_facet	Jochen Zöllner Konrad Sperfeld Christoph Wick Roger Labahn
author_sort	Jochen Zöllner
title	Optimizing Small BERTs Trained for German NER
title_short	Optimizing Small BERTs Trained for German NER
title_full	Optimizing Small BERTs Trained for German NER
title_fullStr	Optimizing Small BERTs Trained for German NER
title_full_unstemmed	Optimizing Small BERTs Trained for German NER
title_sort	optimizing small berts trained for german ner
publisher	MDPI AG
publishDate	2021
url	https://doaj.org/article/c269d264e4014373a18286098ba93af0
work_keys_str_mv	AT jochenzollner optimizingsmallbertstrainedforgermanner AT konradsperfeld optimizingsmallbertstrainedforgermanner AT christophwick optimizingsmallbertstrainedforgermanner AT rogerlabahn optimizingsmallbertstrainedforgermanner
_version_	1718411826881888256

Optimizing Small BERTs Trained for German NER

Ejemplares similares