Agricultural Named Entity Recognition Based on Semantic Aggregation and Model Distillation

With the development of smart agriculture, automatic question and answer (Q&A) of agricultural knowledge is needed to improve the efficiency of agricultural information acquisition. Agriculture named entity recognition plays a key role in automatic Q&A system, which helps obtaining informati...

Descripción completa

Guardado en:

Detalles Bibliográficos
Autores principales:	LI Liangde, WANG Xiujuan, KANG Mengzhen, HUA Jing, FAN Menghan
Formato:	article
Lenguaje:	EN ZH
Publicado:	Editorial Office of Smart Agriculture 2021
Materias:	distant supervision agriculture knowledge graph agriculture q&a system named entity recognition knowledge distillation deep learning bert bi-lstm Agriculture (General) S1-972 Technology (General) T1-995
Acceso en línea:	https://doaj.org/article/ed6ed6cf20804bce8c1c3edb54763743
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

id	oai:doaj.org-article:ed6ed6cf20804bce8c1c3edb54763743
record_format	dspace
spelling	oai:doaj.org-article:ed6ed6cf20804bce8c1c3edb547637432021-11-17T07:52:00ZAgricultural Named Entity Recognition Based on Semantic Aggregation and Model Distillation2096-809410.12133/j.smartag.2021.3.1.202012-SA001https://doaj.org/article/ed6ed6cf20804bce8c1c3edb547637432021-03-01T00:00:00Zhttp://www.smartag.net.cn/article/2021/2096-8094/2096-8094-2021-3-1-118.shtml https://doaj.org/toc/2096-8094With the development of smart agriculture, automatic question and answer (Q&A) of agricultural knowledge is needed to improve the efficiency of agricultural information acquisition. Agriculture named entity recognition plays a key role in automatic Q&A system, which helps obtaining information, understanding agriculture questions and providing answer from the knowledge graph. Due to the scarcity of labeled ANE data, some existing open agricultural entity recognition models rely on manual features, can reduce the accuracy of entity recognition. In this work, an approach of model distillation was proposed to recognize agricultural named entity data. Firstly, massive agriculture data were leveraged from Internet, an agriculture knowledge graph (AgriKG) was constructed. To overcome the scarcity of labeled named agricultural entity data, weakly named entity recognition label on agricultural texts crawled from the Internet was built with the help of AgriKG. The approach was derived from distant supervision, which was used to solve the scarcity of labeled relation extraction data. Considering the lack of labeled data, pretraining language model was introduced, which is fine tuned with existing labeled data. Secondly, large scale pretraining language model, BERT was used for agriculture named entity recognition and provided a pretty well initial parameters containing a lot of basic language knowledge. Considering that the task of agriculture named entity recognition relied heavily on low-end semantic features but slightly on high-end semantic features, an Attention-based Layer Aggregation mechanism for BERT(BERT-ALA) was designed in this research. The aim of BERT-ALA was to adaptively aggregate the output of multiple hidden layers of BERT. Based on BERT-ALA model, Bidirectional LSTM (BiLSTM) and conditional random field (CRF) were coupled to further improve the recognition precision, giving a BERT-ALA+BiLSTM+CRF model. Bi-LSTM improved BERT's insufficient learning ability of the relative position feature, while conditional random field models the dependencies of entity recognition label. Thirdly, since BERT-ALA+BiLSTM+CRF model was difficult to serve online because of the extremely high time and space complexity, BiLSTM+CRF model was used as student model to distill BERT-ALA+BiLSTM+CRF model. It fitted the BERT-ALA+BiLSTM+CRF model's output of BiLSTM layer and CRF layer. The experiment on the database constructed in the research, as well as two open datasets showed that (1) the macro-F1 of the BERT-ALA + BiLSTM + CRF model was improved by 1% compared to the baseline model BERT + BiLSTM + CRF, and (2) compared with the model trained on the original data, the macro-F1 of the distilled student model BiLSTM + CRF was increased by an average of 3.3%, the prediction time was reduced by 33%, and the storage space was reduced by 98%. The experimental results verify the effectiveness of the BERT-ALA and knowledge distillation in agricultural entity recognition.LI LiangdeWANG XiujuanKANG MengzhenHUA JingFAN MenghanEditorial Office of Smart Agriculturearticledistant supervisionagriculture knowledge graphagriculture q&a systemnamed entity recognitionknowledge distillationdeep learningbertbi-lstmAgriculture (General)S1-972Technology (General)T1-995ENZH智慧农业, Vol 3, Iss 1, Pp 118-128 (2021)
institution	DOAJ
collection	DOAJ
language	EN ZH
topic	distant supervision agriculture knowledge graph agriculture q&a system named entity recognition knowledge distillation deep learning bert bi-lstm Agriculture (General) S1-972 Technology (General) T1-995
spellingShingle	distant supervision agriculture knowledge graph agriculture q&a system named entity recognition knowledge distillation deep learning bert bi-lstm Agriculture (General) S1-972 Technology (General) T1-995 LI Liangde WANG Xiujuan KANG Mengzhen HUA Jing FAN Menghan Agricultural Named Entity Recognition Based on Semantic Aggregation and Model Distillation
description	With the development of smart agriculture, automatic question and answer (Q&A) of agricultural knowledge is needed to improve the efficiency of agricultural information acquisition. Agriculture named entity recognition plays a key role in automatic Q&A system, which helps obtaining information, understanding agriculture questions and providing answer from the knowledge graph. Due to the scarcity of labeled ANE data, some existing open agricultural entity recognition models rely on manual features, can reduce the accuracy of entity recognition. In this work, an approach of model distillation was proposed to recognize agricultural named entity data. Firstly, massive agriculture data were leveraged from Internet, an agriculture knowledge graph (AgriKG) was constructed. To overcome the scarcity of labeled named agricultural entity data, weakly named entity recognition label on agricultural texts crawled from the Internet was built with the help of AgriKG. The approach was derived from distant supervision, which was used to solve the scarcity of labeled relation extraction data. Considering the lack of labeled data, pretraining language model was introduced, which is fine tuned with existing labeled data. Secondly, large scale pretraining language model, BERT was used for agriculture named entity recognition and provided a pretty well initial parameters containing a lot of basic language knowledge. Considering that the task of agriculture named entity recognition relied heavily on low-end semantic features but slightly on high-end semantic features, an Attention-based Layer Aggregation mechanism for BERT(BERT-ALA) was designed in this research. The aim of BERT-ALA was to adaptively aggregate the output of multiple hidden layers of BERT. Based on BERT-ALA model, Bidirectional LSTM (BiLSTM) and conditional random field (CRF) were coupled to further improve the recognition precision, giving a BERT-ALA+BiLSTM+CRF model. Bi-LSTM improved BERT's insufficient learning ability of the relative position feature, while conditional random field models the dependencies of entity recognition label. Thirdly, since BERT-ALA+BiLSTM+CRF model was difficult to serve online because of the extremely high time and space complexity, BiLSTM+CRF model was used as student model to distill BERT-ALA+BiLSTM+CRF model. It fitted the BERT-ALA+BiLSTM+CRF model's output of BiLSTM layer and CRF layer. The experiment on the database constructed in the research, as well as two open datasets showed that (1) the macro-F1 of the BERT-ALA + BiLSTM + CRF model was improved by 1% compared to the baseline model BERT + BiLSTM + CRF, and (2) compared with the model trained on the original data, the macro-F1 of the distilled student model BiLSTM + CRF was increased by an average of 3.3%, the prediction time was reduced by 33%, and the storage space was reduced by 98%. The experimental results verify the effectiveness of the BERT-ALA and knowledge distillation in agricultural entity recognition.
format	article
author	LI Liangde WANG Xiujuan KANG Mengzhen HUA Jing FAN Menghan
author_facet	LI Liangde WANG Xiujuan KANG Mengzhen HUA Jing FAN Menghan
author_sort	LI Liangde
title	Agricultural Named Entity Recognition Based on Semantic Aggregation and Model Distillation
title_short	Agricultural Named Entity Recognition Based on Semantic Aggregation and Model Distillation
title_full	Agricultural Named Entity Recognition Based on Semantic Aggregation and Model Distillation
title_fullStr	Agricultural Named Entity Recognition Based on Semantic Aggregation and Model Distillation
title_full_unstemmed	Agricultural Named Entity Recognition Based on Semantic Aggregation and Model Distillation
title_sort	agricultural named entity recognition based on semantic aggregation and model distillation
publisher	Editorial Office of Smart Agriculture
publishDate	2021
url	https://doaj.org/article/ed6ed6cf20804bce8c1c3edb54763743
work_keys_str_mv	AT liliangde agriculturalnamedentityrecognitionbasedonsemanticaggregationandmodeldistillation AT wangxiujuan agriculturalnamedentityrecognitionbasedonsemanticaggregationandmodeldistillation AT kangmengzhen agriculturalnamedentityrecognitionbasedonsemanticaggregationandmodeldistillation AT huajing agriculturalnamedentityrecognitionbasedonsemanticaggregationandmodeldistillation AT fanmenghan agriculturalnamedentityrecognitionbasedonsemanticaggregationandmodeldistillation
_version_	1718425898045145088

Agricultural Named Entity Recognition Based on Semantic Aggregation and Model Distillation

Ejemplares similares