Feature fusion-based collaborative learning for knowledge distillation

Deep neural networks have achieved a great success in a variety of applications, such as self-driving cars and intelligent robotics. Meanwhile, knowledge distillation has received increasing attention as an effective model compression technique for training very efficient deep models. The performanc...

Descripción completa

Guardado en:

Detalles Bibliográficos
Autores principales:	Yiting Li, Liyuan Sun, Jianping Gou, Lan Du, Weihua Ou
Formato:	article
Lenguaje:	EN
Publicado:	SAGE Publishing 2021
Materias:	Electronic computers. Computer science QA75.5-76.95
Acceso en línea:	https://doaj.org/article/e64c67ea766f46eea337ddceeef31f02
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

id	oai:doaj.org-article:e64c67ea766f46eea337ddceeef31f02
record_format	dspace
spelling	oai:doaj.org-article:e64c67ea766f46eea337ddceeef31f022021-12-02T03:05:19ZFeature fusion-based collaborative learning for knowledge distillation1550-147710.1177/15501477211057037https://doaj.org/article/e64c67ea766f46eea337ddceeef31f022021-11-01T00:00:00Zhttps://doi.org/10.1177/15501477211057037https://doaj.org/toc/1550-1477Deep neural networks have achieved a great success in a variety of applications, such as self-driving cars and intelligent robotics. Meanwhile, knowledge distillation has received increasing attention as an effective model compression technique for training very efficient deep models. The performance of the student network obtained through knowledge distillation heavily depends on whether the transfer of the teacher’s knowledge can effectively guide the student training. However, most existing knowledge distillation schemes require a large teacher network pre-trained on large-scale data sets, which can increase the difficulty of knowledge distillation in different applications. In this article, we propose a feature fusion-based collaborative learning for knowledge distillation. Specifically, during knowledge distillation, it enables networks to learn from each other using the feature/response-based knowledge in different network layers. We concatenate the features learned by the teacher and the student networks to obtain a more representative feature map for knowledge transfer. In addition, we also introduce a network regularization method to further improve the model performance by providing a positive knowledge during training. Experiments and ablation studies on two widely used data sets demonstrate that the proposed method, feature fusion-based collaborative learning, significantly outperforms recent state-of-the-art knowledge distillation methods.Yiting LiLiyuan SunJianping GouLan DuWeihua OuSAGE PublishingarticleElectronic computers. Computer scienceQA75.5-76.95ENInternational Journal of Distributed Sensor Networks, Vol 17 (2021)
institution	DOAJ
collection	DOAJ
language	EN
topic	Electronic computers. Computer science QA75.5-76.95
spellingShingle	Electronic computers. Computer science QA75.5-76.95 Yiting Li Liyuan Sun Jianping Gou Lan Du Weihua Ou Feature fusion-based collaborative learning for knowledge distillation
description	Deep neural networks have achieved a great success in a variety of applications, such as self-driving cars and intelligent robotics. Meanwhile, knowledge distillation has received increasing attention as an effective model compression technique for training very efficient deep models. The performance of the student network obtained through knowledge distillation heavily depends on whether the transfer of the teacher’s knowledge can effectively guide the student training. However, most existing knowledge distillation schemes require a large teacher network pre-trained on large-scale data sets, which can increase the difficulty of knowledge distillation in different applications. In this article, we propose a feature fusion-based collaborative learning for knowledge distillation. Specifically, during knowledge distillation, it enables networks to learn from each other using the feature/response-based knowledge in different network layers. We concatenate the features learned by the teacher and the student networks to obtain a more representative feature map for knowledge transfer. In addition, we also introduce a network regularization method to further improve the model performance by providing a positive knowledge during training. Experiments and ablation studies on two widely used data sets demonstrate that the proposed method, feature fusion-based collaborative learning, significantly outperforms recent state-of-the-art knowledge distillation methods.
format	article
author	Yiting Li Liyuan Sun Jianping Gou Lan Du Weihua Ou
author_facet	Yiting Li Liyuan Sun Jianping Gou Lan Du Weihua Ou
author_sort	Yiting Li
title	Feature fusion-based collaborative learning for knowledge distillation
title_short	Feature fusion-based collaborative learning for knowledge distillation
title_full	Feature fusion-based collaborative learning for knowledge distillation
title_fullStr	Feature fusion-based collaborative learning for knowledge distillation
title_full_unstemmed	Feature fusion-based collaborative learning for knowledge distillation
title_sort	feature fusion-based collaborative learning for knowledge distillation
publisher	SAGE Publishing
publishDate	2021
url	https://doaj.org/article/e64c67ea766f46eea337ddceeef31f02
work_keys_str_mv	AT yitingli featurefusionbasedcollaborativelearningforknowledgedistillation AT liyuansun featurefusionbasedcollaborativelearningforknowledgedistillation AT jianpinggou featurefusionbasedcollaborativelearningforknowledgedistillation AT landu featurefusionbasedcollaborativelearningforknowledgedistillation AT weihuaou featurefusionbasedcollaborativelearningforknowledgedistillation
_version_	1718402013552705536

Feature fusion-based collaborative learning for knowledge distillation

Ejemplares similares