Is One Teacher Model Enough to Transfer Knowledge to a Student Model?

Nowadays, the transfer learning technique can be successfully applied in the deep learning field through techniques that fine-tune the CNN’s starting point so it may learn over a huge dataset such as ImageNet and continue to learn on a fixed dataset to achieve better performance. In this paper, we d...

Descripción completa

Guardado en:

Detalles Bibliográficos
Autores principales:	Nicola Landro, Ignazio Gallo, Riccardo La Grassa
Formato:	article
Lenguaje:	EN
Publicado:	MDPI AG 2021
Materias:	deep learning transfer learning loss functions Industrial engineering. Management engineering T55.4-60.8 Electronic computers. Computer science QA75.5-76.95
Acceso en línea:	https://doaj.org/article/b5508f2d11a04b6db13a9ff530052572
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

id	oai:doaj.org-article:b5508f2d11a04b6db13a9ff530052572
record_format	dspace
spelling	oai:doaj.org-article:b5508f2d11a04b6db13a9ff5300525722021-11-25T16:13:20ZIs One Teacher Model Enough to Transfer Knowledge to a Student Model?10.3390/a141103341999-4893https://doaj.org/article/b5508f2d11a04b6db13a9ff5300525722021-11-01T00:00:00Zhttps://www.mdpi.com/1999-4893/14/11/334https://doaj.org/toc/1999-4893Nowadays, the transfer learning technique can be successfully applied in the deep learning field through techniques that fine-tune the CNN’s starting point so it may learn over a huge dataset such as ImageNet and continue to learn on a fixed dataset to achieve better performance. In this paper, we designed a transfer learning methodology that combines the learned features of different teachers to a student network in an end-to-end model, improving the performance of the student network in classification tasks over different datasets. In addition to this, we tried to answer the following questions which are in any case directly related to the transfer learning problem addressed here. Is it possible to improve the performance of a small neural network by using the knowledge gained from a more powerful neural network? Can a deep neural network outperform the teacher using transfer learning? Experimental results suggest that neural networks can transfer their learning to student networks using our proposed architecture, designed to bring to light a new interesting approach for transfer learning techniques. Finally, we provide details of the code and the experimental settings.Nicola LandroIgnazio GalloRiccardo La GrassaMDPI AGarticledeep learningtransfer learningloss functionsIndustrial engineering. Management engineeringT55.4-60.8Electronic computers. Computer scienceQA75.5-76.95ENAlgorithms, Vol 14, Iss 334, p 334 (2021)
institution	DOAJ
collection	DOAJ
language	EN
topic	deep learning transfer learning loss functions Industrial engineering. Management engineering T55.4-60.8 Electronic computers. Computer science QA75.5-76.95
spellingShingle	deep learning transfer learning loss functions Industrial engineering. Management engineering T55.4-60.8 Electronic computers. Computer science QA75.5-76.95 Nicola Landro Ignazio Gallo Riccardo La Grassa Is One Teacher Model Enough to Transfer Knowledge to a Student Model?
description	Nowadays, the transfer learning technique can be successfully applied in the deep learning field through techniques that fine-tune the CNN’s starting point so it may learn over a huge dataset such as ImageNet and continue to learn on a fixed dataset to achieve better performance. In this paper, we designed a transfer learning methodology that combines the learned features of different teachers to a student network in an end-to-end model, improving the performance of the student network in classification tasks over different datasets. In addition to this, we tried to answer the following questions which are in any case directly related to the transfer learning problem addressed here. Is it possible to improve the performance of a small neural network by using the knowledge gained from a more powerful neural network? Can a deep neural network outperform the teacher using transfer learning? Experimental results suggest that neural networks can transfer their learning to student networks using our proposed architecture, designed to bring to light a new interesting approach for transfer learning techniques. Finally, we provide details of the code and the experimental settings.
format	article
author	Nicola Landro Ignazio Gallo Riccardo La Grassa
author_facet	Nicola Landro Ignazio Gallo Riccardo La Grassa
author_sort	Nicola Landro
title	Is One Teacher Model Enough to Transfer Knowledge to a Student Model?
title_short	Is One Teacher Model Enough to Transfer Knowledge to a Student Model?
title_full	Is One Teacher Model Enough to Transfer Knowledge to a Student Model?
title_fullStr	Is One Teacher Model Enough to Transfer Knowledge to a Student Model?
title_full_unstemmed	Is One Teacher Model Enough to Transfer Knowledge to a Student Model?
title_sort	is one teacher model enough to transfer knowledge to a student model?
publisher	MDPI AG
publishDate	2021
url	https://doaj.org/article/b5508f2d11a04b6db13a9ff530052572
work_keys_str_mv	AT nicolalandro isoneteachermodelenoughtotransferknowledgetoastudentmodel AT ignaziogallo isoneteachermodelenoughtotransferknowledgetoastudentmodel AT riccardolagrassa isoneteachermodelenoughtotransferknowledgetoastudentmodel
_version_	1718413278521065472

Is One Teacher Model Enough to Transfer Knowledge to a Student Model?

Ejemplares similares