Gradient Decomposition Methods for Training Neural Networks With Non-ideal Synaptic Devices

While promising for high-capacity machine learning accelerators, memristor devices have non-idealities that prevent software-equivalent accuracies when used for online training. This work uses a combination of Mini-Batch Gradient Descent (MBGD) to average gradients, stochastic rounding to avoid vani...

Descripción completa

Guardado en:

Detalles Bibliográficos
Autores principales:	Junyun Zhao, Siyuan Huang, Osama Yousuf, Yutong Gao, Brian D. Hoskins, Gina C. Adam
Formato:	article
Lenguaje:	EN
Publicado:	Frontiers Media S.A. 2021
Materias:	non-negative matrix factorization gradient data decomposition principal component analysis memristor non-idealities ReRAM Neurosciences. Biological psychiatry. Neuropsychiatry RC321-571
Acceso en línea:	https://doaj.org/article/e757422268bc4a378caf150d006973d5
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

id	oai:doaj.org-article:e757422268bc4a378caf150d006973d5
record_format	dspace
spelling	oai:doaj.org-article:e757422268bc4a378caf150d006973d52021-11-22T05:09:28ZGradient Decomposition Methods for Training Neural Networks With Non-ideal Synaptic Devices1662-453X10.3389/fnins.2021.749811https://doaj.org/article/e757422268bc4a378caf150d006973d52021-11-01T00:00:00Zhttps://www.frontiersin.org/articles/10.3389/fnins.2021.749811/fullhttps://doaj.org/toc/1662-453XWhile promising for high-capacity machine learning accelerators, memristor devices have non-idealities that prevent software-equivalent accuracies when used for online training. This work uses a combination of Mini-Batch Gradient Descent (MBGD) to average gradients, stochastic rounding to avoid vanishing weight updates, and decomposition methods to keep the memory overhead low during mini-batch training. Since the weight update has to be transferred to the memristor matrices efficiently, we also investigate the impact of reconstructing the gradient matrixes both internally (rank-seq) and externally (rank-sum) to the memristor array. Our results show that streaming batch principal component analysis (streaming batch PCA) and non-negative matrix factorization (NMF) decomposition algorithms can achieve near MBGD accuracy in a memristor-based multi-layer perceptron trained on the MNIST (Modified National Institute of Standards and Technology) database with only 3 to 10 ranks at significant memory savings. Moreover, NMF rank-seq outperforms streaming batch PCA rank-seq at low-ranks making it more suitable for hardware implementation in future memristor-based accelerators.Junyun ZhaoSiyuan HuangOsama YousufYutong GaoBrian D. HoskinsGina C. AdamFrontiers Media S.A.articlenon-negative matrix factorizationgradient data decompositionprincipal component analysismemristornon-idealitiesReRAMNeurosciences. Biological psychiatry. NeuropsychiatryRC321-571ENFrontiers in Neuroscience, Vol 15 (2021)
institution	DOAJ
collection	DOAJ
language	EN
topic	non-negative matrix factorization gradient data decomposition principal component analysis memristor non-idealities ReRAM Neurosciences. Biological psychiatry. Neuropsychiatry RC321-571
spellingShingle	non-negative matrix factorization gradient data decomposition principal component analysis memristor non-idealities ReRAM Neurosciences. Biological psychiatry. Neuropsychiatry RC321-571 Junyun Zhao Siyuan Huang Osama Yousuf Yutong Gao Brian D. Hoskins Gina C. Adam Gradient Decomposition Methods for Training Neural Networks With Non-ideal Synaptic Devices
description	While promising for high-capacity machine learning accelerators, memristor devices have non-idealities that prevent software-equivalent accuracies when used for online training. This work uses a combination of Mini-Batch Gradient Descent (MBGD) to average gradients, stochastic rounding to avoid vanishing weight updates, and decomposition methods to keep the memory overhead low during mini-batch training. Since the weight update has to be transferred to the memristor matrices efficiently, we also investigate the impact of reconstructing the gradient matrixes both internally (rank-seq) and externally (rank-sum) to the memristor array. Our results show that streaming batch principal component analysis (streaming batch PCA) and non-negative matrix factorization (NMF) decomposition algorithms can achieve near MBGD accuracy in a memristor-based multi-layer perceptron trained on the MNIST (Modified National Institute of Standards and Technology) database with only 3 to 10 ranks at significant memory savings. Moreover, NMF rank-seq outperforms streaming batch PCA rank-seq at low-ranks making it more suitable for hardware implementation in future memristor-based accelerators.
format	article
author	Junyun Zhao Siyuan Huang Osama Yousuf Yutong Gao Brian D. Hoskins Gina C. Adam
author_facet	Junyun Zhao Siyuan Huang Osama Yousuf Yutong Gao Brian D. Hoskins Gina C. Adam
author_sort	Junyun Zhao
title	Gradient Decomposition Methods for Training Neural Networks With Non-ideal Synaptic Devices
title_short	Gradient Decomposition Methods for Training Neural Networks With Non-ideal Synaptic Devices
title_full	Gradient Decomposition Methods for Training Neural Networks With Non-ideal Synaptic Devices
title_fullStr	Gradient Decomposition Methods for Training Neural Networks With Non-ideal Synaptic Devices
title_full_unstemmed	Gradient Decomposition Methods for Training Neural Networks With Non-ideal Synaptic Devices
title_sort	gradient decomposition methods for training neural networks with non-ideal synaptic devices
publisher	Frontiers Media S.A.
publishDate	2021
url	https://doaj.org/article/e757422268bc4a378caf150d006973d5
work_keys_str_mv	AT junyunzhao gradientdecompositionmethodsfortrainingneuralnetworkswithnonidealsynapticdevices AT siyuanhuang gradientdecompositionmethodsfortrainingneuralnetworkswithnonidealsynapticdevices AT osamayousuf gradientdecompositionmethodsfortrainingneuralnetworkswithnonidealsynapticdevices AT yutonggao gradientdecompositionmethodsfortrainingneuralnetworkswithnonidealsynapticdevices AT briandhoskins gradientdecompositionmethodsfortrainingneuralnetworkswithnonidealsynapticdevices AT ginacadam gradientdecompositionmethodsfortrainingneuralnetworkswithnonidealsynapticdevices
_version_	1718418195226820608

Gradient Decomposition Methods for Training Neural Networks With Non-ideal Synaptic Devices

Ejemplares similares