An adiabatic method to train binarized artificial neural networks

Abstract An artificial neural network consists of neurons and synapses. Neuron gives output based on its input according to non-linear activation functions such as the Sigmoid, Hyperbolic Tangent (Tanh), or Rectified Linear Unit (ReLU) functions, etc.. Synapses connect the neuron outputs to their in...

Descripción completa

Guardado en:

Detalles Bibliográficos
Autores principales:	Yuansheng Zhao, Jiang Xiao
Formato:	article
Lenguaje:	EN
Publicado:	Nature Portfolio 2021
Materias:	Medicine R Science Q
Acceso en línea:	https://doaj.org/article/1e37ac851b52403d9e4e101a2e68ed61
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

id	oai:doaj.org-article:1e37ac851b52403d9e4e101a2e68ed61
record_format	dspace
spelling	oai:doaj.org-article:1e37ac851b52403d9e4e101a2e68ed612021-12-02T16:56:36ZAn adiabatic method to train binarized artificial neural networks10.1038/s41598-021-99191-22045-2322https://doaj.org/article/1e37ac851b52403d9e4e101a2e68ed612021-10-01T00:00:00Zhttps://doi.org/10.1038/s41598-021-99191-2https://doaj.org/toc/2045-2322Abstract An artificial neural network consists of neurons and synapses. Neuron gives output based on its input according to non-linear activation functions such as the Sigmoid, Hyperbolic Tangent (Tanh), or Rectified Linear Unit (ReLU) functions, etc.. Synapses connect the neuron outputs to their inputs with tunable real-valued weights. The most resource-demanding operations in realizing such neural networks are the multiplication and accumulate (MAC) operations that compute the dot product between real-valued outputs from neurons and the synapses weights. The efficiency of neural networks can be drastically enhanced if the neuron outputs and/or the weights can be trained to take binary values $$\pm 1$$ ± 1 only, for which the MAC can be replaced by the simple XNOR operations. In this paper, we demonstrate an adiabatic training method that can binarize the fully-connected neural networks and the convolutional neural networks without modifying the network structure and size. This adiabatic training method only requires very minimal changes in training algorithms, and is tested in the following four tasks: the recognition of hand-writing numbers using a usual fully-connected network, the cat-dog recognition and the audio recognition using convolutional neural networks, the image recognition with 10 classes (CIFAR-10) using ResNet-20 and VGG-Small networks. In all tasks, the performance of the binary neural networks trained by the adiabatic method are almost identical to the networks trained using the conventional ReLU or Sigmoid activations with real-valued activations and weights. This adiabatic method can be easily applied to binarize different types of networks, and will increase the computational efficiency considerably and greatly simplify the deployment of neural networks.Yuansheng ZhaoJiang XiaoNature PortfolioarticleMedicineRScienceQENScientific Reports, Vol 11, Iss 1, Pp 1-8 (2021)
institution	DOAJ
collection	DOAJ
language	EN
topic	Medicine R Science Q
spellingShingle	Medicine R Science Q Yuansheng Zhao Jiang Xiao An adiabatic method to train binarized artificial neural networks
description	Abstract An artificial neural network consists of neurons and synapses. Neuron gives output based on its input according to non-linear activation functions such as the Sigmoid, Hyperbolic Tangent (Tanh), or Rectified Linear Unit (ReLU) functions, etc.. Synapses connect the neuron outputs to their inputs with tunable real-valued weights. The most resource-demanding operations in realizing such neural networks are the multiplication and accumulate (MAC) operations that compute the dot product between real-valued outputs from neurons and the synapses weights. The efficiency of neural networks can be drastically enhanced if the neuron outputs and/or the weights can be trained to take binary values $$\pm 1$$ ± 1 only, for which the MAC can be replaced by the simple XNOR operations. In this paper, we demonstrate an adiabatic training method that can binarize the fully-connected neural networks and the convolutional neural networks without modifying the network structure and size. This adiabatic training method only requires very minimal changes in training algorithms, and is tested in the following four tasks: the recognition of hand-writing numbers using a usual fully-connected network, the cat-dog recognition and the audio recognition using convolutional neural networks, the image recognition with 10 classes (CIFAR-10) using ResNet-20 and VGG-Small networks. In all tasks, the performance of the binary neural networks trained by the adiabatic method are almost identical to the networks trained using the conventional ReLU or Sigmoid activations with real-valued activations and weights. This adiabatic method can be easily applied to binarize different types of networks, and will increase the computational efficiency considerably and greatly simplify the deployment of neural networks.
format	article
author	Yuansheng Zhao Jiang Xiao
author_facet	Yuansheng Zhao Jiang Xiao
author_sort	Yuansheng Zhao
title	An adiabatic method to train binarized artificial neural networks
title_short	An adiabatic method to train binarized artificial neural networks
title_full	An adiabatic method to train binarized artificial neural networks
title_fullStr	An adiabatic method to train binarized artificial neural networks
title_full_unstemmed	An adiabatic method to train binarized artificial neural networks
title_sort	adiabatic method to train binarized artificial neural networks
publisher	Nature Portfolio
publishDate	2021
url	https://doaj.org/article/1e37ac851b52403d9e4e101a2e68ed61
work_keys_str_mv	AT yuanshengzhao anadiabaticmethodtotrainbinarizedartificialneuralnetworks AT jiangxiao anadiabaticmethodtotrainbinarizedartificialneuralnetworks AT yuanshengzhao adiabaticmethodtotrainbinarizedartificialneuralnetworks AT jiangxiao adiabaticmethodtotrainbinarizedartificialneuralnetworks
_version_	1718382785319665664

An adiabatic method to train binarized artificial neural networks

Ejemplares similares