A Novel Restricted Boltzmann Machine Training Algorithm With Dynamic Tempering Chains

Restricted Boltzmann machines (RBMs) are commonly used as pre-training methods for deep learning models. Contrastive divergence (CD) and parallel tempering (PT) are traditional training algorithms of RBMs. However, these two algorithms have shortcomings in processing high-dimensional and complex dat...

Descripción completa

Guardado en:

Detalles Bibliográficos
Autores principales:	Xinyu Li, Xiaoguang Gao, Chenfeng Wang
Formato:	article
Lenguaje:	EN
Publicado:	IEEE 2021
Materias:	Complex data dynamic tempering chains high-dimensional data parallel tempering restricted Boltzmann machine Electrical engineering. Electronics. Nuclear engineering TK1-9971
Acceso en línea:	https://doaj.org/article/670076506bd441819c7d275d7b5041bf
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

id	oai:doaj.org-article:670076506bd441819c7d275d7b5041bf
record_format	dspace
spelling	oai:doaj.org-article:670076506bd441819c7d275d7b5041bf2021-11-19T00:06:03ZA Novel Restricted Boltzmann Machine Training Algorithm With Dynamic Tempering Chains2169-353610.1109/ACCESS.2020.3043599https://doaj.org/article/670076506bd441819c7d275d7b5041bf2021-01-01T00:00:00Zhttps://ieeexplore.ieee.org/document/9288690/https://doaj.org/toc/2169-3536Restricted Boltzmann machines (RBMs) are commonly used as pre-training methods for deep learning models. Contrastive divergence (CD) and parallel tempering (PT) are traditional training algorithms of RBMs. However, these two algorithms have shortcomings in processing high-dimensional and complex data. In particular, the number of temperature chains in PT has a significant impact on the training effect, and the PT algorithm cannot fully utilize parallel sampling from multiple temperature chains for the divergence of the algorithm. The training can quickly converge with fewer temperature chains, but this impacts the accuracy. More temperature chains can help PT achieve higher accuracy in theory, but severe divergence at the beginning of the training may ruin the training result. To exploit fully the advantages of PT and improve the ability of RBMs to process high-dimensional and complex models, this article proposes dynamic tempering chains (DTC). By dynamically changing the number of temperature chains during the training process, DTC starts training with fewer temperature chains and gradually increase the number of temperature chains with training going on, and finally get an accurate RBM. And one-step reconstruction error is proposed to measure the convergence, which can decrease the influence of the dynamic training strategy on reconstruction error. Experiments on MNIST, MNORB, Cifar 10, and Cifar 100 indicate that, compared with PT, the classification accuracy of DTC algorithm improved by up to 8%. DTC quickly converges in the early stage of training because of few exchanges among temperature chains and produces higher accuracy at the end for the global optimum model learned by more temperature chains, especially when learning high-dimensional and complex data. This proves that the DTC algorithm effectively utilizes parallel sampling of multiple temperature chains, overcomes divergence challenges, and further improves the training effect of the RBM.Xinyu LiXiaoguang GaoChenfeng WangIEEEarticleComplex datadynamic tempering chainshigh-dimensional dataparallel temperingrestricted Boltzmann machineElectrical engineering. Electronics. Nuclear engineeringTK1-9971ENIEEE Access, Vol 9, Pp 21939-21950 (2021)
institution	DOAJ
collection	DOAJ
language	EN
topic	Complex data dynamic tempering chains high-dimensional data parallel tempering restricted Boltzmann machine Electrical engineering. Electronics. Nuclear engineering TK1-9971
spellingShingle	Complex data dynamic tempering chains high-dimensional data parallel tempering restricted Boltzmann machine Electrical engineering. Electronics. Nuclear engineering TK1-9971 Xinyu Li Xiaoguang Gao Chenfeng Wang A Novel Restricted Boltzmann Machine Training Algorithm With Dynamic Tempering Chains
description	Restricted Boltzmann machines (RBMs) are commonly used as pre-training methods for deep learning models. Contrastive divergence (CD) and parallel tempering (PT) are traditional training algorithms of RBMs. However, these two algorithms have shortcomings in processing high-dimensional and complex data. In particular, the number of temperature chains in PT has a significant impact on the training effect, and the PT algorithm cannot fully utilize parallel sampling from multiple temperature chains for the divergence of the algorithm. The training can quickly converge with fewer temperature chains, but this impacts the accuracy. More temperature chains can help PT achieve higher accuracy in theory, but severe divergence at the beginning of the training may ruin the training result. To exploit fully the advantages of PT and improve the ability of RBMs to process high-dimensional and complex models, this article proposes dynamic tempering chains (DTC). By dynamically changing the number of temperature chains during the training process, DTC starts training with fewer temperature chains and gradually increase the number of temperature chains with training going on, and finally get an accurate RBM. And one-step reconstruction error is proposed to measure the convergence, which can decrease the influence of the dynamic training strategy on reconstruction error. Experiments on MNIST, MNORB, Cifar 10, and Cifar 100 indicate that, compared with PT, the classification accuracy of DTC algorithm improved by up to 8%. DTC quickly converges in the early stage of training because of few exchanges among temperature chains and produces higher accuracy at the end for the global optimum model learned by more temperature chains, especially when learning high-dimensional and complex data. This proves that the DTC algorithm effectively utilizes parallel sampling of multiple temperature chains, overcomes divergence challenges, and further improves the training effect of the RBM.
format	article
author	Xinyu Li Xiaoguang Gao Chenfeng Wang
author_facet	Xinyu Li Xiaoguang Gao Chenfeng Wang
author_sort	Xinyu Li
title	A Novel Restricted Boltzmann Machine Training Algorithm With Dynamic Tempering Chains
title_short	A Novel Restricted Boltzmann Machine Training Algorithm With Dynamic Tempering Chains
title_full	A Novel Restricted Boltzmann Machine Training Algorithm With Dynamic Tempering Chains
title_fullStr	A Novel Restricted Boltzmann Machine Training Algorithm With Dynamic Tempering Chains
title_full_unstemmed	A Novel Restricted Boltzmann Machine Training Algorithm With Dynamic Tempering Chains
title_sort	novel restricted boltzmann machine training algorithm with dynamic tempering chains
publisher	IEEE
publishDate	2021
url	https://doaj.org/article/670076506bd441819c7d275d7b5041bf
work_keys_str_mv	AT xinyuli anovelrestrictedboltzmannmachinetrainingalgorithmwithdynamictemperingchains AT xiaoguanggao anovelrestrictedboltzmannmachinetrainingalgorithmwithdynamictemperingchains AT chenfengwang anovelrestrictedboltzmannmachinetrainingalgorithmwithdynamictemperingchains AT xinyuli novelrestrictedboltzmannmachinetrainingalgorithmwithdynamictemperingchains AT xiaoguanggao novelrestrictedboltzmannmachinetrainingalgorithmwithdynamictemperingchains AT chenfengwang novelrestrictedboltzmannmachinetrainingalgorithmwithdynamictemperingchains
_version_	1718420612719837184

A Novel Restricted Boltzmann Machine Training Algorithm With Dynamic Tempering Chains

Ejemplares similares