Multispectral Fusion of RGB and NIR Images Using Weighted Least Squares and Convolution Neural Networks

In low light condition, color (RGB) images captured by visible sensors suffer from severe noise causing loss of colors and textures. However, near infrared (NIR) images captured by NIR sensors are robust to noise even in low light condition without color. Since RGB and NIR images are complementary i...

Descripción completa

Guardado en:

Detalles Bibliográficos
Autores principales:	Cheolkon Jung, Qihui Han, Kailong Zhou, Yuanquan Xu
Formato:	article
Lenguaje:	EN
Publicado:	IEEE 2021
Materias:	Image fusion convolution neural networks multispectral near-infrared sensor fusion weighted least squares Electrical engineering. Electronics. Nuclear engineering TK1-9971
Acceso en línea:	https://doaj.org/article/9059961e54964761a1cc27393055f6fc
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

id	oai:doaj.org-article:9059961e54964761a1cc27393055f6fc
record_format	dspace
spelling	oai:doaj.org-article:9059961e54964761a1cc27393055f6fc2021-11-18T00:11:38ZMultispectral Fusion of RGB and NIR Images Using Weighted Least Squares and Convolution Neural Networks2644-132210.1109/OJSP.2021.3122074https://doaj.org/article/9059961e54964761a1cc27393055f6fc2021-01-01T00:00:00Zhttps://ieeexplore.ieee.org/document/9583922/https://doaj.org/toc/2644-1322In low light condition, color (RGB) images captured by visible sensors suffer from severe noise causing loss of colors and textures. However, near infrared (NIR) images captured by NIR sensors are robust to noise even in low light condition without color. Since RGB and NIR images are complementary in low light condition, the multispectral fusion of RGB and NIR images provides a viable solution to low light imaging. In this paper, we propose multispectral fusion of RGB and NIR images using weighted least squares (WLS) and convolution neural networks (CNNs). We combine traditional WLS filtering for layer decomposition and denoising with latest deep learning for image enhancement and texture transfer into the multispectral fusion to take both advantages. We build two networks based on CNN: image enhancement network (IEN) for image enhancement and texture transfer network (TTN) for NIR texture transfer. First, we perform RGB image denoising based on WLS filtering and generate the base layer. We use both RGB and NIR images for WLS filtering as weights to filter out noise in low light RGB images. Second, we conduct IEN to enhance contrast of the base layer. Third, we perform TTN to deliver NIR details completely and naturally to the fusion result. The combination of WLS, TTN and IEN leads to noise reduction, contrast enhancement, and detail preservation in fusion. Experimental results show that the proposed method achieves good performance in both noise reduction and detail transfer as well as outperforms state-of-the-art methods in terms of visual quality and quantitative measurements.Cheolkon JungQihui HanKailong ZhouYuanquan XuIEEEarticleImage fusionconvolution neural networksmultispectralnear-infraredsensor fusionweighted least squaresElectrical engineering. Electronics. Nuclear engineeringTK1-9971ENIEEE Open Journal of Signal Processing, Vol 2, Pp 559-570 (2021)
institution	DOAJ
collection	DOAJ
language	EN
topic	Image fusion convolution neural networks multispectral near-infrared sensor fusion weighted least squares Electrical engineering. Electronics. Nuclear engineering TK1-9971
spellingShingle	Image fusion convolution neural networks multispectral near-infrared sensor fusion weighted least squares Electrical engineering. Electronics. Nuclear engineering TK1-9971 Cheolkon Jung Qihui Han Kailong Zhou Yuanquan Xu Multispectral Fusion of RGB and NIR Images Using Weighted Least Squares and Convolution Neural Networks
description	In low light condition, color (RGB) images captured by visible sensors suffer from severe noise causing loss of colors and textures. However, near infrared (NIR) images captured by NIR sensors are robust to noise even in low light condition without color. Since RGB and NIR images are complementary in low light condition, the multispectral fusion of RGB and NIR images provides a viable solution to low light imaging. In this paper, we propose multispectral fusion of RGB and NIR images using weighted least squares (WLS) and convolution neural networks (CNNs). We combine traditional WLS filtering for layer decomposition and denoising with latest deep learning for image enhancement and texture transfer into the multispectral fusion to take both advantages. We build two networks based on CNN: image enhancement network (IEN) for image enhancement and texture transfer network (TTN) for NIR texture transfer. First, we perform RGB image denoising based on WLS filtering and generate the base layer. We use both RGB and NIR images for WLS filtering as weights to filter out noise in low light RGB images. Second, we conduct IEN to enhance contrast of the base layer. Third, we perform TTN to deliver NIR details completely and naturally to the fusion result. The combination of WLS, TTN and IEN leads to noise reduction, contrast enhancement, and detail preservation in fusion. Experimental results show that the proposed method achieves good performance in both noise reduction and detail transfer as well as outperforms state-of-the-art methods in terms of visual quality and quantitative measurements.
format	article
author	Cheolkon Jung Qihui Han Kailong Zhou Yuanquan Xu
author_facet	Cheolkon Jung Qihui Han Kailong Zhou Yuanquan Xu
author_sort	Cheolkon Jung
title	Multispectral Fusion of RGB and NIR Images Using Weighted Least Squares and Convolution Neural Networks
title_short	Multispectral Fusion of RGB and NIR Images Using Weighted Least Squares and Convolution Neural Networks
title_full	Multispectral Fusion of RGB and NIR Images Using Weighted Least Squares and Convolution Neural Networks
title_fullStr	Multispectral Fusion of RGB and NIR Images Using Weighted Least Squares and Convolution Neural Networks
title_full_unstemmed	Multispectral Fusion of RGB and NIR Images Using Weighted Least Squares and Convolution Neural Networks
title_sort	multispectral fusion of rgb and nir images using weighted least squares and convolution neural networks
publisher	IEEE
publishDate	2021
url	https://doaj.org/article/9059961e54964761a1cc27393055f6fc
work_keys_str_mv	AT cheolkonjung multispectralfusionofrgbandnirimagesusingweightedleastsquaresandconvolutionneuralnetworks AT qihuihan multispectralfusionofrgbandnirimagesusingweightedleastsquaresandconvolutionneuralnetworks AT kailongzhou multispectralfusionofrgbandnirimagesusingweightedleastsquaresandconvolutionneuralnetworks AT yuanquanxu multispectralfusionofrgbandnirimagesusingweightedleastsquaresandconvolutionneuralnetworks
_version_	1718425177464766464

Multispectral Fusion of RGB and NIR Images Using Weighted Least Squares and Convolution Neural Networks

Ejemplares similares