Multi-Modal Deep Learning for Weeds Detection in Wheat Field Based on RGB-D Images
Single-modal images carry limited information for features representation, and RGB images fail to detect grass weeds in wheat fields because of their similarity to wheat in shape. We propose a framework based on multi-modal information fusion for accurate detection of weeds in wheat fields in a natu...
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | article |
| Lenguaje: | EN |
| Publicado: |
Frontiers Media S.A.
2021
|
| Materias: | |
| Acceso en línea: | https://doaj.org/article/a58375e506aa4e719dd6e8865b9706ef |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| id |
oai:doaj.org-article:a58375e506aa4e719dd6e8865b9706ef |
|---|---|
| record_format |
dspace |
| spelling |
oai:doaj.org-article:a58375e506aa4e719dd6e8865b9706ef2021-11-05T13:45:28ZMulti-Modal Deep Learning for Weeds Detection in Wheat Field Based on RGB-D Images1664-462X10.3389/fpls.2021.732968https://doaj.org/article/a58375e506aa4e719dd6e8865b9706ef2021-11-01T00:00:00Zhttps://www.frontiersin.org/articles/10.3389/fpls.2021.732968/fullhttps://doaj.org/toc/1664-462XSingle-modal images carry limited information for features representation, and RGB images fail to detect grass weeds in wheat fields because of their similarity to wheat in shape. We propose a framework based on multi-modal information fusion for accurate detection of weeds in wheat fields in a natural environment, overcoming the limitation of single modality in weeds detection. Firstly, we recode the single-channel depth image into a new three-channel image like the structure of RGB image, which is suitable for feature extraction of convolutional neural network (CNN). Secondly, the multi-scale object detection is realized by fusing the feature maps output by different convolutional layers. The three-channel network structure is designed to take into account the independence of RGB and depth information, respectively, and the complementarity of multi-modal information, and the integrated learning is carried out by weight allocation at the decision level to realize the effective fusion of multi-modal information. The experimental results show that compared with the weed detection method based on RGB image, the accuracy of our method is significantly improved. Experiments with integrated learning shows that mean average precision (mAP) of 36.1% for grass weeds and 42.9% for broad-leaf weeds, and the overall detection precision, as indicated by intersection over ground truth (IoG), is 89.3%, with weights of RGB and depth images at α = 0.4 and β = 0.3. The results suggest that our methods can accurately detect the dominant species of weeds in wheat fields, and that multi-modal fusion can effectively improve object detection performance.Ke XuKe XuKe XuKe XuKe XuYan ZhuYan ZhuYan ZhuYan ZhuYan ZhuWeixing CaoWeixing CaoWeixing CaoWeixing CaoWeixing CaoXiaoping JiangXiaoping JiangXiaoping JiangXiaoping JiangXiaoping JiangZhijian JiangShuailong LiJun NiJun NiJun NiJun NiJun NiFrontiers Media S.A.articleweeds detectionRGB-D imagemulti-modal deep learningmachine learningthree-channel networkPlant cultureSB1-1110ENFrontiers in Plant Science, Vol 12 (2021) |
| institution |
DOAJ |
| collection |
DOAJ |
| language |
EN |
| topic |
weeds detection RGB-D image multi-modal deep learning machine learning three-channel network Plant culture SB1-1110 |
| spellingShingle |
weeds detection RGB-D image multi-modal deep learning machine learning three-channel network Plant culture SB1-1110 Ke Xu Ke Xu Ke Xu Ke Xu Ke Xu Yan Zhu Yan Zhu Yan Zhu Yan Zhu Yan Zhu Weixing Cao Weixing Cao Weixing Cao Weixing Cao Weixing Cao Xiaoping Jiang Xiaoping Jiang Xiaoping Jiang Xiaoping Jiang Xiaoping Jiang Zhijian Jiang Shuailong Li Jun Ni Jun Ni Jun Ni Jun Ni Jun Ni Multi-Modal Deep Learning for Weeds Detection in Wheat Field Based on RGB-D Images |
| description |
Single-modal images carry limited information for features representation, and RGB images fail to detect grass weeds in wheat fields because of their similarity to wheat in shape. We propose a framework based on multi-modal information fusion for accurate detection of weeds in wheat fields in a natural environment, overcoming the limitation of single modality in weeds detection. Firstly, we recode the single-channel depth image into a new three-channel image like the structure of RGB image, which is suitable for feature extraction of convolutional neural network (CNN). Secondly, the multi-scale object detection is realized by fusing the feature maps output by different convolutional layers. The three-channel network structure is designed to take into account the independence of RGB and depth information, respectively, and the complementarity of multi-modal information, and the integrated learning is carried out by weight allocation at the decision level to realize the effective fusion of multi-modal information. The experimental results show that compared with the weed detection method based on RGB image, the accuracy of our method is significantly improved. Experiments with integrated learning shows that mean average precision (mAP) of 36.1% for grass weeds and 42.9% for broad-leaf weeds, and the overall detection precision, as indicated by intersection over ground truth (IoG), is 89.3%, with weights of RGB and depth images at α = 0.4 and β = 0.3. The results suggest that our methods can accurately detect the dominant species of weeds in wheat fields, and that multi-modal fusion can effectively improve object detection performance. |
| format |
article |
| author |
Ke Xu Ke Xu Ke Xu Ke Xu Ke Xu Yan Zhu Yan Zhu Yan Zhu Yan Zhu Yan Zhu Weixing Cao Weixing Cao Weixing Cao Weixing Cao Weixing Cao Xiaoping Jiang Xiaoping Jiang Xiaoping Jiang Xiaoping Jiang Xiaoping Jiang Zhijian Jiang Shuailong Li Jun Ni Jun Ni Jun Ni Jun Ni Jun Ni |
| author_facet |
Ke Xu Ke Xu Ke Xu Ke Xu Ke Xu Yan Zhu Yan Zhu Yan Zhu Yan Zhu Yan Zhu Weixing Cao Weixing Cao Weixing Cao Weixing Cao Weixing Cao Xiaoping Jiang Xiaoping Jiang Xiaoping Jiang Xiaoping Jiang Xiaoping Jiang Zhijian Jiang Shuailong Li Jun Ni Jun Ni Jun Ni Jun Ni Jun Ni |
| author_sort |
Ke Xu |
| title |
Multi-Modal Deep Learning for Weeds Detection in Wheat Field Based on RGB-D Images |
| title_short |
Multi-Modal Deep Learning for Weeds Detection in Wheat Field Based on RGB-D Images |
| title_full |
Multi-Modal Deep Learning for Weeds Detection in Wheat Field Based on RGB-D Images |
| title_fullStr |
Multi-Modal Deep Learning for Weeds Detection in Wheat Field Based on RGB-D Images |
| title_full_unstemmed |
Multi-Modal Deep Learning for Weeds Detection in Wheat Field Based on RGB-D Images |
| title_sort |
multi-modal deep learning for weeds detection in wheat field based on rgb-d images |
| publisher |
Frontiers Media S.A. |
| publishDate |
2021 |
| url |
https://doaj.org/article/a58375e506aa4e719dd6e8865b9706ef |
| work_keys_str_mv |
AT kexu multimodaldeeplearningforweedsdetectioninwheatfieldbasedonrgbdimages AT kexu multimodaldeeplearningforweedsdetectioninwheatfieldbasedonrgbdimages AT kexu multimodaldeeplearningforweedsdetectioninwheatfieldbasedonrgbdimages AT kexu multimodaldeeplearningforweedsdetectioninwheatfieldbasedonrgbdimages AT kexu multimodaldeeplearningforweedsdetectioninwheatfieldbasedonrgbdimages AT yanzhu multimodaldeeplearningforweedsdetectioninwheatfieldbasedonrgbdimages AT yanzhu multimodaldeeplearningforweedsdetectioninwheatfieldbasedonrgbdimages AT yanzhu multimodaldeeplearningforweedsdetectioninwheatfieldbasedonrgbdimages AT yanzhu multimodaldeeplearningforweedsdetectioninwheatfieldbasedonrgbdimages AT yanzhu multimodaldeeplearningforweedsdetectioninwheatfieldbasedonrgbdimages AT weixingcao multimodaldeeplearningforweedsdetectioninwheatfieldbasedonrgbdimages AT weixingcao multimodaldeeplearningforweedsdetectioninwheatfieldbasedonrgbdimages AT weixingcao multimodaldeeplearningforweedsdetectioninwheatfieldbasedonrgbdimages AT weixingcao multimodaldeeplearningforweedsdetectioninwheatfieldbasedonrgbdimages AT weixingcao multimodaldeeplearningforweedsdetectioninwheatfieldbasedonrgbdimages AT xiaopingjiang multimodaldeeplearningforweedsdetectioninwheatfieldbasedonrgbdimages AT xiaopingjiang multimodaldeeplearningforweedsdetectioninwheatfieldbasedonrgbdimages AT xiaopingjiang multimodaldeeplearningforweedsdetectioninwheatfieldbasedonrgbdimages AT xiaopingjiang multimodaldeeplearningforweedsdetectioninwheatfieldbasedonrgbdimages AT xiaopingjiang multimodaldeeplearningforweedsdetectioninwheatfieldbasedonrgbdimages AT zhijianjiang multimodaldeeplearningforweedsdetectioninwheatfieldbasedonrgbdimages AT shuailongli multimodaldeeplearningforweedsdetectioninwheatfieldbasedonrgbdimages AT junni multimodaldeeplearningforweedsdetectioninwheatfieldbasedonrgbdimages AT junni multimodaldeeplearningforweedsdetectioninwheatfieldbasedonrgbdimages AT junni multimodaldeeplearningforweedsdetectioninwheatfieldbasedonrgbdimages AT junni multimodaldeeplearningforweedsdetectioninwheatfieldbasedonrgbdimages AT junni multimodaldeeplearningforweedsdetectioninwheatfieldbasedonrgbdimages |
| _version_ |
1718444248405114880 |