Egocentric Gesture Recognition Using 3D Convolutional Neural Networks for the Spatiotemporal Adaptation of Collaborative Robots

Collaborative robots are currently deployed in professional environments, in collaboration with professional human operators, helping to strike the right balance between mechanization and manual intervention in manufacturing processes required by Industry 4.0. In this paper, the contribution of gest...

Descripción completa

Guardado en:

Detalles Bibliográficos
Autores principales:	Dimitris Papanagiotou, Gavriela Senteri, Sotiris Manitsaris
Formato:	article
Lenguaje:	EN
Publicado:	Frontiers Media S.A. 2021
Materias:	human-robot collaboration gestures actions recognition CNN egocentric vision Neurosciences. Biological psychiatry. Neuropsychiatry RC321-571
Acceso en línea:	https://doaj.org/article/894e810986384c71b181523b91a18bb2
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

id	oai:doaj.org-article:894e810986384c71b181523b91a18bb2
record_format	dspace
spelling	oai:doaj.org-article:894e810986384c71b181523b91a18bb22021-11-30T12:21:28ZEgocentric Gesture Recognition Using 3D Convolutional Neural Networks for the Spatiotemporal Adaptation of Collaborative Robots1662-521810.3389/fnbot.2021.703545https://doaj.org/article/894e810986384c71b181523b91a18bb22021-11-01T00:00:00Zhttps://www.frontiersin.org/articles/10.3389/fnbot.2021.703545/fullhttps://doaj.org/toc/1662-5218Collaborative robots are currently deployed in professional environments, in collaboration with professional human operators, helping to strike the right balance between mechanization and manual intervention in manufacturing processes required by Industry 4.0. In this paper, the contribution of gesture recognition and pose estimation to the smooth introduction of cobots into an industrial assembly line is described, with a view to performing actions in parallel with the human operators and enabling interaction between them. The proposed active vision system uses two RGB-D cameras that record different points of view of gestures and poses of the operator, to build an external perception layer for the robot that facilitates spatiotemporal adaptation, in accordance with the human's behavior. The use-case of this work is concerned with LCD TV assembly of an appliance manufacturer, comprising of two parts. The first part of the above-mentioned operation is assigned to a robot, strengthening the assembly line. The second part is assigned to a human operator. Gesture recognition, pose estimation, physical interaction, and sonic notification, create a multimodal human-robot interaction system. Five experiments are performed, to test if gesture recognition and pose estimation can reduce the cycle time and range of motion of the operator, respectively. Physical interaction is achieved using the force sensor of the cobot. Pose estimation through a skeleton-tracking algorithm provides the cobot with human pose information and makes it spatially adjustable. Sonic notification is added for the case of unexpected incidents. A real-time gesture recognition module is implemented through a Deep Learning architecture consisting of Convolutional layers, trained in an egocentric view and reducing the cycle time of the routine by almost 20%. This constitutes an added value in this work, as it affords the potential of recognizing gestures independently of the anthropometric characteristics and the background. Common metrics derived from the literature are used for the evaluation of the proposed system. The percentage of spatial adaptation of the cobot is proposed as a new KPI for a collaborative system and the opinion of the human operator is measured through a questionnaire that concerns the various affective states of the operator during the collaboration.Dimitris PapanagiotouGavriela SenteriSotiris ManitsarisFrontiers Media S.A.articlehuman-robot collaborationgesturesactionsrecognitionCNNegocentric visionNeurosciences. Biological psychiatry. NeuropsychiatryRC321-571ENFrontiers in Neurorobotics, Vol 15 (2021)
institution	DOAJ
collection	DOAJ
language	EN
topic	human-robot collaboration gestures actions recognition CNN egocentric vision Neurosciences. Biological psychiatry. Neuropsychiatry RC321-571
spellingShingle	human-robot collaboration gestures actions recognition CNN egocentric vision Neurosciences. Biological psychiatry. Neuropsychiatry RC321-571 Dimitris Papanagiotou Gavriela Senteri Sotiris Manitsaris Egocentric Gesture Recognition Using 3D Convolutional Neural Networks for the Spatiotemporal Adaptation of Collaborative Robots
description	Collaborative robots are currently deployed in professional environments, in collaboration with professional human operators, helping to strike the right balance between mechanization and manual intervention in manufacturing processes required by Industry 4.0. In this paper, the contribution of gesture recognition and pose estimation to the smooth introduction of cobots into an industrial assembly line is described, with a view to performing actions in parallel with the human operators and enabling interaction between them. The proposed active vision system uses two RGB-D cameras that record different points of view of gestures and poses of the operator, to build an external perception layer for the robot that facilitates spatiotemporal adaptation, in accordance with the human's behavior. The use-case of this work is concerned with LCD TV assembly of an appliance manufacturer, comprising of two parts. The first part of the above-mentioned operation is assigned to a robot, strengthening the assembly line. The second part is assigned to a human operator. Gesture recognition, pose estimation, physical interaction, and sonic notification, create a multimodal human-robot interaction system. Five experiments are performed, to test if gesture recognition and pose estimation can reduce the cycle time and range of motion of the operator, respectively. Physical interaction is achieved using the force sensor of the cobot. Pose estimation through a skeleton-tracking algorithm provides the cobot with human pose information and makes it spatially adjustable. Sonic notification is added for the case of unexpected incidents. A real-time gesture recognition module is implemented through a Deep Learning architecture consisting of Convolutional layers, trained in an egocentric view and reducing the cycle time of the routine by almost 20%. This constitutes an added value in this work, as it affords the potential of recognizing gestures independently of the anthropometric characteristics and the background. Common metrics derived from the literature are used for the evaluation of the proposed system. The percentage of spatial adaptation of the cobot is proposed as a new KPI for a collaborative system and the opinion of the human operator is measured through a questionnaire that concerns the various affective states of the operator during the collaboration.
format	article
author	Dimitris Papanagiotou Gavriela Senteri Sotiris Manitsaris
author_facet	Dimitris Papanagiotou Gavriela Senteri Sotiris Manitsaris
author_sort	Dimitris Papanagiotou
title	Egocentric Gesture Recognition Using 3D Convolutional Neural Networks for the Spatiotemporal Adaptation of Collaborative Robots
title_short	Egocentric Gesture Recognition Using 3D Convolutional Neural Networks for the Spatiotemporal Adaptation of Collaborative Robots
title_full	Egocentric Gesture Recognition Using 3D Convolutional Neural Networks for the Spatiotemporal Adaptation of Collaborative Robots
title_fullStr	Egocentric Gesture Recognition Using 3D Convolutional Neural Networks for the Spatiotemporal Adaptation of Collaborative Robots
title_full_unstemmed	Egocentric Gesture Recognition Using 3D Convolutional Neural Networks for the Spatiotemporal Adaptation of Collaborative Robots
title_sort	egocentric gesture recognition using 3d convolutional neural networks for the spatiotemporal adaptation of collaborative robots
publisher	Frontiers Media S.A.
publishDate	2021
url	https://doaj.org/article/894e810986384c71b181523b91a18bb2
work_keys_str_mv	AT dimitrispapanagiotou egocentricgesturerecognitionusing3dconvolutionalneuralnetworksforthespatiotemporaladaptationofcollaborativerobots AT gavrielasenteri egocentricgesturerecognitionusing3dconvolutionalneuralnetworksforthespatiotemporaladaptationofcollaborativerobots AT sotirismanitsaris egocentricgesturerecognitionusing3dconvolutionalneuralnetworksforthespatiotemporaladaptationofcollaborativerobots
_version_	1718406633275523072

Egocentric Gesture Recognition Using 3D Convolutional Neural Networks for the Spatiotemporal Adaptation of Collaborative Robots

Ejemplares similares