Face Swapping Consistency Transfer with Neural Identity Carrier

Deepfake aims to swap a face of an image with someone else’s likeness in a reasonable manner. Existing methods usually perform deepfake frame by frame, thus ignoring video consistency and producing incoherent results. To address such a problem, we propose a novel framework Neural Identity Carrier (N...

Descripción completa

Guardado en:

Detalles Bibliográficos
Autores principales:	Kunlin Liu, Ping Wang, Wenbo Zhou, Zhenyu Zhang, Yanhao Ge, Honggu Liu, Weiming Zhang, Nenghai Yu
Formato:	article
Lenguaje:	EN
Publicado:	MDPI AG 2021
Materias:	deepfake generation face swapping consistency transfer Information technology T58.5-58.64
Acceso en línea:	https://doaj.org/article/d8d1184174db460b8c87817c4d6c0d8f
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

id	oai:doaj.org-article:d8d1184174db460b8c87817c4d6c0d8f
record_format	dspace
spelling	oai:doaj.org-article:d8d1184174db460b8c87817c4d6c0d8f2021-11-25T17:40:07ZFace Swapping Consistency Transfer with Neural Identity Carrier10.3390/fi131102981999-5903https://doaj.org/article/d8d1184174db460b8c87817c4d6c0d8f2021-11-01T00:00:00Zhttps://www.mdpi.com/1999-5903/13/11/298https://doaj.org/toc/1999-5903Deepfake aims to swap a face of an image with someone else’s likeness in a reasonable manner. Existing methods usually perform deepfake frame by frame, thus ignoring video consistency and producing incoherent results. To address such a problem, we propose a novel framework Neural Identity Carrier (NICe), which learns identity transformation from an arbitrary face-swapping proxy via a <i>U</i>-<inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mi>N</mi><mi>e</mi><mi>t</mi></mrow></semantics></math></inline-formula>. By modeling the incoherence between frames as noise, NICe naturally suppresses its disturbance and preserves primary identity information. Concretely, NICe inputs the original frame and learns transformation supervised by swapped pseudo labels. As the temporal incoherence has an uncertain or stochastic pattern, NICe can filter out such outliers and well maintain the target content by uncertainty prediction. With the predicted temporally stable appearance, NICe enhances its details by constraining 3D geometry consistency, making NICe learn fine-grained facial structure across the poses. In this way, NICe guarantees the temporal stableness of deepfake approaches and predicts detailed results against over-smoothness. Extensive experiments on benchmarks demonstrate that NICe significantly improves the quality of existing deepfake methods on video-level. Besides, data generated by our methods can benefit video-level deepfake detection methods.Kunlin LiuPing WangWenbo ZhouZhenyu ZhangYanhao GeHonggu LiuWeiming ZhangNenghai YuMDPI AGarticledeepfake generationface swappingconsistency transferInformation technologyT58.5-58.64ENFuture Internet, Vol 13, Iss 298, p 298 (2021)
institution	DOAJ
collection	DOAJ
language	EN
topic	deepfake generation face swapping consistency transfer Information technology T58.5-58.64
spellingShingle	deepfake generation face swapping consistency transfer Information technology T58.5-58.64 Kunlin Liu Ping Wang Wenbo Zhou Zhenyu Zhang Yanhao Ge Honggu Liu Weiming Zhang Nenghai Yu Face Swapping Consistency Transfer with Neural Identity Carrier
description	Deepfake aims to swap a face of an image with someone else’s likeness in a reasonable manner. Existing methods usually perform deepfake frame by frame, thus ignoring video consistency and producing incoherent results. To address such a problem, we propose a novel framework Neural Identity Carrier (NICe), which learns identity transformation from an arbitrary face-swapping proxy via a <i>U</i>-<inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mi>N</mi><mi>e</mi><mi>t</mi></mrow></semantics></math></inline-formula>. By modeling the incoherence between frames as noise, NICe naturally suppresses its disturbance and preserves primary identity information. Concretely, NICe inputs the original frame and learns transformation supervised by swapped pseudo labels. As the temporal incoherence has an uncertain or stochastic pattern, NICe can filter out such outliers and well maintain the target content by uncertainty prediction. With the predicted temporally stable appearance, NICe enhances its details by constraining 3D geometry consistency, making NICe learn fine-grained facial structure across the poses. In this way, NICe guarantees the temporal stableness of deepfake approaches and predicts detailed results against over-smoothness. Extensive experiments on benchmarks demonstrate that NICe significantly improves the quality of existing deepfake methods on video-level. Besides, data generated by our methods can benefit video-level deepfake detection methods.
format	article
author	Kunlin Liu Ping Wang Wenbo Zhou Zhenyu Zhang Yanhao Ge Honggu Liu Weiming Zhang Nenghai Yu
author_facet	Kunlin Liu Ping Wang Wenbo Zhou Zhenyu Zhang Yanhao Ge Honggu Liu Weiming Zhang Nenghai Yu
author_sort	Kunlin Liu
title	Face Swapping Consistency Transfer with Neural Identity Carrier
title_short	Face Swapping Consistency Transfer with Neural Identity Carrier
title_full	Face Swapping Consistency Transfer with Neural Identity Carrier
title_fullStr	Face Swapping Consistency Transfer with Neural Identity Carrier
title_full_unstemmed	Face Swapping Consistency Transfer with Neural Identity Carrier
title_sort	face swapping consistency transfer with neural identity carrier
publisher	MDPI AG
publishDate	2021
url	https://doaj.org/article/d8d1184174db460b8c87817c4d6c0d8f
work_keys_str_mv	AT kunlinliu faceswappingconsistencytransferwithneuralidentitycarrier AT pingwang faceswappingconsistencytransferwithneuralidentitycarrier AT wenbozhou faceswappingconsistencytransferwithneuralidentitycarrier AT zhenyuzhang faceswappingconsistencytransferwithneuralidentitycarrier AT yanhaoge faceswappingconsistencytransferwithneuralidentitycarrier AT hongguliu faceswappingconsistencytransferwithneuralidentitycarrier AT weimingzhang faceswappingconsistencytransferwithneuralidentitycarrier AT nenghaiyu faceswappingconsistencytransferwithneuralidentitycarrier
_version_	1718412133944786944

Face Swapping Consistency Transfer with Neural Identity Carrier

Ejemplares similares