What Does a Language-And-Vision Transformer See: The Impact of Semantic Information on Visual Representations

Neural networks have proven to be very successful in automatically capturing the composition of language and different structures across a range of multi-modal tasks. Thus, an important question to investigate is how neural networks learn and organise such structures. Numerous studies have examined...

Descripción completa

Guardado en:

Detalles Bibliográficos
Autores principales:	Nikolai Ilinykh, Simon Dobnik
Formato:	article
Lenguaje:	EN
Publicado:	Frontiers Media S.A. 2021
Materias:	language-and-vision multi-modality transformer representation learning effect of language on vision self-attention Electronic computers. Computer science QA75.5-76.95
Acceso en línea:	https://doaj.org/article/e246edf91b36449eace5eac40210015e
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

id	oai:doaj.org-article:e246edf91b36449eace5eac40210015e
record_format	dspace
spelling	oai:doaj.org-article:e246edf91b36449eace5eac40210015e2021-12-03T14:17:31ZWhat Does a Language-And-Vision Transformer See: The Impact of Semantic Information on Visual Representations2624-821210.3389/frai.2021.767971https://doaj.org/article/e246edf91b36449eace5eac40210015e2021-12-01T00:00:00Zhttps://www.frontiersin.org/articles/10.3389/frai.2021.767971/fullhttps://doaj.org/toc/2624-8212Neural networks have proven to be very successful in automatically capturing the composition of language and different structures across a range of multi-modal tasks. Thus, an important question to investigate is how neural networks learn and organise such structures. Numerous studies have examined the knowledge captured by language models (LSTMs, transformers) and vision architectures (CNNs, vision transformers) for respective uni-modal tasks. However, very few have explored what structures are acquired by multi-modal transformers where linguistic and visual features are combined. It is critical to understand the representations learned by each modality, their respective interplay, and the task’s effect on these representations in large-scale architectures. In this paper, we take a multi-modal transformer trained for image captioning and examine the structure of the self-attention patterns extracted from the visual stream. Our results indicate that the information about different relations between objects in the visual stream is hierarchical and varies from local to a global object-level understanding of the image. In particular, while visual representations in the first layers encode the knowledge of relations between semantically similar object detections, often constituting neighbouring objects, deeper layers expand their attention across more distant objects and learn global relations between them. We also show that globally attended objects in deeper layers can be linked with entities described in image descriptions, indicating a critical finding - the indirect effect of language on visual representations. In addition, we highlight how object-based input representations affect the structure of learned visual knowledge and guide the model towards more accurate image descriptions. A parallel question that we investigate is whether the insights from cognitive science echo the structure of representations that the current neural architecture learns. The proposed analysis of the inner workings of multi-modal transformers can be used to better understand and improve on such problems as pre-training of large-scale multi-modal architectures, multi-modal information fusion and probing of attention weights. In general, we contribute to the explainable multi-modal natural language processing and currently shallow understanding of how the input representations and the structure of the multi-modal transformer affect visual representations.Nikolai IlinykhSimon DobnikFrontiers Media S.A.articlelanguage-and-visionmulti-modalitytransformerrepresentation learningeffect of language on visionself-attentionElectronic computers. Computer scienceQA75.5-76.95ENFrontiers in Artificial Intelligence, Vol 4 (2021)
institution	DOAJ
collection	DOAJ
language	EN
topic	language-and-vision multi-modality transformer representation learning effect of language on vision self-attention Electronic computers. Computer science QA75.5-76.95
spellingShingle	language-and-vision multi-modality transformer representation learning effect of language on vision self-attention Electronic computers. Computer science QA75.5-76.95 Nikolai Ilinykh Simon Dobnik What Does a Language-And-Vision Transformer See: The Impact of Semantic Information on Visual Representations
description	Neural networks have proven to be very successful in automatically capturing the composition of language and different structures across a range of multi-modal tasks. Thus, an important question to investigate is how neural networks learn and organise such structures. Numerous studies have examined the knowledge captured by language models (LSTMs, transformers) and vision architectures (CNNs, vision transformers) for respective uni-modal tasks. However, very few have explored what structures are acquired by multi-modal transformers where linguistic and visual features are combined. It is critical to understand the representations learned by each modality, their respective interplay, and the task’s effect on these representations in large-scale architectures. In this paper, we take a multi-modal transformer trained for image captioning and examine the structure of the self-attention patterns extracted from the visual stream. Our results indicate that the information about different relations between objects in the visual stream is hierarchical and varies from local to a global object-level understanding of the image. In particular, while visual representations in the first layers encode the knowledge of relations between semantically similar object detections, often constituting neighbouring objects, deeper layers expand their attention across more distant objects and learn global relations between them. We also show that globally attended objects in deeper layers can be linked with entities described in image descriptions, indicating a critical finding - the indirect effect of language on visual representations. In addition, we highlight how object-based input representations affect the structure of learned visual knowledge and guide the model towards more accurate image descriptions. A parallel question that we investigate is whether the insights from cognitive science echo the structure of representations that the current neural architecture learns. The proposed analysis of the inner workings of multi-modal transformers can be used to better understand and improve on such problems as pre-training of large-scale multi-modal architectures, multi-modal information fusion and probing of attention weights. In general, we contribute to the explainable multi-modal natural language processing and currently shallow understanding of how the input representations and the structure of the multi-modal transformer affect visual representations.
format	article
author	Nikolai Ilinykh Simon Dobnik
author_facet	Nikolai Ilinykh Simon Dobnik
author_sort	Nikolai Ilinykh
title	What Does a Language-And-Vision Transformer See: The Impact of Semantic Information on Visual Representations
title_short	What Does a Language-And-Vision Transformer See: The Impact of Semantic Information on Visual Representations
title_full	What Does a Language-And-Vision Transformer See: The Impact of Semantic Information on Visual Representations
title_fullStr	What Does a Language-And-Vision Transformer See: The Impact of Semantic Information on Visual Representations
title_full_unstemmed	What Does a Language-And-Vision Transformer See: The Impact of Semantic Information on Visual Representations
title_sort	what does a language-and-vision transformer see: the impact of semantic information on visual representations
publisher	Frontiers Media S.A.
publishDate	2021
url	https://doaj.org/article/e246edf91b36449eace5eac40210015e
work_keys_str_mv	AT nikolaiilinykh whatdoesalanguageandvisiontransformerseetheimpactofsemanticinformationonvisualrepresentations AT simondobnik whatdoesalanguageandvisiontransformerseetheimpactofsemanticinformationonvisualrepresentations
_version_	1718373238049865728

What Does a Language-And-Vision Transformer See: The Impact of Semantic Information on Visual Representations

Ejemplares similares