Deep saliency models learn low-, mid-, and high-level features to predict scene attention

Abstract Deep saliency models represent the current state-of-the-art for predicting where humans look in real-world scenes. However, for deep saliency models to inform cognitive theories of attention, we need to know how deep saliency models prioritize different scene features to predict where peopl...

Descripción completa

Guardado en:

Detalles Bibliográficos
Autores principales:	Taylor R. Hayes, John M. Henderson
Formato:	article
Lenguaje:	EN
Publicado:	Nature Portfolio 2021
Materias:	Medicine R Science Q
Acceso en línea:	https://doaj.org/article/ce387fe2b53e4987b3f74cda0a3e4bf3
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

id	oai:doaj.org-article:ce387fe2b53e4987b3f74cda0a3e4bf3
record_format	dspace
spelling	oai:doaj.org-article:ce387fe2b53e4987b3f74cda0a3e4bf32021-12-02T17:24:02ZDeep saliency models learn low-, mid-, and high-level features to predict scene attention10.1038/s41598-021-97879-z2045-2322https://doaj.org/article/ce387fe2b53e4987b3f74cda0a3e4bf32021-09-01T00:00:00Zhttps://doi.org/10.1038/s41598-021-97879-zhttps://doaj.org/toc/2045-2322Abstract Deep saliency models represent the current state-of-the-art for predicting where humans look in real-world scenes. However, for deep saliency models to inform cognitive theories of attention, we need to know how deep saliency models prioritize different scene features to predict where people look. Here we open the black box of three prominent deep saliency models (MSI-Net, DeepGaze II, and SAM-ResNet) using an approach that models the association between attention, deep saliency model output, and low-, mid-, and high-level scene features. Specifically, we measured the association between each deep saliency model and low-level image saliency, mid-level contour symmetry and junctions, and high-level meaning by applying a mixed effects modeling approach to a large eye movement dataset. We found that all three deep saliency models were most strongly associated with high-level and low-level features, but exhibited qualitatively different feature weightings and interaction patterns. These findings suggest that prominent deep saliency models are primarily learning image features associated with high-level scene meaning and low-level image saliency and highlight the importance of moving beyond simply benchmarking performance.Taylor R. HayesJohn M. HendersonNature PortfolioarticleMedicineRScienceQENScientific Reports, Vol 11, Iss 1, Pp 1-13 (2021)
institution	DOAJ
collection	DOAJ
language	EN
topic	Medicine R Science Q
spellingShingle	Medicine R Science Q Taylor R. Hayes John M. Henderson Deep saliency models learn low-, mid-, and high-level features to predict scene attention
description	Abstract Deep saliency models represent the current state-of-the-art for predicting where humans look in real-world scenes. However, for deep saliency models to inform cognitive theories of attention, we need to know how deep saliency models prioritize different scene features to predict where people look. Here we open the black box of three prominent deep saliency models (MSI-Net, DeepGaze II, and SAM-ResNet) using an approach that models the association between attention, deep saliency model output, and low-, mid-, and high-level scene features. Specifically, we measured the association between each deep saliency model and low-level image saliency, mid-level contour symmetry and junctions, and high-level meaning by applying a mixed effects modeling approach to a large eye movement dataset. We found that all three deep saliency models were most strongly associated with high-level and low-level features, but exhibited qualitatively different feature weightings and interaction patterns. These findings suggest that prominent deep saliency models are primarily learning image features associated with high-level scene meaning and low-level image saliency and highlight the importance of moving beyond simply benchmarking performance.
format	article
author	Taylor R. Hayes John M. Henderson
author_facet	Taylor R. Hayes John M. Henderson
author_sort	Taylor R. Hayes
title	Deep saliency models learn low-, mid-, and high-level features to predict scene attention
title_short	Deep saliency models learn low-, mid-, and high-level features to predict scene attention
title_full	Deep saliency models learn low-, mid-, and high-level features to predict scene attention
title_fullStr	Deep saliency models learn low-, mid-, and high-level features to predict scene attention
title_full_unstemmed	Deep saliency models learn low-, mid-, and high-level features to predict scene attention
title_sort	deep saliency models learn low-, mid-, and high-level features to predict scene attention
publisher	Nature Portfolio
publishDate	2021
url	https://doaj.org/article/ce387fe2b53e4987b3f74cda0a3e4bf3
work_keys_str_mv	AT taylorrhayes deepsaliencymodelslearnlowmidandhighlevelfeaturestopredictsceneattention AT johnmhenderson deepsaliencymodelslearnlowmidandhighlevelfeaturestopredictsceneattention
_version_	1718380936262844416

Deep saliency models learn low-, mid-, and high-level features to predict scene attention

Ejemplares similares