Novelty is not surprise: Human exploratory and adaptive behavior in sequential decision-making.

Classic reinforcement learning (RL) theories cannot explain human behavior in the absence of external reward or when the environment changes. Here, we employ a deep sequential decision-making paradigm with sparse reward and abrupt environmental changes. To explain the behavior of human participants...

Descripción completa

Guardado en:

Detalles Bibliográficos
Autores principales:	He A Xu, Alireza Modirshanechi, Marco P Lehmann, Wulfram Gerstner, Michael H Herzog
Formato:	article
Lenguaje:	EN
Publicado:	Public Library of Science (PLoS) 2021
Materias:	Biology (General) QH301-705.5
Acceso en línea:	https://doaj.org/article/c394c02f72114f7f930a45a422d833f0
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

id	oai:doaj.org-article:c394c02f72114f7f930a45a422d833f0
record_format	dspace
spelling	oai:doaj.org-article:c394c02f72114f7f930a45a422d833f02021-11-25T05:40:38ZNovelty is not surprise: Human exploratory and adaptive behavior in sequential decision-making.1553-734X1553-735810.1371/journal.pcbi.1009070https://doaj.org/article/c394c02f72114f7f930a45a422d833f02021-06-01T00:00:00Zhttps://doi.org/10.1371/journal.pcbi.1009070https://doaj.org/toc/1553-734Xhttps://doaj.org/toc/1553-7358Classic reinforcement learning (RL) theories cannot explain human behavior in the absence of external reward or when the environment changes. Here, we employ a deep sequential decision-making paradigm with sparse reward and abrupt environmental changes. To explain the behavior of human participants in these environments, we show that RL theories need to include surprise and novelty, each with a distinct role. While novelty drives exploration before the first encounter of a reward, surprise increases the rate of learning of a world-model as well as of model-free action-values. Even though the world-model is available for model-based RL, we find that human decisions are dominated by model-free action choices. The world-model is only marginally used for planning, but it is important to detect surprising events. Our theory predicts human action choices with high probability and allows us to dissociate surprise, novelty, and reward in EEG signals.He A XuAlireza ModirshanechiMarco P LehmannWulfram GerstnerMichael H HerzogPublic Library of Science (PLoS)articleBiology (General)QH301-705.5ENPLoS Computational Biology, Vol 17, Iss 6, p e1009070 (2021)
institution	DOAJ
collection	DOAJ
language	EN
topic	Biology (General) QH301-705.5
spellingShingle	Biology (General) QH301-705.5 He A Xu Alireza Modirshanechi Marco P Lehmann Wulfram Gerstner Michael H Herzog Novelty is not surprise: Human exploratory and adaptive behavior in sequential decision-making.
description	Classic reinforcement learning (RL) theories cannot explain human behavior in the absence of external reward or when the environment changes. Here, we employ a deep sequential decision-making paradigm with sparse reward and abrupt environmental changes. To explain the behavior of human participants in these environments, we show that RL theories need to include surprise and novelty, each with a distinct role. While novelty drives exploration before the first encounter of a reward, surprise increases the rate of learning of a world-model as well as of model-free action-values. Even though the world-model is available for model-based RL, we find that human decisions are dominated by model-free action choices. The world-model is only marginally used for planning, but it is important to detect surprising events. Our theory predicts human action choices with high probability and allows us to dissociate surprise, novelty, and reward in EEG signals.
format	article
author	He A Xu Alireza Modirshanechi Marco P Lehmann Wulfram Gerstner Michael H Herzog
author_facet	He A Xu Alireza Modirshanechi Marco P Lehmann Wulfram Gerstner Michael H Herzog
author_sort	He A Xu
title	Novelty is not surprise: Human exploratory and adaptive behavior in sequential decision-making.
title_short	Novelty is not surprise: Human exploratory and adaptive behavior in sequential decision-making.
title_full	Novelty is not surprise: Human exploratory and adaptive behavior in sequential decision-making.
title_fullStr	Novelty is not surprise: Human exploratory and adaptive behavior in sequential decision-making.
title_full_unstemmed	Novelty is not surprise: Human exploratory and adaptive behavior in sequential decision-making.
title_sort	novelty is not surprise: human exploratory and adaptive behavior in sequential decision-making.
publisher	Public Library of Science (PLoS)
publishDate	2021
url	https://doaj.org/article/c394c02f72114f7f930a45a422d833f0
work_keys_str_mv	AT heaxu noveltyisnotsurprisehumanexploratoryandadaptivebehaviorinsequentialdecisionmaking AT alirezamodirshanechi noveltyisnotsurprisehumanexploratoryandadaptivebehaviorinsequentialdecisionmaking AT marcoplehmann noveltyisnotsurprisehumanexploratoryandadaptivebehaviorinsequentialdecisionmaking AT wulframgerstner noveltyisnotsurprisehumanexploratoryandadaptivebehaviorinsequentialdecisionmaking AT michaelhherzog noveltyisnotsurprisehumanexploratoryandadaptivebehaviorinsequentialdecisionmaking
_version_	1718414542536441856

Novelty is not surprise: Human exploratory and adaptive behavior in sequential decision-making.

Ejemplares similares