Mathematically aggregating experts' predictions of possible futures.

Structured protocols offer a transparent and systematic way to elicit and combine/aggregate, probabilistic predictions from multiple experts. These judgements can be aggregated behaviourally or mathematically to derive a final group prediction. Mathematical rules (e.g., weighted linear combinations...

Descripción completa

Guardado en:

Detalles Bibliográficos
Autores principales:	A M Hanea, D P Wilkinson, M McBride, A Lyon, D van Ravenzwaaij, F Singleton Thorn, C Gray, D R Mandel, A Willcox, E Gould, E T Smith, F Mody, M Bush, F Fidler, H Fraser, B C Wintle
Formato:	article
Lenguaje:	EN
Publicado:	Public Library of Science (PLoS) 2021
Materias:	Medicine R Science Q
Acceso en línea:	https://doaj.org/article/51cfd1033af648108353eb27896efb2e
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

id	oai:doaj.org-article:51cfd1033af648108353eb27896efb2e
record_format	dspace
spelling	oai:doaj.org-article:51cfd1033af648108353eb27896efb2e2021-12-02T20:08:34ZMathematically aggregating experts' predictions of possible futures.1932-620310.1371/journal.pone.0256919https://doaj.org/article/51cfd1033af648108353eb27896efb2e2021-01-01T00:00:00Zhttps://doi.org/10.1371/journal.pone.0256919https://doaj.org/toc/1932-6203Structured protocols offer a transparent and systematic way to elicit and combine/aggregate, probabilistic predictions from multiple experts. These judgements can be aggregated behaviourally or mathematically to derive a final group prediction. Mathematical rules (e.g., weighted linear combinations of judgments) provide an objective approach to aggregation. The quality of this aggregation can be defined in terms of accuracy, calibration and informativeness. These measures can be used to compare different aggregation approaches and help decide on which aggregation produces the "best" final prediction. When experts' performance can be scored on similar questions ahead of time, these scores can be translated into performance-based weights, and a performance-based weighted aggregation can then be used. When this is not possible though, several other aggregation methods, informed by measurable proxies for good performance, can be formulated and compared. Here, we develop a suite of aggregation methods, informed by previous experience and the available literature. We differentially weight our experts' estimates by measures of reasoning, engagement, openness to changing their mind, informativeness, prior knowledge, and extremity, asymmetry or granularity of estimates. Next, we investigate the relative performance of these aggregation methods using three datasets. The main goal of this research is to explore how measures of knowledge and behaviour of individuals can be leveraged to produce a better performing combined group judgment. Although the accuracy, calibration, and informativeness of the majority of methods are very similar, a couple of the aggregation methods consistently distinguish themselves as among the best or worst. Moreover, the majority of methods outperform the usual benchmarks provided by the simple average or the median of estimates.A M HaneaD P WilkinsonM McBrideA LyonD van RavenzwaaijF Singleton ThornC GrayD R MandelA WillcoxE GouldE T SmithF ModyM BushF FidlerH FraserB C WintlePublic Library of Science (PLoS)articleMedicineRScienceQENPLoS ONE, Vol 16, Iss 9, p e0256919 (2021)
institution	DOAJ
collection	DOAJ
language	EN
topic	Medicine R Science Q
spellingShingle	Medicine R Science Q A M Hanea D P Wilkinson M McBride A Lyon D van Ravenzwaaij F Singleton Thorn C Gray D R Mandel A Willcox E Gould E T Smith F Mody M Bush F Fidler H Fraser B C Wintle Mathematically aggregating experts' predictions of possible futures.
description	Structured protocols offer a transparent and systematic way to elicit and combine/aggregate, probabilistic predictions from multiple experts. These judgements can be aggregated behaviourally or mathematically to derive a final group prediction. Mathematical rules (e.g., weighted linear combinations of judgments) provide an objective approach to aggregation. The quality of this aggregation can be defined in terms of accuracy, calibration and informativeness. These measures can be used to compare different aggregation approaches and help decide on which aggregation produces the "best" final prediction. When experts' performance can be scored on similar questions ahead of time, these scores can be translated into performance-based weights, and a performance-based weighted aggregation can then be used. When this is not possible though, several other aggregation methods, informed by measurable proxies for good performance, can be formulated and compared. Here, we develop a suite of aggregation methods, informed by previous experience and the available literature. We differentially weight our experts' estimates by measures of reasoning, engagement, openness to changing their mind, informativeness, prior knowledge, and extremity, asymmetry or granularity of estimates. Next, we investigate the relative performance of these aggregation methods using three datasets. The main goal of this research is to explore how measures of knowledge and behaviour of individuals can be leveraged to produce a better performing combined group judgment. Although the accuracy, calibration, and informativeness of the majority of methods are very similar, a couple of the aggregation methods consistently distinguish themselves as among the best or worst. Moreover, the majority of methods outperform the usual benchmarks provided by the simple average or the median of estimates.
format	article
author	A M Hanea D P Wilkinson M McBride A Lyon D van Ravenzwaaij F Singleton Thorn C Gray D R Mandel A Willcox E Gould E T Smith F Mody M Bush F Fidler H Fraser B C Wintle
author_facet	A M Hanea D P Wilkinson M McBride A Lyon D van Ravenzwaaij F Singleton Thorn C Gray D R Mandel A Willcox E Gould E T Smith F Mody M Bush F Fidler H Fraser B C Wintle
author_sort	A M Hanea
title	Mathematically aggregating experts' predictions of possible futures.
title_short	Mathematically aggregating experts' predictions of possible futures.
title_full	Mathematically aggregating experts' predictions of possible futures.
title_fullStr	Mathematically aggregating experts' predictions of possible futures.
title_full_unstemmed	Mathematically aggregating experts' predictions of possible futures.
title_sort	mathematically aggregating experts' predictions of possible futures.
publisher	Public Library of Science (PLoS)
publishDate	2021
url	https://doaj.org/article/51cfd1033af648108353eb27896efb2e
work_keys_str_mv	AT amhanea mathematicallyaggregatingexpertspredictionsofpossiblefutures AT dpwilkinson mathematicallyaggregatingexpertspredictionsofpossiblefutures AT mmcbride mathematicallyaggregatingexpertspredictionsofpossiblefutures AT alyon mathematicallyaggregatingexpertspredictionsofpossiblefutures AT dvanravenzwaaij mathematicallyaggregatingexpertspredictionsofpossiblefutures AT fsingletonthorn mathematicallyaggregatingexpertspredictionsofpossiblefutures AT cgray mathematicallyaggregatingexpertspredictionsofpossiblefutures AT drmandel mathematicallyaggregatingexpertspredictionsofpossiblefutures AT awillcox mathematicallyaggregatingexpertspredictionsofpossiblefutures AT egould mathematicallyaggregatingexpertspredictionsofpossiblefutures AT etsmith mathematicallyaggregatingexpertspredictionsofpossiblefutures AT fmody mathematicallyaggregatingexpertspredictionsofpossiblefutures AT mbush mathematicallyaggregatingexpertspredictionsofpossiblefutures AT ffidler mathematicallyaggregatingexpertspredictionsofpossiblefutures AT hfraser mathematicallyaggregatingexpertspredictionsofpossiblefutures AT bcwintle mathematicallyaggregatingexpertspredictionsofpossiblefutures
_version_	1718375180820021248

Mathematically aggregating experts' predictions of possible futures.

Ejemplares similares