The Framing of machine learning risk prediction models illustrated by evaluation of sepsis in general wards

Abstract Problem framing is critical to developing risk prediction models because all subsequent development work and evaluation takes place within the context of how a problem has been framed and explicit documentation of framing choices makes it easier to compare evaluation metrics between publish...

Descripción completa

Guardado en:

Detalles Bibliográficos
Autores principales:	Simon Meyer Lauritsen, Bo Thiesson, Marianne Johansson Jørgensen, Anders Hammerich Riis, Ulrick Skipper Espelund, Jesper Bo Weile, Jeppe Lange
Formato:	article
Lenguaje:	EN
Publicado:	Nature Portfolio 2021
Materias:	Computer applications to medicine. Medical informatics R858-859.7
Acceso en línea:	https://doaj.org/article/951bd4e1777041f6af156aa4d5aa40ac
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

id	oai:doaj.org-article:951bd4e1777041f6af156aa4d5aa40ac
record_format	dspace
spelling	oai:doaj.org-article:951bd4e1777041f6af156aa4d5aa40ac2021-11-21T12:05:40ZThe Framing of machine learning risk prediction models illustrated by evaluation of sepsis in general wards10.1038/s41746-021-00529-x2398-6352https://doaj.org/article/951bd4e1777041f6af156aa4d5aa40ac2021-11-01T00:00:00Zhttps://doi.org/10.1038/s41746-021-00529-xhttps://doaj.org/toc/2398-6352Abstract Problem framing is critical to developing risk prediction models because all subsequent development work and evaluation takes place within the context of how a problem has been framed and explicit documentation of framing choices makes it easier to compare evaluation metrics between published studies. In this work, we introduce the basic concepts of framing, including prediction windows, observation windows, window shifts and event-triggers for a prediction that strongly affects the risk of clinician fatigue caused by false positives. Building on this, we apply four different framing structures to the same generic dataset, using a sepsis risk prediction model as an example, and evaluate how framing affects model performance and learning. Our results show that an apparently good model with strong evaluation results in both discrimination and calibration is not necessarily clinically usable. Therefore, it is important to assess the results of objective evaluations within the context of more subjective evaluations of how a model is framed.Simon Meyer LauritsenBo ThiessonMarianne Johansson JørgensenAnders Hammerich RiisUlrick Skipper EspelundJesper Bo WeileJeppe LangeNature PortfolioarticleComputer applications to medicine. Medical informaticsR858-859.7ENnpj Digital Medicine, Vol 4, Iss 1, Pp 1-12 (2021)
institution	DOAJ
collection	DOAJ
language	EN
topic	Computer applications to medicine. Medical informatics R858-859.7
spellingShingle	Computer applications to medicine. Medical informatics R858-859.7 Simon Meyer Lauritsen Bo Thiesson Marianne Johansson Jørgensen Anders Hammerich Riis Ulrick Skipper Espelund Jesper Bo Weile Jeppe Lange The Framing of machine learning risk prediction models illustrated by evaluation of sepsis in general wards
description	Abstract Problem framing is critical to developing risk prediction models because all subsequent development work and evaluation takes place within the context of how a problem has been framed and explicit documentation of framing choices makes it easier to compare evaluation metrics between published studies. In this work, we introduce the basic concepts of framing, including prediction windows, observation windows, window shifts and event-triggers for a prediction that strongly affects the risk of clinician fatigue caused by false positives. Building on this, we apply four different framing structures to the same generic dataset, using a sepsis risk prediction model as an example, and evaluate how framing affects model performance and learning. Our results show that an apparently good model with strong evaluation results in both discrimination and calibration is not necessarily clinically usable. Therefore, it is important to assess the results of objective evaluations within the context of more subjective evaluations of how a model is framed.
format	article
author	Simon Meyer Lauritsen Bo Thiesson Marianne Johansson Jørgensen Anders Hammerich Riis Ulrick Skipper Espelund Jesper Bo Weile Jeppe Lange
author_facet	Simon Meyer Lauritsen Bo Thiesson Marianne Johansson Jørgensen Anders Hammerich Riis Ulrick Skipper Espelund Jesper Bo Weile Jeppe Lange
author_sort	Simon Meyer Lauritsen
title	The Framing of machine learning risk prediction models illustrated by evaluation of sepsis in general wards
title_short	The Framing of machine learning risk prediction models illustrated by evaluation of sepsis in general wards
title_full	The Framing of machine learning risk prediction models illustrated by evaluation of sepsis in general wards
title_fullStr	The Framing of machine learning risk prediction models illustrated by evaluation of sepsis in general wards
title_full_unstemmed	The Framing of machine learning risk prediction models illustrated by evaluation of sepsis in general wards
title_sort	framing of machine learning risk prediction models illustrated by evaluation of sepsis in general wards
publisher	Nature Portfolio
publishDate	2021
url	https://doaj.org/article/951bd4e1777041f6af156aa4d5aa40ac
work_keys_str_mv	AT simonmeyerlauritsen theframingofmachinelearningriskpredictionmodelsillustratedbyevaluationofsepsisingeneralwards AT bothiesson theframingofmachinelearningriskpredictionmodelsillustratedbyevaluationofsepsisingeneralwards AT mariannejohanssonjørgensen theframingofmachinelearningriskpredictionmodelsillustratedbyevaluationofsepsisingeneralwards AT andershammerichriis theframingofmachinelearningriskpredictionmodelsillustratedbyevaluationofsepsisingeneralwards AT ulrickskipperespelund theframingofmachinelearningriskpredictionmodelsillustratedbyevaluationofsepsisingeneralwards AT jesperboweile theframingofmachinelearningriskpredictionmodelsillustratedbyevaluationofsepsisingeneralwards AT jeppelange theframingofmachinelearningriskpredictionmodelsillustratedbyevaluationofsepsisingeneralwards AT simonmeyerlauritsen framingofmachinelearningriskpredictionmodelsillustratedbyevaluationofsepsisingeneralwards AT bothiesson framingofmachinelearningriskpredictionmodelsillustratedbyevaluationofsepsisingeneralwards AT mariannejohanssonjørgensen framingofmachinelearningriskpredictionmodelsillustratedbyevaluationofsepsisingeneralwards AT andershammerichriis framingofmachinelearningriskpredictionmodelsillustratedbyevaluationofsepsisingeneralwards AT ulrickskipperespelund framingofmachinelearningriskpredictionmodelsillustratedbyevaluationofsepsisingeneralwards AT jesperboweile framingofmachinelearningriskpredictionmodelsillustratedbyevaluationofsepsisingeneralwards AT jeppelange framingofmachinelearningriskpredictionmodelsillustratedbyevaluationofsepsisingeneralwards
_version_	1718419259966619648

The Framing of machine learning risk prediction models illustrated by evaluation of sepsis in general wards

Ejemplares similares