Development and validation of an interpretable neural network for prediction of postoperative in-hospital mortality

Abstract While deep neural networks (DNNs) and other machine learning models often have higher accuracy than simpler models like logistic regression (LR), they are often considered to be “black box” models and this lack of interpretability and transparency is considered a challenge for clinical adop...

Descripción completa

Guardado en:

Detalles Bibliográficos
Autores principales:	Christine K. Lee, Muntaha Samad, Ira Hofer, Maxime Cannesson, Pierre Baldi
Formato:	article
Lenguaje:	EN
Publicado:	Nature Portfolio 2021
Materias:	Computer applications to medicine. Medical informatics R858-859.7
Acceso en línea:	https://doaj.org/article/62aa884547394eaa8877a5d5995160f4
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

id	oai:doaj.org-article:62aa884547394eaa8877a5d5995160f4
record_format	dspace
spelling	oai:doaj.org-article:62aa884547394eaa8877a5d5995160f42021-12-02T13:35:39ZDevelopment and validation of an interpretable neural network for prediction of postoperative in-hospital mortality10.1038/s41746-020-00377-12398-6352https://doaj.org/article/62aa884547394eaa8877a5d5995160f42021-01-01T00:00:00Zhttps://doi.org/10.1038/s41746-020-00377-1https://doaj.org/toc/2398-6352Abstract While deep neural networks (DNNs) and other machine learning models often have higher accuracy than simpler models like logistic regression (LR), they are often considered to be “black box” models and this lack of interpretability and transparency is considered a challenge for clinical adoption. In healthcare, intelligible models not only help clinicians to understand the problem and create more targeted action plans, but also help to gain the clinicians’ trust. One method of overcoming the limited interpretability of more complex models is to use Generalized Additive Models (GAMs). Standard GAMs simply model the target response as a sum of univariate models. Inspired by GAMs, the same idea can be applied to neural networks through an architecture referred to as Generalized Additive Models with Neural Networks (GAM-NNs). In this manuscript, we present the development and validation of a model applying the concept of GAM-NNs to allow for interpretability by visualizing the learned feature patterns related to risk of in-hospital mortality for patients undergoing surgery under general anesthesia. The data consists of 59,985 patients with a feature set of 46 features extracted at the end of surgery to which we added previously not included features: total anesthesia case time (1 feature); the time in minutes spent with mean arterial pressure (MAP) below 40, 45, 50, 55, 60, and 65 mmHg during surgery (6 features); and Healthcare Cost and Utilization Project (HCUP) Code Descriptions of the Primary current procedure terminology (CPT) codes (33 features) for a total of 86 features. All data were randomly split into 80% for training (n = 47,988) and 20% for testing (n = 11,997) prior to model development. Model performance was compared to a standard LR model using the same features as the GAM-NN. The data consisted of 59,985 surgical records, and the occurrence of in-hospital mortality was 0.81% in the training set and 0.72% in the testing set. The GAM-NN model with HCUP features had the highest area under the curve (AUC) 0.921 (0.895–0.95). Overall, both GAM-NN models had higher AUCs than LR models, however, had lower average precisions. The LR model without HCUP features had the highest average precision 0.217 (0.136–0.31). To assess the interpretability of the GAM-NNs, we then visualized the learned contributions of the GAM-NNs and compared against the learned contributions of the LRs for the models with HCUP features. Overall, we were able to demonstrate that our proposed generalized additive neural network (GAM-NN) architecture is able to (1) leverage a neural network’s ability to learn nonlinear patterns in the data, which is more clinically intuitive, (2) be interpreted easily, making it more clinically useful, and (3) maintain model performance as compared to previously published DNNs.Christine K. LeeMuntaha SamadIra HoferMaxime CannessonPierre BaldiNature PortfolioarticleComputer applications to medicine. Medical informaticsR858-859.7ENnpj Digital Medicine, Vol 4, Iss 1, Pp 1-9 (2021)
institution	DOAJ
collection	DOAJ
language	EN
topic	Computer applications to medicine. Medical informatics R858-859.7
spellingShingle	Computer applications to medicine. Medical informatics R858-859.7 Christine K. Lee Muntaha Samad Ira Hofer Maxime Cannesson Pierre Baldi Development and validation of an interpretable neural network for prediction of postoperative in-hospital mortality
description	Abstract While deep neural networks (DNNs) and other machine learning models often have higher accuracy than simpler models like logistic regression (LR), they are often considered to be “black box” models and this lack of interpretability and transparency is considered a challenge for clinical adoption. In healthcare, intelligible models not only help clinicians to understand the problem and create more targeted action plans, but also help to gain the clinicians’ trust. One method of overcoming the limited interpretability of more complex models is to use Generalized Additive Models (GAMs). Standard GAMs simply model the target response as a sum of univariate models. Inspired by GAMs, the same idea can be applied to neural networks through an architecture referred to as Generalized Additive Models with Neural Networks (GAM-NNs). In this manuscript, we present the development and validation of a model applying the concept of GAM-NNs to allow for interpretability by visualizing the learned feature patterns related to risk of in-hospital mortality for patients undergoing surgery under general anesthesia. The data consists of 59,985 patients with a feature set of 46 features extracted at the end of surgery to which we added previously not included features: total anesthesia case time (1 feature); the time in minutes spent with mean arterial pressure (MAP) below 40, 45, 50, 55, 60, and 65 mmHg during surgery (6 features); and Healthcare Cost and Utilization Project (HCUP) Code Descriptions of the Primary current procedure terminology (CPT) codes (33 features) for a total of 86 features. All data were randomly split into 80% for training (n = 47,988) and 20% for testing (n = 11,997) prior to model development. Model performance was compared to a standard LR model using the same features as the GAM-NN. The data consisted of 59,985 surgical records, and the occurrence of in-hospital mortality was 0.81% in the training set and 0.72% in the testing set. The GAM-NN model with HCUP features had the highest area under the curve (AUC) 0.921 (0.895–0.95). Overall, both GAM-NN models had higher AUCs than LR models, however, had lower average precisions. The LR model without HCUP features had the highest average precision 0.217 (0.136–0.31). To assess the interpretability of the GAM-NNs, we then visualized the learned contributions of the GAM-NNs and compared against the learned contributions of the LRs for the models with HCUP features. Overall, we were able to demonstrate that our proposed generalized additive neural network (GAM-NN) architecture is able to (1) leverage a neural network’s ability to learn nonlinear patterns in the data, which is more clinically intuitive, (2) be interpreted easily, making it more clinically useful, and (3) maintain model performance as compared to previously published DNNs.
format	article
author	Christine K. Lee Muntaha Samad Ira Hofer Maxime Cannesson Pierre Baldi
author_facet	Christine K. Lee Muntaha Samad Ira Hofer Maxime Cannesson Pierre Baldi
author_sort	Christine K. Lee
title	Development and validation of an interpretable neural network for prediction of postoperative in-hospital mortality
title_short	Development and validation of an interpretable neural network for prediction of postoperative in-hospital mortality
title_full	Development and validation of an interpretable neural network for prediction of postoperative in-hospital mortality
title_fullStr	Development and validation of an interpretable neural network for prediction of postoperative in-hospital mortality
title_full_unstemmed	Development and validation of an interpretable neural network for prediction of postoperative in-hospital mortality
title_sort	development and validation of an interpretable neural network for prediction of postoperative in-hospital mortality
publisher	Nature Portfolio
publishDate	2021
url	https://doaj.org/article/62aa884547394eaa8877a5d5995160f4
work_keys_str_mv	AT christineklee developmentandvalidationofaninterpretableneuralnetworkforpredictionofpostoperativeinhospitalmortality AT muntahasamad developmentandvalidationofaninterpretableneuralnetworkforpredictionofpostoperativeinhospitalmortality AT irahofer developmentandvalidationofaninterpretableneuralnetworkforpredictionofpostoperativeinhospitalmortality AT maximecannesson developmentandvalidationofaninterpretableneuralnetworkforpredictionofpostoperativeinhospitalmortality AT pierrebaldi developmentandvalidationofaninterpretableneuralnetworkforpredictionofpostoperativeinhospitalmortality
_version_	1718392676634591232

Development and validation of an interpretable neural network for prediction of postoperative in-hospital mortality

Ejemplares similares