Data valuation for medical imaging using Shapley value and application to a large-scale chest X-ray dataset

Abstract The reliability of machine learning models can be compromised when trained on low quality data. Many large-scale medical imaging datasets contain low quality labels extracted from sources such as medical reports. Moreover, images within a dataset may have heterogeneous quality due to artifa...

Descripción completa

Guardado en:

Detalles Bibliográficos
Autores principales:	Siyi Tang, Amirata Ghorbani, Rikiya Yamashita, Sameer Rehman, Jared A. Dunnmon, James Zou, Daniel L. Rubin
Formato:	article
Lenguaje:	EN
Publicado:	Nature Portfolio 2021
Materias:	Medicine R Science Q
Acceso en línea:	https://doaj.org/article/115fcccb74764f0687b45ec863822f0b
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

id	oai:doaj.org-article:115fcccb74764f0687b45ec863822f0b
record_format	dspace
spelling	oai:doaj.org-article:115fcccb74764f0687b45ec863822f0b2021-12-02T18:03:26ZData valuation for medical imaging using Shapley value and application to a large-scale chest X-ray dataset10.1038/s41598-021-87762-22045-2322https://doaj.org/article/115fcccb74764f0687b45ec863822f0b2021-04-01T00:00:00Zhttps://doi.org/10.1038/s41598-021-87762-2https://doaj.org/toc/2045-2322Abstract The reliability of machine learning models can be compromised when trained on low quality data. Many large-scale medical imaging datasets contain low quality labels extracted from sources such as medical reports. Moreover, images within a dataset may have heterogeneous quality due to artifacts and biases arising from equipment or measurement errors. Therefore, algorithms that can automatically identify low quality data are highly desired. In this study, we used data Shapley, a data valuation metric, to quantify the value of training data to the performance of a pneumonia detection algorithm in a large chest X-ray dataset. We characterized the effectiveness of data Shapley in identifying low quality versus valuable data for pneumonia detection. We found that removing training data with high Shapley values decreased the pneumonia detection performance, whereas removing data with low Shapley values improved the model performance. Furthermore, there were more mislabeled examples in low Shapley value data and more true pneumonia cases in high Shapley value data. Our results suggest that low Shapley value indicates mislabeled or poor quality images, whereas high Shapley value indicates data that are valuable for pneumonia detection. Our method can serve as a framework for using data Shapley to denoise large-scale medical imaging datasets.Siyi TangAmirata GhorbaniRikiya YamashitaSameer RehmanJared A. DunnmonJames ZouDaniel L. RubinNature PortfolioarticleMedicineRScienceQENScientific Reports, Vol 11, Iss 1, Pp 1-9 (2021)
institution	DOAJ
collection	DOAJ
language	EN
topic	Medicine R Science Q
spellingShingle	Medicine R Science Q Siyi Tang Amirata Ghorbani Rikiya Yamashita Sameer Rehman Jared A. Dunnmon James Zou Daniel L. Rubin Data valuation for medical imaging using Shapley value and application to a large-scale chest X-ray dataset
description	Abstract The reliability of machine learning models can be compromised when trained on low quality data. Many large-scale medical imaging datasets contain low quality labels extracted from sources such as medical reports. Moreover, images within a dataset may have heterogeneous quality due to artifacts and biases arising from equipment or measurement errors. Therefore, algorithms that can automatically identify low quality data are highly desired. In this study, we used data Shapley, a data valuation metric, to quantify the value of training data to the performance of a pneumonia detection algorithm in a large chest X-ray dataset. We characterized the effectiveness of data Shapley in identifying low quality versus valuable data for pneumonia detection. We found that removing training data with high Shapley values decreased the pneumonia detection performance, whereas removing data with low Shapley values improved the model performance. Furthermore, there were more mislabeled examples in low Shapley value data and more true pneumonia cases in high Shapley value data. Our results suggest that low Shapley value indicates mislabeled or poor quality images, whereas high Shapley value indicates data that are valuable for pneumonia detection. Our method can serve as a framework for using data Shapley to denoise large-scale medical imaging datasets.
format	article
author	Siyi Tang Amirata Ghorbani Rikiya Yamashita Sameer Rehman Jared A. Dunnmon James Zou Daniel L. Rubin
author_facet	Siyi Tang Amirata Ghorbani Rikiya Yamashita Sameer Rehman Jared A. Dunnmon James Zou Daniel L. Rubin
author_sort	Siyi Tang
title	Data valuation for medical imaging using Shapley value and application to a large-scale chest X-ray dataset
title_short	Data valuation for medical imaging using Shapley value and application to a large-scale chest X-ray dataset
title_full	Data valuation for medical imaging using Shapley value and application to a large-scale chest X-ray dataset
title_fullStr	Data valuation for medical imaging using Shapley value and application to a large-scale chest X-ray dataset
title_full_unstemmed	Data valuation for medical imaging using Shapley value and application to a large-scale chest X-ray dataset
title_sort	data valuation for medical imaging using shapley value and application to a large-scale chest x-ray dataset
publisher	Nature Portfolio
publishDate	2021
url	https://doaj.org/article/115fcccb74764f0687b45ec863822f0b
work_keys_str_mv	AT siyitang datavaluationformedicalimagingusingshapleyvalueandapplicationtoalargescalechestxraydataset AT amirataghorbani datavaluationformedicalimagingusingshapleyvalueandapplicationtoalargescalechestxraydataset AT rikiyayamashita datavaluationformedicalimagingusingshapleyvalueandapplicationtoalargescalechestxraydataset AT sameerrehman datavaluationformedicalimagingusingshapleyvalueandapplicationtoalargescalechestxraydataset AT jaredadunnmon datavaluationformedicalimagingusingshapleyvalueandapplicationtoalargescalechestxraydataset AT jameszou datavaluationformedicalimagingusingshapleyvalueandapplicationtoalargescalechestxraydataset AT daniellrubin datavaluationformedicalimagingusingshapleyvalueandapplicationtoalargescalechestxraydataset
_version_	1718378734049820672

Data valuation for medical imaging using Shapley value and application to a large-scale chest X-ray dataset

Ejemplares similares