Inflated prediction accuracy of neuropsychiatric biomarkers caused by data leakage in feature selection

Abstract In recent years, machine learning techniques have been frequently applied to uncovering neuropsychiatric biomarkers with the aim of accurately diagnosing neuropsychiatric diseases and predicting treatment prognosis. However, many studies did not perform cross validation (CV) when using mach...

Descripción completa

Guardado en:

Detalles Bibliográficos
Autores principales:	Miseon Shim, Seung-Hwan Lee, Han-Jeong Hwang
Formato:	article
Lenguaje:	EN
Publicado:	Nature Portfolio 2021
Materias:	Medicine R Science Q
Acceso en línea:	https://doaj.org/article/ad61d8cc3ca4460d809b2864f5cb0832
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

id	oai:doaj.org-article:ad61d8cc3ca4460d809b2864f5cb0832
record_format	dspace
spelling	oai:doaj.org-article:ad61d8cc3ca4460d809b2864f5cb08322021-12-02T18:02:48ZInflated prediction accuracy of neuropsychiatric biomarkers caused by data leakage in feature selection10.1038/s41598-021-87157-32045-2322https://doaj.org/article/ad61d8cc3ca4460d809b2864f5cb08322021-04-01T00:00:00Zhttps://doi.org/10.1038/s41598-021-87157-3https://doaj.org/toc/2045-2322Abstract In recent years, machine learning techniques have been frequently applied to uncovering neuropsychiatric biomarkers with the aim of accurately diagnosing neuropsychiatric diseases and predicting treatment prognosis. However, many studies did not perform cross validation (CV) when using machine learning techniques, or others performed CV in an incorrect manner, leading to significantly biased results due to overfitting problem. The aim of this study is to investigate the impact of CV on the prediction performance of neuropsychiatric biomarkers, in particular, for feature selection performed with high-dimensional features. To this end, we evaluated prediction performances using both simulation data and actual electroencephalography (EEG) data. The overall prediction accuracies of the feature selection method performed outside of CV were considerably higher than those of the feature selection method performed within CV for both the simulation and actual EEG data. The differences between the prediction accuracies of the two feature selection approaches can be thought of as the amount of overfitting due to selection bias. Our results indicate the importance of correctly using CV to avoid biased results of prediction performance of neuropsychiatric biomarkers.Miseon ShimSeung-Hwan LeeHan-Jeong HwangNature PortfolioarticleMedicineRScienceQENScientific Reports, Vol 11, Iss 1, Pp 1-7 (2021)
institution	DOAJ
collection	DOAJ
language	EN
topic	Medicine R Science Q
spellingShingle	Medicine R Science Q Miseon Shim Seung-Hwan Lee Han-Jeong Hwang Inflated prediction accuracy of neuropsychiatric biomarkers caused by data leakage in feature selection
description	Abstract In recent years, machine learning techniques have been frequently applied to uncovering neuropsychiatric biomarkers with the aim of accurately diagnosing neuropsychiatric diseases and predicting treatment prognosis. However, many studies did not perform cross validation (CV) when using machine learning techniques, or others performed CV in an incorrect manner, leading to significantly biased results due to overfitting problem. The aim of this study is to investigate the impact of CV on the prediction performance of neuropsychiatric biomarkers, in particular, for feature selection performed with high-dimensional features. To this end, we evaluated prediction performances using both simulation data and actual electroencephalography (EEG) data. The overall prediction accuracies of the feature selection method performed outside of CV were considerably higher than those of the feature selection method performed within CV for both the simulation and actual EEG data. The differences between the prediction accuracies of the two feature selection approaches can be thought of as the amount of overfitting due to selection bias. Our results indicate the importance of correctly using CV to avoid biased results of prediction performance of neuropsychiatric biomarkers.
format	article
author	Miseon Shim Seung-Hwan Lee Han-Jeong Hwang
author_facet	Miseon Shim Seung-Hwan Lee Han-Jeong Hwang
author_sort	Miseon Shim
title	Inflated prediction accuracy of neuropsychiatric biomarkers caused by data leakage in feature selection
title_short	Inflated prediction accuracy of neuropsychiatric biomarkers caused by data leakage in feature selection
title_full	Inflated prediction accuracy of neuropsychiatric biomarkers caused by data leakage in feature selection
title_fullStr	Inflated prediction accuracy of neuropsychiatric biomarkers caused by data leakage in feature selection
title_full_unstemmed	Inflated prediction accuracy of neuropsychiatric biomarkers caused by data leakage in feature selection
title_sort	inflated prediction accuracy of neuropsychiatric biomarkers caused by data leakage in feature selection
publisher	Nature Portfolio
publishDate	2021
url	https://doaj.org/article/ad61d8cc3ca4460d809b2864f5cb0832
work_keys_str_mv	AT miseonshim inflatedpredictionaccuracyofneuropsychiatricbiomarkerscausedbydataleakageinfeatureselection AT seunghwanlee inflatedpredictionaccuracyofneuropsychiatricbiomarkerscausedbydataleakageinfeatureselection AT hanjeonghwang inflatedpredictionaccuracyofneuropsychiatricbiomarkerscausedbydataleakageinfeatureselection
_version_	1718378889308274688

Inflated prediction accuracy of neuropsychiatric biomarkers caused by data leakage in feature selection

Ejemplares similares