A Trust-Based Methodology to Evaluate Deep Learning Models for Automatic Diagnosis of Ocular Toxoplasmosis from Fundus Images

In the automatic diagnosis of ocular toxoplasmosis (OT), Deep Learning (DL) has arisen as a powerful and promising approach for diagnosis. However, despite the good performance of the models, decision rules should be interpretable to elicit trust from the medical community. Therefore, the developmen...

Descripción completa

Guardado en:

Detalles Bibliográficos
Autores principales:	Rodrigo Parra, Verena Ojeda, Jose Luis Vázquez Noguera, Miguel García-Torres, Julio César Mello-Román, Cynthia Villalba, Jacques Facon, Federico Divina, Olivia Cardozo, Verónica Elisa Castillo, Ingrid Castro Matto
Formato:	article
Lenguaje:	EN
Publicado:	MDPI AG 2021
Materias:	deep learning ocular toxoplasmosis machine learning interpretability trust Medicine (General) R5-920
Acceso en línea:	https://doaj.org/article/e9222ee88e094a61ab31e5560289cf84
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

id	oai:doaj.org-article:e9222ee88e094a61ab31e5560289cf84
record_format	dspace
spelling	oai:doaj.org-article:e9222ee88e094a61ab31e5560289cf842021-11-25T17:20:11ZA Trust-Based Methodology to Evaluate Deep Learning Models for Automatic Diagnosis of Ocular Toxoplasmosis from Fundus Images10.3390/diagnostics111119512075-4418https://doaj.org/article/e9222ee88e094a61ab31e5560289cf842021-10-01T00:00:00Zhttps://www.mdpi.com/2075-4418/11/11/1951https://doaj.org/toc/2075-4418In the automatic diagnosis of ocular toxoplasmosis (OT), Deep Learning (DL) has arisen as a powerful and promising approach for diagnosis. However, despite the good performance of the models, decision rules should be interpretable to elicit trust from the medical community. Therefore, the development of an evaluation methodology to assess DL models based on interpretability methods is a challenging task that is necessary to extend the use of AI among clinicians. In this work, we propose a novel methodology to quantify the similarity between the decision rules used by a DL model and an ophthalmologist, based on the assumption that doctors are more likely to trust a prediction that was based on decision rules they can understand. Given an eye fundus image with OT, the proposed methodology compares the segmentation mask of OT lesions labeled by an ophthalmologist with the attribution matrix produced by interpretability methods. Furthermore, an open dataset that includes the eye fundus images and the segmentation masks is shared with the community. The proposal was tested on three different DL architectures. The results suggest that complex models tend to perform worse in terms of likelihood to be trusted while achieving better results in sensitivity and specificity.Rodrigo ParraVerena OjedaJose Luis Vázquez NogueraMiguel García-TorresJulio César Mello-RománCynthia VillalbaJacques FaconFederico DivinaOlivia CardozoVerónica Elisa CastilloIngrid Castro MattoMDPI AGarticledeep learningocular toxoplasmosismachine learning interpretabilitytrustMedicine (General)R5-920ENDiagnostics, Vol 11, Iss 1951, p 1951 (2021)
institution	DOAJ
collection	DOAJ
language	EN
topic	deep learning ocular toxoplasmosis machine learning interpretability trust Medicine (General) R5-920
spellingShingle	deep learning ocular toxoplasmosis machine learning interpretability trust Medicine (General) R5-920 Rodrigo Parra Verena Ojeda Jose Luis Vázquez Noguera Miguel García-Torres Julio César Mello-Román Cynthia Villalba Jacques Facon Federico Divina Olivia Cardozo Verónica Elisa Castillo Ingrid Castro Matto A Trust-Based Methodology to Evaluate Deep Learning Models for Automatic Diagnosis of Ocular Toxoplasmosis from Fundus Images
description	In the automatic diagnosis of ocular toxoplasmosis (OT), Deep Learning (DL) has arisen as a powerful and promising approach for diagnosis. However, despite the good performance of the models, decision rules should be interpretable to elicit trust from the medical community. Therefore, the development of an evaluation methodology to assess DL models based on interpretability methods is a challenging task that is necessary to extend the use of AI among clinicians. In this work, we propose a novel methodology to quantify the similarity between the decision rules used by a DL model and an ophthalmologist, based on the assumption that doctors are more likely to trust a prediction that was based on decision rules they can understand. Given an eye fundus image with OT, the proposed methodology compares the segmentation mask of OT lesions labeled by an ophthalmologist with the attribution matrix produced by interpretability methods. Furthermore, an open dataset that includes the eye fundus images and the segmentation masks is shared with the community. The proposal was tested on three different DL architectures. The results suggest that complex models tend to perform worse in terms of likelihood to be trusted while achieving better results in sensitivity and specificity.
format	article
author	Rodrigo Parra Verena Ojeda Jose Luis Vázquez Noguera Miguel García-Torres Julio César Mello-Román Cynthia Villalba Jacques Facon Federico Divina Olivia Cardozo Verónica Elisa Castillo Ingrid Castro Matto
author_facet	Rodrigo Parra Verena Ojeda Jose Luis Vázquez Noguera Miguel García-Torres Julio César Mello-Román Cynthia Villalba Jacques Facon Federico Divina Olivia Cardozo Verónica Elisa Castillo Ingrid Castro Matto
author_sort	Rodrigo Parra
title	A Trust-Based Methodology to Evaluate Deep Learning Models for Automatic Diagnosis of Ocular Toxoplasmosis from Fundus Images
title_short	A Trust-Based Methodology to Evaluate Deep Learning Models for Automatic Diagnosis of Ocular Toxoplasmosis from Fundus Images
title_full	A Trust-Based Methodology to Evaluate Deep Learning Models for Automatic Diagnosis of Ocular Toxoplasmosis from Fundus Images
title_fullStr	A Trust-Based Methodology to Evaluate Deep Learning Models for Automatic Diagnosis of Ocular Toxoplasmosis from Fundus Images
title_full_unstemmed	A Trust-Based Methodology to Evaluate Deep Learning Models for Automatic Diagnosis of Ocular Toxoplasmosis from Fundus Images
title_sort	trust-based methodology to evaluate deep learning models for automatic diagnosis of ocular toxoplasmosis from fundus images
publisher	MDPI AG
publishDate	2021
url	https://doaj.org/article/e9222ee88e094a61ab31e5560289cf84
work_keys_str_mv	AT rodrigoparra atrustbasedmethodologytoevaluatedeeplearningmodelsforautomaticdiagnosisofoculartoxoplasmosisfromfundusimages AT verenaojeda atrustbasedmethodologytoevaluatedeeplearningmodelsforautomaticdiagnosisofoculartoxoplasmosisfromfundusimages AT joseluisvazqueznoguera atrustbasedmethodologytoevaluatedeeplearningmodelsforautomaticdiagnosisofoculartoxoplasmosisfromfundusimages AT miguelgarciatorres atrustbasedmethodologytoevaluatedeeplearningmodelsforautomaticdiagnosisofoculartoxoplasmosisfromfundusimages AT juliocesarmelloroman atrustbasedmethodologytoevaluatedeeplearningmodelsforautomaticdiagnosisofoculartoxoplasmosisfromfundusimages AT cynthiavillalba atrustbasedmethodologytoevaluatedeeplearningmodelsforautomaticdiagnosisofoculartoxoplasmosisfromfundusimages AT jacquesfacon atrustbasedmethodologytoevaluatedeeplearningmodelsforautomaticdiagnosisofoculartoxoplasmosisfromfundusimages AT federicodivina atrustbasedmethodologytoevaluatedeeplearningmodelsforautomaticdiagnosisofoculartoxoplasmosisfromfundusimages AT oliviacardozo atrustbasedmethodologytoevaluatedeeplearningmodelsforautomaticdiagnosisofoculartoxoplasmosisfromfundusimages AT veronicaelisacastillo atrustbasedmethodologytoevaluatedeeplearningmodelsforautomaticdiagnosisofoculartoxoplasmosisfromfundusimages AT ingridcastromatto atrustbasedmethodologytoevaluatedeeplearningmodelsforautomaticdiagnosisofoculartoxoplasmosisfromfundusimages AT rodrigoparra trustbasedmethodologytoevaluatedeeplearningmodelsforautomaticdiagnosisofoculartoxoplasmosisfromfundusimages AT verenaojeda trustbasedmethodologytoevaluatedeeplearningmodelsforautomaticdiagnosisofoculartoxoplasmosisfromfundusimages AT joseluisvazqueznoguera trustbasedmethodologytoevaluatedeeplearningmodelsforautomaticdiagnosisofoculartoxoplasmosisfromfundusimages AT miguelgarciatorres trustbasedmethodologytoevaluatedeeplearningmodelsforautomaticdiagnosisofoculartoxoplasmosisfromfundusimages AT juliocesarmelloroman trustbasedmethodologytoevaluatedeeplearningmodelsforautomaticdiagnosisofoculartoxoplasmosisfromfundusimages AT cynthiavillalba trustbasedmethodologytoevaluatedeeplearningmodelsforautomaticdiagnosisofoculartoxoplasmosisfromfundusimages AT jacquesfacon trustbasedmethodologytoevaluatedeeplearningmodelsforautomaticdiagnosisofoculartoxoplasmosisfromfundusimages AT federicodivina trustbasedmethodologytoevaluatedeeplearningmodelsforautomaticdiagnosisofoculartoxoplasmosisfromfundusimages AT oliviacardozo trustbasedmethodologytoevaluatedeeplearningmodelsforautomaticdiagnosisofoculartoxoplasmosisfromfundusimages AT veronicaelisacastillo trustbasedmethodologytoevaluatedeeplearningmodelsforautomaticdiagnosisofoculartoxoplasmosisfromfundusimages AT ingridcastromatto trustbasedmethodologytoevaluatedeeplearningmodelsforautomaticdiagnosisofoculartoxoplasmosisfromfundusimages
_version_	1718412503016275968

A Trust-Based Methodology to Evaluate Deep Learning Models for Automatic Diagnosis of Ocular Toxoplasmosis from Fundus Images

Ejemplares similares