Methods and open-source toolkit for analyzing and visualizing challenge results

Abstract Grand challenges have become the de facto standard for benchmarking image analysis algorithms. While the number of these international competitions is steadily increasing, surprisingly little effort has been invested in ensuring high quality design, execution and reporting for these interna...

Descripción completa

Guardado en:

Detalles Bibliográficos
Autores principales:	Manuel Wiesenfarth, Annika Reinke, Bennett A. Landman, Matthias Eisenmann, Laura Aguilera Saiz, M. Jorge Cardoso, Lena Maier-Hein, Annette Kopp-Schneider
Formato:	article
Lenguaje:	EN
Publicado:	Nature Portfolio 2021
Materias:	Medicine R Science Q
Acceso en línea:	https://doaj.org/article/18233fd8c57d48c38215d19972211289
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

id	oai:doaj.org-article:18233fd8c57d48c38215d19972211289
record_format	dspace
spelling	oai:doaj.org-article:18233fd8c57d48c38215d199722112892021-12-02T14:16:17ZMethods and open-source toolkit for analyzing and visualizing challenge results10.1038/s41598-021-82017-62045-2322https://doaj.org/article/18233fd8c57d48c38215d199722112892021-01-01T00:00:00Zhttps://doi.org/10.1038/s41598-021-82017-6https://doaj.org/toc/2045-2322Abstract Grand challenges have become the de facto standard for benchmarking image analysis algorithms. While the number of these international competitions is steadily increasing, surprisingly little effort has been invested in ensuring high quality design, execution and reporting for these international competitions. Specifically, results analysis and visualization in the event of uncertainties have been given almost no attention in the literature. Given these shortcomings, the contribution of this paper is two-fold: (1) we present a set of methods to comprehensively analyze and visualize the results of single-task and multi-task challenges and apply them to a number of simulated and real-life challenges to demonstrate their specific strengths and weaknesses; (2) we release the open-source framework challengeR as part of this work to enable fast and wide adoption of the methodology proposed in this paper. Our approach offers an intuitive way to gain important insights into the relative and absolute performance of algorithms, which cannot be revealed by commonly applied visualization techniques. This is demonstrated by the experiments performed in the specific context of biomedical image analysis challenges. Our framework could thus become an important tool for analyzing and visualizing challenge results in the field of biomedical image analysis and beyond.Manuel WiesenfarthAnnika ReinkeBennett A. LandmanMatthias EisenmannLaura Aguilera SaizM. Jorge CardosoLena Maier-HeinAnnette Kopp-SchneiderNature PortfolioarticleMedicineRScienceQENScientific Reports, Vol 11, Iss 1, Pp 1-15 (2021)
institution	DOAJ
collection	DOAJ
language	EN
topic	Medicine R Science Q
spellingShingle	Medicine R Science Q Manuel Wiesenfarth Annika Reinke Bennett A. Landman Matthias Eisenmann Laura Aguilera Saiz M. Jorge Cardoso Lena Maier-Hein Annette Kopp-Schneider Methods and open-source toolkit for analyzing and visualizing challenge results
description	Abstract Grand challenges have become the de facto standard for benchmarking image analysis algorithms. While the number of these international competitions is steadily increasing, surprisingly little effort has been invested in ensuring high quality design, execution and reporting for these international competitions. Specifically, results analysis and visualization in the event of uncertainties have been given almost no attention in the literature. Given these shortcomings, the contribution of this paper is two-fold: (1) we present a set of methods to comprehensively analyze and visualize the results of single-task and multi-task challenges and apply them to a number of simulated and real-life challenges to demonstrate their specific strengths and weaknesses; (2) we release the open-source framework challengeR as part of this work to enable fast and wide adoption of the methodology proposed in this paper. Our approach offers an intuitive way to gain important insights into the relative and absolute performance of algorithms, which cannot be revealed by commonly applied visualization techniques. This is demonstrated by the experiments performed in the specific context of biomedical image analysis challenges. Our framework could thus become an important tool for analyzing and visualizing challenge results in the field of biomedical image analysis and beyond.
format	article
author	Manuel Wiesenfarth Annika Reinke Bennett A. Landman Matthias Eisenmann Laura Aguilera Saiz M. Jorge Cardoso Lena Maier-Hein Annette Kopp-Schneider
author_facet	Manuel Wiesenfarth Annika Reinke Bennett A. Landman Matthias Eisenmann Laura Aguilera Saiz M. Jorge Cardoso Lena Maier-Hein Annette Kopp-Schneider
author_sort	Manuel Wiesenfarth
title	Methods and open-source toolkit for analyzing and visualizing challenge results
title_short	Methods and open-source toolkit for analyzing and visualizing challenge results
title_full	Methods and open-source toolkit for analyzing and visualizing challenge results
title_fullStr	Methods and open-source toolkit for analyzing and visualizing challenge results
title_full_unstemmed	Methods and open-source toolkit for analyzing and visualizing challenge results
title_sort	methods and open-source toolkit for analyzing and visualizing challenge results
publisher	Nature Portfolio
publishDate	2021
url	https://doaj.org/article/18233fd8c57d48c38215d19972211289
work_keys_str_mv	AT manuelwiesenfarth methodsandopensourcetoolkitforanalyzingandvisualizingchallengeresults AT annikareinke methodsandopensourcetoolkitforanalyzingandvisualizingchallengeresults AT bennettalandman methodsandopensourcetoolkitforanalyzingandvisualizingchallengeresults AT matthiaseisenmann methodsandopensourcetoolkitforanalyzingandvisualizingchallengeresults AT lauraaguilerasaiz methodsandopensourcetoolkitforanalyzingandvisualizingchallengeresults AT mjorgecardoso methodsandopensourcetoolkitforanalyzingandvisualizingchallengeresults AT lenamaierhein methodsandopensourcetoolkitforanalyzingandvisualizingchallengeresults AT annettekoppschneider methodsandopensourcetoolkitforanalyzingandvisualizingchallengeresults
_version_	1718391651931521024

Methods and open-source toolkit for analyzing and visualizing challenge results

Ejemplares similares