Face-to-face panel meetings versus remote evaluation of fellowship applications: simulation study at the Swiss National Science Foundation

Objectives To trial a simplified, time and cost-saving method for remote evaluation of fellowship applications and compare this with existing panel review processes by analysing concordance between funding decisions, and the use of a lottery-based decision method for proposals of similar quality.Des...

Descripción completa

Guardado en:

Detalles Bibliográficos
Autores principales:	Rachel Heyard, Marco Bieri, Katharina Roser
Formato:	article
Lenguaje:	EN
Publicado:	BMJ Publishing Group 2021
Materias:	Medicine R
Acceso en línea:	https://doaj.org/article/3d3dcdfd9fab49f7bf409fda423671ee
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

id	oai:doaj.org-article:3d3dcdfd9fab49f7bf409fda423671ee
record_format	dspace
spelling	oai:doaj.org-article:3d3dcdfd9fab49f7bf409fda423671ee2021-11-29T23:30:04ZFace-to-face panel meetings versus remote evaluation of fellowship applications: simulation study at the Swiss National Science Foundation10.1136/bmjopen-2020-0473862044-6055https://doaj.org/article/3d3dcdfd9fab49f7bf409fda423671ee2021-06-01T00:00:00Zhttps://bmjopen.bmj.com/content/11/5/e047386.fullhttps://doaj.org/toc/2044-6055Objectives To trial a simplified, time and cost-saving method for remote evaluation of fellowship applications and compare this with existing panel review processes by analysing concordance between funding decisions, and the use of a lottery-based decision method for proposals of similar quality.Design The study involved 134 junior fellowship proposals for postdoctoral research (‘Postdoc.Mobility’). The official method used two panel reviewers who independently scored the application, followed by triage and discussion of selected applications in a panel. Very competitive/uncompetitive proposals were directly funded/rejected without discussion. The simplified procedure used the scores of the two panel members, with or without the score of an additional, third expert. Both methods could further use a lottery to decide on applications of similar quality close to the funding threshold. The same funding rate was applied, and the agreement between the two methods analysed.Setting Swiss National Science Foundation (SNSF).Participants Postdoc.Mobility panel reviewers and additional expert reviewers.Primary outcome measure Per cent agreement between the simplified and official evaluation method with 95% CIs.Results The simplified procedure based on three reviews agreed in 80.6% (95% CI: 73.9% to 87.3%) of applicants with the official funding outcome. The agreement was 86.6% (95% CI: 80.6% to 91.8%) when using the two reviews of the panel members. The agreement between the two methods was lower for the group of applications discussed in the panel (64.2% and 73.1%, respectively), and higher for directly funded/rejected applications (range: 96.7%–100%). The lottery was used in 8 (6.0%) of 134 applications (official method), 19 (14.2%) applications (simplified, three reviewers) and 23 (17.2%) applications (simplified, two reviewers). With the simplified procedure, evaluation costs could have been halved and 31 hours of meeting time saved for the two 2019 calls.Conclusion Agreement between the two methods was high. The simplified procedure could represent a viable evaluation method for the Postdoc.Mobility early career instrument at the SNSF.Rachel HeyardMarco BieriKatharina RoserBMJ Publishing GrouparticleMedicineRENBMJ Open, Vol 11, Iss 5 (2021)
institution	DOAJ
collection	DOAJ
language	EN
topic	Medicine R
spellingShingle	Medicine R Rachel Heyard Marco Bieri Katharina Roser Face-to-face panel meetings versus remote evaluation of fellowship applications: simulation study at the Swiss National Science Foundation
description	Objectives To trial a simplified, time and cost-saving method for remote evaluation of fellowship applications and compare this with existing panel review processes by analysing concordance between funding decisions, and the use of a lottery-based decision method for proposals of similar quality.Design The study involved 134 junior fellowship proposals for postdoctoral research (‘Postdoc.Mobility’). The official method used two panel reviewers who independently scored the application, followed by triage and discussion of selected applications in a panel. Very competitive/uncompetitive proposals were directly funded/rejected without discussion. The simplified procedure used the scores of the two panel members, with or without the score of an additional, third expert. Both methods could further use a lottery to decide on applications of similar quality close to the funding threshold. The same funding rate was applied, and the agreement between the two methods analysed.Setting Swiss National Science Foundation (SNSF).Participants Postdoc.Mobility panel reviewers and additional expert reviewers.Primary outcome measure Per cent agreement between the simplified and official evaluation method with 95% CIs.Results The simplified procedure based on three reviews agreed in 80.6% (95% CI: 73.9% to 87.3%) of applicants with the official funding outcome. The agreement was 86.6% (95% CI: 80.6% to 91.8%) when using the two reviews of the panel members. The agreement between the two methods was lower for the group of applications discussed in the panel (64.2% and 73.1%, respectively), and higher for directly funded/rejected applications (range: 96.7%–100%). The lottery was used in 8 (6.0%) of 134 applications (official method), 19 (14.2%) applications (simplified, three reviewers) and 23 (17.2%) applications (simplified, two reviewers). With the simplified procedure, evaluation costs could have been halved and 31 hours of meeting time saved for the two 2019 calls.Conclusion Agreement between the two methods was high. The simplified procedure could represent a viable evaluation method for the Postdoc.Mobility early career instrument at the SNSF.
format	article
author	Rachel Heyard Marco Bieri Katharina Roser
author_facet	Rachel Heyard Marco Bieri Katharina Roser
author_sort	Rachel Heyard
title	Face-to-face panel meetings versus remote evaluation of fellowship applications: simulation study at the Swiss National Science Foundation
title_short	Face-to-face panel meetings versus remote evaluation of fellowship applications: simulation study at the Swiss National Science Foundation
title_full	Face-to-face panel meetings versus remote evaluation of fellowship applications: simulation study at the Swiss National Science Foundation
title_fullStr	Face-to-face panel meetings versus remote evaluation of fellowship applications: simulation study at the Swiss National Science Foundation
title_full_unstemmed	Face-to-face panel meetings versus remote evaluation of fellowship applications: simulation study at the Swiss National Science Foundation
title_sort	face-to-face panel meetings versus remote evaluation of fellowship applications: simulation study at the swiss national science foundation
publisher	BMJ Publishing Group
publishDate	2021
url	https://doaj.org/article/3d3dcdfd9fab49f7bf409fda423671ee
work_keys_str_mv	AT rachelheyard facetofacepanelmeetingsversusremoteevaluationoffellowshipapplicationssimulationstudyattheswissnationalsciencefoundation AT marcobieri facetofacepanelmeetingsversusremoteevaluationoffellowshipapplicationssimulationstudyattheswissnationalsciencefoundation AT katharinaroser facetofacepanelmeetingsversusremoteevaluationoffellowshipapplicationssimulationstudyattheswissnationalsciencefoundation
_version_	1718406857910910976

Face-to-face panel meetings versus remote evaluation of fellowship applications: simulation study at the Swiss National Science Foundation

Ejemplares similares