Identifying and Minimizing Errors in the Measurement of Early Childhood Development: Lessons Learned from the Cognitive Testing of the ECDI2030

Challenges in measuring early childhood development (ECD) at scale have been documented, yet little is known about the specific difficulties related to questionnaire design and question interpretation. The purpose of this paper is to discuss the challenges of measuring ECD at scale in the context of...

Descripción completa

Guardado en:

Detalles Bibliográficos
Autores principales:	Claudia Cappa, Nicole Petrowski, Elga Filipa De Castro, Emily Geisen, Patricia LeBaron, Betania Allen-Leigh, Jean Marie Place, Paul J. Scanlon
Formato:	article
Lenguaje:	EN
Publicado:	MDPI AG 2021
Materias:	early childhood cultural relevance cognitive testing questionnaire design Medicine R
Acceso en línea:	https://doaj.org/article/a16d2fded54644b9b2c8b175e2c62a57
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

id	oai:doaj.org-article:a16d2fded54644b9b2c8b175e2c62a57
record_format	dspace
spelling	oai:doaj.org-article:a16d2fded54644b9b2c8b175e2c62a572021-11-25T17:51:47ZIdentifying and Minimizing Errors in the Measurement of Early Childhood Development: Lessons Learned from the Cognitive Testing of the ECDI203010.3390/ijerph1822121811660-46011661-7827https://doaj.org/article/a16d2fded54644b9b2c8b175e2c62a572021-11-01T00:00:00Zhttps://www.mdpi.com/1660-4601/18/22/12181https://doaj.org/toc/1661-7827https://doaj.org/toc/1660-4601Challenges in measuring early childhood development (ECD) at scale have been documented, yet little is known about the specific difficulties related to questionnaire design and question interpretation. The purpose of this paper is to discuss the challenges of measuring ECD at scale in the context of household surveys and to show how to overcome them. The paper uses examples from the cognitive interviewing exercises that were conducted as part of the methodological work to develop a measure of ECD outcomes, the ECDI2030. It describes the methodological work carried out to inform the selection and improvement of question items and survey implementation tools as a fundamental step to reduce and mitigate systematic measurement error and improve data quality. The project consisted of a total of five rounds of testing, comprising 191 one-on-one, in-depth cognitive interviews across six countries (Bulgaria, India, Jamaica, Mexico, Uganda, and the USA). Qualitative data analysis methods were used to determine matches and mismatches between intention of items and false positives or false negative answers among subgroups of respondents. Key themes emerged that could potentially lead to systematic measurement error in population-based surveys on ECD: (1) willingness of child to perform task versus ability of child to perform task; (2) performing task versus performing task correctly; (3) identifying letters or numbers versus recognizing letters or numbers; (4) consistently performing task versus correctly performing task; (5) applicability of skills being asked versus observability of skills being asked; and (6) language production versus language comprehension. Through an iterative process of testing and subsequent revision, improvements were made to item wording, response options, and interviewer training instructions. Given the difficulties inherent in population-level data collection in the context of global monitoring, this study’s findings confirm the importance of cognitive testing as a crucial step in careful, culturally relevant, and sensitive questionnaire design and as a means to reduce response bias in cross-cultural contexts.Claudia CappaNicole PetrowskiElga Filipa De CastroEmily GeisenPatricia LeBaronBetania Allen-LeighJean Marie PlacePaul J. ScanlonMDPI AGarticleearly childhoodcultural relevancecognitive testingquestionnaire designMedicineRENInternational Journal of Environmental Research and Public Health, Vol 18, Iss 12181, p 12181 (2021)
institution	DOAJ
collection	DOAJ
language	EN
topic	early childhood cultural relevance cognitive testing questionnaire design Medicine R
spellingShingle	early childhood cultural relevance cognitive testing questionnaire design Medicine R Claudia Cappa Nicole Petrowski Elga Filipa De Castro Emily Geisen Patricia LeBaron Betania Allen-Leigh Jean Marie Place Paul J. Scanlon Identifying and Minimizing Errors in the Measurement of Early Childhood Development: Lessons Learned from the Cognitive Testing of the ECDI2030
description	Challenges in measuring early childhood development (ECD) at scale have been documented, yet little is known about the specific difficulties related to questionnaire design and question interpretation. The purpose of this paper is to discuss the challenges of measuring ECD at scale in the context of household surveys and to show how to overcome them. The paper uses examples from the cognitive interviewing exercises that were conducted as part of the methodological work to develop a measure of ECD outcomes, the ECDI2030. It describes the methodological work carried out to inform the selection and improvement of question items and survey implementation tools as a fundamental step to reduce and mitigate systematic measurement error and improve data quality. The project consisted of a total of five rounds of testing, comprising 191 one-on-one, in-depth cognitive interviews across six countries (Bulgaria, India, Jamaica, Mexico, Uganda, and the USA). Qualitative data analysis methods were used to determine matches and mismatches between intention of items and false positives or false negative answers among subgroups of respondents. Key themes emerged that could potentially lead to systematic measurement error in population-based surveys on ECD: (1) willingness of child to perform task versus ability of child to perform task; (2) performing task versus performing task correctly; (3) identifying letters or numbers versus recognizing letters or numbers; (4) consistently performing task versus correctly performing task; (5) applicability of skills being asked versus observability of skills being asked; and (6) language production versus language comprehension. Through an iterative process of testing and subsequent revision, improvements were made to item wording, response options, and interviewer training instructions. Given the difficulties inherent in population-level data collection in the context of global monitoring, this study’s findings confirm the importance of cognitive testing as a crucial step in careful, culturally relevant, and sensitive questionnaire design and as a means to reduce response bias in cross-cultural contexts.
format	article
author	Claudia Cappa Nicole Petrowski Elga Filipa De Castro Emily Geisen Patricia LeBaron Betania Allen-Leigh Jean Marie Place Paul J. Scanlon
author_facet	Claudia Cappa Nicole Petrowski Elga Filipa De Castro Emily Geisen Patricia LeBaron Betania Allen-Leigh Jean Marie Place Paul J. Scanlon
author_sort	Claudia Cappa
title	Identifying and Minimizing Errors in the Measurement of Early Childhood Development: Lessons Learned from the Cognitive Testing of the ECDI2030
title_short	Identifying and Minimizing Errors in the Measurement of Early Childhood Development: Lessons Learned from the Cognitive Testing of the ECDI2030
title_full	Identifying and Minimizing Errors in the Measurement of Early Childhood Development: Lessons Learned from the Cognitive Testing of the ECDI2030
title_fullStr	Identifying and Minimizing Errors in the Measurement of Early Childhood Development: Lessons Learned from the Cognitive Testing of the ECDI2030
title_full_unstemmed	Identifying and Minimizing Errors in the Measurement of Early Childhood Development: Lessons Learned from the Cognitive Testing of the ECDI2030
title_sort	identifying and minimizing errors in the measurement of early childhood development: lessons learned from the cognitive testing of the ecdi2030
publisher	MDPI AG
publishDate	2021
url	https://doaj.org/article/a16d2fded54644b9b2c8b175e2c62a57
work_keys_str_mv	AT claudiacappa identifyingandminimizingerrorsinthemeasurementofearlychildhooddevelopmentlessonslearnedfromthecognitivetestingoftheecdi2030 AT nicolepetrowski identifyingandminimizingerrorsinthemeasurementofearlychildhooddevelopmentlessonslearnedfromthecognitivetestingoftheecdi2030 AT elgafilipadecastro identifyingandminimizingerrorsinthemeasurementofearlychildhooddevelopmentlessonslearnedfromthecognitivetestingoftheecdi2030 AT emilygeisen identifyingandminimizingerrorsinthemeasurementofearlychildhooddevelopmentlessonslearnedfromthecognitivetestingoftheecdi2030 AT patricialebaron identifyingandminimizingerrorsinthemeasurementofearlychildhooddevelopmentlessonslearnedfromthecognitivetestingoftheecdi2030 AT betaniaallenleigh identifyingandminimizingerrorsinthemeasurementofearlychildhooddevelopmentlessonslearnedfromthecognitivetestingoftheecdi2030 AT jeanmarieplace identifyingandminimizingerrorsinthemeasurementofearlychildhooddevelopmentlessonslearnedfromthecognitivetestingoftheecdi2030 AT pauljscanlon identifyingandminimizingerrorsinthemeasurementofearlychildhooddevelopmentlessonslearnedfromthecognitivetestingoftheecdi2030
_version_	1718411938121121792

Identifying and Minimizing Errors in the Measurement of Early Childhood Development: Lessons Learned from the Cognitive Testing of the ECDI2030

Ejemplares similares