Assessing the accuracy of automatic speech recognition for psychotherapy

Abstract Accurate transcription of audio recordings in psychotherapy would improve therapy effectiveness, clinician training, and safety monitoring. Although automatic speech recognition software is commercially available, its accuracy in mental health settings has not been well described. It is unc...

Descripción completa

Guardado en:

Detalles Bibliográficos
Autores principales:	Adam S. Miner, Albert Haque, Jason A. Fries, Scott L. Fleming, Denise E. Wilfley, G. Terence Wilson, Arnold Milstein, Dan Jurafsky, Bruce A. Arnow, W. Stewart Agras, Li Fei-Fei, Nigam H. Shah
Formato:	article
Lenguaje:	EN
Publicado:	Nature Portfolio 2020
Materias:	Computer applications to medicine. Medical informatics R858-859.7
Acceso en línea:	https://doaj.org/article/18d9cddcaf5d45d09a077f644a1611a5
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

id	oai:doaj.org-article:18d9cddcaf5d45d09a077f644a1611a5
record_format	dspace
spelling	oai:doaj.org-article:18d9cddcaf5d45d09a077f644a1611a52021-12-02T17:50:57ZAssessing the accuracy of automatic speech recognition for psychotherapy10.1038/s41746-020-0285-82398-6352https://doaj.org/article/18d9cddcaf5d45d09a077f644a1611a52020-06-01T00:00:00Zhttps://doi.org/10.1038/s41746-020-0285-8https://doaj.org/toc/2398-6352Abstract Accurate transcription of audio recordings in psychotherapy would improve therapy effectiveness, clinician training, and safety monitoring. Although automatic speech recognition software is commercially available, its accuracy in mental health settings has not been well described. It is unclear which metrics and thresholds are appropriate for different clinical use cases, which may range from population descriptions to individual safety monitoring. Here we show that automatic speech recognition is feasible in psychotherapy, but further improvements in accuracy are needed before widespread use. Our HIPAA-compliant automatic speech recognition system demonstrated a transcription word error rate of 25%. For depression-related utterances, sensitivity was 80% and positive predictive value was 83%. For clinician-identified harm-related sentences, the word error rate was 34%. These results suggest that automatic speech recognition may support understanding of language patterns and subgroup variation in existing treatments but may not be ready for individual-level safety surveillance.Adam S. MinerAlbert HaqueJason A. FriesScott L. FlemingDenise E. WilfleyG. Terence WilsonArnold MilsteinDan JurafskyBruce A. ArnowW. Stewart AgrasLi Fei-FeiNigam H. ShahNature PortfolioarticleComputer applications to medicine. Medical informaticsR858-859.7ENnpj Digital Medicine, Vol 3, Iss 1, Pp 1-8 (2020)
institution	DOAJ
collection	DOAJ
language	EN
topic	Computer applications to medicine. Medical informatics R858-859.7
spellingShingle	Computer applications to medicine. Medical informatics R858-859.7 Adam S. Miner Albert Haque Jason A. Fries Scott L. Fleming Denise E. Wilfley G. Terence Wilson Arnold Milstein Dan Jurafsky Bruce A. Arnow W. Stewart Agras Li Fei-Fei Nigam H. Shah Assessing the accuracy of automatic speech recognition for psychotherapy
description	Abstract Accurate transcription of audio recordings in psychotherapy would improve therapy effectiveness, clinician training, and safety monitoring. Although automatic speech recognition software is commercially available, its accuracy in mental health settings has not been well described. It is unclear which metrics and thresholds are appropriate for different clinical use cases, which may range from population descriptions to individual safety monitoring. Here we show that automatic speech recognition is feasible in psychotherapy, but further improvements in accuracy are needed before widespread use. Our HIPAA-compliant automatic speech recognition system demonstrated a transcription word error rate of 25%. For depression-related utterances, sensitivity was 80% and positive predictive value was 83%. For clinician-identified harm-related sentences, the word error rate was 34%. These results suggest that automatic speech recognition may support understanding of language patterns and subgroup variation in existing treatments but may not be ready for individual-level safety surveillance.
format	article
author	Adam S. Miner Albert Haque Jason A. Fries Scott L. Fleming Denise E. Wilfley G. Terence Wilson Arnold Milstein Dan Jurafsky Bruce A. Arnow W. Stewart Agras Li Fei-Fei Nigam H. Shah
author_facet	Adam S. Miner Albert Haque Jason A. Fries Scott L. Fleming Denise E. Wilfley G. Terence Wilson Arnold Milstein Dan Jurafsky Bruce A. Arnow W. Stewart Agras Li Fei-Fei Nigam H. Shah
author_sort	Adam S. Miner
title	Assessing the accuracy of automatic speech recognition for psychotherapy
title_short	Assessing the accuracy of automatic speech recognition for psychotherapy
title_full	Assessing the accuracy of automatic speech recognition for psychotherapy
title_fullStr	Assessing the accuracy of automatic speech recognition for psychotherapy
title_full_unstemmed	Assessing the accuracy of automatic speech recognition for psychotherapy
title_sort	assessing the accuracy of automatic speech recognition for psychotherapy
publisher	Nature Portfolio
publishDate	2020
url	https://doaj.org/article/18d9cddcaf5d45d09a077f644a1611a5
work_keys_str_mv	AT adamsminer assessingtheaccuracyofautomaticspeechrecognitionforpsychotherapy AT alberthaque assessingtheaccuracyofautomaticspeechrecognitionforpsychotherapy AT jasonafries assessingtheaccuracyofautomaticspeechrecognitionforpsychotherapy AT scottlfleming assessingtheaccuracyofautomaticspeechrecognitionforpsychotherapy AT deniseewilfley assessingtheaccuracyofautomaticspeechrecognitionforpsychotherapy AT gterencewilson assessingtheaccuracyofautomaticspeechrecognitionforpsychotherapy AT arnoldmilstein assessingtheaccuracyofautomaticspeechrecognitionforpsychotherapy AT danjurafsky assessingtheaccuracyofautomaticspeechrecognitionforpsychotherapy AT bruceaarnow assessingtheaccuracyofautomaticspeechrecognitionforpsychotherapy AT wstewartagras assessingtheaccuracyofautomaticspeechrecognitionforpsychotherapy AT lifeifei assessingtheaccuracyofautomaticspeechrecognitionforpsychotherapy AT nigamhshah assessingtheaccuracyofautomaticspeechrecognitionforpsychotherapy
_version_	1718379271294025728

Assessing the accuracy of automatic speech recognition for psychotherapy

Ejemplares similares