Bias in Zipf’s law estimators

Abstract The prevailing maximum likelihood estimators for inferring power law models from rank-frequency data are biased. The source of this bias is an inappropriate likelihood function. The correct likelihood function is derived and shown to be computationally intractable. A more computationally ef...

Descripción completa

Guardado en:

Detalles Bibliográficos
Autores principales:	Charlie Pilgrim, Thomas T Hills
Formato:	article
Lenguaje:	EN
Publicado:	Nature Portfolio 2021
Materias:	Medicine R Science Q
Acceso en línea:	https://doaj.org/article/9b29ac7b9d07447eaecece75d6538d88
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

id	oai:doaj.org-article:9b29ac7b9d07447eaecece75d6538d88
record_format	dspace
spelling	oai:doaj.org-article:9b29ac7b9d07447eaecece75d6538d882021-12-02T19:02:27ZBias in Zipf’s law estimators10.1038/s41598-021-96214-w2045-2322https://doaj.org/article/9b29ac7b9d07447eaecece75d6538d882021-08-01T00:00:00Zhttps://doi.org/10.1038/s41598-021-96214-whttps://doaj.org/toc/2045-2322Abstract The prevailing maximum likelihood estimators for inferring power law models from rank-frequency data are biased. The source of this bias is an inappropriate likelihood function. The correct likelihood function is derived and shown to be computationally intractable. A more computationally efficient method of approximate Bayesian computation (ABC) is explored. This method is shown to have less bias for data generated from idealised rank-frequency Zipfian distributions. However, the existing estimators and the ABC estimator described here assume that words are drawn from a simple probability distribution, while language is a much more complex process. We show that this false assumption leads to continued biases when applying any of these methods to natural language to estimate Zipf exponents. We recommend that researchers be aware of the bias when investigating power laws in rank-frequency data.Charlie PilgrimThomas T HillsNature PortfolioarticleMedicineRScienceQENScientific Reports, Vol 11, Iss 1, Pp 1-11 (2021)
institution	DOAJ
collection	DOAJ
language	EN
topic	Medicine R Science Q
spellingShingle	Medicine R Science Q Charlie Pilgrim Thomas T Hills Bias in Zipf’s law estimators
description	Abstract The prevailing maximum likelihood estimators for inferring power law models from rank-frequency data are biased. The source of this bias is an inappropriate likelihood function. The correct likelihood function is derived and shown to be computationally intractable. A more computationally efficient method of approximate Bayesian computation (ABC) is explored. This method is shown to have less bias for data generated from idealised rank-frequency Zipfian distributions. However, the existing estimators and the ABC estimator described here assume that words are drawn from a simple probability distribution, while language is a much more complex process. We show that this false assumption leads to continued biases when applying any of these methods to natural language to estimate Zipf exponents. We recommend that researchers be aware of the bias when investigating power laws in rank-frequency data.
format	article
author	Charlie Pilgrim Thomas T Hills
author_facet	Charlie Pilgrim Thomas T Hills
author_sort	Charlie Pilgrim
title	Bias in Zipf’s law estimators
title_short	Bias in Zipf’s law estimators
title_full	Bias in Zipf’s law estimators
title_fullStr	Bias in Zipf’s law estimators
title_full_unstemmed	Bias in Zipf’s law estimators
title_sort	bias in zipf’s law estimators
publisher	Nature Portfolio
publishDate	2021
url	https://doaj.org/article/9b29ac7b9d07447eaecece75d6538d88
work_keys_str_mv	AT charliepilgrim biasinzipfslawestimators AT thomasthills biasinzipfslawestimators
_version_	1718377284239360000

Bias in Zipf’s law estimators

Ejemplares similares