Improving A/B Testing on the Basis of Possibilistic Reward Methods: A Numerical Analysis

A/B testing is used in digital contexts both to offer a more personalized service and to optimize the e-commerce purchasing process. A personalized service provides customers with the fastest possible access to the contents that they are most likely to use. An optimized e-commerce purchasing process...

Descripción completa

Guardado en:

Detalles Bibliográficos
Autores principales:	Miguel Martín, Antonio Jiménez-Martín, Alfonso Mateos, Josefa Z. Hernández
Formato:	article
Lenguaje:	EN
Publicado:	MDPI AG 2021
Materias:	A/B testing multi-armed bandit stopping criterion numerical analyses Mathematics QA1-939
Acceso en línea:	https://doaj.org/article/b08d1030614046c292f5add6de5aed6a
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

id	oai:doaj.org-article:b08d1030614046c292f5add6de5aed6a
record_format	dspace
spelling	oai:doaj.org-article:b08d1030614046c292f5add6de5aed6a2021-11-25T19:07:21ZImproving A/B Testing on the Basis of Possibilistic Reward Methods: A Numerical Analysis10.3390/sym131121752073-8994https://doaj.org/article/b08d1030614046c292f5add6de5aed6a2021-11-01T00:00:00Zhttps://www.mdpi.com/2073-8994/13/11/2175https://doaj.org/toc/2073-8994A/B testing is used in digital contexts both to offer a more personalized service and to optimize the e-commerce purchasing process. A personalized service provides customers with the fastest possible access to the contents that they are most likely to use. An optimized e-commerce purchasing process reduces customer effort during online purchasing and assures that the largest possible number of customers place their order. The most widespread A/B testing method is to implement the equivalent of RCT (randomized controlled trials). Recently, however, some companies and solutions have addressed this experimentation process as a multi-armed bandit (MAB). This is known in the A/B testing market as dynamic traffic distribution. A complementary technique used to optimize the performance of A/B testing is to improve the experiment stopping criterion. In this paper, we propose an adaptation of A/B testing to account for possibilistic reward (PR) methods, together with the definition of a new stopping criterion also based on PR methods to be used for both classical A/B testing and A/B testing based on MAB algorithms. A comparative numerical analysis based on the simulation of real scenarios is used to analyze the performance of the proposed adaptations in both Bernoulli and non-Bernoulli environments. In this analysis, we show that the possibilistic reward method PR3 produced the lowest mean cumulative regret in non-Bernoulli environments, which proved to have a high confidence level and be highly stable as demonstrated by low standard deviation measures. PR3 behaves exactly the same as Thompson sampling in Bernoulli environments. The conclusion is that PR3 can be used efficiently in both environments in combination with the value remaining stopping criterion in Bernoulli environments and the PR3 bounds stopping criterion for non-Bernoulli environments.Miguel MartínAntonio Jiménez-MartínAlfonso MateosJosefa Z. HernándezMDPI AGarticleA/B testingmulti-armed banditstopping criterionnumerical analysesMathematicsQA1-939ENSymmetry, Vol 13, Iss 2175, p 2175 (2021)
institution	DOAJ
collection	DOAJ
language	EN
topic	A/B testing multi-armed bandit stopping criterion numerical analyses Mathematics QA1-939
spellingShingle	A/B testing multi-armed bandit stopping criterion numerical analyses Mathematics QA1-939 Miguel Martín Antonio Jiménez-Martín Alfonso Mateos Josefa Z. Hernández Improving A/B Testing on the Basis of Possibilistic Reward Methods: A Numerical Analysis
description	A/B testing is used in digital contexts both to offer a more personalized service and to optimize the e-commerce purchasing process. A personalized service provides customers with the fastest possible access to the contents that they are most likely to use. An optimized e-commerce purchasing process reduces customer effort during online purchasing and assures that the largest possible number of customers place their order. The most widespread A/B testing method is to implement the equivalent of RCT (randomized controlled trials). Recently, however, some companies and solutions have addressed this experimentation process as a multi-armed bandit (MAB). This is known in the A/B testing market as dynamic traffic distribution. A complementary technique used to optimize the performance of A/B testing is to improve the experiment stopping criterion. In this paper, we propose an adaptation of A/B testing to account for possibilistic reward (PR) methods, together with the definition of a new stopping criterion also based on PR methods to be used for both classical A/B testing and A/B testing based on MAB algorithms. A comparative numerical analysis based on the simulation of real scenarios is used to analyze the performance of the proposed adaptations in both Bernoulli and non-Bernoulli environments. In this analysis, we show that the possibilistic reward method PR3 produced the lowest mean cumulative regret in non-Bernoulli environments, which proved to have a high confidence level and be highly stable as demonstrated by low standard deviation measures. PR3 behaves exactly the same as Thompson sampling in Bernoulli environments. The conclusion is that PR3 can be used efficiently in both environments in combination with the value remaining stopping criterion in Bernoulli environments and the PR3 bounds stopping criterion for non-Bernoulli environments.
format	article
author	Miguel Martín Antonio Jiménez-Martín Alfonso Mateos Josefa Z. Hernández
author_facet	Miguel Martín Antonio Jiménez-Martín Alfonso Mateos Josefa Z. Hernández
author_sort	Miguel Martín
title	Improving A/B Testing on the Basis of Possibilistic Reward Methods: A Numerical Analysis
title_short	Improving A/B Testing on the Basis of Possibilistic Reward Methods: A Numerical Analysis
title_full	Improving A/B Testing on the Basis of Possibilistic Reward Methods: A Numerical Analysis
title_fullStr	Improving A/B Testing on the Basis of Possibilistic Reward Methods: A Numerical Analysis
title_full_unstemmed	Improving A/B Testing on the Basis of Possibilistic Reward Methods: A Numerical Analysis
title_sort	improving a/b testing on the basis of possibilistic reward methods: a numerical analysis
publisher	MDPI AG
publishDate	2021
url	https://doaj.org/article/b08d1030614046c292f5add6de5aed6a
work_keys_str_mv	AT miguelmartin improvingabtestingonthebasisofpossibilisticrewardmethodsanumericalanalysis AT antoniojimenezmartin improvingabtestingonthebasisofpossibilisticrewardmethodsanumericalanalysis AT alfonsomateos improvingabtestingonthebasisofpossibilisticrewardmethodsanumericalanalysis AT josefazhernandez improvingabtestingonthebasisofpossibilisticrewardmethodsanumericalanalysis
_version_	1718410300351315968

Improving A/B Testing on the Basis of Possibilistic Reward Methods: A Numerical Analysis

Ejemplares similares