Adaptive On-the-Fly Changes in Distributed Processing Pipelines

Distributed data processing systems have become the standard means for big data analytics. These systems are based on processing pipelines where operations on data are performed in a chain of consecutive steps. Normally, the operations performed by these pipelines are set at design time, and any cha...

Descripción completa

Guardado en:

Detalles Bibliográficos
Autores principales:	Toon Albers, Elena Lazovik, Mostafa Hadadian Nejad Yousefi, Alexander Lazovik
Formato:	article
Lenguaje:	EN
Publicado:	Frontiers Media S.A. 2021
Materias:	distributed computing big data applications on-the-fly updates adaptive dynamic systems industrial data management dynamic software updating Information technology T58.5-58.64
Acceso en línea:	https://doaj.org/article/9e3bee21d67445e989faba5585dacf94
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

id	oai:doaj.org-article:9e3bee21d67445e989faba5585dacf94
record_format	dspace
spelling	oai:doaj.org-article:9e3bee21d67445e989faba5585dacf942021-12-01T09:28:11ZAdaptive On-the-Fly Changes in Distributed Processing Pipelines2624-909X10.3389/fdata.2021.666174https://doaj.org/article/9e3bee21d67445e989faba5585dacf942021-11-01T00:00:00Zhttps://www.frontiersin.org/articles/10.3389/fdata.2021.666174/fullhttps://doaj.org/toc/2624-909XDistributed data processing systems have become the standard means for big data analytics. These systems are based on processing pipelines where operations on data are performed in a chain of consecutive steps. Normally, the operations performed by these pipelines are set at design time, and any changes to their functionality require the applications to be restarted. This is not always acceptable, for example, when we cannot afford downtime or when a long-running calculation would lose significant progress. The introduction of variation points to distributed processing pipelines allows for on-the-fly updating of individual analysis steps. In this paper, we extend such basic variation point functionality to provide fully automated reconfiguration of the processing steps within a running pipeline through an automated planner. We have enabled pipeline modeling through constraints. Based on these constraints, we not only ensure that configurations are compatible with type but also verify that expected pipeline functionality is achieved. Furthermore, automating the reconfiguration process simplifies its use, in turn allowing users with less development experience to make changes. The system can automatically generate and validate pipeline configurations that achieve a specified goal, selecting from operation definitions available at planning time. It then automatically integrates these configurations into the running pipeline. We verify the system through the testing of a proof-of-concept implementation. The proof of concept also shows promising results when reconfiguration is performed frequently.Toon AlbersElena LazovikMostafa Hadadian Nejad YousefiAlexander LazovikFrontiers Media S.A.articledistributed computingbig data applicationson-the-fly updatesadaptive dynamic systemsindustrial data managementdynamic software updatingInformation technologyT58.5-58.64ENFrontiers in Big Data, Vol 4 (2021)
institution	DOAJ
collection	DOAJ
language	EN
topic	distributed computing big data applications on-the-fly updates adaptive dynamic systems industrial data management dynamic software updating Information technology T58.5-58.64
spellingShingle	distributed computing big data applications on-the-fly updates adaptive dynamic systems industrial data management dynamic software updating Information technology T58.5-58.64 Toon Albers Elena Lazovik Mostafa Hadadian Nejad Yousefi Alexander Lazovik Adaptive On-the-Fly Changes in Distributed Processing Pipelines
description	Distributed data processing systems have become the standard means for big data analytics. These systems are based on processing pipelines where operations on data are performed in a chain of consecutive steps. Normally, the operations performed by these pipelines are set at design time, and any changes to their functionality require the applications to be restarted. This is not always acceptable, for example, when we cannot afford downtime or when a long-running calculation would lose significant progress. The introduction of variation points to distributed processing pipelines allows for on-the-fly updating of individual analysis steps. In this paper, we extend such basic variation point functionality to provide fully automated reconfiguration of the processing steps within a running pipeline through an automated planner. We have enabled pipeline modeling through constraints. Based on these constraints, we not only ensure that configurations are compatible with type but also verify that expected pipeline functionality is achieved. Furthermore, automating the reconfiguration process simplifies its use, in turn allowing users with less development experience to make changes. The system can automatically generate and validate pipeline configurations that achieve a specified goal, selecting from operation definitions available at planning time. It then automatically integrates these configurations into the running pipeline. We verify the system through the testing of a proof-of-concept implementation. The proof of concept also shows promising results when reconfiguration is performed frequently.
format	article
author	Toon Albers Elena Lazovik Mostafa Hadadian Nejad Yousefi Alexander Lazovik
author_facet	Toon Albers Elena Lazovik Mostafa Hadadian Nejad Yousefi Alexander Lazovik
author_sort	Toon Albers
title	Adaptive On-the-Fly Changes in Distributed Processing Pipelines
title_short	Adaptive On-the-Fly Changes in Distributed Processing Pipelines
title_full	Adaptive On-the-Fly Changes in Distributed Processing Pipelines
title_fullStr	Adaptive On-the-Fly Changes in Distributed Processing Pipelines
title_full_unstemmed	Adaptive On-the-Fly Changes in Distributed Processing Pipelines
title_sort	adaptive on-the-fly changes in distributed processing pipelines
publisher	Frontiers Media S.A.
publishDate	2021
url	https://doaj.org/article/9e3bee21d67445e989faba5585dacf94
work_keys_str_mv	AT toonalbers adaptiveontheflychangesindistributedprocessingpipelines AT elenalazovik adaptiveontheflychangesindistributedprocessingpipelines AT mostafahadadiannejadyousefi adaptiveontheflychangesindistributedprocessingpipelines AT alexanderlazovik adaptiveontheflychangesindistributedprocessingpipelines
_version_	1718405371217838080

Adaptive On-the-Fly Changes in Distributed Processing Pipelines

Ejemplares similares