Hadoop Data Reduction Framework: Applying Data Reduction at the DFS Layer

Big-data processing systems such as Hadoop, which usually utilize distributed file systems (DFSs), require data reduction schemes to maximize storage space efficiency. These schemes have different tradeoffs, and there are no all-purpose schemes applicable to all data. Users must select a suitable sc...

Descripción completa

Guardado en:

Detalles Bibliográficos
Autores principales:	Ryan Nathanael Soenjoto Widodo, Hirotake Abe, Kazuhiko Kato
Formato:	article
Lenguaje:	EN
Publicado:	IEEE 2021
Materias:	Data compression data deduplication distributed file system Hadoop HDFS Electrical engineering. Electronics. Nuclear engineering TK1-9971
Acceso en línea:	https://doaj.org/article/7c5afee34aef437296b7e64a59170164
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

id	oai:doaj.org-article:7c5afee34aef437296b7e64a59170164
record_format	dspace
spelling	oai:doaj.org-article:7c5afee34aef437296b7e64a591701642021-11-20T00:01:55ZHadoop Data Reduction Framework: Applying Data Reduction at the DFS Layer2169-353610.1109/ACCESS.2021.3127499https://doaj.org/article/7c5afee34aef437296b7e64a591701642021-01-01T00:00:00Zhttps://ieeexplore.ieee.org/document/9612160/https://doaj.org/toc/2169-3536Big-data processing systems such as Hadoop, which usually utilize distributed file systems (DFSs), require data reduction schemes to maximize storage space efficiency. These schemes have different tradeoffs, and there are no all-purpose schemes applicable to all data. Users must select a suitable scheme in accordance with their data. To accommodate this requirement, application software or file system (FS) have a fixed selection of these schemes. However, these provided schemes are insufficient for all data types, and when novel schemes emerge, extending the selection can be problematic. If the source code of the application or FS is available, the source code could potentially be extended with extensive labor, but could be virtually impossible without the code maintainers’ assistance. If the source code is unavailable, there is no way to tackle the problem. This paper proposes an unexplored solution through a modular DFS design that eases data reduction scheme usage through existing programming techniques. The advantages of this presented approach are threefold. First, adding new schemes is easy and they are transparent to the application code requiring no extensions to it. Second, the modular structure requires minimal modification to the existing DFSs and performance overhead. Third, users can compile schemes separately from the DFS without the FS or DFS source code. To demonstrate the design’s effectiveness, we implemented it by minimally extending the Hadoop DFS (HDFS) and named it the Hadoop Data Reduction Framework (HDRF). We designed HDRF to work with minimal overhead and tested it extensively. Experimental results indicate that it has negligible overhead over existing approaches. In a number of cases, it can offer up to 48.96% higher throughput while achieving the best result in storage reduction within our tested setups because of the incorporated data reduction schemes.Ryan Nathanael Soenjoto WidodoHirotake AbeKazuhiko KatoIEEEarticleData compressiondata deduplicationdistributed file systemHadoopHDFSElectrical engineering. Electronics. Nuclear engineeringTK1-9971ENIEEE Access, Vol 9, Pp 152704-152717 (2021)
institution	DOAJ
collection	DOAJ
language	EN
topic	Data compression data deduplication distributed file system Hadoop HDFS Electrical engineering. Electronics. Nuclear engineering TK1-9971
spellingShingle	Data compression data deduplication distributed file system Hadoop HDFS Electrical engineering. Electronics. Nuclear engineering TK1-9971 Ryan Nathanael Soenjoto Widodo Hirotake Abe Kazuhiko Kato Hadoop Data Reduction Framework: Applying Data Reduction at the DFS Layer
description	Big-data processing systems such as Hadoop, which usually utilize distributed file systems (DFSs), require data reduction schemes to maximize storage space efficiency. These schemes have different tradeoffs, and there are no all-purpose schemes applicable to all data. Users must select a suitable scheme in accordance with their data. To accommodate this requirement, application software or file system (FS) have a fixed selection of these schemes. However, these provided schemes are insufficient for all data types, and when novel schemes emerge, extending the selection can be problematic. If the source code of the application or FS is available, the source code could potentially be extended with extensive labor, but could be virtually impossible without the code maintainers’ assistance. If the source code is unavailable, there is no way to tackle the problem. This paper proposes an unexplored solution through a modular DFS design that eases data reduction scheme usage through existing programming techniques. The advantages of this presented approach are threefold. First, adding new schemes is easy and they are transparent to the application code requiring no extensions to it. Second, the modular structure requires minimal modification to the existing DFSs and performance overhead. Third, users can compile schemes separately from the DFS without the FS or DFS source code. To demonstrate the design’s effectiveness, we implemented it by minimally extending the Hadoop DFS (HDFS) and named it the Hadoop Data Reduction Framework (HDRF). We designed HDRF to work with minimal overhead and tested it extensively. Experimental results indicate that it has negligible overhead over existing approaches. In a number of cases, it can offer up to 48.96% higher throughput while achieving the best result in storage reduction within our tested setups because of the incorporated data reduction schemes.
format	article
author	Ryan Nathanael Soenjoto Widodo Hirotake Abe Kazuhiko Kato
author_facet	Ryan Nathanael Soenjoto Widodo Hirotake Abe Kazuhiko Kato
author_sort	Ryan Nathanael Soenjoto Widodo
title	Hadoop Data Reduction Framework: Applying Data Reduction at the DFS Layer
title_short	Hadoop Data Reduction Framework: Applying Data Reduction at the DFS Layer
title_full	Hadoop Data Reduction Framework: Applying Data Reduction at the DFS Layer
title_fullStr	Hadoop Data Reduction Framework: Applying Data Reduction at the DFS Layer
title_full_unstemmed	Hadoop Data Reduction Framework: Applying Data Reduction at the DFS Layer
title_sort	hadoop data reduction framework: applying data reduction at the dfs layer
publisher	IEEE
publishDate	2021
url	https://doaj.org/article/7c5afee34aef437296b7e64a59170164
work_keys_str_mv	AT ryannathanaelsoenjotowidodo hadoopdatareductionframeworkapplyingdatareductionatthedfslayer AT hirotakeabe hadoopdatareductionframeworkapplyingdatareductionatthedfslayer AT kazuhikokato hadoopdatareductionframeworkapplyingdatareductionatthedfslayer
_version_	1718419842508259328

Hadoop Data Reduction Framework: Applying Data Reduction at the DFS Layer

Ejemplares similares