Anomaly Detection for the Centralised Elasticsearch Service at CERN

For several years CERN has been offering a centralised service for Elasticsearch, a popular distributed system for search and analytics of user provided data. The service offered by CERN IT is better described as a service of services, delivering centrally managed and maintained Elasticsearch instan...

Descripción completa

Guardado en:

Detalles Bibliográficos
Autores principales:	Jennifer R. Andersson, Jose Alonso Moya, Ulrich Schwickerath
Formato:	article
Lenguaje:	EN
Publicado:	Frontiers Media S.A. 2021
Materias:	anomaly detection elasticsearch LSTM DNN machine learning Information technology T58.5-58.64
Acceso en línea:	https://doaj.org/article/7e6c310348de417ababaed693f0af6e7
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

id	oai:doaj.org-article:7e6c310348de417ababaed693f0af6e7
record_format	dspace
spelling	oai:doaj.org-article:7e6c310348de417ababaed693f0af6e72021-11-16T14:03:13ZAnomaly Detection for the Centralised Elasticsearch Service at CERN2624-909X10.3389/fdata.2021.718879https://doaj.org/article/7e6c310348de417ababaed693f0af6e72021-11-01T00:00:00Zhttps://www.frontiersin.org/articles/10.3389/fdata.2021.718879/fullhttps://doaj.org/toc/2624-909XFor several years CERN has been offering a centralised service for Elasticsearch, a popular distributed system for search and analytics of user provided data. The service offered by CERN IT is better described as a service of services, delivering centrally managed and maintained Elasticsearch instances to CERN users who have a justified need for it. This dynamic infrastructure currently consists of about 30 distinct and independent Elasticsearch installations, in the following referred to as Elasticsearch clusters, some of which are shared between different user communities. The service is used by several hundred users mainly for logs and service analytics. Due to its size and complexity, the installation produces a huge amount of internal monitoring data which can be difficult to process in real time with limited available person power. Early on, an idea was therefore born to process this data automatically, aiming to extract anomalies and possible issues building up in real time, allowing the experts to address them before they start to cause an issue for the users of the service. Both deep learning and traditional methods have been applied to analyse the data in order to achieve this goal. This resulted in the current deployment of an anomaly detection system based on a one layer multi dimensional LSTM neural network, coupled with applying a simple moving average to the data to validate the results. This paper will describe which methods were investigated and give an overview of the current system, including data retrieval, data pre-processing and analysis. In addition, reports on experiences gained when applying the system to actual data will be provided. Finally, weaknesses of the current system will be briefly discussed, and ideas for future system improvements will be sketched out.Jennifer R. AnderssonJose Alonso MoyaUlrich SchwickerathFrontiers Media S.A.articleanomaly detectionelasticsearchLSTMDNNmachine learningInformation technologyT58.5-58.64ENFrontiers in Big Data, Vol 4 (2021)
institution	DOAJ
collection	DOAJ
language	EN
topic	anomaly detection elasticsearch LSTM DNN machine learning Information technology T58.5-58.64
spellingShingle	anomaly detection elasticsearch LSTM DNN machine learning Information technology T58.5-58.64 Jennifer R. Andersson Jose Alonso Moya Ulrich Schwickerath Anomaly Detection for the Centralised Elasticsearch Service at CERN
description	For several years CERN has been offering a centralised service for Elasticsearch, a popular distributed system for search and analytics of user provided data. The service offered by CERN IT is better described as a service of services, delivering centrally managed and maintained Elasticsearch instances to CERN users who have a justified need for it. This dynamic infrastructure currently consists of about 30 distinct and independent Elasticsearch installations, in the following referred to as Elasticsearch clusters, some of which are shared between different user communities. The service is used by several hundred users mainly for logs and service analytics. Due to its size and complexity, the installation produces a huge amount of internal monitoring data which can be difficult to process in real time with limited available person power. Early on, an idea was therefore born to process this data automatically, aiming to extract anomalies and possible issues building up in real time, allowing the experts to address them before they start to cause an issue for the users of the service. Both deep learning and traditional methods have been applied to analyse the data in order to achieve this goal. This resulted in the current deployment of an anomaly detection system based on a one layer multi dimensional LSTM neural network, coupled with applying a simple moving average to the data to validate the results. This paper will describe which methods were investigated and give an overview of the current system, including data retrieval, data pre-processing and analysis. In addition, reports on experiences gained when applying the system to actual data will be provided. Finally, weaknesses of the current system will be briefly discussed, and ideas for future system improvements will be sketched out.
format	article
author	Jennifer R. Andersson Jose Alonso Moya Ulrich Schwickerath
author_facet	Jennifer R. Andersson Jose Alonso Moya Ulrich Schwickerath
author_sort	Jennifer R. Andersson
title	Anomaly Detection for the Centralised Elasticsearch Service at CERN
title_short	Anomaly Detection for the Centralised Elasticsearch Service at CERN
title_full	Anomaly Detection for the Centralised Elasticsearch Service at CERN
title_fullStr	Anomaly Detection for the Centralised Elasticsearch Service at CERN
title_full_unstemmed	Anomaly Detection for the Centralised Elasticsearch Service at CERN
title_sort	anomaly detection for the centralised elasticsearch service at cern
publisher	Frontiers Media S.A.
publishDate	2021
url	https://doaj.org/article/7e6c310348de417ababaed693f0af6e7
work_keys_str_mv	AT jenniferrandersson anomalydetectionforthecentralisedelasticsearchserviceatcern AT josealonsomoya anomalydetectionforthecentralisedelasticsearchserviceatcern AT ulrichschwickerath anomalydetectionforthecentralisedelasticsearchserviceatcern
_version_	1718426433445953536

Anomaly Detection for the Centralised Elasticsearch Service at CERN

Ejemplares similares