Time series database query optimization for anomaly detection

doi:10.3969/j.issn.1000-5641.2023.02.013

Abstract

Abstract:

With the development of the Internet of Things, a large number of sensor devices can be connected to a network. Anomaly detection of data generated by these devices is related to the stability of system services. A time series database is a database system optimized for time series data. As an important component of a monitoring system, time series databases are responsible for storing and querying continuous streams of time series data. The current time series database, however, cannot fully utilize system computing resources and cannot meet the latency requirements when coping with queries from multiple data sources. To address these drawbacks, we redesigned the query execution model of a time series database based on the well-known InfluxDB, and we proposed InfluxDB-PP (parallel processing) as a method to address the aforementioned problems. The experimental results show that InfluxDB-PP reduces query latency by about 85.7% compared to InfluxDB for real-time anomaly data query scenarios.

Key words: monitoring system, time series database, multi-source data query optimization

CLC Number:

TP392

Shuai ZHANG, Huiqi HU, Yaoqiang XU, Xuan ZHOU. Time series database query optimization for anomaly detection[J]. Journal of East China Normal University(Natural Science), 2023, 2023(2): 119-131.

Figures/Tables 13

Fig.1

Fig.2

Fig.3

Fig.4

Fig.5

Fig.6

Fig.7

Fig.8

Fig.9

Fig.10

Fig.11

Fig.12

Fig.13

References 13

1	SANDEEP M, CHANDAVARKAR B R. Data processing in IoT, sensor to cloud: Survey [C]// 2021 12th International Conference on Computing Communication and Networking Technologies (ICCCNT). IEEE, 2021. DOI:10.1109/ICCCNT51525.2021.9579976.
2	EL ZOUKA H A, HOSNI M M. Secure IoT communications for smart healthcare monitoring system. Internet of Things, 2021, 13, 100036.
3	JENSEN S, PEDERSEN T, THOMSEN C. Time series management systems: A survey. IEEE Transactions on Knowledge and Data Engineering, 2017, 29 (11): 2581- 2600.
4	BADER A, KOPP O, FALKENTHAL M. Survey and comparison of open source time series databases [C]// Datenbanksysteme für Business, Technologie und Web (BTW 2017) - Workshopband. 2017: 249-268. Bon: Gesellschaft für Informatik.
5	INFLUXDATA INC. InfluxDB [EB/OL]. [2022-01-05]. https://github.com/influxdata/influxdb.
6	KEOGH E J, LONARDI S, CHIU B. Finding surprising patterns in a time series database in linear time and space [C]// Proceedings of the 8th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 2002: 550-556
7	HU B, CHEN Y P, KEOGH E. Time series classification under more realistic assumptions [C]// Proceedings of the 2013 SIAM International Conference on Data Mining (SDM). SIAM: 2013: 578-586
8	PETITJEAN F, FORESTIER G, WEBB G I, et al. Dynamic time warping averaging of time series allows faster and more accurate classification [C]// 2014 IEEE International Conference on Data Mining. IEEE, 2014: 470-479.
9	PERERA K S, HAHMANN M, LEHNER W, et al. Efficient approximate OLAP querying over time series [C]// Proceedings of the 20th International Database Engineering and Applications Symposium. ACM, 2016: 205-211.
10	PERERA K S, HAHMANN M, LEHNER W, et al. Modeling large time series for efficient approximate query processing [C]// Database Systems for Advanced Applications. Berlin: Springer, 2015: 190-204.
11	SHI X H, FENG Z Z, LI K X, et al. ByteSeries: An in-memory time series database for large-scale monitoring systems [C]// Proceedings of the 11th ACM Symposium on Cloud Computing. ACM, 2020: 60-73.
12	KADIYALA S, SHIRI N. A compact multi-resolution index for variable length queries in time series databases. Knowledge and Information Systems, 2008, 15 (2): 131- 147.
13	HENNESSY J L, PATTERSON D A. Computer Architecture: A Quantitative Approach [M]. 4th ed. [S. l.]: Morgan Kaufmann Publishers, 2006.