面向异常检测的时序数据库查询优化

doi:10.3969/j.issn.1000-5641.2023.02.013

摘要/Abstract

摘要：

随着物联网的发展, 大量传感器设备被接入网络, 这些设备所产生的数据的异常检测关系到系统服务的稳定性. 时序数据库是专门为时序数据优化的数据库系统. 作为监控系统的重要环节, 时序数据库担负着时序数据的管理和查询任务. 但目前时序数据库在处理多个数据源数据的查询上存在着延迟高、没有充分利用系统计算资源的缺点. 针对上述缺点, 基于InfluxDB, 重新设计了时序数据库的查询执行模型, 提出了InfluxDB-PP (parallel processing), 很好地解决了上述问题. 实验结果表明, InfluxDB-PP在实时数据异常查询场景下, 查询时延相较于InfluxDB降低了约85.7%.

关键词: 监控系统, 时序数据库, 多数据源数据查询优化

Abstract:

With the development of the Internet of Things, a large number of sensor devices can be connected to a network. Anomaly detection of data generated by these devices is related to the stability of system services. A time series database is a database system optimized for time series data. As an important component of a monitoring system, time series databases are responsible for storing and querying continuous streams of time series data. The current time series database, however, cannot fully utilize system computing resources and cannot meet the latency requirements when coping with queries from multiple data sources. To address these drawbacks, we redesigned the query execution model of a time series database based on the well-known InfluxDB, and we proposed InfluxDB-PP (parallel processing) as a method to address the aforementioned problems. The experimental results show that InfluxDB-PP reduces query latency by about 85.7% compared to InfluxDB for real-time anomaly data query scenarios.

Key words: monitoring system, time series database, multi-source data query optimization

中图分类号:

TP392

张帅, 胡卉芪, 徐尧强, 周烜. 面向异常检测的时序数据库查询优化[J]. 华东师范大学学报（自然科学版）, 2023, 2023(2): 119-131.

Shuai ZHANG, Huiqi HU, Yaoqiang XU, Xuan ZHOU. Time series database query optimization for anomaly detection[J]. Journal of East China Normal University(Natural Science), 2023, 2023(2): 119-131.

图/表 13

图1

图2

图3

图4

图5

图6

图7

图8

图9

图10

图11

图12

图13

参考文献 13

1	SANDEEP M, CHANDAVARKAR B R. Data processing in IoT, sensor to cloud: Survey [C]// 2021 12th International Conference on Computing Communication and Networking Technologies (ICCCNT). IEEE, 2021. DOI:10.1109/ICCCNT51525.2021.9579976.
2	EL ZOUKA H A, HOSNI M M. Secure IoT communications for smart healthcare monitoring system. Internet of Things, 2021, 13, 100036.
3	JENSEN S, PEDERSEN T, THOMSEN C. Time series management systems: A survey. IEEE Transactions on Knowledge and Data Engineering, 2017, 29 (11): 2581- 2600.
4	BADER A, KOPP O, FALKENTHAL M. Survey and comparison of open source time series databases [C]// Datenbanksysteme für Business, Technologie und Web (BTW 2017) - Workshopband. 2017: 249-268. Bon: Gesellschaft für Informatik.
5	INFLUXDATA INC. InfluxDB [EB/OL]. [2022-01-05]. https://github.com/influxdata/influxdb.
6	KEOGH E J, LONARDI S, CHIU B. Finding surprising patterns in a time series database in linear time and space [C]// Proceedings of the 8th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 2002: 550-556
7	HU B, CHEN Y P, KEOGH E. Time series classification under more realistic assumptions [C]// Proceedings of the 2013 SIAM International Conference on Data Mining (SDM). SIAM: 2013: 578-586
8	PETITJEAN F, FORESTIER G, WEBB G I, et al. Dynamic time warping averaging of time series allows faster and more accurate classification [C]// 2014 IEEE International Conference on Data Mining. IEEE, 2014: 470-479.
9	PERERA K S, HAHMANN M, LEHNER W, et al. Efficient approximate OLAP querying over time series [C]// Proceedings of the 20th International Database Engineering and Applications Symposium. ACM, 2016: 205-211.
10	PERERA K S, HAHMANN M, LEHNER W, et al. Modeling large time series for efficient approximate query processing [C]// Database Systems for Advanced Applications. Berlin: Springer, 2015: 190-204.
11	SHI X H, FENG Z Z, LI K X, et al. ByteSeries: An in-memory time series database for large-scale monitoring systems [C]// Proceedings of the 11th ACM Symposium on Cloud Computing. ACM, 2020: 60-73.
12	KADIYALA S, SHIRI N. A compact multi-resolution index for variable length queries in time series databases. Knowledge and Information Systems, 2008, 15 (2): 131- 147.
13	HENNESSY J L, PATTERSON D A. Computer Architecture: A Quantitative Approach [M]. 4th ed. [S. l.]: Morgan Kaufmann Publishers, 2006.

[1]	刘梦男, 许建秋. 基于查询代价的两级轨迹数据划分算法[J]. 华东师范大学学报（自然科学版）, 2022, 2022(5): 184-194.
[2]	唐永金, 孙家博, 蔡鹏. 基于查询频率的混合事务分析处理数据同步优化[J]. 华东师范大学学报（自然科学版）, 2022, 2022(5): 26-35.
[3]	陈婷, 项兆坤, 徐金凯, 张蓉. 查询优化器连接顺序评估[J]. 华东师范大学学报（自然科学版）, 2022, 2022(5): 48-60.
[4]	游舒泓, 苏仟, 张蓉. 云数据库运行环境的动态模拟[J]. 华东师范大学学报（自然科学版）, 2022, 2022(5): 73-89.
[5]	胡耀艺, 胡卉芪, 周烜, 周傲英. 面向非易失性内存缓冲区的SQLite-CC[J]. 华东师范大学学报（自然科学版）, 2021, 2021(6): 124-134.
[6]	孙家博, 蔡鹏. 日志结构合并树的查询优化技术[J]. 华东师范大学学报（自然科学版）, 2021, 2021(5): 94-103.
[7]	刘文欣, 蔡鹏. 多主数据库中基于分区的并发控制[J]. 华东师范大学学报（自然科学版）, 2021, 2021(5): 84-93.
[8]	项兆坤, 陈婷, 苏仟, 张蓉. 面向OLAP数据库查询处理功能的模糊测试工具[J]. 华东师范大学学报（自然科学版）, 2021, 2021(5): 74-83.
[9]	余阳, 胡卉芪, 周煊. 基于非易失性内存的LSM-tree存储系统优化[J]. 华东师范大学学报（自然科学版）, 2021, 2021(5): 37-47.
[10]	方创新, 宋浩, 林煜明, 周娅. CPU-GPU异构环境下的大规模商品知识查询处理[J]. 华东师范大学学报（自然科学版）, 2021, 2021(5): 157-168.
[11]	丁国浩, 徐辰, 钱卫宁. 面向日志结构化数据存储的高效数据加载[J]. 华东师范大学学报(自然科学版), 2019, 2019(5): 143-158.
[12]	祝朝凡, 郭进伟, 蔡鹏. 基于Paxos的分布式一致性算法的实现与优化[J]. 华东师范大学学报(自然科学版), 2019, 2019(5): 168-177.
[13]	涂云山, 储佳佳, 张耀, 翁楚良. 面向新硬件的数据处理软件技术[J]. 华东师范大学学报(自然科学版), 2018, 2018(5): 30-40,78.
[14]	徐石磊, 魏星, 江红, 钱卫宁, 周傲英. 分布式数据库系统中的并行分组聚合实现[J]. 华东师范大学学报(自然科学版), 2018, 2018(5): 56-66.
[15]	俞文谦, 胡爽, 胡卉芪. 面向Cedar的列存储设计与实现[J]. 华东师范大学学报(自然科学版), 2018, 2018(5): 67-78.