面向异常检测的时序数据库查询优化

doi:10.3969/j.issn.1000-5641.2023.02.013

摘要/Abstract

摘要：

随着物联网的发展, 大量传感器设备被接入网络, 这些设备所产生的数据的异常检测关系到系统服务的稳定性. 时序数据库是专门为时序数据优化的数据库系统. 作为监控系统的重要环节, 时序数据库担负着时序数据的管理和查询任务. 但目前时序数据库在处理多个数据源数据的查询上存在着延迟高、没有充分利用系统计算资源的缺点. 针对上述缺点, 基于InfluxDB, 重新设计了时序数据库的查询执行模型, 提出了InfluxDB-PP (parallel processing), 很好地解决了上述问题. 实验结果表明, InfluxDB-PP在实时数据异常查询场景下, 查询时延相较于InfluxDB降低了约85.7%.

关键词: 监控系统, 时序数据库, 多数据源数据查询优化

Abstract:

With the development of the Internet of Things, a large number of sensor devices can be connected to a network. Anomaly detection of data generated by these devices is related to the stability of system services. A time series database is a database system optimized for time series data. As an important component of a monitoring system, time series databases are responsible for storing and querying continuous streams of time series data. The current time series database, however, cannot fully utilize system computing resources and cannot meet the latency requirements when coping with queries from multiple data sources. To address these drawbacks, we redesigned the query execution model of a time series database based on the well-known InfluxDB, and we proposed InfluxDB-PP (parallel processing) as a method to address the aforementioned problems. The experimental results show that InfluxDB-PP reduces query latency by about 85.7% compared to InfluxDB for real-time anomaly data query scenarios.

Key words: monitoring system, time series database, multi-source data query optimization

中图分类号:

TP392

张帅, 胡卉芪, 徐尧强, 周烜. 面向异常检测的时序数据库查询优化[J]. 华东师范大学学报（自然科学版）, 2023, 2023(2): 119-131.

Shuai ZHANG, Huiqi HU, Yaoqiang XU, Xuan ZHOU. Time series database query optimization for anomaly detection[J]. Journal of East China Normal University(Natural Science), 2023, 2023(2): 119-131.

图/表 13

图1

图2

图3

图4

图5

图6

图7

图8

图9

图10

图11

图12

图13

参考文献 13

1	SANDEEP M, CHANDAVARKAR B R. Data processing in IoT, sensor to cloud: Survey [C]// 2021 12th International Conference on Computing Communication and Networking Technologies (ICCCNT). IEEE, 2021. DOI:10.1109/ICCCNT51525.2021.9579976.
2	EL ZOUKA H A, HOSNI M M. Secure IoT communications for smart healthcare monitoring system. Internet of Things, 2021, 13, 100036.
3	JENSEN S, PEDERSEN T, THOMSEN C. Time series management systems: A survey. IEEE Transactions on Knowledge and Data Engineering, 2017, 29 (11): 2581- 2600.
4	BADER A, KOPP O, FALKENTHAL M. Survey and comparison of open source time series databases [C]// Datenbanksysteme für Business, Technologie und Web (BTW 2017) - Workshopband. 2017: 249-268. Bon: Gesellschaft für Informatik.
5	INFLUXDATA INC. InfluxDB [EB/OL]. [2022-01-05]. https://github.com/influxdata/influxdb.
6	KEOGH E J, LONARDI S, CHIU B. Finding surprising patterns in a time series database in linear time and space [C]// Proceedings of the 8th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 2002: 550-556
7	HU B, CHEN Y P, KEOGH E. Time series classification under more realistic assumptions [C]// Proceedings of the 2013 SIAM International Conference on Data Mining (SDM). SIAM: 2013: 578-586
8	PETITJEAN F, FORESTIER G, WEBB G I, et al. Dynamic time warping averaging of time series allows faster and more accurate classification [C]// 2014 IEEE International Conference on Data Mining. IEEE, 2014: 470-479.
9	PERERA K S, HAHMANN M, LEHNER W, et al. Efficient approximate OLAP querying over time series [C]// Proceedings of the 20th International Database Engineering and Applications Symposium. ACM, 2016: 205-211.
10	PERERA K S, HAHMANN M, LEHNER W, et al. Modeling large time series for efficient approximate query processing [C]// Database Systems for Advanced Applications. Berlin: Springer, 2015: 190-204.
11	SHI X H, FENG Z Z, LI K X, et al. ByteSeries: An in-memory time series database for large-scale monitoring systems [C]// Proceedings of the 11th ACM Symposium on Cloud Computing. ACM, 2020: 60-73.
12	KADIYALA S, SHIRI N. A compact multi-resolution index for variable length queries in time series databases. Knowledge and Information Systems, 2008, 15 (2): 131- 147.
13	HENNESSY J L, PATTERSON D A. Computer Architecture: A Quantitative Approach [M]. 4th ed. [S. l.]: Morgan Kaufmann Publishers, 2006.

[1]	简炜, 胡梓锐, 张蓉. OLAP查询基数预估能力评估[J]. 华东师范大学学报（自然科学版）, 2024, 2024(5): 141-151.
[2]	凌向荣, 翁思扬, 张蓉. 基于锁避免的事务内阻塞分析与调度策略[J]. 华东师范大学学报（自然科学版）, 2024, 2024(5): 152-161.
[3]	连薛超, 刘维, 王清帅, 张蓉. 面向高冲突事务处理的架构设计和优化[J]. 华东师范大学学报（自然科学版）, 2023, 2023(6): 28-38.
[4]	王聪聪, 胡卉芪. 基于持久化内存和共享缓存架构的高性能数据库[J]. 华东师范大学学报（自然科学版）, 2023, 2023(5): 1-10.
[5]	俞融, 杨攀飞, 王清帅, 张蓉. 数据同步机制自适应优化的HTAP数据库原型系统[J]. 华东师范大学学报（自然科学版）, 2023, 2023(5): 11-25.
[6]	梅文娟, 蔡鹏. 面向存算分离架构的混合粒度缓存策略[J]. 华东师范大学学报（自然科学版）, 2023, 2023(5): 26-39.
[7]	刘丹琪, 蔡鹏. 存算分离架构下Part元数据的单独管理策略[J]. 华东师范大学学报（自然科学版）, 2023, 2023(5): 40-50.
[8]	卢皙钰, 刘维, 翁思扬, 李可强, 张蓉. 基于模糊测试生成多样化的数据库隔离级别测试案例[J]. 华东师范大学学报（自然科学版）, 2023, 2023(5): 51-64.
[9]	高歌, 胡卉芪. FeaDB: 基于内存的多版本在线特征存储[J]. 华东师范大学学报（自然科学版）, 2023, 2023(5): 65-76.
[10]	陈现森, 徐辰. 基于openGauss的异构算子加速技术[J]. 华东师范大学学报（自然科学版）, 2023, 2023(5): 90-99.
[11]	官嘉林, 朱艳, 吴庭亮, 陈艳, 张敬伟. 基于大页内存的学习索引内存分配策略[J]. 华东师范大学学报（自然科学版）, 2023, 2023(2): 73-81.
[12]	刘梦男, 许建秋. 基于查询代价的两级轨迹数据划分算法[J]. 华东师范大学学报（自然科学版）, 2022, 2022(5): 184-194.
[13]	唐永金, 孙家博, 蔡鹏. 基于查询频率的混合事务分析处理数据同步优化[J]. 华东师范大学学报（自然科学版）, 2022, 2022(5): 26-35.
[14]	陈婷, 项兆坤, 徐金凯, 张蓉. 查询优化器连接顺序评估[J]. 华东师范大学学报（自然科学版）, 2022, 2022(5): 48-60.
[15]	游舒泓, 苏仟, 张蓉. 云数据库运行环境的动态模拟[J]. 华东师范大学学报（自然科学版）, 2022, 2022(5): 73-89.