A multimode data management method based on Data Fabric

doi:10.3969/j.issn.1000-5641.2023.05.014

Abstract

Abstract:

In the process of government and enterprise evolution, as information technology deepens from informatization into digitization, the data generated by various applications are becoming increasingly multimode, multisource, and massive, thereby posing new challenges to data management. To address these challenges, many new technologies and concepts have emerged in the field of data management. Data Fabric is a method that integrates distributed data storage, processing, and applications into a whole, providing a set of visual interfaces for management. First, we analyzed the technical architecture, characteristics, value, and complete process of managing and applying the multimode data of Data Fabric. Subsequently, we proposed anomaly monitoring methods based on time series indicators as well as log data for multimode and multisource data, whereby the processing speed improved by 33.3% and 42.2%, and F1 score improved by 12.2 pps (percentage points) and 14.8 pps, respectively, using Data Fabric technology. This further demonstrates the efficiency and application value of Data Fabric technology in the newly proposed methods.

Key words: Data Fabric, multimode data management, data virtualization

CLC Number:

TP399

Xinjun ZHENG, Guoliang TIAN, Feihu HUANG. A multimode data management method based on Data Fabric[J]. Journal of East China Normal University(Natural Science), 2023, 2023(5): 164-181.

Figures/Tables 7

Fig.1

Table 1

Table 2

Table 3

Table 4

Table 5

Table 6

References 20

1	LU J, HOLUBOVA I.. Multi-model databases: A new journey to handle the variety of data. ACM Computing Surveys, 2019, 52 (3): 1- 38.
2	ALVORD M M, LU F, DU B, et al.. Big data fabric architecture: How big data and data management frameworks converge to bring a new generation of competitive advantage for enterprises.. ACM SIGMOD, 2016, 1 (1): 1- 15.
3	K2VIEW. What is data fabric? The complete guide [EB/OL]. [2023-05-15]. https://www.k2view.com/what-is-data-fabric.
4	赵国锋, 葛丹凤. 数据虚拟化研究综述 [J]. 重庆邮电大学学报(自然科学版), 2016, 28(4): 494-502.
5	WHITE A, ROLLINGS M. 5 Key actions for IT leaders for effective decision making [EB/OL]. (2023-05-10) [2023-05-15]. https://www.gartner.com/en/publications/what-effective-decision-making-looks-like.
6	GROOMBRIDGE D. Gartner top strategic technology trends for 2022 [EB/OL]. (2021-10-27) [2023-05-15]. https://emtemp.gcom.cloud/ngw/globalassets/en/publications/documents/2022-gartner-top-strategic-technology-trends-ebook.pdf.
7	张元涛, 刘闯. Denodo + AWS: 解密数据编织的核心技术--数据虚拟化 [EB/OL]. [2023-5-15]. https://www.denodo.com.cn/document/denodoaws-decrypt-the-core-technology-of-data-weaving-data-virtualization.
8	GHOSH P. Data Fabric architecture 101 [EB/OL]. (2022-09-27) [2023-05-15]. https://www.dataversity.net/data-fabric-architecture-101.
9	GUPTA A. Data Fabric architecture is key to modernizing data management and integration [EB/OL]. (2021-03-11) [2023-05-15]. https://www.gartner.com/smarterwithgartner/data-fabric-architecture-is-key-to-modernizing-data-management-and-integration.
10	BRODER A Z. On the resemblance and containment of documents [C]// Proceedings of the Co-mpression and Complexity of Sequences 1997. USA: IEEE Computer Society, 1997: 21-29.
11	SUN L, VERSTEEG S, BOZTAS S, et al. Detecting anomalous user behavior using an extended isolation forest algorithm: An enterprise case study. [EB/OL]. (2016-09-21) [2023-05-15]. https://axiv.org/abs/1609. 06676.
12	李存冰, 尹萍, 林杰, 等. 一种面向公共安全领域的多模态数据管理方法及系统: 中国, 202310315483.3 [P]. 2023-03-29.
13	STONEBRAKER M, ILYAS I F.. Data integration: The current status and the way forward. IEEE Data Engineering Bulletin, 2018, 41 (2): 3- 9.
14	DEVLIN J, CHANG M, LEE K, et al. BERT: Pre-training of deep bidirectional transformrs for language understanding [C]// Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2019, 1: 4171–4186.
15	AN J, CHO S.. Variational autoencoder based anomaly detection using reconstruction probability. Special Lecture on IE, 2015, 2 (1): 1- 18.
16	鲁鑫. 2021-2022年中国数字治理研究年度报告 [R/OL]. (2022-09-05) [2023-05-15]. https://www.mtx.cn/#/report?id=1001609.
17	YUHANNA N. The forrester wave™: Enterprise data fabric, Q2 2022 [R/OL]. (2022-06-23) [2023-05-15]. https://reprints2.forrester.com/#/assets/2/73/RES176390/report.
18	唐军, 张林. 一种通过大数据建模实现智能运营及精准营销的方法: 中国, CN201811491274. X [P]. 2023-05-12.
19	易存道. 一种根因分析频繁子图置信度预测方法及系统: 中国, CN202111267296. X [P]. 2021-12-31.
20	GRBOVIC M, CHENG H.. Real-time personalization using embeddings for search rankingat airbnb. ACM SIGKDD, 2018, 1 (1): 311- 320.

设备名	时间	参数名	参数值
DatabaseServer1	2023-05-18 00:00	CPU utilization	45.23%
DatabaseServer1	2023-05-18 00:01	CPU utilization	46.10%
LoadBalancer1	2023-05-18 00:00	Response time	110 ms
LoadBalancer1	2023-05-18 00:01	Response time	112 ms
IoTDevice1	2023-05-18 00:00	Sensor status	"Normal"
IoTDevice1	2023-05-18 00:01	Sensor status	"Normal"
NetworkSwitch1	2023-05-18 00:00	Packet loss rate	0.10%
NetworkSwitch1	2023-05-18 00:01	Packet loss rate	0.15%
WebServer1	2023-05-18 00:00	Requests per minute	3400
WebServer1	2023-05-18 00:01	Requests per minute	3500
Firewall1	2023-05-18 00:00	Connection status	"Secure"
Firewall1	2023-05-18 00:01	Connection status	"Secure"

算法	准确性/%	F1-score/%
优化的孤立森林	91.5	90.4
传统孤立森林	83.7	78.2
One-Class SVM	82.3	70.1
滑动窗口	80.1	68.3

算法	数据处理速度 / (h·TB^–1)		训练时长/s
算法	Data Fabric 方法	非 Data Fabric 方法	Data Fabric 方法	非 Data Fabric 方法
优化的孤立森林	1.62	2.43	2713.24	4182.73
传统孤立森林	1.89	2.12	3562.57	5893.46
One-Class SVM	2.41	2.93	4286.37	6812.49
滑动窗口	1.23	1.78	5124.78	7813.56

设备名	时间	告警信息
DatabaseServer2	2023-05-18 00:00	[ERROR] Database connection lost.
DatabaseServer2	2023-05-18 00:01	[INFO] Database connection restored.
LoadBalancer2	2023-05-18 00:00	<log level="warning">High traffic detected.</log>
LoadBalancer2	2023-05-18 00:01	<log level="info">Traffic back to normal.</log>
NetworkSwitch2	2023-05-18 00:00	Error: Packet loss detected.
NetworkSwitch2	2023-05-18 00:01	Info: Packet transmission restored to normal.
WebServer2	2023-05-18 00:00	2023-05-18 00:00:00, ERROR, Failed to respond to request.
WebServer2	2023-05-18 00:01	2023-05-18 00:01:00, INFO, Request handled successfully.
Firewall2	2023-05-18 00:00	(Firewall2) [Threat_Detected] "Potential DDoS attack."
Firewall2	2023-05-18 00:01	(Firewall2) [Attack_Mitigated] "DDoS attack mitigated."
StorageServer2	2023-05-18 00:00	[StorageServer2] <warn> Disk space running low. </warn>
StorageServer2	2023-05-18 00:01	[StorageServer2] <info> Disk cleanup started. </info>

算法	准确性/%	F1-score/%
BERT + 自编码器	94.6	89.3
传统自编码器	86.5	74.5