Optimization of HTAP data synchronization based on query frequency

doi:10.3969/j.issn.1000-5641.2022.05.003

Abstract

Abstract:

A hybrid transaction analytical processing (HTAP) system must concurrently support both transaction processing and query analysis. To eliminate interference between them, HTAP systems also typically assign different copies of data to both workloads, handling online transaction processing (OLTP) and online analytical processing (OLAP) requests separately, and synchronizing data between the copies based on a log replay. An HTAP system is committed to efficiently synchronizing OLTP data to OLAP, thereby providing a fresher data access service. In addition, the speed of sending and replaying the logs of the tables to be queried is a key factor affecting the freshness of the data. In this paper, using the table grouping based log parallel replay method and the characteristics of the HTAP load, a log sending and replay method is proposed based on the query frequency of the OLAP side. To ensure data consistency, this method improves the processing priority of high-frequency query table logs and achieves efficient log sending and replay capabilities along with a targeted priority display of high-frequency query table data, thereby ensuring the freshness of the HTAP system.

Key words: hybrid transaction analytical processing, data freshness, log replay, conflict detection

CLC Number:

TP392

Yongjin TANG, Jiabo SUN, Peng CAI. Optimization of HTAP data synchronization based on query frequency[J]. Journal of East China Normal University(Natural Science), 2022, 2022(5): 26-35.

Figures/Tables 6

References 14

1	RAZA A, CHRYSOGELOS P, ANADIOTIS A C, et al. Adaptive HTAP through elastic resource scheduling [C]// Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data. 2020: 2043-2054.
2	PSAROUDAKIS I, WOLF F, MAY N, et al. Scaling up mixed workloads: A battle of data freshness, flexibility, and scheduling [C]// Proceedings of the Technology Conference on Performance Evaluation and Benchmarking. 2014: 97-112.
3	KANTER J M, VEERAMACHANENI K. Deep feature synthesis: Towards automating data science endeavors [C]// Proceedings of the 2015 IEEE International Conference on Data Science and Advanced Analytics. 2015: 15652837.
4	LUO Y, WANG M, ZHOU H, et al. Autocross: Automatic feature crossing for tabular data in real-world applications [C]// Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2019: 1936-1945.
5	LAMH T, THIEBAUT J M, SINN M, et al. One button machine for automating feature engineering in relational databases [EB/OL]. (2017-06-01)[2022-06-12]. https://arxiv.org/pdf/1706.00327.pdf.
6	孙家博. HTAP系统中的并行日志回放优化 [D]. 上海: 华东师范大学, 2022.
7	MAKRESHANSKI D, GICEVA J, BARTHELS C, et al. BatchDB: Efficient isolated execution of hybrid OLTP + OLAP workloads for interactive applications [C]// Proceedings of the 2017 ACM International Conference on Management of Data. 2017: 37-50.
8	LAHIRI T, CHAVAN S, COLGAN M, et al. Oracle database in-memory: A dual format in-memory database [C]// Proceedings of the 2015 IEEE 31st International Conference on Data Engineering. 2015: 1253-1258.
9	LARSON P Å, BIRKA A, HANSON E N, et al. Real-time analytical processing with SQL Server. Proceedings of the VLDB Endowment, 2015, 8 (12): 1740- 1751.
10	YANG J, RAE I, XU J, et al. F1 Lightning: HTAP as a service. Proceedings of the VLDB Endowment, 2020, 13 (12): 3313- 3325.
11	HUANG D X, LIU Q, CUI Q, et al. TiDB: A Raft-based HTAP database. Proceedings of the VLDB Endowment, 2020, 13 (12): 3072- 3084.
12	HONG C T, ZHOU D, YANG M, et al. KuaFu: Closing the parallelism gap in database replication [C]// Proceedings of the IEEE 29th International Conference on Data Engineering. 2013: 1186-1195.
13	XIA Y, YU X Y, PAVLO A, et al. Taurus: Lightweight parallel logging for in-memory database management systems (extended version) [EB/OL]. (2020-10-14)[2022-06-08]. https://arxiv.org/pdf/2010.06760.pdf.
14	QIN D, BROWNA D, GOEL A. Scalable replay-based replication for fast databases. Proceedings of the VLDB Endowment, 2017, 10 (13): 2025- 2036.

[1]	Mengnan LIU, Jianqiu XU. Two-level trajectory data partition algorithm based on query cost [J]. Journal of East China Normal University(Natural Science), 2022, 2022(5): 184-194.
[2]	Yaoyi HU, Huiqi HU, Xuan ZHOU, Aoying ZHOU. SQLite-CC based on non-volatile memory cache [J]. Journal of East China Normal University(Natural Science), 2021, 2021(6): 124-134.
[3]	Jiabo SUN, Peng CAI. Query optimization technology based on an LSM-tree [J]. Journal of East China Normal University(Natural Science), 2021, 2021(5): 94-103.
[4]	Wenxin LIU, Peng CAI. Partition-based concurrency control in a multi-master database [J]. Journal of East China Normal University(Natural Science), 2021, 2021(5): 84-93.
[5]	Zhaokun XIANG, Ting CHEN, Qian SU, Rong ZHANG. A fuzzer for query processing functionality of OLAP databases [J]. Journal of East China Normal University(Natural Science), 2021, 2021(5): 74-83.
[6]	Yang YU, Huiqi HU, Xuan ZHOU. Optimization of LSM-tree storage systems based on non-volatile memory [J]. Journal of East China Normal University(Natural Science), 2021, 2021(5): 37-47.
[7]	Chuangxin FANG, Hao SONG, Yuming LIN, Ya ZHOU. Query processing of large-scale product knowledge in a CPU-GPU heterogeneous environment [J]. Journal of East China Normal University(Natural Science), 2021, 2021(5): 157-168.
[8]	DING Guo-hao, XU Chen, QIAN Wei-ning. Efficient data loading for log-structured data stores [J]. Journal of East China Normal University(Natural Sc, 2019, 2019(5): 143-158.
[9]	ZHU Chao-fan, GUO Jin-wei, CAI Peng. Implementation and optimization of a distributed consistency algorithm based on Paxos [J]. Journal of East China Normal University(Natural Sc, 2019, 2019(5): 168-177.
[10]	TU Yun-shan, CHU Jia-jia, ZHANG Yao, WENG Chu-liang. Data processing software technology for new hardware [J]. Journal of East China Normal University(Natural Sc, 2018, 2018(5): 30-40,78.
[11]	XU Shi-lei, WEI Xing, JIANG Hong, QIAN Wei-ning, ZHOU Ao-ying. Implementation of the parallel GroupBy and Aggregation functions in a distributed database system [J]. Journal of East China Normal University(Natural Sc, 2018, 2018(5): 56-66.
[12]	YU Wen-qian, HU Shuang, HU Hui-qi. The designs and implementations of columnar storage in Cedar [J]. Journal of East China Normal University(Natural Sc, 2018, 2018(5): 67-78.
[13]	HUANG Jian-wei, ZHANG Zhao, QIAN Wei-ning. Primary key management in distributed log-structured database systems [J]. Journal of East China Normal University(Natural Sc, 2018, 2018(5): 79-90,119.
[14]	ZHAO Chun-yang, XIAO Bing, GUO Jin-wei, QIAN Wei-ning. Application of the consistency protocol in distributed database systems [J]. Journal of East China Normal University(Natural Sc, 2018, 2018(5): 91-106.
[15]	HE Xiao-long, MA Hai-xin, HE Yu-kun, PANG Tian-ze, ZHAO Qiong. Technology and implementation of a new OLTP system [J]. Journal of East China Normal University(Natural Sc, 2018, 2018(5): 107-119.