Journal of East China Normal University(Natural Science) ›› 2022, Vol. 2022 ›› Issue (5): 26-35.doi: 10.3969/j.issn.1000-5641.2022.05.003

• Blockchain System and Data Management • Previous Articles     Next Articles

Optimization of HTAP data synchronization based on query frequency

Yongjin TANG, Jiabo SUN, Peng CAI*()   

  1. School of Data Science and Engineering, East China Normal University, Shanghai 200062, China
  • Received:2022-07-11 Online:2022-09-25 Published:2022-09-26
  • Contact: Peng CAI E-mail:pcai@dase.ecnu.edu.cn

Abstract:

A hybrid transaction analytical processing (HTAP) system must concurrently support both transaction processing and query analysis. To eliminate interference between them, HTAP systems also typically assign different copies of data to both workloads, handling online transaction processing (OLTP) and online analytical processing (OLAP) requests separately, and synchronizing data between the copies based on a log replay. An HTAP system is committed to efficiently synchronizing OLTP data to OLAP, thereby providing a fresher data access service. In addition, the speed of sending and replaying the logs of the tables to be queried is a key factor affecting the freshness of the data. In this paper, using the table grouping based log parallel replay method and the characteristics of the HTAP load, a log sending and replay method is proposed based on the query frequency of the OLAP side. To ensure data consistency, this method improves the processing priority of high-frequency query table logs and achieves efficient log sending and replay capabilities along with a targeted priority display of high-frequency query table data, thereby ensuring the freshness of the HTAP system.

Key words: hybrid transaction analytical processing, data freshness, log replay, conflict detection

CLC Number: