华东师范大学学报（自然科学版） >

2018 , Vol. 2018 >Issue 5: 67 - 78

DOI: https://doi.org/10.3969/j.issn.1000-5641.2018.05.006

高性能数据库管理

面向Cedar的列存储设计与实现

  • 俞文谦 ,
  • 胡爽 ,
  • 胡卉芪
展开
  • 华东师范大学 数据科学与工程学院, 上海 200062
俞文谦,男,硕士研究生,研究方向为分布式数据库系统.E-mail:wqyu_cs@163.com.

收稿日期: 2018-07-09

  网络出版日期: 2018-09-26

基金资助

国家自然科学基金（61702189）；上海市青年扬帆计划（17YF1427800）

收起

The designs and implementations of columnar storage in Cedar

  • YU Wen-qian ,
  • HU Shuang ,
  • HU Hui-qi
Expand
  • School of Data Science and Engineering, East China Normal University, Shanghai 200062, China

Received date: 2018-07-09

  Online published: 2018-09-26

Fold

摘要

随着数据规模和分析需求的日益增长，数据库面向联机分析处理（On-Line Analytical Processing，OLAP）应用的查询性能变得愈发重要.Cedar是一款基于读写分离架构的分布式关系数据库，由于它主要面向联机事务处理（On-Line Transaction Processing，OLTP）业务，在面对分析处理负载时性能表现不足.对于这个问题，很多研究表明列存储技术能够有效地提高I/O（Input/Output）效率，进而提升分析处理的性能.在Cedar上提出了一种列存储机制，分析了其适用场景并针对这种机制改进了Cedar的数据扫描和批量更新方法.实验结果表明，该机制能大幅度地提升Cedar分析处理性能，并且对事务处理性能的影响控制在10%以内.

关键词： 分布式数据库; 列存储; 联机分析处理(OLAP)

本文引用格式

俞文谦 , 胡爽 , 胡卉芪 . 面向Cedar的列存储设计与实现[J]. 华东师范大学学报（自然科学版）, 2018 , 2018(5) : 67 -78 . DOI: 10.3969/j.issn.1000-5641.2018.05.006

Abstract

With the growing size of data and analytical needs, the query performance of databases for OLAP (On-Line Analytical Processing) applications has become increasingly important. Cedar is a distributed relational database based on read-write decoupled architecture. Since Cedar is mainly oriented to the needs of OLTP (On-Line Transaction Processing) applications, it has insufficient performance for handling analytical processing workloads. To address this issue, many studies have shown that column storage technology can effectively improve the efficiency of I/O (Input/Output) and enhance the performance of analytical processing. This paper presents a column-based storage mechanism in Cedar. The study analyzes applicable scenarios and improves Cedar's data query and batch update methods for this mechanism. The results of an experiment demonstrate that the proposed mechanism can enhance the performance of analytical processing substantially, while limiting the negative impacts on transaction processing performance to within 10%.

Key words： distributed database; column-based storage; OLAP

参考文献

[1] CODD E F, CODD S B, SALLEY C T. Providing OLAP (On-Line Analytical Processing) to User-Analysts:An IT mandate[J]. Codd and Date, 1993, 32:3-5.
[2] 华东师范大学. 面向大型银行应用的高通量可伸缩分布式数据库系统Cedar[DB/OL].[2018-05-16]. https://github.com/daseECNU/Cedar.
[3] COPELAND G P, KHOSHAFIAN S N. A decomposition storage model[C]//ACM SIGMOD International Conference on Management of Data. New York:ACM, 1985:268-279.
[4] ABADI D, MADDEN S, HACHEM N, Column-stores vs. row-stores:How different are they really?//Proceedings of the 2008 ACM SIGMOD international conference on Management of data. New York:ACM, 2008:967-980.
[5] RAMAN V, ATTALURI G, BARBER R, et al. DB2 with BLU acceleration:So much more than just a column store[J]. Proceedings of the VLDB Endowment, 2013, 11:1080-1091.
[6] PETRAKI E, IDREOS S, MANEGOLD S. Holistic Indexing in main-memory column-stores[C]//ACM SIGMOD International Conference on Management of Data. New York:ACM, 2015:1153-1166.
[7] LANG H, FUNKE F, BONCZ P A, et al. Data blocks:Hybrid OLTP and OLAP on compressed storage using both vectorization and compilation[C]//International Conference on Management of Data. New York:ACM, 2016:311-326.
[8] RAMNARAYAN J, MOZAFARI B, WALE S, et al. SnappyData:A hybrid transactional analytical store built on spark[C]//International Conference on Management of Data. New York:ACM, 2016:2153-2156.
[9] LEE J, HAN W S, NA H J, et al. Parallel replication across formats for scaling out mixed OLTP/OLAP workloads in main-memory databases[J]. The VLDB Journal, 2018, 27(3):421-444.
[10] SQream. SQream SQream DB[DB/OL].[2018-06-16]. https://sqream.com/.
[11] ROOT C, MOSTAK T. MapD:A GPU-powered big data analytics and visualization platform[C]//Proceeding of the SIGGRAPH'16 ACM SIGGRAPH 2016 Talks. New York:ACM, 2016:Article No 73. DOI:10.1145/2897839.2927468.
[12] 阳振坤. OceanBase关系数据库架构[J]. 华东师范大学学报(自然科学版), 2014(5):141-148, 163.
[13] 黄贵, 庄明强. OceanBase分布式存储引擎[J].华东师范大学学报(自然科学版), 2014(5):164-172.
[14] GOOGLE. Google Snappy[DB/OL].[2018-06-16]. https://github.com/google/snappy.
[15] SALOMON D. Data Compression:The Complete Reference[M]. New York:Springer-Verlag Inc, 2000.
[16] APACHE. Apache parquet[DB/OL].[2018-06-16]. http://parquet.apache.org/.
Options
文章导航

/

〈 〉