华东师范大学学报(自然科学版)

• 计算机科学 • 上一篇    下一篇

OceanBase 中基于布隆过滤器的连接算法

茅潇潇, 段惠超, 高 明   

  1. 华东师范大学 数据科学与工程研究院, 上海 200062
  • 收稿日期:2016-06-24 出版日期:2016-09-25 发布日期:2016-11-29
  • 通讯作者: 高 明, 男, 副教授, 研究方向为高可用事务处理优化、数据挖掘等. E-mail: mgao@sei.ecnu.edu.cn.
  • 基金资助:

    国家 863 计划项目(2015AA015307)

A join algorithm based on bloom filter in OceanBase

MAO Xiao-xiao, DUAN Hui-chao, GAO Ming   

  1. Institute for Data Science and Engineering, East China Normal University, Shanghai 200062, China
  • Received:2016-06-24 Online:2016-09-25 Published:2016-11-29

RichHTML

14

PDF

865

摘要：

在大数据时代, ``去IOE'' 运动的推进以及``双 11''等活动的兴起对分布式数据库系统提出了更高的要求. OceanBase 是阿里巴巴集团自主研发的开源分布式数据库, 支持海量数据跨行跨表事务, 但是对复杂查询的处理性能仍有待提高, 其中连接操作带来的网络传输严重影响了数据库的性能. 本文提出了一种基于布隆过滤器的连接算法, 通过构建布隆过滤器对右表数据进行过滤, 减少了不必要的数据传输开销, 降低了数据处理带来的内存资源的消耗. 本文在 OceanBase 上实现了该算法, 并通过实验证明, 该算法极大提高了连接操作的效率.

关键词: OceanBase, 连接操作, 布隆过滤器

Abstract:

In the era of big data, the movement of “de-IOE” campaign and the development of activities such as Double 11 have put forward higher request of the performance of distributed database. OceanBase is an open sourced distributed database implemented by Alibaba. It supports for cross-table relational query of massive data but the performance for complex queries remains to be improved. The network transmission overheads caused by join operator seriously influenced the performance of distributed database. This paper proposes a join algorithm based on bloom filter. It filters the data of the right table
by constructing a bloom filter on the join column of the left table. The key point of this algorithm is that it reduces the overhead of unnecessary data transmission and the consumption of memory resources by data processing. We implement this algorithm in OceanBase and the experiment results show that the algorithm can greatly improve the efficiency of join operator.

Key words: OceanBase, join operation, bloom filter