华东师范大学学报(自然科学版) ›› 2014, Vol. 2014 ›› Issue (5): 89-102.doi: 10.3969/j.issn.10005641.2014.05.008

• 计算机科学与技术 • 上一篇    下一篇

支持通信数据查询分析的分布式计算系统

晁平复1，郑芷凌1，房俊华1，张蓉1,2   

  1. 1. 华东师范大学，软件学院，上海，200062; 2. 华东师范大学，数据科学与工程学院，上海，200062
  • 出版日期:2014-09-25 发布日期:2014-11-27
  • 通讯作者: 张蓉，女，副教授，硕士生导师，研究方向为Web数库原理 E-mail:rzhang@sei.ecnu.edu.cn
  • 基金资助:

    国家自然科学基金（61103039）；国家自然科学基金重点项目（61232002）；武汉大学重点实验室开放基金（SKLSE20120916）

Distributed computing system for communication data management

CHAO  Ping-Fu1, ZHENG  Zhi-Ling1, FANG  Jun-Hua1, ZHANG  Rong1,2   

  1. 1. Software Engineering Institute, East China Normal University, Shanghai,200062, China;
    2. Institute for Data Science and Engineering,  East China Normal University, Shanghai,200062, China
  • Online:2014-09-25 Published:2014-11-27

PDF

1875

摘要： 本文详细介绍一种基于分布式内存的通信数据管理平台，可以有效支持针对通信数据的实时处理以及在线查询分析任务.首先，通过对分布式技术、内存技术进行分析，依据通信业务的特点进行技术选型；然后，基于高性能的分布式内存计算环境，设计、实现针对通信数据的实时处理与在线查询分析平台；最后，利用内存技术对平台实现方案进行技术优化.实验结果表明，基于内存的分布式计算系统在查询响应速度、实时数据处理效率、系统资源利用率以及数据吞吐量上较基于硬盘的系统较大的性能提升.

关键词: 通信数据管理, 分布式系统, 内存计算, 实时数据处理

Abstract: In this article, a communication data management platform based on an open sourced resource on clustered system which has the requirement for inmemory data computing is introduced, in order to support real time processing as well as online queries under the massive data volume efficiently. In particular, we firstly give a brief analysis on popular distributed and in memory techniques, provide candidate techniques and test for choosing the approprate ones for our task. Then, we design and implement an online communication data processing and query platform. Finally, we use inmemory techniques to optimize our platform performance. The experimental results indicate that the inmemory distributed computing system not only outperforms the disk based system in both query response time and real time processing speed, but also improves on resource utilization and data throughput.

Key words: communication data management, distributed system, in memory computation, real time data processing

中图分类号: