计算机科学与技术

支持通信数据查询分析的分布式计算系统

  • 晁平复 ,
  • 郑芷凌 ,
  • 房俊华 ,
  • 张蓉
展开
  • 1. 华东师范大学,软件学院,上海,200062; 2. 华东师范大学,数据科学与工程学院,上海,200062

网络出版日期: 2014-11-27

基金资助

国家自然科学基金(61103039);国家自然科学基金重点项目(61232002);武汉大学重点实验室开放基金(SKLSE20120916)

Distributed computing system for communication data management

  • CHAO Ping-Fu ,
  • ZHENG Zhi-Ling ,
  • FANG Jun-Hua ,
  • ZHANG Rong
Expand
  • 1. Software Engineering Institute, East China Normal University, Shanghai,200062, China;
    2. Institute for Data Science and Engineering,  East China Normal University, Shanghai,200062, China

Online published: 2014-11-27

摘要

本文详细介绍一种基于分布式内存的通信数据管理平台,可以有效支持针对通信数据的实时处理以及在线查询分析任务.首先,通过对分布式技术、内存技术进行分析,依据通信业务的特点进行技术选型;然后,基于高性能的分布式内存计算环境,设计、实现针对通信数据的实时处理与在线查询分析平台;最后,利用内存技术对平台实现方案进行技术优化.实验结果表明,基于内存的分布式计算系统在查询响应速度、实时数据处理效率、系统资源利用率以及数据吞吐量上较基于硬盘的系统较大的性能提升.

本文引用格式

晁平复 , 郑芷凌 , 房俊华 , 张蓉 . 支持通信数据查询分析的分布式计算系统[J]. 华东师范大学学报(自然科学版), 2014 , 2014(5) : 89 -102 . DOI: 10.3969/j.issn.10005641.2014.05.008

Abstract

In this article, a communication data management platform based on an open sourced resource on clustered system which has the requirement for inmemory data computing is introduced, in order to support real time processing as well as online queries under the massive data volume efficiently. In particular, we firstly give a brief analysis on popular distributed and in memory techniques, provide candidate techniques and test for choosing the approprate ones for our task. Then, we design and implement an online communication data processing and query platform. Finally, we use inmemory techniques to optimize our platform performance. The experimental results indicate that the inmemory distributed computing system not only outperforms the disk based system in both query response time and real time processing speed, but also improves on resource utilization and data throughput.
文章导航

/