华东师范大学学报(自然科学版) ›› 2014, Vol. 2014 ›› Issue (5): 17-30.doi: 10.3969/j.issn.10005641.2014.05.002

• 计算机科学与技术 • 上一篇    下一篇

集群资源统一管理和调度技术综述

李永峰1，周敏奇1，胡华梁2   

  1. 1.华东师范大学，软件学院，上海，200062 2. 浙江理工大学，经济管理学院，杭州，310018
  • 出版日期:2014-09-25 发布日期:2014-11-27
  • 通讯作者: 胡华梁，男，副教授，硕士生导师，研究方向为数据库应用. E-mail:jmxyandy@126.com
  • 基金资助:

    浙江省自然科学基金（LY12F02044）；国家自然科学基金（61332006）

Survey of resource uniform management and scheduling in cluster

 LI  Yong-Feng1, ZHOU  Min-Qi1, HU  Hua-Liang2   

  1. 1. Software Engineering Institute, East China Normal University, Shanghai,200062, China;
    2. School of Economics and Management, Zhejiang Sci Tech University, Hangzhou,310018, China
  • Online:2014-09-25 Published:2014-11-27

PDF

5511

摘要： 随着互联网的快速发展和大数据的来临，基于数据密集型应用的集群计算框架不断涌现，并且这些计算框架都只面向某一类特定领域的应用.基于这一特点，互联网公司往往需要部署和运行多个计算框架，从而为每个应用选择最优的计算框架.因此，资源统一管理和调度系统作为集群共享平台被提出来.集群资源统一管理和调度系统需要同时支持多种不同计算框架，如何管理集群计算资源和不同计算框架间的资源公平分配成为关键技术难点.不同计算框架的作业是异构的，如何在不同框架间进行作业调度，以充分利用集群资源和提高系统吞吐量，成为了新的挑战.本文针对现有的资源管理系统和应用需求特点，研究和分析了集群资源管理和调度的关键技术，并对现有的集群资源管理技术存在的问题和未来发展进行了探讨.

关键词: 资源管理系统, 资源共享, 作业调度

Abstract: With the rapid development of Internet and the coming of big data, resource management system, a thin resource sharing layer that enables sharing cluster across diverse cluster computing frameworks, by giving frameworks a common interface for accessing cluster resources. For powering both large Internet services and a growing number of dataintensive scientific applications, cluster computing framework will continue emerge, and no framework will be optimal for all applications. Therefore, multiplexing a cluster between frameworks makes significant difference. Deploying and running multiple frameworks in the same cluster, improves utilization and allowing applications to share access to large datasets that may be costly to replicate across clusters. This paper is aimed to illustrate current major techniques of resource management and scheduling in cluster, including resource representation model, resource allocation model and scheduling policy. Finally, current prominent solutions, which have been developed and used by many companies, will be demonstrated, and we then summary and contrast these solutions used in recent years.

Key words: resource management system, resource sharing, job scheduling

中图分类号: