J* E* C* N* U* N* S* ›› 2025, Vol. 2025 ›› Issue (5): 151-161.doi: 10.3969/j.issn.1000-5641.2025.05.014

• Open Source Ecosystem: Development and Governance • Previous Articles    

Open source evaluatology: A framework and methodology for evaluating open source ecosystems based on evaluatology

Shengyu ZHAO1(), Wei WANG2,*(), Fanyu HAN2, Jiaheng PENG2, Lan YOU3   

  1. 1. Department of Computer Science and Technology, Tongji University, Shanghai 201804, China
    2. School of Data Science and Engineering, East China Normal University, Shanghai 200062, China
    3. School of Computer Science and Information Engineering, Hubei University, Wuhan 430062, China
  • Received:2025-01-10 Online:2025-09-25 Published:2025-09-25
  • Contact: Wei WANG E-mail:frank_zsy@tongji.edu.cn;wwang@dase.ecnu.edu.cn

Abstract:

The open source ecosystem, as a critical component of the modern software industry, has garnered increasing attention from both academia and industry regarding its evaluation challenges. However, existing evaluation methods face issues such as inconsistent evaluation standards, lack of theoretical grounding, and poor comparability of evaluation results. Guided by foundational theories of evaluatology, this study introduced a novel interdisciplinary research domain, open source evaluatology, for the first time. It established a theoretical framework and methodological system for evaluating the open source ecosystem. The primary contributions of this paper include the following. Developing the theoretical foundation of open source evaluatology based on the five axioms of evaluatology and defining fundamental concepts, evaluation dimensions, and standards for open source ecosystem evaluation. Designing an evaluation conditions framework comprising five levels: problem definition, task instances, algorithm mechanisms, implementation examples, and supporting systems. A hybrid evaluation model combining statistical and network metrics was proposed. Based on the experiments conducted using the GitHub dataset, this study validated the proposed method from three dimensions: open source repositories, developers, and communities. The results demonstrated that the proposed evaluation model exhibited strong applicability and explanatory power in open source scenarios.

Key words: open source evaluatology, evaluatology, open source ecosystem, evaluation framework, evaluation methodology

CLC Number: