OpenRank contribution evaluation method and empirical study in open-source course

doi:10.3969/j.issn.1000-5641.2024.05.002

Abstract

Abstract:

This study presents an OpenRank-based method for evaluating open-source contributions, designed to address the challenge of quantifying student contributions in open-source projects. Taking the “Open-Source Software Design and Development” course as a case study, we developed a method to assess student contributions in open-source practice. The OpenRank algorithm, which is based on developer collaboration networks, evaluates student contributions in discussions, problem-solving, and coding. Experimental results indicate that OpenRank not only aligns with traditional grading methods but also provides a more comprehensive view of student contributions. Combining OpenRank with traditional grading offers a more scientific and thorough evaluation of student contributions and skills in open-source projects.

Key words: open-source collaboration, open-source contribution, contribution evaluation, empirical research

CLC Number:

TP391

Jie WANG, Wenrui HUANG, Shengyu ZHAO, Xiaoya XIA, Fanyu HAN, Wei WANG, Yanbin ZHANG. OpenRank contribution evaluation method and empirical study in open-source course[J]. Journal of East China Normal University(Natural Science), 2024, 2024(5): 11-19.

Figures/Tables 8

Fig.1

Fig.2

Fig.3

Table 1

Fig.4

Table 2

Table 3

Fig.5

References 19

1	ELLIS H J C, HISLOP G W, JACKSON S, et al.. Team project experiences in humanitarian free and open source software (HFOSS). ACM Transactions on Computing Education, 2015, 15 (4): 18.
2	朝乐门.. 开源课程及数据科学导论的开源. 计算机科学, 2020, 47 (12): 114- 118..
3	荆琦, 冯惠.. 产教融合下的双轨制开源教学模式探索——以北京大学“开源软件开发基础及实践”课程为例. 高等工程教育研究, 2023, 41 (1): 14- 19..
4	黄启春.. 基于特色项目制与开源人才培养的软件工程专业学位研究生创新能力培养实践. 中国信息界, 2024, 22 (2): 25- 27..
5	KAN S H. Metrics and Models in Software Quality Engineering [M]. Boston: Addison-Wesley Professional, 2003.
6	WALSTON C E, FELIX C P.. A method of programming measurement and estimation. IBM Systems Journal, 1977, 16 (1): 54- 73..
7	GOUSIOS G, KALLIAMVAKOU E, SPINELLIS D. Measuring developer contribution from software repository data [C]// Proceedings of the 2008 International Working Conference on Mining Software Repositories. 2008: 129-132.
8	XIA X, WENG Z, WANG W, et al. Exploring activity and contributors on GitHub: Who, what, when, and where [C]// Proceedings of the 2022 29th Asia-Pacific Software Engineering Conference. 2022: 11-20.
9	PINTO G, FERREIRA C, SOUZA C, et al. Training software engineers using open-source software: The students’ perspective [C]// 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering Education and Training. 2019: 147-157.
10	SILVA J, WIESE I, GERMAN D M, et al. A theory of the engagement in open source projects via summer of code programs [C]// Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 2020: 421-431.
11	HU Z, SONG Y, GEHRINGER E F. Open-source software in class: Students’ common mistakes [C]// Proceedings of the 40th International Conference on Software Engineering: Software Engineering Education and Training. 2018: 40-48.
12	HOLMES R, ALLEN M, CRAIG M. Dimensions of experientialism for software engineering education [C]// Proceedings of the 40th International Conference on Software Engineering: Software Engineering Education and Training. 2018: 31-39.
13	DEKOENIGSBERG G. How successful open source projects work, and how and why to introduce students to the open source world [C]// Proceedings of the 2008 21st Conference on Software Engineering Education and Training. 2008: 274-276.
14	TSAY J, DABBISH L, HERBSLEB J. Influence of social and technical factors for evaluating contribution in GitHub [C]// Proceedings of the 36th International Conference on Software Engineering. 2014: 356-366.
15	JOBLIN M, APEL S, HUMSEN C, et al. Classifying developers into core and peripheral: An empirical study on count and network metrics [C]// Proceedings of the 2017 IEEE/ACM 39th International Conference on Software Engineering. 2017: 164-174.
16	CHENG J, GUO J L C. Activity-based analysis of open source software contributors: Roles and dynamics [C]// Proceedings of the 2019 IEEE/ACM 12th International Workshop on Cooperative and Human Aspects of Software Engineering. 2019: 11-18.
17	PAGE L, BRIN S, MOTWANI R, et al. The PageRank citation ranking: Bring order to the web [C]// Proceedings of the 7th International World Wide Web Conference. 1998: 1-17.
18	LI J, WILLETT P.. ArticleRank: A PageRank-based alternative to numbers of citations for analysing citation networks. Aslib Proceedings, 2009, 61 (6): 605- 618.
19	LI Y, LI C, CHEN W. Research on influence ranking of Chinese movie heterogeneous network based on PageRank algorithm [C]// Proceedings of the Web Information Systems and Applications: 15th International Conference. 2018: 344-356.

时间	节点数量/个
时间	用户	Issue	PR
2023-03	75	34	2
2023-04	94	58	42
2023-05	117	54	124

时间	节点数量/个	边数量/条	平均度	网络密度
2023-03	113	616	10.90	0.0487
2023-04	195	940	9.64	0.2485
2023-05	296	1160	7.83	0.0133

学生成绩	Pearson相关系数	Pearson相关系数的$p $值	Spearman相关系数	Spearman相关系数的$p $值
平时分	0.54	1.11×10^–7	0.60	1.07×10^–9
期中作业	0.25	1.95×10^–2	0.28	9.79×10^–3
期末作业	0.31	4.19×10^–3	0.33	1.93×10^–3
总成绩	0.53	1.57×10^–7	0.58	8.19×10^–9