OSS Insight: A platform for open source ecosystem spatiotemporal data analysis and insights

doi:10.3969/j.issn.1000-5641.2025.05.016

Abstract

Abstract:

An open source ecosystem abounds with valuable data, yet extracting insights requires innovative data infrastructure and analytical methods. To address this, OSS Insight was developed that innovatively used the hybrid transactional analytical processing(HTAP) database for efficient storage and query of billions of GitHub event data and offered real-time exploration via a visual interface. It delved into spatiotemporal data analysis, modeling developer behaviors and ecosystem evolution, such as visualizing global contribution patterns. Integrated with large language models(LLMs), it enabled natural language to structured query language(SQL) conversion for intelligent querying. A case study of Kubernetes showcased its capabilities in analyzing developers, project evolution, and organizational collaboration. Experiments proved that OSS Insight efficiently analyzed large-scale open source data, and its LLM-driven interaction simplified data analysis and provided automated insights.

Key words: open source ecosystem, open source insight, spatiotemporal data analysis, HTAP, LLM

CLC Number:

TP391

Xiaowei CHEN, Wei WANG, Fanyu HAN, Guanglei BAO, Fei DONG, Hao HUO, Chen LIU. OSS Insight: A platform for open source ecosystem spatiotemporal data analysis and insights[J]. J* E* C* N* U* N* S*, 2025, 2025(5): 170-182.

Figures/Tables 9

Table 1

Fig.1

Fig.2

Fig.3

Fig.4

Fig.5

Fig.6

Fig.7

Fig.8

References 25

1	KILAMO T, HAMMOUDA I, MIKKONEN T, et al.. From proprietary to open source—Growing an open source ecosystem. Journal of Systems and Software, 2012, 85 (7): 1467- 1478.
2	FRANCO-BEDOYA O, AMELLER D, COSTAL D, et al.. Open source software ecosystems: A systematic mapping. Information and software technology, 2017, 91, 160- 185.
3	VALIEV M, VASILESCU B, HERBSLEB J. Ecosystem-level determinants of sustained activity in open-source projects: A case study of the PyPI ecosystem [C]// Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 2018: 644-655.
4	GITHUB. Octoverse 2024: The state of open source software [EB/OL]. (2024-10-29)[2025-01-10]. https://octoverse.github.com.
5	GOUSIOS G, SPINELLIS D. GHTorrent: GitHub’s data from a firehose [C]// 2012 9th IEEE Working Conference on Mining Software Repositories (MSR). IEEE, 2012: 12-21.
6	GITHUB. GH Archive is a project to record the public GitHub timeline, archive it, and make it easily accessible [EB/OL]. (2011-02-12)[2025-01-10]. https://www.gharchive.org.
7	CNCF. CloudNative DevStats: CNCF-created tool for analyzing and graphing developer contributions [EB/OL]. (2018-01-30)[2025-01-10]. https://devstats.cncf.io.
8	X-lab 开放实验室. OpenDigger: 面向开源协作数字生态的数据挖掘与信息服务系统 [EB/OL]. (2023-07-04)[2025-01-10]. https://arxiv.org/pdf/2311.15204.
9	HUANG D X, LIU Q, CUI Q, et al.. TiDB: A Raft-based HTAP database. Proceedings of the VLDB Endowment, 2020, 13 (12): 3072- 3084.
10	RIEHLE D, BERSCHNEIDER S. A model of open source developer foundations [C]// IFIP International Conference on Open Source Systems. Berlin: Springer, 2012: 15-28.
11	ZHAO S Y, XIA X Y, FITZGERALD B, et al. OpenRank leaderboard: Motivating open source collaborations through social network evaluation in Alibaba [C]// Proceedings of the 46th International Conference on Software Engineering: Software Engineering in Practice. 2024: 346-357.
12	KIAH M L M, HAIQI A, ZAIDAN B B, et al.. Open source EMR software: Profiling, insights and hands-on analysis. Computer Methods and Programs in Biomedicine, 2014, 117 (2): 360- 382.
13	韩凡宇, 毕枫林, 张琰彬, 等. OpenPerf: 面向开源生态可持续发展的数据科学基准测试体系 [J/OL]. 计算机学报, (2024-11-01)[2025-01-10]. http://cjc.ict.ac.cn/online/onlinepaper/hfy-202538104255.pdf.
14	ENGLISH M, EXTON C, RIGON I, et al. Fault detection and prediction in an open-source software project [C]// Proceedings of the 5th International Conference on Predictor Models in Software Engineering. 2009: 17.
15	CALLAHAN T J, TRIPODI I J, STEFANSKI A L, et al.. An open source knowledge graph ecosystem for the life sciences. Scientific Data, 2024, 11, 363.
16	ALAMI A, PARDO R, LINÅKER J.. Free open source communities sustainability: Does it make a difference in software quality?. Empirical Software Engineering, 2024, 29 (5): 114.
17	FRANKE L, LIANG H, FARZANEHPOUR S, et al. An exploratory mixed-methods study on general data protection regulation (GDPR) compliance in open-source software [C]// Proceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement. 2024: 325-336.
18	RASTOGI A, NAGAPPAN N, GOUSIOS G, et al. Relationship between geographical location and evaluation of developer contributions in GitHub [C]// Proceedings of the 12th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement. 2018: 22.
19	WorldofCode(WoC).World of code: an infrastructure for mining the universe of open source VCS data [EB/OL]. (2019-05-26)[2025-01-10]. https://worldofcode.org.
20	CHAOSS. GrimoireLab: Platform for software development analytics and insights [EB/OL]. (2017-09-20)[2025-01-10]. https://chaoss.github.io/grimoirelab.
21	AIT A, IZQUIERDO J L C, CABOT J. An empirical study on the survival rate of GitHub projects [C]// Proceedings of the 19th International Conference on Mining Software Repositories. 2022: 365-375.
22	NISHIURA K, IKEDA K, SASAKURA M, et al. Exploring behavioral trustworthiness of GitHub developers [C]// Proceedings of the 2023 5th World Symposium on Software Engineering. 2023: 92-95.
23	LIMA A, ROSSI L, MUSOLESI M. Coding together at scale: GitHub as a collaborative social network [C]// Proceedings of the International AAAI Conference on Web and Social Media. 2014, 8(1): 295-304.
24	GHAZAL A, LIANG Z Y, BAINS S, et al.. OSSInsight: Scalable GitHub analysis. Proceedings of the VLDB Endowment, 2024, 17 (12): 4321- 4324.
25	张超, 李国良, 冯建华, 等.. HTAP 数据库关键技术综述. 软件学报, 2023, 34 (2): 761- 785.

工具	核心特点	存在的局限
GHTorrent	收集GitHub全量元数据, 支持复杂社会网络分析	仅快照级数据, 更新频率低, 缺乏实时性
GH Archive	提供GitHub事件流, 适合时间序列分析	仅原始事件, 需自行清洗与建模, 门槛高
DevStats	针对云原生项目, 提供标准化统计仪表盘	缺乏通用性和自定义探索
OpenDigger	提供开发者活跃度、影响力等指标数据	主要统计指标有限, 缺乏时空动态分析能力
World of Code	整合多平台数据, 规模大	数据访问复杂, 使用门槛极高
GrimoireLab	支持多源数据分析, 可自建平台	部署与维护复杂, 对用户要求高

[1]	Shengyu ZHAO, Wei WANG, Fanyu HAN, Jiaheng PENG, Lan YOU. Open source evaluatology: A framework and methodology for evaluating open source ecosystems based on evaluatology [J]. J* E* C* N* U* N* S*, 2025, 2025(5): 151-161.
[2]	Sijia KOU, Fengyun YAN, Jing MA. A case study on the application of the automatic labelling of the subject knowledge graph of Chinese large language models: Take morality and law and mathematics as examples [J]. Journal of East China Normal University(Natural Science), 2024, 2024(5): 81-92.
[3]	Jiarui ZHANG, Qiming ZHANG, Fenglin BI, Yanbin ZHANG, Wei WANG, Erjin REN, Haili ZHANG. Locally lightweight course teaching-assistant system based on IPEX-LLM [J]. Journal of East China Normal University(Natural Science), 2024, 2024(5): 162-172.
[4]	Rong YU, Panfei YANG, Qingshuai WANG, Rong ZHANG. An HTAP database prototype with an adaptive data synchronization [J]. Journal of East China Normal University(Natural Science), 2023, 2023(5): 11-25.
[5]	CEN Yuan-jun. Valuation of American continuous-installment put options [J]. Journal of East China Normal University(Natural Sc, 2019, 2019(3): 24-34,62.
[6]	LI Yan-bin, PAN Yan-hong, GU Hang, WANG Lei, SHI Bing, SUN Chen, XIA Fan, DONG Qi-wen, SONG Shu-bin. Design and implementation of graduate student system [J]. Journal of East China Normal University(Natural Sc, 2017, 2017(5): 213-224.