Data application system for multi-source data-aided materialresearch and development

doi:10.3969/j.issn.1000-5641.2022.05.002

Abstract

Abstract:

The material research industry generates a wide variety of data from a wide range of sources. Notably, the data islanding issue pertaining to this scenario demands an application platform that integrates data collection, management, and application so that the value of the data can be better utilized for development. For this purpose, this study proposes a data management system that handles the entire data management lifecycle from collection to application for the material research industry, including thematic database, formula performance prediction, and formula correlation analysis capabilities. Itis expected that this development will promote data-driven innovations and improvements within the industry.

Key words: data management, data collection, data management system

CLC Number:

N945

Yuchen HONG, Yiming YU, Ye WANG, Qiwen DONG. Data application system for multi-source data-aided materialresearch and development[J]. Journal of East China Normal University(Natural Science), 2022, 2022(5): 14-25.

Figures/Tables 11

Fig.1

Fig.2

Fig.3

Fig.4

Fig.5

Fig.6

Table 1

Fig.7

Fig.8

Table 2

Table 3

References 22

1	中华人民共和国工业和信息化部. 工业和信息化部关于工业大数据发展的指导意见 [EB/OL]. 信息技术发展司, (2020-05-13)[2022-07-10]. https://www.miit.gov.cn/xwdt/gxdt/sjdt/art/2020/art_a61849ebec144ebdb91fa9bc5474554c.html.
2	中华人民共和国工业和信息化部. 《工业互联网创新发展行动计划2021—2023年》解读 [EB/OL]. 信息通信管理局, (2021-02-18)[2022-07-10]. https://www.miit.gov.cn/zwgk/zcjd/art/2021/art_8ddb39d0d4134b42aa8fd39a4b9f017e.html.
3	王建民. 工业大数据技术综述. 大数据, 2017, 3 (6): 3- 14.
4	张小钢, 宋峥嵘, 李金来, 等. 材料编码映射辅助系统研究. 中国勘察设计, 2015, (6): 81- 84.
5	何磊, 何亮, 马晓阳. 船体结构钢及焊接材料数据库应用系统设计开发. 材料开发与应用, 2021, 36 (6): 49- 55.
6	The Apache Software Foundation. Apache sqoop [R/OL]. (2019-01-18)[2022-07-10]. https://sqoop.apache.org/.
7	Alibaba. DataX [R/OL]. [2022-07-10]. https://www.aliyun.com/product/bigdata/ide.
8	The Apache Software Foundation. Apache H2O [R/OL]. [2022-07-10]. https://docs.h2o.ai/.
9	南方电网报. 盘活数据资产激活数据价值——南方电网公司加快推进数据要素化资产化进程赋能数字化转型 [EB/OL]. (2021-08-03)[2022-07-10]. http://www.csg.cn/xwzx/2021/gsyw/202108/t20210803_321202.html.
10	HOLZINGER A, STRUGGL K H, DEBEVC M. Applying modelview controller (MVC) in design and development of information systems: An example of smart assistive script breakdown in an ebusiness application [C]// International Conference on e-Business (ICE-B). IEEE, 2010: 1-6.
11	STUURMAN S. Software architecture, java beans [C]// TC2 First Working IFIP Conference on Software Architecture (WICSA1). 1999: 183-200.
12	DRAGONI N, GIALLORENZO S, LLUCH LAFUENTE A, et al. Microservices: Yesterday, today, and tomorrow [EB/OL]. (2019-05-28)[2022-07-10]. https://arxiv.org/abs/1606.04036.
13	SHVACHKO K, KUANG H, RADIA S, et al. The hadoop distributed file system [C]// IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST). IEEE, 2010: 1-10.
14	ORACLE. MySQL binary log [EB/OL]. [2022-07-10]. https://dev.mysql.com/doc/refman/8.0/en/binary-log.html.
15	CHRISTINE M. Plant physiology: Plant biology in the Genome Era. Science, 1998, 281, 331- 332.
16	The Apache Software Foundation. Apache sorl [EB/OL]. [2022-07-10]. https://solr.apache.org/.
17	The Apache Software Foundation. Apache kafka [EB/OL]. [2022-07-10]. https://kafka.apache.org/.
18	The Apache Software Foundation. Apache hive [EB/OL]. [2022-07-10]. https://hive.apache.org/.
19	The Apache Software Foundation. Apache airflow [EB/OL]. [2022-07-10]. https://airflow.apache.org/.
20	The Elastic Corporation. Kibana 用户手册 [EB/OL]. [2022-07-10]. https://www.elastic.co/guide/cn/kibana/current/index.html.
21	王大阜, 邓志文, 贾志勇, 等. 基于ELK构建图书馆多数据源日志管理分析平台. 新世纪图书馆, 2022, (2): 68- 74.
22	The Apache Software Foundation. Apache kibana [EB/OL]. [2022-07-10]. https://www.elastic.co/guide/cn/kibana/current/index.html.

数据类型	排序规则	示例
数值型	数据越大越好	数据在[A,B]区间内, 数值越大得分越高
	数据越小越好	数据在[A,B]区间内, 数值越小得分越高
	越靠近某个数值越好	数据在[A,B]区间内, 数值越靠近C得分越高
枚举型	按类型排序	不同枚举值对应不同分数

用例编号	功能模块	测试步骤	预期结果	实际结果
1	数据采集, 数据录入	1: 调用接口上传数据 2: 数据库中添加数据, 开启数据同步	1: 接收并存储对应数据 2: 获取增量数据并存储	通过
2	数据处理, 定时任务	1: 配置数据过滤流程 2: 创建对应定时任务 3: 开启定时任务	1: 系统保存数据过滤配置 2: 系统显示创建成功 3: 数据过滤结果正确, 能够定时进行	通过
3	日志收集, 系统日志收集	1: 收集系统日志 2: 查看系统日志	1: 系统日志收集正常 2: 能在日志查看界面进行查看	通过
4	数据应用, 专题数据库	1: 配置专题数据库创建规则 2: 点击保存 3: 查看对应数据	1: 页面显示可配置的数据规则 2: 保存成功 3: 数据显示内容正确	通过
5	数据应用, 综合排序	1: 选择对应待排序数据 2: 配置排序规则 3: 运行并查看结果	1: 可选择数据范围正确 2: 页面实现配置规则并能保存成功 3: 运行返回结果正确	通过

功能模块	训练样本量	平均响应时间/ms
配方性能预测	500	252
	1000	385
	2000	510
配方组分推荐	500	128
	1000	213
	2000	386
相关性分析	500	476
相关性分析	1000	793

[1]	Danqi LIU, Peng CAI. Separate management strategies for Part metadata under the storage-computing separation architecture [J]. Journal of East China Normal University(Natural Science), 2023, 2023(5): 40-50.
[2]	Xinjun ZHENG, Guoliang TIAN, Feihu HUANG. A multimode data management method based on Data Fabric [J]. Journal of East China Normal University(Natural Science), 2023, 2023(5): 164-181.
[3]	Sijia DENG, Xing TONG, Haibo TANG, Zhao ZHANG, Cheqing JIN. Blockchain-oriented data management middleware [J]. Journal of East China Normal University(Natural Science), 2021, 2021(5): 60-73.
[4]	TANG Hai-bo, LIN Yu-ming, LI You. A K-Means based clustering algorithm with balanced constraints [J]. Journal of East China Normal University(Natural Sc, 2018, 2018(5): 164-171.
[5]	ZHANG Wen-Jie, PENG Zhi-Yong, PENG Yu-Wei. Application of inmemory data management technology in genealogy information system [J]. Journal of East China Normal University(Natural Sc, 2014, 2014(5): 311-319.
[6]	HUANG Lan, SUN Ke, CHEN Xiao-Zhu, ZHOU Min-Qi. In-memory cluster computing: Interactive data analysis [J]. Journal of East China Normal University(Natural Sc, 2014, 2014(5): 216-227.
[7]	CHAO Ping-Fu, ZHENG Zhi-Ling, FANG Jun-Hua, ZHANG Rong. Distributed computing system for communication data management [J]. Journal of East China Normal University(Natural Sc, 2014, 2014(5): 89-102.