Data calculation and performance optimization of dairy traceability based on Hadoop/Hive

doi:10.3969/j.issn.1000-5641.2018.04.010

Journal of East China Normal University(Natural Sc ›› 2018, Vol. 2018 ›› Issue (4): 99-108.doi: 10.3969/j.issn.1000-5641.2018.04.010

Previous Articles Next Articles

Data calculation and performance optimization of dairy traceability based on Hadoop/Hive

ZHU Shu-xin¹, LI Yue¹, YUAN Pei-sen¹, XU Huan-liang^1,2, WANG Kang¹, XIE Zhong-hong¹

1. College of Information Science and Technology, Nanjing Agricultural University, Nanjing 210095, China;
2. Jiangsu Collaborative Innovation Center of Meat Production and Processing, Quality and Safety Control, Nanjing 210095, China

Received:2017-06-19 Online:2018-07-25 Published:2018-07-19

Abstract

Abstract: In order to enhance the performance of traditional dairy traceability systems for the production data of large-scale enterprise, this paper analyzed the supply chain process of dairy enterprises, key traceability units and traceability information; combining Hadoop/Hive big data technology and distributed database technology, the paper designed and constructed a dairy products traceability framework based on Hadoop/Hive. We built a simulated large-scale data environment and used actual production data to test the system performance. The experimental results showed that after the introduction of the Hadoop/Hive technology system, the average data storage speed, the average data access speed, and the average data exchange rate increased by 87.43%, 27.10% and 58.16%, respectively. The improved traceability system for dairy products is superior to the traditional dairy traceability system in storing and processing large-scale data.

Key words: Hadoop/Hive, dairy products traceability, data calculation, performance optimization

CLC Number:

TP39

ZHU Shu-xin, LI Yue, YUAN Pei-sen, XU Huan-liang, WANG Kang, XIE Zhong-hong. Data calculation and performance optimization of dairy traceability based on Hadoop/Hive[J]. Journal of East China Normal University(Natural Sc, 2018, 2018(4): 99-108.

References

[1] ABOUZIED A, BAJDA-PAWLIKOWSKI K, HUANG J, et al. HadoopDB in action:Building real world applications[C]//ACM SIGMOD International Conference on Management of Data. ACM, 2010:1111-1114.
[2] ABOUZEID A, BAJDA-PAWLIKOWSKI K, ABADI D, et al. HadoopDB:An architectural hybrid of MapReduce and DBMS technologies for analytical workloads[J]. Proceedings of the VLDB Endowment, 2009, 2(1):922-933.
[3] ISMAIL A S, AL-FEEL H, MOKHTAR H M O. Introducing a new arabic endpoint for DBpedia internationalization project[C]//International Database Engineering & Applications Symposium. ACM, 2016:284-289.
[4] TORRES D, SKAF-MOLLI H, MOLLI P. et al. BlueFinder:Recommending wikipedia links using DBpedia properties[C]//Proceedings of the 5th Annual ACM Web Science Conference (WebSci'13). New York:ACM, 2013:413-422. DOI:https://doi.org/10.1145/2464464.2464515.
[5] 叶育鑫, 欧阳丹彤. 混合语义约简和选择估值优化SPARQL[J]. 电子学报, 2010, 38(5):1205-1210.
[6] 王德文, 肖凯, 肖磊. 基于Hive的电力设备状态信息数据仓库[J]. 电力系统保护与控制, 2013(9):125-130.
[7] 曲朝阳, 朱莉, 张士林. 基于Hadoop的广域测量系统数据处理[J]. 电力系统自动化, 2013, 37(4):92-97.
[8] 刘越, 李锦涛, 虎嵩林. 基于代价估计的Hive多维索引分割策略选择算法[J]. 计算机研究与发展, 2016, 53(4):798-810.
[9] 董新华, 李瑞轩, 周湾湾, 等. Hadoop系统性能优化与功能增强综述[J]. 计算机研究与发展, 2013, 50(s2):1-15.
[10] THUSOO A, SARMA J S, JAIN N, et al. Hive:A warehousing solution over a map-reduce framework[J]. Proceedings of the VLDB Endowment, 2009, 2(2):1626-1629.
[11] OLSEN P, BORIT M. How to define traceability[J]. Trends in Food Science & Technology, 2013, 29(2):142-150.
[12] 钱建平, 刘学馨, 杨信廷, 等. 可追溯系统的追溯粒度评价指标体系构建[J]. 农业工程学报, 2014, 30(1):98-104.
[13] SHVACHKO K, KUANG H, RADIA S, et al. The Hadoop distributed file system[C]//Proceedings of the IEEE 26th Symposium on Mass Storage Systems and Technologies. Washington:IEEE Computer Society, 2010:1-10. DOI:10.1109/MSST.2010.5496972.
[14] 张良均. Hadoop大数据分析与挖掘实战[M]. 北京:机械工业出版社, 2016.
[15] 荀亚玲, 张继福, 秦啸. MapReduce集群环境下的数据放置策略[J]. 软件学报, 2015(8):2056-2073.
[16] 叶晓江, 刘鹏. 实战Hadoop2.0从云计算到大数据[M]. 北京:电子工业出版社, 2016.
[17] 张佳兰, 昝林森, 刘永峰, 等. 我国DHI测定现状及存在的问题[J]. 中国牛业科学, 2007, 33(5):56-59.

Data calculation and performance optimization of dairy traceability based on Hadoop/Hive

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 1

Recommended Articles

Metrics

Comments