Journal of East China Normal University(Natural Science) >
Indexing and query technology for steel logistic data
Received date: 2022-07-23
Online published: 2022-09-26
With digital transformation and the development of iron and steel logistics, the scale of iron and steel logistic data has rapidly expanded, and traditional relational databases can no longer meet the storage and query needs. Considering that a distributed not only structured query language (NoSQL) database has a simple expansion capability, fast reading and writing speeds, and low cost, in this study, distributed cloud storage and NoSQL technologies are used to store and build indexes for massive steel logistic data, improving the accuracy of the storage capacity and query performance of the logistic data. First, Spark is used to associate and fuse the data from different sources, and then store and manage the historical and real-time data generated by the freight platform in a hierarchical manner. It then builds spatiotemporal and attribute indexes for the three types of queries mainly involved in steel transportation to achieve an efficient query of multi-source logistic data. Finally, the experimental results based on real steel logistic data show that the proposed scheme is superior to traditional relational database methods in terms of data writing, storage, and querying, and can effectively support the storage and querying of massive logistic data.
Key words: steel logistic data; HBase; spatio-temporal index; query
Tao ZOU , Rongtao QIAN , Jiali MAO . Indexing and query technology for steel logistic data[J]. Journal of East China Normal University(Natural Science), 2022 , 2022(5) : 195 -207 . DOI: 10.3969/j.issn.1000-5641.2022.05.016
1 | XIE D, LI F F, YAO B, et al. Simba: Efficient in-memory spatial analytics [C]// Proceedings of the 2016 International Conference on Management of Data. 2016: 1071-1085. |
2 | SHANG Z Y, LI G L, BAO Z F. DITA: Distributed in-memory trajectory analytics [C]// Proceedings of the 2018 International Conference on Management of Data. 2018: 725-740. |
3 | LI H D, LI G L, LIU J Y, et al. Ratel: Interactive analytics for large scale trajectories [C]// Proceedings of the 2019 International Conference on Management of Data. 2019: 1949-1952. |
4 | FANG Z Q, CHEN L, GAO Y J, et al. Dragoon: A hybrid and efficient big trajectory management system for offline and online analytics. The Very Large Data Bases Journal, 2021, 30, 287- 310. |
5 | NISHIMURA S, DAS S, AGRAWAL D, et al. MD-HBase: A scalable multi-dimensional data infrastructure for location aware services [C]// 2011 IEEE 12th International Conference on Mobile Data Management. IEEE, 2011: 7-16. |
6 | HUGHES J N, ANNEX A, EICHELBERGER C N, et al. GeoMesa: A distributed architecture for spatio-temporal fusion [C]// Geospatial Informatics, Fusion, and Motion Video Analytics V. 2015: 9473. |
7 | GUAN X F, BO C, LI Z Q, et al. ST-Hash: An efficient spatiotemporal index for massive trajectory data in a NoSQL database[C]// 2017 25th International Conference on Geoinformatics. 2017: 17334139. |
8 | LI R Y, HE H J, WANG R B, et al. JUST: JD urban spatio-temporal data engine [C]// Proceedings of the 2020 IEEE 36th International Conference on Data Engineering. IEEE, 2020: 1558-1569. |
9 | CHEN X Y, ZHANG C, GE B, et al. Efficient historical query in HBase for spatio-temporal decision support. International Journal of Computers Communications & Control, 2016, 11 (5): 613- 630. |
10 | XU J Q, BAO Z F, LU H. Continuous range queries over multi-attribute trajectories [C]// 2019 IEEE 35th International Conference on Data Engineering. IEEE, 2019: 1610-1613. |
11 | FENG B, ZHU Q, LIU M W, et al. An efficient graph-based spatio-temporal indexing method for task-oriented multi-modal scene data organization. International Society for Photogrammetry and Remote Sensing International Journal of Geo-Information, 2018, 7 (9): 371. |
/
〈 |
|
〉 |