Small file combination method for spatio-temporal data in smart city

Abstract

The invention discloses a small file combination method for spatio-temporal data in a smart city. A small file is combined to serve as a microscopic data layout mechanism, system I / O performance of the system can be effectively improved, and a user access time delay is reduced. The characteristics of the spatio-temporal data are analyzed, historical user access information is subjected to parametric representation and spatio-temporal attribute extraction, the access information is subjected to hierarchy clustering by an AGNES (AGglomerative NESting) algorithm in a spatio-temporal attribute domain, a clustering result is subjected to weighted calculation based on access density, and a relevant spatio-temporal range is found. Finally, the spatio-temporal range is used for guiding the combination of small files. An experiment result indicates that the algorithm is simple and efficient, and the access efficiency of the spatio-temporal data small file in the system is greatly improved.

Description

technical field [0001] The invention relates to the research field of merging strategies for spatio-temporal data small files in smart cities. Background technique [0002] In a smart city based on the Internet of Things and cloud computing, ubiquitous sensors always generate sensing data containing three inherent attributes of time, space, and type. These data are small in size (usually tens to hundreds of KB) , a wide variety, a large number, high redundancy and dynamic growth over time, which are typical small spatio-temporal data files. [0003] The current mainstream distributed file system focuses on large files in terms of metadata management, data layout, stripe design, cache management and other implementation strategies. At present, the more common distributed file systems include Google GFS, Hadoop HDFS, PVFS, and Luster. These file systems all adopt a centralized data management mechanism with a master-slave structure, and store metadata (data describing data, ...

AI Technical Summary

Problems solved by technology

[0012] However, the existing small file merging strategies all use the characteristics of the file itself (in the above, the reading time of small files, user access characteristics (mainly referring to the number of user visits, stay time, etc.), are directly related to specific files, and belong to the same category. The characteristics of the file itself), can only merge files that have been accessed in history, or simply merge temporally adjacent files, and does not use user access information at a deep level

Images