Data storage and reading method and device

Abstract

The embodiment of the invention relates to the technical field of big data, and provides a data storage and reading method and device. The data storage method comprises the steps of obtaining to-be-stored data; determining a corresponding relationship between each storage unit corresponding to a preset storage format and each sub-data in the to-be-stored data; and storing the to-be-stored data into an out-of-heap memory of the Spark calculation engine by calling a preset data interface according to the corresponding relationship. The data reading method comprises the steps of determining a filtering condition of data required by a to-be-processed service when the to-be-processed service is received; by calling a preset data interface, reading target data meeting a filtering condition froman out-of-heap memory of the Spark calculation engine; and storing the target data into an in-heap memory of a Spark calculation engine, so as to read the target data from the in-heap memory when theto-be-processed service is processed. Compared with the prior art, by applying the scheme provided by the embodiment of the invention, the service processing efficiency of the JVM can be improved.

Description

technical field [0001] The present invention relates to the technical field of big data, in particular to a data storage and reading method and device. Background technique [0002] JVM (Java Virtual Machine, Java Virtual Machine) is a new network computing platform, which is realized by simulating various computer functions on a physical computer. Usually, the JVM uses the Spark computing engine to manage the data involved in the various businesses it processes. And, when the JVM is applied in a data processing system, the Spark computing engine used by the JVM can be included in the data processing system at the same time. [0003] Among them, the Spark computing engine is a fast and general computing engine that supports distributed data sets and can be used for large-scale data processing. It can complete text processing, machine learning, SQL (Structured Query Language, Structured Query Language) queries, etc. Various data processing tasks. [0004] In related techno...

AI Technical Summary

Problems solved by technology

[0006] However, since the JVM will suspend the processing of the current business during the GC process, when Spark frequently initiates GC, it will cause the JVM to frequently suspend the processing of the current business, thereby reducing the business processing efficiency of the JVM

Images