Method and system for cache data decoupling persistence based on Ignite grid

Abstract

The invention discloses a method and a system for cache data decoupling persistence based on Ignite grids. The system comprises a database and a plurality of servers connected with the database. Eachserver comprises a plurality of service nodes, a plurality of file queue catalogs and client nodes. Each service node, each file queue catalog, and each client node are arranged in a one-to-one corresponding manner. Each service node is provided with a plurality of caches. Each cache establishes a corresponding producer. The file queue catalog establishes a queue catalog corresponding to each producer. The cache is used to send updated data to the corresponding producer. The producer is used to update data, package the data, and write the packaged data to the data file in the corresponding queue catalog. The method and the system decouple Ignite cache and the database, and improve cache processing speed and cluster throughput capacity, prevent influence of the database on clusters, and improve stability of the clusters.

Description

technical field [0001] The invention relates to the field of big data processing, in particular to a method and system for decoupling and persisting data based on Ignite grid cache. Background technique [0002] The Apache Ignite memory array organization framework is a high-performance, integrated and distributed memory computing and transaction platform for large-scale data set processing, which has higher performance than traditional disk-based or flash-based technologies. Provides high-performance, distributed in-memory data organization and management functions between applications and different data sources. [0003] In Apache Ignite's data grid, the write through function means that when the cache is updated, it will be automatically persisted, and the data will be persisted to the database. Currently, the write-through method of the Ignite data grid is a write-through method in which the cache and the database are tightly coupled. Even in the write-behind function ...

AI Technical Summary

Problems solved by technology

[0004] In order to solve the problem that the existing cache write-through mode in the data grid of Apache Ignite affects the efficiency and performance of the Ignite cluster due to the database and causes the instability of the Ignite cluster, it provides a method based on Ignite grid cache data decoupling persistence and The system will introduce a local message system similar to the producer-consumer mode between the cache and the database, that is, the producer first writes the cache update data to the local file, and then the consumer writes the data in the local file to the database

Images