Method and system for decoupling and persisting data based on ignite grid cache

Abstract

The present invention discloses a method and system based on Ignite grid cache data decoupling and persistence. The system includes a database and multiple servers connected to the database. Each server includes multiple service nodes, multiple file queue directories, and client nodes. Each service node, file queue directory, and client node are set in one-to-one correspondence. Each service node has multiple caches. Each cache creates a corresponding producer. The file queue directory corresponds to each producer. There is a queue directory corresponding to it, and the cache is used to send its updated data to the corresponding producer, and the producer is used for data update, packaging, and writing the packaged data to the data file in the corresponding queue directory. The invention decouples the Ignite cache and the database, improves the speed of cache processing and the throughput of the cluster, avoids the influence of the database on the cluster, and improves the stability of the cluster.

Description

technical field [0001] The invention relates to the field of big data processing, in particular to a method and system for decoupling and persisting data based on Ignite grid cache. Background technique [0002] The Apache Ignite memory array organization framework is a high-performance, integrated and distributed memory computing and transaction platform for large-scale data set processing, which has higher performance than traditional disk-based or flash-based technologies. Provides high-performance, distributed in-memory data organization and management functions between applications and different data sources. [0003] In Apache Ignite's data grid, the write through function means that when the cache is updated, it will be automatically persisted, and the data will be persisted to the database. Currently, the write-through method of the Ignite data grid is a write-through method in which the cache and the database are tightly coupled. Even in the write-behind function ...

AI Technical Summary

Problems solved by technology

[0004] In order to solve the problem that the existing cache write-through mode in the data grid of Apache Ignite affects the efficiency and performance of the Ignite cluster due to the database and causes the instability of the Ignite cluster, it provides a method based on Ignite grid cache data decoupling persistence and The system will introduce a local message system similar to the producer-consumer mode between the cache and the database, that is, the producer first writes the cache update data to the local file, and then the consumer writes the data in the local file to the database

Images