Message processing method, device and system based on memory mapping and storage medium

A memory mapping and message processing technology, applied in the field of data processing, can solve problems such as large jitter in load balancing consumption, unavailability of cluster nodes, and many storage partitions, so as to improve message interaction processing, reduce message processing performance degradation, and improve processing efficiency. Effect

Pending Publication Date: 2020-09-08
北京青云科技股份有限公司
View PDF0 Cites 1 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] However, the existing message queue storage methods such as Kafka have some problems in processing data storage. There are too many storage partitions, resulting in a rapid decline in data storage efficiency as the number of storage partitions increases. A cluster node is unavailable during the process

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Message processing method, device and system based on memory mapping and storage medium
  • Message processing method, device and system based on memory mapping and storage medium
  • Message processing method, device and system based on memory mapping and storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0036] figure 1 It is a flow chart of a message processing method based on memory mapping provided by Embodiment 1 of the present invention. This embodiment of the present invention is applicable to the situation of fast processing of messages in a distributed cluster. The method can be implemented by a message processing device based on memory mapping Execution, the device can be implemented in hardware and / or software, and is generally applied to the master node in a distributed cluster, see figure 1 , the method provided by the embodiment of the present invention generally includes the following steps:

[0037] Step 101. Store the message in a storage master file, where the storage master file is unique.

[0038] Wherein, the message is the data to be stored in the distributed cluster, which can be received by the master node in the distributed cluster. The master node is the execution node determined according to the Raft or LibraBFT consensus protocol in the distributed...

Embodiment 2

[0047] Embodiment two figure 2 It is a flow chart of a message processing method based on memory mapping provided by Embodiment 2 of the present invention. The embodiment of the present invention is based on the above-mentioned embodiments. The read and write operations of the storage master file are accelerated by establishing a partition index, and based on Preset length restores storage unit from disk to storage master file, see figure 2 , the message processing method based on memory mapping in the embodiment of the present invention includes the following steps:

[0048] Step 201: Store the message in the storage master file according to the received storage request.

[0049] Wherein, the storage request is a request for storing messages, and specifically, the Store thread may independently accept the storage request.

[0050] Specifically, the Store thread monitors the storage request. When the storage request is received, the message is stored in the storage master fi...

Embodiment 3

[0082] Figure 5 It is a flow chart of a message processing method based on memory mapping provided by Embodiment 3 of the present invention. This embodiment of the present invention is applicable to the case of fast processing of messages in a distributed cluster. The method can be implemented by a message processing device based on memory mapping Execution, the device can be implemented in hardware and / or software, and is generally applied to slave nodes in distributed clusters, see Figure 5 , the method provided by the embodiment of the present invention generally includes the following steps:

[0083] Step 301. Obtain the message sent by the master node, and store the message in a storage master file.

[0084] Among them, the master node is the execution node of the distributed cluster, the master node and the current slave node are located in the same group, and the master node is the execution node determined according to the Raft or LibraBFT consensus protocol in the ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a message processing method, device and system based on memory mapping and a storage medium. The method comprises the steps of: storing a message in a storage main file, wherein the storage main file is unique; brushing the message into a disk based on memory mapping and sending the message to a slave node; and generating response information according to the message in thememory mapping and feeding back the response information. According to the embodiment of the invention, based on message processing of memory mapping, message processing performance reduction causedby file partitioning is reduced, message interaction processing between the file and the disk is improved through memory mapping, and the processing efficiency can be improved.

Description

technical field [0001] The embodiments of the present invention relate to the technical field of data processing, and in particular, to a message processing method, device, system and storage medium based on memory mapping. Background technique [0002] With the advancement of Internet technology, data is gradually spreading in all aspects of life, so how to store data quickly and safely has become the focus of current research. The data is stored based on the message queue, and the storage process conforms to the characteristics of sequential writing, supports batch operations, and makes data reading more efficient. The storage feature of the message queue is that the new data is hot data, and the old data is cold data. The data stored in the memory needs to be periodically refreshed to achieve persistent storage. [0003] However, the existing message queue storage methods such as Kafka have some problems in processing data storage. There are too many storage partitions, ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F9/54G06F12/08
CPCG06F12/08G06F9/546
Inventor 张恒
Owner 北京青云科技股份有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap