Data processing method, client and distributed computing system

A technology of distributed computing and distributed system, applied in the field of client, distributed computing system, and data processing method, can solve the problems of prone to errors and many resources, so as to improve accuracy, reduce redis servers, and efficiently execute batches. effect of command

Active Publication Date: 2018-04-24
CHINANETCENT TECH
View PDF8 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] The technical problem to be solved by the present invention is to overcome the defect that the distributed computing system consumes too many system resources and is prone to errors during operation in the prior art, and provides a method that reduces the consumption of cpu, memory and other resources, and can compress data to save A data processing method, a client and a distributed computing system that ensure the accuracy of the data when transferring data while ensuring the accuracy of the data when executing commands in batches

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data processing method, client and distributed computing system
  • Data processing method, client and distributed computing system
  • Data processing method, client and distributed computing system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0057] see figure 1 , the present embodiment provides a distributed computing system 1, the distributed computing system includes 5 clients 11, 1 server 12, the server is a redis server, and the redis server includes several redis instances .

[0058] One of the five clients is a first client, and the first client includes an acquisition module 111, a processing module 112, a sending module 113, a setting module 114, a first judging module 115 and a second judging module Module 116.

[0059] The obtaining module is used to obtain data to be processed.

[0060] The first judging module is used for judging whether the data type of the key exists locally, if so, calling the processing module, otherwise calling the creating module.

[0061] The creation module is used to locally create the data type of the key value, and then call the processing module.

[0062] The processing module is used to process the data to be processed according to a preset rule to obtain a result valu...

Embodiment 2

[0085] see image 3 , this embodiment is basically the same as Embodiment 1, the only difference is:

[0086] Step 200 is executed after step 104 .

[0087] Step 200 , judging whether the time from the current moment to the last time to send data to be processed is greater than 5 seconds, if so, execute step 201 , otherwise return to step 100 .

[0088] Step 201. Send the data to be processed to obtain the result value to the redis server, and then execute step 204.

[0089] The data processing method, client, and distributed computing system of this embodiment can achieve the following effects: reduce the number of redis servers, because in the process of stream computing, the redis commands that will be generated locally will be simplified and compressed according to the data type, and will The commands that need to be processed centrally by the redis server are scattered on various stream computing machines, which can reduce the CPU, memory and other resources consumed by...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a data processing method, a client and a distributed computing system. The distributed system comprises a plurality of clients and a server; the data processing method comprises the following steps: acquiring data to be processed by the client; processing the data to be processed according to a preset rule so as to acquire a result value, wherein the preset rule is to locally calculate data with the same data type and / or same key value in the data to be processed; and sending a result value of the data to be processed after being locally calculated to the server. The data processing method, the client and the distributed computing system provided by the invention can reduce the operational load of the distributed computing system, and reduce resources such as a cpuand a memory and the like needing to be consumed; and can also compress the data to save the bandwidth when transmitting the data, and meanwhile can ensure the accuracy of the data during the batch execution of commands.

Description

technical field [0001] The embodiments of the present invention relate to the technical field of the Internet, and in particular to a data processing method, a client and a distributed computing system. Background technique [0002] The rapid development of the Internet industry has not only brought about explosive growth in data scale, but also made big data more and more streamlined. The traditional batch processing mode is difficult to meet the real-time computing requirements of streaming big data processing. Therefore, More efficient distributed computing systems are being used more and more widely. [0003] During the business processing of streaming computing, the system uses redis for data calculation and temporarily stores the data in redis (a high-performance key-value database, namely key-value database). According to business requirements, during the calculation process, a large amount of data will inevitably generate a large number of redis commands. In order ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): H04L29/08
CPCH04L67/10
Inventor 刘成彦李瑜婷刘华明
Owner CHINANETCENT TECH
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap