Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Data processing method and device, equipment and storage medium

A technology for processing data and storage media, applied in the field of data processing, which can solve the problems of low calculation, unsolvable association between incremental data and full data, and inability to meet timeliness, so as to reduce memory pressure and save network transmission time.

Active Publication Date: 2017-11-24
RUN TECH CO LTD BEIJING
View PDF8 Cites 4 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

Then perform various calculations on the data in the time window to generate result data. Spark streaming provides an associated join mechanism that is also calculated based on the data in the event window, but it cannot be solved based on the association between incremental data and full data.
[0003] At present, the general solution in the industry is to generally rely on external storage, or redis or other traditional databases. The use of nosql databases such as redis generally affects the calculation of low latency and low processing efficiency, while the use of traditional databases cannot meet the requirements of large data volumes. Timeliness, all of the above methods require additional components or equipment, as well as maintenance of related equipment and components

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data processing method and device, equipment and storage medium
  • Data processing method and device, equipment and storage medium
  • Data processing method and device, equipment and storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0057] figure 1 It is a flow chart of a method for processing data provided by Embodiment 1 of the present invention. This embodiment is applicable to the situation of processing data by configuring a client. The method can be executed by a device for processing data. The device It can be implemented by means of software and / or hardware, and is generally integrated in the client.

[0058] The method of Embodiment 1 of the present invention specifically includes:

[0059] Step 110, configure association rules, and upload rule files to preset storage media;

[0060] Wherein, the partitioning of the cache data set according to partition rules includes:

[0061] judging whether the cache data set is cache data according to the association rules;

[0062] If the cached data set is cached data, adding the cached data set to the full data set.

[0063] Optionally, after adding the cached data set to the full data set, it also includes:

[0064] Determine whether there is duplicate...

Embodiment 2

[0087] Such as image 3 As shown, the device includes: a configuration module 310 , a partition module 320 and an association module 330 .

[0088] Configuration module 310, configured to configure association rules, and upload rule files to a preset storage medium;

[0089] A partition module 320, configured to obtain a cached data set, and partition the cached data set according to a partition rule;

[0090] An association module 330, configured to acquire an associated data set, and associate the associated data set according to the association rule;

[0091] Wherein, the partition module 310 is specifically used for:

[0092] judging whether the cache data set is cache data according to the association rules;

[0093] If the cached data set is cached data, adding the cached data set to the full data set;

[0094] Wherein, the device also includes:

[0095] The update module is used to determine whether there is duplicate data when adding the cached data set to the ful...

Embodiment 3

[0109] refer to Figure 4 As shown, the device includes a processor 401, a memory 402, an input device 403, and an output device 404; the number of processors 401 in the device may be one or more, Figure 4 Take a processor 401 as an example; the processor 401, memory 402, input device 403, and output device 404 in the device can be connected by bus or other methods, Figure 4 Take connection via bus as an example.

[0110] The memory 402, as a computer-readable storage medium, can be used to store software programs, computer-executable programs and modules, such as program instructions / modules corresponding to the requested data processing method in the embodiment of the present invention (for example, the requested data processing device The client request acquisition module 401, the key route information acquisition module 402, and the route forwarding information construction module 403) in the system. The processor 401 executes various functional applications and data p...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

An embodiment of the invention discloses a data processing method and device, equipment and a storage medium. Association rules are configured, and a rule file is uploaded to a preset storage medium; a buffered dataset is acquired and is partitioned according to partitioning rules; an association dataset is acquired and is associated according to the association rules. Therefore, stream-type real-time associational computing power is achieved.

Description

technical field [0001] Embodiments of the present invention relate to data processing technologies, and in particular, to a data processing method, device, device, and storage medium. Background technique [0002] Spark streaming is a streaming data processing engine, which provides a micro batch processing mechanism to process data. Then perform various calculations on the data in the time window to generate result data. Spark streaming provides an associated join mechanism that is also calculated based on the data in the event window, but it cannot be solved based on the association between incremental data and full data. [0003] At present, the general solution in the industry is to generally rely on external storage, or redis or other traditional databases. The use of nosql databases such as redis generally affects the calculation of low latency and low processing efficiency, while the use of traditional databases cannot meet the requirements of large data volumes. Tim...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F17/30
CPCG06F16/23G06F16/24539G06F16/24564G06F16/27
Inventor 谢永恒高魁火一莽万月亮
Owner RUN TECH CO LTD BEIJING
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com