Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Distributed data mining and statistics method based on data deduplication

A distributed data and statistical method technology, applied in the field of distributed data mining and statistics based on data deduplication, can solve the problems of high data complexity, slow data processing, and large data volume, so as to improve analysis efficiency and reduce Quantity level, the effect of improving efficiency

Pending Publication Date: 2020-04-24
成都安思科技有限公司
View PDF11 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

Large data volume, high data complexity, and slow data processing seriously affect the efficiency of data mining and statistics

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Distributed data mining and statistics method based on data deduplication
  • Distributed data mining and statistics method based on data deduplication

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0013] The present invention will be further described below in conjunction with accompanying drawing:

[0014] Such as figure 1 As shown, a kind of distributed data mining and statistical method based on data deduplication of the present invention comprises the following steps:

[0015] S1: the system obtains network traffic data packets;

[0016] S2: After the central server performs load balancing, the data packet is split and sent to the distributed server cluster;

[0017] S3: The distributed server cluster traverses all the data in the data packet, and according to the data aggregation and mining conditions configured by the user, each server node in the distributed server cluster judges whether the data is duplicated, if not, keep it, otherwise delete the redundant data;

[0018] S4: the central server merges the deduplicated data;

[0019] S5: The central server traverses the deduplicated data, and judges whether the data is repeated according to the data aggregatio...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a distributed data mining and statistics method based on data deduplication. The method comprises the steps of traversing all data of the data packet by the distributed servercluster, respectively judging whether the data are repeated by each server node in the distributed server cluster according to a data aggregation mining condition configured by a user, if not, retaining the data, and otherwise, deleting the repeated redundant data. According to the method, the original data are processed in a distributed manner, so that the quantity level of the data packets is greatly reduced, the difficulty of configuring complex mining conditions by a user is greatly reduced, the pertinence of data mining is enhanced, and the data mining efficiency is greatly improved.

Description

technical field [0001] The invention belongs to the technical field of network flow analysis, and in particular relates to a distributed data mining and statistical method based on data deduplication. Background technique [0002] With the development of computer technology and the Internet, the increase of broadband speed and the reduction of cost have made people's life and work more closely connected with the network, and the number of network data packets has increased geometrically. For the current network traffic data analysis, even experienced analysts can only analyze at most about 1,700,000 data packets a day. According to statistics, the data packets generated by a family in a week can reach 100 million levels, which is no longer an order of magnitude that can be easily analyzed. If you collect the traffic of an endpoint such as a company, school, government, etc., the data packets will be of an astronomical level. Large data volume, high data complexity, and slo...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F9/50H04L29/08
CPCG06F9/5083G06F9/505H04L67/1001
Inventor 邓金祥王炜代先勇谷峰曾海刚佘朝裕刘洋
Owner 成都安思科技有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com