Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

A method and system for comparing mass data

A mass data and data technology, applied in the field of data processing, can solve problems such as limited host I/O, database suspended animation, and comparison efficiency that cannot meet the requirements, and achieve the effect of improving comparison efficiency

Active Publication Date: 2020-10-02
北京思特奇信息技术股份有限公司
View PDF4 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0007] The traditional comparison method is to import large files into the database, then sort them in the database, then read the data from the database, and compare them in memory. This method puts too much pressure on the database. When there is a large amount of concurrent data, it is easy to cause the database to freeze, and due to the limited I / O of a host, the comparison efficiency cannot meet the requirements.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A method and system for comparing mass data
  • A method and system for comparing mass data
  • A method and system for comparing mass data

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0052] The principles and features of the present invention are described below in conjunction with the accompanying drawings, and the examples given are only used to explain the present invention, and are not intended to limit the scope of the present invention.

[0053] Such as figure 1 As shown, it is a schematic flowchart of a massive data comparison method provided by an embodiment of the present invention, and the method includes the following steps:

[0054] S101, acquiring massive data files to be compared;

[0055] S102, sorting the massive data files according to the pre-stored quick sort algorithm to obtain multiple sub-data files;

[0056] S103, performing data consistency comparison on multiple sub-data files.

[0057] The method for comparing massive data provided by the above-mentioned embodiment effectively solves the problem that the I / O of a single machine is currently limited and multiple The phenomenon of thread comparison can improve the efficiency of d...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention relates to a mass data comparison method and system. The method comprises the following steps that: obtaining a mass data file to be compared; according to a pre-stored quick sorting algorithm, carrying out sorting processing on the mass data file to obtain a plurality of sub data files; and carrying out data consistency comparison on the plurality of sub data files. By use of the mass data comparison method and system provided by the invention, the mass data file is divided into the plurality of sub data files, the plurality of sub data files are independently compared so as to effectively solve an existing phenomenon that a single machine has limited I / O (Input / Output) and can not carry out multi-thread comparison, and data comparison efficiency can be improved.

Description

technical field [0001] The invention relates to the field of data processing, in particular to a method and system for comparing massive data. Background technique [0002] Data comparison refers to the comparison of two or more different sets of data, and the detailed differences of different data can be quickly found and processed effectively. [0003] The traditional comparison method is mainly to arrange the data to be compared in a certain order. Each data record has two fields, the first field is the index field, and the second field is the attribute field. For example, the following is the The right two sets of data: [0004] [0005] Among them, the letter is the index field, and the number is the attribute field. [0006] Then, the comparison method is: obtain the data of the first row for comparison, and find that the index fields and attribute fields are the same, then continue to compare the next row, and find that the index fields of the second row are the ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): G06F16/2458G06F16/28
CPCG06F16/2474G06F16/285
Inventor 温小根
Owner 北京思特奇信息技术股份有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com