Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Big data quantity high performance processing implementing method based on parallel process of split mechanism

A high-performance processing and parallel processing technology, applied in electrical digital data processing, special data processing applications, instruments, etc., to achieve the effect of low investment, easy portability, and low performance dependence

Inactive Publication Date: 2009-08-19
LINKAGE SYST INTEGRATION
View PDF0 Cites 26 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] The purpose of the present invention is to propose a method of parallel processing based on a split mechanism to realize high-performance processing of a large amount of data, and a more complicated SQL statement needs to be executed once for each summary process

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Big data quantity high performance processing implementing method based on parallel process of split mechanism
  • Big data quantity high performance processing implementing method based on parallel process of split mechanism

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0022] By specifying a general interface, it can be conveniently called by the original system, so as to achieve the function of replacing the processing in the original database, and achieve seamless connection with the original system while significantly improving the execution efficiency of the system.

[0023] Several key technologies in the implementation process are as follows:

[0024] One-time reading: In order to achieve the purpose of reducing access to massive data source tables, it is necessary to read all the information required by subsequent summary tables at one time. By first listing the dimensions and index fields required by each summary table, and then taking the union method, a SQL statement for extracting massive source tables can be formulated. No matter how many aggregations there are, only one access to massive data sources is required, minimizing database pressure. In order to achieve the purpose of reducing access to massive data source tables, it i...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a method for realizing large data amount high-performance process, which is based on splitting mechanism parallel processing. A splitting rule is set for the mass data of telegraph tickets to equally split the mass data to be processed into a plurality of files; and the multi-thread and multi-CPU parallel process of a file processing system is adopted. The quick processing of the mass data is as follows: the parallel process of the file processing system is to simulate the database sql algorithm to carry out calculation; an SQL sentence for extracting a mass data source table is established through firstly spreading out the dimensionality and index field required by each collection table and secondly obtaining the unions and then the information required by all the following mass data collection tables is read over; the assembly storing is as follows: after the work for collecting the small files formed while equally splitting a plurality of files is finished, all the result files are combined into large files according to the target table types and then are loaded into the collection tables; and the work can be completed by the peculiar quick accessing instruction of the database.

Description

technical field [0001] The invention belongs to the category of application technology for data processing of massive databases of telecom operators, in particular to a method for realizing high-performance processing of large amount of data through parallel processing. Background technique [0002] Generally speaking, the business inventory data of telecom operators is often massive, especially the inventory data that needs to be aggregated and counted, and the number of records processed every day reaches tens of millions. The usual practice is to pass one or more complex SQL statements in the database and submit them to the database for completion. Such work takes up a lot of time and database resources. [0003] For example, for the daily inventory data generated every day, it is necessary to summarize the records of the daily inventory table according to the specified conditions, and then update them into the summary table. The update method is: if the summary table alr...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F17/30H04Q3/00
Inventor 沈小军庞海东赵懿敏李捷曹晓华
Owner LINKAGE SYST INTEGRATION
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com