Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

System and method for real-time graph data processing based on BSP (Board Support Package) model

A data processing system and data processing technology, applied in the direction of electrical digital data processing, special data processing applications, instruments, etc., can solve the problems of not meeting real-time requirements, unreasonable storage structure, and low efficiency of statistical query, so as to speed up access Performance, access efficiency, and speed-up effects

Active Publication Date: 2013-10-02
INST OF INFORMATION ENG CHINESE ACAD OF SCI
View PDF2 Cites 33 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0009] The technical problem to be solved by the present invention is to provide a real-time graph data processing system and method based on the BSP model, which is used to solve the problems of unreasonable storage structure, low statistical query efficiency, and failure to meet real-time requirements in the existing graph data processing technology and low efficiency

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • System and method for real-time graph data processing based on BSP (Board Support Package) model
  • System and method for real-time graph data processing based on BSP (Board Support Package) model
  • System and method for real-time graph data processing based on BSP (Board Support Package) model

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0067] The principles and features of the present invention are described below in conjunction with the accompanying drawings, and the examples given are only used to explain the present invention, and are not intended to limit the scope of the present invention.

[0068] Existing graph data processing systems generally include three levels:

[0069] The first level is the data storage layer, which is mainly responsible for storing graph data, and at the same time provides an efficient concurrent access interface, providing powerful storage support for graph data processing.

[0070] The second layer is the graph data statistics query layer. This layer is mainly responsible for responding to user query statistics requests. These jobs are characterized by only accessing the graph data once, but the total amount of data accessed is directly related to the size of the job. Therefore, when the cluster system runs multiple jobs, it involves the load balancing problem of the entire ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention relates to a system and method for real-time graph data processing based on a BSP (Board Support Package) model. The system comprises a data storage unit, a graph data inquiring and counting unit and a graph data analyzing and processing unit, wherein the data storage unit is used for preprocessing graph data, storing the graph data according to a memory storage-distributed memory storage-distributed file system three-layer storage structure and generating an operation based on the graph data, the graph data inquiring and counting unit is used for inquiring and counting the graph data, decomposing the operation generated by the data storage unit into a plurality of tasks, distributing the tasks to corresponding calculating nodes in an equalized manner, carrying out statistics on all task calculation results, combining all the task calculation results and returning the combined task calculation results serving as a final result to users, and the graph data analyzing and processing unit is used for enabling all the calculating nodes to execute the decomposed tasks through iterative calculation, realizing the synchronization of iterative calculation of each time through message passing and outputting the task calculation results. The method is used for realizing the real-time graph data processing based on the system. The system and the method have the advantages that the access efficiency is high, the cluster load balancing is kept, the execution efficiency of the BSP model is accelerated, and the like.

Description

technical field [0001] The invention relates to the field of large-scale graph data processing, in particular to a real-time graph data processing system and method based on a BSP model. Background technique [0002] In recent years, with the rapid development and popularization of the SNS (Social Network Service) platform, graph data, which is the data representation of the platform, is also in a state of information expansion. In order to express more information, the expression form of graph data is becoming more and more complex, and the amount of data is also increasing. [0003] At the same time, the number of data items in graph data will be larger, and the relationship between data and data will be more complex, and data does not exist in isolation. Therefore, the storage of graph data will face greater challenges. In addition, how to process such large-scale graph data to achieve the purpose of mining hidden information is also a problem that graph data processing...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30
Inventor 周薇韩冀中戴娇张章
Owner INST OF INFORMATION ENG CHINESE ACAD OF SCI
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com