Streaming data increment processing method and device based on tensor chain decomposition

A streaming data and processing method technology, applied in the field of big data processing, can solve the problems of calculation result explosion, repeated calculation, and low processing efficiency, and achieve the effect of solving intermediate result explosion, fast and accurate update, and improving processing efficiency

Pending Publication Date: 2020-06-05
XIAN UNIV OF POSTS & TELECOMM
View PDF3 Cites 3 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0008] To sum up, there are two problems in the existing incremental processing technology: (1) Explosion of intermediate calculation results; (2) Repeated calculations. At the same time, research on incremental processing of high-dimensional big data mainly considers dynamic update The rapid processing of data, seldom consider the use of new data calculation results to quickly and accurately update the original processing results, and can not systematically describe the relationship between the new data and the existing calculation results, resulting in the existing big data processing Slower, less efficient processing

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Streaming data increment processing method and device based on tensor chain decomposition
  • Streaming data increment processing method and device based on tensor chain decomposition
  • Streaming data increment processing method and device based on tensor chain decomposition

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0057] See figure 1 , figure 1 It is a schematic flowchart of an incremental processing method for streaming data based on tensor chain decomposition provided by an embodiment of the present invention. The method includes the steps of:

[0058] S1. Construct a high-order unified tensor representation model for multi-source heterogeneous data.

[0059] S11. Collect multi-source heterogeneous data.

[0060] Specifically, multi-source heterogeneous data can come from industrial big data smart factory data; according to the data structure, multi-source heterogeneous data can be classified into structured data, semi-structured data and unstructured data.

[0061] See figure 2 , figure 2 It is a schematic diagram of a sub-tensorized representation of structured data provided by an embodiment of the present invention. Structured data refers to data that is logically expressed and realized by a two-dimensional table structure, strictly follows the data format and length specif...

Embodiment 2

[0116] See Figure 9 , Figure 9 It is a schematic structural diagram of an incremental streaming data processing device based on tensor chain decomposition provided by an embodiment of the present invention. The streaming data incremental processing device includes: a model building module, a first tensor chain decomposing module, a second tensor chain decomposing module and a tensor updating module.

[0117] Among them, the model building module is used to build a high-order unified tensor representation model for multi-source heterogeneous data. The first tensor chain decomposition module is used to represent the original data as an original tensor according to the high-order unified tensor representation model, and perform tensor chain decomposition on the original tensor to obtain the first tensor chain format. The second tensor chain decomposing module is used to represent the new data as new tensors according to the high-order unified tensor representation model, and ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention relates to a streaming data increment processing method and device based on tensor chain decomposition, and the method comprises the steps: constructing a high-order unified tensor representation model of multi-source heterogeneous data; representing the original data as an original tensor according to a high-order unified tensor representation model, and performing tensor chain decomposition on the original tensor to obtain a first tensor chain format; representing the newly added data as a newly added tensor according to the high-order unified tensor representation model, and performing tensor chain decomposition on the newly added tensor to obtain a second tensor chain format; and calculating a tensor chain decomposition result of the updated tensor according to the firsttensor chain format and the second tensor chain format. According to the processing method, the original processing result is quickly and accurately updated by utilizing the newly increased data calculation result, the internal relation between the newly increased data and the existing calculation result can be systematically described, meanwhile, the two problems of intermediate result explosionand repeated calculation of incremental processing are solved, and the processing efficiency of big data is improved.

Description

technical field [0001] The invention belongs to a big data processing method, and in particular relates to a stream data incremental processing method and device based on tensor chain decomposition. Background technique [0002] In a traditional industrial cloud architecture, all data from physical equipment is transferred to the cloud for storage and advanced analytics. Since cloud platforms have higher computing power compared to devices at the edge of the network, offloading computationally intensive tasks to core cloud computing platforms is an effective way for data processing. There are various sources of industrial big data, different data structures, different attributes and standards, including production cycle data, relational data from within the enterprise, and unstructured or semi-structured data such as video surveillance data and XML logs. . [0003] The premise and basis for efficient analysis of big data and mining the inherent laws contained in it is the ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/215G06F16/2455G06F16/2458G06F17/16
CPCG06F16/215G06F16/24568G06F16/2465G06F17/16
Inventor 陈彦萍夏虹靳晓东王忠民高聪吕宁
Owner XIAN UNIV OF POSTS & TELECOMM
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap