Method, node and system for loading data to distributed data warehouse

A distributed data and data node technology, applied in the database field, can solve the problems of not having strong computing power, data analysis and distribution, etc., and achieve the effect of high-speed loading

Active Publication Date: 2019-07-30
ALIBABA GRP HLDG LTD
View PDF5 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] On the cloud, a large amount of data is stored in object storage services such as OSS. Object storage services such as OSS do not have strong computing power and cannot be used for data analysis and distribution.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method, node and system for loading data to distributed data warehouse
  • Method, node and system for loading data to distributed data warehouse
  • Method, node and system for loading data to distributed data warehouse

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0044] In order to enable those skilled in the art to better understand the technical solutions in the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described The embodiments are only some of the embodiments of the present application, but not all of them. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without creative efforts shall fall within the scope of protection of this application.

[0045] figure 1It is a flowchart of a method for loading data into a distributed data warehouse according to an embodiment of the present application. figure 1 The method of is executed by a data device or a data node in a distributed data warehouse. It should be understood that the first data node in the embodiment of the present appli...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The embodiment of the invention discloses a method, device and system for loading data to a data warehouse. The method comprises the steps that a first data node receives a data loading request distributed by a management node, and the data loading request carries a reading parameter of the to-be-loaded data, a target data table of the to-be-loaded data in the data warehouse and an identifier of adistributed transaction generated by the management node for the to-be-loaded data; the first data node generates a local transaction of the distributed transaction; the first data node reads the to-be-loaded data according to the reading parameter of the to-be-loaded data; and the first data node pre-writes the data belonging to the first data node in the read to-be-loaded data into the target data table according to a data distribution rule.

Description

technical field [0001] The present application relates to the field of databases, in particular to a method, node and system for loading data into a distributed data warehouse. Background technique [0002] In a distributed data warehouse, data is usually organized in tables. A table will evenly distribute data to all data nodes according to certain rules. For example, according to the hash distribution rule, calculate the hash value of the distribution column of each row of data in the table, take the hash value according to the number of cluster nodes, and send the result to the corresponding data node according to the result of the modulus, so as to achieve data uniformity distributed. The data distribution rules of the data warehouse cause the data in the data files to be loaded by the data warehouse to be scattered to multiple data nodes in disorder. The data in a data file needs to be parsed and the data distribution rules applied, and sent to the corresponding data...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/22G06F16/27G06F16/28
CPCG06F16/22G06F16/278G06F16/283
Inventor 曾文旌张广舟
Owner ALIBABA GRP HLDG LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap