Method and device for data loading

A data loading and data technology, applied in the field of network management, to achieve the effect of improving failover capability

Active Publication Date: 2019-05-31
HANGZHOU DT DREAM TECH
View PDF5 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] However, before the failed processing node fails, some data may have been extracted from the source database and loaded into the target database

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method and device for data loading
  • Method and device for data loading
  • Method and device for data loading

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0035] In view of the problems existing in the prior art, an embodiment of the present invention proposes a data loading method, which can be applied to an ETL scheduling cluster system including multiple processing nodes (eg, processing servers), and each processing node is used to complete the Data extraction, transformation, loading and other processes. by figure 1 It is a schematic diagram of an application scenario of the embodiment of the present invention. The ETL scheduling cluster system may include a processing node 1 , a processing node 2 , a processing node 3 , and a processing node 4 . exist figure 1 , the source database may be an RDBMS database (such as Oracle, MySQL, etc.), and the target database may be a Hadoop database, or, the source database may be a Hadoop database, and the target database may be an RDBMS database.

[0036] When a user sends an ETL request to the ETL scheduling cluster system, the ETL scheduling cluster system can create a task for the ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The present invention provides a data loading method and device, the method comprising: a processing node obtains a subtask to be processed, and determines the data to be loaded corresponding to the subtask; the processing node extracts the subtask from a source database corresponding data to be loaded; the processing node loads the extracted data to be loaded into a temporary table of the target database; after all the data to be loaded corresponding to the subtasks are loaded into the temporary table by the processing node, the All data to be loaded in the temporary table is copied to the target table of the target database. Through the technical solution of the present invention, duplicate data will not be loaded into the target table of the target database, which solves the problem of duplication of data in the target table when the ETL scheduling cluster system recovers from a fault, and improves the Failover of the ETL scheduling cluster system ability.

Description

technical field [0001] The present invention relates to the technical field of network management, and in particular, to a data loading method and device. Background technique [0002] With the advent of the era of big data, there are more and more data exchange requirements between different databases, and ETL (Extract Transform Load) is used to extract data from the source database and load the extracted data into the target database middle. For example, extract data from an RDBMS (Relational Database Management System) database (eg, Oracle, MySQL, etc.), and load the extracted data into a Hadoop (distributed) database. Alternatively, extract data from a Hadoop database and load the extracted data into an RDBMS database. [0003] In the era of big data, in the face of a large number of data extraction and data loading work, a single processing node can no longer meet the needs of users. Usually, multiple processing nodes are required to complete a large amount of data ex...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Patents(China)
IPC IPC(8): G06F16/25
CPCG06F16/254
Inventor 李岩
Owner HANGZHOU DT DREAM TECH
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap