Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Data loading method, device and storage medium, computing device, computing system

A computing device and data loading technology, applied in the Internet field, can solve problems such as input and output bottlenecks, large data loading delays, and limited throughput of network ports

Active Publication Date: 2021-09-14
ALIBABA CLOUD COMPUTING LTD
View PDF5 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

During the data loading process, due to the limited throughput of the network port, it will cause input and output (IO) bottlenecks, resulting in a large delay in data loading.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data loading method, device and storage medium, computing device, computing system
  • Data loading method, device and storage medium, computing device, computing system
  • Data loading method, device and storage medium, computing device, computing system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0033] In order to make the objects, technical solutions, and advantages of the present invention more clearly, the technical solutions in the embodiments of the present invention will be described in contemplation in the embodiments of the present invention, and will be described, and the embodiments described in the embodiments of the present invention will be described. It is a part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, those of ordinary skill in the art will belong to the scope of the present invention without all other embodiments obtained without creative labor.

[0034] Further, the step timing in each of the following method embodiments is only one example, not strictly limited.

[0035] With the widespread application of deep learning, distributed training methods have become the main training methods for responding to large-scale training data sets and neural network models. Bonded belo...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The present invention provides a data loading method, device, storage medium, computing device, and computing system. The method includes: a target computing device in a computing cluster stores data obtained from an external storage system into its memory space, and the memory The space is set to a mode that supports RDMA access, so as to obtain a distributed memory system composed of memory spaces in this mode in each computing device in the computing cluster. In response to the data loading request triggered by the target process in the target computing device, the target computing device sets the target storage space in it as a mode that supports RDMA access, and according to the storage address of the data to be loaded in the distributed memory system, it will be distributed from The data to be loaded obtained from the embedded memory system is loaded into the target storage space. Use the local memory resources of the computing cluster to build an RDMA-based distributed memory system to load data through RDMA, which improves the data loading speed.

Description

Technical field [0001] The present invention relates to the field of Internet technologies, and more particularly to a data loading method, device, and storage medium, a computing device, a computing system. Background technique [0002] In order to address the processing tasks of massive data, distributed system applications are born. In a distributed computing cluster composed of numerous computing devices, each computing device can read some data from an external storage system stored in massive data each time and perform data calculation processing. [0003] In addition, as the data set increases, massive data can be stored in storage systems such as Hadoop Distributed File System, Abbotes HDFS, Object Storage Service, Abbreviation OSS system. [0004] When each computing device in the above computing cluster is loaded, the remote storage system needs to be accessed through the network port to load the data stored. During the data loading process, since the throughput of the ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): G06F9/445G06F9/50G06F13/28G06K9/62
CPCG06F9/44521G06F9/5016G06F9/5027G06F13/28G06F18/214
Inventor 刘小丽董建波李豪曹政
Owner ALIBABA CLOUD COMPUTING LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com