Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Method and apparatus for processing deep learning task in big-data cluster

A deep learning and big data technology, applied in the computer field, can solve problems such as inability to execute, achieve the effect of large data storage and improve execution efficiency

Inactive Publication Date: 2017-03-22
BEIJING QIHOO TECH CO LTD +1
View PDF4 Cites 29 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] It can be seen that deep learning needs to use a large amount of data for calculation, but nowadays big data is often stored on the file system of big data clusters, and the existing deep learning tasks cannot be well executed in big data clusters

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method and apparatus for processing deep learning task in big-data cluster
  • Method and apparatus for processing deep learning task in big-data cluster

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0047] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided for more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0048] figure 1 A schematic flow diagram showing a method for processing deep learning tasks in a large data cluster according to an embodiment of the present invention, as shown in figure 1 As shown, the method includes:

[0049] Step S110, receiving a deep learning task.

[0050] Step S120, allocate at least one node capable of performing the deep learning task from the nodes of the big data cluster.

[0051] Step S130, callin...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a method and apparatus for processing a deep learning task in a big-data cluster. The method comprises the following steps: receiving the deep learning task; distributing at least one node capable of executing the deep learning task from nodes of the big-data cluster; invoking a deep learning database interface, and starting a subtask corresponding to the deep learning task on each distributed node; obtaining data used for the deep learning task from a file system of the big-data cluster; and pushing the obtained data used for the deep learning task to the corresponding subtask for execution, and storing an execution result returned by the subtask at a specific position in the file system of the big-data cluster. According to the technical scheme, the deep learning task can be effectively processed in the big-data cluster, the advantages of parallel execution of big-data cluster tasks and large data storage quantity are utilized, deep learning can be organically combined with big-data calculation, and the execution efficiency of the deep learning task is greatly improved.

Description

technical field [0001] The invention relates to the field of computer technology, in particular to a method and device for processing deep learning tasks in a large data cluster. Background technique [0002] Since "Alpha Dog" defeated Li Shishi, artificial intelligence has received widespread attention. In fact, research on artificial intelligence has been going on for a long time. The concept of deep learning originated from the study of artificial neural networks. Before the emergence of deep learning technology, most of the methods adopted were human beings who spent a lot of energy writing programs, inputting them into machines and then performing predetermined functions. However, deep learning technology allows humans to only write programs that allow machines to learn deeply, and the machines can be realized in a huge In the process of data accumulation, intelligent operation is realized through learning, and its level can be continuously improved in the process of i...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06N99/00
CPCG06N20/00
Inventor 李远策陈永强贾润莹欧阳文
Owner BEIJING QIHOO TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com