Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Method and device for obtaining pre-trained model

A pre-training and model technology, applied in the field of natural language processing and deep learning technology, to achieve good semantic understanding and generalizable representation

Active Publication Date: 2021-12-24
BEIJING BAIDU NETCOM SCI & TECH CO LTD
View PDF10 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

Obviously, most of the existing pre-training methods are only used in single-modal scenarios, such as only for images, or only for text

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method and device for obtaining pre-trained model
  • Method and device for obtaining pre-trained model
  • Method and device for obtaining pre-trained model

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0029] Exemplary embodiments of the present disclosure are described below in conjunction with the accompanying drawings, which include various details of the embodiments of the present disclosure to facilitate understanding, and they should be regarded as exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

[0030] Most of the existing pre-training models can only process single-mode data. For example, the BERT (BidirectionalEncoder Representation from Transformers, bidirectional encoding representation from the converter) model can only learn and process text data. The SimCLR (Simple Framework for Contrastive Learning of Visual Representations) model can only learn and pro...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The disclosure discloses a method and device for acquiring a pre-training model, and relates to natural language processing and deep learning technologies in the technical field of artificial intelligence. The specific implementation scheme is: obtain training data, the training data includes single-mode corpus and multi-mode corpus, wherein the multi-mode corpus includes a corpus pair composed of the first modal corpus and the second modal corpus; The training model performs multi-task training, and the multi-task includes: at least one cross-modal comparative learning task and at least one single-mode learning task; the pre-trained language model obtained in the present disclosure can include single-mode corpus, multi-mode corpus from different forms of corpus. Learning from modulus corpus enables the pre-trained language model to effectively process information of various modalities.

Description

technical field [0001] The present disclosure relates to the field of computer application technology, in particular to natural language processing and deep learning technology in the field of artificial intelligence technology. Background technique [0002] Large-scale pre-trained models have attracted extensive attention due to their strong generalization ability and efficient utilization of large-scale data. Obviously, most of the existing pre-training methods are only used in single-modal scenarios, such as only for images, or only for text. [0003] However, human beings perceive the world through many ways, such as sight, language, sound and so on. The combination of multiple modalities of information enables better information understanding, so an excellent artificial intelligence system should be able to effectively process information of various modalities. Contents of the invention [0004] The present disclosure provides a cross-modal pre-training model acquis...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): G06K9/62G06F40/42G06F40/295
CPCG06N3/08G06V10/803G06N3/045G06F18/214G06N3/04G06F16/9024G06F40/58G06F40/279G06F40/47G06F40/205G06N3/02
Inventor 牛国成李伟高参肖欣延吴华
Owner BEIJING BAIDU NETCOM SCI & TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com