Language model-oriented double-unit search space structure search method

A search space and language model technology, applied in the field of artificial intelligence, can solve the problems of gradient disappearance, interruption of sequence semantic information, difficult back-propagation of gradients at the far end of the sequence, etc., to achieve the effect of increasing continuity and expanding search space.

Pending Publication Date: 2022-01-07
KUNMING UNIV OF SCI & TECH
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] The invention provides a language model-oriented double-unit search space structure search method to solve the problem that when the sequence is long, the gradient at the far end of the sequence is difficult to backpropagate to the current sequence, resulting in gradient disappearance, resulting in semantic information of the sequence interruption problem

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Language model-oriented double-unit search space structure search method
  • Language model-oriented double-unit search space structure search method
  • Language model-oriented double-unit search space structure search method

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0027] Embodiment 1: as Figure 1-Figure 5 As shown, the structure search method of language model-oriented dual-unit search space includes: firstly, constructing a dual-unit search space; secondly, searching on the PTB data set, and selecting the structure with the smallest loss on the verification set during the search process as the structure to be Select the unit structure; finally, enter the evaluation stage, conduct a short-term evaluation on the candidate unit structure obtained in the search stage on the language model task, and obtain the optimal unit structure

[0028] The specific implementation steps of the structure search method based on the dual-unit search space are as follows:

[0029] Step1. A dual-unit search space is proposed for the language model task, a search unit is set, and the final cyclic neural network is formed through the connection of units, and then the search space is constructed;

[0030] The dual-unit search space proposed in Step1 is to co...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention relates to a language model-oriented double-unit search space search method, and relates to the field of artificial intelligence. The search space of the existing search strategy is improved on the basis of the language model task, and the search space more suitable for the language model task is constructed. An information storage unit is added in a recurrent neural network unit to effectively store sequence front-end information, so that a search space is better matched with a language model task, the added unit can relieve the problem that long sequence dependence cannot be solved in a conventional recurrent neural network unit structure, and the continuity of the sequence semantic information is improved. Meanwhile, due to the increase of the units, the search space is directly expanded, and the probability of searching a better network structure is also improved.

Description

technical field [0001] The invention relates to a structure search method for a language model-oriented double-unit search space, and belongs to the technical field of artificial intelligence. Background technique [0002] The design of the search space is the first and extremely important step in the research of neural network structure search. The search space determines the upper and lower limits of the model performance. However, the conflicting relationship between the size of the search space and the search speed and hardware requirements makes its design a dilemma. On the one hand, a huge search space has great potential for network exploration, but requires extremely high hardware support and time consumption; very limited. Therefore, how to define an appropriate search space and achieve the best search effect has become an unsolved problem in the current structure search research. [0003] The research on neural network structure search is still in the preliminar...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06N3/04G06N3/08
CPCG06N3/082G06N3/084G06N3/044
Inventor 余正涛苗育华
Owner KUNMING UNIV OF SCI & TECH
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap