Supercharge Your Innovation With Domain-Expert AI Agents!

Knowledge distillation-based text classification method and system

A text classification and knowledge technology, applied in neural learning methods, character and pattern recognition, instruments, etc., to achieve the effect of ensuring the accuracy of results, improving accuracy, and facilitating deployment and use

Pending Publication Date: 2022-07-29
SHANDONG LANGCHAO YUNTOU INFORMATION TECH CO LTD
View PDF0 Cites 5 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] The technical task of the present invention is to provide a text classification method and system based on knowledge distillation to solve the problem of how to use knowledge distillation and take advantage of the accuracy of complex models to obtain lightweight models with comparable accuracy

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Knowledge distillation-based text classification method and system
  • Knowledge distillation-based text classification method and system
  • Knowledge distillation-based text classification method and system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0089] as attached Figure 5 As shown, this embodiment provides a text classification method based on knowledge distillation, and the method is as follows:

[0090] S1. Obtain unsupervised corpus (data 1) and perform data preprocessing on the unsupervised corpus;

[0091] S2. Obtain a teacher language model (model T) based on large-scale unsupervised corpus training;

[0092] S3. Use the supervised training corpus for the specific classification task to train the teacher language model (model T) on the classification task through fine-tuning, and obtain a trained teacher language model (model T);

[0093] S4. Construct a student model (model S) according to the specific classification task and the trained teacher language model (model T);

[0094] S5. According to the intermediate layer output and final output of the teacher language model (model T), construct a loss function, train the student model (model S), and obtain the final student model (model S);

[0095] S6. Use ...

Embodiment 2

[0147] This embodiment provides a text classification system based on knowledge distillation, and the system includes:

[0148] The first acquisition module is used to acquire the unsupervised corpus (data 1) and perform data preprocessing on the unsupervised corpus;

[0149] The first training module is used to obtain a teacher language model (model T) based on large-scale unsupervised corpus training;

[0150] The second training module is used to use the supervised training corpus (data 2) for the specific classification task to train the teacher language model (model T) on the classification task through fine-tuning, and obtain a trained teacher language model (model T);

[0151] The construction module is used to construct the student model (model S) according to the specific classification task and the trained teacher language model (model T);

[0152] The second acquisition module is used to construct a loss function according to the intermediate layer output and final...

Embodiment 3

[0155] This embodiment also provides an electronic device, including: a memory and a processor;

[0156] Wherein, the memory stores computer execution instructions;

[0157] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the text classification method based on knowledge distillation in any embodiment of the present invention.

[0158] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs), or other programmable logic devices, Discrete gate or transistor logic devices, discrete hardware components, etc. The processor may be a microprocessor or the processor may be any conventional processor or the like.

[0159] The memory can be used to store computer programs and / or modules, and the processor implements various functions of the electronic device...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a text classification method and system based on knowledge distillation, belongs to the technical field of natural language processing, and aims to solve the technical problem of how to obtain a lightweight model with equivalent precision by utilizing knowledge distillation and by virtue of the precision advantage of a complex model. According to the technical scheme, the method specifically comprises the following steps that unsupervised corpora are obtained, and data preprocessing is conducted on the unsupervised corpora; obtaining a teacher language model based on large-scale unsupervised corpus training; performing classification task training on the teacher language model through fining-tuning by using supervised training corpora aiming at a specific classification task to obtain a trained teacher language model; constructing a student model according to the specific classification task and the trained teacher language model; according to the intermediate layer output and the final output of the teacher language model, constructing a loss function, training the student model, and obtaining a final student model; performing text classification prediction by using the final student model; and inputting new data to perform classification structure prediction.

Description

technical field [0001] The invention relates to the technical field of natural language processing, in particular to a text classification method and system based on knowledge distillation. Background technique [0002] In the field of natural language processing (NLP), text classification tasks have a wide range of applications, such as: spam filtering, news classification, sentiment analysis, etc. [0003] Since the advent of BERT, the use of pre-trained language models in downstream tasks through fine-tuning has become more and more a paradigm in the field of natural language processing, achieving excellent results in natural language tasks. But the price of this effect is that the commonly used pre-trained language models, such as BERT, GPT, etc., are trained on the basis of a large amount of corpus through complex network structures, and they are not suitable for hardware in terms of parameter storage and inference speed. Computing resources bring great demands. In sc...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06K9/62G06N3/04G06N3/08G06F40/284
CPCG06N3/088G06F40/284G06N3/042G06F18/2415
Inventor 张烈帅赵振修魏静如解一豪
Owner SHANDONG LANGCHAO YUNTOU INFORMATION TECH CO LTD
Features
  • R&D
  • Intellectual Property
  • Life Sciences
  • Materials
  • Tech Scout
Why Patsnap Eureka
  • Unparalleled Data Quality
  • Higher Quality Content
  • 60% Fewer Hallucinations
Social media
Patsnap Eureka Blog
Learn More

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2025 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com