Lip language recognition method combining graph neural network and multi-feature fusion

Abstract

The invention discloses a lip language recognition method combining a graph neural network and multi-feature fusion. The method comprises the following steps: firstly, extracting and constructing a face change sequence, marking face feature points, correcting a lip deflection angle, performing pre-processing through a trained lip semantic segmentation network, training a lip language recognition network through a graph structure of a single-frame feature point relationship and a graph structure of an adjacent-frame feature point relationship, and finally, generating a lip language recognition result through the trained lip language recognition network. CNN lip features and lip region feature points obtained after CNN extraction and feature fusion are performed on an identification network data set and a lip semantic segmentation network data set are subjected to the extraction and fusion by the GNN lip features obtained after GNN extraction and fusion and then input into BiGRU for identification, and the problems that time sequence feature extraction is difficult and lip feature extraction is affected by external factors are solved; the method effectively extracts the static features of the lip and the dynamic features of the lip change, and has the characteristics of high lip change feature extraction capability, high recognition result accuracy and the like.

Description

technical field [0001] The invention belongs to the field of artificial intelligence and deep learning, and specifically relates to a lip language recognition method combined with a graph neural network and multi-feature fusion. Background technique [0002] With the development of science and technology and the improvement of hardware manufacturing level, artificial intelligence technology based on deep learning has attracted more and more attention from researchers. The field of deep learning includes many subfields, such as machine vision, natural language processing, etc. Lip recognition, which combines machine vision and natural language processing, has gained more and more attention. Lip language recognition has a very wide range of application scenarios, such as liveness detection based on lip features, communication assistance for the hearing-impaired, and voice recovery from traffic cameras. [0003] There are many difficulties in lip language recognition. For exa...

AI Technical Summary

Problems solved by technology

However, most of these methods do not take into account the influence of lip deflection angle, light intensity, light angle and speaker identity information on the lip language recognition task, and the ability of traditional methods to extract information about lip sequence changes is also very limited. The accuracy of language recognition is in a rising bottleneck period

Images