Multi-sensory vehicle-mounted interaction method and system based on multi-modal analysis

Abstract

The invention provides a multi-sensory vehicle-mounted interaction method and system based on multi-modal analysis. The method comprises the following steps: capturing and storing multi-modal source data of a driver in real time through a vehicle-mounted camera, a microphone and a temperature and humidity sensor; analyzing the multi-modal source data of the driver in real time, and extracting feature vectors in the multi-modal source data; splicing the plurality of feature vectors and converting the plurality of feature vectors to the same dimension to obtain processed data; inputting the processed data into a BP neural network for training, and judging the real-time state of the driver; and according to the real-time state of the driver, a corresponding interaction service is actively provided for the driver. By adopting a multi-modal information processing means, the real-time state of a driver can be comprehensively judged, active interaction service is provided, and the accuracy of understanding the emotion and intention of a user is improved; and self-optimization of the vehicle-mounted interaction system is realized through a BP neural network training model.

Description

technical field [0001] The invention relates to the technical field of human-computer interaction, and in particular, to a multi-sensory vehicle-mounted interaction method and system based on multimodal analysis. Background technique [0002] In the stage of human-machine co-driving, multi-sensory channel fusion and interaction will establish a new interactive experience between humans and machines. By collecting and analyzing human expressions, voice, temperature and humidity, etc., the user's emotional state and intention can be comprehensively judged, and passive interaction can be transformed. for active interaction. At the same time, communicating with users in interactive ways of sight, voice, smell, and touch can significantly improve the driving experience. [0003] The Chinese patent application number CN201910764559.4 discloses "an adaptive multi-sensory sleep assistance system based on artificial intelligence", which can judge the sleep state and environmental in...

AI Technical Summary

Problems solved by technology

This technology does not consider the factors of the human body state, and cannot fully and accurately reflect the characteristics of the human body state

[0004] In the Chinese patent with application number CN201480019946.0, a "control method for vehicle functional components used to generate different multi-sensory environments in vehicles" is disclosed. This technology can control the sound, lighting, and fragrance in the car, but only Control each component according to the preset scene, without real-time active interaction features and personalized adjustment functions

Images