Mobile type multi-modal interaction method and device based on enhanced reality

Abstract

The invention discloses a mobile type multi-modal interaction method and device based on enhanced reality. The method comprises the following steps that: through an enhanced reality way, displaying ahuman-computer interaction interface, wherein an enhanced reality scene comprises interaction information, including a virtual object and the like; through the ways of gesture and voice, sending an interaction instruction by a user, comprehending different-modal semantic through a multi-modal fusion method, and carrying out fusion on the modal data of the gesture and the voice to generate a multi-modal fusion interaction instruction; and after a user interaction instruction acts, returning an acting result to an enhanced reality virtual scene, and carrying out information feedback through thechange of the scene. The device of the invention comprises a gesture sensor, a PC (Personal Computer), a microphone, optical transmission type enhanced reality display equipment and a WiFi (Wireless Fidelity) router. The invention provides the mobile type multi-modal interaction method and device based on the enhanced reality, a human-centered thought is embodied, the method and the device are natural and visual, learning load is lowered, and interaction efficiency is improved.

Description

technical field [0001] The invention relates to the technical field of human-computer interaction, in particular to a mobile multi-modal interaction method and device based on augmented reality. Background technique [0002] With the rapid development of computer technology, augmented reality technology (Augmented Reality, AR) has attracted great attention in the consumer market in recent years, and various products have emerged one after another, setting off a wave of visual revolution. Augmented reality technology is a technology that combines real scenes with virtual scenes. Its purpose is to realize the real scene (display environment or user image) and virtual scene (computer-generated virtual environment or virtual objects) through computer graphics and image processing technology. synthesis. [0003] Similarly, multi-modal human-computer interaction technology is currently widely studied in the field of human-computer interaction. The multimodal human-computer inter...

AI Technical Summary

Problems solved by technology

[0004] However, although the current augmented reality technology has a different visual display method from the traditional one, providing more information in a holographic manner, wearable AR devices also have good mobility and portability, but they lack natural, intuitive and efficient interaction methods, usually only through controllers or simple voice or gestures, which makes the user experience poor

The current multi-modal interaction method can unify different sensory modalities to achieve natural, intuitive and efficient interaction, but it is only applied to desktop devices and lacks good portability and mobility.

Images