Augmented reality keyboard and mouse system based on gesture interaction

Abstract

The invention provides an augmented reality keyboard and mouse system based on gesture interaction. The augmented reality keyboard and mouse system comprises an MEMS module, an EMG module, an infraredlight module, an electric tactile stimulation module, an auditory feedback module, a depth image acquisition module, a visual enhancement module and a comprehensive processing module. The comprehensive processing module is connected with the MEMS module, the EMG module, the infrared light module, the electrotactile stimulation module, the auditory feedback module, the depth image acquisition module and the visual enhancement module. The auditory feedback module, the depth image acquisition module and the vision enhancement module are installed on the augmented reality glasses. Compared with atraditional keyboard and a traditional mouse, more natural man-machine interaction can be achieved, a stronger sense of reality is achieved, the manufacturing cost is low, and the period is short. The system is simple and light to wear, high in universality and suitable for various application scenes. Compared with a single-mode interaction method, the multi-mode gesture recognition and touch andauditory feedback combined interaction method based on the MEMS and the EMG is more stable in performance and better in user experience.

Description

technical field [0001] The invention mainly relates to the field of human-computer interaction, in particular to an augmented reality keyboard and three-dimensional mouse human-machine interface system based on gesture interaction. Background technique [0002] Augmented reality technology (Augmented Reality, referred to as AR), is an interactive experience with the real world environment. Objects residing in the real world are "enhanced" by computer-generated visual, auditory, tactile and other sensory information. The superimposed sensory information can be a supplement to the natural environment, or it can be a cover for the natural environment, and it is seamlessly intertwined with the physical world, so that people can get an immersive interactive experience. Augmented reality is used to enhance natural environments or situations and provide perceptually rich experiences. With augmented reality, information from the real world around the user becomes interactive and di...

AI Technical Summary

Problems solved by technology

However, the image-based method is susceptible to interference from natural light, which makes it extremely vulnerable to failure in outdoor environments. Moreover, due to the large range of motion of the human hand in space, fingers will occlude and interfere with each other, and the surface of the hand lacks a strong resolution. Features and other issues, resulting in low accuracy of image-based dynamic gesture tracking system, and it is prone to ambiguity

The voice-based interaction method has strict requirements on the background noise of the environment and the cloud computing network environment. For crowded public places or environments with unstable network environments, the recognition accuracy will be greatly reduced.

Images