A method and device for real-time virtual reproduction of remote sound field

Abstract

The invention discloses a method and device for real-time virtual reproduction of a remote sound field. The method includes: (1) Using an end-to-end convolutional neural network to infer the personalized elevation angle positioning factor from the user's ear image, and based on this, the general HRTF is customized, and the customized personalized HRTF is further applied Based on the synthesis of binaural virtual sound signals; (2) Based on the real-time detection of the user's head position, the playback of dynamic binaural virtual sound signals that track the head position is realized. The above-mentioned personalized customization and real-time dynamic virtual sound signal processing method can enhance the user's sense of presence and immersion. The device includes a remote sound field multi-channel pickup module, a cloud processing module, a client module and a 5G communication module. The implementation of the present invention is based on cloud computing technology and 5G network transmission, which can effectively reduce the computing load of the user end and meet the parallel requirements of multiple users. The present invention can be regarded as a sound module solution based on virtual reality of mobile portable equipment.

Description

technical field [0001] The invention relates to the technical field of 3D virtual sound, in particular to a method and device for real-time virtual reproduction of a remote sound field. Background technique [0002] With the development of the Internet era based on mobile terminals (such as mobile phones and tablets), remote real-time 3D live broadcasting has become a popular way of entertainment, learning and communication. The interactive coordination of audio-visual information is the key to remote real-time 3D live broadcast, and is an important determinant of the user's sense of presence and immersion. At present, compared with the remote real-time transmission of visual information, the maturity and effect of remote real-time transmission of auditory information need to be improved, mainly reflected in the personalized playback of virtual sound and dynamic adjustment of sound field. [0003] In order to fully cover all the acoustic information on the scene, the curren...

AI Technical Summary

Problems solved by technology

However, the existing mainstream systems still mainly use static replay, which assumes that the user's head remains still while watching the live broadcast.

This is mainly limited by two aspects: 1) head tracking equipment is relatively expensive and has not yet become a standard configuration for mobile terminals; 2) network transmission speed is limited

The data volume of dynamic virtual replay is much larger than that of static virtual replay; if dynamic replay is forced to be implemented, there will be sound lag and audio-visual mismatch.

[0005] The above shortcomings restrict the development of remote sound field real-time virtual reproduction technology and the further improvement of related product performance

Images