Workflow

Importance of Spatial Audio In VR Content

Hearing is fundamental to our perception of the surrounding world.  Achieving this effect in virtual reality requires audio that sounds real and authentic.  Implementing spatial audio to create full immersion in 360° video or interactive VR requires capturing audio or a physical acoustic modeling of the space where the scene takes place.  An appropriate soundscape can provide the quickest path to immersion for just about any VR experience, and even removing the visual element, still enables us to sufficiently perceive the surrounding world – giving us a sense of space, time, and presence.  In contrast, the silent experiences, or the ones with incongruent sound would break the sense of presence and immersion, thus immediately removing the suspension of disbelief, and as a result substantially degrading the overall experience.

Spatial sound recording or let’s do it in post?

The use of conventional industry formats such as Mono (single channel) and Stereo (two channels) are a basic requirement, although they are limiting and no longer sufficient to offer full immersion in 360° videos or interactive VR experiences.  The use of spatial audio is the only way to create true three-dimensional audio, which utilises higher number of channels, be it capturing sound on location or through the means of sound design and mixing in post-production.  Depending on the nature of the project both methods are important.  Often to design the full sonic experience in VR, it requires spatial sound recording on set along with sound design and spatialisation of individual elements in post-production such as atmosphere, dialogue, foley, sound effects, and music.

Ambisonic format is the most effective method to capture location sound

There are a number of ways to capture the location sound.  However, the most effective method is to record in an ambisonic format which utilises four channels capturing the sound in all directions, along with discrete sound sources such as dialogue or any required diegetic sounds that are part of the scene.  The latter can then be positioned accordingly within the 3D soundfield by employing specialist spatialisation software within audio editing application or a game engine.  This approach enables VR audio content makers to work with an adequate resolution within the virtual space for positioning sound components across four, 16 or more virtual or physical channels.

Ambisonic sound offers a number of significant benefits that play a crucial role in making experiences as realistic as possible.

-Firstly, sound that was captured in all directions then enables the user to move their head and body while wearing any head-mounted display, and with a use of head-tracking system perceive their own dynamic position within the space in relation to the surrounding environment.

-Second, greater channel count offers more accuracy in positioning individual elements within the 3D space. This avoids everything coming from the same general direction as is common when listening to music, but lacking in realism when comes to creating a metaverse or offering your audience an authentic 360° video experience.

Why is this essential?

The considerations mentioned above are essential due the phenomenon described as a head-related transfer function (HRTF), which is a response that informs how our ears perceive sound from own position in space.  Collectively, head-related functions for both ears give a perception of binaural sound, enabling us to effectively identify a location and a distance of sound sources by constantly receiving sonic information to measure sound intensity and the time difference between sounds arriving to both ears.  We re-create this psychoacoustic process in post-production to then achieve a latency-free, real-time binaural rendering via a close approximation of personal HRTF.  It is essential to take human physiology into consideration when making audience fully immersed and enjoy their experience, be it a story, game or cognitive therapy etc.

The use of audio in marketing campaigns to guide your audience in 360° content

Unlike 2D content where a viewer can see the entire field of view in one direction, the 360° environment presents challenges as well as the opportunities for creative content makers when it comes to constructing the narrative.  Spatial audio can be an effective tool to lead or surprise your audience.  By looking in one direction at any given time, the viewer can easily miss out on what is behind them or sidewards, by implementing sonic cues within the space we can control or suggest a narrative.  By helping our audience to navigate through their point of view, we can ultimately guide them to and encourage them to engage with a specific element within the experience.

What is more important?

When combined effectively, fully integrated visual and sonic perception work in perfect harmony that enables us to see, hear, feel and appreciate the beauty and richness of our world.  Virtual reality already proved its effectiveness in video storytelling, gaming, educational training, social interaction and medical applications.  In order to make any of the above experiences successful, it requires a coherent approach of applying sound and visual content to make it as effective for its purpose as possible – more immersive, more authentic and as the result more engaging, more memorable, more empathetic, more fun and ultimately good enough to have a desire to come back and experience it again and again.

1.618 Team