Cubism of sound

2018

''The city, it sings ... so listen, and there is more to hear''

~ John McGregor

''Through sound, understand where we are living in once again; by listening, find the information hiding in the common events.''

~ Tsan Cheng Wu

We hear and think that we have heard. We are desensitised to the sound around us, but familiarity sometimes differs from understanding. Sounds carry many layers of meanings including materiality, emotions, and social values, and they are often hiding within each other. Eventually, they become something we called ''noise''. I want to find the meaning of sound, by deconstructing, combining, modulating, or even faking it. The goal is not to explain, but to reveal.

Cubism of Sound is an interactive video that uses face detection to play sounds corresponding to the space inside the video according to the viewer’s position. In the video, 6 microphones were placed at different locations in the coffee shop and recorded at the same time. When the viewer moves, the computer will play sounds from the direction and volume accordingly. The content of recordings includes conversations, ambient noise, and machine noise picked carefully by my ambient sound analysis. After recorded the video and the sounds, I added 3 Foley's soundtracks to the existing ones just to emphasise the contract.

Video snapshot

Floor plan and set up positions.

With this work, I wish the viewers to explore within the virtual space by sounds and their imagination, discover the details that are impossible to be seen, and understand the meanings of sounds.

One frequently asked question is ''why am I showing a video when I want to emphasise the hearing perception?''. I did consider using the soundtrack without any visual communication, however, the main issue I am looking at is not that ''we cannot hear when there is sound, but ''we tend to overlook the sounds when we can see''. Therefore, the video is there to create an illusion of the place, but with a restricted view, it provides less information than we think it should be.

Interview with a lady in the coffee shop

In fact, Cubism of Sound is more like an audio recording technology or system than an installation, which only shows a single aspect of many possible applications. For gaming, music, and film industries, it can provide extremely realistic feelings in the scene so that the users are not merely listening to the pre-recorded playbacks, but have the agency to choose which angle they want to understand the narratives from. It challenges traditional media communication norms and breaks one-directional storytelling formulae. For the users, it "dematerialises" their physical body so that they wander in the space more like ghosts or apparitions which encounter everything without physical limitations. When listening to the sounds, users have much room to imagine, think, and question what they hear. Sometimes, sound offers more information than what can be seen.