MaxMSP, vvvv
comments 7

In the Eyes of the Animal – MLF virtualise a forest through the eyes of its creatures

In the Eyes of the Animal is the latest installation by Marshmallow Laser Feast, commissioned by the AND Festival and set in Grizedale Forest, the Lake District forest in the UK. The project takes visitors on a fascinating journey that allows them to fly above the forest canopy, come face-to-face with hi-definition critters and embody various animals as they traverse the Grizedale landscape.

The project is further exploration of Lidar data that the team started earlier this year. After conceptual ground-work, the team scanned Grizedale forest with a Faro X330 lidar scanner that has provided them GPU expensive 800 million points which then decimated to a real-time rendering friendly level.

In the project; Lidar scanning, CT scanning and phoogrammetry techniques co-exist harmoniously even though they are completely different procedures. Each scene of the narrative contains multiple sets of environmental particles that come from Lidar data and dynamic particles that come from come from highly detailed CT Scans of insects and animals. AND Festival visitors are invited to see the forest through the unique eyes of its creatures.

↑ Debug Renderer in vvvv

The ultimate goal is to create an understanding of how these animals process optical information and so give people a chance to reflect on their own visual perceptions of the forest.

Real-time visual engine was built in vvvv with custom HLSL shaders and the audio engine built in MaxMSP with a custom ambisonic library from twobigears. Visual engine generates and renders whole environment in realtime with certain generative elements which makes each experience unique. Visual engine communicates with 3D Audio Engine via OSC to provide positional data as well as head tracking data from the Inertial sensors of VR headset. The sound uses Binaural audio, a technique mimicking the natural functioning of the ear by creating an illusion of 3D space and movement around the head of the listener as immersive as reality can be. Sounds were recorded in Grizedale forest and spatialised using custom built 3D sound tools by company Two Big Ears. Sounds move around the listener at 360° as well as up and down, adapting to the trajectory, head movements and visual experience of the listener. The audio experience was further extended beyond headphones to the whole body through bass vibration. Using the M1 from SubPac, listeners were able to feel their body vibrating as if incarnating breathing, flying and shouting animals. SubPac is an immersive audio device transmitting bass directly to the bones and muscles of the human body. Combination of these data allow 3d audio engine to pair dynamic elements to spatialised sound sources for more immersive VR experience.

The setup of the installation included 7 workstations with Quadro M6000 Graphics card to provide multiple VR experiences simultaneously. Tools used included VVVV, Max MSP, Lidar Scanning, CT Scanning, Photogrammetry, SubPac M1, Oculus Rift DK2, HLSL Shading and DX11. Binaural audio experience was developed by sound artist Antoine Bertin from SounDoesnTravel, a sound-driven creative studio.

The project was created as a feature of the AND Festival using learnings gained from Project Daedalus, a Nesta-funded research project aimed at investigating the creative potential of drones and aerial 360º cameras.

Marshmallow Laser Feast | AND Festival

Commissioner: Abandon Normal Devices and Forestry Commission England’s Forest Art Works
Marshmallow Laser Feast: Robin McNicholas & Barney Steel (Concept and Direction), Ersinhan Ersin and Natan Sinigaglia (Visual Artists), Abraham Manzanares (vvvv Artist), Marc Winklhofer (Designer), Antoine Bertin (Sound Design), Eleanor (Nell) Whitley (Producer), Annatruus Bakker (Assistant Producer), Adam Doherty (MLF Executive Producer) and Nvidia and Sub Pac (Equipment Support).
Abandon Normal Devices: Gabrielle Jenks (Director), Ruth McCullough (Senior Producer) and Tricia Coleman (Touring and Festival Producer)

In the Eyes of the Animal_18

In the Eyes of the Animal_22