Created by Refik Anadol Studio, “Melting Memories” is a series of digital artworks that explore materiality of remembering by offering new insights into the representational possibilities of EEG data collected on the neural mechanisms of cognitive control.
Comprising data paintings, augmented data sculptures and light projections, the project as a whole debuts new advances in technology that enable visitors to experience aesthetic interpretations of motor movements inside a human brain. Each work grows out of the artist’s impressive experiments with the advanced technology tools provided by the Neuroscape Laboratory at the University of California, San Francisco. Neuroscape is a neuroscience center focusing on technology creation and scientific research on brain function of both healthy and impaired individuals. Refik gathers data on the neural mechanisms of cognitive control from an EEG (electroencephalogram) that measures changes in brain wave activity and provides evidence of how the brain functions over time. These data sets constitute the building blocks for the unique algorithms that the artist needs for the multi-dimensional visual structures on display.
The data collection process utilized a 32-channel Enobio and standard protocol configuration. Participants were instructed to focus on specific childhood memories during the recording process. A control recording was also conducted to identify artifacts to later filter with adaptive notch filtering and limiting the frequency range. For analysis the team focused on beta (13-17Hz) and theta (3-7Hz) channels, isolating activation points corresponding to short term and long-term (specifically episodic) memory. Their selections were the Fp1, Fp2, F7, F8, P3, P4, C3, C4, T7, T8, O1, and O2 nodes, which they also used to drive noise parameters within the real-time simulation. For scaling they applied Higuchi’s fractal dimension algorithm and used FFT for a moving average. Recurrent neural nets (via EEGLearn) they used on the recording sessions to generate spectral outputs, which were then utilized as height maps for the visual representation pipeline.
The team found transposing EEG data in to procedural noise forms a really engaging challenge, both technically and conceptually. In the input data and their mapped representation one can find recurrence and rhythm but also hints of higher dimensional structures. They wanted to do this efficiently and in real time and so working on Melting Memories dovetailed nicely with putting the last touches on FieldTrip, an (at the time pre-release) open source GPU library for HLSL/vvvv.
It allowed them to use a composite design pattern to very quickly iterate while producing the aesthetic structures used in the project. This approach enabled them to really explore some deeper procedural functions whilst keeping a completely modular graphics pipeline. This modularity makes it easy and clean for the team to expand on the project’s abstracted content in really interesting ways, such as further integration of machine learning on the source data, evolving rendering techniques and the creation of sculpted physical artifacts.