Created by Sam Chin, Skye Gao, Leonie Dong, Cathy Fang, and Alejandro Lopez-Rodriguez at the MIT Media Lab (Fluid Interfaces Group), [AI]legory of the Cave is an interactive installation featuring an LLM-controlled character who exists only as a shadow.
The piece begins dark, and a physical light switch invites interaction; flipping it on illuminates the stage and awakens the shadow character with a singular goal to turn off the light that defines his existence. Humans place objects in his path—blocks, frames, and stairs—to obstruct him. He kicks, pushes, sometimes gives up, then begins again. The cycle repeats as he discovers changes to his world that he cannot comprehend.
In Plato’s Allegory of the Cave, prisoners are chained facing a wall, seeing only shadows cast by objects behind them. They mistake these shadows for reality, never comprehending the three-dimensional world that produces them. The shadow man is such a prisoner: confined to 2D, he perceives silhouettes but never their source. This mirrors the perceptual limits of AI. An LLM cannot see. It can only read descriptions of sight. It processes language about the world without directly accessing the world itself. The shadow man embodies this gap. Both are confined to representations. The piece is a meditation on AI as cave-dweller – processing patterns without grasping the higher-dimensional reality that casts them.
Sam Chin, Skye Gao, Leonie Dong, Cathy Fang, and Alejandro Lopez-Rodriguez
The interface is written in JavaScript and is web-based. A camera captures the projected shadow and the camera feed is thresholded by color such that the shadow is separated from the background. The figure’s shadow is cancelled out. The world view is converted into ASCII and sent to an LLM, where the shadows, the figure, the ground, and the light switch are represented by different symbols (such as “#”, “P”, “G”, “S”). The LLM is prompted with the scene information and the available movements the figure can make (such as walk, jump, climb, and kick). The goal is for the figure to turn off the light switch. However, the figure is bound by the game-world physics (done by Matter.js), and cannot move through shadows and can stand on top of shadows. The obstacles in the real world are controlled via serial port connection with a microcontroller.
The light switch is connected to a solenoid and can sense when it is turned on or off. The obstacles have magnets embedded inside and are lightweight. When the figure decides to kick an object, a command is sent to a connected ESP32 XIAO which controls an electromagnet. There is a Darlington pair, flyback diode, and bleed resistor to protect the GPIO of the microcontroller. When the electromagnet turns on, it creates a magnetic field which interacts with a magnet embedded in the blocks, causing the blocks to move. The different movements of the blocks are controlled by the relative weight of each of the blocks and the placement of the magnets.
Project Page | Project Page (Sam Chin) | Fluid Interface Group (MIT Media Lab)
