Developed by the Innovation Lab of Milla & Partner GmbH, a German interaction and spatial design agency based in Stuttgart and Berlin, NO_THING is a tracking and mapping framework that uses infrared light to turn portable physical objects into interactive displays. Prepared with reflective foil markers, unassuming surfaces as simple as a sheet of paper can reveal customized, proximity-related content that responds to point and click, drag and drop or spatial gestures.
NO_THING recently premiered at the German Pavilion at the Expo Milano 2015 after four years of R&D. Here, blank sheets of cardboard are turned into handheld “SeedBoards” that seamlessly provide visitors with information about the latest in energy and nutrition (this Expo’s theme is “Feeding the Planet, Energy for Life”) across fourteen different stations. While NO_THING’s immediacy, intuitiveness and tactile, low-tech interface speaks particularly well to general audiences such as Expo’s, the German Pavilion implementation is only one of many applications the developers foresee. Scaleable with budget, technology, and context, NO_THING is ‘bridge technology’, they say, that allows them to experiment with tomorrow’s “holodeck” scenarios today.
↑ NO_THING installed in Milla & Partner’s testing grounds: a blank piece of cardboard becomes an interactive device to display, navigate and trigger media
“We get to do a lot of thinking about the future of interaction,” says Milla & Partner’s Ingo Wörner, who together with Fabian Fuchs spearheaded NO_THING’s in-house development. Having realized award-winning exhibitions, show rooms, and trade shows, their team knows the limitations of the current generation of smart spaces all too well. “We were frustrated with solutions available to us at the time. While the arrival of audio guides, VR, AR or tablets has dramatically improved how contextual information can be accessed within a space, they create largely solitary, device-driven experiences; devices that have to be lugged around and that distract as much as they reveal,” says Ingo. Another problem is technology itself: devices age quickly (particularly when they go through many hands), require maintenance, and updates. With NO_THING, Ingo and Fabian tried to get rid of the technology altogether (they describe the experience as “close to NO_TECH”) by taking full control of the environment. “We wanted to create a smart space that is aware of my presence and that provides relevant information when and where I need it, without me having to carry around any equipment, punch in numbers or navigate a screen. To achieve that we use nothing but light — visible and invisible.”
To project onto a moving target, which is what NO_THING does at its core, the team first built prototypes with Kinect systems, openFrameworks, ARToolKit and OpenCV. To increase tracking precision, stability and overall performance across different light settings, NO_THING now uses IR (infrared) light, OptiTrack motion capture cameras (with in-camera blob tracking) and IR-reflecting foil for the markers. The tracking data can be accessed via a Node.js-based and Socket.IO-connected tracking server and allows a wide range of interactions – from tilting, rotation, and gestures, to movements and accessing pre-defined hotspots. The 3d projection mapping onto the tracked cardboard ‘devices’ is done in Pandoras Box (a commercial media and show control system) using a webkit layer as texture. The advantage of the latter: all content and interaction can be built and delivered with standard web technology.
What goes where: The OptiTrack IR camera module (top left) provides a realtime view of three IR marker sets of three active users being tracked. The camera tracking data is then processed in Pandoras Box’s Widget Designer (lower left) and transferred to a Node.js-based tracking server (right) via OSC. The tracking server provides the webkit layer (rendered in Pandoras Box) with the tracking data via WebSocket.
A ‘holodeck’ degree of freedom demands a lot of the tracking and projection. NO_THING currently performs at an impressive 60 frames per second (a latency of 3 to 4 frames is largely intentional to ensure a stable image) and with a ‘device’ angle of up to 80 degrees. Additional sensors on the device would increase the tracking precision even further.
↑ NO_THING in the making: equipped with infrared light reflective foil markers, NO_THING cardboard ‘devices’ are being tracked with OptiTrack IR cameras
“Given the trend towards miniaturization and mobility of projection technology and seamless connectivity there’s an infinite potential for meaningful mixed-reality applications,” says Fabian about NO_THING’s future. From deploying the next generation display and lighting technology (miniature LEDs, lasers) to interfacing with ‘always on’ devices to the prospects of taking NO_THING outside of “controlled environments” and to the street — “It is as if the future has met us halfway.”
For now, the project’s biggest feat remains the manifestation of the interface. “The surprise transformation of a simple piece of cardboard without any electronics into an interactive device is magical. It gets me every time and personally, it’s what I appreciate the most,” says Fabian. “And while the commercial applications, as evidenced by our Expo 2015 use case, are myriad, I’d really like to see an artistic take on it, maybe as part of an installation or performance.” So do we.
NO_THING’s Expo Milano 2015 setup was developed together with Jan Hüwel of coolux (producers of Pandoras Box).