Featured, Other
comment 1

Z/Z/Z/ – Archiving film, animation and videogame image analysis

Z/Z/Z/ – thumbnail

S.T.U.N. Runner – Z-Projection of Maximum Value for Each Pixel

Z/Z/Z/ is a project hatched by Daniel Rehn and Sarah Caluag dedicated to “describing the dimension of cultural artifacts that are difficult to explain using natural language”. This endeavour deploys a custom visualization workflow to break down footage from film, animation and games and reconstitute this source material into stills and animated GIFs using a range of image analysis techniques. This playful and technically-minded approach to remixing media has yielded an ever-growing database of case studies which includes many important precedents (John Whitney’s “Catalog”, Stanley Kubrick’s “2001, A Spacey Odyssey”, “Astroids” etc.). Daniel and Sarah recently chatted with CAN about their work and they’ve provided us with the following overview of their research.

Could you provide a brief introduction to the Z/Z/Z/ project and how it fits into your creative partnership?

Daniel Rehn & Sarah Caluag: Absolutely. While the underpinnings of Z/Z/Z/ predate our creative partnership, its dual-mission of research and aesthetic-production mirror the goals of our overall practice. The ability to analytically, quantifiably describe these visualizations while also revelling in their beauty is ideal for us. We can can create both large-scale, informative surveys of a medium/era and individual aesthetic layers for re-appropriation in, say, our live visual performances.

Your credits state “Data visualized with ZML + Vilinx” – could you provide the backstory to the tools you are using?

DR: The project’s underlying tools originate with research that Jeremy Douglass and I began in 2008. We met when he came to Calit2 to conduct post-doctoral research and help start Lev Manovich’s Software Studies lab. Jeremy is a perennial collaborator, and we quickly co-founded a games non-profit (Playpower) and a games research lab (Re:Game). The latter focused heavily on gameplay recordings—it was a desire to quantify and analyze those that led us to the possibility of frames as a visualization building block.

Regarding the site itself, I previously designed several big-science-funded ‘curated data repositories’ at the Supercomputer Center. Once the initial pool of images became unmanageable with a basic file structure, I created a similar repository for Z/Z/Z/. I’ve also designed display-walls for ‘situation rooms’ and can envision that leading to a physical installation this project. Our adoption of the data repository (and a situation room potentially) demonstrate how this project delivers on the promise of digital humanities, and also why it’s a natural fit given my hybridized background.

Technical breakdown: Vilinx is a collection of visualization scripts that Jeremy and I continue to extend; he deserves credit for their initial creation. ZML is a custom XML schema designed around a frame stack and related visualizations. ZDB is the web-based data repository where they are stored. I coded ZML/ZDB over the past two years; new features and a new version are already planned, including more visualization types and comparison capabilities.


General Electric, Build a Phone – Extended montage of source frames

General Electric, Build a Phone – Extended montage of source frames

Please talk us through your working process in analyzing one of your favourite studies archived on the site.

DR: With regards to selecting material, our eyes have been trained by these techniques for a few years now. There’s a nice balance between just knowing what artifacts will lead to interesting results and the occasional experimental leap into the aesthetic/compositional unknown.

Star Rider is a futuristic, saturated Laserdisc game from 1983. The footage we analyzed is a run-through of the game’s isolated background visuals without any foreground elements (here’s footage of actual gameplay). This study isn’t overly complex but it does utilize almost all of the techniques well, making it an ideal introductory dissection.

Even though we’ve built a robust toolset, there are still several steps which require human input. For instance: the video must be trimmed to the section that we are analyzing, cropped and color-corrected, then output to frames. Deciding what frame rate to export at is a crucial decision that affects the final output [anywhere from 2–24fps resulting in 800–3000 individual frames] as is choosing how many images to display in an extended montage.

Star Rider – Image analysis composite

Now, for the output (see full-sized versions of the above thumbnails here):

  •  The extended montage is easily read as a chart of the shot-length for each environment as well as the curvature of the road.
  • The keyframe (top→bottom) depicts the density of background objects in the environment. This density is represented in what appear to be colorful wire-harnesses.
  • The average (left→right) provides a sideways view of shot-length as well as a complete color palette sampling and gradient depth.
  • The average (front) demonstrates a “long-exposure photograph” signature that is so common with this technique. It also suggests that—no matter how often it seems like the road is turning—it’s mostly straight throughout the animation.
  • The maximum (front) predictably features the road (dark purple), background objects (white regions), and a surprisingly off-balance, sky-only region (light-pink).

A new feature—which I haven’t had a chance to exploit yet—is the ability to easily generate custom variations of a particular technique. These will appear in a special section in the archive and will allow further pushing individual studies in uniquely appropriate directions.

Z/Z/Z/ appears super-invested in a remix-based approach to exploring a range of mediums. While much of the media analysis on the site is quite beautiful, it is displayed rather matter-of-factly. Should we be reading your website as an archive or a gallery?

DR: The project’s co-investment in aesthetic production and cultural analysis position it as both an archive and a gallery. The primary site features a curated collection of visualizations that point to their full dataset (within data.zprojection.com). Traditionally, you’d have either beautiful images or a research platform, but not both.

Edgerton Marey MuybridgeHarold Edgerton (MIT professor who popularized the stroboscopic flash) swore defiantly that his images were not art. Unfortunately for dear Edgerton, art history has conspired to include his images within its walls. A bullet passing through an apple, a nuclear explosion, and a frozen milk droplet are all too beautiful to ignore, and, yet, they are incredibly revealing scientifically as well. The early, interdisciplinary motion-studies of Edgerton, Marey, and Muybridge comprise much of the foundation upon which Z/Z/Z/ is built.

Your site organizes content by media (animation, cinema, games, TV). In working across mediums, what kind of differences do you notice in the output of your various analytic techniques?

DR: There are technique/medium combinations that reliably produce interesting output. I’ll list a few examples:  Animations with consistent composition and morphing forms produce incredible extended montages. Often times, these montages are akin to a specimen sheet—a visual catalog of animated forms. Pixel-based games with stationary backgrounds produce beautiful averages that emulate long-exposure photographs (monochrome Gameboy titles are particularly suited to this). Meanwhile, polygon and vector-based games result in chaotic maximums that indicate how frequently every on-screen position was exploited. Lastly, any media that feature large, rapidly changing blocks of color produce compelling orthogonal animations where the color slices ‘dance’ and flow.


I see the work of many animation heroes like Norman McLaren and John Whitney nestled in the depths of your archives. In what sense might you consider these selected projects as ancestors to this undertaking?

DR & SC: Z/Z/Z/’s visual heritage is a triangulation of early motion-studies, abstract animation, and Cubism/Futurism. The abstract animators are particularly important to us on an aesthetic level. Their complex compositions and strong use of color naturally lend themselves to our processes. In fact, many of the other works we’ve studies feature similar qualities (computer animation, motion graphics, 2D/polygon videogames). So, in that sense, an ancestral claim is secured and will continue to permeate as the archive expands.

Z/Z/Z/ (note the ‘process 101’ info which further explains the image analysis techniques)
Daniel Rehn / Sarah Caluag 

Mr. Driller – Z-Projection of Average (Mean) Value for Each PixelPatience – Selected keyframe from the midpoint of each seriesEdge – Series of Keyframes, evenly distributed over timeGrey Still Life 2 – Selected keyframe from the midpoint of each series