Leave a comment

Too Much Information – Keiichi Matsuda’s vision for augmented and mixed reality

Keiichi Matsuda stormed onto the scene by releasing a cloud of interface and brand-related visual clutter in a nondescript kitchen. His short film Augmented (hyper)Reality: Domestic Robocop proposed a sardonic endgame for augmented reality (AR) and showed it from point of view of a future consumer, who was somewhat empowered but mostly beleaguered by the tyranny of corporate logos and superfluous information; recipes and didactic instructions hovered in mid-air, social networks transcend flatland and become immersive spaces, and the display resolution of texture maps dipped – making the mediated wallpaper that covered everything a glitchy and unreliable affair. This was 2010, two years before Google would release its hokey ukulele-scored concept video for Google Glass. Just wrapping up his architectural education, London-based Matsuda had staked a convincing claim on what AR could be if it was pushed to its limits – and then some.

Three weeks ago Matsuda released his latest film Hyper-Reality, a crowdfunded short that takes place in the same over-mediated narrative universe of Domestic Robocop and his other earlier works. Set in Medellín, Colombia, it depicts a day in the life of Juliana Restrepo, an underemployed woman at the beck and call of a horrible TaskRabbit-esque job service. In the midst of a existential crisis, she suffers a loyalty points related meltdown in a grocery store and then things get really strange. Fusing reggaetón beats, biometric identity theft, and the elusive prospect of salvation, Matsuda’s Medellín is teeming with life and ‘too much information.’ Riding high of the wave of attention the work received, the director and designer took some time to chat with CAN about the intricacies of his film, and its many implications.

Your 2010 film Augmented (hyper)Reality: Domestic Robocop was certainly sardonic but its tone is noticeably lighter than Hyper-Reality, where precarity looms large. Is this simply due to the narrative arc of the newer work, or is your wariness towards over-mediation increasing?

I’m very close to the project, so I find it difficult to judge the tone, or compare it to my other films. But I suppose Domestic Robocop is a somewhat isolated film, whereas Hyper-Reality is much more ambitious and wider in scope, with characters and situations that allowed me to build out more of that world. I think there’s just a lot more in it. The technology of AR (or its rebranding as ‘mixed reality’) hasn’t significantly changed its vision since my first film, but there have been a lot of other significant shifts in the tech/media landscape; the sharing economy, virtual assistants, gamification and a lot of that patronising and intrusive ’brand personality’ stuff. I suppose my work has always been a way of expressing my concerns around technology, so it follows that the horror level rises in relation to my ambition.

Yeah, you can definitely feel the shift in the landscape between the two works. In the earlier films the ubiquitous media kind of felt like an invasive visual nuisance but the gamification layer in Hyper-Reality is downright oppressive – the protagonist is a ‘job monkey’ managed by bots! To what degree is the film an indictment of the neoliberal ‘solutionism’ of contemporary Silicon Valley?

I suppose the scalability of digital technology means that disruption can happen very quickly. Long-established industries and ways of working get torn down in a matter of years, and replaced by untested, unregulated systems. In this kind of accelerated capitalism, it’s easy to overlook the huge impact that this has on the daily life and routines of the people caught up in those systems. In fact the focus seems to be more on finding new ways to disrupt industries, and less about improving the quality of life for people. At the moment it’s taxi drivers, delivery people and warehouse workers, but it’s not hard to imagine the reach of this brand of capitalism spreading much further. It’s one of the forces that is shaping the human experience, and human experience is the territory of design. I believe that designers have a duty to respond.

Your protagonist Juliana is a 42 year old underemployed woman who lets an algorithm direct her around Medellín as she carries out a litany of menial jobs. Could you talk about the design of her personality and the character arc that takes her from personal shopper to swiping the sign of the cross and adopting Catholicism as the film ends?

I grew up thinking that the world held infinite possibilities, so it was quite a shock to learn that our societies work in very narrow, often backward ways. Conventions and standards are so well defined and we generally stick very close to them, in our media, entertainment and politics (although what is ‘mainstream’ varies between cultures). We start off flexible and open-minded, but gradually entrench ourselves in a single reality. Augmented and mixed reality present really interesting possibilities in this area, by literally allowing you to see the world in different ways. In Augmented City, I was interested in how you could customise your view of the city to reflect your personal tastes; the character in that film fills his world with street art, Georgian architecture and flowers. In Hyper-Reality, the supermarket glitches between two different world-views too, with different virtual architecture and product branding depending on who’s looking at it. But I realised that we are talking about something much more powerful and profound than aesthetic tweaks; the interface actually represents a belief system. Juliana is very passive, having adopted the belief systems presented to her, but external forces give her the chance to re-invent herself. Catholicism is of course a huge part of life for many people in Colombia and other parts of Latin America. It also seemed like the only belief system powerful enough to compete with neoliberal consumer capitalism.

Juliana is very passive, you can kind of sense her waywardness from the “I don’t know what to put here” blurb in her social media profile bio section, and the mostly empty squares where her friends’ avatars should be. Given she never appears onscreen, what challenges did you face in communicating her personality through the POV filming?

In most character driven film or video, we understand character through the faces and body language of the actors. But AR/MR is a very subjective medium, and I knew I wanted to shoot POV. So I had to think about how I could convey a character that you could care about, without seeing her face, in less than six minutes. So the concept is that the character is expressed through her environment. Everything is narrowcasted/personalised to her character; the system knows she is single and overweight, so markets to her accordingly. Her beliefs and frustrations are reflected in the city around her.

↑ “We were taken to the notorious barrio Communa 13 by our guide El Perro (a local graffiti artist); we shot it on a location scout, but it didn’t end up in Hyper-Reality. It’s an amazing location though, so maybe I’ll revisit it for a future film.”

Medellín, Colombia is not the first urban environment that comes to mind when one thinks about science fiction cities. I remember when we last talked in 2014, you were excited about the city. Could you tell our readers a little bit about its character, your connection to communities there, and why it is featured in Hyper-Reality?

Well actually, a big part of it was precisely because it’s not an established ‘science fiction city’. I visited Medellín for the first time in 2012, to speak at FRACTAL, a conference organised by Vivi Trujillo and Hernán Ortiz. They invited both SF writers and experts in emerging technology to lay out their visions, and the event evolved to be a kind of collaborative writing workshop, mapping out future scenarios. Medellín has a very dark history, but there was an amazing energy there, and the local attendees spoke passionately about the future of the city. It’s been going through rapid change, so I was drawn to that dynamism and sense of progress. But it’s also a very beautiful setting. It’s surrounded by mountains and dense jungle, an isolated crater teeming with life. It’s got big economic divides, US tech slowly trickling down through tiny crowded shops in shabby shopping malls, sleazy politicians in bed with developers, super-rich mansions in the hills, futuristic infrastructure set against crumbling barrios, superstitions, folklore and religion. The shops and restaurants spill out onto the streets, and it’s people, (the Paisas) are famously friendly, enthusiastic and welcoming. It has a very palpable energy. I was invited back in 2013, and it just seemed obvious to set the film there. Vivi and Hernán agreed to produce the shoot, and I got a lot of help from their amazing network in the city.

Hyper-Reality, on site and set; the centre image is a screen grab from PFTrack, software that was leaned on heavily in production to facilitate ‘match-moving’ – where a virtual camera matches the movement of a physical camera.

Right, so you fed off the energy of the city – that makes sense. Back to my comment about Medellín not being the ‘city we’d expect’; I think there is a strong case to be made that the best sci-fi projects a distinct vision for a city that is already known. Ridley Scott and Syd Mead’s vision of Los Angeles, 2019 in Blade Runner endures, and L.A. was recently reimagined (and mixed with Shanghai) in Spike Jonze’s Her. What depictions of future cities (cinematic or otherwise) have influenced you?

I suppose I’m attracted to explorative design, that proposes alternative ways of living. My background is in architecture, so a lot of my influence comes from the visionary projects in futurism through to modernism and post-modernism. There are a lot of incredible projects, but Antonio Sant’Elia’s Città Nuova, through to Le Corbusier’s masterplan for Paris, and Walt Disney’s plans for Disneyland and EPCOT stand out. I also love the critical ’60s work of Superstudio, Archigram and Cedric Price, and the Japanese Metabolist movement shortly after. In fiction, Moebius’ cities, Taiyo Matsumoto’s Treasure Town, J.G. Ballard’s various versions of London and other fictional cities in his short stories; I don’t know, there are too many to mention – isn’t that great? Basically, I love design that emerges from an ideology or a system. But most of those are visions from different ages, with different questions and concerns. My Medellín doesn’t look anything like those, but it shares the same exploratory spirit. I hope people will feel that it’s emerged from the issues and ideologies of our time.

While I‘m hard-pressed to compare your visual style to anything else out there, one reference that comes to mind is Black Mirror, specifically the cartoonish “15 Million Merits” episode from its first season. I have to ask, what do you think of Charlie Brooker and his peers’ writing, the show’s production design, and the series in general?

Oh yeah I love Black Mirror and Charlie Brooker’s work in general. I released my first two films in 2010, the year before the first season aired, so I was really happy to see a lot of the issues close to my heart explored in that format. There aren’t many TV shows that make you feel as uncomfortable (actually some of the episodes get a bit too depressing for me); it’s undeniably an important series. I think it really highlights the necessity (and public demand) for the arts to push back against the mostly unchecked spread of techno-utopianism.

hyper-reality-production2↑ Located in Saboneta (an urban centre on the outskirts of Medellín), this ‘Virgin Megastore’ rented Matsuda the effigy of Mary used in Hyper-Reality’s final scene.

I think it is hard to come away from a viewing (or six) of Hyper-Reality without a sense of wonder about the film’s technical virtuosity. I imagine we are talking about hundreds of hours of post-production per minute of onscreen action. However, what might not be apparent to viewers is the wizardry that went into shooting it. Could you talk about the camera rigging and set design that aided you in your world building?

Ha, yes I don’t even want to count the hours. The shoot happened pretty quickly. We shot on a GoPro head-mounted rig, guerrilla style with very little in the way of props or permission!

“After I shouted ‘cut’ on the final take, I saw the make-up girl running towards us shouting something. It turned out that some guys on bikes had pulled up next to our getaway driver, put a gun to his head, and stolen his bike.”

A local bus company called Cootrasana agreed to give us a bus and driver for a day, in exchange for some exposure in the film (you can see their logos everywhere). We then had to get the timings right during rush hour downtown, having the bus stop just as the main character was finishing their existential googling. The other main thing was getting the effigy of the Virgin Mary. You would have thought that it would be difficult to find a statue, let alone a place willing to rent one out, but my producers did a great job. We found not one, but six specialist shops in Sabaneta (just south of Medellín), selling effigies of Jesus, assorted angels, and of course, plenty of virgins. I didn’t know before, but there are many different versions of the virgin Mary. The virgins look different, with different clothes, colours, and decorations, and each is associated with a particular miracle or appearance on Earth. Guadaloupe Maria for example, is dark skinned clothed and in green and gold. We chose a large statue of María Auxiliadora. That scene was the hardest to shoot, as it involved fake blood, a getaway motorbike and driver, and trying to stop people praying to the prop statue. The character had to stumble across the road and see the virgin, but there was so much traffic that it took 21 takes to get a clear path across (even with an assistant trying to stop the traffic further up the road). Word must have got around, because after I shouted ‘cut’ on the final take, I saw the make-up girl running towards us shouting something. It turned out that some guys on bikes had pulled up next to our getaway driver, put a gun to his head, and stolen his bike. It was really intense, I can’t imagine what it was like for him. But no-one got hurt, and he’s fine now; new bike on insurance, and none other than William Gibson retweeted his account of the theft!

An adorable detail that I have to ask about: what was with the ‘virtual pet’ dog in the grocery store scene? What did Juliana buy that rewarded him with a dapper top hat?

The dog came about for a few different reasons. I wanted to frame consumer capitalism as a belief system, and the supermarket is a clear symbol of that. In fact the supermarket literally becomes a cathedral in one of the scenes. But I realised that physical supermarkets might struggle to compete with e-commerce food delivery, so they would want to create incentives. The dog is a kind of virtual pet, that lives in the supermarket and requires you to go and look after it. Physical products come with virtual products attached, so buying shampoo for yourself might also give your pet a bath. It’s a way for shops to build emotional bonds with their customers, and guide them towards spending more (premium products come with premium rewards). I was also interested in the idea that the augmented city could be home to both humans and virtual characters, like NPCs in an online multiplayer game. Instead of having an app for booking a holiday, you could summon a virtual assistant that walks next to you and helps you book. The city would be filled with these assistants, some humanoid, some animals, some more fantastical. Finally, the film is broadly structured around three games; the portable game at the start, the building-scale game with the virtual pet, and at the largest scale, the worldwide game of Catholicism. The hat came with the pineapple, you can see it being advertised if you look closely.

So now that Hyper-Reality has been released, what are the next steps for your studio?

It’s been crazy. Across all the official uploads (Vimeo, YouTube, Facebook), it’s been viewed over three million times, and even the bootlegs are in the hundreds of thousands! We’ve been working recently on concept design consultancy, which has been great, but I’ve got a mountain of emails to get through, so who knows what’s buried in there. I’ve got a lot of ideas for projects; responsive sculptures, VR experiences, and of course I want to make more Hyper-Reality stuff. I’m looking for sponsors, so please get in touch if you’d like to get involved or find out more.

Project Page | Keiichi Matsuda

Leave a Reply

Your email address will not be published. Required fields are marked *