Found In Translation – Experiencing the process of machine translation

Created by TheGreenEyl, “Found in Translation” is an interactive, immersive installation for the exhibition ‘traNslatioNs – Understanding Misunderstanding’ at 21_21 DESIGN SIGHT in Tokyo. Using their own spoken sentences, visitors viscerally experience the process of machine translation. Visualizations show how the machine learning model clusters words from different languages by semantic similarity, and translations are presented typographically and auditorily across 24 languages.

The spatial environment consists of 24 screen panels and 24 audio speakers distributed across the room. The panels are organized and color-coded by language family, and each panel represents one language in text, diagrams and sound. In the center of the room is a microphone on a pedestal. When a visitor approaches the microphone the center panel shows a question as a prompt. Once an answer is spoken into the microphone a visualization shows the multi-lingual machine learning model that is used for translations. Word by word and sentence by sentence it becomes apparent which words from which languages are clustered together.

The visualizations travel across the entire room, showing a spatialized version of a small multiple diagram across the room. Each diagram in turn animates into the text version of the translation, as the translated text is spoken in voice.

Hardware used includes 3x 4K projectors, 24 channel audio, interface speaker, microphone and distance sensor. Software includes Max MSP for sound, ElectronWeb technologies: HTML, CSS, Javascript, D3, Three.js. The whole installation is practically one big browser window. Also used is Google speech-to-text API, Google Translate API and Google text-to-speech API. The machine learning model visualization is based on Recent Advances in Google Translate, BERT and the embeddings projector.

Project Page | TheGreenEyl | 21_21 DESIGN SIGHT

Full Credits: Studio TheGreenEyl: Richard The (Creative Direction), Richard The and Frédéric Eyl (Concept), Andreas Schmelas (Software), Marian Mentrup (Sound Design), Pam Anantrungroj (Spatial Design, Richard The and Calen Chung (Visual Design) // Google Creative Lab: Amit Pitaru (Creative Direction), Cassie Hamilton (Team Lead), Jasmin Rubinowitz (Machine Learning), Ben Wiley (Writing) // 21_21 Design Sight: Dominique Chen (Curator), Mamie Nishida (Program Officer), Luftzug (On-site production) // Video Production: Maco Film (Production & Editing), Jun Oshima and Chris Nicholls (Cinematography)




Leave a Reply