openFrameworks Sound TouchDesigner
Leave a comment

Expressions – Paint and pixel matière at micro-scale

Created by Kynd in collaboration with Yu Miyashita (Sound), ‘Expressions’ is a series of artworks exploring the physicality of thick and bold paint-like dynamic constructs that emerge from illuminated digital space revealing an intricate play of shapes, light and shadow. The video, titled ‘Accelerated Expressions’ (video below) is a culmination of experiments and a dynamic counterpart driven by Yu Miyashita’s recent composition, in contrast to more ambient versions made for living room (FRAMED) and street (Dong Gallery) installations. All graphics are procedurally generated in real time, no post-editing.

Expressionism and Neo Expressionism, Kynd has long been inspired by the notion of digital paint, and when at art school, the fellow students would often use various materials to create texture (matière in Japanese-French) for their paintings. Calcium carbonate was most commonly used to the paint to increase the thickness. There were many other oils and substances used to achieve unique characteristics. Though Kynd was not a big fan of intensive use of the method since it was often used to cover up immature technique, lack of detail, etc. or tend to distract the artist from the subject (concrete or abstract), he doesn’t deny there was a certain joy and satisfaction in playing with these materials and seeing  expressions they make. At the sam time, Kynd has also been particularly attracted to miniature representations of the world. From architecture models, or city plans to Japanese gardens designed to mimic nature.  Very objective camera angle from the top would make the construction appear like an imaginary landscape, or perhaps Bonsai made by non-human intelligence like a Martian. This type of work also reminded him of Koyaanisqatsi. If Koyaanisqatsi is a time-lapse at global scale, ‘Expressions’ might be a time-lapse of personal memory at a microscopic scale.

Lighting plays a significant role in this video adding colors, depicting contours and casting intricate shadows. They imply the environment and what is happening outside of the canvas. Be it the sunlight through a day, an industrial set up or an explosion. Thinking about the word “expression”, there is a slight allusion for expressionism and neo-expressionism, but he would rather think of it as a physical term “to force out (something, such as the juice of a fruit) by pressure”.

The video is created procedurally in real time with no post-editing or post processing. With some optimization and restructuring, and by adding controls this can be live-performance ready. After Effects is used only to add music and for encode.

→ Construction and Rendering

Kynd tried to combine the best of 2D and 3D graphics to enable real-time creation of quite intricate surfaces and rendering with depths and interplay of light and shadow. The method enables him to still utilize a broad range of 2D techniques and add further depths to it (both literally and figuratively).

In essence the set up is quite simple and basic. Despite the rendering result appearing to have depths in 3D space, the date provided to the rendering process consists of just two 2D images – color map and height map. Different elements draw onto these two maps layering different shapes.

This method gives great advantages over certain constraints:

  • It is easier to create very intricate shapes and details without managing geometries = huge number of triangles and quads. 
  • It is fast and lightweight (on CPU – all the heavy weight lifting happen in GPU/Fragment shades)
  • It allows use of many 2D image processing methods, such as blurring, bleeding, deformation, blending, etc. that are useful to simulate paints and other materials. Shapes can be easily merged together, since it is just to compare the height from the ground level.  Metaballs-like smooth operation is also cheap in 2D.

Constraints of this method are:

  • Most obviously floating objects are not possible. Everything has to be built upon the canvas. This also limits the camera angle. 
  • Camera angle is limited. To get proper rendering result, the camera has to be in a certain range of angle relatively close to perpendicular to the surface.

Once all the construction is done, the renderer marches the rays in the space defined by these two maps and applies lighting and shadows.

→ Graphic Elements

The video composists of mostly two kinds of graphical elements. Some graphical elements are not necessarily directly visible. They can be negative shapes that curve or cut through other positive shapes. Light sources can be both autonomous or sound reactive.

Autonomous Elements act irrespective of the sound input. They are placed to the respective scenes to set up the foundation and create base movement.

Reactive Elements respond to the sound. Each reactive element has a selective part (e.g., frequency) of music associated.

→ Tools

Primarily TouchDesigner, with supportive use of openFrameworks and WebGL for prototyping.

Almost every graphics are created in fragment shaders written in GLSL, so this project could have been done on any other tools that support programming with OpenGL. TouchDesigner provides a good preview of each step in the process (the result from each single shader pass) and configurability (re-connecting shaders differently). For Kynd, the drawback of TouchDesigner or most of other node based tools is that they tend to force a certain way of thinking and make it hard to do beyond what existing nodes are designed for. Otherwise they tend to require too many nodes to do relatively simple things compared to their code equivalents. openFrameworks for Kynd is still the most flexible tool in that sense, and that was why most of the early prototyping was done using it. However once basic structure was figured out, he found these constraints don’t apply much to the project since the majority of the logic are in shader codes anyway. The benefit listed above surpassed the disadvantage.

→ Music

The music is composed purposefully for the video to create a unique sequence of scenes to introduce a variety of dynamics. Headphones strongly recommended, recordings of live improvisations on “Plogue Bidule” were collaged on “Ableton Live”. Aim, to sculpt the time and space as vividly as possible in stereo-channel in 2 minutes especially for those who get tired of repetitive structure and/or rigidness.

Kynd has kindly shared many work-in-progress videos with CAN, way too many to include here. We have made a small selection here (full screen recommended + sound on) but you can find many more by visiting the links below, in particular Kynd’s instagram or Twitter.

Kynd | Yu Miyashita | Instagram |on FRAMED

Leave a Reply

Your email address will not be published. Required fields are marked *