Out of Berlin, a city known for great design and shitty winter weather, comes Buchstabengewitter (letter storm), a generative typography project that transforms three-dimensional linear structures into all the letters of the alphabet. Developed by designer and programmer Ingo Italic the dynamic type display is only the latest experiment out of Letters Are My Friends, a research laboratory he runs together with Bärbel Bold. A place where, as the laser-cut sign outside says, “type meets new technology”

Continue reading.... Buchstabengewitter – Dynamic Type That Morphs from A to Z [vvvv]

Bowyer is a new company by Vadik Marmeladov, Sergey Filippov, a man known as “The Client”, and Ilya Kolganov. In this interview we talk through their first collaborations, philosophy, process, commercial projects, and future plans.

“Craftmanship is attention to details. Luxury is attention to unnecessary details.”

Typical phone games and apps retain a certain average appearance with a target on mass-market appeal. Design is usually not a priority, and as a result, often mediocre. Unlike the fashion industry, for instance, app development is young enough that nichés have not yet developed fully. Seeing this gap in the industry, Bowyer wished to target luxury markets.

“About 2 years ago while working on a commercial website we came up with a simple app for a luxury brand,” Vadik explained. The screen would be completely black, and through in-app purchases, individual diamonds could be bought and populated on the screen. The concept would be a reflection of physical diamonds: holding no real value other than the inherent scarcity and cost of ownership.

Continue reading.... An Interview with Bowyer – UIDesign + Funware + Luxury [Theory, iPad, Interview]

(This essay was commissioned by Centre national des arts plastiques for Graphisme en France 2012)

- Edited by Casey Reas and Chandler McWilliams -

Technical mastery and innovation are part of the rich history of visual design. The printing press is the quintessential example of how a shift in design technology can ripple through society. In the Twenty-First Century, innovation in design often means pushing the role of computers within the visual arts in new directions. Writing software is something that’s not typically associated with the work of a visual designer, but there’s a growing number of designers who write custom software as a component of their work. Over the last decade, through personal experience, We’ve learned many of the benefits and pitfalls of writing code as a component of a visual arts practice, but our experience doesn’t cover the full spectrum. Custom software is changing typography, photography, and composition and is the foundation for new categories of design practice that includes design for networked media (web browsers, mobile phones, tablets) and interactive installations. Most importantly, designers writing software are pushing design thinking into new areas. To cut to the core of the matter, we asked a group of exceptional designers two deceptively simple questions:

1. Why do you write your own software rather than only use existing software tools?

2. How does writing your own software affect your design process and also the visual qualities of the final work?

Continue reading.... Designing Programs [Theory]

The Product, a Berlin-based design studio with a focus on objects, space and interaction, was commissioned by Volkswagen to develop a set of visuals for an interactive musical performance for the premiere of the New Beetle at the 2011 IAA motor show (September 15-25, 2011, Frankfurt am Main). What the client got instead was Soundmachines – a custom-built instrument for performing electronic music by DJing visual patterns on record-sized discs. Honk for ingenuity

Continue reading.... Soundmachines [Arduino, Processing, Objects]

From Roger Ebert’s pedantic proclamation that “video games can never be art” to the clichéd fawning over the truckloads of revenue generated by each new release in the Modern Warfare series, gaming consistently inspires overarching conversations about media and culture. At this point, these ‘big conversations’ should surprise no one, as with each passing year gaming becomes less esoteric and permeates more and more demographic groups (e.g. the popularity of social games on Facebook, senior citizens embracing the Wii as an exercise platform, etc.). So while gaming may be everywhere, it is strange that it is often difficult to locate conversations about it that speak to how we actually integrate play and simulation into our everyday experience. What can games tell us about relaxation, work and routine? What do they have to say about movement and the body? How might we subvert gaming conventions through pranks and humour? Ian Bogost’s recent book How To Do Things With Videogames thoughtfully considers questions like these while endeavouring to re-frame the medium through a series of focused, topical texts that draw on familiar and engaging points of reference.

Organized as 20 bite-sized chapters, How To Do Things With Videogames carefully considers how gaming has been leveraged to explore sex, art, politics, branding and boredom – all the touchstones of contemporary life. Within each of these articles, Bogost carefully blends accessible pop culture references with illustrative gaming examples as the basis of his ruminations on how the medium functions as a cultural mirror. The “feel and weight” of Go pieces sets the stage for a meditation on haptic feedback, a FPS shootout set in the Manchester Cathedral serves as a gateway into a conversation about awe and reverence. Bogost’s knowledge of game history is encyclopedic and it is hard to come away from this book without a renewed appreciation for just how weird and wonderful game design can get – some of the more obscure references to SimHacks, mundane minigames and naive game tourism are priceless.

There are many compelling moments within the text, I’ve picked out two that I found particularly provocative.

Red Dead Redemption / Screen capture: Red Dead Wiki

In the chapter on “transit” Bogost sketches out a history of the moving image that considers panoramas, how the advent of rail altered the experience of landscape and the broader implications of movement in games:

Instead of looking forward to a future in which the risky, laborious process of traversing a space could be lessened, in-videogame transit re-creates a past in which reality had not yet been dissolved into bits, but had to be traversed deliberately. Like the panorama show, the transit simulation is a kind of replacement therapy for an inaccessible experience of movement.

Could this thesis be any more clearly articulated than in Red Dead Redemption, where the 21st century leisure class faux-nostalgically gallop across a simulated American Southwest on horseback?

How To Do Things With Videogames also unpacks how game design can complement and challenge corporate identity. The following passage is culled from a consideration of how games put brand identity under the microscope and (for me) it really evoked memories of the recent high speed turfing of Molleindustria’s Phone Story from the App Store.

Of course, unauthorized brand abuse in large commercial games might not be possible or desirable. But brands’ cultural values offer a bridge between visual appearance and game mechanics. In some cases, our understanding of particular rules of interaction has become bonded to products or services.

Phone Story sketched out a damning narrative of the consumer electronics industry’s reliance on conflict minerals and dubious labour practices and tells this story as playable narrative. The fact that a game that explicitly took aim at Apple’s supply chain ethics was so quickly ‘disappeared’ underscores the degree to which gamespace is contested (branding) ground. Bogost’s analysis of this milieu weaves together several examples of promotional games, an analysis of Monopoly tokens, commentary on Obama’s 2008 in-game ad purchases and also considers examples of “anti-advergame” critical resistance.

How To Do Things With Videogames is a lightning fast read and the book’s success is largely due to both brevity and charm. As a topical ‘scan’ of an entire medium, the undertaking is noteworthy for clearly articulating down to earth approaches for reconsidering the politics and experience of play. Bogost’s conclusion describes an ongoing process whereby games are becoming “more ordinary and familiar” and the cultural currency of ‘the gamer’ as a distinct subculture is fading – the tone and execution of this work certainly support this forecast. While a delight to muse over, this text should be read as a serious reconsideration of ‘first principles’ for anyone who plays, designs or avoids gaming on a regular basis.

Purchase on amazon.com / amazon.co.uk

We’ve heard it plenty of times when people are talking about working with the Kinect: “we’ll just get the point cloud and turn it a mesh”. You may have even thought that yourself at some point, “this is going to be easy” Well, it’s actually fairly difficult to do and especially difficult to do quickly, because a mesh is a complex object to create and update in realtime. In a mesh of either triangles or quads, each surface needs to be correctly oriented to it’s neighboring surfaces and to the camera. When working with a point cloud coming from the Kinect you have several thousand points which means several thousand pieces of geometry needing to be created all at once. The typical approach that people use is to just display the points and not worry about trying to make surfaces or shapes out of those points. While this is fine for some projects, we’d like to see more people dig into making real shapes from their Kinect imagery so we’re going to show you how to do that in two different ways. This is a fairly advanced tutorial that focuses a little more on theory and implementation rather than a specific toolset. What we’re going to talk about in this article can be applied to any of the creative coding toolkits, but we’ll be making projects for Cinder to demonstrate. If you don’t know anything about Cinder, I suggest you take a look here: http://libcinder.org/ and check it out. With that said, let’s get into some theory.

Point clouds are big and connecting those points in a 3D shape is slow. We need to make geometry and we need to make it fast, so we’re going to look at two approaches to using shaders to either generate our geometry and to position the RGB data that we receive from the camera. To do that we’re going to use shaders because they can be faster and they more neatly separate the code that accesses our peripherals, i.e. the Kinect, from the code that creates our geometry and displays the mesh.

A quick introduction to shaders

This is a very quick introction to shaders, for more detail there are a lot of resources that you can look at: The Orange Book, Nehe, even glsl.heroku.com/. For now though, we just want to make sure you get the basics: GLSL (Graphics Language Shading Language) is an abbreviation for the official OpenGL Shading Language. GLSL is a high-level programming language that’s similar to C/C++ for several parts of the graphics card. With GLSL, you can code short programs, called shaders, which are executed on the GPU. GLSL can be fairly confusing because graphics card support is a little bit unreliable, i.e. what works on one card may not work on another, and because the differences between versions is significant. However, whether you’re working with GL ES on mobile devices or WebGL, or with full GLSL in a Cinder application, the basics are the same, and an understanding of how the shading pipeline functions will serve you well in any OpenGL platform or with other shading tools like CG, HLSL, or ARB.

Before we get started though, you might want to know what version of GLSL you’re running. This isn’t mandatory, but it is pretty handy. Downloading and running the following: http://www.realtech-vr.com/glview/ will let you know what version of GLSL your computer supports. If you aren’t interested in that, or you already know, feel free to skip that step.

So, first, the basics: a shading language is a special programming language adapted to easily map on shader programming. Those kinds of languages usually have special data types such as color and normal. Because of the various target markets of 3D graphics, different shading languages have been developed: fragment shaders, geometry shaders, vertex shaders. In this article we’re interested in GLSL. GLSL shaders themselves are just text that is passed to the graphics card drivers for compilation from within an application using the OpenGL API’s entry points. Shaders can be created on the fly from within an application or read in as text files, but they must be sent to the driver in the form of a text string.

There are three fundamental steps in shading: read and modify any vertices, create or modify any geometry, and read and modify any pixel data. The OpenGL shader pipeline looks roughly like this:

Let’s break that down a bit.

Vertex Shaders

A vertex shader has attributes about a location in space or vertex, which means not only the actual coordinates of that location but also its color, how any textures should be mapped onto it, and how the vertices are modified in the operation. A vertex shader can change the positions of each vertex, the number of lighting computations per vertex, and the color that will be applied to each vertex.

Your application certainly doesn’t need to do all of these operations to use a vertex shader. As you’ll see in the next code example, you don’t need to apply lights to your objects. Even though you don’t need to use all the functionality of the vertex shader, when you create and apply a vertex shader, your program will assume that you’re replacing all the functionality of your other vertex transformations. If you apply changes to particular vertex, the vertex shader will probably change those according to the instructions contained in the vertex shader. When a vertex shader is used, it becomes responsible for replacing all the needed functionality of this stage of the pipeline.

The vertex processor processes each vertex individually and doesn’t store any data about the other vertices that it has already processed or that it will process. It is responsible for at least modifying (or not) the position of the vertex and then writing it to the next shaders, usually transforming the vertex with the ModelView and Projection matrices. It does this by writing to the gl_Position, which is where OpenGL stores the location of the vertex. A vertex processor has access to OpenGL state, so it can perform operations that involve lighting for instance, and use materials, and can even access textures.
So, what does a vertex shader look like? They can be quite simple. Take a look at the following example. It’s a single function that sets the value of each vertex passed to it. To be nice to everyone, no matter what graphics card they’re running, we need to make a small split here, showing two different ways of doing things.

GLSL 1.2

void main() {
    vec4 v = vec4(gl_Vertex);

As a quick example, the x position of the vertex is shifted over by 1 pixel:

v.x += 1;
Now the final position of the vertex is set:
    gl_Position = gl_ModelViewProjectionMatrix * v;
}

GLSL 1.3+

uniform mat4 mvp; // this is set for the entire shader, it doesn’t change for each point
in vec4 vertexIn; // this is set for each vertex, i.e it changes for each point

void main() {
    v.x += 1.;

Now the final position of the vertex is set:

gl_Position = mvp * vertexIn;
}

Here the vertex that is currently being operated on is used to create a new vertex: As a quick example, the x position of the vertex is shifted over by an arbitrary amount. That’s a very simple vertex shader that doesn’t alter your image in any immediately noticeable way, but it does show you how the vertex shader functions. In 1.2 the vertex passed to the shader can be accessed within that method as gl_Vertex, and its final position after the operation can be set using the gl_Position variable:

Geometry Shader

A geometry shader can generate new graphics primitives like points, lines, and triangles, from those primitives that were sent to the graphics card from the CPU. This means that you could get a point and turn it into a triangle or even a bunch of triangles, or get a line and turn it into a rectangle, or do real-time extrusion.

Geometry shader programs are executed after the vertex shader and before the fragment shader. As an example, if your oF application creates triangles and passes them to the vertex shader, then your geometry shader will receive one triangle at a time. What you do with those is up to you. You could, for instance, turn each triangle into 2 triangles or add additional points around each triangle. As long as you declare what your geometry shader will be getting in and what it will be churning out, everything will be passed to the fragment shader to be colored in without a problem. A very simple geometry shader looks like this:

GLSL 1.2

void main() {
  for(int i = 0; I < gl_VerticesIn; ++i) {
     gl_FrontColor = gl_FrontColorIn[i];
     gl_Position = gl_PositionIn[i];
    EmitVertex();
  }
}

GLSL 1.3+

in vec4 vertex;
out vec4 color;
void main() {
  for(int i = 0; i < gl_VerticesIn; ++i) {
    gl_Position = gl_PositionIn[i];
    color = vec4( i / 4., 1 - i/4., 0, 1.);
    EmitVertex();
  }
  EmitPrimitive();
}

The number of vertices being passed in with the gl_VerticesIn variable will be the same as the GL type that you’re using. If you’re using points, it’ll be one. If you’re using quads, it’ll be four. gl_Position is where you want to put the vertex that you’re creating and EmitVertex() is how you say that you’re done with a vertex. EmitPrimitive is how you tell the geometry shader that you’re done with creating geometry all together. The diagram below shows how you would create a quad out of GL_TRIANGLE_STRIP from a GL_POINT. This means that the vertex shader receives a point, manipulates its position, and then passes the coordinates of that point to the geometry shader. The geometry shader then generates the 4 points to create the quad using the received position, calls EmitPrimitive() when the quad is complete, and then passes the data on to the fragment shader.

Fragment Shader

The fragment shader is somewhat misleadingly named because what it really allows you to do is to change values assigned to each pixel. The vertex shader operates on the vertices, and the fragment shader operates on the pixels. By the time the fragment shader gets information passed into it by the graphics card, the color of a particular pixel has already been computed and in the fragment shader can be combined with an element like a lighting effecting, a fog effect, or a blur among many other options. The usual end result of this stage per fragment is a color value and a depth for the fragment.

The inputs of this stage are the pixel’s location and the fragment’s depth, whether it will be visible or altered by an effect, and color values.
Just like vertex shaders, fragment shaders must also have a main() function. Just as the vertex shader provides you access to the current vertex using the gl_Vertex variable, the fragment shader provides you access to the current fragment, that is, the fragment that the fragment shader is working with when it is called, using the gl_FragColor variable. Like the gl_vertex, this is a four-dimensional vector that lets you set the RGB and alpha properties of the fragment. In the following example, the color of the fragment is set to a medium gray with full alpha:

GLSL 1.2

#version 120
void main() {
        gl_FragColor = vec4(0.5, 0.5, 0.5, 1.0);
    }

GLSL 1.3+

#version 150
out vec4 color;
void main(void)
{
    color = vec4(1.0, 0.0, 0.0, 1.0);
}

Notice how each of the channels is set using a floating point number from 0.0 to 1.0. The gl_FragColor is what is passed to the renderer to be displayed on the screen.

What we want is to be able to get a point cloud from the IR camera of the kinect and an RGB image from the RGB camera, to display them you need to assemble them. You can do this on the CPU or you can do it on the GPU, but at some point everything needs to go to the GPU so it can be drawn by the graphics card. Of the types of shaders that we’re going to focus more on the geometry shader, since our goal is to be able to make a mesh out of the points that we get quickly and in a lightweight way.

Probably the best way to understand this is to dig into some small demos of fun things that you can do when putting together points and shaders.

To save space and to keep you interested, we’re going to just look at the important parts of the applciations that are available for download and closer browsing here.

The first application is a simple one, to introduce the basics and set the stage for what we’re going to do later on. As you can see in the picture below, we’ve got a control panel that allow you to control the drawing and the result shown behind the control panel.

We’re actually doing all the drawing in the geometry shader so we’ll unpack how that’s done. First, the Cinder application .cpp file. The github link is here. Take a look at the loadShaders() method:

// Find maximum number of output vertices for geometry shader
	int32_t maxGeomOutputVertices;
	glGetIntegerv(GL_MAX_GEOMETRY_OUTPUT_VERTICES_EXT, & maxGeomOutputVertices);

This is important: every graphics card will have a maximum number of vertices that it can output. For most modern cards this is 1024 or even more, but you want to set your shader to output only as many as you need. In this case, since we’re just demonstrating the basics, we’ll go with the max number that the card supports but in practice, you never want to do this. This is passed to the geometry shader when you actually load it onto graphics card. Each programming framework, Processing, WebGL, OF, Cinder, etc, has a slightly different syntax for this, but the basic idea is always the same: load the vertex shader, the fragment shader, and optionally the geometry shader. If you load a geometry shader you need to say what type of OpenGL primitive you’re passing to the geometry shader, what kind of OpenGL primitive you’re outputting, and the number of vertices that you’re going to be outputting. Here’s the cinder implementation that we’re using:

mShaderPassThru = gl::GlslProg(
	loadResource(RES_SHADER_VERT_150),
	loadResource(RES_SHADER_FRAG_150),
	loadResource(RES_SHADER_GEOM_150),
	GL_POINTS, GL_POINTS, 1); // note: 1 point out

mShaderTransform = gl::GlslProg(
	loadResource(RES_SHADER_VERT_150),
	loadResource(RES_SHADER_FRAG_150),
	loadResource(RES_SHADER_GEOM_150),
	GL_POINTS, GL_TRIANGLE_STRIP, maxGeomOutputVertices
	); // note max points out

The draw() method in the first application is quite simple, draw a bunch of points and you’re good to go.

// Draw points
	mShader.bind();
	glBegin(GL_POINTS);
	for (vector::const_iterator pointIt = mPoints.cbegin(); pointIt != mPoints.cend(); ++pointIt)
		gl::vertex(* pointIt);
	glEnd();
	mShader.unbind();

Again, that’s Cinder, but actual shader can be used in any programming framework, so let’s dig into that a little bit because the real trick here is making a geometry shader for points.

#version 120
#extension GL_EXT_geometry_shader4 : enable
#extension GL_EXT_gpu_shader4 : enable 

// Constants to match those in CPP
const int SHAPE_CIRCLE = 2;
const int SHAPE_SQUARE = 1;
const int SHAPE_TRIANGLE = 0;

// Uniforms
uniform float aspect;
uniform int shape;
uniform float size;
uniform bool transform;

// Kernel
void main(void)
{
	// Key on shape
	switch (shape)
	{

		case SHAPE_TRIANGLE:

To draw a triangle, we add the size times the cos and sin of each angle to the original position. After each point is set, we call EmitVertex() to the newly created vertex to the geometry that will be sent to the fragment shader. Be sure to draw counter-clockwise.

gl_Position = gl_PositionIn[0] + vec4(cos(radians(270.0)) * size, sin(radians(270.0)) * size * aspect, 0.0, 0.0);
		EmitVertex();
		gl_Position = gl_PositionIn[0] + vec4(cos(radians(150.0)) * size, sin(radians(150.0)) * size * aspect, 0.0, 0.0);
		EmitVertex();
		gl_Position = gl_PositionIn[0] + vec4(cos(radians(30.0)) * size, sin(radians(30.0)) * size * aspect, 0.0, 0.0);
		EmitVertex();

Call EndPrimitive() to close the shape. In this case we don’t strictly need this since we’re only drawing one primitive and the GLSL program can figure out that we’re done making our primitive, but when you have more than one primitive, it’s vitally important to call this whenever you’ve finished one of whatever type of primitive you’re creating.

EndPrimitive();

		break;
		case SHAPE_SQUARE:

To draw a square, we will draw two triangles, just like above. The two triangles will share vertices, so we’ll define the vertices in advance.

vec4 vert0 = vec4(cos(radians(315.0)) * size, sin(radians(315.0)) * size * aspect, 0.0, 0.0);
		vec4 vert1 = vec4(cos(radians(225.0)) * size, sin(radians(225.0)) * size * aspect, 0.0, 0.0);
		vec4 vert2 = vec4(cos(radians(135.0)) * size, sin(radians(135.0)) * size * aspect, 0.0, 0.0);
		vec4 vert3 = vec4(cos(radians(45.0)) * size, sin(radians(45.0)) * size * aspect, 0.0, 0.0);

			// Draw the left triangle
			gl_Position = gl_PositionIn[0] + vert0;
			EmitVertex();
			gl_Position = gl_PositionIn[0] + vert1;
			EmitVertex();
			gl_Position = gl_PositionIn[0] + vert3;
			EmitVertex();

			// Close this triangle
			EndPrimitive();

			// And now the right one
			gl_Position = gl_PositionIn[0] + vert3;
			EmitVertex();
			gl_Position = gl_PositionIn[0] + vert1;
			EmitVertex();
			gl_Position = gl_PositionIn[0] + vert2;
			EmitVertex();

			// Close the second triangle to form a quad
			EndPrimitive();

			break;
			case SHAPE_CIRCLE:

			// To draw a circle, we'll iterate through 24
			// segments and form triangles
			float twoPi = 6.283185306;
			float delta = twoPi / 24.0;
			for (float theta = 0.0; theta < twoPi; theta += delta)
			{
					// Draw a triangle to form a wedge of the circle
				gl_Position = gl_PositionIn[0] + vec4(cos(theta) * size, sin(theta) * size * aspect, 0.0, 0.0);
				EmitVertex();
				gl_Position = gl_PositionIn[0];
				EmitVertex();
				gl_Position = gl_PositionIn[0] + vec4(cos(theta + delta) * size, sin(theta + delta) * size * aspect, 0.0, 0.0);
				EmitVertex();
				EndPrimitive();
			}

			break;

		}

	}
}

So that’s pretty interesting, but next let’s make boxes for all those points and then things will get more interesting.

Actually, if you’ve done any OpenGL, making a cube using a geometry shader is going to look very familiar since it involves almost the same code as the glVertex() version, only with the addition of EmitVertex() and EmitPrimitive(). First though, we’re going to dig into the Cinder application to show a few new features.

We’re going to use a Vertex Buffer Object to create objects and points that we can work with in the geometry shader. If VBOs aren’t familiar to you and you want to learn more about how to work with them in Cinder, check out http://www.creativeapplications.net/tutorials/guide-to-meshes-in-cinder-cinder-tutorials/

First, set up the mesh using a vbo layout object:

// Set VBO layout, we use this to set up how the VBO is configured in the GPU memory
	mVboLayout.setStaticIndices();
	mVboLayout.setStaticPositions();

	// Iterate through the grid dimensions
	for (int32_t y = 0; y < mMeshHeight; y++) {
		for (int32_t x = 0; x < mMeshWidth; x++)
		{

			// Set the index of the vertex in the VBO so it is
			// numbered left to right, top to bottom
			mVboIndices.push_back(x + y * mMeshWidth);

			// Set the position of the vertex in world space
			mVboVertices.push_back(Vec3f((float)x - (float)mMeshWidth * 0.5f, (float)y - (float)mMeshHeight * 0.5f, 0.0f));

		}
	}

Now we actually add all the indices and positions to the VBOMesh:

mVboMesh = gl::VboMesh(mVboVertices.size(), mVboIndices.size(), mVboLayout, GL_POINTS);
	mVboMesh.bufferIndices(mVboIndices);
	mVboMesh.bufferPositions(mVboVertices);

Normally you do some modificiation of the VBO on the CPU in the update() method, but we’re interested in moving as much of our processing as possible to the GPU, so we’re not doing any of that. We do however need to be able to pass variables into our shader so we can generate the wave and correctly position the view onto the mesh. We do this using Uniforms, which as mentioend eaelir in the article are values passed into the shader on a per shader basis. Not every shader (vertex, geometry, fragment) needs to read them, but they are available for all shaders. Here’s how we pass all the values into the shaders and then draw the VBOMesh:

// Bind and configure shader
	mShader.bind();
	mShader.uniform("alpha", mMeshAlpha);
	mShader.uniform("amp", mMeshWaveAmplitude);
	mShader.uniform("eyePoint", mEyePoint);
	mShader.uniform("lightAmbient", mLightAmbient);
	mShader.uniform("lightDiffuse", mLightDiffuse);
	mShader.uniform("lightPosition", mLightPosition);
	mShader.uniform("lightSpecular", mLightSpecular);
	mShader.uniform("mvp", gl::getProjection() * gl::getModelView());
	mShader.uniform("phase", mElapsedSeconds);
	mShader.uniform("rotation", mBoxRotationSpeed);
	mShader.uniform("scale", mMeshScale);
	mShader.uniform("dimensions", Vec4f(mBoxDimensions));
	mShader.uniform("shininess", mLightShininess);
	mShader.uniform("speed", mMeshWaveSpeed);
	mShader.uniform("transform", mBoxEnabled);
	mShader.uniform("uvmix", mMeshUvMix);
	mShader.uniform("width", mMeshWaveWidth);

	// Draw VBO
	gl::draw(mVboMesh);

	// Stop drawing
	mShader.unbind();

Some of these lines are worth annotating a bit further. For instance:

mShader.uniform(“mvp”, gl::getProjection() * gl::getModelView());

In previous versions of GLSL, you could access the ModelView matrix through a built in variable. In GLSL 1.5 though you simply pass the matrix in using a uniform. This line:

mShader.uniform(“eyePoint”, mEyePoint);

allows us to position our generated geometry to the camera. This is really powerful because all you need is a 3D vector and you can correctly calculate the eye perspective onto a scene, I’m sure you can imagine how this will be very useful when we finally get to the Kinect portion of this tutorial. That eye position is vital, because it’s how we use the place that you want to look when we generate all the geometry. In the geometry shader you can see how these are all put to use:

uniform vec4 dimensions;
uniform mat4 mvp;
uniform float rotation;
uniform bool transform;

// Input attributes from the vertex shader
in vec4 vertex[];

// Output attributes
out vec4 normal;
out vec4 position;
out vec2 uv;

// Adds a vertex to the current primitive
void addVertex(vec2 texCoord, vec4 vert, vec4 norm)
{

	// Assign values to output attributes
	normal = norm;
	uv = texCoord;
	position = vert;
	gl_Position = vert;

	// Create vertex
	EmitVertex();

}

// Kernel
void main(void)
{

Now we need to calculate what the position should be based on the view so that we can send that on to the fragment shader.

position = mvp * vertex[0];

	// Transform point to box
	if (transform)
	{

		// Use Y position for phase
		float angle = vertex[0].y * rotation;

The rotation matrix we’re making here is what allows us to correctly position each box after twisting it to fit the shape of our wave. If these are a bit fuzzy for you, I find this to be an excellent refresher or introduction.

// Define rotation matrix
		mat4 rotMatrix;
		rotMatrix[0].x = 1.0 + cos(angle);
		rotMatrix[0].y = 1.0 + -sin(angle);
		rotMatrix[0].z = 1.0 + -rotMatrix[0].x;
		rotMatrix[0].w = 0.0;
		rotMatrix[1].x = 1.0 + -rotMatrix[0].z;
		rotMatrix[1].y = 1.0 + -rotMatrix[0].y;
		rotMatrix[1].z = 1.0 + -rotMatrix[0].x;
		rotMatrix[1].w = 0.0;
		rotMatrix[2].x = 0.0;
		rotMatrix[2].y = 0.0;
		rotMatrix[2].z = 1.0;
		rotMatrix[2].w = 0.0;
		rotMatrix[3].x = 0.0;
		rotMatrix[3].y = 0.0;
		rotMatrix[3].z = 0.0;
		rotMatrix[3].w = 1.0;

Now we need vertices for our box. This can broken down as:
vertex = the view matrix * (our position from the vertex shader + (the size of the box * the corner of the box that we're generating) * the amount that it should be twisted)

// Define vertices
		vec4 vert0 = mvp * (vertex[0] + vec4(dimensions * vec4(-1.0, -1.0, -1.0, 0.0)) * rotMatrix); // 0 ---
		vec4 vert1 = mvp * (vertex[0] + vec4(dimensions * vec4( 1.0, -1.0, -1.0, 0.0)) * rotMatrix); // 1 +--
		vec4 vert2 = mvp * (vertex[0] + vec4(dimensions * vec4(-1.0,  1.0, -1.0, 0.0)) * rotMatrix); // 2 -+-
		vec4 vert3 = mvp * (vertex[0] + vec4(dimensions * vec4( 1.0,  1.0, -1.0, 0.0)) * rotMatrix); // 3 ++-
		vec4 vert4 = mvp * (vertex[0] + vec4(dimensions * vec4(-1.0, -1.0,  1.0, 0.0)) * rotMatrix); // 4 --+
		vec4 vert5 = mvp * (vertex[0] + vec4(dimensions * vec4( 1.0, -1.0,  1.0, 0.0)) * rotMatrix); // 5 +-+
		vec4 vert6 = mvp * (vertex[0] + vec4(dimensions * vec4(-1.0,  1.0,  1.0, 0.0)) * rotMatrix); // 6 -++
		vec4 vert7 = mvp * (vertex[0] + vec4(dimensions * vec4( 1.0,  1.0,  1.0, 0.0)) * rotMatrix); // 7 +++

		// Define normals
		vec4 norm0 = mvp * vec4( 1.0,  0.0,  0.0, 0.0); // Right
		vec4 norm1 = mvp * vec4( 0.0,  1.0,  0.0, 0.0); // Top
		vec4 norm2 = mvp * vec4( 0.0,  0.0,  1.0, 0.0); // Front
		vec4 norm3 = mvp * vec4(-1.0,  0.0,  0.0, 0.0); // Left
		vec4 norm4 = mvp * vec4( 0.0, -1.0,  0.0, 0.0); // Bottom
		vec4 norm5 = mvp * vec4( 0.0,  0.0, -1.0, 0.0); // Back

		// Define UV coordinates
		vec2 uv0 = vec2(0.0, 0.0); // --
		vec2 uv1 = vec2(1.0, 0.0); // +-
		vec2 uv2 = vec2(1.0, 1.0); // ++
		vec2 uv3 = vec2(0.0, 1.0); // -+

Now we get in to simply making the sides of a cube:

// left
		addVertex(uv1, vert5, norm3);
		addVertex(uv2, vert7, norm3);
		addVertex(uv0, vert4, norm3);
		addVertex(uv3, vert6, norm3);
		EndPrimitive();

		// Right
		addVertex(uv3, vert3, norm0);
		addVertex(uv0, vert1, norm0);
		addVertex(uv2, vert2, norm0);
		addVertex(uv1, vert0, norm0);
		EndPrimitive();

		// Top
		addVertex(uv1, vert7, norm1);
		addVertex(uv2, vert3, norm1);
		addVertex(uv0, vert6, norm1);
		addVertex(uv3, vert2, norm1);
		EndPrimitive();

		// Bottom
		addVertex(uv1, vert1, norm4);
		addVertex(uv2, vert5, norm4);
		addVertex(uv0, vert0, norm4);
		addVertex(uv3, vert4, norm4);
		EndPrimitive();

		// Front
		addVertex(uv2, vert3, norm2);
		addVertex(uv3, vert7, norm2);
		addVertex(uv1, vert1, norm2);
		addVertex(uv0, vert5, norm2);
		EndPrimitive();

		// Back
		addVertex(uv2, vert6, norm5);
		addVertex(uv1, vert2, norm5);
		addVertex(uv3, vert4, norm5);
		addVertex(uv0, vert0, norm5);
		EndPrimitive();

	}
	else
	{

		// Pass-thru
		gl_Position = position;
		EmitVertex();
		EndPrimitive();

	}

}

Now the fragment shader and yes, this seems a bit complex, but is actually fairly simple.

#version 150

// Uniforms
uniform float alpha;
uniform vec3 eyePoint;
uniform vec4 lightAmbient;
uniform vec4 lightDiffuse;
uniform vec4 lightSpecular;
uniform vec3 lightPosition;
uniform float shininess;
uniform bool transform;
uniform float uvmix;

The geometry shader is outputting a normal, position, and uv coordinate for the fragment shader to use in coloring:

// Input attributes
in vec4 normal;
in vec4 position;
in vec2 uv;

And here’s what is actually output:

// Output attributes
out vec4 color;

// Kernel
void main(void)
{
	// Transform point to box
	if (transform)
	{

		// Initialize color
		color = vec4(0.0, 0.0, 0.0, 0.0);

		// Normalized eye position
		vec3 eye = normalize(-eyePoint);

Lights are really a light position and a reflection vector. That reflection needs to take the position of the light and normal of the surface into account to look “right”, so we normalize the reflected light vector using the reflect() method, which is built into GLSL.

// Calculate light and reflection positions
		vec3 light = normalize(lightPosition.xyz - position.xyz);
		vec3 reflection = normalize(-reflect(light, normal.xyz));

This is where what people think of as materials are actually generated: putting together the value of a light, a surface color, and a reflection together mathematically. That becomes the color of our pixel and voila, our geometry is generated and colored correctly to match what we’d expect from the color of the box and the light shining onto it.

// Calculate ambient, diffuse, and specular values
		vec4 ambient = lightAmbient;
		vec4 diffuse = clamp(lightDiffuse * max(dot(normal.xyz, light), 0.0), 0.0, 1.0);
		vec4 specular = clamp(lightSpecular * pow(max(dot(reflection, eye), 0.0), 0.3 * shininess), 0.0, 1.0); 

		// Set color from light
		color = ambient + diffuse + specular;

		// Mix with UV map
		color = mix(color, vec4(uv.s, uv.t, 1.0, 1.0), uvmix);

		// Set alpha
		color.a = color.a * alpha;

	}
	else
	{

		// Color point white
		color = vec4(1.0, 1.0, 1.0, 1.0);

	}

}

Here’s another view of what that looks like:

So now you’ve seen how to make geometry and give it color, we’re going to let you take a breather and in part 2, generate some geometry from a Kinect point cloud data using these techniques. It’s gonna be awesome :)

Puppet Parade is an interactive installation by Emily Gobeille and Theo Watson of Design I/O that allows children to use their arms to puppeteer larger than life creatures projected on the wall in front of them. This dual interactive setup allows children to perform alongside the puppets, blurring the line between the ‘audience’ and the puppeteers and creating an endlessly playful dialogue between the children in the space and the children puppeteering the creatures.

Continue reading.... Puppet Parade [openFrameworks]

All of the visuals in the above video were created using NyArtoolkit for Processing. NyARToolkit is an augmented reality toolkit built with 100% pure Java. It is derived from ARToolkit-2.72.1. Like Processing itself it’s open source and free! In this tutorial you will learn how to use it to place computer generated imagery correctly onto real world footage. To do this in real-time NyArtoolkit uses markers – black and white images – to determine the three-dimensional position and orientation in the real world. Most likely you will have seen something like this before, but now you will learn how to do it yourself using freely available software.

All right so let’s start with the general setup. What do you need to get started?

Continue reading.... Augmented Reality with Processing [Tutorial, Processing]

Another year of iPhone/iPad development but there haven’t been many apps that have caught our eye as there have been in the past. Nevertheless few that stood above the rest, both in their conception and how they make us think about technology. As an increasing number of apps saturate the AppStore, now in their 100s of thousands, we’d like to remain optimistic that this platform will continue to grow and not just by large development studios and those that reiterate but rather as an exciting, creative and above all – “innovative” place to push ideas to.

10. Composite [iPad, openFrameworks]

Inspired by the neo-dadaist collages of Robert Rauschenberg, James Alliban’s Composite allows you to remix your surroundings to create graphic compositions. Users can paint pictures using live video stream by simply pointing their iPad towards your subject drawing over it. Pixels are captured and transfered onto canvas. A different way of looking at painting.

–

9. Photo/Nykto [iPad, openFrameworks, Games]

Photo/Nykto is an experimental game conceived by Annelore Schneider and Douglas Edric Stanley as part of the “Unterplay” project at the Master Media Design —HEAD, Genève. Edric describes at as a game for nyktophobes and photophobes. It is played by switching on and off the lights in order to avoid reaching the edge of the screen. Photo/Nykto is one of the few apps that explores interaction with the device beyond the device itself. We like to think of it as “spatial interface” one that allows iPad to become an aggregator of physical activity.

Soon in the AppStore

–

8. Planetary [Cinder, iPad]

Created by the collective consisting of Ben Cerveny, Tom Carden, Jesper Sparre Andersen and Robert Hodgin, Planetary by Bloom is a way to explore your music collection using planetary system. Application, created using Cinder framework allows you to navigate dynamically created by information about the music on your iPad. Fly thorough the stars that represent your favourite artists, visit planets (albums) or listen to the moons (tracks).

–

7. Windosill [iPad, Games]

Previously available for Mac and Windows, we are glad to see Windosill by Patrick Smith (Vectorpark.com) finally make it’s way to the iOS.  An adventure puzzle game filled with peculiar objects, your task is to resolve abstract but yet logical puzzles to progress from room to room. For fans of the original, this is not just a simple port. The Windosill for the iPad was built as a native iOS app and customized the behavior of every element to the iPad’s touch interface. Bonus features include a sketchbook gallery of concept and development artwork, the option to instantly skip to any room you’ve already completed and two special settings, unlocked when you complete Windosill: Complex Gravity, which allows you to manipulate objects by tipping your iPad, and See-Thru Mode, which makes everything translucent (giving you a sneak-peek into how Windosill is put together). Fantastic addition to the AppStore.

–

6. OscilloScoop [iPhone, iPad, Sound]

Created by Scott Snibbe and Graham McDermott and originally designed by Lukas Girling, “OscilloScoop” is a culmination of about 15 years’ effort trying to create musical creation tools that is more like a video game. The process began back in the 90′s at a research lab when Scott Snibbe and Lukas Girling worked briefly with Brian Eno and Laurie Anderson on some of the concepts. The app, playfully called OscilloScoop, presents a trio of brightly colored stacked spinning crowns. Touching a crown trims or builds up its edge, like a spinning disk of clay. As you carve into this disc, the music changes, and you effortlessly produce hip hop, techno, electro, and other recognisable forms of music. Scott Snibble describes the process much like a DJ spinning records, but you create original music of your own, rather than merely cutting between tracks.

–

5. The Infinite Adventure Machine [MaxMSP, iPad]

The Infinite Adventure Machine by David Benqué is a computer program which generates fairy-tale plots. Based on the work of Vladimir Propp, who reduced the structure of russian folk-tales to 31 basic functions, the project addresses the difficulties of automatic story generation which David explains remain an unsolved problem for computer science.

Not available – Prototype

–

4. field [iPhone, iPad, openFrameworks, Sound]

Created by Rainer Kohlberger with sound by Wilm Thoben, field is an abstract audiovisual app that uses realtime camera feed as input. Brightness, saturation and color are interpreted, and translated into a constructed grid. The realtime image triggers different sounds as you pan around. Included are five different modes which you can switch through by double tapping the screen. Very addictive and playful take on  live video as medium to generate both visual and sound patterns.

–

3. Last Clock [iPad]

Originally created in 2002 by Jussi Ängeslevä and Ross Cooper when at the Royal College of Arts in London and developed for the iOS by NewMediology (Danqing Shi), Last clock is a clock app that uses popular slit scan technique to keep you in factual time, human time and remote time. Just like any other analog clock, the app has three hands: one for seconds, minutes and hours. The hands, however, are made of a slice of live video that gets scanned to the clockface. With different refresh rate for the three hands, the three time circles reflect the rhythms of the space at different temporal resolutions. The app also allows you to stream the last clock camera feed over the internet.

–

2. Sword & Sworcery EP [Games, iPhone, iPad]

The long awaited “21st century interpretation of the archetypical old school videogame adventure” by Superbrothers finally hit the AppStore this year. The response was amazing quickly became one of the most popular games of the year. If you haven’t had a chance to play with it, xmas break seems like a perfect opportunity. Easily the best iOS game this year.

–

1. Björk – Biophilia [iPhone, iPad, Sound]

Having seen many apps over the last few years, I don’t think anyone expected Biophilia. If you don’t know about it, is an iPhone/iPad release of Björk’s latest album created in collaboration with Scott Snibbe and her longtime design collaborators M/M (Paris). Comprising a suite of musical pieces and interactive artworks, Biophilia is a unique experience where different elements are weaved together with both sensitivity and precision. The experience is unified, building on different layers of visuals and sound. It’s a simple must! (Read also our special Making-of Virus)

It’s that time of the year when we slowly begin wrapping up the 2011. Before we say goodbye, lets take a quick moment to look back at some of the best and most memorable projects of 2011. There is no mechanism in deciding these and neither-nor all projects here are spectacular and magnificent but instead we feel they offer a brief insight and a form of introduction to what may lie ahead.

10. MIT Media Lab identity [Processing]

In 2010 MIT Media Lab approached TheGreenEyl and E Roon Kang to update their identity for the lab’s 25th anniversary. The team developed an algorithmic logo using Processing in an effort to capture the dynamism and diversity of the MIT Media Lab. The invisible grid around which different things happen symbolizes the environment the MIT Media Lab provides, academically, physically and intellectually.

9. Tele-Present Water [Arduino, MaxMSP]

Created by David Bowen, Tele-Present Water installation draws information from the intensity and movement of the water in a remote location. The wave intensity and frequency is scaled and transferred to the mechanical grid structure resulting in a simulation of the physical effects caused by the movement of water from this distant location. Elegance and ephemerality of data beautifully staged by David Bowen.

8. ExtraFile File Formats [Mac]

In the world populated by file formats and continued fight over ownership, open source and value of data, Kim Asendorf created ExtraFile, a conceptional software with practical usage. It’s main intention is to offer an alternative to the static system of image file formats. It is also an attempt to address an alternative image file format, a piece of art, far away from the mainstream and commercial standards.

7. Ping! Augmented Pixel

In the decade when videogames were born, everything virtual looked like rectangular blocks. From today’s perspective, the representation of a tennis court in the earliest videogames is hard to distinguish from a soccer or a basketball field. ‘PING! – Augmented Pixel’ is a videogame by Niklas Roy and one of the very few augmented reality projects that really explores the space between physical and digital. Niklas built a custom piece of hardware the intersects the signal between the video camera and television, inserting a classic style Pong game that you control using a finger positioned between the two mediums. Simplicity and ingenuity at it’s best.

6. Little Printer [Objects]

2011 wil be remembered as the year of thermal printers. Just as we have forgotten about the tech, a wave of projects have spread across the web. One that especially stands out is the Little Printer by BERG and giving us an “alternative” insight into how we may interact with digital information int he future. Little Printer may live in your home, bringing you news, puzzles and gossip from friends. Use your smartphone to set up subscriptions and Little Printer will gather them together to create a timely mini-newspaper.

5. Written Images [openFrameworks, Processing]

Although first launched in 2010, Written Images book finally landed on our desks, in it’s full glory it’s an unique artifact, printed in very limited numbers, is the first of it’s kind – a ‘programmed book’. Created in collaboration with more than 70 media artists and developers from across the world, Written Images is continuously regenerated for the digital printing process, offering each reader a unique experience. Each artist programed an image creating application that is generated for each print of the book.

4. Cascade [Processing]

Cascade is a by NYTimes R&D department that allows precise analysis of the structures that underly sharing activity on the web. Initiated by Mark Hansen and working with Jer Thorp and Jake Porway (Data Scientist at the Times) the team spent 6 months building the tool to understand how information propagates through the social media space. While initially applied to New York Times stories and information, the tool and its underlying logic may be applied to any publisher or brand interested in understanding how its messages are shared.

3. Particles [openFrameworks, Arduino, Events]

Particles was an installation by Daito Manabe and Motoi Ishibashi exhibited at the Yamaguchi Center for Arts and Media [YCAM]. The installation centers around a spiral-shaped rail construction on which a number of balls with built-in LEDs and xbee transmitters are rolling while blinking in different time intervals. The idea was driven by simple desire to both understand and represent particle behaviour in physical form. Besides the sheer complexity of the structure installed, the position of each ball is determined via total of 17 control points on the rail. Every time a ball passes through one of them, the respective ball’ s positional information is transmitted via a built-in infrared sensor. During the time the ball travels between one control points to the next, this position is calculated based on its average speed. The data for regulating the balls’ luminescence are divided by the control point segments and are switched every time a ball passes on a control point. Fantastic work Daito + Motoi!

2. Communion [Cinder]

Earlier this year on show at the La Gaîté Lyrique in Paris and created by Universal Everything with FIELD, Communion is a room sized installation creating almost a 360º environment. The final piece includes an array or evolving creatures going through stages of development – evolution from simple to complex with human like properties with generative behaviours, deeply immersive and “a celebration of an audio visual synesthetic experience”. Read our special.

1. Solar Sinter [Objects, Arduino]

Solar Sinter by Markus Kayser is most definitely one of the most inspiring projects this year, aiming to raise questions about, energy and manufacturing. Markus designed a 3d printed that uses solar rays and silicia and heated sand to solidify it as glass in any shape or form. A combination of Arduino, ReplicatorG software, laptop computer and a bunch of custom made components, Markus took the printer to Sahara desert near Siwa, Egypt, for a two week testing period. CAN is proud to be the first to write about the project, the video on Vimeo now measures almost a million views.