We’ve heard it plenty of times when people are talking about working with the Kinect: “we’ll just get the point cloud and turn it a mesh”. You may have even thought that yourself at some point, “this is going to be easy” Well, it’s actually fairly difficult to do and especially difficult to do quickly, because a mesh is a complex object to create and update in realtime. In a mesh of either triangles or quads, each surface needs to be correctly oriented to it’s neighboring surfaces and to the camera. When working with a point cloud coming from the Kinect you have several thousand points which means several thousand pieces of geometry needing to be created all at once. The typical approach that people use is to just display the points and not worry about trying to make surfaces or shapes out of those points. While this is fine for some projects, we’d like to see more people dig into making real shapes from their Kinect imagery so we’re going to show you how to do that in two different ways. This is a fairly advanced tutorial that focuses a little more on theory and implementation rather than a specific toolset. What we’re going to talk about in this article can be applied to any of the creative coding toolkits, but we’ll be making projects for Cinder to demonstrate. If you don’t know anything about Cinder, I suggest you take a look here: http://libcinder.org/ and check it out. With that said, let’s get into some theory.

Point clouds are big and connecting those points in a 3D shape is slow. We need to make geometry and we need to make it fast, so we’re going to look at two approaches to using shaders to either generate our geometry and to position the RGB data that we receive from the camera. To do that we’re going to use shaders because they can be faster and they more neatly separate the code that accesses our peripherals, i.e. the Kinect, from the code that creates our geometry and displays the mesh.

A quick introduction to shaders

This is a very quick introction to shaders, for more detail there are a lot of resources that you can look at: The Orange Book, Nehe, even glsl.heroku.com/. For now though, we just want to make sure you get the basics: GLSL (Graphics Language Shading Language) is an abbreviation for the official OpenGL Shading Language. GLSL is a high-level programming language that’s similar to C/C++ for several parts of the graphics card. With GLSL, you can code short programs, called shaders, which are executed on the GPU. GLSL can be fairly confusing because graphics card support is a little bit unreliable, i.e. what works on one card may not work on another, and because the differences between versions is significant. However, whether you’re working with GL ES on mobile devices or WebGL, or with full GLSL in a Cinder application, the basics are the same, and an understanding of how the shading pipeline functions will serve you well in any OpenGL platform or with other shading tools like CG, HLSL, or ARB.

Before we get started though, you might want to know what version of GLSL you’re running. This isn’t mandatory, but it is pretty handy. Downloading and running the following: http://www.realtech-vr.com/glview/ will let you know what version of GLSL your computer supports. If you aren’t interested in that, or you already know, feel free to skip that step.

So, first, the basics: a shading language is a special programming language adapted to easily map on shader programming. Those kinds of languages usually have special data types such as color and normal. Because of the various target markets of 3D graphics, different shading languages have been developed: fragment shaders, geometry shaders, vertex shaders. In this article we’re interested in GLSL. GLSL shaders themselves are just text that is passed to the graphics card drivers for compilation from within an application using the OpenGL API’s entry points. Shaders can be created on the fly from within an application or read in as text files, but they must be sent to the driver in the form of a text string.

There are three fundamental steps in shading: read and modify any vertices, create or modify any geometry, and read and modify any pixel data. The OpenGL shader pipeline looks roughly like this:

Let’s break that down a bit.

Vertex Shaders

A vertex shader has attributes about a location in space or vertex, which means not only the actual coordinates of that location but also its color, how any textures should be mapped onto it, and how the vertices are modified in the operation. A vertex shader can change the positions of each vertex, the number of lighting computations per vertex, and the color that will be applied to each vertex.

Your application certainly doesn’t need to do all of these operations to use a vertex shader. As you’ll see in the next code example, you don’t need to apply lights to your objects. Even though you don’t need to use all the functionality of the vertex shader, when you create and apply a vertex shader, your program will assume that you’re replacing all the functionality of your other vertex transformations. If you apply changes to particular vertex, the vertex shader will probably change those according to the instructions contained in the vertex shader. When a vertex shader is used, it becomes responsible for replacing all the needed functionality of this stage of the pipeline.

The vertex processor processes each vertex individually and doesn’t store any data about the other vertices that it has already processed or that it will process. It is responsible for at least modifying (or not) the position of the vertex and then writing it to the next shaders, usually transforming the vertex with the ModelView and Projection matrices. It does this by writing to the gl_Position, which is where OpenGL stores the location of the vertex. A vertex processor has access to OpenGL state, so it can perform operations that involve lighting for instance, and use materials, and can even access textures.
So, what does a vertex shader look like? They can be quite simple. Take a look at the following example. It’s a single function that sets the value of each vertex passed to it. To be nice to everyone, no matter what graphics card they’re running, we need to make a small split here, showing two different ways of doing things.

GLSL 1.2

void main() {
    vec4 v = vec4(gl_Vertex);

As a quick example, the x position of the vertex is shifted over by 1 pixel:

v.x += 1;
Now the final position of the vertex is set:
    gl_Position = gl_ModelViewProjectionMatrix * v;
}

GLSL 1.3+

uniform mat4 mvp; // this is set for the entire shader, it doesn’t change for each point
in vec4 vertexIn; // this is set for each vertex, i.e it changes for each point

void main() {
    v.x += 1.;

Now the final position of the vertex is set:

gl_Position = mvp * vertexIn;
}

Here the vertex that is currently being operated on is used to create a new vertex: As a quick example, the x position of the vertex is shifted over by an arbitrary amount. That’s a very simple vertex shader that doesn’t alter your image in any immediately noticeable way, but it does show you how the vertex shader functions. In 1.2 the vertex passed to the shader can be accessed within that method as gl_Vertex, and its final position after the operation can be set using the gl_Position variable:

Geometry Shader

A geometry shader can generate new graphics primitives like points, lines, and triangles, from those primitives that were sent to the graphics card from the CPU. This means that you could get a point and turn it into a triangle or even a bunch of triangles, or get a line and turn it into a rectangle, or do real-time extrusion.

Geometry shader programs are executed after the vertex shader and before the fragment shader. As an example, if your oF application creates triangles and passes them to the vertex shader, then your geometry shader will receive one triangle at a time. What you do with those is up to you. You could, for instance, turn each triangle into 2 triangles or add additional points around each triangle. As long as you declare what your geometry shader will be getting in and what it will be churning out, everything will be passed to the fragment shader to be colored in without a problem. A very simple geometry shader looks like this:

GLSL 1.2

void main() {
  for(int i = 0; I < gl_VerticesIn; ++i) {
     gl_FrontColor = gl_FrontColorIn[i];
     gl_Position = gl_PositionIn[i];
    EmitVertex();
  }
}

GLSL 1.3+

in vec4 vertex;
out vec4 color;
void main() {
  for(int i = 0; i < gl_VerticesIn; ++i) {
    gl_Position = gl_PositionIn[i];
    color = vec4( i / 4., 1 - i/4., 0, 1.);
    EmitVertex();
  }
  EmitPrimitive();
}

The number of vertices being passed in with the gl_VerticesIn variable will be the same as the GL type that you’re using. If you’re using points, it’ll be one. If you’re using quads, it’ll be four. gl_Position is where you want to put the vertex that you’re creating and EmitVertex() is how you say that you’re done with a vertex. EmitPrimitive is how you tell the geometry shader that you’re done with creating geometry all together. The diagram below shows how you would create a quad out of GL_TRIANGLE_STRIP from a GL_POINT. This means that the vertex shader receives a point, manipulates its position, and then passes the coordinates of that point to the geometry shader. The geometry shader then generates the 4 points to create the quad using the received position, calls EmitPrimitive() when the quad is complete, and then passes the data on to the fragment shader.

Fragment Shader

The fragment shader is somewhat misleadingly named because what it really allows you to do is to change values assigned to each pixel. The vertex shader operates on the vertices, and the fragment shader operates on the pixels. By the time the fragment shader gets information passed into it by the graphics card, the color of a particular pixel has already been computed and in the fragment shader can be combined with an element like a lighting effecting, a fog effect, or a blur among many other options. The usual end result of this stage per fragment is a color value and a depth for the fragment.

The inputs of this stage are the pixel’s location and the fragment’s depth, whether it will be visible or altered by an effect, and color values.
Just like vertex shaders, fragment shaders must also have a main() function. Just as the vertex shader provides you access to the current vertex using the gl_Vertex variable, the fragment shader provides you access to the current fragment, that is, the fragment that the fragment shader is working with when it is called, using the gl_FragColor variable. Like the gl_vertex, this is a four-dimensional vector that lets you set the RGB and alpha properties of the fragment. In the following example, the color of the fragment is set to a medium gray with full alpha:

GLSL 1.2

#version 120
void main() {
        gl_FragColor = vec4(0.5, 0.5, 0.5, 1.0);
    }

GLSL 1.3+

#version 150
out vec4 color;
void main(void)
{
    color = vec4(1.0, 0.0, 0.0, 1.0);
}

Notice how each of the channels is set using a floating point number from 0.0 to 1.0. The gl_FragColor is what is passed to the renderer to be displayed on the screen.

What we want is to be able to get a point cloud from the IR camera of the kinect and an RGB image from the RGB camera, to display them you need to assemble them. You can do this on the CPU or you can do it on the GPU, but at some point everything needs to go to the GPU so it can be drawn by the graphics card. Of the types of shaders that we’re going to focus more on the geometry shader, since our goal is to be able to make a mesh out of the points that we get quickly and in a lightweight way.

Probably the best way to understand this is to dig into some small demos of fun things that you can do when putting together points and shaders.

To save space and to keep you interested, we’re going to just look at the important parts of the applciations that are available for download and closer browsing here.

The first application is a simple one, to introduce the basics and set the stage for what we’re going to do later on. As you can see in the picture below, we’ve got a control panel that allow you to control the drawing and the result shown behind the control panel.

We’re actually doing all the drawing in the geometry shader so we’ll unpack how that’s done. First, the Cinder application .cpp file. The github link is here. Take a look at the loadShaders() method:

// Find maximum number of output vertices for geometry shader
	int32_t maxGeomOutputVertices;
	glGetIntegerv(GL_MAX_GEOMETRY_OUTPUT_VERTICES_EXT, & maxGeomOutputVertices);

This is important: every graphics card will have a maximum number of vertices that it can output. For most modern cards this is 1024 or even more, but you want to set your shader to output only as many as you need. In this case, since we’re just demonstrating the basics, we’ll go with the max number that the card supports but in practice, you never want to do this. This is passed to the geometry shader when you actually load it onto graphics card. Each programming framework, Processing, WebGL, OF, Cinder, etc, has a slightly different syntax for this, but the basic idea is always the same: load the vertex shader, the fragment shader, and optionally the geometry shader. If you load a geometry shader you need to say what type of OpenGL primitive you’re passing to the geometry shader, what kind of OpenGL primitive you’re outputting, and the number of vertices that you’re going to be outputting. Here’s the cinder implementation that we’re using:

mShaderPassThru = gl::GlslProg(
	loadResource(RES_SHADER_VERT_150),
	loadResource(RES_SHADER_FRAG_150),
	loadResource(RES_SHADER_GEOM_150),
	GL_POINTS, GL_POINTS, 1); // note: 1 point out

mShaderTransform = gl::GlslProg(
	loadResource(RES_SHADER_VERT_150),
	loadResource(RES_SHADER_FRAG_150),
	loadResource(RES_SHADER_GEOM_150),
	GL_POINTS, GL_TRIANGLE_STRIP, maxGeomOutputVertices
	); // note max points out

The draw() method in the first application is quite simple, draw a bunch of points and you’re good to go.

// Draw points
	mShader.bind();
	glBegin(GL_POINTS);
	for (vector::const_iterator pointIt = mPoints.cbegin(); pointIt != mPoints.cend(); ++pointIt)
		gl::vertex(* pointIt);
	glEnd();
	mShader.unbind();

Again, that’s Cinder, but actual shader can be used in any programming framework, so let’s dig into that a little bit because the real trick here is making a geometry shader for points.

#version 120
#extension GL_EXT_geometry_shader4 : enable
#extension GL_EXT_gpu_shader4 : enable 

// Constants to match those in CPP
const int SHAPE_CIRCLE = 2;
const int SHAPE_SQUARE = 1;
const int SHAPE_TRIANGLE = 0;

// Uniforms
uniform float aspect;
uniform int shape;
uniform float size;
uniform bool transform;

// Kernel
void main(void)
{
	// Key on shape
	switch (shape)
	{

		case SHAPE_TRIANGLE:

To draw a triangle, we add the size times the cos and sin of each angle to the original position. After each point is set, we call EmitVertex() to the newly created vertex to the geometry that will be sent to the fragment shader. Be sure to draw counter-clockwise.

gl_Position = gl_PositionIn[0] + vec4(cos(radians(270.0)) * size, sin(radians(270.0)) * size * aspect, 0.0, 0.0);
		EmitVertex();
		gl_Position = gl_PositionIn[0] + vec4(cos(radians(150.0)) * size, sin(radians(150.0)) * size * aspect, 0.0, 0.0);
		EmitVertex();
		gl_Position = gl_PositionIn[0] + vec4(cos(radians(30.0)) * size, sin(radians(30.0)) * size * aspect, 0.0, 0.0);
		EmitVertex();

Call EndPrimitive() to close the shape. In this case we don’t strictly need this since we’re only drawing one primitive and the GLSL program can figure out that we’re done making our primitive, but when you have more than one primitive, it’s vitally important to call this whenever you’ve finished one of whatever type of primitive you’re creating.

EndPrimitive();

		break;
		case SHAPE_SQUARE:

To draw a square, we will draw two triangles, just like above. The two triangles will share vertices, so we’ll define the vertices in advance.

vec4 vert0 = vec4(cos(radians(315.0)) * size, sin(radians(315.0)) * size * aspect, 0.0, 0.0);
		vec4 vert1 = vec4(cos(radians(225.0)) * size, sin(radians(225.0)) * size * aspect, 0.0, 0.0);
		vec4 vert2 = vec4(cos(radians(135.0)) * size, sin(radians(135.0)) * size * aspect, 0.0, 0.0);
		vec4 vert3 = vec4(cos(radians(45.0)) * size, sin(radians(45.0)) * size * aspect, 0.0, 0.0);

			// Draw the left triangle
			gl_Position = gl_PositionIn[0] + vert0;
			EmitVertex();
			gl_Position = gl_PositionIn[0] + vert1;
			EmitVertex();
			gl_Position = gl_PositionIn[0] + vert3;
			EmitVertex();

			// Close this triangle
			EndPrimitive();

			// And now the right one
			gl_Position = gl_PositionIn[0] + vert3;
			EmitVertex();
			gl_Position = gl_PositionIn[0] + vert1;
			EmitVertex();
			gl_Position = gl_PositionIn[0] + vert2;
			EmitVertex();

			// Close the second triangle to form a quad
			EndPrimitive();

			break;
			case SHAPE_CIRCLE:

			// To draw a circle, we'll iterate through 24
			// segments and form triangles
			float twoPi = 6.283185306;
			float delta = twoPi / 24.0;
			for (float theta = 0.0; theta < twoPi; theta += delta)
			{
					// Draw a triangle to form a wedge of the circle
				gl_Position = gl_PositionIn[0] + vec4(cos(theta) * size, sin(theta) * size * aspect, 0.0, 0.0);
				EmitVertex();
				gl_Position = gl_PositionIn[0];
				EmitVertex();
				gl_Position = gl_PositionIn[0] + vec4(cos(theta + delta) * size, sin(theta + delta) * size * aspect, 0.0, 0.0);
				EmitVertex();
				EndPrimitive();
			}

			break;

		}

	}
}

So that’s pretty interesting, but next let’s make boxes for all those points and then things will get more interesting.

Actually, if you’ve done any OpenGL, making a cube using a geometry shader is going to look very familiar since it involves almost the same code as the glVertex() version, only with the addition of EmitVertex() and EmitPrimitive(). First though, we’re going to dig into the Cinder application to show a few new features.

We’re going to use a Vertex Buffer Object to create objects and points that we can work with in the geometry shader. If VBOs aren’t familiar to you and you want to learn more about how to work with them in Cinder, check out http://www.creativeapplications.net/tutorials/guide-to-meshes-in-cinder-cinder-tutorials/

First, set up the mesh using a vbo layout object:

// Set VBO layout, we use this to set up how the VBO is configured in the GPU memory
	mVboLayout.setStaticIndices();
	mVboLayout.setStaticPositions();

	// Iterate through the grid dimensions
	for (int32_t y = 0; y < mMeshHeight; y++) {
		for (int32_t x = 0; x < mMeshWidth; x++)
		{

			// Set the index of the vertex in the VBO so it is
			// numbered left to right, top to bottom
			mVboIndices.push_back(x + y * mMeshWidth);

			// Set the position of the vertex in world space
			mVboVertices.push_back(Vec3f((float)x - (float)mMeshWidth * 0.5f, (float)y - (float)mMeshHeight * 0.5f, 0.0f));

		}
	}

Now we actually add all the indices and positions to the VBOMesh:

mVboMesh = gl::VboMesh(mVboVertices.size(), mVboIndices.size(), mVboLayout, GL_POINTS);
	mVboMesh.bufferIndices(mVboIndices);
	mVboMesh.bufferPositions(mVboVertices);

Normally you do some modificiation of the VBO on the CPU in the update() method, but we’re interested in moving as much of our processing as possible to the GPU, so we’re not doing any of that. We do however need to be able to pass variables into our shader so we can generate the wave and correctly position the view onto the mesh. We do this using Uniforms, which as mentioend eaelir in the article are values passed into the shader on a per shader basis. Not every shader (vertex, geometry, fragment) needs to read them, but they are available for all shaders. Here’s how we pass all the values into the shaders and then draw the VBOMesh:

// Bind and configure shader
	mShader.bind();
	mShader.uniform("alpha", mMeshAlpha);
	mShader.uniform("amp", mMeshWaveAmplitude);
	mShader.uniform("eyePoint", mEyePoint);
	mShader.uniform("lightAmbient", mLightAmbient);
	mShader.uniform("lightDiffuse", mLightDiffuse);
	mShader.uniform("lightPosition", mLightPosition);
	mShader.uniform("lightSpecular", mLightSpecular);
	mShader.uniform("mvp", gl::getProjection() * gl::getModelView());
	mShader.uniform("phase", mElapsedSeconds);
	mShader.uniform("rotation", mBoxRotationSpeed);
	mShader.uniform("scale", mMeshScale);
	mShader.uniform("dimensions", Vec4f(mBoxDimensions));
	mShader.uniform("shininess", mLightShininess);
	mShader.uniform("speed", mMeshWaveSpeed);
	mShader.uniform("transform", mBoxEnabled);
	mShader.uniform("uvmix", mMeshUvMix);
	mShader.uniform("width", mMeshWaveWidth);

	// Draw VBO
	gl::draw(mVboMesh);

	// Stop drawing
	mShader.unbind();

Some of these lines are worth annotating a bit further. For instance:

mShader.uniform(“mvp”, gl::getProjection() * gl::getModelView());

In previous versions of GLSL, you could access the ModelView matrix through a built in variable. In GLSL 1.5 though you simply pass the matrix in using a uniform. This line:

mShader.uniform(“eyePoint”, mEyePoint);

allows us to position our generated geometry to the camera. This is really powerful because all you need is a 3D vector and you can correctly calculate the eye perspective onto a scene, I’m sure you can imagine how this will be very useful when we finally get to the Kinect portion of this tutorial. That eye position is vital, because it’s how we use the place that you want to look when we generate all the geometry. In the geometry shader you can see how these are all put to use:

uniform vec4 dimensions;
uniform mat4 mvp;
uniform float rotation;
uniform bool transform;

// Input attributes from the vertex shader
in vec4 vertex[];

// Output attributes
out vec4 normal;
out vec4 position;
out vec2 uv;

// Adds a vertex to the current primitive
void addVertex(vec2 texCoord, vec4 vert, vec4 norm)
{

	// Assign values to output attributes
	normal = norm;
	uv = texCoord;
	position = vert;
	gl_Position = vert;

	// Create vertex
	EmitVertex();

}

// Kernel
void main(void)
{

Now we need to calculate what the position should be based on the view so that we can send that on to the fragment shader.

position = mvp * vertex[0];

	// Transform point to box
	if (transform)
	{

		// Use Y position for phase
		float angle = vertex[0].y * rotation;

The rotation matrix we’re making here is what allows us to correctly position each box after twisting it to fit the shape of our wave. If these are a bit fuzzy for you, I find this to be an excellent refresher or introduction.

// Define rotation matrix
		mat4 rotMatrix;
		rotMatrix[0].x = 1.0 + cos(angle);
		rotMatrix[0].y = 1.0 + -sin(angle);
		rotMatrix[0].z = 1.0 + -rotMatrix[0].x;
		rotMatrix[0].w = 0.0;
		rotMatrix[1].x = 1.0 + -rotMatrix[0].z;
		rotMatrix[1].y = 1.0 + -rotMatrix[0].y;
		rotMatrix[1].z = 1.0 + -rotMatrix[0].x;
		rotMatrix[1].w = 0.0;
		rotMatrix[2].x = 0.0;
		rotMatrix[2].y = 0.0;
		rotMatrix[2].z = 1.0;
		rotMatrix[2].w = 0.0;
		rotMatrix[3].x = 0.0;
		rotMatrix[3].y = 0.0;
		rotMatrix[3].z = 0.0;
		rotMatrix[3].w = 1.0;

Now we need vertices for our box. This can broken down as:
vertex = the view matrix * (our position from the vertex shader + (the size of the box * the corner of the box that we're generating) * the amount that it should be twisted)

// Define vertices
		vec4 vert0 = mvp * (vertex[0] + vec4(dimensions * vec4(-1.0, -1.0, -1.0, 0.0)) * rotMatrix); // 0 ---
		vec4 vert1 = mvp * (vertex[0] + vec4(dimensions * vec4( 1.0, -1.0, -1.0, 0.0)) * rotMatrix); // 1 +--
		vec4 vert2 = mvp * (vertex[0] + vec4(dimensions * vec4(-1.0,  1.0, -1.0, 0.0)) * rotMatrix); // 2 -+-
		vec4 vert3 = mvp * (vertex[0] + vec4(dimensions * vec4( 1.0,  1.0, -1.0, 0.0)) * rotMatrix); // 3 ++-
		vec4 vert4 = mvp * (vertex[0] + vec4(dimensions * vec4(-1.0, -1.0,  1.0, 0.0)) * rotMatrix); // 4 --+
		vec4 vert5 = mvp * (vertex[0] + vec4(dimensions * vec4( 1.0, -1.0,  1.0, 0.0)) * rotMatrix); // 5 +-+
		vec4 vert6 = mvp * (vertex[0] + vec4(dimensions * vec4(-1.0,  1.0,  1.0, 0.0)) * rotMatrix); // 6 -++
		vec4 vert7 = mvp * (vertex[0] + vec4(dimensions * vec4( 1.0,  1.0,  1.0, 0.0)) * rotMatrix); // 7 +++

		// Define normals
		vec4 norm0 = mvp * vec4( 1.0,  0.0,  0.0, 0.0); // Right
		vec4 norm1 = mvp * vec4( 0.0,  1.0,  0.0, 0.0); // Top
		vec4 norm2 = mvp * vec4( 0.0,  0.0,  1.0, 0.0); // Front
		vec4 norm3 = mvp * vec4(-1.0,  0.0,  0.0, 0.0); // Left
		vec4 norm4 = mvp * vec4( 0.0, -1.0,  0.0, 0.0); // Bottom
		vec4 norm5 = mvp * vec4( 0.0,  0.0, -1.0, 0.0); // Back

		// Define UV coordinates
		vec2 uv0 = vec2(0.0, 0.0); // --
		vec2 uv1 = vec2(1.0, 0.0); // +-
		vec2 uv2 = vec2(1.0, 1.0); // ++
		vec2 uv3 = vec2(0.0, 1.0); // -+

Now we get in to simply making the sides of a cube:

// left
		addVertex(uv1, vert5, norm3);
		addVertex(uv2, vert7, norm3);
		addVertex(uv0, vert4, norm3);
		addVertex(uv3, vert6, norm3);
		EndPrimitive();

		// Right
		addVertex(uv3, vert3, norm0);
		addVertex(uv0, vert1, norm0);
		addVertex(uv2, vert2, norm0);
		addVertex(uv1, vert0, norm0);
		EndPrimitive();

		// Top
		addVertex(uv1, vert7, norm1);
		addVertex(uv2, vert3, norm1);
		addVertex(uv0, vert6, norm1);
		addVertex(uv3, vert2, norm1);
		EndPrimitive();

		// Bottom
		addVertex(uv1, vert1, norm4);
		addVertex(uv2, vert5, norm4);
		addVertex(uv0, vert0, norm4);
		addVertex(uv3, vert4, norm4);
		EndPrimitive();

		// Front
		addVertex(uv2, vert3, norm2);
		addVertex(uv3, vert7, norm2);
		addVertex(uv1, vert1, norm2);
		addVertex(uv0, vert5, norm2);
		EndPrimitive();

		// Back
		addVertex(uv2, vert6, norm5);
		addVertex(uv1, vert2, norm5);
		addVertex(uv3, vert4, norm5);
		addVertex(uv0, vert0, norm5);
		EndPrimitive();

	}
	else
	{

		// Pass-thru
		gl_Position = position;
		EmitVertex();
		EndPrimitive();

	}

}

Now the fragment shader and yes, this seems a bit complex, but is actually fairly simple.

#version 150

// Uniforms
uniform float alpha;
uniform vec3 eyePoint;
uniform vec4 lightAmbient;
uniform vec4 lightDiffuse;
uniform vec4 lightSpecular;
uniform vec3 lightPosition;
uniform float shininess;
uniform bool transform;
uniform float uvmix;

The geometry shader is outputting a normal, position, and uv coordinate for the fragment shader to use in coloring:

// Input attributes
in vec4 normal;
in vec4 position;
in vec2 uv;

And here’s what is actually output:

// Output attributes
out vec4 color;

// Kernel
void main(void)
{
	// Transform point to box
	if (transform)
	{

		// Initialize color
		color = vec4(0.0, 0.0, 0.0, 0.0);

		// Normalized eye position
		vec3 eye = normalize(-eyePoint);

Lights are really a light position and a reflection vector. That reflection needs to take the position of the light and normal of the surface into account to look “right”, so we normalize the reflected light vector using the reflect() method, which is built into GLSL.

// Calculate light and reflection positions
		vec3 light = normalize(lightPosition.xyz - position.xyz);
		vec3 reflection = normalize(-reflect(light, normal.xyz));

This is where what people think of as materials are actually generated: putting together the value of a light, a surface color, and a reflection together mathematically. That becomes the color of our pixel and voila, our geometry is generated and colored correctly to match what we’d expect from the color of the box and the light shining onto it.

// Calculate ambient, diffuse, and specular values
		vec4 ambient = lightAmbient;
		vec4 diffuse = clamp(lightDiffuse * max(dot(normal.xyz, light), 0.0), 0.0, 1.0);
		vec4 specular = clamp(lightSpecular * pow(max(dot(reflection, eye), 0.0), 0.3 * shininess), 0.0, 1.0); 

		// Set color from light
		color = ambient + diffuse + specular;

		// Mix with UV map
		color = mix(color, vec4(uv.s, uv.t, 1.0, 1.0), uvmix);

		// Set alpha
		color.a = color.a * alpha;

	}
	else
	{

		// Color point white
		color = vec4(1.0, 1.0, 1.0, 1.0);

	}

}

Here’s another view of what that looks like:

So now you’ve seen how to make geometry and give it color, we’re going to let you take a breather and in part 2, generate some geometry from a Kinect point cloud data using these techniques. It’s gonna be awesome :)

All of the visuals in the above video were created using NyArtoolkit for Processing. NyARToolkit is an augmented reality toolkit built with 100% pure Java. It is derived from ARToolkit-2.72.1. Like Processing itself it’s open source and free! In this tutorial you will learn how to use it to place computer generated imagery correctly onto real world footage. To do this in real-time NyArtoolkit uses markers – black and white images – to determine the three-dimensional position and orientation in the real world. Most likely you will have seen something like this before, but now you will learn how to do it yourself using freely available software.

All right so let’s start with the general setup. What do you need to get started?

Continue reading.... Augmented Reality with Processing [Tutorial, Processing]

Last year, onedotzero approached Joanie Lemercier of AntiVJ to be part of one of their event, a festival they organised at empac, upstate New York, with a selection of screenings, installations and live performances. The installation, now on show at the China Millennium Monument Museum of Digital Arts onedotzero event, has received fantastic feedback and high acclaim. Just as the team published video describing the project (above), we asked Joanie few questions about the installation.

Joanie: The original plan was to fly to empac to do a 3 weeks residency, to develop a new project from scratch, which would involve projection mapping onto objects, a sound track by minimal techno producer Sleeparchive, and potentially a live performance on the day of the opening. As the video explains, the original idea and schedule felt apart when the volcano erupted, and the 3 weeks long residency turned into just 5 days on site, to setup the installation and prepare a live performance…

Continue reading.... EYJAFJALLAJÖKULL [vvvv, Events, Environment, Inspiration]

(A Generative Art Lost Chapter)

Through the process of writing Generative Art there were various tangents and miscellaneous-mad-shit that, usually in the name of brevity and clarity, ended up on the cutting room floor. What follows is one such tangent reworked into a brief tutorial. In the week that the Processing 2.0 Beta gets its release this would seem an appropriate time to share this on CAN, as it mentions one of P5-2′s coolest features: publishing to HTML5.

The tutorial is in three sections:

1. “… A Practical Guide Using HTML5″

To me, the why is always as important as the how. So first allow me to explain why this is relevant, even though it didn’t make the book.

2. Processing to JavaScript – The Ridiculously Easy Way

HTML Canvas for Processing Users. The two minute version.

3. Generative Art in JavaScript – The Harder, But Still Ridiculously Easy, Way

Cutting out the middle man. How you might get started with nothing but a JavaScript enabled browser to hand.

Continue reading.... Generative Art In HTML5 [Processing, JavaScript, Tutorial]

Numbers, a record label originating from Glasgow, had the opportunity to showcase artists at this year’s Sonar Festival in Barcelona. The artists Redinho, Spencer, Deadboy, Jackmaster, and Lory D each had a spot in the set.

Adam Rodgers, co-founder of Numbers and Remote-Location approached Thomas Traum, Adam Finlay and myself to create live visuals for the show.

We compared a number of technologies and frameworks– OpenFrameworks, Cinder, Unity3D, Cocos2d, WebGL, Processing, and good ol’ Flash. Doing something in 3D was a priority, and since we only had a 2-3 week timeline, the decision really came down to what would be the fastest to develop.

Continue reading.... Visuals for Sonar Festival + VDMX to Unity3D [Tutorials]

In the decade where videogames were born, everything virtual looked like rectangular blocks. From today’s perspective, the representation of a tennis court in the earliest videogames is hard to distinguish from a soccer or a basketball field.
‘PING! – Augmented Pixel’ is a seventies style videogame, that adds a layer of digital information and oldschool aesthetics to a video signal: A classic rectangular video game ball moves across a video image. Whenever the ball hits something dark, it bounces off. The game itself has no rules and no goal. Like GTA, it provides a free environment in which anything is possible. And like Sony’s Eyetoy, it uses a video camera as game controller.

What I found interesting when I developed this game, is, that it could have been made already in the seventies. The technology that I used for it is (in a way) similar to what Atari used for the first Pong. It becomes even more awkward, if you think that the electronic components for capturing and evaluating a video signal are cheaper than the rotary game controllers that Atari used. But still, from an economic point of view it makes sense that Eyetoys weren’t the ultimate controllers of thirty-something years ago, as a video camera was probably very hard to afford back in the days.

For those who want to know how it works:

The game is programed with AVR-GCC on an ATmega8 microcontroller that runs with 16MHz. The controller gets basic videosignal synchronisation information from an LM1881 sync separator that triggers two hardware interrupts. One for a new image, the other one for a new line. The controller evaluates the brightness around the pixel (/ball) via its comparator input. Drawing the white image overlay is realized with a simple pull-up resistor in the signal line.

Continue reading.... PING! Augmented Pixel [Tutorials, Games]

Would you like to create what you see in those videos? Well, read on! Because in this article I will show you how you can do just that using Processing and Toxiclibs. As Processing’s biggest open source collection of libraries, Toxiclibs can assist you in areas like geometry, physics, math and color. With so much code candy for the taking, the libs can still be a bit daunting for many people, especially Processing beginners. That’s why – in addition to great functionality and documentation – clear and inspiring examples on how to use the library are so important. Fortunately the collection of code examples bundled with the libs is growing steadily. I hope my examples can add to that and be helpful to those learning how to use this wonderful collection of code which is shared and continuously developed by Karsten Schmidt.

I’ve already shared the two code examples from the first video on my blog. As the video shows these concern creating, picking and dragging polygon shapes (example 1) and the destruction of voronoi tesselated circles (example 2). The full source code and a more detailed explanation of those two examples can be found HERE. In this follow-up I will share the source code for the two brand new examples you see in the second video. This time I’m venturing a little deeper into the physics capabilities of Toxiclibs, more specifically the VerletSpring2D class (example 3) and we will explore a whole new area of the libs, namely the color library (example 4). All of the examples are commented as much as possible. So by running them and looking through the code, you should be able to understand what’s going on. The rest of this blog post can be considered additional background information ranging from general description to specific pointers. Note (06/05/2011): Karsten came up with some useful suggestions to further improve the code, which of course I was happy to apply. This explains why the visuals when running the code may differ ever so slightly from what you see in the movie.

Example 3: The Infinite Rope

What’s better than a rope? Exactly. An INFINITE rope! This example demonstrates both the creation and efficient removal of particles, springs and behaviors. The three pillars of the physics system. Specifically it uses the VerletSpring2D class, which can connect two VerletParticles in space. For simplicity it’s kept 2D, but everything with regard to physics in Toxiclibs also has a 3D equivalent. Let me describe the way it works and some of the specific choices and solutions. When the mouse is dragged a new particle is created and connected to the last one, effectively making a digital rope. Release the mouse to start a new rope. Push behavior is added to each particle to make it look a little more realistic. For aesthetic reasons, the color of every segment is determined by the direction of each spring.

The most important part of the sketch however, is the code that removes off-screen objects. This is absolutely imperative to keep things running smoothly. Let me elaborate on two specific aspects with regard to the removeOffscreen() function. First, it’s running backwards through the for loop! This is because we are removing things from the list. Meaning the list is getting shorter while you are going through it. Therefore you need to go backwards to prevent problems and to make sure you cover every item in the list. Second, notice that I remove behavior(i+1) for particle(i). The reason is that the first behavior on the list is the gravity we added in setup(). Therefore the behavior of particle 1 can be found in position 2 of the behavior list and so on.

//  Toxiclibs Code Example: The Infinite Rope
//  by Amnon Owed (05/05/2011)
//  minor refactorings by Karsten Schmidt (06/05/2011)

import processing.opengl.*;

import toxi.physics2d.behaviors.*;
import toxi.physics2d.*;
import toxi.geom.*;
import toxi.color.*;

VerletPhysics2D physics;
VerletParticle2D prev;

int continuous,current; // variables to create a new continuous line on each mouse drag
 
void setup() {
  size(1280,720,OPENGL);
  physics = new VerletPhysics2D();
  // add gravity in positive Y direction
  physics.addBehavior(new GravityBehavior(new Vec2D(0,0.1)));
  // set the stroke weight of the line
  strokeWeight(2);
}
 
void draw() {
  background(255);
  // update all the physics stuff (particles, springs, gravity)
  physics.update();
 
  // draw a line segment for each spring and change the color of it based on the x position
  for(VerletSpring2D s : physics.springs) {
    // map the direction of each spring to a hue
    float currHue=map(s.b.sub(s.a).heading(),-PI,PI,0,1);
    // define a color in HSV and convert into ARGB format (32bit packed integer)
    stroke(TColor.newHSV(currHue,1,1).toARGB());
    line(s.a.x,s.a.y,s.b.x,s.b.y);
  }
 
  // remove stuff that is off the screen to keep things running smoothly ;-)
  removeOffscreen();
}
 
void removeOffscreen() {
  // remove off-screen springs
  for (Iterator<VerletSpring2D> i=physics.springs.iterator(); i.hasNext();) {
    VerletSpring2D s=i.next();
    if (s.a.y > height+100 || s.b.y > height+100) {
      i.remove();
    }
  }
 
  // remove off-screen particles & behaviors
  for (int i=physics.particles.size()-1; i>=0; i--) {
    VerletParticle2D p = physics.particles.get(i);
    if (p.y > height+200) {
      physics.removeParticle(p);
      ParticleBehavior2D b = physics.behaviors.get(i+1);
      physics.removeBehavior(b);
    }
  }
}
 
void mouseDragged() {
  // create a locked (unmovable) particle at the mouse position
  VerletParticle2D p = new VerletParticle2D(mouseX,mouseY);
  p.lock();
  // if there is at least one particle and this is the current continuous line
  if (physics.particles.size() > 0 && continuous == current) {
    // get the previous particle (aka the last in the list)
    VerletParticle2D prev = physics.particles.get(physics.particles.size()-1);
    // create a spring between the previous and the current particle of length 10 and strength 1
    VerletSpring2D s = new VerletSpring2D(p,prev,10,1);
    // add the spring to the physics system
    physics.addSpring(s);
  } else {
    current = continuous;
  }
  // unlock previous particle
  if (prev!=null) {
    prev.unlock();
  }
  // add the particle to the physics system
  physics.addParticle(p);
  // create a forcefield around this particle with radius 20 and force -1.5 (aka push)
  ParticleBehavior2D b = new AttractionBehavior(p,20,-1.5);
  // add the behavior to the physics system (will be applied to all particles)
  physics.addBehavior(b);
  // make current particle the previous one...
  prev=p;
}
 
void mouseReleased() {
  if (prev!=null) {
    prev.unlock();
  }
  continuous++;
}

Example 4: NamedColors

Aesthetically somewhat similar, but technically completely different, this example is meant to demonstrate how to use TColors in general and NamedColors in particular. If Vec2D/Vec3D is the heart of the geometry lib, then you could say TColor is the heart of the color lib. It’s the basis for much greater possibilities such as color ranges, themes and gradients. But to grasp this part of Toxiclibs, I think it’s best to start with a basic example. For this I chose the NamedColors since they are less abstract than working with numbers alone. In the color portion of Toxiclibs there is a list of 143 NamedColors that you can use. They have names like azure, darkturquoise, lavender, orange and last but not least peachpuff. When working with TColors it’s important to remember that you need to convert them into something that can be used in a Processing fill() or stroke() function. In this example you can see that every time the color is actually used, it’s converted into a packed ARGB int using the toARGB() function.

So let me walk through the sketch real quick. In setup() all the names are loaded into an ArrayList of strings for the purpose of sorting them alphabetically. Running the sketch presents you with the full color palette. I’ve applied some ZoomLensInterpolation to bring out the selected color (mouseX) and made it move up and down with the user (mouseY). There are some checks to make sure both the name and it’s background are kept within the boundaries of the screen. The right mouse button changes the background color, while the left mouse button creates a colorWorm at the mouse position. Press any key to toggle the palette on/off, the mouse functionality will keep working. The colorWorm is basically a list of up to 25 points, a direction and a certain color. It starts at the mouse position and then moves randomly, adding new points along the way. Since the directional change is limited to 30 degrees, it will generally keep going into a certain direction instead of wriggling around the same spot. To make it a little smoother all the points are loaded into a Spline2D which is then subdivided. From the vertices that come out, the line is drawn.

//  Toxiclibs Code Example: NamedColors
//  by Amnon Owed (05/05/2011)
//  minor refactorings by Karsten Schmidt (06/05/2011)

import processing.opengl.*;
import toxi.geom.*;
import toxi.color.*;
import toxi.math.*;

ArrayList <String> names = new ArrayList <String> ();
ArrayList <ColorWorm> worms = new ArrayList <ColorWorm> ();

ZoomLensInterpolation zoomLens = new ZoomLensInterpolation();

boolean showColorPalette = true;
int selectedColorID;

// screen center
Vec2D center;

// background color (readonly colors can't be modified)
ReadonlyTColor bg;

void setup() {
  size(1280, 720, OPENGL);
  center = new Vec2D(width/2, height/2);
  // create a list of all the Toxiclibs NamedColors
  names = NamedColor.getNames();
  // sort it alphabetically
  Collections.sort(names);
  textFont(createFont("SansSerif", 28));
  // set zoom lens to a dilating characteristic
  // setting the first parameter to a negative value will create a bundling effect
  zoomLens.setLensStrength(0.45, 1);
  // set the background color to deepskyblue
  bg = NamedColor.getForName("deepskyblue");
}

void draw() {
  // convert the bg color into ARGB color format (32bit packed integer)
  background(bg.toARGB());

  // run through all the worms (backwards cause we are also removing some from the list)
  for (Iterator<ColorWorm> i=worms.iterator(); i.hasNext();) {
    ColorWorm w = i.next();
    // if the worm's last point is 'off the screen' remove the worm
    // distanceToSquared() is faster than distanceTo() since it avoids
    // the square root calculation and we don't need precise values here...
    if (w.points.get(0).distanceToSquared(center) > 640000) {
      i.remove();
    } 
    else {
      // otherwise update and display the worm
      w.update();
      w.display();
    }
  }

  // set the zoom location based on the normalized mouseX (0.0 .. 1.0 interval)
  float normX=(float)mouseX / width;
  // interpolate focal point to new mouse position (15% step per frame)
  zoomLens.setLensPos(normX, 0.15);
  // determine the selected color based on mouseX
  // by finding which color area contains mouseX
  float focalX=zoomLens.interpolate(0, width, normX);
  for (int i=0, num=names.size()-1; i<=num; i++) {
    float x=zoomLens.interpolate(0, width, (float)i/num);
    float x2=zoomLens.interpolate(0, width, (float)(i+1)/num);
    // select color if focalX is between x and x2
    if (focalX >= x && focalX < x2) {
      selectedColorID=i;
      break;
    }
  }

  // toggle the color palette
  if (showColorPalette) {
    drawColorPalette();
  }

  if (mousePressed) {
    // Create worms with the LEFT mouse button
    if (mouseButton == LEFT) {
      Vec2D mouse = new Vec2D(mouseX, mouseY);
      ReadonlyTColor c = NamedColor.getForName(names.get(selectedColorID));
      worms.add(new ColorWorm(mouse, c));
      // Change the background color with the RIGHT or MIDDLE mouse button
    } 
    else {
      bg = NamedColor.getForName(names.get(selectedColorID));
    }
  }
}

// Press ANY key to toggle the color palette
void keyPressed() {
  showColorPalette = !showColorPalette;
}

class ColorWorm {
  List <Vec2D> points = new ArrayList <Vec2D> ();
  Vec2D direction;
  TColor c;

  ColorWorm(Vec2D origin, ReadonlyTColor c) {
    // at the origin point (mouseX,mouseY)
    points.add(origin);
    // create a copy of the readonly color for later manipulation
    this.c = c.copy();
    // create a random direction
    direction = Vec2D.randomVector();
  }

  void update() {
    // every second frame (not too fast, not too slow)
    if (frameCount % 2 == 0) {
      // create a new point given the last point's coordinates
      Vec2D p = points.get(points.size()-1).copy();
      // rotate the direction randomly somewhere between -30 and 30 degrees
      direction.rotate(radians(random(-30, 30)));
      // create a movement vector in that direction with a random magnitude between 15 and 30
      Vec2D move = direction.getNormalizedTo(random(15, 30));
      // move the point in that direction and with that distance
      p.addSelf(move);
      // add the new point to the list
      points.add(p);
    }

    // truncate at 25 points (remove the oldest point)
    while (points.size () > 25) {
      points.remove(0);
    }
  }

  void display() {
    // need at least 3 points to construct a spline
    if (points.size()>2) {
      // create Spline2D from the points
      Spline2D s = new Spline2D(points);
      // subdivide it by 8 into a list of vertices
      List <Vec2D> vertices = s.computeVertices(8);
      noFill();
      strokeWeight(2);
      // draw a line through all the vertices
      beginShape();
      for (int i=0,num=vertices.size()-1; i<=num; i++) {
        Vec2D v = vertices.get(i);
        // the position in the list determines the transparency of the segment
        c.setAlpha(map(i, 0, num, 0, 1));
        // convert the color into ARGB color format (32bit packed integer)
        stroke(c.toARGB());
        vertex(v.x, v.y);
      }
      endShape();
    }
  }
}

void drawColorPalette() {
  noStroke();

  // display all the colors over the width of the screen
  for (int i=0,num=names.size()-1; i<=num; i++) {
    // determine the color swatch's position & width based on
    // it's relative position and the zoom location (mouseX)
    float x = zoomLens.interpolate(0, width, (float)i / num);
    float x2 = zoomLens.interpolate(0, width, (float)(i+1) / num);
    // convert the color into ARGB color format (32bit packed integer)
    fill(NamedColor.getForName(names.get(i)).toARGB());
    // move the colors vertically with mouseY
    rect(x, mouseY-15, x2-x, 30);
  }

  // get the name of the selectedColor
  String name = names.get(selectedColorID);
  float ascent = textAscent();
  float textwidth = textWidth(name);
  // keep the text and it's background fill within screen boundaries
  float x = min(mouseX, width-textwidth-8);
  float y = min(mouseY + 52, height-4);
  // draw a white text background
  fill(255);
  rect(x, y-ascent-4, textwidth+8, ascent+8);
  // draw a black text
  fill(0);
  text(name, x+4, y);
}

That concludes this round of code sharing. For all things Toxiclibs go to toxiclibs.org. A description of how to install contributed libraries for Processing can be found here. If you would like to know if and how Toxiclibs can help you with your project, but are unsure of where to start, I suggest asking for help in the Processing forum. There are quite a few people over there (including Karsten himself sometimes) who can help you out with advice and maybe even code examples. So good luck and get creative! ;-)

Introduction

In this tutorial, you’ll learn how to sign and publish a Processing/Android sketch to the Android Market.
Note that the signing process should be automated and accessible from the PDE quite soon.
You can follow this specific issue here and get an alert when the bug is fixed.

> Pre-requisites (installation steps are covered here):
- Processing 1.5 installed
- Android SDK

> Pre-requisites for Windows users
- Java SE (http://www.oracle.com/technetwork/java/javase/downloads)
- Ant (http://ant.apache.org/manual/install.html)

Step One- Create and export your sketch

If you haven’t already, I encourage you to first read Jer Thorp’s introduction to Mobile development using Processing for how to setup your environment and run your first sketch.

Another page worth visiting is the Processing for Android Wiki

That being said, you can use this sketch as a starting point and modify it to fit your needs.

[sourcecode language="java"]
float radius = 50.0;
float X, Y;
float nX, nY;
int delay = 16;

void setup() {
size(screenWidth, screenHeight);
orientation(PORTRAIT);
background(255);
X = screenWidth / 2;
Y = screenHeight /3;
nX = X;
nY = Y;
}

void draw() {
radius = radius + sin( frameCount / 4 );
X+=(nX-X)/delay;
Y+=(nY-Y)/delay;
stroke(0, 50);
noFill();
ellipse( X, Y, radius, radius );
}

public boolean surfaceTouchEvent(MotionEvent event) {
int numPointers = event.getPointerCount();
if (numPointers == 2) {
background(255);
}
else if (numPointers == 1) {
int pointerId = event.getPointerId(0);
nX = event.getX(0);
nY = event.getY(0);
}
return super.surfaceTouchEvent(event);
}

[/sourcecode]

Start Processing, switch to the Android mode, copy the sketch above and save it.

To export your sketck to an Android Project, from the PDE menu, choose File > Export Android Project
This will create a new Project folder named “android” at the root level of your sketch.

Step Two – Create a key to sign your app

You’ll need a private key to sign your app that you can interactively create using the following command where
“myAndroidKey” is the name you choose for your key and “myAndroidKeyAlias” the alias for this key.

[sourcecode language="bash"]
keytool -genkey -v -keystore myKey.keystore -alias myKeyAlias -keyalg RSA -validity 10000
[/sourcecode]

After answering a couple of questions, you should end up with a .keystore file.
Make sure you remember the key name and alias as well as the password you choose.
The password will be mandatory if you want to later be able to update your app once published.
This key might be used to sign other apps, so you can copy it somewhere and use it when needed without having to generate a new one each time.

Step Three – Configure your project for signing

Important: each time you export your sketch to an Android Project, if the project already exists, il will be renamed into “android-timestampsomething” and a new “android” project will replace it. This means that every modifications that you will do below will be lost, unless you copy the modified files to the new Android project or somehow automate this process.

First copy the .keystore file you generated at step two to the root level of your android folder.

To automate the signing and alignment of the app when building for release (you’ll just be asked for the keystore password), you have to modify the build.properties file and add the following lines (replace the values accordingly):

[sourcecode language="text"]
key.store=myKey.keystore
key.alias=myKeyAlias
[/sourcecode]

Save these changes.

Step Four – Modify the AndroidManifest.xml

Open the AndroidManifest.xml file and set the value of the android:debuggable attribute to false:

[sourcecode language="xml"]
<application android:label="AndroidSketch"
android:icon="@drawable/icon"
android:debuggable="false">
[/sourcecode]

If required, you can change your app’s version by modifying the manifest element attributes.
The values are set by default to the following:
android:versionName=”1.1″ android:versionCode=”1″
If you ever update your app, you’ll have to increment the versionName (1.2 or 2.0) and increment the versionCode (2).
Save these changes.

Step Five- Customize your app icon

Processing provides a default icon in 3 different sizes for your app
You can of course replace them with some others of your choice.
They are located in the following folders:

[sourcecode language="text"]
android/res/drawable/icon.png (48×48 pixels)
android/res/drawable-hdpi/icon.png (72×72 pixels)
android/res/drawable/icon.png (36×36 pixels)
[/sourcecode]

Step Six- Build and sign your app for release

Now, go to the “android” project root and run the following command:

[sourcecode language="bash"]
ant release
[/sourcecode]

Then, enter the password when asked for it.
Make sure you get this message in the end.

[sourcecode language="bash"]BUILD SUCCESSFUL[/sourcecode]

You now have a signed and aligned app in the android/bin folder (ie: myApp-release.apk), ready to be published to the Android Market

Step Seven – Upload your app to the Android Market

The most painful step (for your wallet) of this tutorial: if not already, you’ll have to sign up for an Android Developer account for $25.

Once completed, you can directly publish your app. You’ll need to provide screenshots, a high resolution application icon and some other information to be published along with your application.

You’re done.

I hope to hear from you very soon with some great Android apps that we can review here.

photo by David Cuartielles

Written by Joshua Noble, with images by Robert Hodgin

This article is going to cover two things in great detail: vertices and meshes and how they are handled in Cinder. There are a few different names for things that may be new to you in this tutorial which I’ll lay out for you right at the beginning: vertices, mesh, TriMesh, VboMesh, and VBOs. If any of those are already familiar to you and you simply want implementation details on how they work, feel free to skip ahead or around, I won’t be offended. For those of you who want more background, we’ll proceed in an orderly fashion.
So, let’s begin at the beginning: the vertex. In Cinder we use the OpenGL coordinate system which is a right-handed system. Initially the world is oriented at the upper left hand corner of your screen, so the x Axis points right, y Axis points downward, z Axis points into the screen (away from the user). A vertex is a point in geometrical shape, or, a vertex is a point in 3D space that has x, y, and z properties that determine where it is in relation to the 0, 0, 0 point (you can think of this as the center) of the world. For instance, a vertex with the values 100, 100, 100, describes a location 100 pixels to the right on the x axis, down on the y axis, and back on the z axis from the origin of the world in which it is located. In your Cinder application, unless you change it, the world is oriented at the upper left hand corner of your screen. All shapes are the result of the connections between vertices. So, a line is made up of the connections between two vertices, a pyramid is a construction made of the connections between five vertices, a cube is made up of the connections between eight vertices.

What do these vertices look like in code? Something like this:


gl::vertex( Vec3f(300, 100, 0) ); // creates an OpenGL vertex from a Vec3f


Those vertices are passed to your graphics card and your graphics card fill in the spaces in between them in a processing usually called “the rendering pipeline”. The rendering pipeline goes more or less like this:
1. Say how you’re going to connect all the points.
2. Make some points.
3. Say that you’re done making points.
Saying it in code, and more specficially in Cinder code, looks like this:


glBegin(GL_QUAD_STRIP); // start drawing
gl::vertex(Vec3f(20, 20, 0)); // repeated a lot of times with different positions
glEnd(); // stop drawing


You may be thinking: “I’ll just make eight vertices and voila: a cube.” Not so quick. There’s a hitch and that hitch is that the OpenGL renderer has different ways of connecting the vertices that you pass to it and none are as efficient as to only need eight vertices to create a cube. You’ve probably seen a version of the following image somewhere before:

Creating eight vertices and expecting the GL_QUAD_STRIP to connect them is going to lead to something looking like this:

Why is that you ask? Well, first off because we used the GL_QUAD_STRIP but it’s also because there isn’t an easy way to tell your graphics card that you want to connect all those points into a cube. Generally you have to create your points to fit the drawing mode that you’ve selected because of what’s called “winding”. A vertex gets connected to another vertex in the order that the mode does it’s winding and this means that you might need multiple vertices in a given location to create the shape you want. The cube, for example, requires eighteen vertices, not the eight that you would expect. If you note the order of vertices in the GL chart above you’ll see that all of them use their vertices slightly differently (in particular you should make note of the GL_TRIANGLE_STRIP example). Drawing a shape requires that you keep track of which drawing mode is being used and which order your vertices are declared in. If you’re thinking: “it would be nice if there were an abstraction layer for this” you’re thinking right. Enter the mesh, which is really just an abstraction of the vertex and drawing mode that we started with but which has the added bonus of managing the draw order for you. That may seem insignificant at first, but it provides some real benefits when working with complex geometry.

TriMesh

The TriMesh represents a series of vertices connected with the same connection algorithm as the GL_TRIANGLE_STRIP. It’s a convienent way to keep track of multiple complex objects, draw and scale them easily, and manage which vertices create which faces of a model. Let’s get the simplest example out the way first, drawing a square consisting of two triangles. This requires, as you’d imagine, four vertices that represent the vertices of each triangle.


// Position and Color for the four corners
mesh.appendVertex( Vec3f( 10, 10, 0 ) );
mesh.appendColorRGB( Color( 1, 0, 0 ) );
mesh.appendVertex( Vec3f( 10, 300, 0 ) );
mesh.appendColorRGB( Color( 0, 1, 0 ) );
mesh.appendVertex( Vec3f( 300, 300, 0 ) );
mesh.appendColorRGB( Color( 0, 0, 1 ) );
mesh.appendVertex( Vec3f( 300, 10, 0 ) );
mesh.appendColorRGB( Color( 0, 0, 0 ) );

// Two triangles to create our square
mesh.appendTriangle( vIdx0, vIdx1, vIdx2 );
mesh.appendTriangle( vIdx0, vIdx2, vIdx3 );


So far, so good. Now, let’s extend that a little further and build something in 3D. A simple cube should suffice to demonstrate how the mesh connects all the vertices that it contains. At first glance, the next block of code (140 lines) might seem daunting, but all we are doing is repeating the previous code block 6 times, once per side. Since each side is comprised of 2 triangles, we draw a total of 12 triangles.


// Create the points of our cube
Vec3f v0(-100,-100,-100);
Vec3f v1( 100,-100,-100);
Vec3f v2( 100, 100,-100);
Vec3f v3(-100, 100,-100);
Vec3f v4(-100,-100, 100);
Vec3f v5( 100,-100, 100);
Vec3f v6( 100, 100, 100);
Vec3f v7(-100, 100, 100);

}


Accessing the vertices in the TriMesh is a little different than you might imagine it at first, because you can’t directly access the vertices of the mesh and then alter them. Instead, you make a copy of the vertices using getVertices(), which returns a Vector of the vertices contained by the TriMesh as Vec3f objects and then modifying the values in that vector. To update the vertices in the mesh itself you then need to clear the mesh and append all the vertices again. The same goes for any modifications to the RGB color array as well. Look at the following:


void TriMeshSampleApp::update()
{
if( mMesh.getNumVertices() == 0 ) return;

// go to the next triangle pair
j+=4;
}
}


The result?

Want to use a gl::Texture to texture the cube? Not so difficult. First, you need to set the texture co-ordinates of each vertex in the TriMesh. This requires that you use texture coordinates, not space coordinates or pixel co-ordinates. So, you may think your image is 500 pixels high and 500 pixels wide, but texture co-ordinates say that your texture is 1×1 with what you think of as upper-left corner (0, 0), still being (0, 0), but the bottom-right corner being (1,1). All the values between the 0, 0 and 1,1 points are scalar, meaning that 0.5 will be the midpoint of the image. So if you assign the texture co-ordinate 0.5, 0.5 to a vertex, the middle of the image will appear at that point. The picture below will hopefully help clarify this:

So, now, how to put our Trimesh and gl::Texture together? When creating the vertices using the appendVertex() and appendColorRGB() methods, use the appendTexCoord() method to create a texture co-ordinate. This indicates the location in the texture that you want to appear at that location in space:


mesh.appendVertex(v0);
mesh.appendColorRGB( Color(1, 0, 0) );
mesh.appendTexCoord(Vec2f(0, 0));
mesh.appendVertex(v1);
mesh.appendColorRGB( Color(0, 0, 1) );
mesh.appendTexCoord(Vec2f(1, 0));
mesh.appendVertex(v2);
mesh.appendColorRGB( Color(1, 0, 0) );
mesh.appendTexCoord(Vec2f(1, 1));
mesh.appendVertex(v3);
mesh.appendColorRGB( Color(0, 1, 0) );
mesh.appendTexCoord(Vec2f(0, 1));


Now, to apply a gl::Texture to this TriMesh, call gl::Texture bind(), draw your TriMesh, and then call gl::Texture unbind() and you’re done.


tex.enableAndBind();
gl::draw(mesh);
tex.unbind();


You can even animate the positions of the texture coordinates by copying, modifying, and then re-appending them in the same way that the vertices and RGB coordinates were updated. But there’s more to the TriMesh than just a nice way to store all the vertices that you create because, let’s face it, lining up vertices is a drag and there are better and more fun ways to make vertices, for instance, using a 3d modeling program. To that end, you can import and export OBJ files. So, in my favorite 3d modeling program (the opensource and free one) Blender, I’ll create a classic platonic solid, the isodecahedron, and then export it as an .obj file:

Now I can import it using an instance of the ObjLoader class like so:


ObjLoader loader( loadResource( SOLID ) );
loader.load( *mesh );


and draw it to the screen by passing it to gl::draw():


gl::draw(mesh);


Here it is with some random colors applied to it:

You can even manipulate it and export it using the TriMesh::writeObject() method:


mesh.write( writeFileStream(meshFileName, true); );


Now, if you thought that was all there was to working with meshes in Cinder you would be wrong. As I mentioned earlier, the way that geometry is drawn on the graphics card is by sending points and telling the card how to connect those points. This might remind you a little of how the Textures work in OpenGL: load some data up there, draw it by referring to it using its TextureId. So then, naturally you might wonder, what if we just stored the points on the card? Introducing the Vertex Buffer Object, aka VBO:

VBO

A VBO is a way of storing all of the data of vertex data on the graphics card. You’ve perhaps heard of Vertex Arrays and Display Lists and the VBO is similiar to both of these, but with a few advantages that we’ll go over very quickly. Vertex Arrays just let you store all the vertex data in an array on the client side, that is, on the CPU side and then send it to the graphics card when you’re ready to draw it. The downside of that is that you’re still storing the data on the client side and sending it over to the graphics card. So, instead of making all of our vertex data in what’s called “immediate mode”, which means between a glBegin() and glEnd() pair, you can just store vertex data in an arrays and you can draw geometric primitives by dereferencing the array elements with array indices. The Display List is a similar technique, using an array to store the created geometry, with the crucial difference that a Display List lives solely on the graphics card. This means that once you’ve created the vertex data for geometry, you can send it the graphics card and draw it simply by referencing the id of the stored data. The downside is that display lists can’t be modified. Once they’ve been sent to the card, you need to load them from the card, modify them, and then resend them to the card to see your changes applied. Since one of the conveniences of moving things to the graphics card is reducing the amount of traffic between the graphics card and the rest of your system. The VBO operates quite similarly to the Display List, with the advantage of allowing you to modify the geometry data on the graphics card without downloading all of it at once.

Now, since we’re focusing on Cinder, we’ll be focusing on the VboMesh class that Cinder uses to wrap the core OpenGL functionality. When you create a VboMesh, much like in the TriMesh, you need to decide whether the mesh is going to contain color information, texture coordinate information, but, there is another element to the VboMesh: whether the data is dynamic. Thius is particularly important because it determines whether you’re able to manipulate vertices in the mesh after creating it and it matters because using dynamic data means that the VboMesh will generate two copies of the data: one on the graphics card and one in the memory of your application. So, if you don’t need to update the data after it’s been loaded then you you don’t need to do anything special and if you do want to create dynamic data, you’ll need to create a gl::VboMesh::Layout object and pass it to the VboMesh. So, for instance, if you want static positions and static colors you would create a Layout object with those properties set on it, declare a VboMesh and then assign it to the result of a call to VboMesh(). This is because the VboMesh, like so many other things in Cinder, is a shared pointer, which means that it has to be created before you can use it, you can return it from a method without problems, and also that you don’t have to worry about deallocating the VboMesh once you’ve created it.
Here’s one of the constructors for the VboMesh:


VboMesh( size_t numVertices, size_t numIndices, Layout layout, GLenum primitiveType );


so that would be used like this:


gl::VboMesh mesh;
gl::VboMesh::Layout layout;
layout.setStaticPositions();
layout.setStaticColorsRGBA();
mesh = gl::VboMesh( 24, 20, layout, GL_QUADS );


Now, the numVertices and numIndices need a little explanantion. You’ll recall from the TriMesh that the number of indices and the number of vertices aren’t directly correlated. You’ll almost always need to create more indices than vertices because there are more indices than discrete locations required for most geometry, but that depends on what kind of geometry you’re creating. This brings us to one of the more confusing elements of working with VboMesh::Layout, which is the idea of indices and dynamic indices in particular. The indices set the location of an Vec3f in the order of drawing elements. Remember how in the first example before introducing the TriMesh, you saw how the order that the vertices were created in affected how they were connected the other vertices? Well, the index for a vertex in the VboMesh works much the same way, indicating where the vertex is in relation to the other vertices and how it should be connected to them. In the graphic below the ways to draw a square using GL_QUADS is shown at the top and left and using GL_TRIANGLE_STRIP at the bottom and right.

Cinder always requires that you create the indices for a mesh before you create the positions for that mesh, because conceptually a vertice without a index can’t be properly integrated into the mesh and vice versa. The two modes of storing positions means that the process for working with VboMeshes that dynamic positions versus those that have static positions is slightly different. We’ll do the static first:


gl::VboMesh::Layout layout;
layout.setStaticIndices();
layout.setDynamicColorsRGBA();
layout.setStaticPositions();

int vertCount = 24;
int quadCount = 6;
mesh = gl::VboMesh(vertCount, quadCount * 4, layout, GL_QUADS);


When you’re using static positions you can simply assign positions using a vector of Vec3f objects and calling VboMesh::bufferPositions(). So, our VboMesh creation code looks like this, note the indices being assigned before the positions:


vector indices;
int i=0;
while(i < 24){
indices.push_back(i);
i++;
}

// now we can buffer positions
mesh.bufferPositions(positions);


This creates the same old boring looking white cube. Until we update the colors. This requires retrieving the color values from the mesh and using an iterator to examine the values, which is a familiar pattern in Cinder. The VertexIter object is created to iterate over all the values in the mesh, you assign to the result of VboMesh::mapVertexBuffer() like so:


void VboTutorial::update()
{
float g = sin(getElapsedSeconds());
float b = cos(getElapsedSeconds());
gl::VboMesh::VertexIter iter = mesh.mapVertexBuffer();
for( int x = 0; x < 24; ++x ) {
//positions.at(x) *= 1.001;
iter.setColorRGBA( ColorA( 1 - (g+b/3), g, b, 0.5) );
++iter;
}
}


Dynamic positions are handled in the same way as dynamic colors. To create the dynamic positions for the simple cube in the VboMesh, setting up the VboMesh is going to look familiar, assigning the indices via a vector of uints, but the actual updating of the positions is going to look quite different because you can’t pass a position values into the VboMesh by using VboMesh::bufferPositions(). Instead, you create an VertexIter to iterate over and modify the positions:


void VboTutorialApp::setup()
{
gl::VboMesh::Layout layout;
layout.setStaticIndices();
layout.setDynamicColorsRGBA();
layout.setDynamicPositions();

mesh.bufferIndices(indices);
}


Note that creating the VboMesh only buffers the indices and not any positions. Creating position data is done via a VertexIter, for instance, in the update() of the application:


void VboTutorialApp::update()
{
gl::VboMesh::VertexIter iter = mesh.mapVertexBuffer();
for( int idx = 0; idx < numberOfVertices; ++idx ) {
iter.setPosition( Vec3f( xVal, yVal, zVal) );
++iter;
}
}


The positions are automatically generated to be unit vector values, stored both on the graphics card and in the application memory, When you modify the VertexIter you mark a certain range of data as needing to be uploaded to the graphics card. For those interested in the underlying functionality, this uses the stored local copy of the vertex and index information to pass to glBufferDataARB(). The values are only uploaded to the card when index, color, or position data is modified saving time and space on any draw operation when changes haven’t been made. This is an important nuance because the idea of keeping records both on the graphics card and in the application memory is used throughout the VboMesh, color data, texture positions, and even indices. If you create dynamic texture positions using either setDynamicTexCoords2d() if you’re using 2d coordinates or setDynamicTexCoords3d() if you’re using 3d coordinates, you can reposition a texture on a mesh dynamically. There are some key differences between 2d textures and 3d textures that you can read up more on here . To move a texture around the x axis of the mesh, create a VertexIter and then modify the texture using setTexCoord3d2() or setTexCoord2d2() depending on the texture type that you’re using. For instance, shifting a 2d texture around:


// dynmaically generate our new positions based on a simple sine wave for mesh
gl::VboMesh::VertexIter iter2 = mVboMesh2.mapVertexBuffer();
for( int x = 0; x < VERTICES_X; ++x ) {
for( int z = 0; z < VERTICES_Z; ++z ) {
float height = sin( z / (float)VERTICES_Z * zFreq + x / (float)VERTICES_X * xFreq + offset ) / 5.0f;
iter2.setPosition( Vec3f( x / (float)VERTICES_X, height, z / (float)VERTICES_Z ) + Vec3f( 0, 0.5, 0 ) );
iter.setTexCoord2d2( Vec2f( x / (float)VERTICES_X, z / (float)VERTICES_Z ));
++iter2;
}
}


You can also copy the values from one VboMesh to another, retrieving values from a mesh using the VboMesh::getIndexVbo() to get the indices from a mesh and VboMesh::getStaticVbo() to get the vertex data from the mesh. The VboMesh can be constructed like so assuming that there’s another VboMesh called otherMesh:


mesh = gl::VboMesh( numberVertices, numberQuads * 4, otherMesh.getLayout(), GL_QUADS, &otherMesh.getIndexVbo(), NULL, &otherMesh.getDynamicVbo());


You may be noticing the last two parameters, which are pointers to either the static or dynamic VBO data from the source VBO.


VboMesh( size_t numVertices, size_t numIndices, Layout layout, GLenum primitiveType, Vbo *indexBuffer, Vbo *staticBuffer, Vbo *dynamicBuffer );


Which you use will depend on how the source has been created, using the dynamic properties from a source VBO requires that the source has dynamic positionsVBO with dynamic properties. Using the static values from a buffer means copying the values over so that they can be manipulated independently. This means that the target mesh will have the same properties as the source mesh until they are altered. Using the dynamic properties of a mesh means that the target mesh is using the same data as the source and changes to the source will be reflected in the target mesh. This makes it easy to reflect changes across meshes, for instance color effects or textures, but makes your meshes dependent on the source mesh.

For the last trick of interoperation between the mesh types in Cinder, a VboMesh can load a mesh from a TriMesh instance, which means that you can easily import and export an .obj file to a VBO with ease, for instance:


ObjLoader loader( loadResource( OBJ_FILE_RESOURCE ));
loader.load(&trimesh);
gl::VboMesh mesh = gl::VboMesh(trimesh);


And that gets you from a modeling tool to a VBO quite painlessly.

As a final demonstration of how much is built into the VBO and mesh handling in Cinder you can easily create normals and set the normals for a VBO using the VertexIter object and its setNormal() method. A normal is simply a vector that’s perpendicular to the face of a triangle or quad in a mesh or other piece of OpenGL geometry and they’re quite important if you plan to use lighting of any kind when rendering a TriMesh or VBO to the screen. The normal essentially tells OpenGL how any light is to be reflected off of a particular face. This allows you to create shading, depth, and other real world lighting effects. It’s also a little out of the scope of this guide, so I’ll point you to a chapter from OpenGL Programming aka “The Red Book” Using the VertexIter allows you set the normal position for each vertex in a VBO (or in a TriMesh for that matter) by iterating through the vertices of the VBO and setting each normal with an appropriate value. The algorithm here was taken from the http://www.opengl.org/wiki/Calculating_a_Surface_Normal OpenGL wiki which explains the algorithm in a little bit greater detail, as well as providing an alternative algorithm.


Vec3f v0, v1, v2;

}
}
}


So hopefully you’re feeling a little more comfortable not only with creating and manipulating meshes in Cinder, but also with underestanding what some of the underlying mechanics of a mesh and OpenGL geometry are as well. This guide has barely scratched the surface, but it should be enough to get you started. For more information you can check the excellent references at songho or to the reference for the TriMesh and VBO mesh. Have fun, and make something beautiful.

–

Joshua Noble is a writer, designer, and programmer based in Portland, Oregon and New York City. He’s the author of, most recently, Programming Interactivity and the forthcoming book Research for Living.

In this little tutorial we’re going to follow a path that starts with a file on the filesystem, say a PNG file, and ends with the image being drawn to the screen using OpenGL.

[sourcecode language="cpp"]
// probably in your App’s setup() method
gl::Texture texture = loadImage( "image.jpg" );

// and in your App’s draw()
gl::draw( texture );
[/sourcecode]

Even though that code simply loads and draws an image, there’s actually quite a lot going on behind the scenes. The first line creates a new OpenGL texture from the result of loadImage(). This function is used for loading the various image formats Cinder knows about, and can be used to read from files, URLs, resources and other sources, as we’ll see later. And the second line actually draws the gl::Texture to your window using OpenGL.

These two lines highlight one of the core characteristics of Cinder’s design: data, its I/O (Input/Output) and manipulation are carefully separated. Objects and functions are designed for distinct logical points in the process of loading, manipulating, and transferring data. For example, loadImage() is a standalone function, not a member of gl::Texture. Similarly, a gl::Texture doesn’t draw itself – it’s designed to represent image data, but not to manipulate or render it. To draw a gl::Texture, we use a standalone function as well, gl::draw(). As we explore more of Cinder’s image capabilities, we’ll look at some of the power and convenience this division of labor brings.

Continuing this theme of the separation of responsibilities, we introduce the Surface class. This class represents a block of bitmap data on the CPU, just as the gl::Texture represents a block of bitmap data on the GPU. That’s all each does and that’s a good thing. When you want to use a bitmap in OpenGL, you need to load it onto the GPU. Luckily Cinder makes this easy by allowing you to pass the bitmap to an OpenGL Texture by constructing a gl::Texture directly from a Surface:

[sourcecode language="cpp"]
Surface mySurface; // initialized elsewhere
gl::Texture texture = gl::Texture( mySurface );
[/sourcecode]

The rest of this article is going to run through a few different topics in greater depth: Surfaces, Channels, I/O, and Textures, so feel free to skip around in a technologically enhanced fashion or to read along in the time honored “beginning to end” fashion.

Surface

The Surface comes in two varieties: an 8 bit unsigned integer and 32 bit floating point high dynamic range version. This means each of the color components – red, green, blue and sometimes alpha – are represented using either an 8 bit uint8_t or a 32 bit float. These two types are Surface8u and Surface32f, respectively. The majority of the time a Surface8u is just what you need. However if you’re doing some advanced image processing, or you want to make use of high dynamic range images, Surface32f is your best bet. Also it’s worth noting Surface is just a convenient synonym for Surface8u – you can use either name for the class in your code.

The most important thing to recognize about the Surface is where it lives: on the CPU. Any manipulations that you’d like to do using C++ code (as opposed something like an OpenGL shader), such as filtering, replacing sections of bitmaps, or tweaking values of pixels, should all be performed using a Surface. You declare a Surface like so:

[sourcecode language="cpp"]
Surface8u regularSurface; // an empty 8 bit surface
Surface32f hdrSurface; // an empty 32 bit high dynamic range surface
[/sourcecode]

When you declare a Surface without assigning it or constructing it, it’s empty. The memory hasn’t been set aside for it yet and trying to use aSurface before it’s been allocated is going to lead to your code throwing an exception. This is akin to trying to sit down before a chair is beneath your hindquarters: best to put something there first. To allocate the memory for a Surface you can call the constructor like so:

[sourcecode language="cpp"]
Surface mySurface( 640, 480, true ); // width, height, alpha?
[/sourcecode]

So what’s that doing? Simply setting aside memory. Surfaces are contiguous in memory, which means that when you look at an image you see a grid of pixels, but the memory is simply a block of bytes filled with numeric values. In our case above, we’ve asked for a Surface that is 640 pixels wide, 480 pixels tall, has an alpha channel. It’s worth noting that not all Surfaces have an alpha channel. Let’s also look at a slightly different way to allocate a Surface:

[sourcecode language="cpp"]
Surface mySurface( 640, 480, true, SurfaceChannelOrder::RGBA ); // width, height, alpha?, channel order
[/sourcecode]

Here we’ve asked that the color channels of a pixel be ordered in memory as red-green-blue-alpha. This last parameter is optional (notice that we left it out earlier) and it defaults to something reasonable based on whether you have an alpha channel or not. If we had passed something different like SurfaceChannelOrder::BGRA our pixels would be represented in blue-green-red-alpha order in memory instead. Why would we ever want that? Well many graphics APIs – such as Apple’s Quartz, Microsoft’s GDI+, Cairo, or OpenCV prefer or even require that pixels be ordered in a manner other than RGBA. This feature is part of what allows Cinder Surfaces to seamlessly interoperate with these and other APIs.

In the image below, the small area outlined by the white box is blown up on the right hand side to show the red, green and blue values, as well as the alpha, which is depicted as a block of white to indicate that the pixel is 100% opaque (a value of 255).

The application deals with the bitmap data sequentially…

… and stores it in memory as an array of numbers (in this case, bytes).

The idea to take away is that the Surface is, at its heart, a block of raw data that is interpreted as a rectangular array of pixels. When you’re creating or working with images, you’re simply moving numbers around and the Surface is really a collection of methods to help you work with that data more easily.

For advanced users, if you’re familiar with C++ templates then you’ll recognize what you see when you open up the Surface.h file – the core type is a class SurfaceT, templated on uint8_t or float. However understanding the implementation is definitely not necessary to understand how to use it. Also, as a side note, Cinder supports a 3rd less common Surface type named Surface16u, but not all of Cinder’s image manipulation code supports it – it’s there primarily as an intermediate representation (handy for things like the depth information of the Microsoft Kinect).

Copying Images

There are a few different ways to create a new image from the data in another image. First up, using the clone() method:

[sourcecode language="cpp"]
newSurface = oldSurface.clone();
[/sourcecode]

This function creates a new Surface with the same dimensions and layout, and then copies all of its pixels into the new Surface. Let’s compare that to a simple assignment:

[sourcecode language="cpp"]
newSurface = oldSurface;
[/sourcecode]

This gives us a very different result. While clone() allocated a new block of memory and then duplicated the pixels of oldSurface, the assigment statement above causes newSurface to point to the exact same pixels in memory that oldSurface does. Without delving too deep into the topic, this feature is what makes it safe and fast for you to do things like return a Surface as the result of a function, or to create an STLvector<Surface>, for example. In the example above, oldSurface and newSurface are able to safely sort out memory managment between themselves automatically. So if oldSurface goes away, it won’t take the image data that newSurface is now pointing to with it. However if botholdSurface and newSurface go away, the memory they were sharing is freed for you automatically. Last one out turns off the lights. You’ll find this design technique used throughout Cinder. If you’re into the Design Pattern literature, you might know this techique as the handle-body idiom. Or if you are familiar with shared_ptr or other reference-counted pointers, you can think of Surface, gl::Texture and many other classes in Cinder as transparent shared_ptr‘s.

We can also copy just a section of a Surface using copyFrom(). This code copies the left half of oldSurface:

[sourcecode language="cpp"]
Surface newSurface( oldSurface.getWidth() / 2, oldSurface.getHeight(), false );
newSurface.copyFrom( oldSurface, newSurface.getBounds() );
[/sourcecode]

That second parameter is an Area, Cinder’s class for representing discrete rectangles, and it specifies the region to copy the pixels from.

You can also load Surfaces from files, using loadImage(), or construct them from a gl::Texture:

[sourcecode language="cpp"]
Surface myPicture = loadImage( "myPicture.png" );

gl::Texture myTexture; // initialized elsewhere
Surface fromTex( myTexture );
[/sourcecode]

both of which are discussed in greater detail in the I/O section. These methods are appropriate when you’re copying big blocks of data. But what if you want to manipulate a single pixel, or even more surgically, a single channel value within a pixel?

Manipulating Surfaces

Something that you might want to do with a Surface is walk the pixels, that is, iterate over each RGB value to perform an operation on it. For the sake of a simple example we’ll invert all the pixels in a Surface:

[sourcecode language="cpp"]
Surface bitmap( loadImage( "image.jpg" ) );
Area area( 0, 0, 500, 500 );

Surface::Iter iter = surface->getIter( area );
while( iter.line() ) {
while( iter.pixel() ) {
iter.r() = 255 – iter.r();
iter.g() = 255 – iter.g();
iter.b() = 255 – iter.b();
}
}
[/sourcecode]

Here we construct an instance of a helpful class, Surface::Iter, which is used to iterate the pixels of an Area of a Surface. It is designed to be used in a nested while-loop; the outer loop makes use of the Iter‘s line() routine, while the inner loop uses the pixel() routine. By using these loops, we can access the red, green, blue and alpha values of each pixel in succession. Another thing you might want to do is manipulate those pixels using the values of other pixels in the bitmap. The iterator helps you with a little syntactic sugar to allow you to access the pixels around the pixel that you’re currently iterating over. The same way that you can access the red, green, blue or alpha value of the pixel that you want to set using r(), g(), b(), or a(), you can also get the pixel values with a relative x and y offset using overloaded versions of those same methods.

For instance, in the image below, if the Iter is currently pointing at the pixel labeled A in the image, then r(0,-1) will access the red value of the pixel labeled B, and r(1,1), will access the red value of the pixel labeled C in the image.

This saves you the trouble of either using multiple iterators (messy) or maintaining multiple pointers (awful). The Surface does providegetPixel() and setPixel() methods to return the Color of a pixel at a single point, but the accessor methods of Iter are faster and preferable. As an example, if you wanted to create a twirl effect in an image, you could do something like the following:

[sourcecode language="cpp"]
void TwirlSampleApp::twirl( Surface *surface, Area area, float maxAngle )
{
// make a clone of the surface
Surface inputSurface = surface->clone();

// we’ll need to iterate the inputSurface as well as the output surface
Surface::ConstIter inputIter( inputSurface.getIter() );
Surface::Iter outputIter( surface->getIter( area ) );

float maxDistance = area.getSize().length() / 2;
Vec2f mid = ( area.getUL() + area.getLR() ) / 2;

while( inputIter.line() && outputIter.line() ) {
while( inputIter.pixel() && outputIter.pixel() ) {
Vec2f current = inputIter.getPos() – mid;
float r = current.length();
float twirlAngle = r / maxDistance * maxAngle;
float angle = atan2( current.y, current.x );
Vec2f outSample( r * cos( angle + twirlAngle ), r * sin( angle + twirlAngle ) );
Vec2i out = outSample – current;

outputIter.r() = inputIter.rClamped( out.x, out.y );
outputIter.g() = inputIter.gClamped( out.x, out.y );
outputIter.b() = inputIter.bClamped( out.x, out.y );
}
}
}
[/sourcecode]

Without getting into too much detail, the algorithm converts the coordinates of each pixel into polar coordinates, adds a value to the angle based on how far it is from the center, and then converts this back to rectangular coordinates. One thing to note in that code is the use of rClamped() and friends. Unlike the normal versions, the clamped variants of these accessors are safe to use when you might be illegally accessing pixels that don’t exist in the bitmap. For example, trying to access the pixel to the left of a row’s left-most pixel would be illegal normally, but rClamped() will return the red value of the left-most pixel, clamping the x & y coordinates to the boundaries of the image.

You may have noticed that channels in a bitmap get special attention, so much so that they have their own class to help you work with them more easily. Let’s look at the Channel class itself.

Channel

You can think of a Surface as a collection of images in its own right – one for the red, the green, the blue and sometimes alpha. To interact withSurfaces in that way in Cinder, we can use the Channel class. You may be familiar with this view from Photoshop:

A Surface is structured to conceptually replicate this. In the normal cases you don’t need to interact with individual Channels, but in certain instances it can be very powerful. If you want to create a Surface from a Channel, you can just construct it:

[sourcecode language="cpp"]
Surface surface( channel );
[/sourcecode]

You’ll get a grayscale image automatically because the red, green and blue channels will all be set to the same values as the Channel instance passed in. If you want to turn a Surface into a Channel, making your color image into a single grayscale channel, you just do the inverse:

[sourcecode language="cpp"]
Channel channel( surface );
[/sourcecode]

In this case a high quality grayscale interpretation is automatically made of your RGB data. We say “high quality” because the red, green and blue are weighted to mimick the way your eye perceives luminance (derived from the Rec. 709 high definition video spec, for those interested in such things). Using a Channel instead of a Surface can also be a smart move when you simply need a lightweight grayscale image, to work with the face detection of OpenCV for example. This saves on both memory and processing time.

You can also mix the 8u and 32f variants of the Channel and Surface as well without causing problems:

[sourcecode language="cpp"]
Channel8u myChannel( … );
Surface32f myHdrSurface( myChannel );
[/sourcecode]

A potential use for the Channel would be to do some sort of masking on the CPU, for instance, using the red channel of a Surface to set the alpha of another Surface:

[sourcecode language="cpp"]
// only uses the red pixels of the mask Surface
void ChannelDemoApp::surfaceMaskImage( const Surface &mask, Surface *target )
{
Surface::ConstIter maskIter( mask.getIter() ); // using const because we’re not modifying it
Surface::Iter targetIter( target->getIter() ); // not using const because we are modifying it
while( maskIter.line() && targetIter.line() ) { // line by line
while( maskIter.pixel() && targetIter.pixel() ) { // pixel by pixel
float maskValue = maskIter.r() / 255.0f;

targetIter.r() *= maskValue;
targetIter.g() *= maskValue;
targetIter.b() *= maskValue;
}
}
}
[/sourcecode]

That could be accelerated and made more general by passing a Channel instead of a Surface:

[sourcecode language="cpp"]
void ChannelDemoApp::channelMaskImage( const Channel &mask, Surface *target )
{
Channel::ConstIter maskIter = mask.getIter(); // using const because we’re not modifying it
Surface::Iter targetIter( target->getIter() ); // not using const because we are modifying it
while( maskIter.line() && targetIter.line() ) { // line by line
while( maskIter.pixel() && targetIter.pixel() ) { // pixel by pixel
float maskValue = maskIter.v() / 255.0f;

targetIter.r() *= maskValue;
targetIter.g() *= maskValue;
targetIter.b() *= maskValue;
}
}
}
[/sourcecode]

As a side note, this could be optimized substantially, so don’t take the above code to be an example of fast image manipulation.

The value of a Channel at any position can be retrieved using the v() (v for value) function of Channel::Iter. Many of the image processing functions in Cinder can also be used with single channels, allowing you to, for instance, perform an edge detection function using only the blue channel of a Surface:

[sourcecode language="cpp"]
gl::Texture createEdgeTexture( const Channel &src )
{
Channel temp( src.getWidth(), src.getHeight() );
cinder::ip::edgeDetectSobel( src, &temp );
return gl::Texture( temp );
}
[/sourcecode]

This would be called like so:

[sourcecode language="cpp"]
myTexture = createEdgeTexture( simpleSurface.getChannelBlue() );
[/sourcecode]

In summary, the Channel is a lightweight tool for working with a particular color channel from a Surface, or as a standalone grayscale image. Copying, image processing operations, or other time critical operations that don’t require all the data from a bitmap are best done with the Channel.

I/O Operations

Very often you’ll want to read or write an image file, which is a topic we’ll explore in this section. In the beginning of this article you saw loading images using the loadImage() method. This function can be used to load from file paths:

[sourcecode language="cpp"]
loadImage( "data/image.png" );
[/sourcecode]

or, with a slight variation, from a URL:

[sourcecode language="cpp"]
loadImage( loadUrl( "http://site.com/image.png" ) );
[/sourcecode]

We can also use this method to load images stored in resources (described in a separate guide):

[sourcecode language="cpp"]
loadImage( loadResource( RES_LOGO_IMAGE ) );
[/sourcecode]

If you look up loadImage() in the Cinder documentation, you’ll see it returns an ImageSourceRef. The way this class (and its cousin, ImageTargetRef) work under the hood will be the topic of a future tutorial, but we can benefit from them without understanding their inner workings. In short, an ImageSource is able to represent any sort of image, even the ones we can’t manipulate directly using Channels orSurfaces. You can pass an ImageSourceRef to the constructor of gl::Textures, Channels, Surfaces and several other classes like our OpenCV wrapper classes. The power of this design – a separate class to represent a generic source of an image – is that a source and its destination can negotiate between themselves to create the best representation. For example, let’s imagine you have a grayscale 8-bit image stored in a PNG file you’d like to turn into a gl::Texture. By using code like the following, you get the most optimal representation:

[sourcecode language="cpp"]
gl::Texture myTexture = loadImage( "grayscale.png" );
[/sourcecode]

OpenGL can store textures in a special grayscale-only mode which saves valuable GPU memory. The ImageSource which comes back fromloadImage() can tell the constructor of gl::Texture the things it needs to know to allocate such a representation, and it does so automatically.

And what about the other way around – writing an image out as a file? As you might have guessed, it’s done with a function called writeImage():

[sourcecode language="cpp"]
writeImage( "/path/to/image.png", surfaceToBeWritten );
[/sourcecode]

Cinder automatically infers the sort of image you want from the file extension. You can also force the image to be written with a certain format by supplying the extension as a string for the 3rd parameter:

[sourcecode language="cpp"]
writeImage( "/path/to/image_without_extension", surfaceToBeWritten, "jpg" );
[/sourcecode]

It’s not just Surfaces that can be saved. For example, you can pass a Channel to writeImage() and the I/O machinery will automatically understand it’s dealing with grayscale data and will use an optimized variant of say, PNG for you. Or if you pass a Channel32f, writeImage() will use the 16-bit grayscale mode of PNG to try to preserve as much of your precision as possible. As a fun fact, you can also pass a gl::Texture towriteImage(), or you can convert between file formats with code like this:

[sourcecode language="cpp"]
writeImage( "output.png", loadImage( "input.jpg" ) );
[/sourcecode]

GL::Texture

As we’ve seen, the gl::Texture represents an image on your graphics card. The class features all the things that you would expect of an OpenGL texture: an ID (sometimes called a name in the OpenGL docs), functions to bind() and unbind(), an update() method to load new data into the Texture, methods to initialize the texture, etc.

There are a few conceptual things to keep in mind when working with a Texture. Fundamentally it’s just a bitmap object, but your program has handed it off to another piece of hardware, the GPU, which has its own memory and processors. Copying from CPU to GPU (by for example, constructing a gl::Texture from a Surface) is much slower than say, copying from one Surface to another, so you should do this judiciously. However once the image data is on the GPU it’s ready for any sort of use with OpenGL.

The actual data of a gl::Texture is stored on the GPU very similarly to the way a Surface stores its bitmap data in memory. You’ll recall that the Surface provides an iterator that can be used to walk the bitmap data and access the pixel data or channel data at any point. Doing the same in the gl::Texture is a different matter. As you are likely familiar, modern GPUs implement a fairly rigid multi-stage process known as the “graphics pipeline” (discussed in Wikipedia here). If you want to access the pixels of a texture there are a few different techniques that you can use, all of which are a little beyond the scope of this article. The most common and recommendation-worthy would be in a Fragment Shader, which are implemented in Cinder using the gl::GlslProg class.

You’ve seen how gl::Textures are drawn:

[sourcecode language="cpp"]
gl::draw( mProcessedImageTex, getWindowBounds() );
[/sourcecode]

The first parameter is of course the Texture that you want to draw, and the optional second is a Rectf that indicates how large and where the texture should be drawn. It’s important that a gl::Texture be filled with bitmap data before you try to draw it – a gl::Texture can also point to nothing, just as a Surface can. You can populate a gl::Texture a few different ways, such as by constructing the gl::Texture from a Surface:

[sourcecode language="cpp"]
texture = gl::Texture( surfaceInstance );
[/sourcecode]

or by initiailizing the Texture with some dummy data:

[sourcecode language="cpp"]
gl::Texture texture(500, 500);
[/sourcecode]

Of course if you just draw this, your program won’t crash, but you’ll just see junk data – whatever random memory happens to be on the graphics card at the moment. Cool if you’re making glitch art, not if you’re not.

You can also initialize the Texture with a gl::Texture::Format object that contains specs for how to build the Texture. For instance, by default agl::Texture will simply clamp the Texture to its boundaries. If you wanted the texture to repeat, you could pass in a Texture::Format object like so:

[sourcecode language="cpp"]
gl::Texture::Format fmt;
fmt.setWrap( GL_REPEAT );
texture = gl::Texture( someSurface, fmt );
[/sourcecode]

The differences between clamping and repeating can be seen in the image below:

There are other features that you can set with the Format object too – mipmapping for instance. What’s that you may ask? Take a look at the following images:

Notice the edges on the left. As the English say: that’s not cricket. We want smoothed edges as the gl::Texture itself is scaled down. This is where mipmapping comes in: the GPU can generate multiple copies of Texture data when you create it and that data can be swapped in and out when needed for minification or magnification in your application.

A mipmap is a miniaturized version of the larger bitmap that helps you reduce artifacts when scaling. The smaller the texture is drawn, the smaller the mipmap that the GPU will select. As an example: a texture has a basic size of 256 by 256 pixels, so a mipmap set will be generated with a series of 8 images, each one-fourth the total area of the previous one: 128×128 pixels, 64×64, 32×32, 16×16, 8×8, 4×4, 2×2, 1×1 (a single pixel).

Artifacts are reduced since the mipmap images are effectively already anti-aliased, taking some of the burden off the real-time renderer and potentially improving performance.

[sourcecode language="cpp"]
gl::Texture::Format fmt;
fmt.enableMipmapping( true );
fmt.setMinFilter( GL_LINEAR_MIPMAP_LINEAR );
[/sourcecode]

You’ll notice we added a call to setMinFilter() to tell it use our mipmap. In general Cinder is designed for you to use the OpenGL constants directly (like GL_LINEAR_MIPMAP_LINEAR) so if these are unfamiliar to you, we’d recommend you pick up a book like The OpenGL Programming Guide.

And that’s that. As promised: image data from the filesystem, to the CPU, to the graphics card, and finally to your screen. Now go make something and have fun. And thanks to Flickr user Trey Ratcliff, whose beautiful photograph is used throughout this tutorial.

–

Joshua Noble is a writer, designer, and programmer based in Portland, Oregon and New York City. He’s the author of, most recently, Programming Interactivity and the forthcoming book Research for Living.