Augmented Reality with #Processing - Tutorial by Amnon Owed

CreativeApplications.Net reports innovation and catalogues projects, tools and platforms relevant to the intersection of art, media and technology. Read more here.

t f r

About
- CreativeApplications.Net is your daily dose of inspiring apps for Mac, PC and iPhone...
- Read more..
- Contact
- Advertise
Categories
- Popular CAN Categories:
- iPhone
- Mac
- Windows
- WebApp
- Games
- Processing
- openFrameworks
- iPad
Archives
- Look up past posts by date or category
- Archives
Jobs
- On the Creative Application Network employers within the creative and technology fields are able to connect to skilled and talented job-seekers worldwide through direct access to thousands of daily visitors.
- Browse Job Board
- Submit
- About / Instructions and Terms and Conditions
Apps.CAN
- Apps.CreativeApplications.Net is a site dedicated to the collection of Apps for Mac, iPhone and iPad created by artists around the globe and published by CreativeApplications.Net.
- Apps.CreativeApplications.Net
Education
- Creative Application Network offers a number of resources to help you get started in the field of creative coding and physical computing.
- Books
- Tools
- Tutorials
Support
- Creative Application Network is always looking for support. See below how you can help.
- Support / Donate
Follow
- Join, Submit and Share your projects with CAN using these social networks
Subscribe to
- RSS Subscribers

> 20933
22/12/2011

78 c
AR, au, augmented reality, code, example, examples, GSVideo, NyArtoolkit, opensource, Processing, tutorial

Augmented Reality with Processing [Tutorial, Processing]

All of the visuals in the above video were created using NyArtoolkit for Processing. NyARToolkit is an augmented reality toolkit built with 100% pure Java. It is derived from ARToolkit-2.72.1. Like Processing itself it’s open source and free! In this tutorial you will learn how to use it to place computer generated imagery correctly onto real world footage. To do this in real-time NyArtoolkit uses markers – black and white images – to determine the three-dimensional position and orientation in the real world. Most likely you will have seen something like this before, but now you will learn how to do it yourself using freely available software.

All right so let’s start with the general setup. What do you need to get started?

1. Processing 1.5.1
A Processing tutorial wouldn’t really work without Processing, now would it? So if you haven’t already, go HERE and download Processing for your OS. These examples were written for Processing 1.5.1 (the current stable build) but also with the upcoming Processing 2.0 in mind. I have personally tested all these examples in Processing 2.0a4 (alpha 4) and they work. When the time comes, you will probably notice that the default examples from the NyArtoolkit break in the Processing 2.0, due to the changes in the internal structure of the renderers. P3D is gone and there is a new (and vastly improved) OPENGL renderer in town. Go HERE to see the major changes in Processing 2.0. Just know that I have done my utmost to make sure these augmented reality code examples should run both under the current and future Processing builds.

2. NyArtoolkit for Processing 1.1.6
What it’s all about! Go HERE and download nyar4psg-1.1.6.zip. If you don’t know how to install contributed libraries in Processing, go HERE for instructions.

3. GSVideo 1.0.0
The second and third example use GSVideo to get a webcam stream into Processing. You could use alternatives if you want. I have chosen GSVideo because it works well and also because it will be integrated into Processing 2.0 as the new default video/capture library. So future readers, you might not need to seperately download this. For all of us living in the now, go HERE and download GSVideo for your OS. Again, if you don’t know how to install contributed libraries in Processing, go HERE for instructions.

4. Patterns
To make this whole thing work you will need black and white patterns. The NyArtoolkit includes two patterns (Hiro and Kanji), but these MultiMarker code examples can handle many more. There are a couple of online services that let you make your own markers and train them. But in my opinion, this is too much hassle. Fortunately you can download an ARToolkit Patternmaker from the university of Utah, which comes with 100 patterns and the associated training files! Go HERE to download the ARToolkit Patternmaker. Unpack this file and you will find 100 images (patternMaker\examples\gif) and 100 corresponding pattern files (patternMaker\examples\ARToolKit_Patterns). Very useful and these are the ones I used in the video. Very important: rename the first 10 pattern files to two-digits so they are loaded correctly in alphanumeric order in Processing. Also, if you use for example marker number 30, make sure to increase the number of loaded markers in these code example with the numMarkers variable.

All right, time to recap. So at this point you have Processing, the NyArtoolkit and GSVideo installed. In addition, you have downloaded the Patternmaker and therefore are in possession of 100 quality markers and the corresponding .patt files. If this is correct, you are all set and ready to get started with the code examples. All code examples are fully commented and kept as straight-forward as possible in order to maximize the educational value.

Example 1: Basic
The first example is basic, but holds all of the important techniques that are necessary for more advanced uses of the NyArtoolkit. I’ve purposefully started with an example that is image based for two reasons. First, because it will allow you to focus on getting the library working without any webcam troubles. ;) Second, to show that the NyArtoolkit is in essence image-based. This makes it extremely flexible, because any input will work! Be it a static image, an image sequence, a webcam capture or a movie. This is for example how I made the movie accompanying this blog post.

If you input the following image (place it in the sketch’s data subdirectory)…

…into the first code example, you should end up with something like this…

// Augmented Reality Basic Example by Amnon Owed (21/12/11)
// Processing 1.5.1 + NyARToolkit 1.1.6

import java.io.*; // for the loadPatternFilenames() function
import processing.opengl.*; // for OPENGL rendering
import jp.nyatla.nyar4psg.*; // the NyARToolkit Processing library

// a central location is used for the camera_para.dat and pattern files, so you don't have to copy them to each individual sketch
// Make sure to change both the camPara and the patternPath String to where the files are on YOUR computer
// the full path to the camera_para.dat file
String camPara = "C:/Users/mainframe/Documents/Processing/libraries/nyar4psg/data/camera_para.dat";
// the full path to the .patt pattern files
String patternPath = "C:/Users/mainframe/Documents/Processing/libraries/nyar4psg/patternMaker/examples/ARToolKit_Patterns";
// the dimensions at which the AR will take place. with the current library 1280x720 is about the highest possible resolution.
// it will work just as well at a lower resolution such 640x360, in some case a lower resolution even seems to work better.
int arWidth = 1280;
int arHeight = 720;
// the number of pattern markers (from the complete list of .patt files) that will be detected, here the first 10 from the list.
int numMarkers = 10;

MultiMarker nya;
float displayScale;
color[] colors = new color[numMarkers];
float[] scaler = new float[numMarkers];
PImage input, inputSmall;

void setup() {
  size(1280, 720, OPENGL); // the sketch will resize correctly, so for example setting it to 1920 x 1080 will work as well
  // create a text font for the coordinates and numbers on the boxes at a decent (80) resolution
  textFont(createFont("Arial", 80));
  // load the input image and create a copy at the resolution of the AR detection (otherwise nya.detect will throw an assertion error!)
  input = loadImage("input.jpg");
  inputSmall = input.get();
  inputSmall.resize(arWidth, arHeight);
  // to correct for the scale difference between the AR detection coordinates and the size at which the result is displayed
  displayScale = (float) width / arWidth;
  // create a new MultiMarker at a specific resolution (arWidth x arHeight), with the default camera calibration and coordinate system
  nya = new MultiMarker(this, arWidth, arHeight, camPara, NyAR4PsgConfig.CONFIG_DEFAULT);
  // set the delay after which a lost marker is no longer displayed. by default set to something higher, but here manually set to immediate.
  nya.setLostDelay(1);
  String[] patterns = loadPatternFilenames(patternPath);
  // for the selected number of markers, add the marker for detection
  // create an individual color and scale for that marker (= box)
  for (int i=0; i<numMarkers; i++) {
    nya.addARMarker(patternPath + "/" + patterns[i], 80);
    colors[i] = color(random(255), random(255), random(255), 160); // random color, always at a transparency of 160
    scaler[i] = random(0.5, 1.9); // scaled at half to double size
  }
}

void draw() {
  background(0); // a background call is needed for correct display of the marker results
  image(input, 0, 0, width, height); // display the image at the width and height of the sketch window
  nya.detect(inputSmall); // detect markers in the input image at the correct resolution (incorrect resolution will give assertion error)
  drawMarkers(); // draw the coordinates of the detected markers (2D)
  drawBoxes(); // draw boxes on the detected markers (3D)
}

// this function draws the marker coordinates, note that this is completely 2D and based on the AR dimensions (not the final display size)
void drawMarkers() {
  // set the text alignment (to the left) and size (small)
  textAlign(LEFT, TOP);
  textSize(10);
  noStroke();
  // scale from AR detection size to sketch display size (changes the display of the coordinates, not the values)
  scale(displayScale);
  // for all the markers...
  for (int i=0; i<numMarkers; i++) {
    // if the marker does NOT exist (the ! exlamation mark negates it) continue to the next marker, aka do nothing
    if ((!nya.isExistMarker(i))) { continue; }
    // the following code is only reached and run if the marker DOES EXIST
    // get the four marker coordinates into an array of 2D PVectors
    PVector[] pos2d = nya.getMarkerVertex2D(i);
    // draw each vector both textually and with a red dot
    for (int j=0; j<pos2d.length; j++) {
      String s = "(" + int(pos2d[j].x) + "," + int(pos2d[j].y) + ")";
      fill(255);
      rect(pos2d[j].x, pos2d[j].y, textWidth(s) + 3, textAscent() + textDescent() + 3);
      fill(0);
      text(s, pos2d[j].x + 2, pos2d[j].y + 2);
      fill(255, 0, 0);
      ellipse(pos2d[j].x, pos2d[j].y, 5, 5);
    }
  }
}

// this function draws correctly placed 3D boxes on top of detected markers
void drawBoxes() {
  // set the AR perspective uniformly, this general point-of-view is the same for all markers
  nya.setARPerspective();
  // set the text alignment (full centered) and size (big)
  textAlign(CENTER, CENTER);
  textSize(20);
  // for all the markers...
  for (int i=0; i<numMarkers; i++) {
    // if the marker does NOT exist (the ! exlamation mark negates it) continue to the next marker, aka do nothing
    if ((!nya.isExistMarker(i))) { continue; }
    // the following code is only reached and run if the marker DOES EXIST
    // get the Matrix for this marker and use it (through setMatrix)
    setMatrix(nya.getMarkerMatrix(i));
    scale(1, -1); // turn things upside down to work intuitively for Processing users
    scale(scaler[i]); // scale the box by it's individual scaler
    translate(0, 0, 20); // translate the box by half (20) of it's size (40)
    lights(); // turn on some lights
    stroke(0); // give the box a black stroke
    fill(colors[i]); // fill the box by it's individual color
    box(40); // the BOX! ;-)
    noLights(); // turn off the lights
    translate(0, 0, 20.1); // translate to just slightly above the box (to prevent OPENGL uglyness)
    noStroke();
    fill(255, 50);
    rect(-20, -20, 40, 40); // display a transparent white rectangle right above the box
    translate(0, 0, 0.1); // translate to just slightly above the rectangle (to prevent OPENGL uglyness)
    fill(0);
    text("" + i, -20, -20, 40, 40); // display the ID of the box in black text centered in the rectangle
  }
  // reset to the default perspective
  perspective();
}

// this function loads .patt filenames into a list of Strings based on a full path to a directory (relies on java.io)
String[] loadPatternFilenames(String path) {
  File folder = new File(path);
  FilenameFilter pattFilter = new FilenameFilter() {
    public boolean accept(File dir, String name) {
      return name.toLowerCase().endsWith(".patt");
    }
  };
  return folder.list(pattFilter);
}

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

// Augmented Reality Basic Example by Amnon Owed (21/12/11)

// Processing 1.5.1 + NyARToolkit 1.1.6

import java.io.*; // for the loadPatternFilenames() function

import processing.opengl.*; // for OPENGL rendering

import jp.nyatla.nyar4psg.*; // the NyARToolkit Processing library

// a central location is used for the camera_para.dat and pattern files, so you don't have to copy them to each individual sketch

// Make sure to change both the camPara and the patternPath String to where the files are on YOUR computer

// the full path to the camera_para.dat file

String camPara = "C:/Users/mainframe/Documents/Processing/libraries/nyar4psg/data/camera_para.dat";

// the full path to the .patt pattern files

String patternPath = "C:/Users/mainframe/Documents/Processing/libraries/nyar4psg/patternMaker/examples/ARToolKit_Patterns";

// the dimensions at which the AR will take place. with the current library 1280x720 is about the highest possible resolution.

// it will work just as well at a lower resolution such 640x360, in some case a lower resolution even seems to work better.

int arWidth = 1280;

int arHeight = 720;

// the number of pattern markers (from the complete list of .patt files) that will be detected, here the first 10 from the list.

int numMarkers = 10;

MultiMarker nya;

float displayScale;

color[] colors = new color[numMarkers];

float[] scaler = new float[numMarkers];

PImage input, inputSmall;

void setup() {

size(1280, 720, OPENGL); // the sketch will resize correctly, so for example setting it to 1920 x 1080 will work as well

// create a text font for the coordinates and numbers on the boxes at a decent (80) resolution

textFont(createFont("Arial", 80));

// load the input image and create a copy at the resolution of the AR detection (otherwise nya.detect will throw an assertion error!)

input = loadImage("input.jpg");

inputSmall = input.get();

inputSmall.resize(arWidth, arHeight);

// to correct for the scale difference between the AR detection coordinates and the size at which the result is displayed

displayScale = (float) width / arWidth;

// create a new MultiMarker at a specific resolution (arWidth x arHeight), with the default camera calibration and coordinate system

nya = new MultiMarker(this, arWidth, arHeight, camPara, NyAR4PsgConfig.CONFIG_DEFAULT);

// set the delay after which a lost marker is no longer displayed. by default set to something higher, but here manually set to immediate.

nya.setLostDelay(1);

String[] patterns = loadPatternFilenames(patternPath);

// for the selected number of markers, add the marker for detection

// create an individual color and scale for that marker (= box)

for (int i=0; i<numMarkers; i++) {

nya.addARMarker(patternPath + "/" + patterns[i], 80);

colors[i] = color(random(255), random(255), random(255), 160); // random color, always at a transparency of 160

scaler[i] = random(0.5, 1.9); // scaled at half to double size

}

void draw() {

background(0); // a background call is needed for correct display of the marker results

image(input, 0, 0, width, height); // display the image at the width and height of the sketch window

nya.detect(inputSmall); // detect markers in the input image at the correct resolution (incorrect resolution will give assertion error)

drawMarkers(); // draw the coordinates of the detected markers (2D)

drawBoxes(); // draw boxes on the detected markers (3D)

}

// this function draws the marker coordinates, note that this is completely 2D and based on the AR dimensions (not the final display size)

void drawMarkers() {

// set the text alignment (to the left) and size (small)

textAlign(LEFT, TOP);

textSize(10);

noStroke();

// scale from AR detection size to sketch display size (changes the display of the coordinates, not the values)

scale(displayScale);

// for all the markers...

for (int i=0; i<numMarkers; i++) {

// if the marker does NOT exist (the ! exlamation mark negates it) continue to the next marker, aka do nothing

if ((!nya.isExistMarker(i))) { continue; }

// the following code is only reached and run if the marker DOES EXIST

// get the four marker coordinates into an array of 2D PVectors

PVector[] pos2d = nya.getMarkerVertex2D(i);

// draw each vector both textually and with a red dot

for (int j=0; j<pos2d.length; j++) {

String s = "(" + int(pos2d[j].x) + "," + int(pos2d[j].y) + ")";

fill(255);

rect(pos2d[j].x, pos2d[j].y, textWidth(s) + 3, textAscent() + textDescent() + 3);

fill(0);

text(s, pos2d[j].x + 2, pos2d[j].y + 2);

fill(255, 0, 0);

ellipse(pos2d[j].x, pos2d[j].y, 5, 5);

}

// this function draws correctly placed 3D boxes on top of detected markers

void drawBoxes() {

// set the AR perspective uniformly, this general point-of-view is the same for all markers

nya.setARPerspective();

// set the text alignment (full centered) and size (big)

textAlign(CENTER, CENTER);

textSize(20);

// for all the markers...

for (int i=0; i<numMarkers; i++) {

// if the marker does NOT exist (the ! exlamation mark negates it) continue to the next marker, aka do nothing

if ((!nya.isExistMarker(i))) { continue; }

// the following code is only reached and run if the marker DOES EXIST

// get the Matrix for this marker and use it (through setMatrix)

setMatrix(nya.getMarkerMatrix(i));

scale(1, -1); // turn things upside down to work intuitively for Processing users

scale(scaler[i]); // scale the box by it's individual scaler

translate(0, 0, 20); // translate the box by half (20) of it's size (40)

lights(); // turn on some lights

stroke(0); // give the box a black stroke

fill(colors[i]); // fill the box by it's individual color

box(40); // the BOX! ;-)

noLights(); // turn off the lights

translate(0, 0, 20.1); // translate to just slightly above the box (to prevent OPENGL uglyness)

noStroke();

fill(255, 50);

rect(-20, -20, 40, 40); // display a transparent white rectangle right above the box

translate(0, 0, 0.1); // translate to just slightly above the rectangle (to prevent OPENGL uglyness)

fill(0);

text("" + i, -20, -20, 40, 40); // display the ID of the box in black text centered in the rectangle

}

// reset to the default perspective

perspective();

}

// this function loads .patt filenames into a list of Strings based on a full path to a directory (relies on java.io)

String[] loadPatternFilenames(String path) {

File folder = new File(path);

FilenameFilter pattFilter = new FilenameFilter() {

public boolean accept(File dir, String name) {

return name.toLowerCase().endsWith(".patt");

}

};

return folder.list(pattFilter);

}

Example 2: Dynamic
Time to get a little more dynamic. Both with regard to the input (webcam) and the output (moving graphics). Instead of a static box, this example shows a moving mountain terrain on each marker. I’ve added some suggestions in the code where you could tweak the look of this example. The mountain will grow (up to a certain height) while the marker is in sight and will flatten if the marker is no longer detected. You can see this behavior very clearly in the video above. Try it out yourself, by running the code example below in front of a webcam!

// Augmented Reality Dynamic Example by Amnon Owed (21/12/11)
// Processing 1.5.1 + NyARToolkit 1.1.6 + GSVideo 1.0

import java.io.*; // for the loadPatternFilenames() function
import processing.opengl.*; // for OPENGL rendering
import jp.nyatla.nyar4psg.*; // the NyARToolkit Processing library
import codeanticode.gsvideo.*; // the GSVideo library

// a central location is used for the camera_para.dat and pattern files, so you don't have to copy them to each individual sketch
// Make sure to change both the camPara and the patternPath String to where the files are on YOUR computer
// the full path to the camera_para.dat file
String camPara = "C:/Users/mainframe/Documents/Processing/libraries/nyar4psg/data/camera_para.dat";
// the full path to the .patt pattern files
String patternPath = "C:/Users/mainframe/Documents/Processing/libraries/nyar4psg/patternMaker/examples/ARToolKit_Patterns";
// the dimensions at which the AR will take place. with the current library 1280x720 is about the highest possible resolution.
int arWidth = 640;
int arHeight = 360;
// the number of pattern markers (from the complete list of .patt files) that will be detected, here the first 10 from the list.
int numMarkers = 10;

// the resolution at which the mountains will be displayed
int resX = 60;
int resY = 60;
// this is a 2 dimensional float array that all the displayed mountains use during their update-to-draw routine
float[][] val = new float[resX][resY];

GSCapture cam;
MultiMarker nya;
float[] scaler = new float[numMarkers];
float[] noiseScale = new float[numMarkers];
float[] mountainHeight = new float[numMarkers];
float[] mountainGrowth = new float[numMarkers];

void setup() {
  size(1280, 720, OPENGL); // the sketch will resize correctly, so for example setting it to 1920 x 1080 will work as well
  cam = new GSCapture(this, 1280, 720); // initializing the webcam capture at a specific resolution (correct/possible settings depends on YOUR webcam)
  cam.start(); // start capturing
  noStroke(); // turn off stroke for the rest of this sketch :-)
  // create a new MultiMarker at a specific resolution (arWidth x arHeight), with the default camera calibration and coordinate system
  nya = new MultiMarker(this, arWidth, arHeight, camPara, NyAR4PsgConfig.CONFIG_DEFAULT);
  // set the delay after which a lost marker is no longer displayed. by default set to something higher, but here manually set to immediate.
  nya.setLostDelay(1);
  String[] patterns = loadPatternFilenames(patternPath);
  // for the selected number of markers, add the marker for detection
  // create an individual scale, noiseScale and maximum mountainHeight for that marker (= mountain)
  for (int i=0; i<numMarkers; i++) {
    nya.addARMarker(patternPath + "/" + patterns[i], 80);
    scaler[i] = random(0.8, 1.9); // scaled a little smaller or bigger
    noiseScale[i] = random(0.02, 0.075); // the perlin noise scale to make it look nicely mountainy
    mountainHeight[i] = random(75, 150); // the maximum height of a mountain
  }
}

void draw() {
  // if there is a cam image coming in...
  if (cam.available()) {
    cam.read(); // read the cam image
    background(0); // a background call is needed for correct display of the marker results
    image(cam, 0, 0, width, height); // display the image at the width and height of the sketch window
    // create a copy of the cam image at the resolution of the AR detection (otherwise nya.detect will throw an assertion error!)
    PImage cSmall = cam.get();
    cSmall.resize(arWidth, arHeight);
    nya.detect(cSmall); // detect markers in the image
    drawMountains(); // draw dynamically flowing mountains on the detected markers (3D)
  }
}

// this function draws correctly placed 3D 'mountains' on top of detected markers
// while the mountains are displayed they grow (up to a certain point), while not displayed they return to the zero-state
void drawMountains() {
  // set the AR perspective uniformly, this general point-of-view is the same for all markers
  nya.setARPerspective();
  // turn on some general lights (without lights it also looks pretty cool, try commenting it out!)
  lights();
  // for all the markers...
  for (int i=0; i<numMarkers; i++) {
    // if the marker does NOT exist (the ! exlamation mark negates it)...
    if ((!nya.isExistMarker(i))) {
      // if the mountainGrowth is higher than zero, decrease by 0.05 (return to the zero-state), then continue to the next marker
      if (mountainGrowth[i] > 0) { mountainGrowth[i] -= 0.05; }
      continue;
    }
    // the following code is only reached and run if the marker DOES EXIST
    // if the mountainGrowth is lower than 1, increase by 0.03
    if (mountainGrowth[i] < 1) { mountainGrowth[i] += 0.03; }
    // the double for loop below sets the values in the 2 dimensional float array for this mountain, based on it's noiseScale, mountainHeight and index (i).
    float xoff = 0.0;
    for (int x=0; x<resX; x++) {
      xoff += noiseScale[i];
      float yoff = 0;
      for (int y=0; y<resY; y++) {
        yoff += noiseScale[i];
        val[x][y] = noise(i*10+xoff+frameCount*0.05, yoff) * mountainHeight[i]; // this sets the value
        float distance = dist(x, y, resX/2, resY/2);
        distance = map(distance, 0, resX/2, 1, 0);
        if (distance < 0) { distance = -distance; } // this line causing the four corners to flap upwards (try commenting it out or setting it to zero)
        val[x][y] *= distance; // in the default case this makes the value approach zero towards the outer ends (try commenting it out to see the difference)
      }
    }

    // get the Matrix for this marker and use it (through setMatrix)
    setMatrix(nya.getMarkerMatrix(i));
    scale(1, -1); // turn things upside down to work intuitively for Processing users
    scale(scaler[i]); // scale the mountain by it's individual scaler
    translate(-resX/2, -resY/2); // translate to center the mountain on the marker
    // for the full resolution...
    for (int x=0; x<resX-1; x++) {
      for (int y=0; y<resY-1; y++) {
        // each face is a Shape with a fill color, together they make a colored mountain
        fill(x*20+y*20, 255-x*5, y*5);
        beginShape();
        vertex(x, y, val[x][y] * mountainGrowth[i]);
        vertex((x+1), y, val[x+1][y] * mountainGrowth[i]);
        vertex((x+1), (y+1), val[x+1][y+1] * mountainGrowth[i]);
        vertex(x, (y+1), val[x][y+1] * mountainGrowth[i]);
        endShape(CLOSE);
      }
    }
  }
  // reset to the default perspective
  perspective();
}

// this function loads .patt filenames into a list of Strings based on a full path to a directory (relies on java.io)
String[] loadPatternFilenames(String path) {
  File folder = new File(path);
  FilenameFilter pattFilter = new FilenameFilter() {
    public boolean accept(File dir, String name) {
      return name.toLowerCase().endsWith(".patt");
    }
  };
  return folder.list(pattFilter);
}

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

// Augmented Reality Dynamic Example by Amnon Owed (21/12/11)

// Processing 1.5.1 + NyARToolkit 1.1.6 + GSVideo 1.0

import java.io.*; // for the loadPatternFilenames() function

import processing.opengl.*; // for OPENGL rendering

import jp.nyatla.nyar4psg.*; // the NyARToolkit Processing library

import codeanticode.gsvideo.*; // the GSVideo library

// a central location is used for the camera_para.dat and pattern files, so you don't have to copy them to each individual sketch

// Make sure to change both the camPara and the patternPath String to where the files are on YOUR computer

// the full path to the camera_para.dat file

String camPara = "C:/Users/mainframe/Documents/Processing/libraries/nyar4psg/data/camera_para.dat";

// the full path to the .patt pattern files

String patternPath = "C:/Users/mainframe/Documents/Processing/libraries/nyar4psg/patternMaker/examples/ARToolKit_Patterns";

// the dimensions at which the AR will take place. with the current library 1280x720 is about the highest possible resolution.

int arWidth = 640;

int arHeight = 360;

// the number of pattern markers (from the complete list of .patt files) that will be detected, here the first 10 from the list.

int numMarkers = 10;

// the resolution at which the mountains will be displayed

int resX = 60;

int resY = 60;

// this is a 2 dimensional float array that all the displayed mountains use during their update-to-draw routine

float[][] val = new float[resX][resY];

GSCapture cam;

MultiMarker nya;

float[] scaler = new float[numMarkers];

float[] noiseScale = new float[numMarkers];

float[] mountainHeight = new float[numMarkers];

float[] mountainGrowth = new float[numMarkers];

void setup() {

size(1280, 720, OPENGL); // the sketch will resize correctly, so for example setting it to 1920 x 1080 will work as well

cam = new GSCapture(this, 1280, 720); // initializing the webcam capture at a specific resolution (correct/possible settings depends on YOUR webcam)

cam.start(); // start capturing

noStroke(); // turn off stroke for the rest of this sketch :-)

// create a new MultiMarker at a specific resolution (arWidth x arHeight), with the default camera calibration and coordinate system

nya = new MultiMarker(this, arWidth, arHeight, camPara, NyAR4PsgConfig.CONFIG_DEFAULT);

// set the delay after which a lost marker is no longer displayed. by default set to something higher, but here manually set to immediate.

nya.setLostDelay(1);

String[] patterns = loadPatternFilenames(patternPath);

// for the selected number of markers, add the marker for detection

// create an individual scale, noiseScale and maximum mountainHeight for that marker (= mountain)

for (int i=0; i<numMarkers; i++) {

nya.addARMarker(patternPath + "/" + patterns[i], 80);

scaler[i] = random(0.8, 1.9); // scaled a little smaller or bigger

noiseScale[i] = random(0.02, 0.075); // the perlin noise scale to make it look nicely mountainy

mountainHeight[i] = random(75, 150); // the maximum height of a mountain

}

void draw() {

// if there is a cam image coming in...

if (cam.available()) {

cam.read(); // read the cam image

background(0); // a background call is needed for correct display of the marker results

image(cam, 0, 0, width, height); // display the image at the width and height of the sketch window

// create a copy of the cam image at the resolution of the AR detection (otherwise nya.detect will throw an assertion error!)

PImage cSmall = cam.get();

cSmall.resize(arWidth, arHeight);

nya.detect(cSmall); // detect markers in the image

drawMountains(); // draw dynamically flowing mountains on the detected markers (3D)

}

// this function draws correctly placed 3D 'mountains' on top of detected markers

// while the mountains are displayed they grow (up to a certain point), while not displayed they return to the zero-state

void drawMountains() {

// set the AR perspective uniformly, this general point-of-view is the same for all markers

nya.setARPerspective();

// turn on some general lights (without lights it also looks pretty cool, try commenting it out!)

lights();

// for all the markers...

for (int i=0; i<numMarkers; i++) {

// if the marker does NOT exist (the ! exlamation mark negates it)...

if ((!nya.isExistMarker(i))) {

// if the mountainGrowth is higher than zero, decrease by 0.05 (return to the zero-state), then continue to the next marker

if (mountainGrowth[i] > 0) { mountainGrowth[i] -= 0.05; }

continue;

}

// the following code is only reached and run if the marker DOES EXIST

// if the mountainGrowth is lower than 1, increase by 0.03

if (mountainGrowth[i] < 1) { mountainGrowth[i] += 0.03; }

// the double for loop below sets the values in the 2 dimensional float array for this mountain, based on it's noiseScale, mountainHeight and index (i).

float xoff = 0.0;

for (int x=0; x<resX; x++) {

xoff += noiseScale[i];

float yoff = 0;

for (int y=0; y<resY; y++) {

yoff += noiseScale[i];

val[x][y] = noise(i*10+xoff+frameCount*0.05, yoff) * mountainHeight[i]; // this sets the value

float distance = dist(x, y, resX/2, resY/2);

distance = map(distance, 0, resX/2, 1, 0);

if (distance < 0) { distance = -distance; } // this line causing the four corners to flap upwards (try commenting it out or setting it to zero)

val[x][y] *= distance; // in the default case this makes the value approach zero towards the outer ends (try commenting it out to see the difference)

}

// get the Matrix for this marker and use it (through setMatrix)

setMatrix(nya.getMarkerMatrix(i));

scale(1, -1); // turn things upside down to work intuitively for Processing users

scale(scaler[i]); // scale the mountain by it's individual scaler

translate(-resX/2, -resY/2); // translate to center the mountain on the marker

// for the full resolution...

for (int x=0; x<resX-1; x++) {

for (int y=0; y<resY-1; y++) {

// each face is a Shape with a fill color, together they make a colored mountain

fill(x*20+y*20, 255-x*5, y*5);

beginShape();

vertex(x, y, val[x][y] * mountainGrowth[i]);

vertex((x+1), y, val[x+1][y] * mountainGrowth[i]);

vertex((x+1), (y+1), val[x+1][y+1] * mountainGrowth[i]);

vertex(x, (y+1), val[x][y+1] * mountainGrowth[i]);

endShape(CLOSE);

}

// reset to the default perspective

perspective();

}

// this function loads .patt filenames into a list of Strings based on a full path to a directory (relies on java.io)

String[] loadPatternFilenames(String path) {

File folder = new File(path);

FilenameFilter pattFilter = new FilenameFilter() {

public boolean accept(File dir, String name) {

return name.toLowerCase().endsWith(".patt");

}

};

return folder.list(pattFilter);

}

Example 3: OOP
The third example serves two purposes. First, you should realise that every (3D) sketch you can run in Processing, can be used in an augmented reality context. The only thing that AR changes is the perspective/camera. So every sketch which holds 3D shapes could work. To show this, I’ve taken an example that comes with Processing, the RGBCube, and integrated it – quite easily – into in AR context. The second purpose of this example, is to show the use of AR detection within an object-oriented (OOP) sketch. As sketches get a little more advanced, many people will turn to classes. It’s very easy to combine AR with classes. In this example it’s done through an ID, which is the link between the object (here a spinning RGBCube) and the corresponding marker. See the code how it works. In this example there are two part / tabs. The main sketch and the ARObject class. You will need both to run the example.

Main Sketch

// Augmented Reality RGBCube OOP Example by Amnon Owed (21/12/11)
// Processing 1.5.1 + NyARToolkit 1.1.6 + GSVideo 1.0

import java.io.*; // for the loadPatternFilenames() function
import processing.opengl.*; // for OPENGL rendering
import jp.nyatla.nyar4psg.*; // the NyARToolkit Processing library
import codeanticode.gsvideo.*; // the GSVideo library

MultiMarker nya;
GSCapture cam;

// this is the arraylist that holds all the objects
ArrayList <ARObject> ars = new ArrayList <ARObject> ();

// a central location is used for the camera_para.dat and pattern files, so you don't have to copy them to each individual sketch
// Make sure to change both the camPara and the patternPath String to where the files are on YOUR computer
// the full path to the camera_para.dat file
String camPara = "C:/Users/mainframe/Documents/Processing/libraries/nyar4psg/data/camera_para.dat";
// the full path to the .patt pattern files
String patternPath = "C:/Users/mainframe/Documents/Processing/libraries/nyar4psg/patternMaker/examples/ARToolKit_Patterns";
// the dimensions at which the AR will take place. with the current library 1280x720 is about the highest possible resolution.
int arWidth = 640;
int arHeight = 360;
// the number of pattern markers (from the complete list of .patt files) that will be detected, here the first 10 from the list.
int numMarkers = 10;
// the maximum rotation speed (x, y, z) at which the RGBCubes will rotate
float mS = 0.2;

void setup() {
  size(1280, 720, OPENGL); // the sketch will resize correctly, so for example setting it to 1920 x 1080 will work as well
  cam = new GSCapture(this, 1280, 720); // initialize the webcam capture at a specific resolution (correct and/or possible settings depend on YOUR webcam)
  cam.start(); // start capturing
  // initialize the MultiMarker at a specific resolution (make sure to input images for detection EXACTLY at this resolution)
  nya = new MultiMarker(this, arWidth, arHeight, camPara, NyAR4PsgConfig.CONFIG_DEFAULT);
  // set the delay after which a lost marker is no longer displayed. by default set to something higher, but here manually set to immediate.
  nya.setLostDelay(1);
  // load the pattern filenames (markers)
  String[] patterns = loadPatternFilenames(patternPath);
  // for the selected number of markers...
  for (int i=0; i<numMarkers; i++) {
    // add the marker for detection
    nya.addARMarker(patternPath + "/" + patterns[i], 80);
    // and create an ARObject with the corresponding 'ID'
    ars.add(new ARObject(i));
  }
  // set the color range to 1 (instead of 255), saves typing for the coloring of the cube
  colorMode(RGB, 1);
  // turn off stroke for the rest of the sketch
  noStroke();
}

void draw() {
  // if there is a cam image coming in...
  if (cam.available()) {
    cam.read(); // read the cam image
    background(0); // a background call is needed for correct display of the marker results
    image(cam, 0, 0, width, height); // display the image at the width and height of the sketch window
    // create a copy of the cam image at the resolution of the AR detection (otherwise nya.detect will throw an assertion error!)
    PImage cSmall = cam.get();
    cSmall.resize(arWidth, arHeight);
    nya.detect(cSmall); // detect markers in the image
    // set the AR perspective uniformly, this general point-of-view is the same for all markers
    nya.setARPerspective();
    // run all the ARObjects's in the arraylist => most things are handled inside the ARObject (see the class for more info)
    for (ARObject ar : ars) { ar.run(); }
    // reset to the default perspective
    perspective();
  }
}

// this function loads .patt filenames into a list of Strings based on a full path to a directory (relies on java.io)
String[] loadPatternFilenames(String path) {
  File folder = new File(path);
  FilenameFilter pattFilter = new FilenameFilter() {
    public boolean accept(File dir, String name) {
      return name.toLowerCase().endsWith(".patt");
    }
  };
  return folder.list(pattFilter);
}

// Augmented Reality RGBCube OOP Example by Amnon Owed (21/12/11)

// Processing 1.5.1 + NyARToolkit 1.1.6 + GSVideo 1.0

import java.io.*; // for the loadPatternFilenames() function

import processing.opengl.*; // for OPENGL rendering

import jp.nyatla.nyar4psg.*; // the NyARToolkit Processing library

import codeanticode.gsvideo.*; // the GSVideo library

MultiMarker nya;

GSCapture cam;

// this is the arraylist that holds all the objects

ArrayList <ARObject> ars = new ArrayList <ARObject> ();

// a central location is used for the camera_para.dat and pattern files, so you don't have to copy them to each individual sketch

// Make sure to change both the camPara and the patternPath String to where the files are on YOUR computer

// the full path to the camera_para.dat file

String camPara = "C:/Users/mainframe/Documents/Processing/libraries/nyar4psg/data/camera_para.dat";

// the full path to the .patt pattern files

String patternPath = "C:/Users/mainframe/Documents/Processing/libraries/nyar4psg/patternMaker/examples/ARToolKit_Patterns";

// the dimensions at which the AR will take place. with the current library 1280x720 is about the highest possible resolution.

int arWidth = 640;

int arHeight = 360;

// the number of pattern markers (from the complete list of .patt files) that will be detected, here the first 10 from the list.

int numMarkers = 10;

// the maximum rotation speed (x, y, z) at which the RGBCubes will rotate

float mS = 0.2;

void setup() {

size(1280, 720, OPENGL); // the sketch will resize correctly, so for example setting it to 1920 x 1080 will work as well

cam = new GSCapture(this, 1280, 720); // initialize the webcam capture at a specific resolution (correct and/or possible settings depend on YOUR webcam)

cam.start(); // start capturing

// initialize the MultiMarker at a specific resolution (make sure to input images for detection EXACTLY at this resolution)

nya = new MultiMarker(this, arWidth, arHeight, camPara, NyAR4PsgConfig.CONFIG_DEFAULT);

// set the delay after which a lost marker is no longer displayed. by default set to something higher, but here manually set to immediate.

nya.setLostDelay(1);

// load the pattern filenames (markers)

String[] patterns = loadPatternFilenames(patternPath);

// for the selected number of markers...

for (int i=0; i<numMarkers; i++) {

// add the marker for detection

nya.addARMarker(patternPath + "/" + patterns[i], 80);

// and create an ARObject with the corresponding 'ID'

ars.add(new ARObject(i));

}

// set the color range to 1 (instead of 255), saves typing for the coloring of the cube

colorMode(RGB, 1);

// turn off stroke for the rest of the sketch

noStroke();

}

void draw() {

// if there is a cam image coming in...

if (cam.available()) {

cam.read(); // read the cam image

background(0); // a background call is needed for correct display of the marker results

image(cam, 0, 0, width, height); // display the image at the width and height of the sketch window

// create a copy of the cam image at the resolution of the AR detection (otherwise nya.detect will throw an assertion error!)

PImage cSmall = cam.get();

cSmall.resize(arWidth, arHeight);

nya.detect(cSmall); // detect markers in the image

// set the AR perspective uniformly, this general point-of-view is the same for all markers

nya.setARPerspective();

// run all the ARObjects's in the arraylist => most things are handled inside the ARObject (see the class for more info)

for (ARObject ar : ars) { ar.run(); }

// reset to the default perspective

perspective();

}

// this function loads .patt filenames into a list of Strings based on a full path to a directory (relies on java.io)

String[] loadPatternFilenames(String path) {

File folder = new File(path);

FilenameFilter pattFilter = new FilenameFilter() {

public boolean accept(File dir, String name) {

return name.toLowerCase().endsWith(".patt");

}

};

return folder.list(pattFilter);

}

ARObject

// class that defines the AROBject, both the AR detection and display are handled inside this class
class ARObject {
  int ID; // keep track of the current the ID of the object (corresponds with the ID i of the marker)
  PVector rot, speed; // in this example the cube has a certain rotation and rotates at a certain speed
  
  ARObject(int ID) {
    this.ID = ID; // set the ID
    rot = new PVector(random(TWO_PI), random(TWO_PI), random(TWO_PI)); // random x, y, z rotation
    speed = new PVector(random(-mS, mS), random(-mS, mS), random(-mS, mS)); // random x, y, z speed (within maxSpeed boundaries)
  }

  void run() {
    // always keep rotating (even when the marker is NOT detected)
    rot.add(speed);
    // checks the object's corresponding marker through the ID
    // if the marker is found, display the cube
    if (nya.isExistMarker(ID)) { display(); }
  }

  // the display in this example shows a colored, rotating RGBCube
  void display () {
    // get the Matrix for this marker and use it (through setMatrix)
    setMatrix(nya.getMarkerMatrix(ID));
    scale(1, -1); // turn things upside down to work intuitively for Processing users

    // hover the cube a little above the real-world marker image
    translate(0, 0, 30);

    // rotate the cube in 3 dimensions
    rotateX(rot.x);
    rotateY(rot.y);
    rotateZ(rot.z);

    // scale - as with the the color range - to save typing with the coordinates (and make it much easier to change the size)
    scale(15);

    // a cube made out of 6 quads
    // the 1 range can be used for both the color and the coordinates as a result of color range and scale (see earlier)
    beginShape(QUADS);

    fill(0, 1, 1); vertex(-1,  1,  1);
    fill(1, 1, 1); vertex( 1,  1,  1);
    fill(1, 0, 1); vertex( 1, -1,  1);
    fill(0, 0, 1); vertex(-1, -1,  1);
  
    fill(1, 1, 1); vertex( 1,  1,  1);
    fill(1, 1, 0); vertex( 1,  1, -1);
    fill(1, 0, 0); vertex( 1, -1, -1);
    fill(1, 0, 1); vertex( 1, -1,  1);

    fill(1, 1, 0); vertex( 1,  1, -1);
    fill(0, 1, 0); vertex(-1,  1, -1);
    fill(0, 0, 0); vertex(-1, -1, -1);
    fill(1, 0, 0); vertex( 1, -1, -1);

    fill(0, 1, 0); vertex(-1,  1, -1);
    fill(0, 1, 1); vertex(-1,  1,  1);
    fill(0, 0, 1); vertex(-1, -1,  1);
    fill(0, 0, 0); vertex(-1, -1, -1);

    fill(0, 1, 0); vertex(-1,  1, -1);
    fill(1, 1, 0); vertex( 1,  1, -1);
    fill(1, 1, 1); vertex( 1,  1,  1);
    fill(0, 1, 1); vertex(-1,  1,  1);

    fill(0, 0, 0); vertex(-1, -1, -1);
    fill(1, 0, 0); vertex( 1, -1, -1);
    fill(1, 0, 1); vertex( 1, -1,  1);
    fill(0, 0, 1); vertex(-1, -1,  1);

    endShape();
  }
}

// class that defines the AROBject, both the AR detection and display are handled inside this class

class ARObject {

int ID; // keep track of the current the ID of the object (corresponds with the ID i of the marker)

PVector rot, speed; // in this example the cube has a certain rotation and rotates at a certain speed

ARObject(int ID) {

this.ID = ID; // set the ID

rot = new PVector(random(TWO_PI), random(TWO_PI), random(TWO_PI)); // random x, y, z rotation

speed = new PVector(random(-mS, mS), random(-mS, mS), random(-mS, mS)); // random x, y, z speed (within maxSpeed boundaries)

}

void run() {

// always keep rotating (even when the marker is NOT detected)

rot.add(speed);

// checks the object's corresponding marker through the ID

// if the marker is found, display the cube

if (nya.isExistMarker(ID)) { display(); }

}

// the display in this example shows a colored, rotating RGBCube

void display () {

// get the Matrix for this marker and use it (through setMatrix)

setMatrix(nya.getMarkerMatrix(ID));

scale(1, -1); // turn things upside down to work intuitively for Processing users

// hover the cube a little above the real-world marker image

translate(0, 0, 30);

// rotate the cube in 3 dimensions

rotateX(rot.x);

rotateY(rot.y);

rotateZ(rot.z);

// scale - as with the the color range - to save typing with the coordinates (and make it much easier to change the size)

scale(15);

// a cube made out of 6 quads

// the 1 range can be used for both the color and the coordinates as a result of color range and scale (see earlier)

beginShape(QUADS);

fill(0, 1, 1); vertex(-1, 1, 1);

fill(1, 1, 1); vertex( 1, 1, 1);

fill(1, 0, 1); vertex( 1, -1, 1);

fill(0, 0, 1); vertex(-1, -1, 1);

fill(1, 1, 1); vertex( 1, 1, 1);

fill(1, 1, 0); vertex( 1, 1, -1);

fill(1, 0, 0); vertex( 1, -1, -1);

fill(1, 0, 1); vertex( 1, -1, 1);

fill(1, 1, 0); vertex( 1, 1, -1);

fill(0, 1, 0); vertex(-1, 1, -1);

fill(0, 0, 0); vertex(-1, -1, -1);

fill(1, 0, 0); vertex( 1, -1, -1);

fill(0, 1, 0); vertex(-1, 1, -1);

fill(0, 1, 1); vertex(-1, 1, 1);

fill(0, 0, 1); vertex(-1, -1, 1);

fill(0, 0, 0); vertex(-1, -1, -1);

fill(0, 1, 0); vertex(-1, 1, -1);

fill(1, 1, 0); vertex( 1, 1, -1);

fill(1, 1, 1); vertex( 1, 1, 1);

fill(0, 1, 1); vertex(-1, 1, 1);

fill(0, 0, 0); vertex(-1, -1, -1);

fill(1, 0, 0); vertex( 1, -1, -1);

fill(1, 0, 1); vertex( 1, -1, 1);

fill(0, 0, 1); vertex(-1, -1, 1);

endShape();

}

Troubleshooting & General Tips
- If you get an “Exception occurred at Multimarker.Multimarker” you probably haven’t changed the camPara String. In all the examples above it points to the location of the file on MY computer. You have to change it to the location of the file on YOUR computer. Also make sure you have (the latest) NyArtoolkit for Processing 1.1.6 installed.
- NullPointerException with regard to the patterns. Most likely the path to the patterns is incorrect. Use forward slashes in the pattern path. Don’t put a slash at the end.
- You get no error messages, but the sketch window remains grey. This means there is no incoming webcam imagery or perhaps no webcam at all. Make sure your webcam is working correctly.
- You get no error messages, the marker is clearly on the screen, but not detected. Most likely you haven’t loaded it’s pattern file. In the code examples you load the first ‘numMarkers’ from the list. Your marker apparantly is alphanumerically not within the range of loaded patterns. Make sure it is and it’s loaded.
- For bigger issues (errors, questions), please use the Processing forum.

I hope you have learned something and this tutorial has been useful to get you off to a running start with augmented reality. Of course this is only the tip of the iceberg. There is a whole world of possibilities. Good luck with these code examples and more!

Best iPhone and iPad Projects of 2011

Fireplace [Mac, Windows]

http://twitter.com/gPanevino G

I think there’s something missing in example n.2, function drawMountains().
for is bad formatted
for (int i=0; i 0) { mountainGrowth[i] -= 0.05; }
continue;
http://twitter.com/dGoligorsky ʎʞsɹoƃıןoƃ pıʌɐp

Thanks for creating this tool and for documenting everything! I’m trying to get the examples running and I’m getting an exception error in Example 1 in the “nya” definition for creating a new MultiMarker. Error is “Exception occurred at Multimarker.Multimarker”
Amnon P5

Yeah, you were right. WordPress ate part of the code ;-) due to some character interpretation issue or something. I’ve corrected the post above, should be all right now. Thanks for pointing it out.
Amnon P5

The error “Exception occurred at Multimarker.Multimarker” means you are using an older version of the NyARToolkit. The MultiMarker class was introduced in the latest version. Download NyARToolkit 1.1.6.
Haris N

I agree with the “G”.

Same problem.
http://twitter.com/gPanevino G

Ok now example 2 is working right, so cool! I think wordpress also cut some piece in example 3 in setup() function.
p.s. I was also getting Multimaker error before I customized paths definition in the code properly as for my Processing directory.
Amnon P5

Great to hear you’ve gotten example 2 to work! Indeed, after you pointed out the issue with that example, I noticed a large part of example 3 main sketch was missing as well. I’m working with Filip to get the correct code for that example up as well. So a little patience, but should be corrected asap. ;-D
http://www.creativeapplications.net Filip

Just installed highlighter that should make reading code easier. We’ll get there eventually =]
Amnon P5

All right everyone, to conclude the above discussion. The new highligher is awesome. It makes the code look good, but more importantly it keeps the code intact! I’ve corrected all the code in this blog post and personally tested it (by copying it from this page for a trial run) and everything worked. Good to see some interest in this topic so quickly ;D now you can get started with the correct code examples. Good luck!
http://twitter.com/ghostpressbed Derek J. Kinsman

GStreamer is giving me nothing but trouble. Attempts at reinstalling it are failing. Need lunch. Derp. Otherwise, it’s a fantastic tutorial. The first example worked great.
Amnon P5

Thanks Derek. If you have an alternative and/or preferred method of getting a webcam image into Processing (instead of GSVideo) you could of course use that. The changes to the examples would be negligable. As I wrote, NyArtoolkit is image-based, so it doesn’t really matter which route you take to get the image there. :D
http://twitter.com/MickD MickD

Really well put together tutorial. Thanks.
InkRunner

Hi! I’m getting the same MultiMarker error but I got the latest NyARTtoolkit 1.1.6 :S can’t find the problem :(
Amnon P5

InkRunner and others. I have a suggestions with regard to the MultiMarker error you could try that may fix it. Change the path of the String camPara to where the file is on your computer. I see on your (InkRunner) screenshot that both the camPara
path and the pattern file path are exactly like I posted it here. But
that is where they are located on my computer. You have to check where
they are on YOUR computer and adapt the paths in the sketch accordingly.

Some general suggestions to make sure the library is working:
1. Make sure you have removed/deleted all older versions of the library from your libraries directory.
2. Rename the main library directory to nyar4psg (when you unpack it the directory is named nyar4psg-1.1.6 so remove the last part of the name to make the directory name equal to the library’s jar filename).
Haris N

As Amnon P5 mention you have to change the path for camPara and markers.
Everything works fine :)
http://www.screenr.com/1MQs
InkRunner

you were totally right! :)

Thank you very much! this gave me so much ideas! :) I’ll be sharing.
Adrianmb

wow!! excellent tutorial ;)
Aart Rost

Hi Amnon! Cool tutorial, thanks for sharing!

To the guys with exceptions: It’s probably in the file paths. Just drop the files and folders in the data folder of the sketch and use dataPath if you need the full file path (only the patterns need that).

This is the first example: http://aartrost.nl/meuk/augumented_reality1.zip rewritten to work without absolute paths. Should work everywhere if you’ve installed the libraries :)
Aart Rost

Try renaming the “gstreamer-0.10″ folder to just “gstreamer”. Not sure if that works but that’s the first thing I would do. Numbers and dots in folder names can be troublesome
k.naresh89

hello amnon p5…..but i would like to bring to your notice that even after making sure the nyartoolkit is at 1.1.6 i am still getting the error similar to
ʎʞsɹoƃıןoƃ pıʌɐp…”
Exception occurred at Multimarker.Multimarker”
any help would be appreciated
and i sincerely thank you for your efforts to bring AR tuts to programmers….;)
Amnon P5

To solve the
MultiMarker error make sure to change the path of the
String camPara to where the camera_para.dat file is on YOUR computer (because in the example code it points to the location where it is on my computer). Also see the earlier comments on this page about it.
http://twitter.com/tecnotic tecnotic

Hi, thanks for the examples.

Unfortunatly I obtain that error:”GLException: Method “glBlendEquation” not available”

I am using Processing 2.0 and I think that I have installed correctly the libraries. Could you suggest me something to follow?
Amnon P5

@ tecnotic: Processing 2.0 is not out yet. Only early alpha versions, you don’t mention which one you use. I have sucessfully run the examples under alpha 1 and alpha 4 (the latest at this time). In general, the alpha’s have proven to be fairly unstable (after all they are alpha and are spread with this fair warning). Wait until the release of the actual stable release of 2.0 before you rely on it. For now, use 1.5.1.

Having said that, I am puzzled by the blend error, since blending is not used in any of the above examples. For general support or issues with the 2.0 alpha’s please use the Processing forum and the bug tracker at google code.
Pavlov1234

nice! i was wondering: which camera you used for the webcam example? the image is pretty, with nice depth of field. was it made by a SLR photo camera?!
Amnon P5

@ Pavlov1234: Yes it was. All of the footage in the movie was shot with a Canon EOS 600D (Rebel T3i).
Pavlov1234

ah cool. but that means you haven’t really used the canon as a webcam, right? you shot all the footage of the markers previously and later used it for the AR magic.
Amnon P5

That’s right. While testing different code I used my webcam in realtime. But for the purpose of the showcase I rewrote all the sketches to work with existing high quality footage.
http://www.jumbocortex.com/think mkedave

bitchin skill to add to the resume.
http://twitter.com/ghostpressbed Derek J. Kinsman

Ah, thanks Amnon. I figured I could get rid of GStreamer. Not sure when I broke it though. I really like GSVideo. Not sure what broke it, so I guess I’ll be debugging the computer before getting any work done. :/
Roberto233

Hi and thanks so much for posting this! I’m very excited to start trying this, but am having the exact same issue on the first example (as ʎʞsɹoƃıןoƃ pıʌɐp posted) and I do indeed have the NyARTToolkit 1.1.6 library installed. I’m using Processing 2.0a4 (alpha 4). I was wondering if you think this issue could be caused by something else? Thanks again.
tomh

Eventually I renamed the folder to nyar4psg The code example 1 on the page has windows formatted directory paths, so I adjusted them to match mine – e.g.String camPara = “/Users/yourusernamehere/Documents/Processing/libraries/nyar4psg/data/camera_para.dat”;// the full path to the .patt pattern filesString patternPath = “/Users/yourusernamehere/Documents/Processing/libraries/nyar4psg/patternMaker/examples/ARToolKit_Patterns”;Then I had to copy the folder called patternMaker into the nyar4psg folder, which isn’t mentioned in the post…

Quick test of the ARToolkit inside Processing:
http://vimeo.com/34605217
Amnon P5

Roberto233: I have updated the blog post to include some more details on the multimarker exception and also updated my first reply to this problem in the comments. Once again, the solution is to change the camPara String to point to the file on YOUR computer. Similarly, the patternPath needs to reflect the location of the files on YOUR computer.

tomh: You are correct. As mentioned in the comments below and the updated blog post, the paths need to point to the files on YOUR computer.

In hindsight, seeing the number of questions about it, I guess I should
have added this more explicitly to the post in the first place. Live and learn! Happy coding everybody! :P
http://www.arminunruh.de/ Armin Unruh

Thanks! Your examples helped me make an application for drawing a simple sound sculpture:
http://vimeo.com/34752161
Simmonsstummer

awesome! i managed to work the static one but in the second i get: “the function start() does not exists” referred to the cam.start()
any suggestion?
Mark Hester

Great Tutorial and a fantastic library.
Thanks, I’m enjoying playing around with it!
One quick question – would it be possible to link the AR markers with a 3D model from a 3D CAD package.
For example if I design something in CAD and save it as an STL or similar 3D file format could I play the file using procesing and link it to an AR marker?
Amnon P5

Yes. You can use for example Toxiclibs’s STLReader to load the STL file into Processing. Once it is in Processing, you can use it. This should be fairly easy.
Juan Patino

Great tutorial! Nice accident happened when the webcam thought a frame in the background was a pattern! Thanks!!!
crg

what is that “import java.io*”? i successfully installed the 2 libraries you mentioned, also got the generator but when ir run the code i get this :
java.lang.NullPointerException at processing.app.Library.getClassPath(Library.java:298) at processing.mode.java.JavaBuild.preprocess(JavaBuild.java:390) at processing.mode.java.JavaBuild.preprocess(JavaBuild.java:197) at processing.mode.java.JavaBuild.build(JavaBuild.java:156) at processing.mode.java.JavaBuild.build(JavaBuild.java:135) at processing.mode.java.JavaMode.handleRun(JavaMode.java:176) at processing.mode.java.JavaEditor$20.run(JavaEditor.java:481) at java.lang.Thread.run(Thread.java:680)i would preciate it very much if you could explain me what’s happening
Amnon P5

java.io is part of the default java and contains some classes used in the loadPatternFilenames() function such as File and FilenameFilter. Some Processing versions already automatically include these, so an explicit import statement is not required, but for some it is required. So that’s why I added it explicitly. Your error points to a deeper issue that I suspect is either caused by the Processing version you use (very old version or very new alpha?) or the java installation on your computer. Try downloading Processing 1.5.1 STANDARD version (which includes java) for your operating system. If this does not work, please pinpoint the problem (remove all non-relevant code) and post to the Processing forum with more details (which line gives the error, which OS, which Processing version). Good luck.
crg

my processing version is the 1.5.1 and java is correctly updated as i double checked it. don’t know why or how, but reinstalled the IDE and first example now works perfectly.
thanks a lot for your time
Kristian_tapia

Hi everyone i tried to run the second program using processing and I get a strange error Unsatisfied LinkError: Unable to load library ‘ gstreamer-0.10′: dlopen (lib streamer-0.10.dylib,9): image not found any ideas? I’ve been searching for a solution but nothing….hope you can help I’m using Mac OSX 10.7.2
ggh

This was a lot of fun. In the end, everything worked perfect for me. I ran into a few issues, but nothing a Google search couldn’t solve. I would love to keep experimenting with this and I was wondering if anyone knew of some more resources I could use to learn about AR in processing. I guess my ultimate goal would be to combine AR in processing and my arduino.
lukas valiauga

how did you get nyARToolkit installed into libraries. I got 1.5.1processing, and 1.1.6nyartoolkit. When I add it to contributed libreries folder, after restarting program it just does not show up. Please help me !
Stkr

On example 1…. I keep getting a “NullPointerException” error with the following,

“The file “input.jpg” is missing or inaccessible, make sure the URL is valid or that the file has been added to your sketch and is readable.
Exception in thread “Animation Thread” java.lang.NullPointerException
    at sketch_jan26a.setup(sketch_jan26a.java:52)
    at processing.core.PApplet.handleDraw(Unknown Source)
    at processing.core.PApplet.run(Unknown Source)
    at java.lang.Thread.run(Thread.java:662)”

I’m new to this whole coding/AR thing, what files/code do I need to fix/add/change to get the example to work for me? Help would be appreciated, thanks!
Amnon P5

Hi Kristian. Since this is a specific issue on your system in relation to GSVideo, I suggest you use the Processing forum to look for a solution.
Amnon P5

Hi ggh. You could check out the early release of Making Things See by Greg Borenstein.
Amnon P5

Hi lukas. Change the directory name from “nyar4psg-1.1.6″ into “nyar4psg” (to match the jar file name in the library folder). Then it should get picked up by Processing.
Amnon P5

Hi stkr. As the text in example 1 explains you need to place the image that I provided (the one above with only the black and white markers) in the sketch’s data subdirectory. The file needs to be named “input.jpg”.
Stkr

Hello Amnon, for some reason it still gives me the same error message, I renamed the picture posted above (http://www.creativeapplications.net/wp-content/uploads/2011/12/Amnon-Owed-ARtut-01.jpg) (this is the image needed, right?) to ‘input.jpg’ like you said and put it in the data folder.

(path was libraries->documents->processing->sketch_jan26a->data (and the input.jpg file is in the data folder) that was the correct location right?
maria

Thanks for this!! It’s great. However, I can’t make it work on my computer. I don’t really understand how the webcam works. I have a mac, I don’t know if the code changes at all. My sketch window is grey and I get the following message:

This version of Processing only supports libraries and JAR files compiled for Java 1.5.A library used by this sketch was compiled for Java 1.6 or later, and needs to be recompiled to be compatible with Java 1.5.Exception in thread “Animation Thread” java.lang.UnsupportedClassVersionError: Bad version number in .class file at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:676) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:124) at java.net.URLClassLoader.defineClass(URLClassLoader.java:260) at java.net.URLClassLoader.access$100(URLClassLoader.java:56) at java.net.URLClassLoader$1.run(URLClassLoader.java:195) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:188) at java.lang.ClassLoader.loadClass(ClassLoader.java:317) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:280) at java.lang.ClassLoader.loadClass(ClassLoader.java:252) at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:375) at nyart.setup(nyart.java:59) at processing.core.PApplet.handleDraw(PApplet.java:1608) at processing.core.PApplet.run(PApplet.java:1530) at java.lang.Thread.run(Thread.java:655)

CreativeApplications.Net is your daily dose of inspiring apps for Mac, PC and iPhone...

Popular CAN Categories:

Look up past posts by date or category

On the Creative Application Network employers within the creative and technology fields are able to connect to skilled and talented job-seekers worldwide through direct access to thousands of daily visitors.

Apps.CreativeApplications.Net is a site dedicated to the collection of Apps for Mac, iPhone and iPad created by artists around the globe and published by CreativeApplications.Net.

Creative Application Network offers a number of resources to help you get started in the field of creative coding and physical computing.

Creative Application Network is always looking for support. See below how you can help.

Join, Submit and Share your projects with CAN using these social networks

Subscribe to

16,512

RSS Subscribers

Augmented Reality with Processing [Tutorial, Processing]