Please disable AdBlock. CAN is an ad-supported site that takes hundreds of hours and thousands of dollars to sustain. Read More.
HOLO is a biannual magazine about emerging trajectories in art, science, and technology brought to you by the people behind CAN. Learn more!

Augmented Reality with Processing [Tutorial, Processing]

All of the visuals in the above video were created using NyArtoolkit for Processing. NyARToolkit is an augmented reality toolkit built with 100% pure Java. It is derived from ARToolkit-2.72.1. Like Processing itself it’s open source and free! In this tutorial you will learn how to use it to place computer generated imagery correctly onto real world footage. To do this in real-time NyArtoolkit uses markers – black and white images – to determine the three-dimensional position and orientation in the real world. Most likely you will have seen something like this before, but now you will learn how to do it yourself using freely available software.

All right so let’s start with the general setup. What do you need to get started?

1. Processing 1.5.1
A Processing tutorial wouldn’t really work without Processing, now would it? So if you haven’t already, go HERE and download Processing for your OS. These examples were written for Processing 1.5.1 (the current stable build) but also with the upcoming Processing 2.0 in mind. I have personally tested all these examples in Processing 2.0a4 (alpha 4) and they work. When the time comes, you will probably notice that the default examples from the NyArtoolkit break in the Processing 2.0, due to the changes in the internal structure of the renderers. P3D is gone and there is a new (and vastly improved) OPENGL renderer in town. Go HERE to see the major changes in Processing 2.0. Just know that I have done my utmost to make sure these augmented reality code examples should run both under the current and future Processing builds.

2. NyArtoolkit for Processing 1.1.6
What it’s all about! Go HERE and download nyar4psg-1.1.6.zip. If you don’t know how to install contributed libraries in Processing, go HERE for instructions.

3. GSVideo 1.0.0
The second and third example use GSVideo to get a webcam stream into Processing. You could use alternatives if you want. I have chosen GSVideo because it works well and also because it will be integrated into Processing 2.0 as the new default video/capture library. So future readers, you might not need to seperately download this. For all of us living in the now, go HERE and download GSVideo for your OS. Again, if you don’t know how to install contributed libraries in Processing, go HERE for instructions.

4. Patterns
To make this whole thing work you will need black and white patterns. The NyArtoolkit includes two patterns (Hiro and Kanji), but these MultiMarker code examples can handle many more. There are a couple of online services that let you make your own markers and train them. But in my opinion, this is too much hassle. Fortunately you can download an ARToolkit Patternmaker from the university of Utah, which comes with 100 patterns and the associated training files! Go HERE to download the ARToolkit Patternmaker. Unpack this file and you will find 100 images (patternMaker\examples\gif) and 100 corresponding pattern files (patternMaker\examples\ARToolKit_Patterns). Very useful and these are the ones I used in the video. Very important: rename the first 10 pattern files to two-digits so they are loaded correctly in alphanumeric order in Processing. Also, if you use for example marker number 30, make sure to increase the number of loaded markers in these code example with the numMarkers variable.

All right, time to recap. So at this point you have Processing, the NyArtoolkit and GSVideo installed. In addition, you have downloaded the Patternmaker and therefore are in possession of 100 quality markers and the corresponding .patt files. If this is correct, you are all set and ready to get started with the code examples. All code examples are fully commented and kept as straight-forward as possible in order to maximize the educational value.

Example 1: Basic
The first example is basic, but holds all of the important techniques that are necessary for more advanced uses of the NyArtoolkit. I’ve purposefully started with an example that is image based for two reasons. First, because it will allow you to focus on getting the library working without any webcam troubles. ;) Second, to show that the NyArtoolkit is in essence image-based. This makes it extremely flexible, because any input will work! Be it a static image, an image sequence, a webcam capture or a movie. This is for example how I made the movie accompanying this blog post.

If you input the following image (place it in the sketch’s data subdirectory)…

…into the first code example, you should end up with something like this…


// Augmented Reality Basic Example by Amnon Owed (21/12/11)
// Processing 1.5.1 + NyARToolkit 1.1.6

import java.io.*; // for the loadPatternFilenames() function
import processing.opengl.*; // for OPENGL rendering
import jp.nyatla.nyar4psg.*; // the NyARToolkit Processing library

// a central location is used for the camera_para.dat and pattern files, so you don't have to copy them to each individual sketch
// Make sure to change both the camPara and the patternPath String to where the files are on YOUR computer
// the full path to the camera_para.dat file
String camPara = "C:/Users/mainframe/Documents/Processing/libraries/nyar4psg/data/camera_para.dat";
// the full path to the .patt pattern files
String patternPath = "C:/Users/mainframe/Documents/Processing/libraries/nyar4psg/patternMaker/examples/ARToolKit_Patterns";
// the dimensions at which the AR will take place. with the current library 1280x720 is about the highest possible resolution.
// it will work just as well at a lower resolution such 640x360, in some case a lower resolution even seems to work better.
int arWidth = 1280;
int arHeight = 720;
// the number of pattern markers (from the complete list of .patt files) that will be detected, here the first 10 from the list.
int numMarkers = 10;

MultiMarker nya;
float displayScale;
color[] colors = new color[numMarkers];
float[] scaler = new float[numMarkers];
PImage input, inputSmall;

void setup() {
size(1280, 720, OPENGL); // the sketch will resize correctly, so for example setting it to 1920 x 1080 will work as well
// create a text font for the coordinates and numbers on the boxes at a decent (80) resolution
textFont(createFont("Arial", 80));
// load the input image and create a copy at the resolution of the AR detection (otherwise nya.detect will throw an assertion error!)
input = loadImage("input.jpg");
inputSmall = input.get();
inputSmall.resize(arWidth, arHeight);
// to correct for the scale difference between the AR detection coordinates and the size at which the result is displayed
displayScale = (float) width / arWidth;
// create a new MultiMarker at a specific resolution (arWidth x arHeight), with the default camera calibration and coordinate system
nya = new MultiMarker(this, arWidth, arHeight, camPara, NyAR4PsgConfig.CONFIG_DEFAULT);
// set the delay after which a lost marker is no longer displayed. by default set to something higher, but here manually set to immediate.
nya.setLostDelay(1);
String[] patterns = loadPatternFilenames(patternPath);
// for the selected number of markers, add the marker for detection
// create an individual color and scale for that marker (= box)
for (int i=0; i nya.addARMarker(patternPath + "/" + patterns[i], 80);
colors[i] = color(random(255), random(255), random(255), 160); // random color, always at a transparency of 160
scaler[i] = random(0.5, 1.9); // scaled at half to double size
}
}

void draw() {
background(0); // a background call is needed for correct display of the marker results
image(input, 0, 0, width, height); // display the image at the width and height of the sketch window
nya.detect(inputSmall); // detect markers in the input image at the correct resolution (incorrect resolution will give assertion error)
drawMarkers(); // draw the coordinates of the detected markers (2D)
drawBoxes(); // draw boxes on the detected markers (3D)
}

// this function draws the marker coordinates, note that this is completely 2D and based on the AR dimensions (not the final display size)
void drawMarkers() {
// set the text alignment (to the left) and size (small)
textAlign(LEFT, TOP);
textSize(10);
noStroke();
// scale from AR detection size to sketch display size (changes the display of the coordinates, not the values)
scale(displayScale);
// for all the markers...
for (int i=0; i // if the marker does NOT exist (the ! exlamation mark negates it) continue to the next marker, aka do nothing
if ((!nya.isExistMarker(i))) { continue; }
// the following code is only reached and run if the marker DOES EXIST
// get the four marker coordinates into an array of 2D PVectors
PVector[] pos2d = nya.getMarkerVertex2D(i);
// draw each vector both textually and with a red dot
for (int j=0; j String s = "(" + int(pos2d[j].x) + "," + int(pos2d[j].y) + ")";
fill(255);
rect(pos2d[j].x, pos2d[j].y, textWidth(s) + 3, textAscent() + textDescent() + 3);
fill(0);
text(s, pos2d[j].x + 2, pos2d[j].y + 2);
fill(255, 0, 0);
ellipse(pos2d[j].x, pos2d[j].y, 5, 5);
}
}
}

// this function draws correctly placed 3D boxes on top of detected markers
void drawBoxes() {
// set the AR perspective uniformly, this general point-of-view is the same for all markers
nya.setARPerspective();
// set the text alignment (full centered) and size (big)
textAlign(CENTER, CENTER);
textSize(20);
// for all the markers...
for (int i=0; i // if the marker does NOT exist (the ! exlamation mark negates it) continue to the next marker, aka do nothing
if ((!nya.isExistMarker(i))) { continue; }
// the following code is only reached and run if the marker DOES EXIST
// get the Matrix for this marker and use it (through setMatrix)
setMatrix(nya.getMarkerMatrix(i));
scale(1, -1); // turn things upside down to work intuitively for Processing users
scale(scaler[i]); // scale the box by it's individual scaler
translate(0, 0, 20); // translate the box by half (20) of it's size (40)
lights(); // turn on some lights
stroke(0); // give the box a black stroke
fill(colors[i]); // fill the box by it's individual color
box(40); // the BOX! ;-)
noLights(); // turn off the lights
translate(0, 0, 20.1); // translate to just slightly above the box (to prevent OPENGL uglyness)
noStroke();
fill(255, 50);
rect(-20, -20, 40, 40); // display a transparent white rectangle right above the box
translate(0, 0, 0.1); // translate to just slightly above the rectangle (to prevent OPENGL uglyness)
fill(0);
text("" + i, -20, -20, 40, 40); // display the ID of the box in black text centered in the rectangle
}
// reset to the default perspective
perspective();
}

// this function loads .patt filenames into a list of Strings based on a full path to a directory (relies on java.io)
String[] loadPatternFilenames(String path) {
File folder = new File(path);
FilenameFilter pattFilter = new FilenameFilter() {
public boolean accept(File dir, String name) {
return name.toLowerCase().endsWith(".patt");
}
};
return folder.list(pattFilter);
}

Example 2: Dynamic
Time to get a little more dynamic. Both with regard to the input (webcam) and the output (moving graphics). Instead of a static box, this example shows a moving mountain terrain on each marker. I’ve added some suggestions in the code where you could tweak the look of this example. The mountain will grow (up to a certain height) while the marker is in sight and will flatten if the marker is no longer detected. You can see this behavior very clearly in the video above. Try it out yourself, by running the code example below in front of a webcam!

// Augmented Reality Dynamic Example by Amnon Owed (21/12/11)
// Processing 1.5.1 + NyARToolkit 1.1.6 + GSVideo 1.0

import java.io.*; // for the loadPatternFilenames() function
import processing.opengl.*; // for OPENGL rendering
import jp.nyatla.nyar4psg.*; // the NyARToolkit Processing library
import codeanticode.gsvideo.*; // the GSVideo library

// a central location is used for the camera_para.dat and pattern files, so you don't have to copy them to each individual sketch
// Make sure to change both the camPara and the patternPath String to where the files are on YOUR computer
// the full path to the camera_para.dat file
String camPara = "C:/Users/mainframe/Documents/Processing/libraries/nyar4psg/data/camera_para.dat";
// the full path to the .patt pattern files
String patternPath = "C:/Users/mainframe/Documents/Processing/libraries/nyar4psg/patternMaker/examples/ARToolKit_Patterns";
// the dimensions at which the AR will take place. with the current library 1280x720 is about the highest possible resolution.
int arWidth = 640;
int arHeight = 360;
// the number of pattern markers (from the complete list of .patt files) that will be detected, here the first 10 from the list.
int numMarkers = 10;

// the resolution at which the mountains will be displayed
int resX = 60;
int resY = 60;
// this is a 2 dimensional float array that all the displayed mountains use during their update-to-draw routine
float[][] val = new float[resX][resY];

GSCapture cam;
MultiMarker nya;
float[] scaler = new float[numMarkers];
float[] noiseScale = new float[numMarkers];
float[] mountainHeight = new float[numMarkers];
float[] mountainGrowth = new float[numMarkers];

void setup() {
size(1280, 720, OPENGL); // the sketch will resize correctly, so for example setting it to 1920 x 1080 will work as well
cam = new GSCapture(this, 1280, 720); // initializing the webcam capture at a specific resolution (correct/possible settings depends on YOUR webcam)
cam.start(); // start capturing
noStroke(); // turn off stroke for the rest of this sketch :-)
// create a new MultiMarker at a specific resolution (arWidth x arHeight), with the default camera calibration and coordinate system
nya = new MultiMarker(this, arWidth, arHeight, camPara, NyAR4PsgConfig.CONFIG_DEFAULT);
// set the delay after which a lost marker is no longer displayed. by default set to something higher, but here manually set to immediate.
nya.setLostDelay(1);
String[] patterns = loadPatternFilenames(patternPath);
// for the selected number of markers, add the marker for detection
// create an individual scale, noiseScale and maximum mountainHeight for that marker (= mountain)
for (int i=0; i nya.addARMarker(patternPath + "/" + patterns[i], 80);
scaler[i] = random(0.8, 1.9); // scaled a little smaller or bigger
noiseScale[i] = random(0.02, 0.075); // the perlin noise scale to make it look nicely mountainy
mountainHeight[i] = random(75, 150); // the maximum height of a mountain
}
}

void draw() {
// if there is a cam image coming in...
if (cam.available()) {
cam.read(); // read the cam image
background(0); // a background call is needed for correct display of the marker results
image(cam, 0, 0, width, height); // display the image at the width and height of the sketch window
// create a copy of the cam image at the resolution of the AR detection (otherwise nya.detect will throw an assertion error!)
PImage cSmall = cam.get();
cSmall.resize(arWidth, arHeight);
nya.detect(cSmall); // detect markers in the image
drawMountains(); // draw dynamically flowing mountains on the detected markers (3D)
}
}

// this function draws correctly placed 3D 'mountains' on top of detected markers
// while the mountains are displayed they grow (up to a certain point), while not displayed they return to the zero-state
void drawMountains() {
// set the AR perspective uniformly, this general point-of-view is the same for all markers
nya.setARPerspective();
// turn on some general lights (without lights it also looks pretty cool, try commenting it out!)
lights();
// for all the markers...
for (int i=0; i // if the marker does NOT exist (the ! exlamation mark negates it)...
if ((!nya.isExistMarker(i))) {
// if the mountainGrowth is higher than zero, decrease by 0.05 (return to the zero-state), then continue to the next marker
if (mountainGrowth[i] > 0) { mountainGrowth[i] -= 0.05; }
continue;
}
// the following code is only reached and run if the marker DOES EXIST
// if the mountainGrowth is lower than 1, increase by 0.03
if (mountainGrowth[i] < 1) { mountainGrowth[i] += 0.03; }
// the double for loop below sets the values in the 2 dimensional float array for this mountain, based on it's noiseScale, mountainHeight and index (i).
float xoff = 0.0;
for (int x=0; x xoff += noiseScale[i];
float yoff = 0;
for (int y=0; y yoff += noiseScale[i];
val[x][y] = noise(i*10+xoff+frameCount*0.05, yoff) * mountainHeight[i]; // this sets the value
float distance = dist(x, y, resX/2, resY/2);
distance = map(distance, 0, resX/2, 1, 0);
if (distance < 0) { distance = -distance; } // this line causing the four corners to flap upwards (try commenting it out or setting it to zero)
val[x][y] *= distance; // in the default case this makes the value approach zero towards the outer ends (try commenting it out to see the difference)
}
}

// get the Matrix for this marker and use it (through setMatrix)
setMatrix(nya.getMarkerMatrix(i));
scale(1, -1); // turn things upside down to work intuitively for Processing users
scale(scaler[i]); // scale the mountain by it's individual scaler
translate(-resX/2, -resY/2); // translate to center the mountain on the marker
// for the full resolution...
for (int x=0; x for (int y=0; y // each face is a Shape with a fill color, together they make a colored mountain
fill(x*20+y*20, 255-x*5, y*5);
beginShape();
vertex(x, y, val[x][y] * mountainGrowth[i]);
vertex((x+1), y, val[x+1][y] * mountainGrowth[i]);
vertex((x+1), (y+1), val[x+1][y+1] * mountainGrowth[i]);
vertex(x, (y+1), val[x][y+1] * mountainGrowth[i]);
endShape(CLOSE);
}
}
}
// reset to the default perspective
perspective();
}

// this function loads .patt filenames into a list of Strings based on a full path to a directory (relies on java.io)
String[] loadPatternFilenames(String path) {
File folder = new File(path);
FilenameFilter pattFilter = new FilenameFilter() {
public boolean accept(File dir, String name) {
return name.toLowerCase().endsWith(".patt");
}
};
return folder.list(pattFilter);
}

Example 3: OOP
The third example serves two purposes. First, you should realise that every (3D) sketch you can run in Processing, can be used in an augmented reality context. The only thing that AR changes is the perspective/camera. So every sketch which holds 3D shapes could work. To show this, I’ve taken an example that comes with Processing, the RGBCube, and integrated it – quite easily – into in AR context. The second purpose of this example, is to show the use of AR detection within an object-oriented (OOP) sketch. As sketches get a little more advanced, many people will turn to classes. It’s very easy to combine AR with classes. In this example it’s done through an ID, which is the link between the object (here a spinning RGBCube) and the corresponding marker. See the code how it works. In this example there are two part / tabs. The main sketch and the ARObject class. You will need both to run the example.

Main Sketch

// Augmented Reality RGBCube OOP Example by Amnon Owed (21/12/11)
// Processing 1.5.1 + NyARToolkit 1.1.6 + GSVideo 1.0

import java.io.*; // for the loadPatternFilenames() function
import processing.opengl.*; // for OPENGL rendering
import jp.nyatla.nyar4psg.*; // the NyARToolkit Processing library
import codeanticode.gsvideo.*; // the GSVideo library

MultiMarker nya;
GSCapture cam;

// this is the arraylist that holds all the objects
ArrayList ars = new ArrayList ();

// a central location is used for the camera_para.dat and pattern files, so you don't have to copy them to each individual sketch
// Make sure to change both the camPara and the patternPath String to where the files are on YOUR computer
// the full path to the camera_para.dat file
String camPara = "C:/Users/mainframe/Documents/Processing/libraries/nyar4psg/data/camera_para.dat";
// the full path to the .patt pattern files
String patternPath = "C:/Users/mainframe/Documents/Processing/libraries/nyar4psg/patternMaker/examples/ARToolKit_Patterns";
// the dimensions at which the AR will take place. with the current library 1280x720 is about the highest possible resolution.
int arWidth = 640;
int arHeight = 360;
// the number of pattern markers (from the complete list of .patt files) that will be detected, here the first 10 from the list.
int numMarkers = 10;
// the maximum rotation speed (x, y, z) at which the RGBCubes will rotate
float mS = 0.2;

void setup() {
size(1280, 720, OPENGL); // the sketch will resize correctly, so for example setting it to 1920 x 1080 will work as well
cam = new GSCapture(this, 1280, 720); // initialize the webcam capture at a specific resolution (correct and/or possible settings depend on YOUR webcam)
cam.start(); // start capturing
// initialize the MultiMarker at a specific resolution (make sure to input images for detection EXACTLY at this resolution)
nya = new MultiMarker(this, arWidth, arHeight, camPara, NyAR4PsgConfig.CONFIG_DEFAULT);
// set the delay after which a lost marker is no longer displayed. by default set to something higher, but here manually set to immediate.
nya.setLostDelay(1);
// load the pattern filenames (markers)
String[] patterns = loadPatternFilenames(patternPath);
// for the selected number of markers...
for (int i=0; i // add the marker for detection
nya.addARMarker(patternPath + "/" + patterns[i], 80);
// and create an ARObject with the corresponding 'ID'
ars.add(new ARObject(i));
}
// set the color range to 1 (instead of 255), saves typing for the coloring of the cube
colorMode(RGB, 1);
// turn off stroke for the rest of the sketch
noStroke();
}

void draw() {
// if there is a cam image coming in...
if (cam.available()) {
cam.read(); // read the cam image
background(0); // a background call is needed for correct display of the marker results
image(cam, 0, 0, width, height); // display the image at the width and height of the sketch window
// create a copy of the cam image at the resolution of the AR detection (otherwise nya.detect will throw an assertion error!)
PImage cSmall = cam.get();
cSmall.resize(arWidth, arHeight);
nya.detect(cSmall); // detect markers in the image
// set the AR perspective uniformly, this general point-of-view is the same for all markers
nya.setARPerspective();
// run all the ARObjects's in the arraylist => most things are handled inside the ARObject (see the class for more info)
for (ARObject ar : ars) { ar.run(); }
// reset to the default perspective
perspective();
}
}

// this function loads .patt filenames into a list of Strings based on a full path to a directory (relies on java.io)
String[] loadPatternFilenames(String path) {
File folder = new File(path);
FilenameFilter pattFilter = new FilenameFilter() {
public boolean accept(File dir, String name) {
return name.toLowerCase().endsWith(".patt");
}
};
return folder.list(pattFilter);
}

ARObject

// class that defines the AROBject, both the AR detection and display are handled inside this class
class ARObject {
int ID; // keep track of the current the ID of the object (corresponds with the ID i of the marker)
PVector rot, speed; // in this example the cube has a certain rotation and rotates at a certain speed

ARObject(int ID) {
this.ID = ID; // set the ID
rot = new PVector(random(TWO_PI), random(TWO_PI), random(TWO_PI)); // random x, y, z rotation
speed = new PVector(random(-mS, mS), random(-mS, mS), random(-mS, mS)); // random x, y, z speed (within maxSpeed boundaries)
}

void run() {
// always keep rotating (even when the marker is NOT detected)
rot.add(speed);
// checks the object's corresponding marker through the ID
// if the marker is found, display the cube
if (nya.isExistMarker(ID)) { display(); }
}

// the display in this example shows a colored, rotating RGBCube
void display () {
// get the Matrix for this marker and use it (through setMatrix)
setMatrix(nya.getMarkerMatrix(ID));
scale(1, -1); // turn things upside down to work intuitively for Processing users

// hover the cube a little above the real-world marker image
translate(0, 0, 30);

// rotate the cube in 3 dimensions
rotateX(rot.x);
rotateY(rot.y);
rotateZ(rot.z);

// scale - as with the the color range - to save typing with the coordinates (and make it much easier to change the size)
scale(15);

// a cube made out of 6 quads
// the 1 range can be used for both the color and the coordinates as a result of color range and scale (see earlier)
beginShape(QUADS);

fill(0, 1, 1); vertex(-1, 1, 1);
fill(1, 1, 1); vertex( 1, 1, 1);
fill(1, 0, 1); vertex( 1, -1, 1);
fill(0, 0, 1); vertex(-1, -1, 1);

fill(1, 1, 1); vertex( 1, 1, 1);
fill(1, 1, 0); vertex( 1, 1, -1);
fill(1, 0, 0); vertex( 1, -1, -1);
fill(1, 0, 1); vertex( 1, -1, 1);

fill(1, 1, 0); vertex( 1, 1, -1);
fill(0, 1, 0); vertex(-1, 1, -1);
fill(0, 0, 0); vertex(-1, -1, -1);
fill(1, 0, 0); vertex( 1, -1, -1);

fill(0, 1, 0); vertex(-1, 1, -1);
fill(0, 1, 1); vertex(-1, 1, 1);
fill(0, 0, 1); vertex(-1, -1, 1);
fill(0, 0, 0); vertex(-1, -1, -1);

fill(0, 1, 0); vertex(-1, 1, -1);
fill(1, 1, 0); vertex( 1, 1, -1);
fill(1, 1, 1); vertex( 1, 1, 1);
fill(0, 1, 1); vertex(-1, 1, 1);

fill(0, 0, 0); vertex(-1, -1, -1);
fill(1, 0, 0); vertex( 1, -1, -1);
fill(1, 0, 1); vertex( 1, -1, 1);
fill(0, 0, 1); vertex(-1, -1, 1);

endShape();
}
}

Troubleshooting & General Tips
- If you get an "Exception occurred at Multimarker.Multimarker" you probably haven't changed the camPara String. In all the examples above it points to the location of the file on MY computer. You have to change it to the location of the file on YOUR computer. Also make sure you have (the latest) NyArtoolkit for Processing 1.1.6 installed.
- NullPointerException with regard to the patterns. Most likely the path to the patterns is incorrect. Use forward slashes in the pattern path. Don’t put a slash at the end.
- You get no error messages, but the sketch window remains grey. This means there is no incoming webcam imagery or perhaps no webcam at all. Make sure your webcam is working correctly.
- You get no error messages, the marker is clearly on the screen, but not detected. Most likely you haven't loaded it's pattern file. In the code examples you load the first 'numMarkers' from the list. Your marker apparantly is alphanumerically not within the range of loaded patterns. Make sure it is and it's loaded.
- For bigger issues (errors, questions), please use the Processing forum.

I hope you have learned something and this tutorial has been useful to get you off to a running start with augmented reality. Of course this is only the tip of the iceberg. There is a whole world of possibilities. Good luck with these code examples and more!

    • http://twitter.com/Darkmatterial Alex Nolla

      Just tried the tutorial, and everything works nicely, except that when I press play and the window opens the video crashes :( I can see the cubes and move them but the background is complete black. Could it be my web cam?
      I have windows 7 (64 bit)
      Thanks

    • http://twitter.com/Darkmatterial Alex Nolla

      The image crashes but no error message. And I can still see and move the cubes, what could it be?

    • http://twitter.com/Darkmatterial Alex Nolla

      I will build an AR sculpture in your honor!

    • Guest

      many ppl did make questition no one thar answer ???
      what kind of comment are this?
      all ppl want build , but no one want share his experience ??

      this is not a good , for future of net !

    • bob Desnos

      many people requesting informationbut because no one answers?
      many unnecessary comments
      no constructive
      if that ‘the future of the network…

    • Will

      Hello!

      I have a problem with gstreamer, processing couldn’t recognize

    • Will

      I have found the solution, you should check all gstreamer app in synaptic package (ubuntu), later, you should apply and restart your pc

    • Cara

      Hi,
      I’ve some issues loading the NyAR4…
      I tried the “still having trouble” of the install library faq but now it says it cannot load jp.
      does anyone help me? thanks

    • Andy Crowe

      Is there a way to rewrite the code so that it picks up the pattern files from the data folder and not needing to do a filepath? That seems to be the only hiccup for me… Help would be appreciated :D

    • Andy Crowe

      Excellent Aart! This worked for me. I knew I had to figure out to rewrite the code to pick up the pattern folder in the data folder.

      Thanks again!

    • Thiag Bussiki

      For the Black Screen solution, change “background(0)” to “background(inputSmall)”

    • zack

      Hi I created an AR Code based off of this tutorial, I can get it to work in processing 1.5 but not 2.0, For some reason in the latest version I can’t get it to open up the cam, any idea why? There are no error messages. Here is my code below

      import processing.video.*;
      import jp.nyatla.nyar4psg.*;

      Movie myMovie;
      Capture cam;
      MultiMarker nya;

      void setup() {
      size(1280,720,P3D);
      colorMode(RGB, 100);
      println(MultiMarker.VERSION);
      cam=new Capture(this,1280,720);
      nya=new MultiMarker(this,width,height,”camera_para.dat”,NyAR4PsgConfig.CONFIG_PSG);
      nya.addARMarker(“marker16.pat”,80);//id=0
      nya.addARMarker(“marker17.pat”,80);//id=1
      myMovie = new Movie(this, “fullrender.mp4″);
      myMovie.loop();
      myMovie.play();
      }

      void draw()
      {
      if (cam.available() !=true) {
      return;
      }
      cam.read();
      nya.detect(cam);
      background(0);
      //frustumを考慮した背景描画
      for(int i=0;i<2;i++){
      if((!nya.isExistMarker(i))){
      continue;
      }
      nya.beginTransform(i);
      scale(0.1,0.1,1);
      rotateZ(PI/2);
      translate(0,0,20);
      image(myMovie, -640, -360);
      nya.endTransform();
      }
      }

    • Brendan Flynn

      aaaaahhhhhhhh I cant wait, you are a legend.

    • Geert

      Hi, i am at the 2nd example and i am getting a nullpointerexeption at this line:

      nya.addARMarker(patternPath + “/” + patterns[i], 80);

      I have directed the pattern maker and cam para.dat file correctly, and GSvideo starts running, but i get a long and very confusing error message..

      Anyone with a bit of help?

      Begcomment />

    • Brendan Flynn

      First Sketch. Basic Example. having issue. nya = new MultiMarker(this, arWidth, arHeight, camPara, NyAR4PsgConfig.CONFIG_DEFAULT); Highlights this line and gives me the error – Exception occurred at MultiMarker.MultiMarker Any suggestions welcome.

    • Brendan Flynn

      I have same problem, you solve it ?

    • Brendan Flynn

      bloody hell, all mine was i left the c: in the path, on the Mac. :(

    • Brendan Flynn

      as usual, silly mistake, in the paths I left C: and I am using a Mac, I was like “c drive, i dont have a c:” :) works a charm now,

    • Brendan Flynn

      Also would like to add Amnon, is a beautiful person. Thank You again, kind Sir.

    • Nicola_97

      i want to do the example of the rgb cube but i can’t undertand what i have to modify for make the script correct

    • Nicola_97

      this is the problems

    • Ruitao Yang

      Great job!

    • Jaadu

      Hi… excellent tutorial……i tried first sketch on windowXP with processing 2.0b6….first basic example

      having issue. nya.addARMarker(patternPath + “/” + patterns[i], 80); Highlights this line and gives me the error – NullPointer Exception

      Exception in thread “Animation Thread”

      java.lang.NullPointerException

    • Jaadu

      Hi..

      I am trying to do this with the 3D file. But don’t know how to map 3D files with patterns……in my case 3D image is just stay on screen and there is no tracking. There is no response by image for the pattern. your early help will be appreciated….thanking you in anticipation.

      Here is my code

      import processing.opengl.*;

      import javax.media.opengl.*;

      import java.io.*;

      import processing.video.*;

      import jp.nyatla.nyar4psg.*;

      import mri.*;

      int arWidth = 640;

      int arHeight = 360;

      V3dsScene vscene;

      Capture cam;

      MultiMarker nya;

      String camPara = “C:/Documents and Settings/javed.hasan/My Documents/Processing/libraries/nyar4psg/data/camera_para.dat”;

      String patternPath = “C:/Documents and Settings/javed.hasan/My Documents/Processing/libraries/nyar4psg/patternMaker/examples/ARToolKit_Patterns”;

      int numMarkers = 10;

      void setup(){

      size(640,480,OPENGL);

      hint( ENABLE_OPENGL_4X_SMOOTH );

      smooth();

      frameRate( 60 );

      try{

      vscene=new V3dsScene(this,”Worker Chainsaw N100612.3DS”);

      GL gl = ((PGraphicsOpenGL)g).beginGL();

      ((PGraphicsOpenGL)g).endGL();

      vscene = new V3dsScene( gl, dataPath(“Worker Chainsaw N100612.3DS”) );

      vscene.useMaterial(true);

      }

      catch(Exception e){

      System.out.println(e);

      System.exit(0);

      }

      colorMode(RGB,100);

      println(MultiMarker.VERSION);

      cam=new Capture(this,640,480);

      camera();

      nya = new MultiMarker(this, arWidth, arHeight, camPara, NyAR4PsgConfig.CONFIG_DEFAULT);

      nya.setLostDelay(1);

      // load the pattern filenames (markers)

      String[] patterns = loadPatternFilenames(patternPath);

      // for the selected number of markers…

      for (int i=0; i most things are handled inside the ARObject (see the class for more info)

      drawMarkers();

      // reset to the default perspective

      perspective();

      }

      }

      void drawMarkers()

      {

      float time = millis() * 0.001;

      // background( cam );

      perspective( -PI*0.2, 4.0/3.0, 1, 5000 );

      GL _gl = ((PGraphicsOpenGL)g).beginGL();

      camera( 0, -190, -250, 0, -3*sin(time)*2, 8, 0, 1, 0 );

      // in case you want to use a custom GL object (usually for eclipse projects)

      _gl.glRotatef( 10*time, 0, 1, 0 );

      vscene.draw( _gl );

      ((PGraphicsOpenGL)g).endGL();

      rotateY( radians(10*time) );

      }

      // this function loads .patt filenames into a list of Strings based on a full path to a directory (relies on java.io)

      String[] loadPatternFilenames(String path) {

      File folder = new File(path);

      FilenameFilter pattFilter = new FilenameFilter() {

      public boolean accept(File dir, String name) {

      return name.toLowerCase().endsWith(“.patt”);

      }

      };

      return folder.list(pattFilter);

      }

    • Jaadu

      Hi..

      I am trying to upload .OBJ & .3DS files and track them with pattern image. I don’t know how to map coordinates of 3D file with pattern image……i try hard but the result is poor tracking

      Here is my code

      any help would be appreciated…

      import java.io.*; // for the loadPatternFilenames() function

      import processing.opengl.*; // for OPENGL rendering

      import jp.nyatla.nyar4psg.*; // the NyARToolkit Processing library

      import codeanticode.gsvideo.*; // the GSVideo library

      import codeanticode.glgraphics.*;

      import saito.objloader.*;

      MultiMarker nya;

      GSCapture cam;

      // declare that we need a OBJModel and we’ll be calling it “model”

      OBJModel model;

      // these booleans will be used to turn on and off bits of the OBJModel

      //boolean bTexture = true;

      //boolean bStroke = false;

      //boolean bMaterial = false;

      String camPara = “C:/Documents and Settings/javed.hasan/My Documents/Processing/libraries/nyar4psg/data/camera_para.dat”;

      // the full path to the .patt pattern files

      String patternPath = “C:/Documents and Settings/javed.hasan/My Documents/Processing/libraries/nyar4psg/patternMaker/examples/ARToolKit_Patterns”;

      // the dimensions at which the AR will take place. with the current library 1280×720 is about the highest possible resolution.

      int arWidth = 640;

      int arHeight = 360;

      // the number of pattern markers (from the complete list of .patt files) that will be detected, here the first 10 from the list.

      int numMarkers = 10;

      void setup()

      {

      size(640, 480, P3D);

      // making an object called “model” that is a new instance of OBJModel

      model = new OBJModel(this,”Lancer.obj”, “relative”);

      cam = new GSCapture(this, 640, 480); // initialize the webcam capture at a specific resolution (correct and/or possible settings depend on YOUR webcam)

      camera(); // start capturing

      // initialize the MultiMarker at a specific resolution (make sure to input images for detection EXACTLY at this resolution)

      nya = new MultiMarker(this, arWidth, arHeight, camPara, NyAR4PsgConfig.CONFIG_DEFAULT);

      // set the delay after which a lost marker is no longer displayed. by default set to something higher, but here manually set to immediate.

      nya.setLostDelay(1);

      // load the pattern filenames (markers)

      String[] patterns = loadPatternFilenames(patternPath);

      // for the selected number of markers…

      for (int i=0; i<numMarkers; i++) {

      // add the marker for detection

      nya.addARMarker(patternPath + "/" + patterns[i], 80);

      // turning on the debug output (it's all the stuff that spews out in the black box down the bottom)

      model.enableDebug();

      // model.translateToCenter();

      model.scale(.9);

      // model.printModelInfo();

      noStroke();

      }

      }

      void draw()

      {

      if (cam.available()) {

      cam.read(); // read the cam image

      background(cam); // a background call is needed for correct display of the marker results

      image(cam, 0, 0, width, height); // display the image at the width and height of the sketch window

      // create a copy of the cam image at the resolution of the AR detection (otherwise nya.detect will throw an assertion error!)

      PImage cSmall = cam.get();

      cSmall.resize(arWidth, arHeight);

      nya.detect(cSmall); // detect markers in the image

      // set the AR perspective uniformly, this general point-of-view is the same for all markers

      nya.setARPerspective();

      drawimage();

      // reset to the default perspective

      perspective();

      }

      }

      void drawimage()

      {

      float time = millis() * 0.0001;

      nya.setARPerspective();

      lights();

      for (int i=0; i<numMarkers; i++) {

      // if the marker does NOT exist (the ! exlamation mark negates it)…

      if ((!nya.isExistMarker(i))) { continue; }

      //this will do nothing until the model material is turned off

      // fill(255,0,255);

      setMatrix(nya.getMarkerMatrix(i));

      pushMatrix();

      translate(width/2, height/2, 1);

      rotateY( -3*sin(time)*2);

      model.draw();

      popMatrix();

      }

      }

      String[] loadPatternFilenames(String path) {

      File folder = new File(path);

      FilenameFilter pattFilter = new FilenameFilter() {

      public boolean accept(File dir, String name) {

      return name.toLowerCase().endsWith(".patt");

      }

      };

      return folder.list(pattFilter);

      }