Featured, openFrameworks, Tutorials
comments 16

Arduino + Servo + openCV Tutorial [openFrameworks]

contour

One of the my favorite things about creativeapplications.net has always been the small tags one can find beneath the name of an application indicating among other things, the technology used to create it. That little nod to the process and to all the work that went into creating the libraries and techniques that an artist or designer uses helps not only contextualize the work but it also helps give recognition to everyone who has contributed their time and expertise to building tools for creative expression in code. Figuring that some of the readers might be interested in learning a little more about these frameworks I’ve put together a quick walk-through of how to connect up two of those tools that one so often sees attached to the names of the projects profiled here: openFrameworks and Arduino.

Many of you are probably familiar with at least the basics of the oF and Arduino projects, but for anyone not familiar here’s the briefest of brief explanations: oF is a C++ library for creative coding that helps artists and designers quickly create powerful applications for Windows, Linux, or OSX. Arduino is both a library and a microprocessor platform for prototyping and tinkering with microprocessors and embedded computing that simplifies the process of connecting and programming a fantastic range of components and electronic devices. Connecting the two of them allows you to create applications that not only use processing-heavy libraries like the OpenCV library for Computer Vision (more on that later) but also connect to and communicate with electronics out in the physical world. In this tutorial we’ll be using motion tracking in an openFrameworks application to control servo motors connected to an Arduino board.

Arduino

The Arduino is a few different things: an IDE, a library for embedded computing, and a microprocessor and that can be a little confusing because people often use the terms interchangeably but after a little experience you’ll see how they all work together.

For this tutorial you’ll need a few things:

1 x Arduino-compatible device
1 x Trossen Servokit
1 x USB cable
1 x Breadboard and wires to connect the Servos to the Arduino

You’ll also need to have the Arduino IDE installed on your computer as well as an IDE and the openFrameworks libraries to work with openFrameworks. You can get the Arduino IDE here and oF here

Serial Communication

Communication between an Arduino board and an openFrameworks application is conducted through the Serial port. The most common setup looks like this: an Arduino gets connected to a computer via a USB cable. The Arduino application starts listening for data at a certain baud rate (read: speed) and the openFrameworks starts listening on a certain port (since a computer has multiple) and at a certain baud rate. Voila, your Arduino is talking to your computer.

Arduino and servo-motors

A servo is a type of motor but with a few added advantages. Most important:the pulse of a signal to determine how far to rotate within its range. The most common range for a servo motor is 180 degrees but there are other types as well and the servo can move from any point in that range to any other point clockwise or counterclockwise, making it a powerful tool for controlling anything that needs a controllable, repeatable movement. Robots of all kinds, puppets,  factory machinery of all kinds, all make extensive use of servos to function.

To connect a servo motor to the Arduino controller, you need to provide the servo with power from the 5V pin on the Arduino controller, to ground the servo, and to send commands to the servo PWM port. To control the servo, you send it a command every 20 milliseconds. The servo responds to a short pulse (1ms or less) by moving to one extreme, and it moves around 180 degrees in the other direction when it receives long pulses (around 2ms or more). The amount that the servo moves is determined by the length of time that the pulse is sent to the servo (Figure 11-7). You should be aware that there is no standard for the exact relationship between pulse width and position; different servos may need slightly longer or shorter pulses.

Servo motors have three wires: power, ground, and signal, as Figure 11-8 shows. The power wire is typically red and should be connected to the 5V pin on the Arduino board, though some servo motors require more power than the Arduino can provide. The ground wire is typically black or brown and gets connected to a ground pin on the Arduino board. The yellow (or white) wire of the servo goes to the digital pin that you’re using to control the servo.

Now, let’s take a look at the code involved:

 #include <Servo.h>
 

Here are the two instances of the Servo object that will control each servo motors:

Servo vert;
Servo horz;
byte xy[2]; // this array is what the Serial data will be written to

To start controlling the servo, simply call the attach() method of the Servo object:

void setup() {
   horz.attach(3);
   vert.attach(5);
   Serial.begin(9600); // start the communication between the oF app and the Arduino
}

In the loop(), simply check for any data on the Serial port, and if anything has arrived, use it to set the positions of the servos with the write() method.

void loop() {
   if(Serial.available() > 1) {
      xy[0] = Serial.read();
      xy[1] = Serial.read();
      // now that we have the xy we can go ahead
      // and write them to the serial
      float ratio = 180.0/255.0;
      horz.write(xy[0] * ratio);
      vert.write(xy[1] * ratio);
      delay(20); // make sure we give it a second to move
      }
}

openFrameworks and OpenCV

As mentioned earlier, openFrameworks is a C++ library for creative coding and that means that in order to get started working with it, you’ll need a way to compile your application for your computer and operating system. If you take a look at openframeworks.cc/downloads you’ll find a download for your operating system and chosen IDE as well as instructions on how to get started. For this tutorial, you’ll want to get the FAT download. That is, the download that contains all the additional addons, including the OpenCV library that we’ll be using. If you run into any problems the community on the forums at openframeworks.cc/forums are incredibly helpful. You might want to take a moment to get set up if you’re not already.

A team of developers at Intel headed by Gary Bradski started developing OpenCV in 1999. In 2007, Intel released version 1.0 as an open source project. Since it’s an open source library, it’s free to download, and you can use and modify it for your own purposes. Some of the core features that it provides are an interface to a camera, object detection and tracking, and face and gesture recognition. The library itself is written in C and is a little tricky to get started using right away without another library or frame- work to interface with it. Luckily, Stefan Hechenberger and Zachary Lieberman developed the ofxOpenCV add-on for openFrameworks to help anyone working with oF use OpenCV easily in their projects.

Movement tracking in OpenCV is done through what is commonly called a blob, which is a data object that represents a contiguous area of a frame. Blobs can be all sorts of different things: hands, faces, a flashlight, a ball. Blobs are usually identified by the contrast caused by the difference between adjoining pixels. A group of adjoining pixels that have the same relationship to another group of adjoining pixels, such as an area of darker pixels that is bordered by an area of whiter pixels, usually indicates a specific object. To do that in your application you retrieve an image from the camera and pass it to an object called a contourFinder that looks for contours and then tries to create blobs from those. The image below shows an image passed to a ContourFinder instance and the detected contours:

Now we’re ready to look at some code. I’m only going to highlight certain areas of the code, since everything else is available in the download for this tutorial. Like the Arduino application every oF applicaiton has a setup() method that is called when the it starts up. In our case, we want to set up the camera that we’ll be using to find blobs and start the communication between the Arduino and oF application:

void servoApp::setup() {
   threshold = 60;
   bLearnBakground = true;

The enumerateDevices() method prints out all of the devices attached to your computer and allows you to see what your oF application thinks is attached. This is great way to avoid typos in the device name, which can be a little unfriendly at times.

   serial.enumerateDevices();
   //serial.setup("COM4"); // windows will look something like this.
   serial.setup("/dev/cu.usbserial-A9007KHo",9600); // mac osx looks like this
   //serial.setup("/dev/ttyUSB0", 9600); //linux looks like this

In the final video I’m using two cameras: one to read to do motion tracking and one attached to my servo kit to show the movement of the servos. That’s why you see vidGrabber and vidGrabber2. If you’re not using two cameras, feel free to comment all the code for vidGrabber2 as it isn’t doing anything vital to motion tracking or communicating with the Arduino.

   vidGrabber.setDeviceID(5); // this isn't necessary if you're only using one camera
   vidGrabber.initGrabber( 320, 240 );
   colorImg.allocate( 320, 240 );
   grayImg.allocate( 320, 240 );
   bgImg.allocate( 320, 240 );
   // here's the setup for the second camera
   vidGrabber2.setDeviceID(4); // this isn't necessary if you're only using one camera
   vidGrabber2.initGrabber(320, 240, true);
}

Like the Arduino applications’ loop() method, an oF application has an update() method that is called every frame. This is where you want to do any data processing or reading from or writing to peripheral devices like cameras or Arduinos. The grabFrame() method of the videoGrabber is what tells the underlying operating system to get a new frame from the camera and store it in memory so that it can be analyzed and displayed.

[sourcecode]
void servoApp::update() {
vidGrabber.grabFrame();
vidGrabber2.grabFrame();
int i, len, largest;

Usually you’ll want to check whether the frame is new since the application might be operating at a slightly different speed than the device driver for your camera. That’s another thing that oF makes very easy, with the aptly named isFrameNew() method. Another thing worth pointing out is that the images are all converted to grayscale before they’re passed to the contourFinder. This is because grayscale images are inherently smaller than color images, black/white requiring less data to represent a pixel than Red/Green/Blue.


   if( vidGrabber.isFrameNew() ) {
      colorImg = vidGrabber.getPixels();
      colorImg.mirror( false, true );
      grayImg = colorImg;
      if( bLearnBakground ) {
        bgImg = grayImg;
        bLearnBakground = false;
      }
      grayImg.absDiff( bgImg );
      grayImg.blur( 11 );
      grayImg.threshold( threshold );

Now, after converting the image to greyscale it gets passed to the contourFinder to process the image and see whether there are any blobs. The parameters for the findContours() method are:

image – the image to process
minSize – the smallest size of a blob
maxSize – the largest size of a blob
numberOfBlobs – the number of blobs to look for
findHoles – whether to look for holes in the blobs or not

      contourFinder.findContours( grayImg, 50, 20000, 10, false );
      largest = -1;
      i = 0;

Now that the contours have been processed we can try to figure out which is the largest since that’s probably the one we’re interested in.

      if(contourFinder.blobs.size() > 0) {
         int len = contourFinder.blobs.size() - 1;
         while(i<len){
            if(largest == -1) {
               largest = i;
            } else if(contourFinder.blobs.at(i).area > contourFinder.blobs.at(largest).area) {
               largest = i;
            }
            i++;
      }

This is important: we don’t want to send data to the serial port on every frame because the Arduino runs much more slowly than the oF app. The ofGetFrameNum method gives the number of frames that the application has been running. The modulo operator (%) returns the remaineder of the number on the left when divided by the number on the right. In this case, the net effect of this if() statement is to only write data to the serial once every 4 frames, or about 15 frames a second, which is about as much as the Arduino can process.

      if(ofGetFrameNum() % 4 == 0) {
         if(largest != -1) {
            serial.writeByte(contourFinder.blobs.at(largest).centroid.y/240 * 255);
            serial.writeByte(contourFinder.blobs.at(largest).centroid.x/320 * 255);
          }
      }
}

The rest of the code is fairly straightforward and includes comments to help you understand what’s going on.

You can download the code for both the Arduino and oF apps at: http://thefactoryfactory.com/ca_of_arduino.zip

This short video shows how the servos are connected to the Arduino board and how you might use the code:

Joshua Noble is a writer, designer, and programmer based in Portland, Oregon and New York City. He’s the author of, most recently, Programming Interactivity and the forthcoming book Research for Living.

  • http://r0botics.com

    Very nice! thank you for the tutorial it's very interesting

  • http://www.facebook.com/profile.php?id=551533666 Marcus Åslund

    Great article and an inspiring project! Could you please specify which parts you used for the Trossen servos setup? Thanks!

  • joshua noble
  • Sermad

    Very very cool. Is your website up?

  • http://twitter.com/ElectronicsFun Electronics Is Fun

    Great tutorial, it shows a great use of the arduino.
    Thank You,
    GuruSantiago

    Want to learn more about electronics?

    The GuruSantiago can help. Checkout his videos here:

    http://www.youtube.com/user/ElectronicsIsFun
    And follow him on twitter @ElectronicsFun

  • Sharonvella401

    will you please do a tutorial about sending data from matlab simulink to arduino? on the internet there is a lack of such tutorials. thanks … it will be very helpful.

  • Ouazar Yahia

    Hi
    Can I interface Arduino with a program in c + + which is included in opencv

  • wofty

    Hello, I have only the video test with hands, and no webcam. What I must modify for using a webcam ?
    Thanks

  • http://www.facebook.com/profile.php?id=771095600 Abishek Hariharan

    can the camera attached to the servos be used for the tracking instead ? , or does this method require a static camera ?

  • http://www.facebook.com/profile.php?id=646606626 Jonathan Mercieca

    Do you physically connect a wire from serial port of computer to arduino (through a RS-232 driver/receiver) or do you use the Arduino Software Serial Interface?

  • http://profiles.google.com/franck.bouyon Franck BOUYON

    Hi,do you know where can i find the same code for Processing.
    Thanks

  • http://www.facebook.com/profile.php?id=771095600 Abishek Hariharan

    I implemented your code(modified to suit) on an arduino mega 1280 , with a JR ES 539 servo setup , the motion i get is very erratic and doesnt seem to be tracking at all. help.

  • Valentin_dr

    wher do i find the all code you write…i`m very interested in that project.

  • Jonathan

    What kind of camera are you using? Where is it mounted? All i can see on your servo is what looks like a reflector? It would be really helpful if you could elaborate on your setup. Thanks a million for this great tutorial!

  • Jonathan

    Sorry i missed the explaination that you are using two cameras, one for tracking and one to show movement. I am still curious as to the camera type that you are using? Can you use a wireless webcam and input the video from the wireless network and still get that image processed in OF?

    Thank you again for your time!

  • Conoco75

     What OF version of the FAT download were using?