Please disable AdBlock. CAN is an ad-supported site that takes hundreds of hours and thousands of dollars to sustain. Read More.
HOLO is a biannual magazine about emerging trajectories in art, science, and technology brought to you by the people behind CAN. Learn more!

FaceShift Studio experiments by Kyle McDonald

Kyle McDonald has been exploring possibilities of face tracking using FaceShift Studio, an affordable markerless facial performance capture in development since 2009 and presented presented over the years at the SIGGRAPH conference. The tutorial below show you how to connect it to other apps in realtime via OSC with FaceShiftOSC, and connect to openFrameworks via his ofxFaceShift addon.

Kyle’s tutorial begins with the instruction how to use your Kinect to generate a nurb model of your face. Once you have gone through the process, FaceMesh is able to translate your face movements directly onto the three dimensional model in realtime. The model is so precise that besides capturing your facial expressions extremely well it is also capturing your eye movement.

The next step is to capture the movements though the OSC server and reflect the same in the openFrameworks app Kyle has written. All the data is incoming, even the eye movement. Finally, the OSC data is fed into Ableton, and you are able to generate sounds using your facial movement.

Kyle has made all the code and examples available online so you can have a play.

FaceShift intro
FaceShiftOSC setup
ofxFaceShift overview
Sound design from

Kyle McDonald

    • Joel Colombo

      This is wicked, thanks for sharing this stuff! Although I’m a designer, I’m learning a lot :)

    • john

      this is fascinating, you should make an app for this is would be major

    • Fernando Magalhães