openFrameworks, Sound, Tutorials
comments 3

FaceShift Studio experiments by Kyle McDonald

faceshift01 copy

Kyle McDonald has been exploring possibilities of face tracking using FaceShift Studio, an affordable markerless facial performance capture in development since 2009 and presented presented over the years at the SIGGRAPH conference. The tutorial below show you how to connect it to other apps in realtime via OSC with FaceShiftOSC, and connect to openFrameworks via his ofxFaceShift addon.

Kyle’s tutorial begins with the instruction how to use your Kinect to generate a nurb model of your face. Once you have gone through the process, FaceMesh is able to translate your face movements directly onto the three dimensional model in realtime. The model is so precise that besides capturing your facial expressions extremely well it is also capturing your eye movement.

The next step is to capture the movements though the OSC server and reflect the same in the openFrameworks app Kyle has written. All the data is incoming, even the eye movement. Finally, the OSC data is fed into Ableton, and you are able to generate sounds using your facial movement.

Kyle has made all the code and examples available online so you can have a play.

FaceShift intro
FaceShiftOSC setup
ofxFaceShift overview
Sound design from

Kyle McDonald

  • This is wicked, thanks for sharing this stuff! Although I’m a designer, I’m learning a lot :)

  • john

    this is fascinating, you should make an app for this is would be major

  • mindblowing