Kyle McDonald has been exploring possibilities of face tracking using FaceShift Studio, an affordable markerless facial performance capture in development since 2009 and presented presented over the years at the SIGGRAPH conference. The tutorial below show you how to connect it to other apps in realtime via OSC with FaceShiftOSC, and connect to openFrameworks via his ofxFaceShift addon.
Kyle’s tutorial begins with the instruction how to use your Kinect to generate a nurb model of your face. Once you have gone through the process, FaceMesh is able to translate your face movements directly onto the three dimensional model in realtime. The model is so precise that besides capturing your facial expressions extremely well it is also capturing your eye movement.
The next step is to capture the movements though the OSC server and reflect the same in the openFrameworks app Kyle has written. All the data is incoming, even the eye movement. Finally, the OSC data is fed into Ableton, and you are able to generate sounds using your facial movement.
Kyle has made all the code and examples available online so you can have a play.
FaceShift intro faceshift.com/
FaceShiftOSC setup github.com/kylemcdonald/ofxFaceShift/downloads
ofxFaceShift overview github.com/kylemcdonald/ofxFaceShift
Sound design from soundcloud.com/robclouth/
OSCulator osculator.net/
