SketchSynth – Drawable user interface by Billy Keyes

SketchSynth by Billy Keyes is a drawable OSC control panel which lets anyone create their own control panels with just a marker and a piece of paper. Once drawn, the controller sends Open Sound Control (OSC) messages to anything that can receive them; in this case, a synthesizer running in Pure Data.

SketchSynth is built using openFrameworks and makes heavy use of Kyle McDonald’s ofxCv addon. A standard webcam looks down at the paper and detects both controls and hands. To prevent confusion, there are two modes: EDIT and PLAY. The system does nothing in EDIT mode, allowing you do draw controls. When you switch to PLAY mode, the controls are detected. Controls are detected by finding contours (blobs) in an edge-detected version of the image. The camera and projector are aligned by clicking on the four corners of a projected rectangle (as seen by the camera) when the program starts up.