Visual programming languages, languages that create programs by the manipulation of graphical elements, as opposed to specifying lines of text, have seen an increased popularity in recent years both in audio and video synthesis. Some of the more well-known environments, ones that are regularly used for projects that are featured on CAN, include VVVV (real-time motion graphics and physical IO) MAX/MSP (real-time music and multimedia), Pure Data (ostensibly an open source equivalent of MAX/MSP) and Quartz Composer (video synthesis for MAC).
Visual programming owes its many of its conventions for the representation of information and programs from Flowcharts – a lesser used term for these kinds of environments is Data-flow Programming. VPL’s date back to the late 60’s. A good example is the GraIL system (GRaphical Input Language) a flowchart language entered on a graphics tablet developed by the Rand Corporation in 1969.
A program (Patch) typically consists of a set of objects (Nodes) that are connected to each other via their inputs and outputs (Pins). In this case I’ve used the specific terminology of the VVVV environment, but the essentials are the same for other VPLs.
The constraints of patch-based visual programming can give rise to some interesting aesthetic configurations. Various factors such as the legitimate connections between nodes, types of information flow paths, the arrangement of input and output devices are some of the constraints that drive the schematic of the patch and the spatialisation of the elements within it. In one sense these kinds of environments could be seen as self-organising systems – where functionality drives local behaviours and interrelationships of nodes and connections.
While ruminating over the aesthetics of patches I wondered what were the equivalents (in textual programming) of best practices and programming design patterns and if there were any models? If there were such models being implemented on a wider basis we would see similarities in the schematics in different programmers patches. More so an agreed system of placement of nodes and connections would render patches universally readable by the developer community as a whole.
In textual programming there is a long history of best practices, optimisation techniques and standardisation – ways to make the code modular, reusable and manageable over large teams of programmers. Languages such as C++, Java and Actionscript 3 have settled on the Object Orientated Model. Programming Patterns are the paragon of optimisation – ‘ general and reusable solutions to a commonly occurring problems that ‘typically show relationships and interactions between classes or objects, without specifying the final application classes or objects that are involved’.
PerfMeter – Kalle Karlen (shows strict top-to-bottom data-flow with horizontal groupings of nodes with similar functions)
I talked to some of the VVVV community, as well as some other well-know visual programmers to gain some insight into their preferences for optimisation and the importance of ‘visual programming design patterns’ if indeed such a thing existed. I’ve also outlined some of my own personal methods for optimising the layout of patches.
Well respected user Kalle Karlen of the VVVV community, who has a VVVV node named after himself, generously sent me screen grabs of many of his patches. Some of his key points were:
– The dataflow in the patch should be arranged vertically.
– Comments are absolutely necessary if you patch in collaboration with others.
– Showing only important pins helps keeping track of data flow. [Pins can be set to zero visibility in VVVV]
– A preference not to use curved and VHV (Vertical-Horizontal-Vertical) connectors as design elements until a patch is very close to final perfection – and then only use curved connectors to indicate data feedback loops.
An interesting additional to Kalle’s working method is his his use of the UI to create a block diagram of empty sub patches that act purely as a visual aid in the early planning stage of a project.
Joreg, one of the developers behind VVVV, and who was responsible for developing the GUI as part of his diploma thesis also mentioned the importance of using top to bottom data flows. Since visual programming is so much more about data flow than text-based programming languages a clearly discernible top-to-bottom structure for the direction data-flow should be applied. It makes logical sense to have inputs (such as communication with external devices, keyboards and XML) at the top of a patch and outputs (such as a renderer and sound output) at the bottom. In general nodes of a similar function should be aligned on the Y axis. It further makes sense to group nodes that work together in a tight configuration indicating their interrelationships. These groupings can also be considered as possible candidates for subpatches.
Nearly everyone I talked said that they would first get the algorithm patched and then, if time permitted, do some re-arrangements to tidy the patch up into a readable format. Eventually any complex patch will outgrow the size of the screen without scrolling the window. In this circumstance it is probably a good time to start thinking about sub-patching especially if a part of the patch can be made into a module and reused in other projects. Joreg mentioned that in VVVV one of the most wanted features is the ability to ‘make selected nodes into a subpatch’ which would increase the efficiency of modularizing a patch.
When a system becomes very complex its often good to implement the finite state machine model to describe the systems state(s) at a particular instant. David Brüll a lead VVVV software designer who usually works with different teams of patchers, programmers and designers on professional projects recommended the use a state designer such as QFSM when planning a patch. He also recommended that all logic related to states of the system to be place in a top level sub-patch. In this sense the structure of a patch might already seem to mimic a design pattern seen in OOP languages (such as MVC) where different aspects of a program are separated and isolated from one another.
Not everybody works in a rectilinear fashion, and VPL’s allow much more liberty in regards to personal style than textual programming. Woeishi Lean works with a dense and extreme clustered style which he says enables him to keep the data flow as exposed as possible. He prefers not to create deep hierarchies of sub-patches as he feel this can cause him to lose the overview of the data flow.
‘The main data starts at the upper left and the flow moves down vertically, additional data comes in from the right side. My nodes are often edge on edge and grouped to logical functions so it’s easier to grasp all the node names from top to bottom”
In considering the configuration of patches I wondered if there was any examples of isomorphic similarities between spatialisation of elements in a patch and the functionality of patch. One contender for this is the simulation of a recursive function by creating a video feedback loop. Clearly in the above patch you can see a cluster of nodes that generate an ‘input’, a circuit of renderers which represent the recursive loop (as the ‘output’ of the renderer is fed-back to another renderer, and so on) and then a final output, as the branch of video-data flow is directed out of the loop to another renderer.
Moving away from VVVV to PureData we arrive at Chun Lee’s lesson in austerity. ‘0xA – expr~’ is musical release, the entirety of which was generated nearly exclusively with one object in PureData, the [expr~] object. The tracks vary from saw-tooth drone to noise systems music all with an 8-bit chip-tune feel.
Regarding optimisation Chun had this to say ‘I tend not to think of patterns – Pd was never really designed with scalability in mind, so ultimately any effort in trying to enforce some kind of pattern so that codes can be easily reused/organised, would seem a bit like a work around (or too much effort for very little gain)’,
All thought not written in a VPL environment Dave Griffiths genetic programming synthesizer, FastBreeder, works backwards creating a visual representation of the users ‘grown’ code structure. The sound generated is therefore related to the structures of the nodes which are automatically generated using an algorithm implementing a form of a nested two-pi layout.
Paul Prudence is a audio-visual performer and installation artist working with computational and generative environments. His work, which had been shown internationally, focuses on the ways in which sound, space and form can be combined to create live-cinematic visual-music experiences. Paul maintains the research weblog Dataisnature in which he writes about the interrelationships between natural processes, computational systems and procedural-based art practices