Mac, Tutorials
comments 46

Visuals for Sonar Festival + VDMX to Unity3D [Tutorials]

sonar-151

Numbers, a record label originating from Glasgow, had the opportunity to showcase artists at this year’s Sonar Festival in Barcelona. The artists Redinho, Spencer, Deadboy, Jackmaster, and Lory D each had a spot in the set. Adam Rodgers, co-founder of Numbers and Remote-Location approached Thomas Traum, Adam Finlay and myself to create live visuals for the show. We compared a number of technologies and frameworks– OpenFrameworks, Cinder, Unity3D, Cocos2d, WebGL, Processing, and good ol’ Flash. Doing something in 3D was a priority, and since we only had a 2-3 week timeline, the decision really came down to what would be the fastest to develop.

Despite my own lack of experience with the platform, Unity3D stood out as the clear choice. Unityscript, essentially their modified Javascript, ended up being quick to learn (particularly if you’ve done JS or Flash in the past.) Alternatively, Unity supports C# and Boo. It was only a matter of days before I was churning out decent prototypes for possible scenes. In addition, Tommi and both Adams could use Cinema4D as a quick way to contribute assets to the project. Unity’s native inclusion of 3D rigidbody support ended up saving us days of work. Rather than having to rely on an external library, out of the box 3D objects can simply have the attribute toggled on. Mass, friction, directional and angular velocity vectors– it’s all there. Many of the visuals were driven simply by sending the volume straight into the velocity values.

The app runs as a standalone Mac / PC executable file. Audio comes in via OSC from VDMX which handles spectrum bracketing, gain, and smoothing. Have a look at the tutorial if you’d like more info about this setup. We met only once to discuss scene ideas, the rest sort of spilled out of “it would be cool if…” comments. Some scenes, like the gravity balls, were pretty much done in a few hours. Others like the ragdoll falling through space, which had 9 separate asset modes, took over a week to get right. The scene with falling extruded text was created literally the day of the show in the hotel room. In the end, I probably wrote less lines of code than in a standard Flash microsite.

In the end there were a dozen scenes, each with at least 3 cameras, along with various modes to toggle. The keyboard was literally packed with toggles and options. All this in about 2 weeks of work, not bad at all considering the product that came out of it. One additional benefit of using Unity is that now we have the opportunity to easily port it to iPhone or iPad with very little difficulty.

Credits: NumbersRemote-LocationThomas TraumAdam Finlay and Mike Tucker

 

 

A Brief Tutorial on Creating a Sound Visualizer with Unity3D

Note: These directions are directed toward Mac users.

Part I. Preparing the Audio for OSC

We’re going to use OSC to convert music into variables that we’ll use as our input for the visualizer. 1) Download and install Soundflower: http://cycling74.com/products/soundflower/

This reroutes audio inputs and outputs. Perfect for what we’re doing.

2) For now we’ll use the computer’s output as our input.

  •  Go to Sound in your System Preferences
  •  In the Output tab, select Soundflower (2ch) (see picture)

3) Download and install VDMX demo: http://www.vidvox.net/

The demo won’t let you save files, but that won’t be necessary for our project.

4) Download the example project files from github: https://github.com/miketucker/UnityAudioVisualizerTutorial

Direct link to zip file: https://github.com/miketucker/UnityAudioVisualizerTutorial/zipball/master

5) Open the .vdmx5 file with VDMX. Only use the other vdmx file if you happen to be running an old version…

6) Try playing music from any app (Spotify, iTunes, etc) and you should see the wave forms in the Audio Analysis 1 window.

7) Go into preferences, and ensure there is a OSC Output port like the one pictured below.

 

Congrats, we’re done setting up the audio. There’s a lot to explore in VDMX– but this is enough to get some interesting visuals.

 

Part II. The App

1) If you don’t already have it, go download Unity Indie (it’s free): http://unity3d.com/

There’s a few things in Pro version that we used, such as Depth of Field, Bloom effects, etc.

But none if that is really necessary for what we’re doing in this project.

2) Open the same folder we used earlier  ( https://github.com/miketucker/UnityAudioVisualizerTutorial )

3) Open the file ExampleOscApp/Assets/Scene1.unity, this should start up Unity3D with the first scene opened

4) In the Hierarchy window you’ll see a list of names, these are the objects inside the scene.

Each object can have scripts attached to them. I’ll briefly run through each object and the scripts attached.

Camera – It’s from this view that we’ll see the running app.

GUILayer, Flare Layer, Audio Listener: These are by default added to every camera. Don’t worry about them right now.

Script – Rotate Around: I wrote this, It makes the camera spin around the centre, it also goes up and down with a sine wave.

Capsule, Cube, Cylinder, and Sphere – These are 3D objects in the scene. We’re going to have them scale to the music.

Script – ScaleObject: This is where we get to have fun. Only 2 lines of code to get it scaling to our OSC music!  The public variable oscChannel is a number 0-3 based on our original 4 bars from VDMX. 0 is bass, 1 and 2 are midtones, and 3 is treble.

Point light – just a light to give the 3d objects some depth. OSCStuff – this connects OSC from VDMX to our Unity app.

Notice how the IP and ports match the ones we set in VDMX.

The code was found on this forum: http://forum.unity3d.com/threads/16882-MIDI-or-OSC-for-Unity-Indie-users/page2

5) Try running the scene by pressing the Play button. The shapes should bounce to the music!

Part III. Onwards and Upwards

1) Once you wrap your head around how Unity deals with project structures it’s time to move onto Scene 2.

2) This scene is the exact scene from our Sonar Festival project. Very simple to make, with some beautiful outcomes.

3) There’s some new things going on here:

• Keyboard Manager: I used this class to map keyboard keys to different functions

• Multiple cameras: These can be switched in playback by KeyboardManager

• Duplicating objects: the Dupe object is duplicated using the Icosahedron vertices

• Physics: Each sphere has a rigidbody which means it will bump into other spheres

• Changing materials in playback: If you hit the J, K, L, and semicolon keys in playback, the spheres will change colors. This is triggered by KeyboardManager and can be found in the AttractionObject class

4) During playback, jump back to VDMX and play around with those values, it’s a great way to control the visuals.

5) Thanks and hope you enjoyed the introduction!

— Here are some more images from the Sonar Festival:

  • Eduard Prats Molner

    That’s a fantastic post Mike! Thanks for sharing this!

  • Martijn

    PC-version would be nice also for the tutorial!
    Furthermore, very cool stuff there

  • Mike Tucker

    hey Martijn (and PC owners), linekernal.net sent a VVVV patch that will send sound data to Unity, although I haven’t had a chance to try it yet: http://linekernel.net/www/files/insteadOfVdmx.v4p

  • MartinSkre

    This is very inspiring! Thank you so much. :)

  • MartinSkre

    Hm weird, the scenes don’t work for me at all in the Unity pro trial, all objects disappear when I press play. Just thought I’d let you know!

  • Dave Pentecost

    Nice work, good tutorial – and I’m glad to see that I may have contributed in some small way, by adapting other work for that OSC script in the Unity forums! If you come to New York, we’ll do this in our 30ft digital dome (a year from now when we’ve finished building it)!

  • Daniel

    just amazing guys.
    congrats for such a great idea!

  • nrlnd

    UnityEngine.Object:CompareBaseObjects(Object, Object)
    UnityEngine.Object:op_Inequality(Object, Object)
    Osc:Cancel() (at Assets/Plugins/Osc.cs:188)
    Osc:Finalize() (at Assets/Plugins/Osc.cs:176)

    Bohoo : ( using vdvmx beta 8 and unity 3.4

  • polyphant

    Same as user above! Thanks for taking the time, awesome tutorial!

  • Mike Tucker

    If you’re not seeing it visuals in Unity then there’s a problem with the VDMX bit..
    check steps 6,7, and 8 in the tutorial.

    and once you’re certain…
    open the OSCReceiver.js file and add this to ListenEvent function:
    Debug.Log(oscMessage.Values[0]);

    Upon running the app you should see number values showing up in the Console.

    Regarding the errors..
    they can come up anytime an OSC message isn’t sent. It even happens sometimes when the app is running correctly. On line 182 of Osc.cs I added this to stop the errors:   if(!OscPacketIO) return;

    lemme know!

  • Mike Tucker

    If you’re not seeing it visuals in Unity then there’s a problem with the VDMX bit..
    go back to Part 1 of the tutorial and check steps 6, 7, and 8.

    and once you’re certain…
    open the OSCReceiver.js file and add this to ListenEvent function:
    Debug.Log(oscMessage.Values[0]);

    Upon running the app you should see number values showing up in the Console.

    Regarding the errors..
    they can come up anytime an OSC message isn’t sent. It even happens sometimes when the app is running correctly. On line 182 of Osc.cs I added this to stop the errors:   if(!OscPacketIO) return;

    lemme know!

  • Mike Tucker

    Sorry to hear that, have a look at my response above.

  • nrlnd

    seams like unity 3.4 broke the script. what unity ver did you use 3.1?

  • nrlnd

    ThreadAbortExceptionSystem.Net.Sockets.SocketException: The descriptor is not a socket

  • nrlnd

    YAY finally …  started to work when i set the real ip of my mbp..  pff : D local was the part that was screwing me up 

  • Denberghvanmartijn

    i hace exact the same problem but dont understand what you are saying, can you help me out ?
    give a bit more detail on what you mean with ‘mbp’

  • LineKernel

    sorry , i just realized the link for the vvvv file is not working 
    this link should be working 

    http://linekernel.net/files/vvvv.unity.zip

  • LineKernel
  • Any

    Hey, thanks. In Unity the 3d-Objects are ‘Missing (Mono Script)’. Where to find that?

  • thevidness

    It’s interesting how that one thread on the Unity3D forums aggregated everybody who searched for OSC and Unity3D. so i was part of that bunch and i just wanted to share that i gotten around to create a rudimentary node for the visual scripting environment uScript inside Unity3D. this allows for a more convenient way of routing and handling the messages inside Unity3D.
    Have a look if you’re interested: http://uscript.net/forum/viewtopic.php?f=21&t=1089

  • Ian Jones

    Works for me! Thanks so much for sharing your project, it totally got me on the right track. I got Unity 3.4 (free not pro) receiving OSC messages from Quartz Composer under OSX 10.6

  • Ian Jones

    Quick Q: Did you use separate scenes and LoadLevel() between them? How did you handle scene transitions?

  • zencoder

    I wanted to see the app in action.  Found it on you tube and thought I’d share the link.

  • zencoder

    Found a better link to a video here:

  • Jóhannes Gunnar Þorsteinsson

    There seems to be some changes in the most recent version of VDMX which makes this tutorial really hard to follow. I did find this tutorial here http://itp.nyu.edu/~jms1024/myblog/soundflower-vdmx5-b8-unity3d/ although thanks to my lack of experience with VDMX and any other similar software I was unable to follow it. Could someone update this blogpost or point me in the right direction?

  • http://twitter.com/miketucker Mike Tucker

    Hey everyone, things were broken when VDMX released a new version.
    I’ve just updated the files and tutorial to work with Unity 3.5 and VDMX5. Sorry it took me so long. Please let me know if you run into further problems.

  • Jóhannes Gunnar Þorsteinsson

    Perfect timing! Thank you very much!

  • http://twitter.com/XY01 Brad Hammond

    Nice work, good to see more real time stuff out there and unity being used, it is a great platform.

    Also now you can now do all your audio analysis native in unity as of 3.5. 
    I am currently building an audio reactive visual synth called Splash, using Unity and have it all running smoothly inside of Unity. 
    https://vimeo.com/41237294 
    https://vimeo.com/41237294 

    It is currently in alpha, if anyone would like to get on board with testing the alpha, first release goes out in 3 weeks, you can sign up at. 
    http://www.facebook.com/XY01Design/app_190322544333196

  • Denzyl B

    Hi I am a PC user but think i am misunderstanding how this works.

    I have tried finding PC alternatives for soundflower and vdmx5 for PC but with no luck with a workable PC work around.

  • Jóhannes Gunnar Þorsteinsson

    Check out linekernal.net link. in Mike Tucker’s comment. Also, I have a faint memory of cross platform software called Jack that can be used to route audio between programs, inputs and outputs. I could be wrong though but I thought i’d mention it.

  • Denzyl B

    well see that link did nothing there is a working on later in the posts but it looks like all a get is JPEG of a picture of what i think i’m being instructed to do and a file type that looks like its meant to be opened with that VDMX but thats for mac only isn’t it?. but not sure what to do with it ?
    linekernel.net/files/vvvv.unity.zip
    ?

  • TimTim

    Hi Mike,

    this tuto seems old, but maybe you can help me, I have two errors :

    the descriptor is not a socket.

    CompareBaseObjects can only be called from the main thread.

    Can you explain me these errors?
    I already try to change port and IP. nothing change.

    I really need your work for a school project. If you can help me it will be very nice.

    Thanks a lot

  • http://twitter.com/phoenixperry phoenix perry

    Hey guys – it would honestly take you two seconds to properly export your unity project… hint, hint, hint….

  • http://benmorrow.info/ Ben Morrow

    Same here!

  • http://benmorrow.info/ Ben Morrow

    I get these errors

  • William Chak

    Hello Mike,

    Thanks so much for sharing! Brilliant tutorial!

    It was great meeting you at Resonate too.

    I’m having the same issue with VDMX. It seems to be not talking to Unity (ver 4).

    The problem might be related to recent updates of VDMX and Unity. Alternatively. you know you mentioned about the getSpectrumData technique, would you mind to kindly share this with the rest of us too?

    Cheers!
    Will

  • William Chak

    Hello Mike,

    Thanks so much for sharing! Brilliant tutorial!

    It was great meeting you at Resonate too.

    I’m having the same issue with VDMX. It seems to be not talking to Unity (ver 4).

    The problem might be related to recent updates of VDMX and Unity. Alternatively. you know you mentioned about the getSpectrumData technique, would you mind to kindly share this with the rest of us too?

    Cheers!
    Will

  • William Chak

    Hello Mike,

    Thanks so much for sharing! Brilliant tutorial!

    It was great meeting you at Resonate too.

    I’m having the same issue with VDMX. It seems to be not talking to Unity (ver 4).

    The problem might be related to recent updates of VDMX and Unity. Alternatively. you know you mentioned about the getSpectrumData technique, would you mind to kindly share this with the rest of us too?

    Cheers!
    Will

  • William Chak

    Hello Mike,

    Thanks so much for sharing! Brilliant tutorial!

    It was great meeting you at Resonate too.

    I’m having the same issue with VDMX. It seems to be not talking to Unity (ver 4).

    The problem might be related to recent updates of VDMX and Unity. Alternatively. you know you mentioned about the getSpectrumData technique, would you mind to kindly share this with the rest of us too?

    Cheers!
    Will

  • William Chak

    Hello Mike,

    Thanks so much for sharing! Brilliant tutorial!

    It was great meeting you at Resonate too.

    I’m having the same issue with VDMX. It seems to be not talking to Unity (ver 4).

    The problem might be related to recent updates of VDMX and Unity. Alternatively. you know you mentioned about the getSpectrumData technique, would you mind to kindly share this with the rest of us too?

    Cheers!
    Will

  • William Chak

    Hello Mike,

    Thanks so much for sharing! Brilliant tutorial!

    It was great meeting you at Resonate too.

    I’m having the same issue with VDMX. It seems to be not talking to Unity (ver 4).

    The problem might be related to recent updates of VDMX and Unity. Alternatively. you know you mentioned about the getSpectrumData technique, would you mind to kindly share this with the rest of us too?

    Cheers!
    Will

  • William Chak

    Hello Mike,

    Thanks so much for sharing! Brilliant tutorial!

    It was great meeting you at Resonate too.

    I’m having the same issue with VDMX. It seems to be not talking to Unity (ver 4).

    The problem might be related to recent updates of VDMX and Unity. Alternatively. you know you mentioned about the getSpectrumData technique, would you mind to kindly share this with the rest of us too?

    Cheers!
    Will

  • William Chak

    Hello Mike,

    Thanks so much for sharing! Brilliant tutorial!

    It was great meeting you at Resonate too.

    I’m having the same issue with VDMX. It seems to be not talking to Unity (ver 4).

    The problem might be related to recent updates of VDMX and Unity. Alternatively. you know you mentioned about the getSpectrumData technique, would you mind to kindly share this with the rest of us too?

    Cheers!
    Will

  • Aldo Manetti

    Many Thanks!! i`m downloading, and testing at night.
    Just what i`m looking for and artistic project that i`m doing… sound improvisation, realtime design/art and human interaction.

    thanks again….i hope it will work for me hehe :)

  • http://martinlynge.dk Martin Lynge Jensen

    Worked for me with Unity 4.5.2 and VDMX5. It did initially not. Got the same warning and error as pointed out in many of the comments and scale (0,0,0) on all objects, but then noticed that the port in VDMX OSC preference was set to 1234 and not 12345. Thanks for the tutorial.

  • PlanetYJ

    Great intro to your vdmx/unity approach thanks! Can anyone confirm if they got all the cameras working in playback via the key press? I only seem to have the keys for changing the objects colors.. Right now its stuck on main camera.