For the Grammy awards 2010, musician & artist Imogen Heap wanted a dress that would display the tweets and photos coming from her fans in realtime, so she could take her fans with her onto the red carpet. Moritz Waldemeyer designed a flexible LED ribbon which she could wear, and I developed the controlling software, an iPod Touch application that she could carry in her bag that would collect all the tweets and photos from the net, and send the information to the custom LED ribbon.
I designed and programmed visuals (projections) for the OKGo performance and Fendi installation at Design Miami 2009. OKGo were playing modified Les Paul guitars mounted with laser beams in the headstock, designed by Fendi in collaboration with Moritz Waldemeyer. Using a PC equipped with a high-speed firewire camera, I developed custom software to track the laser beams and generate visuals around the spots they hit the wall. These visuals were also audio-reactive, responding to the live audio feed coming from the sounddesk.
More images & video coverage coming soon.
P.S. For additional laser tracking goodness, checkout the Graffiti Research Lab's Laser Tag.
MSA Remote is a remote control application for iPhone & iPod Touch that sends OSC messages over the wifi network. This allows you to control any OSC supporting applications such as Max/MSP/Jitter, PureData, Reaktor, VDMX, vvvv, Resolume, Quartz Composer etc. as well as any custom applications such as interactive installations, audio/visual performances, multitouch capable surfaces etc.
By mapping the OSC to midi on desktop (e.g. using OSCulator) allows further control of any application which supports midi such as Ableton Live, Cubase, Logic Pro, 3DSMax etc. In addition, developers can easily integrate OSC into their applications knowing it can be controlled remotely.
The application can be distributed to visitors, guests, members of the public etc. to interact with an installation or performance, or used by dedicated performers.
- Multitouch information sent using standard TUIO protocol for instant integration with existing TUIO clients
- Accelerometer data for each axis (x, y, z) is sent
- 64 faders (8 pages of 8 faders)
- 64 triggers (8 pages of 8 triggers)
- 108 key (9 octaves) VELOCITY SENSITIVE polyphonic keyboard. Yes, the harder you hit the keys, the greater the velocity.
- Settings are automatically saved and restored
- Multitouch area orientation can be set as desired
- All information on protocols are documented in the app
Controlling VDMX & Ableton Live simultaneously
In this video, OSCulator is routing the OSC (& TUIO) messages coming from MSA Remote to midi and forwarding to Ableton Live (for audio) and VDMX (for visuals) simultaneously. Nothing is done in post, the same signal is controlling both audio and video. Video demonstrates faders, triggers, and velocity sensitive keys - yes velocity sensitive. The harder you hit the keys, the louder the sound (or whatever you want to map it to).
Controlling Java/processing.org fluid simulation with TUIO
Source code for the processing.org sketch can be found here:
Controlling custom multitouch paint application with TUIO
Graffiti Wall is a product that Tangible Interaction had already developed a while ago and have been using commercially. It was built on TUIO from the beginning, so when I passed on a copy of MSA Remote (via ad-hoc), they were able to immediately integrate it with their app without any modifications.
Camille Troillard, the mastermind behind OSCulator was kind enough to add some features (and sample patches) to his app to make handling the data coming from MSA Remote and converting to midi as seamless as possible.
The /msaremote/fader protocol is recognized and faders are automatically setup to forward to midiCC with same number as the slider.
The /msaremote/trigger protocol is recognized and triggers are automatically setup to forward to midi notes with constant velocity.
The /msaremote/bank protocol is recognized and triggers are automatically setup to forward to midi notes with constant velocity.
In the Osculator/Sample Patches/iPhone folder there are two sample patches which map incoming msaremote keyboard OSC messages to midi notes with velocity. The basic patch maps each key to the relevant midi note, while the advanced patch has the keys setup so you can assign each key to a different event.
Max/MSP
Blair Neal aka vimeo.com/laserpilot has created a Max MSP template (for Max5) to handle the OSC Messages coming from MSA Remote. Cheers Blair!
This is a library for solving real-time fluid dynamics simulations based on Navier-Stokes equations and Jos Stam's paper on Real-Time Fluid Dynamics for Games. While I wrote the library primarily for processing it has no dependency on processing libraries and the source can be used with any Java application.
The video below is a demo of a processing sketch using MSAFluid, being controlled by MSA Remote on iPhone (You can view the video in HD and download a 1080p version at vimeo).
ofxMSAPhysics is a C++ 3D particle/constraint based physics library for openFrameworks. It uses a very similar api to the traer.physics library for processing to make getting into it as easy as possible.
Version 2.0a is now available for testing.
Main features include
particles
springs
attractions (+ve or -ve)
collision
replay saving and load from disk (temporarily disabled in current alpha release)
custom particles (extend ofxMSAParticle and add to the system)
custom constraints (extend ofxMSAConstraint and add to the system)
custom force fields (extend ofxMSAParticleUpdater and add to the system)
custom drawing (extend ofxMSAParticleDrawer and add to the system)
An interactive, generative, audio-reactive visualization app for the iPhone and iPod Touch (what a mouthful!) (This is the continuation of what I originally called "The Meshulator").
Use your fingers to create and interact with abstract geometry in 2D or 3D. Very simple to use, just slide your fingers over the screen and try different gestures. Tilt the device to look at the shape from different angles. Options for two different kinds of audio-reactive behaviour (microphone input - iPhone only): deforming based on audio, or completely generative driven by audio. If you feel adventurous, you can play with plenty of customizable parameters to adjust coloring, physics, shape etc. Or if that feels too intimidating just click 'randomize' to make the settings random and experiment.
It was written with a custom version of openFrameworks (pre-006) and an early version of the ofxiPhone addon. Information on openFrameworks and ofxiPhone at memo.tv/ofxiphone.
"Reincarnation" is an off-shoot from a visual performance for the Rambert Dance Company's "Iatrogenesis" at the Queen Elizabeth Hall, South Bank, London UK - visuals for the latter directed by my good friends at flat-e.
The project began when I started working with footage of the Rambert dancers. Inspired by the footage, I wrote custom software to track the motion and generate the visuals you see in the video below... and "Reincarnation" was born.
For the Rambert's "Iatrogenesis" piece, I created similar visuals, abstract visual layers containing subtle hints of human forms and motion, which would tie in with the movement of the dancers on stage. These were used by flat-e and composited with various other layers and projected on a see-through gauze in front of the stage.
The video (and music) you see below is not representative of the visuals (and music) of the Rambert's and flat-e's "Iatrogenesis". This is just a standalone piece born from working with the Rambert choreography.
When the clip starts, you probably won't recognize a human shape at first, but your eyes and mind will be searching, seeking mental connections between abstract shapes and recognizable patterns, like looking for shapes in clouds. You'll be questioning what you see, is that it? is it sitting? is it crouching? is it kneeling? Then all of a sudden, it'll be crystal clear. Then you'll try and keep it in focus, following it as it moves around, tracking each limb, using the motion to construct an image of the parts you can't see. It'll fade in and out of clarity. At times you'll be clinging onto just the tip of it's hand swinging round, trying to identify any other recognizable parts. You might see another arm or leg and grab onto it, fighting not to lose it. Then it'll be crystal clear again, and then all of a sudden vanish, literally in a puff of smoke, and your eyes will start searching again.
Working with Seeper, we created a vision driven interactive stand for Brandwidth and Toyota IQ. The stand is touring shopping centers in the UK and currently at the Westfield Shopping center in London.
I'm delighted to announce that my "Gold" installation has been selected to be shown at the Tent London exhibition as part of the London Design Festival. 24-27th September 2009, at the Truman Brewery, London, UK. Stay tuned for more information.
The video below is an early demo of the installation.
“Gold” is an interactive installation which explores our obsession with super-stardom, and the extravagance that accompanies it. Through a ‘magic mirror’, revel in a world of excess where you are the super-star. Shower in glittery gold, experience almost omnipotent powers as you materialize, morph and dematerialize into pure sparkling gold dust. Immortalize yourself as a shimmering golden statue, before you collapse and fade away.
The installation uses custom software written with openFrameworks and the OpenCV computer vision library. The software analyzes the video feed from infra-red cameras in real-time and generates 1080p HD output using OpenGL.
Working with Arnold Agency and Todd Vanderlin+Ryan Habbyshaw from their R&D team, we created an interactive display for Citizens bank in branches in major cities in the US.
Using motion tracking, the pedestrians can interact with the display, signaling birds to come flying in and drop coins, grow plants, create wind to blow the plants around and spray pollen etc. The display is also time-reactive, automatically theming itself depending on time of day.
Added loop mode configurable from the preferences.plist
-----------------
MSAQTPlayer is a Mega Super Awesome (& mega super basic), fullscreen / multiple output Quicktime player with fast, greater-than-4096-pixels support.
I wrote this app for an installation because I needed to play a 4,800 x 600 Quicktime file across 2 outputs on my graphics card, each feeding 3 triple heads to go into 6 projectors, and I couldn't find a single application that could do it.
Most video applications which can output across multiple video-outs cannot play files larger than the maximum size of a texture (usually 4096 pixels). And even if the movie resolution fits in a texture, they are very slow because they convert each frame to a texture and upload to the graphics card. The only application I found which could play a large file is Quicktime Pro, but that cannot output across multiple video-outs.
MSA QTPlayer is a native Cocoa app which can do this (using QTKit and OSX 10.5) .
My Secret Heart is a music and film installation & performance commissioned by Streetwise Opera with music composed by Mira Calix and sound design by David Sheppard. Working with video artists Flat-e, we created a film to accompany the 48 minute performance, as well as versions for an installation and short film.
Streetwise Opera
Streetwise Opera are a charity who use music as a tool to help people who have experienced homelessness move forward in their lives. They run a weekly music programme, resident in 10 homeless centres around the country - and also stage an annual production which gives their performers the chance to star in quality shows where there are high-expectations, no compromise and no patronising. The voices you hear in the music, and people you see in the film, are from Streetwise workshops around the UK. 100+ Streetwise performers also sang at the My Secret Heart premiere at the Royal Festival Hall in December 2008. My Secret Heart is about their story.
The film
The film has an abstract narrative derived from individual conversations with each of the Streetwise performers. It is a direct emotional response to their stories combined with the haunting beauty of Mira Calix's composition. Instead of focusing on a specific plot, the film embarks on a complex journey through various states of emotion, starting from pre-birth through birth, curiosity, exploration, excitement, playfulness; through to fear, anxiety and isolation. While it maintains a relatively dark and eerie mood overall, intertwined with the feelings of desperation are strong elements of hope.
The visuals were designed and created primarily with custom software written in C++/openFrameworks, with some Quartz Composer elements, rendered AfterEffects sequences and live action footage. The custom C++ app is audio-reactive and user-interactive, allowing the visuals to be 'performed' live with full control over the behaviour of the virtual inhabitants of the cylindrical aquarium-like rig.
Over the course of a few months, and after many conversations with Mira Calix and listening to the soundtrack over and over and over again, we decided roughly what the visuals should do and what kind of behaviours we wanted the visuals to perform at specific points in the song. After a lengthy coding period, I had an application that when you ran, did... nothing, but it had the potential to do everything I wanted. The application was a live performance tool with full control over its environment as well as audio playback and control, and an input recording / playback system.
Once the application was complete, I sat down with Robin from flat-e, and pressed 'play' on the app - this started the music playback and the physics recorder. While the music was playing we could control the inhabitants of the virtual world with many sliders, knobs, touchpads, mouse etc. As the music was playing we would respond in realtime by sending messages to make them move gracefully, erratically, flocking together, swimming apart, getting excited, slowing down, speeding up, telling them to die, slowly start twitching, come alive, swim to the surface, sink to the bottom etc - our actions being recorded gave us the ability to later go back and scrub to certain positions in the song and overdub and mix new behaviours we might have missed in the first round. In the end we found that actually we had to do little to no editing. The best overall performance was the one we recorded in a single 50 minute take.
The sensation of performing and recording the visuals was that of actually directing a film with thousands of virtual actors, commanding an army, digital puppetry - an approach I'm sure I will be revisiting in the very near future.
Early tests of visuals on the 'aquarium' (rig built by Gaianova):
"Roots" is an interactive musical/visual installation for the Brick Table tangible and multi-touch interface, where multiple people can collaborate in making generative music in a dynamic & visually responsive environment. It is a collaborative effort between myself and the Brick Table creators Jordan Hochenbaum & Owen Vallis. It will premiere at the Minitek Music + Innovation Festival September 12-14, 2008 in New York.
The essence of the interaction, is that you control parameters of a chaotic environment - which affect the behaviour of its inhabitants - which create and control music.
To breakdown very briefly without going into much detail:
There are vinelike structures branching and wandering around on the table. They live and move in an environment governed by chaos.
Audio is triggered and controlled entirely by how and where the branches move.
You - the user - control various parameters of the chaotic environment. Parameters which range from introducing varying amounts of order, to simply changing certain properties to let the chaos evolve in different directions.
There are varying levels of interaction, ranging from traditional one-to-one correlations - 'this movement I make creates that sound', but also to more complex relationships along the lines of 'this movement I make affects the environment in this way which sends the music into that direction where it evolves with a life of its own'. The visuals are purely generative, as is the audio, and as user you can play with the parameters of that system and watch and listen to the results...
Demo of drawing with roots:
Demo of using fiducials to create magnetic force fields:
This is an 'early current state of app' demo for a multi-discipline event I'm working on with Streetwise Opera, Mira Calix and fellow visualists Flat-e, to be showcased at the Royal Festival Hall later this year with quite a few more venues lined up.
The app was written in Processing 0135 and is running realtime at 60fps, though if I add another couple hundred eels it does drop, so I may switch to OpenFrameworks if performance does become an issue (which it probably will). There are occasional freezes in the video which happened while capturing the screen so that is a bit annoying.
I'm controlling the eels using the mouse, keyboard and Quartz Composer (just simple sliders sending OSC to vary some parameters - similar to the 'magnetic force fields' video - I'm quite into this technique now, very quick and easy to setup, and you can have loads of sliders with descriptive names at your disposal to play with, and adjust your internal variables in realtime for tweaking heaven).
The final show will have many many more features, both in the digital realm, and physical... more info coming soon...
I was sitting minding my own business, exploring accumulation buffers in Quartz Composer, when all of a sudden I just zoned out, and next thing I know, I found myself staring at Him on my screen. His Noodly Appendages came down and touched me, and guided my hands, connecting Quartz Composer's very own noodles in His Image.
His Noodly Screen Saver runs on Mac OSX 10.5 (Leopard) and you can download it below.
"Pi" is an interactive audio/visual installation commissioned by Trash City of the Glastonbury Festival to be shown at the festival in June 2008.
Working with arts and technology collective Seeper, our concept was to take a 50ft tent, and convert it into a giant audio/visual instrument - all of the music, audio and visuals inside the tent are generated and controlled purely by the movements of the occupants.
The space was divided into 6 zones. Two of the zones were purely visual, this was the waiting area. Here people could dance, chill, run about and do what they pleased. Two cameras tracked their movement and applied it to the fluid/particles visuals - so people could 'throw' plasma balls at each other, or send colorful waves propagating around the space. The other 4 zones had the same visual interactions, but in addition were also connected to an audio system. Each of these four zones was allocated an instrument type (drums/beats/percussion, pads, bass, strings etc.), and movement within these zones would also trigger notes or beats - depending on precisely where in the zone the movement was triggered. A lot of effort went into designing the sounds and notes triggered to make sure the end result would almost always sound pleasant and not be complete cacophony.
The first psychedelic fluid/particles interaction prototype developed in processing.org:
Camera -> osc/midi interaction tests (developed in Quartz Composer):
The two concepts strung together and written in C++ with openFrameworks:
A test in motion detection in Quartz Composer 3.0.
The music is all generated in real-time by me waving my fingers, hands and arms around (or in fact any motion) in front of a standard web-cam. No post-processing was done on the audio or the video.
The concept is by no means new, but still fun nevertheless - and I'm quite happy with this implementation. I'm using a very simple frame difference technique and generating midi notes based on where-ever there is movement (actually, as QC3 cannot send midi notes I had to send the data as OSC and use OSCulator to forward them as midi).
I was at the Adobe Air Tour 2008 in London today, and while most of it was very interesting, there were moments which were rather dry... so in my boredom I created this little Quartz Composition.
Last fall I was at Flash on the Beach in Brighton, and Craig Swann was demonstrating some amazing things he'd been doing in Flash and Max MSP/Jitter. I remembered one of them today and decided to try and create something similar in Quartz Composer, the results are quite fun.
Basically it captures a very narrow slice of video input (in this case my webcam) every frame and creates a picture out of lots of slices from different times, essentially capturing a segment of time in one picture. The composition here can do it horizontally or vertically. You could try radial and other methods too.
Leopard only I'm afraid...
** Update **
Barth informs me that this is called slit-scanning, and lots of people have done amazing things with this technique! More info at Golan Levin's Slit-scanning page.
This is a little test using GLSL in Quartz Composer 3.0, and controlling via VDMX. All happening in realtime and completely audio-reactive with no post production or timeline animations etc. The potential is humongous and very exciting!!
Soundtrack "Caliper Remote" by Autechre (from LP5 - 1998)
Who needs autechre when you have a bunch of mad girls!
P.S. I have hours of footage of this if anyone is interested :P
Another example of an AS3 3D Sprite based particle system. This one is using my 3D sprite engine but will port this to papervision asap. See it here.
The source code is by no means generic enough to be considered an engine or library, but is work in progress. I will extend the Node3D, SpringyNode3D classes to use Papervision3D and the Billboards for the rendering, and will definitely carry on working on the particle systems. I've also been converting a lot of noise functions to AS3 and will post them soon... its all getting quite exciting in the flash world! :P
A 3D particles system using Action Script 3. See it here. This does not use Papervision 3D or any other 3D engine but just a simple custom 3D sprite based engine. Not sure whether I should spend time enhancing this or just use Papervision for this kind of stuff. I've used it a bit and it looks great (will post some stuff soon), but I don't know if there is any unnecessary overheads for just 3D sprite stuff. When I get a chance I"ll recreate this demo in Papervision and compare the results.
Onset assistant director and visual effects supervisor, also responsible for post production on 60second commercial for clothing brand Riva. 3-day shoot in studio (mainly greenscreen shoot) in south France. Post production work involved keying, tracking, virtual set building, stabilizing, compositing, particles, cloth simulations and motion graphics.
