More information at www.msavisuals.com/webcam_piano_2

A while ago I'd posted an example and source code for using Vertex Arrays, Vertex Buffer Objects and Point Sprites in openFrameworks. This was for openFrameworks 005 and needed some mods to the core and other hacks to get it to do what we needed. In the current version of openframeworks (006+) a lot of the required functionality has been moved to the core and so we don't need the extra classes MSAImage and MSATexture, or to hack the core. The updated example is attached and can be downloaded from below.

P.S. An example on particle system with OpenCL for even more performance (updating the particles on the GPU) can be found here.

I've recently needed to work with Midi Time Code (MTC) and could not find any code to parse the midi messages and construct an SMPTE timecode. Closest I got was finding this documentation (which is pretty good) on how the data is encoded in the bits of 8 bytes sent over 2 SMPTE frames, each byte sent at quarter frame intervals. From that I wrote the code below (I've only really tested the 25 fps). The code is from an openframeworks application but should work with any C/C++ code.

P.S. Some info on bits, bytes and nibbles here.

class ofxMidiEventArgs: public ofEventArgs{
public:
    int     port;
    int     channel;
    int     status;
    int     byteOne;
    int     byteTwo;
    double  timestamp;
};
 
#define kMTCFrames      0
#define kMTCSeconds     1
#define kMTCMinutes     2
#define kMTCHours       3
 
// callback for when a midi message is received
void newMidiMessage(ofxMidiEventArgs& eventArgs){
 
    if(eventArgs.status == 240) {                       // if this is a MTC message...
        // these static variables could be globals, or class properties etc.
        static int times[4]     = {0, 0, 0, 0};                 // this static buffer will hold our 4 time componens (frames, seconds, minutes, hours)
        static char *szType     = "";                           // SMPTE type as string (24fps, 25fps, 30fps drop-frame, 30fps)
        static int numFrames    = 100;                          // number of frames per second (start off with arbitrary high number until we receive it)
 
        int messageIndex        = eventArgs.byteOne >> 4;       // the high nibble: which quarter message is this (0...7).
        int value               = eventArgs.byteOne & 0x0F;     // the low nibble: value
        int timeIndex           = messageIndex>>1;              // which time component (frames, seconds, minutes or hours) is this
        bool bNewFrame          = messageIndex % 4 == 0;
 
 
        // the time encoded in the MTC is 1 frame behind by the time we have received a new frame, so adjust accordingly
        if(bNewFrame) {
            times[kMTCFrames]++;
            if(times[kMTCFrames] >= numFrames) {
                times[kMTCFrames] %= numFrames;
                times[kMTCSeconds]++;
                if(times[kMTCSeconds] >= 60) {
                    times[kMTCSeconds] %= 60;
                    times[kMTCMinutes]++;
                    if(times[kMTCMinutes] >= 60) {
                        times[kMTCMinutes] %= 60;
                        times[kMTCHours]++;
                    }
                }
            }           
            printf("%i:%i:%i:%i | %s\n", times[3], times[2], times[1], times[0], szType);
        }           
 
 
        if(messageIndex % 2 == 0) {                             // if this is lower nibble of time component
            times[timeIndex]    = value;
        } else {                                                // ... or higher nibble
            times[timeIndex]    |=  value<<4;
        }
 
 
        if(messageIndex == 7) {
            times[kMTCHours] &= 0x1F;                               // only use lower 5 bits for hours (higher bits indicate SMPTE type)
            int smpteType = value >> 1;
            switch(smpteType) {
                case 0: numFrames = 24; szType = "24 fps"; break;
                case 1: numFrames = 25; szType = "25 fps"; break;
                case 2: numFrames = 30; szType = "30 fps (drop-frame)"; break;
                case 3: numFrames = 30; szType = "30 fps"; break;
                default: numFrames = 100; szType = " **** unknown SMPTE type ****";
            }
        }
    }
}

Recently I've been playing a lot with OpenCL, the new API / framework designed to handle cross-platform parallel computing (i.e. a simple way of running code simultaneously on all cores of your CPU, GPU or other processors). Implementations have been cropping up this year in NVidia drivers or ATI drivers, but most famously it's included with Mac OSX 10.6 Snow Leopard.

To cut a long story short I've been working on a simple-to-use C++ wrapper for some of the most common functions, imaginatively called ofxOpenCL and here is a little demo of 1 million particles running at 100-200fps.

NOTE: The Vimeo compression destroys most of the particles, so I suggest downloading the quicktime directly from the vimeo page at http://www.vimeo.com/7332496

Frame-rate is around 100-200fps running on a macbook pro with GF 9600GT. That's 100-200fps on a laptop! (albeit a pretty decent one), but I'm dying to try this on a GF 285 GTX - which has 7.5x the number of cores, 2.5x the fillrate and 3.5x the memory bandwidth - for only £250!!

The kernel for this is surprisingly simple:

__kernel void updateParticleWithoutCollision(__global Particle* pIn, __global float2* pOut, const float2 mousePos, const float2 dimensions){
	int id = get_global_id(0);
	__global Particle *p = &pIn[id];
 
	float2 diff = mousePos - pOut[id];
	float invDistSQ = 1.0f / dot(diff, diff);
	diff *= 300.0f * invDistSQ;
 
	p->vel += (dimensions*0.5 - pOut[id]) * CENTER_FORCE2 - diff* p->mass;
	pOut[id] += p->vel;
	p->vel *= DAMP2;
 
	float speed2 = dot(p->vel, p->vel);
	if(speed2<MIN_SPEED2) pOut[id] = mousePos + diff * (1 + p->mass);
}

This example is based on Rui's opencl example at http://vimeo.com/7298380.

Discussion on the matter at http://www.openframeworks.cc/forum/viewtopic.php?f=10&t=2728&p=15107#p15...

source code for ofxOpenCL and the above example at
http://code.google.com/p/ofxmsaof/downloads/list
(the SVN is likely to be more recent).

The quickest game I've ever developed on any platform.

Next steps, to do a 9 page panoramic ;)

P.S. Do not expect to see this in the App store anytime soon!

Rejection

So my MSA Remote app has been rejected AGAIN, for the second time! This time because my startup image "infringes an Apple Trademark Image". Aaargh!

I had paid special attention to not include the sliver outline and home button in case it would be an issue, but it wasn't good enough. Apparently Apple own rectangles with rounded corners at a specific radius.

But like they say, Never give up, never surrender. I've resubmitted with a much tighter crop (I know, a lot uglier :S but it'll have to do for now), if this doesn't get approved, I think I'll just give it all up and retire to a little fishing village (maybe I should do that anyway).

P.S. You can see the previous post about the first rejection here.

Demo, Velocity Sensitive Keys and Patents

In the meantime I've uploaded a new video, this time demoing controlling Ableton Live (for audio) and VDMX (for visuals) and using faders, triggers, and velocity sensitive keys - yes velocity sensitive. The harder you hit the keys, the louder the sound (or whatever you want to map it to).

Coincidentally, in a tweet from cdmblogs I saw these guys have developed a very similar "patent pending" technique. They're very young, all in their early twenty's. So respect guys for all you've developed and the whole business venture etc. but a tip, please don't try and patent things like this. It's bad for mankind.

And for those wondering how it works, it's not rocket science, it's accelerometer. The harder you hit the screen, the bigger jolt on the accelerometer. And yes it does work on hard surfaces because you still get an internal jolt. You just need to do some filtering and check change of acceleration on finger touchDown. Neat I know ;) patent worthy? I think not.

In this video, OSCulator is routing the OSC (& TUIO) messages coming from MSA Remote to midi and forwarding to Ableton Live and VDMX simultaneously. Nothing is done in post, the same signal is controlling both audio and video.

This is a set of C++ classes for solving and displaying real-time fluid dynamics simulations based on Navier-Stokes equations and Jos Stam's paper on Real-Time Fluid Dynamics for Games. The solver class has no dependencies on openFrameworks and can be used in any C++ project. The drawer class extends ofBaseDraws and contains an ofTexture for seamless integration with openFrameworks drawing routines. Also included in the addon is a ofxMSAParticleUpdater class which allows the fluid solver to be easily plugged into ofxMSAPhysics as a force field.

Here are some videos of my msafluid for processing library out in the wild.

Superfluid vs Particle from jimi hertz on Vimeo.

MSA Fluids test from den ivanov on Vimeo.

MSAFluid on a MultiTouch table from xTUIO from Sandor Rozsa on Vimeo.

About

This is a library for solving real-time fluid dynamics simulations based on Navier-Stokes equations and Jos Stam's paper on Real-Time Fluid Dynamics for Games. While I wrote the library primarily for processing it has no dependency on processing libraries and the source can be used with any Java application.

C++ version for openFrameworks can be found here.

The video below is a demo of a processing sketch using MSAFluid, being controlled by MSA Remote on iPhone (You can view the video in HD and download a 1080p version at vimeo).

Superfluid vs Particle from jimi hertz on Vimeo.

MSA Fluids test from den ivanov on Vimeo.

MSAFluid on a MultiTouch table from xTUIO from Sandor Rozsa on Vimeo.

ofxMSAPhysics is a C++ 3D particle/constraint based physics library for openFrameworks. It uses a very similar api to the traer.physics library for processing to make getting into it as easy as possible.

Version 2.0a is now available for testing.

Main features include

• particles
• springs
• attractions (+ve or -ve)
• collision
• replay saving and load from disk (temporarily disabled in current alpha release)
• custom particles (extend ofxMSAParticle and add to the system)
• custom constraints (extend ofxMSAConstraint and add to the system)
• custom force fields (extend ofxMSAParticleUpdater and add to the system) 
• custom drawing (extend ofxMSAParticleDrawer and add to the system)

Made with openFrameworks.

Version 1.1 is now in the appStore. New version brings serious optimizations thus max amount of dust is now tripled.

This is a port of my original gold dust app (vimeo.com/2281077) to iPhone. Of course it doesn't translate exactly but this is my first attempt to capture the essence of the original demo. Unfortunately I don't think this video manages to capture the behavior of the glittery gold dust either, but it's free so you can try it for yourself :P.

It was written with a custom version of openFrameworks (pre-006) and an early version of the ofxiPhone addon. Information on openFrameworks and ofxiPhone at memo.tv/ofxiphone.

Get it from the iTunes App Store here.

Made with openFrameworks.

Over the years OpenGL has developed lots of different ways of sending vertex data to the graphics card: Immediate mode (glBegin / glVertex / glEnd etc.), Display Lists, Vertex Arrays, Vertex Buffer Objects etc. The first of these (Immediate Mode) is quite inefficient, so it's been dropped in OpenGL ES and depreciated in OpenGL 3.0. Instead we are to use Vertex Arrays or VBO's (Vertex Buffer Objects). They are way more efficient, but generally not as straightforward to setup and use.

For this reason I wrote a C++ class that wraps up the functionality of Vertex Arrays (and soon VBO's) in an immediate mode style syntax. I.e. you can carry on calling glBegin / glNormal / setColor / glTexCoord / glVertex / glEnd etc. but instead of sending the data on every function call, it just caches them all in client-side arrays, and sends it all in one go when you call glEnd(). Later you can call draw() to redraw the information you've put in. Thus improving performance on desktop systems, and allowing immediate mode style syntax on embedded systems such as the iPhone. (The previous post about the Meshulator uses this class).

Early tests of Meshmerizer for iPhone

Screengrabs can be seen at flickr.com/photos/memotv/sets/72157612949492089/
(they look so much more aliased than they do on the iPhone screen, I should do some FSAA)

Made with openFrameworks.

UPDATE:

This code was for openframeworks 005. Things have changed for openframeworks 006 and it is a lot simpler now. See the updated example here.

Here is some sample code for using (and comparing performance with) Vertex Arrays, VBO's and Point Sprites with openFrameworks 005 to have half a million particles at 60fps (on Mac Pro) or 1M particles at 30fps. Note you will need to hack the core and change ofTexture to only use GL_TEXTURE_2D and never GL_TEXTURE_RECTANGLE_ARB (hopefully with oF 006 hacking the core will not be necessary).
P.S. More performance can be gained by splitting the particle update into multiple threads or moving onto the GPU.
check testApp::keyPressed to see what keys do what...

Update

I'm delighted to announce that my "Gold" installation has been selected to be shown at the Tent London exhibition as part of the London Design Festival. 24-27th September 2009, at the Truman Brewery, London, UK. Stay tuned for more information.

The video below is an early demo of the installation.

“Gold” is an interactive installation which explores our obsession with super-stardom, and the extravagance that accompanies it. Through a ‘magic mirror’, revel in a world of excess where you are the super-star. Shower in glittery gold, experience almost omnipotent powers as you materialize, morph and dematerialize into pure sparkling gold dust. Immortalize yourself as a shimmering golden statue, before you collapse and fade away.

The installation uses custom software written with openFrameworks and the OpenCV computer vision library. The software analyzes the video feed from infra-red cameras in real-time and generates 1080p HD output using OpenGL.

An iPhone adaptation of this can be found here

Source code for the particle system in this demo (minus the fancy effects) can be found at http://memo.tv/vertex_arrays_vbos_and_point_sprites_with_openframeworks
This demonstrates how to use VBOs, Vertex Arrays and Point Sprites.

This is a demo of projection mapping with VDMX & Quartz Composer inspired by deepvisual's tutorial of doing it in modul8 (uk.youtube.com/watch?v=2bRfdn9lNO8).

VDMX unfortunately doesn't have this feature built-in, but fortunately has beautiful integration with Quartz Composer - allowing me to build a quad warper in QC using a GLSL vertex shader, which should be super fast.

Also, around the 4:30 mark you'll see me masking the video on the box in the back. This is also using a custom Quartz Composition which allows 4 point mask creation. Usage is almost identical to the QuadWarper, but instead of warping the image it just applies a mask, or you can invert the mask and it cuts a chunk out. You could do the same by creating new layers, grouping, using as a layer mask etc. but its a a bit more hassle I think. Using the QuadMask is a lot quicker and you can put multiple QuadMasks on the same layer to draw more complex masks.

** Update 27/10/08 - Version 1.1 **

Added loop mode configurable from the preferences.plist

-----------------

MSAQTPlayer is a Mega Super Awesome (& mega super basic), fullscreen / multiple output Quicktime player with fast, greater-than-4096-pixels support.

I wrote this app for an installation because I needed to play a 4,800 x 600 Quicktime file across 2 outputs on my graphics card, each feeding 3 triple heads to go into 6 projectors, and I couldn't find a single application that could do it.

Most video applications which can output across multiple video-outs cannot play files larger than the maximum size of a texture (usually 4096 pixels). And even if the movie resolution fits in a texture, they are very slow because they convert each frame to a texture and upload to the graphics card. The only application I found which could play a large file is Quicktime Pro, but that cannot output across multiple video-outs.

MSA QTPlayer is a native Cocoa app which can do this (using QTKit and OSX 10.5) .

Well I finally caved in and bought an iPhone - and my favorite feature (and main reason for buying it) is of course the multi-touch capabilities. So currently OSCemote is my favourite app. Apart from having a few sliders and knobs which transmit OSC (similar to TouchOSC), it also has a multitouch pad which sends out TUIO messages, so any app which responds to TUIO (E.g. anything written with reactivision api) will respond. So I had to try out my visualisation for the Roots Project! Up and running in 5 minutes! Awesome! (I had to rotate the coordinates in my processing code though to have the long end of the iphone screen map to the long end of my desktop screen, slightly annoying that this isn't an option in the app... hopefully soon :P).

I'm very much into creating intuitive interactivity with minimum dependency on a controlled enviroment - so the experience can easily be recreated elsewhere with minimal hardware & setup (which is why I generally prefer optical flow analysis over blob tracking if I can, for vision related projects). So a conversation in the vidvox forums about painting in Quartz Composer using the Wiimote but without using the IR sensor really sparked my interest.

This is a demo for a traer like physics library for C++/openFrameworks.
I wrote the lib for a little project with Todd Vanderlin while in Linz, Austria at Ars Electronic 2008 (vimeo.com/1707467). I tried to keep the same API as Traer (so processing code using traer can easily be ported to oF) and it is basically a particle system with springs, attractions, gravity etc (but uses verlet integration instead of RK4).

In this demo I am interacting with the app using the keys:

A set of C++ template classes for doing various types of interpolations on data with any number of dimensions. You can feed the system an arbitrary number of data (data can be simple types like float, int or complex types like structs, classes), then resample at any resolution, or ask for the value at any percentage along the data - or just draw it - including splines in 3D.
This is useful for creating and drawing splines (in any dimensions), or creating smooth animation paths from keyframes (again, in any dimensions).

Made with openFrameworks.

Source code on SVN here

This is an 'early current state of app' demo for a multi-discipline event I'm working on with Streetwise Opera, Mira Calix and fellow visualists Flat-e, to be showcased at the Royal Festival Hall later this year with quite a few more venues lined up.

The app was written in Processing 0135 and is running realtime at 60fps, though if I add another couple hundred eels it does drop, so I may switch to OpenFrameworks if performance does become an issue (which it probably will). There are occasional freezes in the video which happened while capturing the screen so that is a bit annoying.

I'm controlling the eels using the mouse, keyboard and Quartz Composer (just simple sliders sending OSC to vary some parameters - similar to the 'magnetic force fields' video - I'm quite into this technique now, very quick and easy to setup, and you can have loads of sliders with descriptive names at your disposal to play with, and adjust your internal variables in realtime for tweaking heaven).

The final show will have many many more features, both in the digital realm, and physical... more info coming soon...

I strongly recommend watching the HD version at http://www.vimeo.com/1582196

This is a very early version of a GPU based depth-of-field GLSL shader and sample Processing code. Adjust some parameters and it can also be used to give the scene a nice backlight/glow effect.

This is a demo of creating and visualizing magnetic (kind of) fields in Processing and controlling with a tangible multitouch table and Quartz Composer. It gets more interesting after the 1 minute mark :P

I recommend watching the video in HD at http://www.vimeo.com/1569676

The demo came about as a digression off the Roots project I'm working on with Jordan & Owen - makers of the Bricktable (http://bricktable.wordpress.com/). You can read more about the Roots project at http://www.memo.tv/roots_creating_and_visualising_generative_music_on_a_... and http://bricktable.wordpress.com/about/what-is-roots/ .

I was sitting minding my own business, exploring accumulation buffers in Quartz Composer, when all of a sudden I just zoned out, and next thing I know, I found myself staring at Him on my screen. His Noodly Appendages came down and touched me, and guided my hands, connecting Quartz Composer's very own noodles in His Image.

His Noodly Screen Saver runs on Mac OSX 10.5 (Leopard) and you can download it below.

For those who do not have a clue what this is about, I suggest reading this: venganza.org/about
and this: venganza.org/about/open-letter

P.S. you can watch the above demo video in HD at vimeo.

RAmen.

Thanks to the windy ways of the web, I've found myself working with some truly talented musicians/techies/electronics experts over on the otherside of the pond in California, on a very exciting interactive, generative audio/visual project. The number of traditional instruments they have and play wasn't enough for them, so they decided to build their own, as one does when in that situation - one of which is the bricktable, a tangible and multi-touch table - and instrument.

I've worked on a number of interesting interactive audio projects, but the approach in this one is quite different and i"m very excitied to be working with the bricktable guys on it.

In one line: You control parameters of a chaotic environment - which affect the behaviour of its inhabitants - which create and control music. 

To breakdown very briefly without going into much detail:

I've always wondered (well at least since Quartz Composer 3.0 was released with Leopard) the performance difference between using the new and much easier to use Math Expression patch, and the old Math patch - and in fact compared to using JavaScript.

So I created the attached test composition and found some surprising results (at least I found them surprising, though in retrospect I can understand why :P).

First of all, all 3 methods are pretty quick, and are unlikely to be a bottleneck. Unless you are using the operations in an Iterator patch with a lot of iterations you won't notice any difference.

The figures below are for 2nd gen 2.33Ghz Macbook Pro:

Quartz Composer is a great piece of software for many things. It has a lot of features which really allow you to create amazing things very quickly. It also has some 'features' which allow you to lose your hair very quickly. One of these 'features' is loading images from within an iterator.

You'd think it was quite straightforward, just send a different string (either generated within the iterator or loaded from XML etc.) to the Image Downloader, but alas QC has other plans. It always loads the same image whatever string you send it!

So I started playing with the House of Cards data in OpenFrameworks, but looks like I'm not gonna have time to finish it for a while. So I thought I'd post a skeleton if anyone else wants to play.

Also added a Processing source file as well using the BIN data (P5_HoC_bin_v1.zip). Demo video below (all interactions are controlled with the mouse in realtime - pulling/pushing etc.