The installation includes three computers: a body projection PC, a front projection PC and a back projection PC.
Participants stand on the stage, and their skeletons are tracked by Kinect and OpenNI+NiTE (ofxOpenNI) on the body projection PC. The body projection PC is connected to a projector facing to the stage which is calibrated by customized mapamok. Tracked hand coordinates are warped to the projector coordinate system, and fluid is generated on the warped position with an initial velocity corresponding to the hand velocity (ofxFluid). Thus, virtual fluid is projected on the hands and the residual can be projected onto the body. To avoid projection on faces, black masks are overlaid on the faces tracked by NiTE.
The hand coordinates are sent to the front projection PC through OSC for projection on a sculpture. This PC renders similar fluid simulation. OSC is proxied to a Unity3d app which renders ripples by a customized water asset. The scuplture projection PC blends fluid and ripples by Syphon and Resolume, which can be easily controlled by the artists. Finally, the blended video is warped by MadMapper for projection.
The back projection PC projects user contributed videos and images on the other side of the sculpture.