After writing the last article, I found a solution for the sync problems. The principle is to make sure that the internal clocks of all the equipment is synchronized (for example, check time.is before recording). This is tricky for action cameras or camcorders but you can always correct the offset later. For video recordings, you can access the metadata of the video file by MediaInfo to check the “Encoded Date.” Convert this date and time into epoch timestamp on Epech Unix Time Stamp Converter and use the converted value on Piecemaker2. Now the video’s timestamp is aligned to the time in the real world (up to a second of precision). I am planning to write a Python script to automate this process if I can find an appropriate library (TODO1).
The rest of the procedure is quite simple as we only need to stick to the epoch timestamp and no need to correct offsets. I edited Pathrefinder on the dev branch so that it stores the geometry data every frame with epoch timestamp. This can be further processed by a Python script to format into TRAC file, which can be read by Piecemeta. Note that Piecemeta does not store timestamps, so do not forget to set the timestamp on Piecemaker2. To do this, take the timestamp of the first frame of the geometry data and use this timestamp for the score scene on Piecemaker2.
For Kinect tracking, use OSCeleton to output the joint data. Before, I recorded the Kinect stream by Kinect Studio, but then, it is more difficult to recover the original timestamps from the recording. Rather I recommend to output the data in real-time so that no adjustment is needed. The log file can be converted by this script to TRAC file ready to be uploaded to Piecemeta. Note that I could not find a way to output csv from Pandas library with epoch timestamp so the timestamp is formated as hour-minute-second-milisecond and not epoch. When you set the timestamp on Piecemaker2, please use the value from the original OSCeleton log file (note that its timestamps are epoch but in milliseconds not seconds).
At last, you can use a Processing sketch to play back the video, skeletal tracking and Pathfinder score data in sync. The current Processing sketch is a quick and dirty hack so it should be simplified (TODO2).