Strobing

Our first exploration in visualizing motion is based on Marey's multiple-exposure approach, which we denote as "strobing". We extract the desired subject from individual frames of a video sequence and composite them into a single image.

Tools

Creating strobing visualizations requires several steps. The first is matting: the extraction of foreground objects from the video frames.  Recent research performed at UW [4,5] provide algorithms for matting in still-images and video, focusing on difficult cases where the foreground object has soft edges (such as wispy hair). We used these tools for some of our examples, but for the most part found them too complex for our needs. For simple foreground elements with hard edges, we simply used Adobe Photoshop.  To do this, we copied the object along with the surrounding background, used the "magnetic lasso" to select the foreground object, and then feathered the edges to create soft falloff before deleting the excess.

The second common step is background reconstruction. Many of the video sequences we used contain camera movement.  Strobe images created using traditional photography must use a fixed camera, lest the backgrund features streak and overlap.  Digital strobing gives us the capability to handle this, by overlaying the foreground sprites over a common background image stitched together from the many frames of video.  The automation of this task has been the focus of recent research [6]; however, we did not have access to any working code or the various commercial software packages for building mosaics.  Thus, for many sequences, we constructed these background plates by hand in Photoshop (see Figure 6 for a sequence from Point Break).  For others, we simply discarded the background information.

backgroung plate
Figure 6: A background mosaic. Video

Fixed frequency Strobing

Our first approach was to emulate Marey; foreground sprites were extracted at a regular frequency and composited into a single image.  One degree of freedom inherent in this process is the frequency of sprites.  In the following three visualizations (Figure 7) of the Point Break sequence, this frequency was adjusted; every second, fourth, or sixth frame was used.  

Freq 2
Freq 4
Freq 6

Figure 7: Three fixed-frequency strobing visualizations, using sprites from every second, fourth, or sixth frame, respectively.  Video
In our user studies, viewers consistently preferred the first visualization, citing the overlap among images as helping ground the fact that it was all a single motion.  In the visualizations with more intermittent strobing, in the words of one viewer, "it looks like there are a bunch of clones running after one another."  The further apart the sprites, the harder it seems it is to make the mental leap that this image is sampling the motion in time.  Therefore, the spacing of sprites is a crucial factor in the efficacy of a strobing visualization.  Determing the ideal frequency is difficult in traditional photography, since it must be chosen ahead of time.  Luckily, on the computer, this optimum frequency selection can be determined after filming the scene.

Nevertheless, note that in the above sequence, the two parts of the action (running and climbing) happen at different speeds, and so the optimum strobing frequency for the running results in significant overlap during the second half.  This overlap, which one viewer called "a jumbled mess," doesn't convey the subtleties of the actual action taking place.

A similar problem can be seen in the following visualizations, which show a character jumping into water (Figure 8); the sequence is taken from the movie Iron Giant.  In the first visualization, every sprite is shown; this is effective towards the end, but unnecessarily busy in the beginning.  The second uses every third frame; this is effective in the beginning, but clearly undersamples the end.

Iron Giant 1Iron Giant 2

Figure 8: Two fixed-frequency visualizations showing every frame and every third frame, respectively.  Video

Intelligent frequency strobing

We thus drew the conclusion that intelligently selecting the frames to place in the visualization could possibly be more effective than simple frequency strobing.  This is clearly not possible with photography, and so we begin to illustrate the real power of using computers over traditional techniques.  For the following two visualizations (Figures 9,10), we hand-selected the sprites using the metric that we should maximize the number of images while minimizing their overlap.

intel point break
Figure 9: Intelligent strobing visualization. Video


intel iron giant
Figure 10: Intelligent strobing visualization. Video
These visualizations clearly reduce the jumbled overlap that can be present in fixed-frequency strobing.  At the same time, they do so at the expense of the timing information present in the previous visualizations.  Viewers found the technique unnecessary for the Iron Giant sequence (Figure 10), where the timing was deemed more important than individual image clarity.  In the Point Break sequence (Figure 9), viewers preferred the intelligent strobing version to the fixed, but still expressed some issues with timing: "It's definitely clearer that he's climbing and more pleasing to watch, but you do kind of get the feeling that he just hopped right up the fence."

With intelligent strobing, one can no longer rely on sprite spacing to inform speed information.  But what if we encode that information using some other degree of freedom?  One solution (Figure 11) that we have begun to explore is using transparency to reinject the lost timing information.  Using the same images, we now adjust each sprite's opacity based on how many neighbors had to be removed.  The running images stay relatively transparent, whereas the climbing sprites get "burned" into the image, giving the sense that more time was spent in that area, without decreasing clarity.


trans point break
Figure 11: Using Transparency to encode lost timing information.  Video

Though intelligent strobing can reduce overlap in many sequences, there are still many for which any kind of strobing is doomed to failure.  The following visualization (Figure 12) depicts a Paul Taylor dance sequence.  Despite carefully choosing of keyframe poses, there is simply too much overlap in the same area of the image to be comprehensible.


Paul Taylor strobing

Figure 12:  Intelligent strobing visualization with considerable overlap. Video

Mobile sprites

We can further exploit the freedom of the digital domain if we relax the constraint that sprites must remain in their original location.  To avoid overlap we simply move the images to space them further apart.  In the following visualizations (Figure 13), we space the images from right to left and vice versa:


Paul Taylor right to left

Taylor right to left

Figure 13: Mobile sprite visualizations, leftwards and rightwards.  Video

Each direction has its own advantages.  The motion of the character is generally leftwards, making the composition of the first visualization more realistic and compact.  However, people are generally used to reading left to right, making it more obvious in the second visualization in which pose the character starts.  Though the compositing of the sprites denotes the ordering (newer images above older), at a glance the top visualization can be easily be mistakenly read backwards.

In the same way that intelligent strobing loses some of the temporal context, mobile sprites loses the spatial context. Viewers found these visualizations vastly more understandable than the simple strobing example (Figure 12), but remarked that without careful study, they seemed like two completely different motions: clearly, much of the information is still being lost.

The following visualization (Figure 14) depicts another dance sequence.  The motion of the character is generally left to right, but the overlap is still distracting.  The sequence is particularly suited for the spacing technique (Figure 15): we can increase the left to right motion of the character, reducing overlap, without losing a sense of the motion.

ya

Figure 14: Intelligent strobing visualization. Video

a

Figure 15: Mobile sprite version. Video

Earlier, we used transparency to encode a specific variable, namely a sprite's velocity.  The human visual system is adept at discerning overlapping translucent layers, and so we can also use this in a more abstract sense to help deal with overlap.  This allows us to overload the same image real estate with several layers of information.  The following visualization (Figure 16) is the same as the top visualization in Figure 13, but the transparency of the sprites range from zero to fifty percent and back to zero.


transparency

Figure 16: Leftward mobile sprites with transparency visualization. Video
Mobile sprites, described in this section, can be seen as a midpoint in the continuum between strobing and comic books, where the sprites are spaced so far apart that they are in different images. Both methods lose the spatial context of simple strobing, but this is less of an issue in comics, where the lack of context is made explicit. In that way, one can avoid the false illusion of directional movement that can be present in the above examples.

NEXT: Comics