The modern Internet contains incredible quantities of digital video
available for viewing. With the advent of inexpensive
consumer-level digital video cameras many people have chosen to record
their moments in the form of digital video. The proliferation of
surveillance cameras has only added to this growing mountain.
Video can be viewed as a very large three-dimensional data set[1] across the axes of two-dimensional space and
time.
It is often inpractical to view this data set in its original form: the
moving image. A security agent may wish to quickly comprehend the
motion in several long video sequences. In many cases, the content
of a video or animation needs to be shown in a single image for a
printed medium such as a newspaper or article. A particularily
common requirement in the field of computer graphics is the need to show
human figure animation in a figure to form part of an academic
publication. Finally, video consumes significant bandwidth; in
many cases it is more economical to visualize the video in a single
image.
The field of video summarization is a large and heavily researched one.
There has also been significant research into the visualization of
very large data sets. However, few in the the research
communities have considered the problem of visualization of video.
If video is a very large data set, perhaps novel slices and views
of this data could yield insights? Indeed, planar slices of the
video cube have been explored [1]; the
results, however, are not effective as a visualization technique.
Instead, we take the approach of selecting data from video in a
non-planar fashion. We consider the restricted case of video
displaying human motion, with the goal of creating a single image that
conveys the motion to the viewer. This is
an incredibly challenging problem; information filling three dimensions
must be compressed into a two-dimensional form. It is easy to
quickly overwhelm the limited real estate of a single image with the
information of human motion.
Indeed, the scope of the problem quickly overwhelmed our original plan
to automate the creation of one or two types of visualizations.
The range of possible visualization components and techniques is
immense, and we would have little justification in choosing one.
Instead, we chose to explore the space of possible motion
visualization techniques, creating exemplars by
hand using both available software and mini-tools created for specific
tasks. In this way, we were able to evaluate the effectiveness of
different visualization techniques, narrowing down our toolbox through
informal user feedback. Throughout the process of creating
these images, we held interviews with fellow grad
students. We presented them with the video clip, followed by several
visualizations, and asked them to comment on which they felt best
captured the essence of the motion and which they found aesthetically
pleasing. At the same time, we encouraged them to critize the
techniques and to suggest alternative ways to encode the motion in 2D.
Through this process, we were about to single out certain
visualization techniques as being particularly promising, and hope
that this exploration can form a springboard for future research
automating the most effective of the methods studied.
Related work
While few in the scientific communities have considered the problem of
visualizing human motion, artists have long been interested the problem.
We relate a few of the relevant efforts which influenced our
approach.
Strobing
The most relevant historical reference to our research is the work of
French photographer Etienne-Jules Marey (1830-1904). Marey devoted
his entire career to developing photographic techniques to capture
movement across time. An excellent summarization of his work can
be found in the book "Picturing Time." [2].
He developed a variety of special-purpose devices to accomplish
multiple-exposure photographs that captured a subject in multiple
positions across time in a single photograph. An example of his
work is shown in Figure 1.
Figure 1: Two multiple exposure
photographs by Etienne-Jules Marey.
Marey's work inspired two other relevant photographers. One is
Harold Edgerton, who used electronic strobe flashes to achieve similar
effects. Indeed, most of our visualization techniques are inspired
by this multiple-exposure and strobing approach. However, both
Marey and Edgerton were limited by analog technology; we are able to
expand the range of possibilities using digital approaches. The
photographer Eadweard Muybridge was also influenced by Marey. He
was interested in using photography to study human motion, and captured
several studies similar to the one showed in Figure 2.
Muybridge and Marey influenced a variety of artists, perhaps most
notably in Duchamp's famous motion study "Nude Descending a Staircase."
(Figure 3) Muybridge added to Marey's
approach by using multiple simultaneous perspectives. His style
of separating individual moments into frames is highly reminiscent of
comic books, which brings us to our other major inspiration.
Figure
2: Muybridge motion study entitled "Descending stairs and
turning around"
Figure 3: Marcel Duchamp's
"Nude Descending a Staircase"
Comic books
Comic book artists have long been faced with the challenge of conveying
human motion in static form. An excellent discussion of this issue
is found in Scott McCloud's "Understanding Comics." [4]
He notes that a single frame in a comic book rarely depicts a
single moment in time, but rather slices across the space-time volume.
An example of such a space-time frame from this book is shown in Figure 4.
Figure 4: A comic strip frame
that covers a range of time.
Comic book artists have developed a number of techniques to convey the
notion of human motion in a single frame. The most common are
"motion lines"; diagrammatic elements that indicate motion.
Examples are shown in Figure 5. We will
attempt to make such of such motion indicators in our visualizations.
Figure 5: Several examples of
motion lines.
We will focus on the contrasting techniques of comic strips and
multiple-exposure strobing. Strobing is highly effective in
showing motion in a compressed form and can situate the relevant action
poses in a unified context. Unfortunately, in slow moving sequences,
overlap between images can be confusing and overload information into too
small a space. Comic strips do not suffer from this overloading, but
require more space and, more importantly, can lose spatial context between
the images.
NEXT: Strobing
