Authoring effective depictions of reality by combining multiple samples of the plenoptic function

Aseem Agarwala

Cameras are powerful tools for depicting of the world around us, but the images they produce are interpretations of reality that often fail to resemble our intended interpretation. The recent digitization of photography and video offers the opportunity to move beyond the traditional constraints of analog photography and improve the processes by which light in a scene is mapped into a depiction. In this thesis, I explore one approach to addressing this challenge. My approach begins by capturing multiple digital photographs and videos of a scene. I present algorithms and interfaces that identify the best pieces of this captured imagery, as well as algorithms for seamlessly fusing these pieces into new depictions that are better than any of the originals. As I show in this thesis, the results are often much more effective than what could be achieved using traditional techniques.

I apply this approach to three projects in particular. The ”photomontage” system is an interactive tool that partially automates the process of combining the best features of a stack of images. The interface encourages the user to consider the input stack of images as a single entity, pieces of which exhibit the user’s desired interpretation of the scene. The user authors a composite image (photomontage) from the input stack by specifying high-level objectives it should exhibit, either globally or locally through a painting interface. I describe an algorithm that then constructs a depiction from pieces of the input images by maximizing the user-specified objectives while also minimizing visual artifacts. In the next project, I extend the photomontage approach to handle sequences of photographs captured from shifted viewpoints. The output is a multi-viewpoint panorama thatis particularly useful for depicting scenes too long to effectively image with the single-viewpoint perspective of a traditional camera. In the final project, I extend my approach to video sequences. I introduce the “panoramic video texture”, which is a video with a wide field of view that appears to play continuously and indefinitely. The result is a new medium that combines the benefits of panoramic photography and video to provide a more immersive depiction of a scene. The key challenge in this project is that panoramic video textures are created from the input of a single video camera that slowly pans across the scene; thus, although only a portion of the scene has been imaged at any given time, the output must simultaneously portray motion throughout the scene. Throughout this thesis, I demonstrate how these novel techniques can be used to create expressive visual media that effectively depicts the world around us.

Aseem Agarwala. "Authoring effective depictions of reality by combining multiple samples of the plenoptic function", Doctoral thesis, University of Washington, 2006.

2006 ACM Doctoral Dissertation Award, Honorable Mention
2006 William Chan Memorial Dissertation Award

Reading committee
David Salesin, Michael Cohen, Steven Seitz