Postdoctoral Research Associate with Steve Seitz

PhD 2011 from Columbia University

Co-advised by Peter Belhumeur & Shree Nayar

neeraj@cs.washington.edu

Computer Science and Engineering

University of Washington

Box 352350, Paul G. Allen Center 282

Seattle, WA 98195-2350

Research Overview

I work at the intersection of computer vision and machine learning, developing techniques that leverage large collections of real-world images for a variety of applications. I am particularly interested in the use of intermediate representations such as parts and attributes, both for improving performance on classical vision tasks such as search and recognition, as well as for creating novel applications such as automatic face replacement and exploration of image collections. Much of my work is directed toward human faces, although I am also very interested in the growing area of fine-grained visual categorization, which seeks to distinguish categories within a particular domain. One such project is Leafsnap, an electronic field guide that lets users identify plant species by photographing their leaves. The freely-available apps have received over a million downloads.


Watch a recent talk I gave at UNC-Chapel Hill describing my PhD work.


neeraj headshot

News

July 25, 2013: LFPW dataset available again from my website
The Labeled Face Parts in the Wild (LFPW) dataset is temporarily available from my webpage, as the KBVT site on which it was previously hosted has been undergoing renovations.
October 14, 2012: ECCV 2012 Leafsnap presentation slides released
The slides for our ECCV 2012 presentation on Leafsnap are now available.
August 01, 2012: ECCV 2012 paper on Leafsnap released
Our ECCV 2012 paper on Leafsnap is now available. This will be an oral presentation in Florence, Italy this October.
June 20, 2012: Multi-attribute spaces project page and software
Project page for multi-attributes paper added to website, as well as link to attribute calibration software.

Curriculum Vitae

CV (pdf)
Citations to my work

Talks (see all)

CSE 590V: Computer vision seminar (Fall 2011)

Selected Projects (see all)

Leafsnap: An Electronic Field Guide

Summer 2009 - Fall 2012
Leafsnap: An Electronic Field Guide

Automatically identifying plant species using photos of leaves taken in the field by mobile devices

Multi-Attribute Spaces: Calibration for Attribute Fusion and Similarity Search

Calibrating the outputs of multiple attribute classifiers into normalized multi-attribute spaces for fusion and similarity search

Localizing Parts of Faces Using a Consensus of Exemplars

Robustly localizing face parts using local detectors and non-parametric, exemplar-based global models

Attribute and Simile Classifiers for Face Verification

Identifying faces using describable visual attributes of the face, automatically detected using attribute and simile classifiers

FaceTracer: A Search Engine for Large Collections of Images with Faces

Searching face images using automatically-trained classifiers of descriptions of face and image attributes

Face Swapping: Automatically Replacing Faces in Photographs

Using large collections of images to automatically replace faces in photos

Selected Software (see all)

Attribute Calibration Software

C/C++/Python code to calibrate attribute classifier outputs using the w-score formulation, based on the Extreme Value Theory. For non-commercial uses only.

Leafsnap

Spring 2011 - Fall 2012
Leafsnap

The Leafsnap homepage, with links to download the iPhone and iPad apps, as well as browse the list of species and their images.

MugHunt Online Face Search Engine

Spring 2010 - Fall 2012
MugHunt Online Face Search Engine

An online face search engine built using this research. This search engine is a joint collaboration with Securics, Inc..

Databases

Labeled Face Parts in the Wild (LFPW)

1,432 Images with 29 points labeled on each image

PubFig: Public Figures Face Database

60,000 face images with several face verification benchmarks & 65 automatically computed attribute labels for 42,000 images

FaceTracer Database

15,000 face images with detailed pose and fiducial information & 5,000 manual attribute labels

Selected Publications (see all)

  1. "Localizing Parts of Faces Using a Consensus of Exemplars,"
    IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI),
    vol. 35, no. 12, pp. 2930--2940, December 2013.
  2. "How reliable are your visual attributes?,"
    Proceedings of SPIE Volume 8712: Defense, Security, and Sensing,
    April 2013.
  3. "Leafsnap: A Computer Vision System for Automatic Plant Species Identification," (oral presentation)
    Proceedings of the 12th European Conference on Computer Vision (ECCV),
    October 2012.
  4. "Multi-Attribute Spaces: Calibration for Attribute Fusion and Similarity Search,"
    Proceedings of the 25th IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
    June 2012.
  5. "Fusing with Context: a Bayesian Approach to Combining Descriptive Attributes," (oral presentation)
    Proceedings of the IEEE International Joint Conference on Biometrics (IJCB),
    October 2011.
  6. "Two faces are better than one: Face recognition in group photographs,"
    Proceedings of the IEEE International Joint Conference on Biometrics (IJCB),
    October 2011.
  7. "Describable Visual Attributes for Face Verification and Image Search,"
    IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI),
    vol. 33, no. 10, pp. 1962--1977, October 2011.
  8. "Describable Visual Attributes for Face Images," (PhD Thesis)
    Technical Report CUCS-035-11, Department of Computer Science, Columbia University,
    August 2011.
  9. "Localizing Parts of Faces Using a Consensus of Exemplars,"
    Proceedings of the 24th IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
    June 2011.
  10. "Attribute and Simile Classifiers for Face Verification," (oral presentation)
    Proceedings of the 12th IEEE International Conference on Computer Vision (ICCV),
    October 2009.
  11. "FaceTracer: A Search Engine for Large Collections of Images with Faces,"
    Proceedings of the 10th European Conference on Computer Vision (ECCV),
    pp. 340--353, October 2008.
  12. "What is a Good Nearest Neighbors Algorithm for Finding Similar Patches in Images?,"
    Proceedings of the 10th European Conference on Computer Vision (ECCV),
    pp. 364--378, October 2008.
  13. "Face Swapping: Automatically Replacing Faces in Photographs," (oral presentation)
    Dmitri Bitouk, Neeraj Kumar, Samreen Dhillon, Peter N. Belhumeur, Shree K. Nayar,
    ACM Transactions on Graphics (Proceedings of SIGGRAPH),
    vol. 27, no. 3, August 2008.