Leafsnap: An Electronic Field Guide

Keywords: species identification, electronic field guide, curvature histograms, education, leafsnap, smithsonian, botany
Summer 2009 - Fall 2012

Description

Studying the world's rapidly-dwindling biodiversity and educating the public about these issues is an important task for botanists. One of the primary ways to help with this task is to automate the tedious and error-prone process of identifying existing plant species. Computer vision techniques are now sufficiently sophisticated that we can do this reliably using an electronic field guide. The guide runs as an application on a mobile device such as the iPhone, allowing users to take photos which are sent to a server where they are automatically recognized. Including high-quality photos taken by the non-profit Finding Species and expert curation by botanists at the Smithsonian Institution, this field guide becomes a state-of-the-art and authoritative resource for both botanists and the general public.

The benefits to the end-user are obvious and significant: they now have available at their fingertips access to a field guide with high-quality, attractive photos, advanced search capabilities, and automatic recognition of all plant species in their area. The app also includes compelling games that help train users in learning to recognize and distinguish different plant species on the basis of their leaves, fruits, or flowers. The whole package also makes a much better introduction to botany and taxonomy for children. Existing, printed field guides, while adequate for recognition by adults and those with experience in classification, are often too difficult for children to use. Perhaps even more importantly, they are not very inviting to kids, and the diagramatic images of plants they contain are neither compelling nor life-like.

Leafsnap turns users into citizen scientists, automatically sharing images, species identifications, and geo-coded stamps of species locations with a community of scientists who will use the stream of data to map and monitor the ebb and flow of flora nationwide.

The recognition process, developed by the research groups of Peter Belhumeur at Columbia University and David Jacobs at the University of Maryland, consists of:

  1. Segmenting the image to obtain a binary image separating the leaf from the background. This is currently implemented using an Expectation-Maximization framework, estimating foreground and background color distributions in the HSV colorspace.
  2. Extracting features from the binarized image for compactly and discriminatively representing the shape of the leaf. We use histograms of curvature over scale as the feature representation, robustly and efficiently implemented using integral measures of curvature.
  3. Comparing the features to those from a labeled database of leaf images and returning the species with the closest matches. Due to the discriminative power of the features and the size of our labeled dataset, we use a simple nearest neighbor (NN) approach with the L1-norm.
This whole process is completed in under 5 seconds, and can be highly parallelized across many machines.

Publications

  • "Leafsnap: A Computer Vision System for Automatic Plant Species Identification," (oral presentation)
    Proceedings of the 12th European Conference on Computer Vision (ECCV),
    October 2012.
  • "Describable Visual Attributes for Face Images," (PhD Thesis)
    Technical Report CUCS-035-11, Department of Computer Science, Columbia University,
    August 2011.

Software

Leafsnap

Leafsnap:

The Leafsnap homepage, with links to download the iPhone and iPad apps, as well as browse the list of species and their images.

Videos

Leafsnap Introduction

Leafsnap Introduction:

This video introduces Leafsnap, including how to use it, its purpose, and more!

ECCV 2012 Leafsnap Video Preview

ECCV 2012 Leafsnap Video Preview:

This 30-second clip was created to play at ECCV 2012 on the monitors throughout the conference venue. It shows the full pipeline of using Leafsnap for visual identification (no audio).
IntoMobile Leafsnap demo

IntoMobile Leafsnap demo:

The website IntoMobile demonstrated our app on their website. Here is the video they made.

Gizmodo Leafsnap for iPad demo

Gizmodo Leafsnap for iPad demo:

This video shows an interview conducted by Gizmodo where we demonstrated the iPad version of Leafsnap for them.
Lessons from Photographing and Identifying the World's Plant Species

Lessons from Photographing and Identifying the World's Plant Species:

This is a talk given Prof. Peter Belhumeur describing the inception and development of this project. It was presented at ICCP 2011.

Images

EM-based Segmentation in the HSV Colorspace

EM-based Segmentation in the HSV Colorspace:

To segment the image in the top left into the one on the top-right, we run several rounds of Expectation-Maximization using the SV channels of the HSV colorspace. Each of the 3 bottom rows shows one iteration of the algorithm. The left column shows the current status of each pixel, and the right column shows the distribution of pixels in the SV colorspace and the current segmentation.
Why Curvature Histogram Features?

Why Curvature Histogram Features?:

As a motivating example, consider these 4 different leaf species. The two on top have the same rough shape, as do the two on the bottom. However, the two on the left have smooth boundaries, while the two on the right have serrated edges. So if we construct histograms of the curvature values along the edges at a rough scale and at a fine scale, it will be sufficient to distinguish all 4 species. In practice, we compute these at 25 different scales to get high discriminability.
Computation of Curvature Histograms Using Integral Measures

Computation of Curvature Histograms Using Integral Measures:

We compute curvature values using an integral measure as opposed to the usual differential method. This is done by placing a circle centered at the given point and counting the fraction of pixels that are inside the object. The radius of the circle gives a natural notion of scale. This measure is robust to various sources of noise, is rotation-invariant, and by first resizing images to a constant segmented area, it becomes roughly scale-invariant as well. We extract histograms at 25 different scales.
Recognition Accuracy on the Northeast

Recognition Accuracy on the Northeast:

This graph shows the accuracy of our system evaluated on tree species in the northeast US. Each point represents the percentage of times that the correct match is within the first x results. So the first point shows a rank-1 accuracy of around 70%, the second point a rank-2 accuracy of around 82%, and so on.
Guided Tour of Leafsnap - Home Screen

Guided Tour of Leafsnap - Home Screen:

The home screen of Leafsnap, showing a random image and links to all major operations. There is also a link to various games that help train users in distinguishing different species. Our hypothesis that a few weeks of playing these games can drastically increase a user's skill at identifying tree species -- perhaps even more than a trained botanist!
Guided Tour of Leafsnap - Browse Screen

Guided Tour of Leafsnap - Browse Screen:

The browse screen of Leafsnap lets users browse through all species. Users can toggle between thumbnails of leaves, flowers, and fruits to quickly find particular species by visual inspection. They can also sort the list by scientific name, or first or last word of common name.
Guided Tour of Leafsnap - Search

Guided Tour of Leafsnap - Search:

Users can search for specific species by typing in parts of the scientific or common name to filter the list of species shown. Additionally, they can set their location in the options, filtering the list by only the species that exist in that location.
Guided Tour of Leafsnap - Detail View

Guided Tour of Leafsnap - Detail View:

Clicking on a species shows the detail view for that species, with high-quality images of its leaf, flower, fruit, seed, petiole, bark, etc.
Guided Tour of Leafsnap - Detail Quality

Guided Tour of Leafsnap - Detail Quality:

The detail photos are available in very high quality, showing even the hairs on the petiole!
Guided Tour of Leafsnap - Snap It!

Guided Tour of Leafsnap - Snap It!:

Clicking 'Snap It!' allows users to take a photo of a leaf and send it to the server for automatic recognition. The photo must have a single leaf on a fully-white background, so that segmentation can work successfully on the image.
Guided Tour of Leafsnap - Snap It! Results

Guided Tour of Leafsnap - Snap It! Results:

The server returns a set of ranked matches. Ideally, the correct match will be first in the list, although this is not always true; however, the correct match is almost always within the top 10.
Guided Tour of Leafsnap - Snap It! Verification 1

Guided Tour of Leafsnap - Snap It! Verification 1:

To make the final identification, the user can click on a result to see the detail page and verify if it's the right species by looking at other characteristics of the tree.
Guided Tour of Leafsnap - Snap It! Verification 2

Guided Tour of Leafsnap - Snap It! Verification 2:

Characteristics such as the tree bark or even the petiole are often very discriminative. While it is too difficult to automatically recognize these characteristics, the presence of high-quality photos in the app allows users to use these cues themselves by visually inspecting the tree in front of them.
Guided Tour of Leafsnap - Snap It! Verification 2

Guided Tour of Leafsnap - Snap It! Verification 2:

Textual information about the species, including habitat, growth pattern, bloom time, and coverage information, can often be useful at this stage.
Guided Tour of Leafsnap - Labeling a Leaf

Guided Tour of Leafsnap - Labeling a Leaf:

Once the user has made the final identification, they can label the leaf by swiping across the correct match. This will send the identification to our server, where we plan to use it in the future to increase our recognition database, assuming we can first confirm the accuracy of the label in some way.
Guided Tour of Leafsnap - Collection

Guided Tour of Leafsnap - Collection:

Labeling a leaf also shows it with the right species name in the collection tab of the app, where the user can examine their past collection history. Each leaf shown here also contains the time the leaf was collected.
Guided Tour of Leafsnap - Map

Guided Tour of Leafsnap - Map:

Leaves in the user's collection are also viewable on a map, as we store GPS information along with each image. This way, a user can track their collection history over both time and space.