ECEP 596, Autumn 2019 Homework 3: Creating Panoramas

DUE: November 10, 2019 (Sunday) 11:59 P.M.

Late date: November 13, 2019 (Wednesday) 11:59 P.M. (Late policy: 10% penalty per day)

Total points: 30 (+3 extra credit)

Before you start:
Download the assignment's files here.

Assignment:
For this assignment you'll be implementing a panorama image stitcher. This requires the implementation of a Harris corner detector, RANSAC alignment and an image stitcher. Feel free to reuse any code from your last assignment. CONVERT TO GRAY TONE to find derivatives. Do not use the green channel.

In the project, you should only have to update one file "Project3.cpp. " The buttons in the UI will call the corresponding functions in "Project3.cpp. "

What to turn in:

To receive credit for the project, you need to turn in HERE the 'Project3.cpp' file and a PDF file (name it as report.pdf) containing the requested images for each task with proper captions.

Tasks:

Step 1: (10 pts) Implement the Harris corner detector. To do this, you'll need to write:

void HarrisCornerDetector (QImage image, double sigma, double thres, CIntPt **cornerPts, int &numCornersPts, QImage &imageDisplay)

image: the input image,
sigma: the standard deviation for the Gaussian,
thres: the threshold for detection corners,
cornerPts: the returned corner points,
numCornersPts: the number of points returned,
imageDisplay: image returned to display for debugging.

Compute the Harris corner detector using the following steps:
i. Compute x and y derivatives of the image, use them to produce 3 images (I_x^2, I_y^2, and I_x*I_y).
ii. Smooth each image with gaussian blur. The three images represent the Harris matrix H for each pixel.
iii. Compute corner response function R = Det(H)/Tr(H), and threshold R. Try threshold 50 on the UI.
iv. Find local maxima of the response function using nonmaximum supression [use 5x5 window].
We also provide Convolution and GaussianBlurImage functions for you to use.

Required: Open "Boxes.png" and compute the Harris corners. Save an image "1a.png" showing the Harris response on "Boxes.png" (you'll need to scale the response of the Harris detector to lie between 0 and 255). Open "Rainier1.png" with tab "Image 1" selected, and open "Rainier2.png" with tab "Image 2" selected. Use DrawCornerPoints to overlay red crosses on the images.

Step 2: (10 pts) Implement the following keypoint matching function:

void MatchCornerPoints (Qimage image1, Cintpt *cornerPts1, int numCornerPts1, Qimage image2, Cintpt *cornerPts2, int numCornerPts2, CMatches **matches, int &numMatches, Qimage &image1Display, Qimage &image2Display)

image1: the first input image,
image2: the second (match from image1 to image2),
cornerPts1: a vector of corner points found in image1,
cornerPts2: a vector of corner points found in image2,
numCornerPts1: the number of corner points in image1,
numCornerPts2: the number of corner points in image2,
matches: a vector of matches; each match has X and Y coordinates from each image

To do this, you'll need to follow these steps:
a. Compute the descriptors for each corner point. This code has already been written for you.
b. For each corner point in image 1, find its best match in image 2. The best match is defined as the closest distance (L1-norm distance. )
c. Add the pair of matching points to "matches".
d. Display the matches using DrawMatches (code is already written. ) You should see many correct and incorrect matches.

Required: Open "Rainier1.png" and "Rainier2.png", compute the Harris corner detector and find matches. Save the images "2a.png" and "2b.png" respectively. (Use DrawMatches to overlay green lines on the images. )

Step 3: (8 pts) Compute the homography between the images using RANSAC (Szeliski, Section 6.1.4). You will need to write:

void RANSAC (Cmatches *matches, int numMatches, int numIterations, double inlierThreshold, double hom[3][3], double homInv[3][3], Qimage &image1Display, Qimage &image2Display)

matches: a set of numMatches matches,
numIterations: the number of times to iterate,
inlierThreshold: a real number so that the distance from a projected point to the match is less than its square,
hom: the homography and homInv its inverse,
Image1Display, Image2Display: images that hold the matches to display.

This function takes a list of potentially matching points between two images and returns the homography transformation that relates them. To do this follow these steps:
a. Iteratively do the following for "numIterations" times: (try 200 on the UI)
          i. Randomly select 4 pairs of potentially matching points from "matches".
          ii. Compute the homography relating the four selected matches with the function "ComputeHomography. "
          iii. Using the computed homography, compute the number of inliers using "ComputeInlierCount".
          iv. If this homography produces the highest number of inliers, store it as the best homography.
b. Given the highest scoring homography, once again find all the inliers. Compute a new refined homography using all of the inliers (not just using four points as you did previously. ) Compute an inverse homography as well (the fourth term of the function ComputeHomography should be false), and return their values in "hom" and "homInv".
c. Display the inlier matches using "DrawMatches".

We also provide two helper functions for you to use:

void Project(double x1, double y1, doube &x2, double &y2, double h[3][3])

This should project point (x1, y1) using the homography h. Return the projected point (x2, y2). See the slides for details on how to project using homogeneous coordinates.

int ComputeInlierCount(double h[3][3], Cmatches *matches, int numMatches, double inlierThreshold).

This is a helper function for RANSAC that computes the number of inlying points given a homography h. That is, project the first point in each match using the function "Project". If the projected point is less than the distance inlierThreshold (default is 5) from the second point, it is an inlier. Return the total number of inliers.

Required: Open "Rainier1.png" with tab "Image 1" selected, and open "Rainier2.png" with tab "Image 2" selected. Compute the Harris corner detector, find matches and run RANSAC. Save the images "3a.png" and "3b.png" of the found matches . (Use DrawMatches to overlay green lines on the images. ) You should only see correct matches, i.e. , all the incorrect matches from the previous step should be removed. If you see all or some incorrect matches try running RANSAC with a larger number of iterations. You may try tuning the other parameters as well.

Step 4: (2 pts) Stitch the images together using the computed homography. To do this, please directly call the Stitch function in the UI.:

Required: Open "Rainier1.png" with tab "Image 1" selected, and open "Rainier2.png" with tab "Image 2" selected. Compute the Harris corner detector, find matches, run RANSAC and stitch the images. Save the stitched image as "4.png". It should look like the image "Stitched.png".

Bells and Whistles (extra credit)
(Whistle = 1 point, Bell = 2 points)

Create a panorama that stitches together the six Mt. Rainier photographs, i.e. , Rainier1.png, ... Painier6.png. The final result should look similar to "AllStitched.png".

Create your own panorama using three or more images. You must capture the images yourself, and not find them on the web. I would recommend downsampling them before stitching, i.e. , make them approximately the same size as the images in the homework assignment.