mosaic : Automatic 2D Image Mosaics

CIS581 Project at UPenn

Fall 2016

I am fascinated by photographic technology, from the manufacturing of CMOS sensors to the mechanisms of film cameras, so I was particularly excited to have the chance to implement an end-to-end image stitching algorithm that automatically transformed a series of images into a panorama for CIS581: Computer Vision (CV) and Computational Photography.

Automatically creating a panorama from a video is an involved process (see image gallery below for more details):

Find the corners in each image (using a Harris Corner Detector)
Determine the strength of each corner (using MATLAB's cornermetric function)
Downsample to strong, spatially distributed corners (using Adaptive Non-Maximal Suppression, or ANMS)
Build a list of feature descriptors for each image (by selecting a grid around each corner, subsampling, and normalizing)
Match feature descriptors between consecutive images (using kNN search on a k-d tree)
Estimate the homography (by maximizing matched feature inliers via RANSAC)
Refine the homography (using least-squares on those inliers)
Transform each image by that homography
Stitch the images together (by overlaying the corners and blending the pixels)
Recompose the panorama around the center image to reduce warp effects

Step 1-3: Corners detected and downsampled.

Step 4: Feature descriptor built for each corner.

Step 9, 10: Original image stitch (left) and recomposed image stitch (right).

Step 1-3: Corners detected and downsampled.

1/6

I refined my implementation of the mosaic algorithm as I tested it on my own image sequences. For example, I upgraded the ANMS algorithm to find local maxima using a sliding filter and then used a sorted search through those maxima to determine which corners to suppress - these modifications sped up ANMS by 6x and reduced it from ~90% to ~55% of total computation time. I also implemented functionality that allowed me to generate a panorama from a continuous video, and automatically select good frames to stitch together. See below for some examples of panoramas-from-video generated by my mosaic algorithm.

PANORAMA FROM VIDEO EXAMPLE: BIG SUR, CA

PANORAMA FROM VIDEO EXAMPLE: MILLS CANYON, CA