uw0s home/ thesis/ parcel-tracker/ meddisc/ ideapad15arr-opencore/ deck-builder/

Removing unwanted objects from photos

A web-based implementation of my diploma thesis on semi-automated object removal from photographs [code] [demo].

Method Overview

The method combines computer vision techniques to achieve the goal and consists of five main stages.

Image Alignment

Feature detection is performed using the ORB (Oriented FAST and Rotated BRIEF) algorithm [1]. The original thesis implementation used SURF (Speeded-Up Robust Features) [2].

1. Detect keypoints in both the original and secondary images
2. Match keypoint descriptors between images using a nearest neighbor search
3. Calculate homography matrix from matched keypoint pairs
4. Apply a projective transformation to align the secondary image with the original

The homography matrix for the projective transformation has the form:

$$H=\left[\begin{array}{ccc}{h}_{11}& h_{12}& h_{13}\\ h_{21}& h_{22}& h_{23}\\ h_{31}& h_{32}& 1\end{array}\right]$$

Region Selection

The user defines the Region of Interest (ROI) containing the unwanted object using a polygon selection tool.

Region Segmentation

K-means clustering is applied to segment both:

• ROI: The selected region in the aligned secondary image
• Surrounding Region (SR): The area around the ROI in the original image

Color Harmonization

Each segment in the ROI is matched with the most similar segment in the SR based on color similarity. The matching criterion uses Euclidean distance in RGB color space:

$$d_E=\sqrt{{(R_2-R_1)}^2+{(G_2-G_1)}^2+{(B_2-B_1)}^2}$$

Color correction is then performed using one of two methods:

• Statistical Color Correction: Matches the mean and standard deviation of the color distributions between corresponding segments [3].
• Histogram Matching: Aligns the full color histograms between corresponding segments.

Region Replacement

The color-harmonized segments from the aligned secondary image are used to replace their corresponding segments in the original image.

References

1. E. Rublee, V. Rabaud, K. Konolige and G. Bradski, "ORB: An efficient alternative to SIFT or SURF," 2011 International Conference on Computer Vision, Barcelona, Spain, 2011, pp. 2564-2571. [doi]
2. H. Bay, T. Tuytelaars and L. Van Gool, "SURF: Speeded Up Robust Features," in Computer Vision – ECCV 2006, Berlin, Heidelberg, Springer Berlin Heidelberg, 2006, pp. 404-417. [doi]
3. E. Reinhard, M. Adhikhmin, B. Gooch and P. Shirley, "Color transfer between images," IEEE Computer Graphics and Applications, vol. 21, no. 4, pp. 34-41, 2001. [doi]