STUDY DOC
Panorama Stitching (RANSAC based)
1. Imports & Config:
•
•
•
•
•
•
import cv2 This line imports the OpenCV (Open Source Computer Vision Library), which is
essential for computer vision tasks such as reading and writing images, manipulating pixel
data, and performing various image processing operations.
import pandas as pd Pandas is imported for data manipulation and analysis. It's used to
work with structured data, like tables, which can be useful for organizing image metadata or
storing features extracted from images.
import numpy as np NumPy is fundamental for numerical operations in Python. It's used to
handle arrays and matrices, which is how images are represented in a computer as a grid of
pixel values.
import matplotlib.pyplot as plt This library is used for creating static, animated, or
interactive visualizations. In this context, it is likely used to display images, plot results of data
analysis, or visualize machine learning model performance.
from sklearn.linear_model import RANSACRegressor, LinearRegression This line
imports two specific machine learning models from scikit-learn:
o LinearRegression is a basic model used for linear regression, a common technique
for modeling the relationship between a dependent variable and one or more
independent variables.
o RANSACRegressor (RANdom SAmple Consensus) is a robust regression model
that is particularly useful for fitting data that contains a significant number of outliers,
which can be common in computer vision applications.
import os The os module provides a way to interact with the operating system. It's often used
to manage file paths, read and write files, and navigate directory structures, which is
necessary for loading and saving images or other project-related files.
2. I/O & Preprocessing
•
Data Capture:
o Equipment used: Nikon Coolpix P900, Tripod.
o Subject: Building view from student’s apartment.
•
Methodology:
o Click images from left to right frame, keeping 30-60% overlap.
•
Number of images captured:
o 11 images
1|Page
o
•
Images saved in Google Drive for reading through Google Colab
Data Import and Visualization:
o
Library Imports and Image Loading
▪ Imports necessary libraries: The code imports cv2 for image processing,
pandas and numpy for data handling, matplotlib.pyplot for plotting, and os for
file system operations.
▪ Loads images from a folder: A function named load_images_from_folder
reads image files from a specified directory and stores them in a dictionary.
o
Image Visualization
▪ Creates a gallery: The code sets up a grid (4 columns wide) to display
multiple images.
▪ Iterates and displays: It loops through the loaded images, converts them from
OpenCV's BGR color format to matplotlib's RGB, and displays each image in
a subplot with its filename as the title.
▪ Enhances display: The code hides the axes and adjusts the layout to ensure
a clean, well-organized presentation of all the images.
2|Page
•
Preprocessing:
3|Page
4|Page
o
o
•
The preprocess_image function takes an image and optional camera parameters
as input. It performs three main steps:
▪ Resizing: The function resizes the input image to a consistent size, in this
case, 640x480 pixels. This step is crucial for standardizing images before
they are fed into a computer vision model.
▪ Undistortion: It corrects for lens distortion, a common issue in photography
where straight lines appear curved. This step is conditional; it only runs if a
camera matrix (which describes the camera's internal properties) and
distortion coefficients are provided.
▪ Grayscale Conversion: The function converts the image from color (BGR
format) to grayscale. This simplifies the image data from three channels to
one, which can be computationally more efficient for many computer vision
tasks.
Processing and Visualization
▪ The code then processes all images stored in the image_dict by calling the
preprocess_image function on each one and storing the results in a new
dictionary, processed_image_dict.
▪ Finally, it creates a gallery view to display the processed images using
matplotlib. It plots the grayscale images in a grid, with each subplot showing
a single image and its corresponding filename as the title. This allows for a
quick visual inspection of the preprocessing results.
Feature Extraction
o SIFT Keypoint Detection
This is the core of the code's computer vision task. It uses the SIFT (Scale-Invariant Feature
Transform) algorithm, a powerful method for finding unique and distinct points in an image
that are invariant to scale, rotation, and lighting changes.
▪ A SIFT detector is initialized with cv2.SIFT_create().
▪ A loop iterates through the preprocessed images to detect keypoints and
compute their descriptors using sift.detectAndCompute(). Keypoints are the
locations of these unique features, while descriptors are numerical
representations of the features.
▪ The code then draws these detected keypoints onto the images to visualize
them.
5|Page
o
Result/Visualization
Dense Feature Coverage: The SIFT algorithm successfully found a large
number of keypoints across the entire image, particularly concentrated on the
high-contrast structural elements of the buildings.
▪ Scale and Orientation Invariance: The detected keypoints (indicated by
circles of varying sizes and radiating lines) demonstrate SIFT's ability to find
features that are robust to changes in scale and rotation, a core strength of
the algorithm.
▪ Focus on Structural Details: The majority of the keypoints are concentrated
on stable features like window frames, building corners, and balcony railings.
These are ideal for SIFT because they are less susceptible to change than
textures or lighting.
▪ Consistency Across Images: SIFT appears to have identified similar types of
features across the different images, which is essential for applications like
image matching or stitching multiple images together.
▪ Robustness to Repetitive Patterns: Despite the repetitive nature of the
windows and balconies, SIFT has successfully identified unique features,
indicating its effectiveness in distinguishing between similar visual elements
within a complex scene.
Key Trade-off
▪ Computational cost versus feature robustness.
▪ The SIFT algorithm is highly effective at finding features that are invariant to
scale, rotation, and lighting changes. However, this robustness comes at a
high computational cost.
▪ Detecting and computing descriptors for a large number of keypoints in highresolution images can be slow, making real-time applications challenging.
▪
o
6|Page
▪
•
The code's explicit preprocessing steps (resizing and converting to grayscale)
aim to mitigate this by reducing the image's data size, improving processing
speed at the potential cost of losing some fine-grained detail.
Matching
o
Matcher Type: The code uses cv2.BFMatcher(), which stands for Brute-Force
Matcher. This is a straightforward matcher that compares the descriptor of a feature
from the first image with all descriptors from the second image to find the best match.
7|Page
o
o
o
o
Metric: The metric used by the Brute-Force Matcher is the Euclidean distance (L2
norm) by default. It measures the "distance" between two descriptors, with a smaller
distance indicating a better, more similar match.
k-NN Ratio and Cross-Check:
▪ k-NN (k-Nearest Neighbors): The code uses bf.knnMatch(descriptors1,
descriptors2, k=2). This finds the two nearest neighbors for each descriptor
from the first image in the second image's set of descriptors.
▪ Cross-Check: A true cross-check, which would ensure that the best match for
A in B is also the best match for B in A, is not explicitly implemented. Instead,
the code relies on the Lowe's ratio test as a filtering mechanism.
Outliers and Ratio Test Effect:
▪ Outliers: Outliers are incorrect matches that arise from visually similar but
unrelated features, or from noise. The ratio test is designed to filter these out.
▪ Ratio Test: The code applies Lowe's ratio test (if m.distance < 0.75 *
n.distance). This is a powerful technique for removing outliers. It compares
the distance to the best match (m.distance) with the distance to the secondbest match (n.distance). If the ratio is below a certain threshold (e.g., 0.75), it
means the best match is significantly better than the next best, and is likely a
correct match. Matches with a high ratio are discarded as potential outliers.
Visualizations/Results:
8|Page
9|Page
▪
▪
▪
High Number of Matches: The code successfully found a high number of
corresponding SIFT keypoints between most of the adjacent image pairs, as
indicated by the numerous lines connecting the images. This suggests the
images have significant overlap and that SIFT is a robust algorithm for this
type of scene.
Concentration of Matches: The majority of the good matches are clustered on
the stable, high-contrast features of the buildings, such as window frames
and edges. This is expected, as these are the most reliable features for SIFT
to detect and match.
Visible Mismatches (Outliers): Despite the use of the ratio test, some
incorrect matches (outliers) are still visible. These are the lines that cross
over or connect to visually unrelated parts of the two images. For example,
some lines connect features on the buildings to seemingly random points in
the sky, indicating a mismatch.
3. RANSAC Homography – normalization, thresholds, confidence; plotting inlier mask.
•
RANSAC: The code uses RANSAC (RANdom SAmple Consensus), an iterative method to
estimate the homography matrix. It works by randomly selecting minimal subsets of four matches,
computing a homography from them, and then checking how many other matches "agree" with
this model. This process is repeated to find the model that fits the most matches, effectively
discarding a majority of incorrect matches (outliers) that would otherwise skew the result.
• Thresholds: The ransacReprojThreshold is a key parameter set to 5.0. This value defines the
maximum allowed distance, in pixels, between a transformed source point and its corresponding
destination point for a match to be considered an inlier. A match is an inlier if its reprojection error
is below this threshold, meaning it fits the estimated homography model well.
• Confidence: The confidence parameter, though commented out, is typically set to a high value
(e.g., 0.995). It represents the desired probability that the final estimated homography is free from
outliers. A higher confidence level may result in more iterations to find a better model.
• Normalization: The code mentions that point normalization is often handled internally by
cv2.findHomography. Normalization is a preprocessing step that centers the points and scales
them to a consistent range. This is done to improve the numerical stability of the homography
estimation, ensuring the algorithm converges to a more accurate solution.
• Inlier Mask Plotting
The core function is cv2.drawMatches, which now uses a matchesMask to control the color of each
match line.
o The Mask: The mask is a list of 0s and 1s, generated by the cv2.findHomography
function. A value of 1 indicates an inlier (a good match that fits the homography
model), and a value of 0 indicates an outlier (a bad match that does not).
o Color-Coding: drawMatches is configured to draw inliers in green (matchColor=(0,
255, 0)) and outliers in red (singlePointColor=(0, 0, 255)). This visual distinction
provides a clear way to see how effective RANSAC was at filtering out bad matches.
10 | P a g e
11 | P a g e
•
Visualization/Results:
o
o
o
High Number of Matches and Inliers: The results consistently show a large number of
matches, ranging from 486 to 569. The inlier percentage is also very high, generally
above 85% and often around 90%, indicating that a significant majority of the found
matches are geometrically consistent and correct. This suggests the algorithm is
robust and the images are well-suited for this type of feature matching.
Consistent Feature Location: The green lines representing the matches are primarily
concentrated on the structural elements of the buildings, such as windows, balconies,
and the corners of the walls. This indicates that the algorithm is successfully
identifying and matching salient and consistent features across different images. The
repetitive patterns of the building's facade are likely providing many strong, reliable
feature points.
Low Root Mean Square Error (RMSE): The RMSE values are consistently low,
ranging from 0.39 to 0.51. A low RMSE signifies that the geometric transformation
12 | P a g e
(e.g., a homography or fundamental matrix) calculated from the matched points has a
very small reprojection error. In simple terms, it means the matched points align very
well with each other in 3D space, which further confirms the accuracy and quality of
the matches.
4. Reference selection: Pick the frame with the highest total inliers as the reference.
13 | P a g e
5. Composition:
• Global Homography Composition: The code calculates a global homography for each
image relative to a designated reference frame. It does this by composing the pairwise
homographies (H_pair) sequentially. For images before the reference, it iterates
backward and multiplies H_pair to accumulate the transformation. For images after the
reference, it iterates forward but uses the inverse of the previously accumulated global
homography to calculate the new one. This ensures all images are transformed to a
single, consistent coordinate system.
• Canvas Size and Bounding Box Calculation: To determine the final panorama's
dimensions, the code warps the four corners of every image using its calculated global
homography. By finding the minimum and maximum x and y coordinates across all these
warped corners, it establishes the global bounding box of the stitched image. A margin is
then added to this bounding box to create the final panorama_width and
panorama_height.
• Coordinate System Translation: The calculated bounding box can have negative
coordinates. To ensure all image content is visible on the final canvas, a translation
matrix is created. This matrix shifts the entire coordinate system so that the top-left corner
of the bounding box (i.e., min_x, min_y) is moved to (0, 0). This translation matrix is then
composed with each of the global homographies to create a new set of translated global
homographies (global_H_matrices_translated).
• Preparing for Final Warping: The final result of this code is a dictionary of
global_H_matrices_translated and the final panorama_width and panorama_height.
These are the essential inputs for the next step of the panorama stitching process, where
each image will be warped and blended onto the final panorama canvas. This preparation
ensures that all images are correctly positioned within a single, positive-coordinate
canvas, preventing image clipping.
14 | P a g e
15 | P a g e
16 | P a g e
6. Warp & Blend:
•
•
•
•
•
Image Warping and Masking: The code uses cv2.warpPerspective to transform each original
image onto a single, large canvas based on its pre-calculated global homography. This is the
core step that aligns all images. A binary mask is created for each warped image, identifying
its non-black pixels. These masks are crucial for the subsequent blending and exposure
compensation steps.
Feather Blending for Seamless Transitions: Instead of simply pasting images, the code
implements a basic feather blending technique. It iterates through all warped images,
accumulating their pixel values into a blended_panorama array and their corresponding
normalized mask values into an accumulator. By dividing the blended_panorama by the
accumulator, the code creates a weighted average of the overlapping regions, which smooths
out seams and color differences between images.
Multi-Band Blending for Better Quality Transitions: It decomposes images into frequency
bands using Laplacian pyramids and blends each band separately. This technique effectively
handles misalignments and seams across different scales, allowing for smoother and more
natural-looking transitions in the final panorama, particularly for fine details and textures,
which are contained in the higher-frequency bands.
Basic Exposure Compensation: The code attempts to correct for lighting differences between
images by applying a simple exposure compensation. It calculates the average intensity of
the warped reference frame and the overall blended panorama. The difference between these
averages is then added to every pixel of the panorama, adjusting its brightness to match the
reference image. This is a basic but effective way to improve the visual consistency of the
final panorama.
Final Cropping and Visualization: After blending and exposure compensation, the stitched
image often has black borders due to the warping process. The code finds the non-zero pixels
(the actual image content) in the panorama and uses cv2.boundingRect to determine the
tightest possible rectangle around them. It then crops the image to this bounding box,
removing the empty black space. The final cropped panorama is then displayed using
matplotlib.
17 | P a g e
18 | P a g e
19 | P a g e
Feather Blending output:
Multi-Bland Blending output:
7. Diagnostics:
20 | P a g e
As we see there are lot of black pixels from panorama canvas. While the panorama image is
generated, these black pixels also need to be diagnosed and addressed through appropriate
cropping.
We track points where width of non-zero pixels are minimum to find the appropriate cropping box
dimensions.
• Finding Non-Black Pixels: The first step is to convert the blended_panorama to grayscale and
then use cv2.findNonZero() to locate all pixels that contain actual image data. These pixels
are represented as a list of coordinates, effectively defining the shape of the stitched image
content within the larger canvas.
• Standard Bounding Box Cropping: The code correctly uses cv2.boundingRect() on the coords
to find the smallest upright rectangle that encloses all the image data. This is the standard
and correct approach for cropping a warped image to its minimal size. The coordinates (x, y)
and dimensions (w, h) of this bounding box are then used to crop the blended_panorama,
removing the black borders.
• Flawed Custom Cropping Logic: A significant part of the code introduces custom logic to find
a "minimal" crop. It iterates through the unique x-coordinates of the image data, finds the
minimum and maximum y-coordinates for each x, and calculates the difference (diff). The
goal is to find the x-column that has the smallest vertical span (diff). This approach is incorrect
because it only considers a single vertical line and doesn't guarantee the best overall crop.
The results from this flawed logic (dtt['x'], dtt['y_min'], dtt['y_max'], and dtt['diff']) are then used
to crop the image, overriding the correct bounding box found by cv2.boundingRect().
•
Display and Visualization: The final part of the code visualizes the cropped panorama using
matplotlib. It displays the result with a title indicating that it has been cropped to the "bounding
21 | P a g e
box of non-zero pixels," which is somewhat misleading given the custom and incorrect logic
applied for the final crop.
o
o
Feature Blended Cropped Panorama
Multi-Band Blended Cropped Panorama
22 | P a g e