Hands-on: Perspective Correction and Camera Calibration

In computer vision, camera calibration is a crucial process that involves determining the camera's intrinsic and extrinsic parameters and correcting for lens distortions. This allows for accurate 3D measurements and analysis. Perspective correction, on the other hand, rectifies images that have been distorted due to the viewing angle, transforming them into a "bird's eye view." These techniques are fundamental for various applications, including Augmented Reality (AR), robotics, 3D reconstruction, object recognition, and document scanning.

What is Camera Calibration?

Camera calibration is the process of estimating the following properties of a camera:

• Intrinsic Parameters: These describe the internal characteristics of the camera and how it projects 3D points onto a 2D image plane. They include:
  • Focal Length: The distance from the optical center to the image sensor.
  • Optical Center: The principal point (the projection of the camera's optical center onto the image plane).
  • Distortion Coefficients: Parameters that model deviations from ideal pinhole camera geometry, such as radial and tangential lens distortions.
• Extrinsic Parameters: These define the camera's position and orientation in 3D space relative to a world coordinate system. They are represented by a rotation vector and a translation vector.

By understanding these parameters, we can accurately map points between the 3D world and the 2D image, and vice-versa.

Prerequisites

Before you begin, ensure you have OpenCV installed.

pip install opencv-python

Step-by-Step Camera Calibration using a Chessboard Pattern

A common and effective method for camera calibration is to use a precisely manufactured pattern, such as a chessboard. The known dimensions and geometric regularity of the chessboard allow us to reliably detect its corners in images.

Step 1: Prepare Object Points

We need to define the 3D coordinates of the chessboard corners in a real-world coordinate system. For a chessboard of (rows, cols) squares, the 3D points are typically represented in a plane (e.g., Z=0).

import numpy as np
import cv2
import glob

# Define the dimensions of the chessboard (number of inner corners)
chessboard_size = (9, 6) # For a 9x6 grid of squares, there are 9x6 inner corners

# Prepare object points. In a real world, these are at some known distance.
# For simplicity, we assume the chessboard is on the XY plane (Z=0).
# The points will be (0,0,0), (1,0,0), ..., (8,5,0) in arbitrary units.
objp = np.zeros((np.prod(chessboard_size), 3), np.float32)
# objp[:, :2] assigns coordinates like (0,0), (1,0), ..., (8,0), (0,1), ... to the first two columns.
objp[:, :2] = np.indices(chessboard_size).T.reshape(-1, 2)

# Lists to store object points and image points from each calibration image
objpoints = []  # 3D points in real world space
imgpoints = []  # 2D points in image plane

Step 2: Read Images and Detect Corners

Capture multiple images of the chessboard from different angles and distances. For each image, convert it to grayscale and attempt to find the chessboard corners.

# Load calibration images
# Ensure you have a folder named 'calib_images' containing your chessboard images (.jpg)
images = glob.glob('calib_images/*.jpg')

for fname in images:
    img = cv2.imread(fname)
    if img is None:
        print(f"Error: Could not read image {fname}")
        continue

    gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

    # Find the chessboard corners
    # ret: True if corners were found, False otherwise
    # corners: array of detected corner points
    ret, corners = cv2.findChessboardCorners(gray, chessboard_size, None)

    # If corners are found, refine them and store
    if ret:
        objpoints.append(objp)
        # Refine corner locations for sub-pixel accuracy
        corners = cv2.cornerSubPix(gray, corners, (11, 11), (-1, -1), (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 30, 0.001))

        imgpoints.append(corners)

        # Draw and display the detected corners on the image
        cv2.drawChessboardCorners(img, chessboard_size, corners, ret)
        cv2.imshow('Corners', img)
        cv2.waitKey(100) # Wait for 100ms

cv2.destroyAllWindows()

Step 3: Calibrate the Camera

Once you have collected sufficient data (multiple images with detected corners), you can calibrate the camera using the cv2.calibrateCamera() function. This function uses the 3D object points and their corresponding 2D image points to estimate the camera's intrinsic and extrinsic parameters.

# Calibrate the camera
# gray.shape[::-1] provides the image dimensions (height, width) in reverse order
# None, None are placeholders for initial guesses for camera_matrix and dist_coeffs
ret_calib, camera_matrix, dist_coeffs, rvecs, tvecs = cv2.calibrateCamera(
    objpoints, imgpoints, gray.shape[::-1], None, None
)

The outputs are:

• ret_calib: Reprojection error. A lower value indicates better calibration.
• camera_matrix: The intrinsic camera matrix (K).
• dist_coeffs: The distortion coefficients (k1, k2, p1, p2, k3, ...).
• rvecs: Rotation vectors for each image (extrinsic parameters).
• tvecs: Translation vectors for each image (extrinsic parameters).

Step 4: Save and Use Calibration Results

It's good practice to save the calibrated camera parameters for later use.

# Print the calibration results
print("Camera Matrix:\n", camera_matrix)
print("\nDistortion Coefficients:\n", dist_coeffs)

# Save the calibration data to a file
# This can be loaded later to undistort images without re-calibrating
np.savez("calib_data.npz", camera_matrix=camera_matrix, dist_coeffs=dist_coeffs)
print("\nCalibration data saved to calib_data.npz")

Step 5: Undistort an Image

To correct the lens distortion in a new image, you can use the saved calibration parameters. cv2.undistort() applies the calculated distortion correction. cv2.getOptimalNewCameraMatrix() can be used to refine the camera matrix, potentially cropping the image to remove black borders resulting from the undistortion.

# Load a test image for undistortion
img_test = cv2.imread('calib_images/test.jpg') # Replace with a path to an image from your set
if img_test is None:
    print("Error: Could not read test image for undistortion.")
else:
    h, w = img_test.shape[:2]

    # Get the optimal new camera matrix
    # The alpha parameter (1 in this case) controls the scaling of the output image.
    # alpha=0 means all pixels are kept. alpha=1 means as many pixels as possible are kept.
    new_camera_matrix, roi = cv2.getOptimalNewCameraMatrix(
        camera_matrix, dist_coeffs, (w, h), 1, (w, h)
    )

    # Undistort the image
    undistorted = cv2.undistort(img_test, camera_matrix, dist_coeffs, None, new_camera_matrix)

    # Display the original and undistorted images
    cv2.imshow('Original Image', img_test)
    cv2.imshow('Undistorted Image', undistorted)

    cv2.waitKey(0)
    cv2.destroyAllWindows()

Perspective Correction (Bird's Eye View)

Perspective correction is used to transform an image so that the perspective distortion is removed, making parallel lines appear parallel and right angles appear as right angles. This is often used to convert images of documents, whiteboards, or road surfaces into a flat, top-down view.

The process involves identifying four points in the source image (defining the quadrilateral that needs to be straightened) and mapping them to corresponding points in the destination image (typically a rectangle).

Step 1: Define Source and Destination Points

You need to manually identify or programmatically detect four corner points of the object/region in the source image that you want to rectify. These points define the perspective transformation. The destination points define the shape and size of the output image.

import numpy as np
import cv2

# Load an image for perspective correction
# Assume 'perspective.jpg' contains an image with perspective distortion
img_persp = cv2.imread('perspective.jpg')
if img_persp is None:
    print("Error: Could not read perspective image.")
else:
    h_persp, w_persp = img_persp.shape[:2]

    # Define the source points (corners of the object in the original image)
    # These are in the format [[x1, y1], [x2, y2], [x3, y3], [x4, y4]]
    # The order is important: typically top-left, top-right, bottom-left, bottom-right.
    pts_src = np.float32([[100, 200], [400, 200], [100, 500], [400, 500]])

    # Define the destination points (corners of the desired output image)
    # These define the rectangle into which the source quadrilateral will be transformed.
    # The width and height should ideally match the aspect ratio of the original quadrilateral.
    pts_dst = np.float32([[0, 0], [300, 0], [0, 400], [300, 400]])

    # Display the source points on the image to verify
    img_display_pts = img_persp.copy()
    for pt in pts_src:
        cv2.circle(img_display_pts, (int(pt[0]), int(pt[1])), 5, (0, 0, 255), -1) # Red circles
    cv2.imshow("Source Points", img_display_pts)

Step 2: Apply Perspective Transform

Use cv2.getPerspectiveTransform() to compute the 3x3 transformation matrix based on the source and destination points. Then, use cv2.warpPerspective() to apply this matrix to the image, creating the perspective-corrected (warped) image.

    # Calculate the perspective transformation matrix
    matrix = cv2.getPerspectiveTransform(pts_src, pts_dst)

    # Apply the perspective transformation
    # The third argument (300, 400) specifies the size of the output image
    warped = cv2.warpPerspective(img_persp, matrix, (300, 400))

    # Display the original and warped images
    cv2.imshow('Original Image (Perspective)', img_persp)
    cv2.imshow('Warped Image (Bird\'s Eye View)', warped)

    cv2.waitKey(0)
    cv2.destroyAllWindows()

Summary of Key OpenCV Functions

SEO Keywords

Camera calibration OpenCV tutorial, Lens distortion correction, Intrinsic and extrinsic camera parameters, Chessboard calibration OpenCV, cv2.calibrateCamera example, Image undistortion with OpenCV, Perspective correction OpenCV, Bird's eye view transformation, cv2.getPerspectiveTransform usage, Camera matrix and distortion coefficients.

Frequently Asked Questions (FAQ)

What is camera calibration and why is it important in computer vision?

Camera calibration is the process of estimating a camera's internal properties (intrinsic parameters) and its pose (extrinsic parameters) in the world. It's vital for tasks requiring accurate 3D information, such as 3D reconstruction, stereo vision, object pose estimation, and augmented reality, as it corrects for distortions and provides a precise mapping between 3D world coordinates and 2D image coordinates.

What are intrinsic and extrinsic camera parameters?

• Intrinsic parameters relate to the camera's internal geometry: focal length, optical center, and lens distortion coefficients. They describe how a camera maps 3D points to 2D image points, independent of the camera's position in space.
• Extrinsic parameters describe the camera's position and orientation in 3D space relative to a world coordinate system. They are represented by a rotation vector and a translation vector.

How does the chessboard pattern help in camera calibration?

A chessboard pattern has a grid of squares with known dimensions and perfect right angles at the corners. This geometric regularity allows for reliable detection of its corners in an image, even with slight distortions. By knowing the precise 3D locations of these corners in a world coordinate system and their corresponding 2D locations in the image, we can solve for the camera's intrinsic and extrinsic parameters.

What is the role of distortion coefficients in camera calibration?

Distortion coefficients (like radial distortion k1, k2, k3 and tangential distortion p1, p2) model the non-linear deviations of a real lens from the ideal pinhole camera model. Radial distortion causes straight lines to appear curved (barrel or pincushion distortion), while tangential distortion occurs when the lens is not perfectly perpendicular to the image sensor. These coefficients are essential for accurately undistorting images.

How do you use OpenCV’s cv2.calibrateCamera function?

You call cv2.calibrateCamera() with lists of 3D object points (objpoints) and their corresponding 2D image points (imgpoints) collected from multiple calibration images. You also provide the shape of the calibration images. The function returns the camera matrix, distortion coefficients, and the rotation and translation vectors for each image.

How can you undistort an image using camera calibration results?

After calibrating, you obtain the camera_matrix and dist_coeffs. You can then use cv2.undistort(image, camera_matrix, dist_coeffs, None, new_camera_matrix) to remove the lens distortion from any new image. Optionally, cv2.getOptimalNewCameraMatrix() can be used to refine the camera matrix for a cleaner output.

What is perspective correction and how is it applied?

Perspective correction is the process of transforming an image to remove distortions caused by the camera's viewing angle, resulting in a "bird's eye view." It's applied by defining four source points in the original image that outline the area to be corrected and four destination points that define the desired rectangular output. A perspective transform matrix is computed and then applied using cv2.warpPerspective().

How do cv2.getPerspectiveTransform and cv2.warpPerspective work?

cv2.getPerspectiveTransform(src_pts, dst_pts) calculates a 3x3 affine transformation matrix that maps the points in src_pts to dst_pts. cv2.warpPerspective(image, matrix, output_size) then uses this matrix to warp the input image into a new image of output_size, effectively transforming the perspective.

What are practical applications of camera calibration and perspective correction?

• Camera Calibration: 3D reconstruction, augmented reality, stereo vision (depth perception), robot navigation and localization, 3D object measurement, virtual reality.
• Perspective Correction: Document scanning and digitization, license plate recognition, self-driving car perception (understanding road geometry), whiteboard digitization, image stitching.

How do you choose source and destination points for perspective transforms?

Source Points: Identify the four corners of the object or region in the image that you want to straighten. This often requires manual selection or automated feature detection (e.g., detecting the corners of a document or a whiteboard). The order of these points (e.g., top-left, top-right, bottom-right, bottom-left) is crucial. Destination Points: Define the desired shape and size of the output image. These points typically form a rectangle. The dimensions of this rectangle should correspond to the expected aspect ratio of the straightened object. For example, if straightening a document, the destination points would form a perfect rectangle with the expected width and height of the document.