Geometric Camera Calibration

Plan

1. Review Perspective Projection: Understand the mathematical model of how cameras form images.
2. Geometric Camera Calibration: Determine the camera’s parameters.
   • Indirect Calibration: First solve for the overall projection matrix, then extract individual parameters.
   • Direct Calibration: Solve for parameters more directly (often involving non-linear methods or specific assumptions).
   • Multi-planes Method (Zhang’s Method): A popular technique using a planar calibration target.
   • Example: Using the Matlab Camera Calibration Toolbox.
3. Catadioptric Sensing: (Briefly mentioned, covered in a different slide deck - e.g., using mirrors).
4. Other Methods (Not Covered in Detail):
   • Vanishing points-based method.
   • Self-calibration (using only image correspondences without a known target).

Review: Perspective Projection Model

Camera Parameters

A camera’s imaging process is modeled by several parameters:

• Extrinsic Parameters: Define the camera’s position and orientation in the 3D world.
  • Rotation ($\mathbf{R}$): A 3x3 rotation matrix describing the camera’s orientation relative to the world coordinate system.
  • Translation ($\mathbf{T}$): A 3x1 vector describing the position of the camera’s optical center relative to the world origin.
• Intrinsic Parameters: Define the camera’s internal optical and geometric properties. Encoded in the Camera Matrix ($\mathbf{K}$).
  • Focal Length ($f$): Distance from the optical center to the image plane. Often expressed in pixel units ($\alpha =f{s}_x$, $\beta =f{s}_y$).
  • Principal Point ($c_x,c_y$ or $u_0,v_0$): The point where the optical axis intersects the image plane, usually near the image center. Coordinates are in pixels.
  • Pixel Size ($s_x,s_y$): Physical size of a pixel (e.g., mm/pixel). Incorporated into $\alpha ,\beta$.
  • Skew Angle ($\theta$): Angle between the image sensor’s x and y axes. Usually close to 90 degrees, making the skew factor ($s=-\alpha \cot \theta$) close to zero for modern cameras.

Important Term: Intrinsics vs. Extrinsics: Intrinsics describe the camera itself; Extrinsics describe its pose in the world.

(Note: Parameter definitions, especially intrinsics, can vary slightly between texts/toolboxes.)

Projection Equation

The relationship between a 3D world point ${\mathbf{X}}_w=[X_w,Y_w,Z_w{]}^T$ and its corresponding 2D image point $\mathbf{x}=[u,v{]}^T$ is described by the perspective projection equation. Using homogeneous coordinates:

• 3D world point: ${\mathbf{P}}_w=[X_w,Y_w,Z_w,1{]}^T$
• 2D image point: $\mathbf{p}=[u,v,1{]}^T$

The projection is given by:

$$\lambda \mathbf{p}=\mathbf{K}[\mathbf{R}|\mathbf{T}]{\mathbf{P}}_w$$

Where:

• $\lambda$ is a scale factor (depth).
• $\mathbf{K}$ is the $3\times 3$ intrinsic matrix.
• $[\mathbf{R}|\mathbf{T}]$ is the $3\times 4$ extrinsic matrix (combining rotation $\mathbf{R}$ and translation $\mathbf{T}$).
• The combined $3\times 4$ matrix $\mathbf{M}=\mathbf{K}[\mathbf{R}|\mathbf{T}]$ is called the Projection Matrix.

The Intrinsic Matrix (K)

A common form for the intrinsic matrix $\mathbf{K}$ is:

$$\mathbf{K}=\left[\begin{array}{ccc}\alpha & s& u_0\\ 0& \beta & v_0\\ 0& 0& 1\end{array}\right]$$

Where:

• $\alpha =f\cdot k$ (focal length in x-pixel units)
• $\beta =f\cdot l$ (focal length in y-pixel units)
• $s=-\alpha \cot \theta$ (skew parameter, often 0)
• $(u_0,v_0)$ is the principal point.

$\mathbf{K}$ has 5 degrees of freedom (DoF): $\alpha ,\beta ,u_0,v_0,s$. If skew is zero and pixels are square ($\alpha =\beta$), only 3 DoF remain.

Normalized Image Plane

Conceptually, the projection can be seen as first projecting the 3D point onto a “normalized” image plane located at $f=1$. The coordinates on this plane are $(\hat{X},\hat{Y})=(X/Z,Y/Z)$. The intrinsic matrix $\mathbf{K}$ then maps these normalized coordinates to pixel coordinates, accounting for focal length, principal point, pixel size, and skew.

Properties of Perspective Projection

• Distant objects appear smaller.
• Points project to points.
• Lines project to lines.
• Vanishing Points: Projections of points at infinity. Parallel lines in 3D appear to converge at a vanishing point in the image.
• Angles are not preserved.
• Parallel lines in 3D are not generally parallel in the image (they meet at a vanishing point).

Geometric Camera Calibration

Goal: To estimate the intrinsic ($\mathbf{K}$) and extrinsic ($[\mathbf{R}|\mathbf{T}]$) parameters of a camera. Equivalently, estimate the $3\times 4$ projection matrix $\mathbf{M}$ and decompose it.

The Calibration Problem

• Input: A set of $n$ known 3D points ${\mathbf{P}}_{w,i}$ (e.g., corners on a calibration rig/checkerboard) and their corresponding measured 2D image projections ${\mathbf{p}}_i$.
• Output: The intrinsic matrix $\mathbf{K}$ and the extrinsic matrix $[\mathbf{R}|\mathbf{T}]$.

How many points are needed?

• The projection matrix $\mathbf{M}$ has 12 entries but is defined only up to an overall scale factor, meaning it has 11 degrees of freedom (DoF).
• Each 3D-to-2D point correspondence provides 2 linear constraints on the entries of $\mathbf{M}$.
• Therefore, we need at least $11/2=5.5$ points. Since we need a whole number of points, we need at least 6 points (providing 12 constraints) to solve linearly for $\mathbf{M}$.
• Exam Tip: Remember that $\mathbf{M}$ has 11 DoF and requires at least 6 point correspondences for a unique solution (assuming points are in general position).

Indirect Calibration (DLT Approach)

This approach first solves for the projection matrix $\mathbf{M}$ and then decomposes it into $\mathbf{K}$, $\mathbf{R}$, and $\mathbf{T}$.

1. Formulate Linear Equations: Start with the projection equation $\lambda {\mathbf{p}}_i=\mathbf{M}{\mathbf{P}}_{w,i}$. Let ${\mathbf{p}}_i=[u_i,v_i,1{]}^T$ and $\mathbf{M}=\left[\begin{array}{c}{\mathbf{m}}_1^T\\ {\mathbf{m}}_2^T\\ {\mathbf{m}}_3^T\end{array}\right]$, where ${\mathbf{m}}_j$ are the rows of $\mathbf{M}$. Writing out the components: $\lambda u_i={\mathbf{m}}_1^T{\mathbf{P}}_{w,i}$ $\lambda v_i={\mathbf{m}}_2^T{\mathbf{P}}_{w,i}$ $\lambda ={\mathbf{m}}_3^T{\mathbf{P}}_{w,i}$ Eliminate $\lambda$ to get two linear equations in the elements of $\mathbf{M}$ for each point $i$: $u_i({\mathbf{m}}_3^T{\mathbf{P}}_{w,i})-({\mathbf{m}}_1^T{\mathbf{P}}_{w,i})=0$ $v_i({\mathbf{m}}_3^T{\mathbf{P}}_{w,i})-({\mathbf{m}}_2^T{\mathbf{P}}_{w,i})=0$

2. Set up Homogeneous Linear System: Stack the equations from $n\ge 6$ points into a single matrix equation:

   $$\mathcal{P}\mathbf{m}=0$$

   Where:

   • $\mathcal{P}$ is a $2n\times 12$ matrix constructed from the known ${\mathbf{P}}_{w,i}$ and measured $u_i,v_i$.

   Structure of P for one point i (two rows) $P_i$ = [$X_i$, $Y_i$, $Z_i$, $1$] $ro{w}_1$ = [ $0^T$, $-P_i^T$, $v_i\ast P_i^T$ ] $ro{w}_2$ = [ $P_i^T$, $0^T$, $-u_i\ast P_i^T$ ] P stacks row1, row2 for all i

   • $\mathbf{m}$ is a $12\times 1$ vector containing the elements of $\mathbf{M}$ (e.g., stacked rows or columns).

3. Solve using SVD: This is a homogeneous linear system ($A\mathbf{x}=0$). The solution $\mathbf{m}$ that minimizes $||\mathcal{P}\mathbf{m}|{|}^2$ subject to $||\mathbf{m}|{|}^2=1$ is found using Singular Value Decomposition (SVD). Compute the SVD of $\mathcal{P}$: $\mathcal{P}=\mathbf{UD}{\mathbf{V}}^T$. The solution $\mathbf{m}$ is the last column of $\mathbf{V}$, which corresponds to the smallest singular value of $\mathcal{P}$. Important Term: DLT (Direct Linear Transformation): This method of solving for transformations (like $\mathbf{M}$) by setting up and solving $\mathcal{P}\mathbf{m}=0$ using SVD is known as DLT.

4. Extract Parameters from M: Once $\mathbf{m}$ is found, reshape it into the $3\times 4$ matrix $\mathbf{M}$. We know $\mathbf{M}=\rho [\mathbf{A}|\mathbf{b}]=\rho \mathbf{K}[\mathbf{R}|\mathbf{T}]$, where $\rho$ is an unknown scale factor.

   • Decompose $\mathbf{A}=\rho \mathbf{KR}$. This can be done using RQ decomposition (similar to QR). Since $\mathbf{K}$ is upper triangular and $\mathbf{R}$ is orthogonal, RQ decomposition finds these factors.
   • The scale factor $\rho$ can be determined (e.g., by requiring $K_{33}=1$).
   • The translation is then $\mathbf{T}=\frac{1}{\rho }{\mathbf{K}}^{-1}\mathbf{b}$.
   • Intrinsic Parameters: Extracted from $\mathbf{K}$.
   • Extrinsic Parameters: $\mathbf{R}$ and $\mathbf{T}$.
   • Faugeras’ Theorem (1993): Provides conditions on $\mathbf{M}$ (specifically on its $3\times 3$ left part $\mathbf{A}$) for it to be a valid perspective projection matrix (e.g., $\det (\mathbf{A})\ne 0$) and for specific constraints like zero-skew.

5. Degenerate Cases: The basic DLT method requires the 3D points ${\mathbf{P}}_{w,i}$ to be non-coplanar. If they are coplanar, the solution for $\mathbf{M}$ is not unique. Special methods (like Zhang’s) handle planar targets.

Handling Lens Distortion

Real lenses introduce distortions not modeled by the ideal pinhole model.

• Types:
  • Radial Distortion: Most significant, causes straight lines to appear curved, especially near image edges. (Barrel distortion: magnification decreases with distance from center; Pincushion distortion: magnification increases).
  • Spherical Aberration, Chromatic Aberration (less commonly modeled in basic calibration).
• Modeling: Radial distortion is often modeled by adjusting the image coordinates $(u,v)$ based on their distance $r=\sqrt{(u-u_0{)}^2+(v-v_0{)}^2}$ from the principal point. Let $(u_d,v_d)$ be the distorted (measured) coordinates and $(u_u,v_u)$ be the undistorted (ideal pinhole) coordinates. $$\begin{gathered} u_u=u_d+(u_d-u_0)(k_1{r}_d^2+k_2{r}_d^4+...) \\ {v}_u=v_d+(v_d-v_0)(k_1{r}_d^2+k_2{r}_d^4+...) \end{gathered}$$ Where $r_d$ is the radius using distorted coordinates, and $k_1,k_2,...$ are the radial distortion coefficients. Tangential distortion parameters ($p_1,p_2$) can also be added.
• Impact: Including distortion parameters makes the calibration problem non-linear.
• Solution Strategy:
  1. Initial Guess: Estimate $\mathbf{M}$ (and thus $\mathbf{K},\mathbf{R},\mathbf{T}$) using the linear DLT method, ignoring distortion.
  2. Non-linear Refinement: Minimize the reprojection error over all parameters (intrinsics $\mathbf{K}$, extrinsics $[\mathbf{R}|\mathbf{T}]$ for each view, and distortion coefficients $k_1,k_2,...$) using an iterative optimization algorithm like Levenberg-Marquardt. Important Term: Reprojection Error: The geometric distance (e.g., Euclidean distance in pixels) between the measured 2D image point ${\mathbf{p}}_i$ and the projected 2D point ${\hat{\mathbf{p}}}_i$ obtained by projecting the corresponding 3D point ${\mathbf{P}}_{w,i}$ using the current estimate of the camera parameters ($\mathbf{K},\mathbf{R},\mathbf{T}$, distortion). ${\min }_{\mathbf{K},\mathbf{R},\mathbf{T},k_1,k_2,...}{\sum }_i||{\mathbf{p}}_i-{\hat{\mathbf{p}}}_i(\mathbf{K},\mathbf{R},\mathbf{T},k_1,k_2,...,{\mathbf{P}}_{w,i})|{|}^2$

Direct Calibration (Alternative Approach, e.g., Tsai)

These methods attempt to solve for parameters more directly, sometimes making simplifying assumptions (like zero distortion, known aspect ratio, known principal point) to derive linear solutions for some parameters, followed by non-linear optimization for others. The slides outline one such method (Slides 52-59) involving solving for parts of the extrinsic parameters linearly using SVD, determining scale factors, orthogonalizing the rotation matrix, and finally solving for focal length and $T_z$. These often require careful handling of scale and signs.

Multi-Plane Calibration (Zhang’s Method)

A very popular and practical method that uses a planar calibration target (e.g., a checkerboard) shown at several different orientations.

1. Homography: For a planar target (assume $Z_w=0$), the relationship between 3D world points ${\mathbf{P}}_w=[X_w,Y_w,0,1{]}^T$ and image points $\mathbf{p}$ simplifies. Let ${\tilde{\mathbf{P}}}_w=[X_w,Y_w,1{]}^T$. The projection is given by a homography $\mathbf{H}$: $$\lambda \mathbf{p}=\mathbf{K}[{\mathbf{r}}_1{\mathbf{r}}_2{\mathbf{r}}_3|\mathbf{T}]\left[\begin{array}{c}{X}_w\\ Y_w\\ 0\\ 1\end{array}\right]=\mathbf{K}[{\mathbf{r}}_1{\mathbf{r}}_2|\mathbf{T}]\left[\begin{array}{c}{X}_w\\ Y_w\\ 1\end{array}\right]=\mathbf{H}{\tilde{\mathbf{P}}}_w$$ Where $\mathbf{H}=\mathbf{K}[{\mathbf{r}}_1{\mathbf{r}}_2|\mathbf{T}]$ is a $3\times 3$ matrix, and ${\mathbf{r}}_1,{\mathbf{r}}_2$ are the first two columns of the rotation matrix $\mathbf{R}$.
2. Estimate Homographies: For each image ($k$) of the planar target, estimate the homography ${\mathbf{H}}_k$ relating the known 2D points on the target plane to the measured 2D image points using DLT.
3. Constraints on Intrinsics: Let $\mathbf{H}=[{\mathbf{h}}_1{\mathbf{h}}_2{\mathbf{h}}_3]$. We have $[{\mathbf{h}}_1{\mathbf{h}}_2{\mathbf{h}}_3]=\rho \mathbf{K}[{\mathbf{r}}_1{\mathbf{r}}_2\mathbf{t}]$. Using the properties that ${\mathbf{r}}_1,{\mathbf{r}}_2$ are orthonormal columns of $\mathbf{R}$:
   • ${\mathbf{r}}_1^T{\mathbf{r}}_2=0$
   • ${\mathbf{r}}_1^T{\mathbf{r}}_1={\mathbf{r}}_2^T{\mathbf{r}}_2=1$ (same length) Substitute ${\mathbf{r}}_1=\frac{1}{\rho }{\mathbf{K}}^{-1}{\mathbf{h}}_1$ and ${\mathbf{r}}_2=\frac{1}{\rho }{\mathbf{K}}^{-1}{\mathbf{h}}_2$. This yields two linear constraints on the matrix $\mathbf{B}={\mathbf{K}}^{-T}{\mathbf{K}}^{-1}$ (which is symmetric) per homography:
   • ${\mathbf{h}}_1^T{\mathbf{K}}^{-T}{\mathbf{K}}^{-1}{\mathbf{h}}_2=0$
   • ${\mathbf{h}}_1^T{\mathbf{K}}^{-T}{\mathbf{K}}^{-1}{\mathbf{h}}_1={\mathbf{h}}_2^T{\mathbf{K}}^{-T}{\mathbf{K}}^{-1}{\mathbf{h}}_2$
4. Solve for Intrinsics (K): Since $\mathbf{B}$ is symmetric ($3\times 3$), it has 6 unique elements. Each homography gives 2 constraints. Stacking constraints from $n\ge 3$ views allows solving linearly for the vector of elements in $\mathbf{B}$. Once $\mathbf{B}$ is known, $\mathbf{K}$ can be recovered using Cholesky decomposition (since $\mathbf{B}={\mathbf{K}}^{-T}{\mathbf{K}}^{-1}$).
5. Solve for Extrinsics ([R|T]): Once $\mathbf{K}$ is known, the extrinsics for each view $k$ can be easily computed from the corresponding homography ${\mathbf{H}}_k$.
6. Refinement: Perform non-linear minimization of reprojection error (as described before) to refine $\mathbf{K}$, all $[{\mathbf{R}}_k|{\mathbf{T}}_k]$, and potentially lens distortion parameters. Exam Tip: Zhang’s method is important. Key ideas: planar target → homography, constraints from rotation matrix properties on intrinsics, linear solve for ${\mathbf{K}}^{-T}{\mathbf{K}}^{-1}$, non-linear refinement.

Self-Calibration

• Goal: Calibrate the camera using only point correspondences between multiple images, without a known calibration object.
• Assumptions: Typically assumes a static scene and fixed intrinsic parameters across views.
• Method: Relies on constraints from projective geometry, often involving the “image of the absolute conic.”
• Characteristics: Highly flexible (no special setup needed) but generally less accurate and robust than methods using calibration targets. More complex mathematically.