Memos: Tübingen ML Open Crouse Image Formation 1

Image Formation 1: Primitives and Transformations

Credits: Tübingen Machine Learning | Computer Vision - Andreas Geiger
Computer Vision - Lecture 2.1 (Image Formation: Primitives and Transformations) - YouTube

1.1 Primitives and Transformations

• Geometric primitives are the basic building blocks used to describe 3D shapes
• Introduction of points, lines and planes
• Introduction of the most basic transformations

2D points

2D points can be written in inhomogeneous coordinates as

$$\mathrm{x} = \begin{pmatrix} x \\ y \\ \end{pmatrix} \in \mathbb{R}^2$$

or in homogeneous coordinates as

$$\tilde{\mathrm{x}} = \begin{pmatrix} \tilde{x} \\ \tilde{y} \\ \tilde{w} \\\end{pmatrix} \in \mathbb{P}^2$$

where $\mathbb{p}^2 = \mathbb{R}^3 / {(0,0,0)}$ is called projective space.

An inhomogeneous vector x is converted to a homogeneous vector x as followed:

$$\tilde{\mathrm{x}} = \begin{pmatrix} \tilde{x} \\ \tilde{y} \\ \tilde{w} \\\end{pmatrix} = \begin{pmatrix} x \\ y \\ 1 \\\end{pmatrix} = \begin{pmatrix} \mathrm{x} \\ 1 \\\end{pmatrix} = \bar{\mathrm{x}}$$

with augmented vector $\bar{\mathrm{x}}$. To convert in the opposite direction, we divide by $\tilde{w}$.

$$\bar{\mathrm{x}} = \begin{pmatrix} \mathrm{x} \\ 1 \\\end{pmatrix} = \begin{pmatrix} x \\ y \\ 1 \\\end{pmatrix} = \frac{1}{\tilde{w}}\tilde{\mathrm{x}} = \frac{1}{\tilde{w}}\begin{pmatrix} \tilde{x} \\ \tilde{y} \\ \tilde{w} \\\end{pmatrix} = \begin{pmatrix} \tilde{x}/ \tilde{w}\\ \tilde{y}/\tilde{w} \\ 1 \\\end{pmatrix}$$

Homogeneous points whose last element is $\tilde{w} =0$ are called ideal points or points at infinity. These points can’t be represented with inhomogeneous coordinates.

2D lines

2D lines can also be expressed using homogeneous coordinates $\tilde{\mathrm{l}} = (a,b,c)^{\top}$:

$$\{ \bar{\mathrm{x}} \bigm| \tilde{\mathrm{l}}^{\top}\bar{\mathrm{x}}=0 \} \Leftrightarrow \{ x,y \bigm| ax + by + c = 0 \}$$

We can normalize $\tilde{\mathrm{l}}$ so that $\tilde{\mathrm{l}} = (n_x,n_y,d)^{\top}=(\mathrm{n},d)^{\top}$ with $\|\mathrm{n}\|_2 = 1$. In this case, $\mathrm{n}$ is the normal vector perpendicular to the line and $d$ is its distance to the origin.

An exception is the line at infinity $\tilde{\mathrm{l}}_{\infty} = (0,0,1)^{\top}$ which passes through all ideal points.

Cross Product

Cross product expressed as the product of a skew-symmetric matrix and a vector:

$$\mathrm{a}\times\mathrm{b} = [\mathrm{a}]\times\mathrm{b} = \begin{bmatrix} 0 & -a_3 & a_2 \\ a_3 & 0 & -a_1 \\ -a_2 & 1 & 0 \end{bmatrix} \begin{pmatrix} b_1 \\ b_2 \\ b_3 \end{pmatrix} = \begin{pmatrix} a_2b_3-a_3b_2\\ a_3b_1-a_1b_3 \\ a_1b_2-a_2b_1 \end{pmatrix}$$

Remark: Squared brackets are matrices.

2D Line Arithmetic

In homogeneous coordinates, the intersection of two lines is given by:

$$\tilde{\mathrm{x}}=\tilde{\mathrm{l}}_1\times\tilde{\mathrm{l}}_2$$

Similarly, the line joining two points can be compactly written as:

$$\tilde{\mathrm{l}}=\bar{\mathrm{x}}_1\times\bar{\mathrm{x}}_2$$

The symbol $\times$ denotes the cross product.

2D Conics

More complex algebraic objects can be represented using polynomial homogeneous equations. For example, conic sections (arising as the intersection of a plane and a3D cone) can be written using quadric equations:

$$\{ \bar{\mathrm{x}} \bigm| \bar{\mathrm{x}}^{\top}Q\bar{\mathrm{x}}=0 \}$$

3D Points

3D points can be written in inhomogeneous coordinates as

$$\mathrm{x}= \begin{pmatrix}x\\y\\z\end{pmatrix} \in \mathbb{R}^3$$

or in homogeneous coordinates as

$$\tilde{\mathrm{x}}= \begin{pmatrix}\tilde{x}\\\tilde{y}\\\tilde{z}\\\tilde{w}\end{pmatrix} \in \mathbb{P}^3$$

with Projective space $\mathbb{P}^3= \mathbb{R}^4/{(0,0,0,0)}$.

3D Planes

3D planes can also be represented as homogeneous coordinates $\tilde{\mathrm{m}}=(a,b,c,d)^{\top}$:

$$\{ \bar{\mathrm{x}} \bigm|\tilde{\mathrm{m}}^{\top}\bar{\mathrm{x}}=0 \} \Leftrightarrow \{ x,y,z\bigm| ax+by+cz+d=0 \}$$

Again, we can normalize $\tilde{\mathrm{m}}$ so that $\tilde{\mathrm{m}} = (n_x,n_y,n_z,d)^{\top}=(\mathrm{n},d)^{\top}$ with $\|\mathrm{n}\|_2 = 1$. In this case, $\mathrm{n}$ is the normal perpendicular to the plane and $d$ is its distance to the origin.

An exception is the plane at infinity $\tilde{\mathrm{m}}_{\infty} = (0,0,0,1)^{\top}$ which passes through all ideal points (=points at infinity) for which $\tilde{w}=0$.

3D lines

3D lines are less elegant than either 2D lines or 3D planes. One possible representation is to express points on a line as a linear combination of two points $p$ and $q$ on the line:

$$\{\mathrm{x}\bigm| \mathrm{x}=(1-\lambda)\mathrm{p}+\lambda\mathrm{q}\wedge\lambda\in\mathbb{R} \}$$

However, this representation uses 6 parameters for 4 degrees of freedom.
Alternative minimal representations are the two-plane parameterization or Pluecker coordinates. See Szeliski, Chapter 2.1.

3D Quadrics

The 3D analog of 2D conics is a quadric surface:

$$\{ \bar{\mathrm{x}} \bigm| \bar{\mathrm{x}}^{\top}Q\bar{\mathrm{x}}=0 \}$$

Useful in the study of multi-view geometry. Also serves as useful modeling primitives (spheres, ellipsoids, cylinders).

2D Transformations

Translation: (2D Translation of the Input, 2 DF)

$$\mathrm{x}'=\mathrm{x}+t \Leftrightarrow \bar{\mathrm{x}}'= \begin{bmatrix} I & t \\ 0^{\top} & 1 \end{bmatrix} \bar{\mathrm{x}}$$

• Using homogeneous representations allows to chain/invert transformations
• Augmented vectors $\bar{\mathrm{x}}$ can always be replaced by general homogeneous ones $\tilde{\mathrm{x}}$

Euclidean: (2D Translation + 2D Rotation, 3 DF)

$$\mathrm{x}'=R\mathrm{x}+t \Leftrightarrow \bar{\mathrm{x}}'= \begin{bmatrix} R & t \\ 0^{\top} & 1 \end{bmatrix} \bar{\mathrm{x}}$$

• $R \in SO(2)$ is a rotation matrix and $s$ is an arbitrary scale factor
• The similarity transform preserves angles between lines

Affine: (2D Linear Transformation, 6 DF)

$$\mathrm{x}'=A\mathrm{x}+t \Leftrightarrow \bar{\mathrm{x}}'= \begin{bmatrix} A & t \\ 0^{\top} & 1 \end{bmatrix} \bar{\mathrm{x}}$$

• $A \in \mathbb{R}^{2\times2}$ is an arbitrary $2\times2$ matrix
• Parallel lines remain parallel under affine transformations

Perspective: (Homography, 8DF)

$$\tilde{\mathrm{x}}'=\tilde{H}\tilde{\mathrm{x}} \qquad \begin{pmatrix} \bar{\mathrm{x}}=\frac{1}{\tilde{w}}\tilde{\mathrm{x}} \end{pmatrix}$$

• $\tilde{\mathrm{H}} \in \mathbb{R}^{3\times3}$ is an arbitrary homogeneous $3\times3$ matrix (specified up to scale)
• Perspective transformations preserve straight lines

2D Transformations on Co-Vectors

Considering any perspective 2D transformation

$$\tilde{\mathrm{x}}' = \tilde{\mathrm{H}}\tilde{\mathrm{x}}$$

the transformed 2D line equation is given by:

$$\tilde{\mathrm{l}}'^{\top}\tilde{\mathrm{x}}' =\tilde{\mathrm{l}}'^{\top}\tilde{H}\tilde{\mathrm{x}}' =(\tilde{\mathrm{H}}^{\top}\tilde{l}')^{\top}\tilde{\mathrm{x}}' =\tilde{\mathrm{l}}^{\top}\tilde{\mathrm{x}} =0$$

Therefore, we have:

$$\tilde{\mathrm{l}}' =\tilde{\mathrm{H}}^{-\top}\tilde{\mathrm{l}}$$

Thus, the action of a projective transformation on a co-vector such as a 2D line or 3D normal can be represented by the transposed inverse of the matrix.

Overview of 2D Transformation

Transformation	Matrix	DF	Preserves
translation	$\begin{bmatrix}I&t\end{bmatrix}_{2\times3}$	2	orientation
rigid	$\begin{bmatrix}R&t\end{bmatrix}_{2\times3}$	3	lengths
similarity	$\begin{bmatrix}sR&t\end{bmatrix}_{2\times3}$	4	angles
affine	$\begin{bmatrix}A\end{bmatrix}_{2\times3}$	6	parallelism
projective	$\begin{bmatrix}\tilde{\mathrm{H}}\end{bmatrix}_{3\times3}$	8	straight lines

• Transformations form nested set of groups
• Interpret as restricted $3\times3$ matrices operating on 2D homogeneous coordinates
• Transformations preserve properties below

Overview of 3D Transformation

Transformation	Matrix	DF	Preserves
translation	$\begin{bmatrix}I&t\end{bmatrix}_{3\times4}$	3	orientation
rigid	$\begin{bmatrix}R&t\end{bmatrix}_{3\times4}$	6	lengths
similarity	$\begin{bmatrix}sR&t\end{bmatrix}_{3\times4}$	7	angles
affine	$\begin{bmatrix}A\end{bmatrix}_{3\times4}$	12	parallelism
projective	$\begin{bmatrix}\tilde{\mathrm{H}}\end{bmatrix}_{4\times4}$	15	straight lines

• 3D transformations are defined analogously to 2D transformations
• $3\times4$ matrices are extended with a fourth $\begin{bmatrix}0^{\top} 1\end{bmatrix}$ row for homogeneous transforms
• Transformations preserve properties below

Direct Linear Transform for Homography Estimation

Q: How can we estimate a homography from a set of 2D correspondences?
Let $\mathcal{X}=\{\tilde{\mathrm{x_i}},\tilde{\mathrm{x'_i}}\}^{N}_{i=1}$ denote a set of $N$ 2D-to-2D correspondences related by $\tilde{\mathrm{x_i'}}=\tilde{\mathrm{H}}\tilde{\mathrm{x_i}}$. As the correspondence vectors are homogeneous, they have the same direction but differ in magnitude. Thus, the equation above can be expressed as$\tilde{\mathrm{x_i}'}\times\tilde{\mathrm{H}}\tilde{\mathrm{x_i}}=0$.
Using $\tilde{\mathrm{h}^{\top}_k}$ to denote the k’th row of $\tilde{\mathrm{H}}$, this can be rewritten as a linear equation in $\tilde{\mathrm{h}}$:

$$\underbrace{ \begin{bmatrix} 0^{\top} & -\tilde{w_i'}\tilde{\mathrm{x_i}^{\top}} & \tilde{y'}\tilde{\mathrm{x_i}}^{\top}\\ \tilde{w'}\tilde{\mathrm{x_i}}^{\top} & 0^{\top} & -\tilde{x'}\tilde{\mathrm{x_i}}^{\top}\\ -\tilde{y'}\tilde{\mathrm{x_i}}^{\top} & \tilde{x'}\tilde{\mathrm{x_i}}^{\top} & 0^{\top} \\ \end{bmatrix} }_{\mathrm{A_i}} \underbrace{ \begin{bmatrix} \tilde{h_1}\\ \tilde{h_2}\\ \tilde{h_3} \end{bmatrix}}_{\tilde{h}}=0$$

Each point correspondence yields two equations. Stacking all equations int a $2N\times9$ dimensional matrix $\mathrm{A}$ leads to the following constrained least squares problem:

$$\tilde{h}^*=argmin_{\tilde{h}}\ \|\mathrm{A\tilde{h}}\|^2_2+\lambda(\|\tilde{h}\|^2_2-1) = argmin_{\tilde{h}}\ \tilde{h}^{\top}\mathrm{A}^{\top}\mathrm{A}\tilde{h}+\lambda(\tilde{h}^{\top}\tilde{h}-1)$$

where we have fixed $\|\mathrm{\tilde{h}}\|^2_2=1$ as $\tilde{\mathrm{H}}$ is homogeneous (i.e., defined only up to scale) and the trivial solution to $\tilde{h}=0$ is not of interest. The solution to the above optimization problem is the singular vector corresponding to the smallest singular value of $\mathrm{A}$ (i.e., the last column of $\mathrm{V}$ when decomposing $\mathrm{A=UDV^{\top}}$, see also Deep Learning lecture 11.2). The resulting algorithm is called Direct Linear Transformation.

1.2 Geometric Image Formation

Origins of the Pinhole Camera

• In a physical pinhole camera the image is projected up-side down onto the image plane which is located behind the focal point
• When modeling perspective projection, we assume the image plane in front
• Both models are equivalent, with appropriate change of image coordinates

Projection Models

• Orthographic Projection
• Perspective Projection

Orthographic Projection

Orthographic projection of a 3D point $\mathrm{x_c}\in\mathbb{R}^3$ to pixel coordinates $\mathrm{x_s}\in\mathbb{R}^3$

• The x and y axes of the camera and image coordinate systems are shared
• Light rays are parallel to the z-coordinate of the camera coordinate system
• During projection, the z-coordinate is dropped, x and y remain the same

An orthographic projection simply drops the z component of the 3D point in camera coordinates $\mathrm{x_c}$ to obtain the corresponding 2D point on the image plane (=screen) $\mathrm{x_s}$.

$$\mathrm{x_s}= \begin{bmatrix} 1&0&0\\ 0&1&0\end{bmatrix}\mathrm{x_c} \quad\Leftrightarrow\quad \bar{\mathrm{x_s}} \begin{bmatrix} 1&0&0&0\\ 0&1&0&0\\ 0&0&0&1\end{bmatrix}\bar{\mathrm{x_c}}$$

Orthography is exact for telecentric lenses and an approximation for telephoto lenses. After projection, the distance of the 3D point from the image can’t be recovered.

We usually scale orthography:

$$\mathrm{x_s}= \begin{bmatrix} s&0&0\\ 0&s&0\end{bmatrix}\mathrm{x_c} \quad\Leftrightarrow\quad \bar{\mathrm{x_s}} \begin{bmatrix} s&0&0&0\\ 0&s&0&0\\ 0&0&0&s\end{bmatrix}\bar{\mathrm{x_c}}$$

Here, the unit for $s$ is px/m or px/mm to convert metric 3D points into pixels.

Perspective Projection

Perspective projection of a 3D point $\mathrm{x_c}\in\mathbb{R}^3$ to pixel coordinates $\mathrm{x_s}\in\mathbb{R}^3$

• Light rays passes through the camera center, the pixel $\mathrm{x_s}$ and the point $\mathrm{x_c}$
• Convention: the principal axis (orthogonal to image plane) aligns with the z-axis

In perspective projection, 3D points in camera coordinates are mapped to the image plane by dividing their z component and multiplying with the focal length:

$$\begin{pmatrix}x_s\\y_s\end{pmatrix} = \begin{pmatrix}fx_c/z_c\\fy_c/z_c\end{pmatrix} \quad\Leftrightarrow\quad \tilde{\mathrm{x}}_s =\begin{bmatrix} f&0&0&0\\0&f&0&0\\0&0&1&0\\ \end{bmatrix}\bar{\mathrm{x}}_c$$

Note that this projection is linear when using homogeneous coordinates. After the projection, it is not possible to recover the distance of the 3D point from the image.

• To ensure positive pixel coordinates, a principal point offset c is usually added
• This moves the image coordinate system to the corner of the image plane

The complete perspective projection model is given by:

$$\begin{pmatrix}x_s\\y_s\end{pmatrix} = \begin{pmatrix}fx_c/z_c + sx_c/z_c + c_x\\fy_c/z_c + c_y\end{pmatrix} \quad\Leftrightarrow\quad \tilde{\mathrm{x}}_s =\begin{bmatrix} f_x&s&c_x&0\\0&f_y&c_y&0\\0&0&1&0\\ \end{bmatrix}\bar{\mathrm{x}}_c$$

• The left $3\times3$ submatrix of the projection matrix is called calibration matrix $\mathrm{K}$
• The parameters of $\mathrm{K}$ are called camera intrinsics (as opposed to extrinsic pose)
• Here, $f_x$ and $f_y$ are independent, allowing for different pixel aspect ratios
• The skew $s$ arises due to the sensor not mounted perpendicular to the optical axis
• In practice, we often set $f_x=f_y$ and $s=0$, but model $c=(c_x,c_y)^{\top}$

Chaining Transformations

Let $\mathrm{K}$ be the calibration matrix (intrinsics) and $[R|t]$ the camera pose (extrinsics).
We chain both transformations to project a point in world coordinates to the image:

$$\tilde{\mathrm{x}}_s =\begin{bmatrix} \mathrm{K} & 0\end{bmatrix}\bar{\mathrm{x}}_c =\begin{bmatrix} \mathrm{K} & 0\end{bmatrix} \begin{bmatrix} \mathrm{K} & 0\\0^{\top} & 1\end{bmatrix} \bar{\mathrm{x}}_w =\mathrm{K}\begin{bmatrix} R & t\end{bmatrix}\bar{\mathrm{x}}_w =\mathrm{P}\bar{\mathrm{x}}_w$$

Full Rank Representation

It is sometimes preferable to use a full rank $4\times4$ projection matrix:

$$\tilde{\mathrm{x}}_s =\begin{bmatrix} \mathrm{K} & 0\\0^{\top} & 1\end{bmatrix} \begin{bmatrix} \mathrm{R} & t\\0^{\top} & 1\end{bmatrix} \bar{\mathrm{x}}_w =\mathrm{P}\bar{\mathrm{x}}_w$$

Now, the homogeneous vector $\tilde{\mathrm{x}}_s$ is a 4D vector and must be normalized wrt. Its 3rd entry to obtain inhomogeneous image pixels:

$$\bar{\mathrm{x}}_s=\tilde{\mathrm{x}}_s/z_s =(x_s/z_s,y_s/z_s,1,1/z_s)^{\top}$$

Note that the 4th component of the inhomogeneous 4D vector is the inverse depth. If the inverse depth is known, a 3D point can be retrieved from its pixel coordinates via $\tilde{\mathrm{x}}_w=\tilde{\mathrm{P}}^{-1}\bar{\mathrm{x}}_w$ and subsequent normalization of $\tilde{\mathrm{x}}_w$ wrt. its 4th entry.

Lens Distortion

The assumption of linear projection is violated in practice due to the properties of the camera lens which introduces distortions. Both radial and tangential distortion effects can be modeled relatively easily: Let $x=x_c/z_c,y=y_c/z_c$ and $r_2=x^2+y^2$. The distorted point is obtained as:

$$\mathrm{x}' =\underbrace{(1+\mathcal{k}_1r^2+\mathcal{k}_2r^4)}_{ Radial\ Distortion }\begin{pmatrix}x\\y\end{pmatrix} + \underbrace{ \begin{pmatrix} 2\mathcal{k}_3xy+\mathcal{k}_4(r^2+2x^2)\\ 2\mathcal{k}_4xy+\mathcal{k}_3(r^2+2y^2) \end{pmatrix} }_{Tangential\ Distortion}$$

$$\mathrm{x}_s= \begin{pmatrix} f_xx'+c_x\\f_yy'+c_y \end{pmatrix}$$

Images can be undistorted such that the perspective projection model applies. More complex distortion models must be used for wide-angle lenses (e.g., fisheye).