Vision sensors measure electromagnetic radiation (typically visible or near-infrared) arriving at a detector array. Compared to 1D range sensors (Lesson 3), cameras output a 2D field of measurements, enabling robots to infer structure, motion, and semantics after processing (processing is a Chapter 8 topic). We focus here on the sensor-level models.

Let a 3D point in the camera coordinate frame be \( \mathbf{P}_c = [X, Y, Z]^T \in \mathbb{R}^3 \) with \( Z > 0 \) in front of the camera. A vision sensor produces some measurement \( \mathbf{z} \) related by a sensor model \( \mathbf{z} = h(\mathbf{P}_c) + \boldsymbol{\eta} \), where \( \boldsymbol{\eta} \) is noise (statistics in Lesson 5).

flowchart TD
  W["3D world radiation"] --> L["Lens / optics"]
  L --> S["Sensor array (pixels)"]
  S --> D["Digitization & timing"]
  D --> O["Output: image / depth map"]
  O --> R["Robot software stack"]
        

We now study three major classes: (i) passive RGB cameras, (ii) active depth cameras (ToF, structured light), and (iii) stereo cameras (passive depth).

An RGB camera measures intensity in three bands (R,G,B). The most common geometric model is the pinhole camera, which maps 3D points to 2D image coordinates by central projection.

Define normalized image coordinates \( x = X/Z \), \( y = Y/Z \). With focal lengths \( f_x, f_y \) (in pixels) and principal point \( (c_x, c_y) \), pixel coordinates are

\[ \begin{bmatrix} u \\ v \\ 1 \end{bmatrix} = \underbrace{\begin{bmatrix} f_x & 0 & c_x \\ 0 & f_y & c_y \\ 0 & 0 & 1 \end{bmatrix}}_{\mathbf{K}} \begin{bmatrix} x \\ y \\ 1 \end{bmatrix} = \mathbf{K} \begin{bmatrix} X/Z \\ Y/Z \\ 1 \end{bmatrix}. \]

The matrix \( \mathbf{K} \) is the intrinsic matrix; its parameters are estimated by calibration (Chapter 8, Lesson 2). In homogeneous form:

\[ s\begin{bmatrix} u \\ v \\ 1 \end{bmatrix} = \mathbf{K}\begin{bmatrix} X \\ Y \\ Z \end{bmatrix}, \quad s = Z. \]

Key implication: RGB cameras do not measure depth directly because scaling by \( Z \) is lost. Many different 3D points on the ray \( \lambda[X,Y,Z]^T \), \( \lambda > 0 \), project to the same pixel.

A simple noise model for a pixel measurement is \( \tilde{u} = u + \eta_u \), \( \tilde{v} = v + \eta_v \), where often \( \eta_u,\eta_v \sim \mathcal{N}(0,\sigma_{px}^2) \). Linearizing the projection gives first-order covariance propagation:

\[ \mathbf{\Sigma}_{uv} \approx \mathbf{J}_{uv}\,\mathbf{\Sigma}_{XYZ}\,\mathbf{J}_{uv}^T + \sigma_{px}^2 \mathbf{I}_2, \quad \mathbf{J}_{uv} = \frac{\partial(u,v)}{\partial(X,Y,Z)}. \]

This Jacobian-based linearization reappears in sensor fusion contexts (only conceptual in Chapter 8).

Active depth sensors emit light (usually IR) and estimate distance per pixel. Two dominant methods are: (a) Time-of-Flight (ToF) and (b) Structured Light.

3.1 Time-of-Flight (ToF)

A ToF camera modulates emitted light and measures return time (or phase shift). If a pulse travels to an object and back in time \( \Delta t \), depth is

\[ Z = \frac{c\,\Delta t}{2}, \]

where \( c \approx 3\times 10^8 \text{ m/s} \). In continuous-wave ToF, depth comes from phase:

\[ Z = \frac{c}{4\pi f_m}\,\Delta \phi, \]

with modulation frequency \( f_m \) and phase delay \( \Delta \phi \in [0,2\pi) \). Ambiguity appears when \( Z \) exceeds \( Z_{\max} = \frac{c}{2 f_m} \), leading to phase wrapping.

If timing noise is \( \eta_t \sim \mathcal{N}(0,\sigma_t^2) \), then depth noise follows by linear propagation:

\[ \sigma_Z^2 = \left(\frac{\partial Z}{\partial \Delta t}\right)^2 \sigma_t^2 = \left(\frac{c}{2}\right)^2 \sigma_t^2. \]

3.2 Structured Light

Structured-light cameras project a known pattern and infer depth by observing its deformation. Geometrically this is equivalent to stereo between a “virtual projector camera” and the real camera. The depth formula is therefore stereo-like (derived in Section 4).

Practical notes: Active depth performs best indoors; strong sunlight can saturate the IR channel. Surfaces with low reflectance or multi-path reflections increase depth noise.

Stereo uses two laterally separated cameras with baseline \( B \). Each camera follows the pinhole model. Under rectification (epipolar lines horizontal), corresponding points differ mainly in horizontal pixel coordinate.

flowchart TD
  CL["Left camera center"] ---|baseline B| CR["Right camera center"]
  P["3D point"] --> CL
  P --> CR
  CL --> IL["Left image u_L"]
  CR --> IR["Right image u_R"]
  IL --> DISP["Disparity d = u_L - u_R"]
  DISP --> DEPTH["Depth Z = f*B/d"]
        

Let the left and right image coordinates be \( u_L \) and \( u_R \) (same row). Disparity is \( d = u_L - u_R \). By similar triangles:

\[ Z = \frac{f B}{d}, \]

Proof (similar triangles):

In the rectified setup, the point projects to normalized x-coordinates \( x_L = X/Z \) and \( x_R = (X-B)/Z \). Multiplying by \( f \): \( u_L = f X/Z \), \( u_R = f (X-B)/Z \). Subtracting gives \( d = u_L - u_R = fB/Z \), hence \( Z = fB/d \).

Stereo depth uncertainty grows with range. If disparity noise is \( \eta_d \sim \mathcal{N}(0,\sigma_d^2) \), then

\[ \sigma_Z^2 = \left(\frac{\partial Z}{\partial d}\right)^2\sigma_d^2 = \left(\frac{fB}{d^2}\right)^2\sigma_d^2 = \left(\frac{Z^2}{fB}\right)^2\sigma_d^2. \]

So \( \sigma_Z \propto Z^2 \): far objects have rapidly increasing depth noise.

Vision sensors are chosen by matching task constraints to sensing physics:

• RGB camera: rich texture/color, cheap, low power; no direct depth. Works in many lighting conditions but suffers from glare/low light.
• Active depth: direct per-pixel range; sensitive to ambient IR, reflective surfaces, limited outdoors. Depth noise usually grows roughly linearly with distance (ToF) but can show multi-path artifacts.
• Stereo: passive depth outdoors-friendly; needs texture and accurate correspondence; depth noise grows as \(Z^2\).

A high-level “information rate” proxy is pixel count times frame rate: \( R_{info} \approx N_{px}\,f_s \). Higher \(R_{info}\) improves reactivity but increases bandwidth and processing load (software trade-offs in Chapters 10–12).

These examples show how to read camera streams. They are not perception algorithms. We use OpenCV as a cross-platform library; ROS/ROS2 wrappers are used in later chapters.

6.1 Python (OpenCV)

import cv2

# RGB camera (index 0)
cap = cv2.VideoCapture(0)
if not cap.isOpened():
    raise RuntimeError("Cannot open camera")

while True:
    ok, frame = cap.read()
    if not ok:
        break

    # frame is BGR in OpenCV; convert to RGB if needed
    rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
    cv2.imshow("RGB stream", frame)

    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

cap.release()
cv2.destroyAllWindows()
      

6.2 C++ (OpenCV)

#include <opencv2/opencv.hpp>
#include <iostream>

int main() {
    cv::VideoCapture cap(0);
    if (!cap.isOpened()) {
        std::cerr << "Cannot open camera\n";
        return -1;
    }

    cv::Mat frame;
    while (true) {
        cap >> frame;
        if (frame.empty()) break;

        cv::imshow("RGB stream", frame);
        if (cv::waitKey(1) == 'q') break;
    }
    return 0;
}
      

6.3 Java (OpenCV Java Bindings)

import org.opencv.core.*;
import org.opencv.videoio.VideoCapture;
import org.opencv.highgui.HighGui;

public class CameraBringUp {
    static { System.loadLibrary(Core.NATIVE_LIBRARY_NAME); }

    public static void main(String[] args) {
        VideoCapture cap = new VideoCapture(0);
        if (!cap.isOpened()) {
            System.out.println("Cannot open camera");
            return;
        }

        Mat frame = new Mat();
        while (true) {
            cap.read(frame);
            if (frame.empty()) break;

            HighGui.imshow("RGB stream", frame);
            if (HighGui.waitKey(1) == 'q') break;
        }
        cap.release();
        HighGui.destroyAllWindows();
    }
}
      

6.4 MATLAB (Image Acquisition Toolbox)

% RGB camera bring-up (requires Image Acquisition Toolbox)
cam = webcam(1);  % choose device index
for k = 1:300
    img = snapshot(cam);
    imshow(img); drawnow;
end
clear cam;
      

6.5 Simulink (Conceptual Block Use)

In Simulink, equivalent bring-up uses: From Video Device → Color Space Conversion → Display. Depth cameras are accessed via vendor blocks or by streaming depth images into a MATLAB Function block for custom handling (details in later labs).

6.6 Depth and Stereo Streams (Note)

Depth cameras often provide a second stream (16-bit depth). Stereo setups typically provide left/right images; computing disparity is a Chapter 8 concept. Still, OpenCV provides a ready function later: \(d = \texttt{StereoBM}(\text{left},\text{right})\).

Problem 1 (Projection Scaling): Show that if a 3D point \( \mathbf{P}_c = [X,Y,Z]^T \) projects to pixel \( (u,v) \) under the pinhole model, then any point \( \lambda \mathbf{P}_c \) with \( \lambda > 0 \) projects to the same pixel.

Solution:

Under pinhole projection, \( u = f_x X/Z + c_x \), \( v = f_y Y/Z + c_y \). For \( \lambda\mathbf{P}_c = [\lambda X, \lambda Y, \lambda Z]^T \), we get \( u' = f_x (\lambda X)/(\lambda Z) + c_x = u \) and similarly \( v' = v \). So RGB cameras do not encode depth without extra assumptions.

Problem 2 (Stereo Depth Derivation): In a rectified stereo pair with focal length \( f \) and baseline \( B \), derive the depth \( Z = fB/d \), where \( d \) is disparity.

Solution:

From Section 4: \( u_L = f X/Z \), \( u_R = f(X-B)/Z \). Thus \( d=u_L-u_R = fB/Z \), giving \( Z=fB/d \).

Problem 3 (Noise Growth in Stereo): Suppose disparity noise variance is \( \sigma_d^2 \). Prove that depth standard deviation grows quadratically with range: \( \sigma_Z \propto Z^2 \).

Solution:

With \( Z = fB/d \), \( \frac{\partial Z}{\partial d} = -fB/d^2 \). First-order propagation yields \( \sigma_Z^2 = (fB/d^2)^2\sigma_d^2 \). Since \( d=fB/Z \), substitute: \( \sigma_Z^2 = (Z^4/(fB)^2)\sigma_d^2 \). Therefore \( \sigma_Z \propto Z^2 \).

Problem 4 (ToF Timing Precision): A ToF camera has timing standard deviation \( \sigma_t \). Derive \( \sigma_Z \) for depth.

Solution:

From \( Z = c\Delta t/2 \), \( \sigma_Z^2 = (c/2)^2 \sigma_t^2 \), so \( \sigma_Z = (c/2)\sigma_t \).

Problem 5 (Selection Reasoning): A mobile robot must perceive obstacles outdoors at 20 m and can carry moderate compute. Choose between RGB, active depth, and stereo, and justify using Section 5 and the noise models.

Solution:

Active depth struggles outdoors due to sunlight/IR interference and often has limited range. RGB alone provides no depth. Stereo is passive and outdoor-friendly; although \( \sigma_Z \propto Z^2 \) increases noise at 20 m, a sufficiently large baseline \( B \) and focal length \( f \) mitigate error. Thus stereo is the best high-level choice.

We modeled RGB cameras with pinhole projection and showed why depth is not directly observable. We introduced active depth sensing via ToF and structured light, and derived stereo triangulation with its key uncertainty scaling. These sensor-level models set the stage for calibration and perception pipelines in Chapter 8.