A perception pipeline is an ordered (often branching) sequence of modules that transforms raw sensor signals into task-relevant representations: features, object states, scene geometry, or robot-local maps. A typical pipeline includes: (i) acquisition and synchronization, (ii) calibration and preprocessing, (iii) feature extraction, (iv) estimation/fusion, and (v) outputs to planners/controllers.

flowchart TD
  S0["Sensors: camera, lidar, imu, encoders"] --> S1["Acquisition & time sync"]
  S1 --> S2["Calibration apply (intrinsic/extrinsic)"]
  S2 --> S3["Preprocess (denoise, rectify, resample)"]
  S3 --> S4["Feature extract (edges, corners, keypoints)"]
  S4 --> S5["Estimate / fuse (solve for states)"]
  S5 --> S6["Perception output for control"]
  S3 -->|optional branch| S7["Segmentation / clustering"]
  S7 --> S5
        

Mathematically, a pipeline is a composition of operators: \( \mathcal{P} = \mathcal{O}_m \circ \cdots \circ \mathcal{O}_2 \circ \mathcal{O}_1 \), where each \( \mathcal{O}_i \) consumes a data structure and produces another. This composition perspective helps analyze error propagation and latency.

Let \( \mathbf{x} \in \mathbb{R}^d \) denote a physical quantity of interest (e.g., a landmark location, robot attitude, or object pose parameters). A sensor provides measurements \( \mathbf{y} \in \mathbb{R}^p \) described by the general model

\[ \mathbf{y} = \mathbf{h}(\mathbf{x}) + \mathbf{v}, \qquad E[\mathbf{v}] = \mathbf{0}. \]

In this chapter we avoid full probabilistic inference, yet the disturbance \( \mathbf{v} \) is still useful to model noise effects. When \( \mathbf{h} \) is approximately linear around an operating point \( \mathbf{x}_0 \), we use first-order linearization:

\[ \mathbf{h}(\mathbf{x}) \approx \mathbf{h}(\mathbf{x}_0) + \mathbf{H}(\mathbf{x}-\mathbf{x}_0), \quad \mathbf{H} = \left.\frac{\partial \mathbf{h}}{\partial \mathbf{x}}\right|_{\mathbf{x}_0}. \]

Preprocessing aims to produce a cleaned signal \( \tilde{\mathbf{y}} \). A common deterministic denoiser is a linear discrete-time filter:

\[ \tilde{\mathbf{y}}[k] = \sum_{i=0}^{M} b_i \mathbf{y}[k-i] - \sum_{j=1}^{N} a_j \tilde{\mathbf{y}}[k-j], \]

where \( b_i \) and \( a_j \) define an FIR (if all \( a_j=0 \)) or IIR filter. Students knowing linear control can view this as an LTI system with transfer function

\[ G(z) = \frac{\sum_{i=0}^{M} b_i z^{-i}}{1+\sum_{j=1}^{N} a_j z^{-j}}. \]

Key point: filtering changes noise statistics but also introduces latency. If the filter has group delay \( d_g \) samples, then a pipeline’s timing budget must include a delay \( d_g T_s \), where \( T_s \) is sampling period.

Feature extraction converts cleaned signals into structured descriptors. For a grayscale image \( I(u,v) \), edge-like features are based on spatial gradients:

\[ \nabla I = \begin{bmatrix} I_u \\ I_v \end{bmatrix}, \qquad I_u = \frac{\partial I}{\partial u},\;\; I_v = \frac{\partial I}{\partial v}. \]

A classical local corner measure uses the (smoothed) structure tensor \( \mathbf{S} \):

\[ \mathbf{S}(u,v)= \sum_{(i,j)\in \mathcal{N}(u,v)} w(i,j) \begin{bmatrix} I_u(i,j)^2 & I_u(i,j)I_v(i,j)\\ I_u(i,j)I_v(i,j) & I_v(i,j)^2 \end{bmatrix}. \]

Let \( \lambda_1,\lambda_2 \) be eigenvalues of \( \mathbf{S} \). A corner exists if both are large, since that indicates intensity changes in two independent directions. A simple scalar test is \( \min(\lambda_1,\lambda_2) > \gamma \) for threshold \( \gamma \).

Noise amplification proof (sketch). Suppose pixel noise is additive: \( I = I^\star + n \) with \( E[n]=0 \) and variance \( \sigma_n^2 \). Finite-difference gradient \( I_u \approx I(u+1,v)-I(u,v) \) yields noise term \( n(u+1,v)-n(u,v) \). Since independent noise differences add variances,

\[ \operatorname{Var}[I_u] = \operatorname{Var}[I_u^\star] + \operatorname{Var}[n(u+1,v)-n(u,v)] = \operatorname{Var}[I_u^\star] + 2\sigma_n^2. \]

Hence gradient-based features double noise variance before smoothing, motivating preprocessing filters in real pipelines.

After features are extracted, robots estimate states by solving a deterministic inverse problem. With linearized model \( \mathbf{y} \approx \mathbf{H}\mathbf{x} + \mathbf{v} \), the least-squares (LS) estimate is

\[ \hat{\mathbf{x}}_{\text{LS}} = \arg\min_{\mathbf{x}}\|\mathbf{y}-\mathbf{H}\mathbf{x}\|_2^2 = (\mathbf{H}^\top\mathbf{H})^{-1}\mathbf{H}^\top\mathbf{y}, \]

assuming \( \mathbf{H}^\top\mathbf{H} \) is invertible.

Unbiasedness proof. If \( E[\mathbf{v}]=\mathbf{0} \) and \( \mathbf{H} \) is deterministic,

\[ E[\hat{\mathbf{x}}_{\text{LS}}] = (\mathbf{H}^\top\mathbf{H})^{-1}\mathbf{H}^\top E[\mathbf{H}\mathbf{x}+\mathbf{v}] = (\mathbf{H}^\top\mathbf{H})^{-1}\mathbf{H}^\top \mathbf{H}\mathbf{x} = \mathbf{x}. \]

Thus LS is unbiased under our assumptions.

Multi-sensor fusion (deterministic). Suppose two sensors produce independent LS estimates \( \hat{\mathbf{x}}_1, \hat{\mathbf{x}}_2 \) with error covariances \( \mathbf{\Sigma}_1, \mathbf{\Sigma}_2 \). A fused estimate is the weighted LS solution:

\[ \hat{\mathbf{x}}_f = (\mathbf{\Sigma}_1^{-1}+\mathbf{\Sigma}_2^{-1})^{-1} (\mathbf{\Sigma}_1^{-1}\hat{\mathbf{x}}_1+\mathbf{\Sigma}_2^{-1}\hat{\mathbf{x}}_2). \]

Optimality (Gauss–Markov / BLUE). Consider linear unbiased estimators of form \( \hat{\mathbf{x}} = \mathbf{A}_1\hat{\mathbf{x}}_1+\mathbf{A}_2\hat{\mathbf{x}}_2 \) with unbiasedness requiring \( \mathbf{A}_1+\mathbf{A}_2=\mathbf{I} \). The fused covariance is

\[ \mathbf{\Sigma}_f = \mathbf{A}_1\mathbf{\Sigma}_1\mathbf{A}_1^\top +\mathbf{A}_2\mathbf{\Sigma}_2\mathbf{A}_2^\top. \]

Minimizing \( \operatorname{tr}(\mathbf{\Sigma}_f) \) under the constraint yields \( \mathbf{A}_1 = \mathbf{\Sigma}_f \mathbf{\Sigma}_1^{-1} \), \( \mathbf{A}_2 = \mathbf{\Sigma}_f \mathbf{\Sigma}_2^{-1} \), which implies the fusion formula above. Therefore the fused estimate has minimum variance among linear unbiased estimators.

Real robots operate under timing constraints. Model a pipeline as a directed acyclic graph (DAG) of modules. Let module \( i \) have execution time \( t_i \). If modules execute serially, end-to-end latency is

\[ L_{\text{serial}} = \sum_{i=1}^{m} t_i. \]

If some stages run in parallel branches that rejoin at the estimator, latency is the sum along the critical path:

\[ L_{\text{DAG}} = \max_{p\in\mathcal{Paths}} \sum_{i\in p} t_i. \]

flowchart LR
  A["Acquire (t1)"] --> B["Preprocess (t2)"]
  B --> C["Features A (t3)"]
  B --> D["Features B (t4)"]
  C --> E["Fuse/Estimate (t5)"]
  D --> E
  E --> F["Output (t6)"]
        

In the diagram, two branches run after preprocessing. The critical path is \( t_1+t_2+\max(t_3,t_4)+t_5+t_6 \). If the control loop period is \( T_c \), a necessary condition for usefulness is \( L_{\text{DAG}} < T_c \).

Pipeline stability analogy: Control students can view perception latency as a pure delay in the closed loop. Larger delays reduce phase margin, so perception designers trade accuracy for lower latency when needed.

We implement a simplified pipeline: (i) load an image, (ii) preprocess by smoothing, (iii) extract corners, (iv) estimate a 2D point from multiple measurements using weighted least squares fusion.

6.1 Python (NumPy + OpenCV)

import cv2
import numpy as np

# (1) Acquire
img = cv2.imread("scene.png", cv2.IMREAD_GRAYSCALE)

# (2) Preprocess: Gaussian smoothing
img_s = cv2.GaussianBlur(img, (5, 5), 1.2)

# (3) Feature extraction: Shi-Tomasi corners (structure tensor based)
corners = cv2.goodFeaturesToTrack(img_s, maxCorners=200, qualityLevel=0.01, minDistance=8)
corners = np.squeeze(corners)  # shape: (N,2)

# (4) Deterministic fusion example:
# Suppose two sensors give noisy position estimates x1, x2 with covariances S1, S2
x1 = np.array([1.2, 0.5])
x2 = np.array([1.0, 0.9])
S1 = np.diag([0.04, 0.09])
S2 = np.diag([0.16, 0.04])

W1 = np.linalg.inv(S1)
W2 = np.linalg.inv(S2)
xf = np.linalg.inv(W1 + W2) @ (W1 @ x1 + W2 @ x2)

print("Fused estimate:", xf)
      

6.2 C++ (OpenCV + Eigen)

#include <opencv2/opencv.hpp>
#include <Eigen/Dense>
#include <iostream>

int main() {
    // (1) Acquire
    cv::Mat img = cv::imread("scene.png", cv::IMREAD_GRAYSCALE);

    // (2) Preprocess
    cv::Mat img_s;
    cv::GaussianBlur(img, img_s, cv::Size(5,5), 1.2);

    // (3) Feature extraction
    std::vector<cv::Point2f> corners;
    cv::goodFeaturesToTrack(img_s, corners, 200, 0.01, 8);

    // (4) Weighted LS fusion
    Eigen::Vector2d x1(1.2, 0.5), x2(1.0, 0.9);
    Eigen::Matrix2d S1, S2;
    S1 << 0.04, 0.0, 0.0, 0.09;
    S2 << 0.16, 0.0, 0.0, 0.04;

    Eigen::Matrix2d W1 = S1.inverse();
    Eigen::Matrix2d W2 = S2.inverse();
    Eigen::Vector2d xf = (W1 + W2).inverse() * (W1 * x1 + W2 * x2);

    std::cout << "Fused estimate: " << xf.transpose() << std::endl;
    return 0;
}
      

6.3 Java (OpenCV Java + EJML)

import org.opencv.core.*;
import org.opencv.imgcodecs.Imgcodecs;
import org.opencv.imgproc.Imgproc;
import org.opencv.features2d.*;
import org.ejml.simple.SimpleMatrix;

public class PerceptionPipeline {
    static { System.loadLibrary(Core.NATIVE_LIBRARY_NAME); }

    public static void main(String[] args) {
        // (1) Acquire
        Mat img = Imgcodecs.imread("scene.png", Imgcodecs.IMREAD_GRAYSCALE);

        // (2) Preprocess
        Mat imgS = new Mat();
        Imgproc.GaussianBlur(img, imgS, new Size(5,5), 1.2);

        // (3) Feature extraction (goodFeaturesToTrack)
        Mat corners = new Mat();
        Imgproc.goodFeaturesToTrack(imgS, corners, 200, 0.01, 8);

        // (4) Weighted fusion with EJML
        SimpleMatrix x1 = new SimpleMatrix(2,1,true, new double[]{1.2,0.5});
        SimpleMatrix x2 = new SimpleMatrix(2,1,true, new double[]{1.0,0.9});
        SimpleMatrix S1 = SimpleMatrix.diag(0.04, 0.09);
        SimpleMatrix S2 = SimpleMatrix.diag(0.16, 0.04);

        SimpleMatrix W1 = S1.invert();
        SimpleMatrix W2 = S2.invert();
        SimpleMatrix xf = (W1.plus(W2)).invert().mult(W1.mult(x1).plus(W2.mult(x2)));

        System.out.println("Fused estimate: " + xf);
    }
}
      

6.4 MATLAB / Simulink

% (1) Acquire
img = imread('scene.png');
if size(img,3) == 3, img = rgb2gray(img); end

% (2) Preprocess
img_s = imgaussfilt(img, 1.2);

% (3) Feature extraction
corners = detectMinEigenFeatures(img_s, 'MinQuality', 0.01);
pts = corners.selectStrongest(200).Location;

% (4) Weighted least squares fusion
x1 = [1.2; 0.5];  x2 = [1.0; 0.9];
S1 = diag([0.04, 0.09]);  S2 = diag([0.16, 0.04]);

W1 = inv(S1); W2 = inv(S2);
xf = inv(W1 + W2) * (W1*x1 + W2*x2);

disp('Fused estimate:'); disp(xf);

% Simulink note:
% Use blocks: Image From File -> Gaussian Filter -> Corner Detector
% -> MATLAB Function block implementing xf formula.
      

These snippets mirror the conceptual pipeline stages from Sections 2–4. In larger robots, each stage is a separate software module with measured execution times and explicit data interfaces.

Problem 1 (Derive deterministic fusion weights): Two sensors provide unbiased estimates \( \hat{\mathbf{x}}_1, \hat{\mathbf{x}}_2 \) with covariances \( \mathbf{\Sigma}_1, \mathbf{\Sigma}_2 \). Among all linear unbiased estimators \( \hat{\mathbf{x}} = \mathbf{A}_1\hat{\mathbf{x}}_1 + \mathbf{A}_2\hat{\mathbf{x}}_2 \) satisfying \( \mathbf{A}_1+\mathbf{A}_2=\mathbf{I} \), find the estimator minimizing \( \operatorname{tr}(\mathbf{\Sigma}_f) \).

Solution: Substitute \( \mathbf{A}_2=\mathbf{I}-\mathbf{A}_1 \) into \( \mathbf{\Sigma}_f \):

\[ \mathbf{\Sigma}_f(\mathbf{A}_1)= \mathbf{A}_1\mathbf{\Sigma}_1\mathbf{A}_1^\top +(\mathbf{I}-\mathbf{A}_1)\mathbf{\Sigma}_2(\mathbf{I}-\mathbf{A}_1)^\top. \]

Differentiate \( \operatorname{tr}(\mathbf{\Sigma}_f) \) w.r.t. \( \mathbf{A}_1 \) and set to zero (matrix calculus):

\[ 2\mathbf{A}_1\mathbf{\Sigma}_1 - 2(\mathbf{I}-\mathbf{A}_1)\mathbf{\Sigma}_2 = \mathbf{0} \;\Rightarrow\; \mathbf{A}_1(\mathbf{\Sigma}_1+\mathbf{\Sigma}_2)=\mathbf{\Sigma}_2. \]

Thus \( \mathbf{A}_1 = \mathbf{\Sigma}_2(\mathbf{\Sigma}_1+\mathbf{\Sigma}_2)^{-1} \), \( \mathbf{A}_2 = \mathbf{\Sigma}_1(\mathbf{\Sigma}_1+\mathbf{\Sigma}_2)^{-1} \). Substituting gives the fused form from Section 4.

Problem 2 (Pipeline latency): A pipeline has serial stages with times \( t_1=5 \) ms, \( t_2=7 \) ms, \( t_3=6 \) ms, then two parallel branches with times \( t_4=10 \) ms and \( t_5=4 \) ms that rejoin before output stage \( t_6=3 \) ms. Compute \( L_{\text{DAG}} \).

Solution: The critical path includes the slower parallel branch:

\[ L_{\text{DAG}} = t_1+t_2+t_3+\max(t_4,t_5)+t_6 = 5+7+6+\max(10,4)+3 = 31 \text{ ms}. \]

If the control loop is 25 ms, the pipeline violates timing and must be simplified or parallelized further.

Problem 3 (Gradient noise amplification): Assume independent pixel noise with variance \( \sigma_n^2 \). Show that the variance of a forward-difference gradient \( I_u = I(u+1,v)-I(u,v) \) includes an added term \( 2\sigma_n^2 \).

Solution: Write \( I=I^\star+n \). Then \( I_u = I_u^\star + n(u+1,v)-n(u,v) \). Because independent random variables add variances,

\[ \operatorname{Var}[n(u+1,v)-n(u,v)] = \operatorname{Var}[n(u+1,v)] + \operatorname{Var}[n(u,v)] = \sigma_n^2 + \sigma_n^2 = 2\sigma_n^2. \]

Therefore gradients increase noise variance, motivating smoothing first.

Problem 4 (Time synchronization error): Two sensors sample a moving scalar signal \( s(t) \). Sensor 2 lags by \( \Delta t \). Using first-order Taylor expansion, derive the measurement mismatch \( e \approx \dot{s}(t)\Delta t \).

Solution: Sensor 1 measures \( s(t) \), sensor 2 measures \( s(t-\Delta t) \). Expand:

\[ s(t-\Delta t) \approx s(t) - \dot{s}(t)\Delta t. \]

Hence mismatch is \( e = s(t)-s(t-\Delta t) \approx \dot{s}(t)\Delta t \). Faster signals (large \( \dot{s}(t) \)) require tighter synchronization.

We assembled the perception ideas of this chapter into real robot pipelines. We modeled acquisition, preprocessing, feature extraction, and deterministic estimation/fusion using linearization and least squares. We also analyzed end-to-end latency through critical-path timing, emphasizing practical accuracy–compute–delay trade-offs. These principles are the basis of robust perception stacks in modern robots.