In semantic segmentation, the perception system assigns a class label to every pixel of an image (or point in a cloud). For an RGB(-D) image defined on a discrete grid \( \Omega \subset \mathbb{Z}^2 \) and a finite set of semantic labels \( \mathcal{C} = \{1,\dots,C\} \), a segmentation is a function

\[ y : \Omega \to \mathcal{C}, \quad y(p) = c \;\text{means pixel } p \text{ belongs to class } c . \]

For manipulation, typical labels include object of interest, support surface (table, shelf), background, and sometimes obstacles. A segmentation determines which 3D points are safe to grasp, which belong to the table, and which must be avoided. Connecting to previous lessons in this chapter, we interpret segmentation as an intermediate representation between raw point clouds and object pose estimation or tracking.

A common representation is a one-hot tensor \( \mathbf{Y} \in \{0,1\}^{|\Omega|\times C} \), where pixel \( p \) and class \( c \) entry is

\[ Y_{p,c} = \begin{cases} 1, & \text{if } y(p) = c,\\ 0, & \text{otherwise.} \end{cases} \]

We will focus on discriminative models \( p_\theta(y \mid x) \) that directly map an image (or RGB-D observation) \( x \) to a distribution over labelings \( y \).

flowchart TD
  I["RGB-D image"] --> F["Feature extractor (CNN or similar)"]
  F --> SCORES["Per-pixel class scores"]
  SCORES --> SMASK["2D semantic mask"]
  SMASK --> P3D["Back-project to 3D points"]
  P3D --> OBJ["Object point cloud for target class"]
  OBJ --> G["Grasp / contact region selection"]
  G --> MPC["Manipulator planner uses segmented object"]
        

Let \( \mathbf{Y} = \{Y_p\}_{p\in\Omega} \) be discrete random variables taking values in \( \mathcal{C} \). A widely used conditional random field (CRF) model assigns a Gibbs distribution conditioned on the observed image \( x \):

\[ p_\theta(y \mid x) = \frac{1}{Z(x;\theta)} \exp\!\big(-E_\theta(y,x)\big), \]

where \( Z(x;\theta) \) is the partition function and the energy typically decomposes into unary and pairwise terms over a pixel neighborhood system \( \mathcal{N} \):

\[ E_\theta(y,x) = \sum_{p\in\Omega} \psi_p(y_p;x,\theta) + \sum_{(p,q)\in\mathcal{N}} \psi_{p,q}(y_p,y_q;x,\theta). \]

Unary terms \( \psi_p(y_p;x,\theta) \) encode the cost of assigning class \( y_p \) to pixel \( p \) based on local appearance or deep features. In modern systems these often come from a deep network:

\[ \psi_p(c;x,\theta) = -\log p_\theta(Y_p = c \mid x), \]

where \( p_\theta(Y_p = c \mid x) \) is produced by a softmax layer.

Pairwise terms enforce spatial coherence. A common Potts model is

\[ \psi_{p,q}(y_p,y_q;x,\theta) = \lambda \,\mathbf{1}\{y_p \neq y_q\} \exp\!\left( -\frac{\|I_p - I_q\|_2^2}{2\sigma_I^2} -\frac{\|p - q\|_2^2}{2\sigma_p^2} \right), \]

where \( I_p \) is the color at pixel \( p \). Neighboring pixels with similar colors and positions are encouraged to share labels.

Maximum a posteriori (MAP) inference solves

\[ \hat{y} = \arg\max_{y} p_\theta(y\mid x) = \arg\min_{y} E_\theta(y,x). \]

For particular classes of energies (submodular pairwise potentials) this minimization can be solved exactly by s-t graph cuts. In robotics, graph-cut based segmentation remains attractive for real-time CPU implementations when deep models are not available, or as a refinement of network predictions.

Most current systems use fully convolutional networks (FCNs) or encoder–decoder architectures to produce dense label probabilities. Given an image \( x \), a network with parameters \( \theta \) outputs per-pixel logits \( z_{p,c}(x;\theta) \), organized as a tensor \( \mathbf{Z} \in \mathbb{R}^{|\Omega|\times C} \).

Applying the softmax at each pixel:

\[ p_\theta(Y_p = c \mid x) = \frac{\exp\!\big(z_{p,c}(x;\theta)\big)} {\sum_{c'=1}^{C} \exp\!\big(z_{p,c'}(x;\theta)\big)}. \]

For a single training example with ground-truth labels \( y_p \), the pixel-wise cross-entropy loss is

\[ \ell(x,y;\theta) = - \sum_{p\in\Omega} \log p_\theta(Y_p = y_p \mid x). \]

For class-imbalanced scenes (e.g., small objects on a large table), weighted cross-entropy can reduce bias toward dominant classes:

\[ \ell_{\text{wce}}(x,y;\theta) = - \sum_{p\in\Omega} w_{y_p}\, \log p_\theta(Y_p = y_p \mid x), \]

where \( w_c > 0 \) emphasize rare classes.

The gradient of the unweighted loss with respect to logits has a simple closed form. Let \( \ell_p(\theta) = -\log p_\theta(Y_p = y_p \mid x) \) and denote the Kronecker delta by \( \delta_{c,y_p} \). Then

\[ \frac{\partial \ell_p}{\partial z_{p,c}} = p_\theta(Y_p = c \mid x) - \delta_{c,y_p}, \]

which implies that backpropagation for segmentation is structurally identical to that for multi-class classification, but applied at every pixel.

For evaluation, the intersection-over-union (IoU) for class \( c \) is defined by confusion counts \( \text{TP}_c, \text{FP}_c, \text{FN}_c \):

\[ \operatorname{IoU}_c = \frac{\text{TP}_c} {\text{TP}_c + \text{FP}_c + \text{FN}_c}, \quad \text{mIoU} = \frac{1}{C}\sum_{c=1}^{C} \operatorname{IoU}_c. \]

IoU is more informative than pixel accuracy for robotics, since imprecise boundaries around objects of interest can significantly affect grasp success, even if the majority of background pixels are correctly classified.

Manipulation requires reasoning in 3D. Given synchronized RGB and depth images and a calibrated pinhole camera with intrinsic matrix \( \mathbf{K} \in \mathbb{R}^{3\times 3} \), each pixel \( p = (u,v) \) with depth \( d_p > 0 \) back-projects to a 3D point in the camera frame:

\[ \tilde{p} = \begin{bmatrix} u \\ v \\ 1 \end{bmatrix},\quad \mathbf{P}_{\text{cam}}(p) = d_p\,\mathbf{K}^{-1}\tilde{p} \in \mathbb{R}^3. \]

If the homogeneous transform from camera frame to robot base frame is known from calibration, \( \mathbf{T}^{\text{base}}_{\text{cam}} \in \mathrm{SE}(3) \), we obtain

\[ \tilde{\mathbf{P}}_{\text{cam}} = \begin{bmatrix} \mathbf{P}_{\text{cam}} \\ 1 \end{bmatrix},\quad \tilde{\mathbf{P}}_{\text{base}} = \mathbf{T}^{\text{base}}_{\text{cam}} \tilde{\mathbf{P}}_{\text{cam}}, \]

and the 3D position in the robot base frame is \( \mathbf{P}_{\text{base}} \), the first three components of \( \tilde{\mathbf{P}}_{\text{base}} \).

Given a semantic mask for class \( c^\star \), e.g. a target object, define the set of pixels belonging to that class:

\[ \mathcal{M}_{c^\star} = \{ p\in\Omega : y(p) = c^\star \}. \]

The corresponding object point cloud in the base frame is

\[ \mathcal{P}_{c^\star} = \{ \mathbf{P}_{\text{base}}(p) : p \in \mathcal{M}_{c^\star} \}. \]

A simple 3D descriptor for grasp planning is the centroid and covariance of this cloud:

\[ \boldsymbol{\mu} = \frac{1}{|\mathcal{P}_{c^\star}|} \sum_{\mathbf{P}\in\mathcal{P}_{c^\star}} \mathbf{P}, \quad \mathbf{\Sigma} = \frac{1}{|\mathcal{P}_{c^\star}|} \sum_{\mathbf{P}\in\mathcal{P}_{c^\star}} (\mathbf{P}-\boldsymbol{\mu})(\mathbf{P}-\boldsymbol{\mu})^\top. \]

These quantities summarize object size and orientation and can be passed to a grasp synthesis module (later chapters) or used to place heuristic top-down grasps aligned with principal directions.

In practice, semantic segmentation runs as a node within a robotics middleware (e.g., ROS). The node subscribes to camera topics, performs inference on a GPU or CPU, and publishes dense labels and object masks in both 2D and 3D. A typical data flow is:

flowchart TD
  A["Camera topics '/camera/rgb' + '/camera/depth'"] --> B["Segmentation node"]
  B --> C["2D class probabilities"]
  C --> D["2D semantic mask (target class)"]
  B --> E["Per-pixel depth + intrinsics"]
  E --> F["3D back-projection in base frame"]
  D --> F
  F --> G["3D object mask point cloud"]
  G --> H["Grasp pose generator"]
  H --> J["Manipulator motion planner / controller"]
        

The following sections give language-specific implementations of the segmentation core and its interaction with robotic software stacks.

We implement a compact FCN-style network in PyTorch and wrap it as a ROS node. For brevity, we consider a fixed number of classes and an input image already resized to a canonical resolution.

import torch
import torch.nn as nn
import torch.nn.functional as F
import numpy as np
import cv2

# Optional: ROS interface (comment out if not using ROS)
try:
    import rospy
    from sensor_msgs.msg import Image
    from cv_bridge import CvBridge
    ROS_AVAILABLE = True
except ImportError:
    ROS_AVAILABLE = False


class SmallSegNet(nn.Module):
    def __init__(self, num_classes: int = 4):
        super().__init__()
        # Encoder
        self.conv1 = nn.Conv2d(3, 32, kernel_size=3, padding=1)
        self.conv2 = nn.Conv2d(32, 64, kernel_size=3, padding=1)
        self.pool = nn.MaxPool2d(kernel_size=2, stride=2)
        # Bottleneck
        self.conv3 = nn.Conv2d(64, 128, kernel_size=3, padding=1)
        # Decoder (simple upsampling)
        self.up1 = nn.ConvTranspose2d(128, 64, kernel_size=2, stride=2)
        self.up2 = nn.ConvTranspose2d(64, 32, kernel_size=2, stride=2)
        self.logits = nn.Conv2d(32, num_classes, kernel_size=1)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        # x: (N, 3, H, W) in [0,1]
        x = F.relu(self.conv1(x))
        x = self.pool(x)          # (N, 32, H/2, W/2)
        x = F.relu(self.conv2(x))
        x = self.pool(x)          # (N, 64, H/4, W/4)
        x = F.relu(self.conv3(x)) # (N, 128, H/4, W/4)
        x = F.relu(self.up1(x))   # (N, 64, H/2, W/2)
        x = F.relu(self.up2(x))   # (N, 32, H,   W)
        x = self.logits(x)        # (N, C, H, W)
        return x


def load_model(weights_path: str, num_classes: int) -> SmallSegNet:
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    model = SmallSegNet(num_classes=num_classes).to(device)
    state_dict = torch.load(weights_path, map_location=device)
    model.load_state_dict(state_dict)
    model.eval()
    return model


def segment_rgb_image(model: SmallSegNet, rgb: np.ndarray) -> np.ndarray:
    """
    rgb: uint8 array with shape (H, W, 3), BGR or RGB.
    Returns: uint8 mask with shape (H, W), each entry is class index.
    """
    device = next(model.parameters()).device
    img = cv2.cvtColor(rgb, cv2.COLOR_BGR2RGB)
    img_f = img.astype(np.float32) / 255.0
    tensor = torch.from_numpy(img_f.transpose(2, 0, 1)).unsqueeze(0).to(device)
    with torch.no_grad():
        logits = model(tensor)
        probs = F.softmax(logits, dim=1)
        mask = torch.argmax(probs, dim=1).squeeze(0).cpu().numpy().astype(np.uint8)
    return mask


def backproject_to_base(mask: np.ndarray,
                        depth: np.ndarray,
                        K: np.ndarray,
                        T_base_cam: np.ndarray,
                        target_class: int) -> np.ndarray:
    """
    Back-project pixels of target_class into 3D base frame.
    depth: depth image in meters, shape (H, W).
    K: 3x3 camera intrinsics.
    T_base_cam: 4x4 homogeneous transform.
    Returns: array of 3D points, shape (N, 3).
    """
    ys, xs = np.where(mask == target_class)
    if ys.size == 0:
        return np.zeros((0, 3), dtype=np.float32)

    # Build homogeneous pixel coordinates
    ones = np.ones_like(xs, dtype=np.float32)
    pix = np.stack([xs.astype(np.float32),
                    ys.astype(np.float32),
                    ones], axis=0)  # (3, N)

    depths = depth[ys, xs].astype(np.float32)  # (N,)
    K_inv = np.linalg.inv(K)
    # Camera-frame coordinates
    P_cam = (K_inv @ pix) * depths  # (3, N)
    P_cam_h = np.vstack([P_cam, ones])  # (4, N)
    # Transform to base frame
    P_base_h = T_base_cam @ P_cam_h
    P_base = P_base_h[:3, :].T  # (N, 3)
    return P_base


# Optional ROS node wrapper
def run_ros_node(weights_path: str,
                 num_classes: int,
                 target_class: int):
    if not ROS_AVAILABLE:
        raise RuntimeError("ROS and cv_bridge are not available in this environment.")
    rospy.init_node("semantic_segmentation_node")
    bridge = CvBridge()
    model = load_model(weights_path, num_classes)

    K = np.eye(3, dtype=np.float32)
    T_base_cam = np.eye(4, dtype=np.float32)

    pub_mask = rospy.Publisher("/segmentation/mask", Image, queue_size=1)

    def callback(msg: Image):
        rgb = bridge.imgmsg_to_cv2(msg, desired_encoding="bgr8")
        mask = segment_rgb_image(model, rgb)
        mask_msg = bridge.cv2_to_imgmsg(mask, encoding="mono8")
        pub_mask.publish(mask_msg)

    sub = rospy.Subscriber("/camera/rgb/image_raw", Image, callback, queue_size=1)
    rospy.spin()


if __name__ == "__main__":
    # Example standalone usage (no ROS)
    dummy_img = np.zeros((480, 640, 3), dtype=np.uint8)
    model = SmallSegNet(num_classes=4)
    model.eval()
    mask = segment_rgb_image(model, dummy_img)
    print("Mask shape:", mask.shape)
      

For training, the loss function from Section 3 can be implemented directly with torch.nn.CrossEntropyLoss, optionally with class weights. The backproject_to_base function then converts a selected class mask into a 3D point cloud usable by manipulation modules developed in later chapters.

In C++, OpenCV's DNN module can load an ONNX semantic segmentation model and integrate it with ROS. The following simplified node loads a network and publishes an 8-bit mask image:

#include <ros/ros.h>
#include <image_transport/image_transport.h>
#include <cv_bridge/cv_bridge.h>
#include <sensor_msgs/Image.h>

#include <opencv2/opencv.hpp>
#include <opencv2/dnn.hpp>

class SegmentationNode {
public:
    SegmentationNode(const std::string& onnx_path,
                     int num_classes)
        : it_(nh_), num_classes_(num_classes)
    {
        net_ = cv::dnn::readNetFromONNX(onnx_path);
        sub_ = it_.subscribe("/camera/rgb/image_raw", 1,
                             &SegmentationNode::imageCallback, this);
        pub_ = it_.advertise("/segmentation/mask", 1);
    }

    void spin() {
        ros::spin();
    }

private:
    void imageCallback(const sensor_msgs::ImageConstPtr& msg) {
        cv_bridge::CvImageConstPtr cv_ptr;
        try {
            cv_ptr = cv_bridge::toCvShare(msg, "bgr8");
        } catch (cv_bridge::Exception& e) {
            ROS_ERROR("cv_bridge exception: %s", e.what());
            return;
        }

        cv::Mat img = cv_ptr->image;
        cv::Mat blob = cv::dnn::blobFromImage(
            img, 1.0 / 255.0, cv::Size(320, 240),
            cv::Scalar(0, 0, 0), true, false
        );

        net_.setInput(blob);
        cv::Mat scores = net_.forward(); // shape: (1, C, H, W)

        int C = num_classes_;
        int H = scores.size[2];
        int W = scores.size[3];

        cv::Mat mask(H, W, CV_8UC1);
        const float* data = reinterpret_cast<const float*>(scores.data);

        for (int y = 0; y < H; ++y) {
            for (int x = 0; x < W; ++x) {
                int best_class = 0;
                float best_score = data[y * W * C + x * C + 0];
                for (int c = 1; c < C; ++c) {
                    float val = data[y * W * C + x * C + c];
                    if (val > best_score) {
                        best_score = val;
                        best_class = c;
                    }
                }
                mask.at<unsigned char>(y, x) = static_cast<unsigned char>(best_class);
            }
        }

        cv::Mat mask_resized;
        cv::resize(mask, mask_resized, img.size(), 0, 0, cv::INTER_NEAREST);

        sensor_msgs::ImagePtr out_msg =
            cv_bridge::CvImage(msg->header, "mono8", mask_resized).toImageMsg();
        pub_.publish(out_msg);
    }

    ros::NodeHandle nh_;
    image_transport::ImageTransport it_;
    image_transport::Subscriber sub_;
    image_transport::Publisher pub_;
    cv::dnn::Net net_;
    int num_classes_;
};

int main(int argc, char** argv) {
    ros::init(argc, argv, "semantic_segmentation_node_cpp");
    std::string onnx_path = "/path/to/model.onnx";
    int num_classes = 4;
    SegmentationNode node(onnx_path, num_classes);
    node.spin();
    return 0;
}
      

For 3D projection, one can subscribe to the depth image and camera info, then implement the back-projection equations from Section 4 using cv::Mat and Eigen or another linear algebra library for homogeneous transforms.

Java is less common in low-level manipulation, but Java-based robotic stacks (and Android-based systems) can still benefit from segmentation. The following example performs a simple two-class segmentation by color thresholding using OpenCV's Java API; it can serve as a placeholder for a deep model:

import org.opencv.core.Core;
import org.opencv.core.CvType;
import org.opencv.core.Mat;
import org.opencv.core.Scalar;
import org.opencv.imgcodecs.Imgcodecs;
import org.opencv.imgproc.Imgproc;

public class SimpleColorSegmentation {
    static {
        System.loadLibrary(Core.NATIVE_LIBRARY_NAME);
    }

    public static void main(String[] args) {
        String inputPath = "scene.png";
        Mat bgr = Imgcodecs.imread(inputPath);
        if (bgr.empty()) {
            System.err.println("Could not load image.");
            return;
        }

        Mat hsv = new Mat();
        Imgproc.cvtColor(bgr, hsv, Imgproc.COLOR_BGR2HSV);

        // Example: segment pixels that are "table" colored (e.g., brown)
        Scalar lower = new Scalar(5, 50, 50);
        Scalar upper = new Scalar(20, 255, 255);
        Mat mask = new Mat(hsv.rows(), hsv.cols(), CvType.CV_8UC1);
        Core.inRange(hsv, lower, upper, mask);

        Imgcodecs.imwrite("mask_table.png", mask);
        System.out.println("Saved binary mask to mask_table.png");
    }
}
      

A deep segmentation network can be accessed via Java bindings to libraries such as TensorFlow or through JNI wrappers around C++ code. The key robotics ideas (projection to 3D, integration with planners) remain governed by the mathematical formulation in Section 4.

MATLAB's Computer Vision Toolbox provides high-level functions for semantic segmentation that integrate naturally with Simulink models of robotic systems. Assuming a pre-trained network net for a set of classes:

% Load RGB image
rgb = imread("scene.png");

% Load pre-trained semantic segmentation network (e.g., DeepLab v3+)
load("trainedSegNet.mat","net");

% Perform pixel-wise classification
[C, scores] = semanticseg(rgb, net);

% Visualize one class mask, e.g., "targetObject"
targetClass = "targetObject";
mask = C == targetClass;
imshowpair(rgb, mask, "montage");

% Suppose we have a registered depth image and camera intrinsics K
depth = imread("depth.png");      % depth in meters stored as single
depth = single(depth);
K = [fx 0 cx; 0 fy cy; 0 0 1];

% Back-project mask pixels into 3D camera frame
[H, W] = size(mask);
[u, v] = meshgrid(0:W-1, 0:H-1);
u = single(u); v = single(v);

idx = find(mask);
u_vec = u(idx);
v_vec = v(idx);
d_vec = depth(idx);

pix = [u_vec.'; v_vec.'; ones(1, numel(u_vec), "single")];
Kinv = inv(K);
P_cam = Kinv * pix .* d_vec.;

% Transform into base frame with T_base_cam (4x4)
T_base_cam = eye(4, "single");
P_cam_h = [P_cam; ones(1, size(P_cam,2), "single")];
P_base_h = T_base_cam * P_cam_h;
P_base = P_base_h(1:3,:).';

% P_base is an N-by-3 array of 3D points for the target object
      

In Simulink, a block diagram can be constructed where a semantic segmentation block feeds a point cloud processing block that computes object pose and passes it to a manipulator controller subsystem, naturally connecting perception to actuation within a single model-based environment.

Wolfram Mathematica supports image segmentation via both classical and neural approaches. A basic semantic segmentation experiment can be carried out as follows:

(* Load an RGB image *)
img = Import["scene.png"];

(* Simple clustering-based segmentation into k regions *)
k = 4;
seg = Colorize @ ImageSegmentation[img, k];

(* Assign integer labels to each region *)
labels = MorphologicalComponents[seg];

(* Visualize a particular region, e.g., label 1 *)
targetLabel = 1;
mask = Binarize[labels, {targetLabel - 0.5, targetLabel + 0.5}];
Row[{img, Colorize[mask]}]

(* Example of defining a small fully convolutional network *)
net = NetChain[{
    ConvolutionLayer[16, {3, 3}, "PaddingSize" -> 1],
    ElementwiseLayer["ReLU"],
    ConvolutionLayer[32, {3, 3}, "PaddingSize" -> 1],
    ElementwiseLayer["ReLU"],
    ConvolutionLayer[4, {1, 1}] (* 4 semantic classes *)
},
   "Input" -> NetEncoder[{"Image", ColorSpace -> "RGB"}],
   "Output" -> NetDecoder[{"Class", Range[4]}]
];

(* Training would require a dataset of {image, labelImage} pairs *)
      

For robotics use, the segmented 2D mask can be exported and combined with external tools (e.g., a separate kinematics or motion planning pipeline) using Wolfram's symbolic and numeric capabilities for reasoning about geometry.

Problem 1 (Gradient of Pixel-wise Cross-Entropy): Consider a segmentation network producing logits \( z_{p,c} \) at pixel \( p \). Let \( p_{p,c} = \exp(z_{p,c}) / \sum_{c'} \exp(z_{p,c'}) \) and the loss \( \ell_p = -\log p_{p,y_p} \). Derive \( \partial \ell_p / \partial z_{p,c} \).

Solution: Write \( \ell_p = -z_{p,y_p} + \log(\sum_{c'} \exp(z_{p,c'})) \). Differentiating with respect to \( z_{p,c} \) gives

\[ \frac{\partial \ell_p}{\partial z_{p,c}} = -\mathbf{1}\{c = y_p\} + \frac{\exp(z_{p,c})}{\sum_{c'} \exp(z_{p,c'})} = p_{p,c} - \mathbf{1}\{c = y_p\}. \]

Stacking over all pixels and classes recovers the standard softmax cross-entropy gradient used in deep learning libraries.

Problem 2 (IoU in Terms of Confusion Matrix): Let \( n_{ij} \) be the number of pixels with true class \( i \) and predicted class \( j \). Express \( \operatorname{IoU}_c \) for class \( c \) in terms of \( n_{ij} \).

Solution: For class \( c \), we have \( \text{TP}_c = n_{cc} \). False positives are predictions that equal \( c \) but ground truth is not \( c \): \( \text{FP}_c = \sum_{i\neq c} n_{ic} \). False negatives are ground truth pixels of class \( c \) predicted as something else: \( \text{FN}_c = \sum_{j\neq c} n_{cj} \). Therefore

\[ \operatorname{IoU}_c = \frac{n_{cc}} {n_{cc} + \sum_{i\neq c} n_{ic} + \sum_{j\neq c} n_{cj}}. \]

Problem 3 (Unary-Only CRF and Independent Pixels): Show that if the pairwise terms vanish, \( \psi_{p,q} \equiv 0 \), then the MAP labeling decomposes into independent per-pixel decisions.

Solution: With no pairwise terms, the energy reduces to

\[ E(y,x) = \sum_{p\in\Omega} \psi_p(y_p;x). \]

Thus \( \hat{y} = \arg\min_{y} \sum_{p} \psi_p(y_p;x) \). Since \( y_p \) only appears in \( \psi_p \), the minimization factorizes:

\[ \hat{y}_p = \arg\min_{c\in\mathcal{C}} \psi_p(c;x), \quad \forall p\in\Omega. \]

This is exactly the per-pixel classification rule \( \hat{y}_p = \arg\max_c p_\theta(Y_p = c \mid x) \). Pairwise terms therefore encode all spatial coupling in the CRF.

Problem 4 (Back-Projection Jacobian): Consider the back-projection function \( \mathbf{P}_{\text{cam}}(u,v,d) = d\,\mathbf{K}^{-1}[u,v,1]^\top \). Compute the Jacobian of \( \mathbf{P}_{\text{cam}} \) with respect to the depth \( d \) and discuss its implication for depth noise near the camera.

Solution: Write \( \mathbf{P}_{\text{cam}}(u,v,d) = d\,\mathbf{q}(u,v) \), where \( \mathbf{q}(u,v) = \mathbf{K}^{-1}[u,v,1]^\top \) is independent of \( d \). Then

\[ \frac{\partial \mathbf{P}_{\text{cam}}}{\partial d} = \mathbf{q}(u,v). \]

Thus a small depth perturbation \( \delta d \) induces a 3D perturbation \( \delta \mathbf{P}_{\text{cam}} = \mathbf{q}(u,v)\,\delta d \). Close to the optical axis, \( \mathbf{q}(u,v) \) is almost aligned with the camera's z-axis, so small depth errors mainly affect distance along the line of sight. This is critical when using segmented depth points to estimate object height above a table: errors in \( d \) directly propagate to grasp pose estimation.

Problem 5 (Effect of Class Weights on the Loss Landscape): Consider the weighted loss \( \ell_{\text{wce}} \) from Section 3. Show that increasing \( w_c \) for a rare class \( c \) effectively rescales the gradient contributions from pixels of that class, and interpret this in terms of optimization dynamics.

Solution: For a pixel with label \( y_p = c \), the derivative with respect to \( z_{p,k} \) is

\[ \frac{\partial \ell_{\text{wce}}}{\partial z_{p,k}} = w_c\big(p_{p,k} - \mathbf{1}\{k=c\}\big). \]

Relative to the unweighted case, every pixel of class \( c \) contributes a factor \( w_c \) to the gradient. Larger \( w_c \) therefore increases the effective learning rate for that class, causing optimization to focus on reducing errors for rare classes, at the cost of slower convergence for abundant classes.

We formulated semantic segmentation as a structured prediction problem over pixel-wise labels, introduced CRF-style energy models, and derived the gradients underlying deep segmentation networks. We showed how to evaluate segmentations with IoU-based metrics, and how to project 2D masks into the robot base frame to obtain object point clouds for manipulation. Finally, we implemented segmentation components in Python, C++, Java, MATLAB/Simulink, and Wolfram Mathematica, emphasizing their integration with robotic middleware and 3D geometry. Subsequent lessons will build on these ideas to represent affordances and action-conditioned perception.