In rigid-body manipulation, a robot typically estimates a relatively low-dimensional configuration (pose, joint angles) with sufficient accuracy to treat the state as approximately known. For deformable objects (cloth, cables, bags), the underlying configuration space is extremely high-dimensional, and large parts of the object are often occluded or poorly sensed.

Planning in this setting must explicitly reason about partial observability: the robot chooses actions \( a_t \) based on a belief distribution over latent states \( s_t \), rather than a single point estimate. This leads naturally to the POMDP formulation introduced earlier in the course, but specialized here to deformable object manipulation.

flowchart TD
  S["Latent deformable state s_t (full mesh/curve)"]
    --> P["Perception pipeline: depth, tactile, vision features"]
  P --> O["Observation o_t (partial, noisy)"]
  O --> B["Belief update: b_t = p(s_t | history)"]
  B --> PL["Planner in belief space"]
  PL --> A["Action a_t (grasp, pull, push)"]
  A --> S2["Next latent state s_(t+1)"]
  S2 --> P
        

The main questions we address are:

• How to formally define a POMDP for deformable objects?
• How to represent and update beliefs over very high-dimensional states?
• How to plan approximately in belief space while retaining useful guarantees?

Recall that a finite-horizon POMDP is a tuple \( (\mathcal{S},\mathcal{A},\mathcal{O},T,Z,R,\gamma) \) with state space \( \mathcal{S} \), action space \( \mathcal{A} \), observation space \( \mathcal{O} \), transition kernel \( T(s' \mid s,a) \), observation kernel \( Z(o \mid s',a) \), reward \( R(s,a) \), and discount \( \gamma \in (0,1) \).

For deformable manipulation, we typically factor the state as \( s_t = (x_t, q_t) \), where:

• \( x_t \): configuration of the deformable (e.g., cloth mesh);
• \( q_t \): robot configuration (joints, gripper state).

The robot chooses actions \( a_t \in \mathcal{A} \) (joint velocities, Cartesian motions, grasp toggles). We write the dynamics as

\[ p(s_{t+1} \mid s_t, a_t) \;=\; T(s_{t+1} \mid s_t, a_t), \quad p(o_{t+1} \mid s_{t+1}, a_t) \;=\; Z(o_{t+1} \mid s_{t+1}, a_t). \]

A key difficulty is that \( x_t \) is extremely large. In practice we work with a lower-dimensional state representation from Lesson 2, \( \phi(x_t) \in \mathbb{R}^d \), and define \( s_t = (\phi(x_t), q_t) \) as the POMDP state. Planning then seeks a policy \( \pi \) mapping beliefs to actions maximizing expected discounted reward:

\[ J^\pi(b_0) \;=\; \mathbb{E}\Bigg[ \sum_{t=0}^{\infty} \gamma^t R(s_t, a_t) \,\Big|\, b_0, \pi \Bigg], \quad \pi^\star \in \arg\max_\pi J^\pi(b_0), \]

where \( b_0 \) is the initial belief over states.

The belief at time \( t \) is a probability distribution \( b_t(s) = p(s_t = s \mid h_t) \) over states given the history \( h_t = (o_0,a_0,o_1,a_1,\dots,o_t) \). After taking action \( a_t \) and observing \( o_{t+1} \), the belief is updated via Bayes' rule:

\[ \tilde{b}_{t+1}(s') \;=\; \sum_{s \in \mathcal{S}} T(s' \mid s, a_t)\,b_t(s), \quad b_{t+1}(s') \;=\; \eta\, Z(o_{t+1} \mid s', a_t)\,\tilde{b}_{t+1}(s'), \]

where \( \eta \) is a normalizing constant ensuring \( \sum_{s'} b_{t+1}(s') = 1 \).

In continuous deformable state spaces, these sums become integrals and \( b_t \) is typically approximated by:

• Gaussian beliefs: \( b_t \approx \mathcal{N}(\mu_t, \Sigma_t) \);
• particle beliefs: \( b_t \approx \sum_{i=1}^N w_t^i \delta_{s_t^i} \);
• learned latent distributions in a low-dimensional embedding.

A fundamental result is that the belief is a sufficient statistic for optimal control:

Proposition 1 (Belief Sufficiency). For a POMDP with dynamics \( T \) and observations \( Z \), the conditional distribution \( b_t = p(s_t \mid h_t) \) is a sufficient statistic for computing an optimal policy. In other words, there exists an optimal policy \( \pi^\star \) that depends on \( h_t \) only through \( b_t \).

Sketch of proof. Consider the belief-state process \( \{b_t\} \). Using the update above, the next belief \( b_{t+1} \) depends only on \( b_t \), the chosen action \( a_t \), and the new observation \( o_{t+1} \). Thus the process \( \{b_t\} \) is Markov given actions. The expected return from time \( t \) onward can be written as a function of \( b_t \) alone by marginalizing over the latent state \( s_t \). Therefore, dynamic programming can be carried out on the belief space, and an optimal policy need not depend on the entire history.

Define the value function over beliefs under a policy \( \pi \) as

\[ V^\pi(b) \;=\; \mathbb{E}\Bigg[ \sum_{t=0}^{\infty} \gamma^t R(s_t, a_t) \,\Big|\, b_0 = b, \pi \Bigg]. \]

The optimal value function satisfies the belief Bellman equation:

\[ V^\star(b) = \max_{a \in \mathcal{A}} \Bigg\{ R(b,a) + \gamma \sum_{o \in \mathcal{O}} p(o \mid b,a)\, V^\star\big(\tau(b,a,o)\big) \Bigg\}, \]

where \( R(b,a) = \sum_{s} b(s)\,R(s,a) \) and \( \tau(b,a,o) \) is the belief update. For finite \( \mathcal{S} \), a classical result states:

Proposition 2 (Piecewise-Linear and Convex). For a finite POMDP, the finite-horizon optimal value function \( V_t^\star(b) \) is piecewise-linear and convex over the belief simplex. There exists a finite set of "alpha-vectors" \( \Gamma_t = \{\alpha^1,\dots,\alpha^K\} \subset \mathbb{R}^{|\mathcal{S}|} \) such that

\[ V_t^\star(b) = \max_{\alpha \in \Gamma_t} \alpha^\top b. \]

Sketch of proof (induction on horizon).

1. At horizon \( t=0 \), the value is just the expected immediate reward, a linear function of \( b \).
2. Assume \( V_{t+1}^\star(b) \) is piecewise-linear and convex. Substituting into the Bellman update and expanding the expectation over \( o \) shows that \( V_t^\star(b) \) is a maximum of finitely many linear functions of \( b \) (obtained by backing up each alpha-vector through the dynamics and observations).
3. A maximum of linear functions is convex and piecewise-linear, completing the induction.

For deformable manipulation, this structure is useful conceptually but impractical to compute exactly because \( |\mathcal{S}| \) and \( |\Gamma_t| \) explode combinatorially. This motivates approximate methods such as point-based value iteration and QMDP, which we now discuss.

Exact POMDP planning is intractable for most real robotic systems, especially when the deformable state is high-dimensional. Common approximations include:

• QMDP. Assume the system becomes fully observable after one step, solve the underlying fully observable MDP, and pick actions maximizing the belief-averaged Q-values.
• Point-based value iteration (PBVI, SARSOP). Maintain a finite set of representative beliefs and perform backups only on those points.
• Online Monte Carlo tree search (POMCP-like). Sample state trajectories from the belief, build a look-ahead tree in belief space, and choose actions based on estimated returns.

For deformable manipulation, online methods are attractive: the robot replans at each step using the current belief (obtained from simulation or learned models) and a truncated look-ahead horizon, rather than computing a global policy.

flowchart TD
  B["Current belief b_t (particles / latent embedding)"]
    --> SAMP["Sample states s_t^i ~ b_t"]
  SAMP --> TREE["Rollout simulations with candidate actions"]
  TREE --> UPD["Estimate Q(b_t, a) by Monte Carlo"]
  UPD --> SEL["Select action a_t = argmax_a Q(b_t, a)"]
  SEL --> EXEC["Execute on robot, observe o_(t+1)"]
  EXEC --> B2["Update belief b_(t+1)"]
  B2 --> B
        

This online loop can be implemented on top of deformable simulators (e.g., PyBullet, SOFA, MuJoCo) or learned differentiable simulators, with the belief updated by importance weighting of particles or by a learned observation model.

We illustrate a small discrete POMDP representing a 1D "deformable" state: a cable can be in three latent configurations \( \mathcal{S} = \{s_0, s_1, s_2\} \) (e.g., "straight", "slightly entangled", "heavily entangled"). The robot chooses actions "pull" or "shake", gets noisy observations, and we plan using QMDP. In practice, this logic would be embedded in a ROS/OMPL pipeline with deformable simulation (e.g., PyBullet soft bodies).

import numpy as np

# State indices: 0=straight, 1=slightly entangled, 2=heavily entangled
S = 3
A = 2  # 0=pull, 1=shake
O = 2  # 0="good tension", 1="bad tension"

gamma = 0.95

# Transition model T[s, a, s']
T = np.zeros((S, A, S))
# Pull tends to improve state but risks damage from heavily entangled
T[0, 0] = [0.9, 0.1, 0.0]
T[1, 0] = [0.6, 0.3, 0.1]
T[2, 0] = [0.1, 0.4, 0.5]

# Shake tends to randomize towards slightly entangled
T[0, 1] = [0.7, 0.3, 0.0]
T[1, 1] = [0.2, 0.6, 0.2]
T[2, 1] = [0.0, 0.7, 0.3]

# Observation model Z[a, s', o]
Z = np.zeros((A, S, O))
# "Good tension" more likely when state is straight or slightly entangled
Z[:, 0, :] = [[0.9, 0.1],  # after pull
              [0.8, 0.2]]  # after shake
Z[:, 1, :] = [[0.7, 0.3],
              [0.6, 0.4]]
Z[:, 2, :] = [[0.3, 0.7],
              [0.4, 0.6]]

# Reward R[s, a]: prefer straight cable, penalize heaviness
R = np.array([
    [+1.0, +0.5],   # straight
    [-0.2, -0.1],   # slightly entangled
    [-1.0, -0.5],   # heavily entangled
])

def belief_predict(b, a):
    """One-step prediction b'(s') = sum_s T[s,a,s'] b(s)."""
    return T[:, a, :].T.dot(b)

def belief_update(b, a, o):
    """Bayes update using observation o after action a."""
    bp = belief_predict(b, a)  # predictive belief
    # likelihood p(o | s', a)
    lik = Z[a, :, o]
    num = lik * bp
    if np.sum(num) < 1e-12:
        # Numerical safeguard
        return np.ones_like(b) / len(b)
    return num / np.sum(num)

def value_iteration_mdp(max_iter=1000, tol=1e-6):
    """Solve fully observable MDP for QMDP."""
    V = np.zeros(S)
    for it in range(max_iter):
        Q = R + gamma * np.einsum("sasp->sa", T * V[np.newaxis, np.newaxis, :])
        V_new = np.max(Q, axis=1)
        if np.max(np.abs(V_new - V)) < tol:
            break
        V = V_new
    Q = R + gamma * np.einsum("sasp->sa", T * V[np.newaxis, np.newaxis, :])
    return V, Q

V_mdp, Q_mdp = value_iteration_mdp()

def qmdp_action(b):
    """QMDP action: maximize expected Q(s,a) under belief b."""
    # expected Q for each action
    Qa = b.dot(Q_mdp)  # shape (A,)
    return int(np.argmax(Qa))

# Example usage
b = np.array([1/3, 1/3, 1/3], dtype=float)  # initial belief
obs_seq = [0, 1, 0]  # hypothetical tension measurements

for t, o in enumerate(obs_seq):
    a = qmdp_action(b)
    print(f"t={t}, belief={b}, action={a}")
    b = belief_update(b, a, o)

print("Final belief:", b)
      

In a real deformable-manipulation system, T and Z would be estimated from simulation rollouts (e.g., PyBullet soft bodies) or learned models, and the belief would be over a latent representation learned in previous chapters.

The following C++ snippet sketches a belief update for a discrete POMDP, suitable for integration with OMPL (Open Motion Planning Library) and ROS-based deformable simulators. We assume a small discrete state space representing coarse deformable configurations.

#include <vector>
#include <cmath>
#include <numeric>

using Belief = std::vector<double>;
using Matrix = std::vector<std::vector<double>>;

// T[a][s][s_next]
using TransitionModel = std::vector<std::vector<std::vector<double>>>;
// Z[a][s_next][o]
using ObservationModel = std::vector<std::vector<std::vector<double>>>;

Belief predictBelief(const Belief& b,
                     const TransitionModel& T,
                     int a) {
    const int S = static_cast<int>(b.size());
    Belief bp(S, 0.0);
    for (int s = 0; s < S; ++s) {
        for (int sp = 0; sp < S; ++sp) {
            bp[sp] += T[a][s][sp] * b[s];
        }
    }
    return bp;
}

Belief updateBelief(const Belief& b,
                    const TransitionModel& T,
                    const ObservationModel& Z,
                    int a, int o) {
    Belief bp = predictBelief(b, T, a);
    const int S = static_cast<int>(b.size());
    Belief bnew(S, 0.0);
    double norm = 0.0;
    for (int sp = 0; sp < S; ++sp) {
        bnew[sp] = Z[a][sp][o] * bp[sp];
        norm += bnew[sp];
    }
    if (norm < 1e-12) {
        // fallback: uniform
        std::fill(bnew.begin(), bnew.end(), 1.0 / S);
        return bnew;
    }
    for (int sp = 0; sp < S; ++sp) {
        bnew[sp] /= norm;
    }
    return bnew;
}

// Example: inside a ROS node controlling a robot arm that manipulates a rope,
// you would call updateBelief after each observation and use an OMPL-based
// planner that chooses actions based on the current belief.
      

In a full implementation, TransitionModel and ObservationModel would either be learned from data or derived from a deformable dynamics library such as Drake, DART, or a finite-element-based simulator.

In Java, we might integrate with ROSJava and a simulation engine (e.g., jMonkeyEngine-based) to manage beliefs for deformable states at a high level.

public class BeliefState {
    private final int S;
    private double[] belief;

    // T[a][s][sNext]
    private final double[][][] T;
    // Z[a][sNext][o]
    private final double[][][] Z;

    public BeliefState(double[] initialBelief,
                       double[][][] T,
                       double[][][] Z) {
        this.S = initialBelief.length;
        this.belief = initialBelief.clone();
        this.T = T;
        this.Z = Z;
    }

    public double[] getBelief() {
        return belief.clone();
    }

    public void update(int action, int observation) {
        double[] bp = new double[S];
        for (int s = 0; s < S; ++s) {
            for (int sp = 0; sp < S; ++sp) {
                bp[sp] += T[action][s][sp] * belief[s];
            }
        }
        double[] bnew = new double[S];
        double norm = 0.0;
        for (int sp = 0; sp < S; ++sp) {
            bnew[sp] = Z[action][sp][observation] * bp[sp];
            norm += bnew[sp];
        }
        if (norm < 1e-12) {
            for (int sp = 0; sp < S; ++sp) {
                bnew[sp] = 1.0 / S;
            }
        } else {
            for (int sp = 0; sp < S; ++sp) {
                bnew[sp] /= norm;
            }
        }
        belief = bnew;
    }

    // Example policy: QMDP-like argmax_a sum_s b(s) Q[s][a]
    public int selectAction(double[][] Q) {
        int A = Q[0].length;
        double bestVal = Double.NEGATIVE_INFINITY;
        int bestA = 0;
        for (int a = 0; a < A; ++a) {
            double qa = 0.0;
            for (int s = 0; s < S; ++s) {
                qa += belief[s] * Q[s][a];
            }
            if (qa > bestVal) {
                bestVal = qa;
                bestA = a;
            }
        }
        return bestA;
    }
}
      

This class can be instantiated within a ROSJava node that subscribes to tactile and vision topics, updates the belief after each observation, and publishes desired gripper or arm commands based on a high-level planner.

MATLAB with Robotics System Toolbox and Simscape Multibody is often used for rapid prototyping. Below we show a script implementing belief update and QMDP action selection. The core functions can be wrapped in a MATLAB Function block in Simulink to form a planning component in a larger robot model.

function [b_next, a_qmdp] = deformable_qmdp_step(b, a_prev, o, T, Z, Q)
% b      : current belief column vector (Sx1)
% a_prev : last action index (1..A)
% o      : new observation index (1..O)
% T      : SxSxA transition probabilities (T(s_next, s, a))
% Z      : SxAxO observation probabilities (Z(s_next, a, o))
% Q      : SxA MDP Q-values for QMDP
%
% b_next : updated belief
% a_qmdp : next action from QMDP

S = size(T, 1);
A = size(T, 3);

% Prediction: b_pred(s') = sum_s T(s', s, a_prev) * b(s)
b_pred = zeros(S,1);
for sp = 1:S
    for s = 1:S
        b_pred(sp) = b_pred(sp) + T(sp,s,a_prev) * b(s);
    end
end

% Correction: multiply by likelihood Z(s', a_prev, o)
lik = Z(:,a_prev,o);
b_next = lik .* b_pred;
norm_const = sum(b_next);

if norm_const < 1e-12
    b_next = ones(S,1) / S;
else
    b_next = b_next / norm_const;
end

% QMDP action selection: maximize sum_s b_next(s) * Q(s,a)
Qa = zeros(A,1);
for a = 1:A
    Qa(a) = b_next' * Q(:,a);
end
[~, a_qmdp] = max(Qa);
      

In Simulink, this function becomes a block receiving b, the previous action and the new observation. The block outputs an updated belief and the next action, which can feed into a lower-level trajectory generator in a Robot Arm block.

In Wolfram Mathematica, we can conveniently represent the POMDP components as associations and carry out exact or numeric belief updates symbolically. This is useful for theoretical exploration of small deformable POMDPs.

(* States, actions, observations *)
states = {s0, s1, s2};
actions = {pull, shake};
observations = {good, bad};

(* Transition T[s, a, s'] and observation Z[a, s', o] as rules *)
T = Association[
  {s0, pull} -> {s0 -> 0.9, s1 -> 0.1, s2 -> 0.0},
  {s1, pull} -> {s0 -> 0.6, s1 -> 0.3, s2 -> 0.1},
  {s2, pull} -> {s0 -> 0.1, s1 -> 0.4, s2 -> 0.5},
  {s0, shake} -> {s0 -> 0.7, s1 -> 0.3, s2 -> 0.0},
  {s1, shake} -> {s0 -> 0.2, s1 -> 0.6, s2 -> 0.2},
  {s2, shake} -> {s0 -> 0.0, s1 -> 0.7, s2 -> 0.3}
];

Z = Association[
  {pull, s0} -> {good -> 0.9, bad -> 0.1},
  {pull, s1} -> {good -> 0.7, bad -> 0.3},
  {pull, s2} -> {good -> 0.3, bad -> 0.7},
  {shake, s0} -> {good -> 0.8, bad -> 0.2},
  {shake, s1} -> {good -> 0.6, bad -> 0.4},
  {shake, s2} -> {good -> 0.4, bad -> 0.6}
];

(* Belief is an association b[s] *)
b0 = Association[s0 -> 1/3, s1 -> 1/3, s2 -> 1/3];

predictBelief[b_, act_] := Module[{bp},
  bp = Association[Table[sp -> 0.0, {sp, states}]];
  Do[
    With[{tp = T[{s, act}]},
      Do[
        bp[sp] = bp[sp] + tp[sp] * b[s],
        {sp, states}
      ]
    ],
    {s, states}
  ];
  bp
];

updateBelief[b_, act_, obs_] := Module[{bp, bnew, norm},
  bp = predictBelief[b, act];
  bnew = Association[
    Table[
      sp -> Z[{act, sp}][obs] * bp[sp],
      {sp, states}
    ]
  ];
  norm = Total[Values[bnew]];
  If[norm < 10^-12,
    Association[Table[sp -> 1.0/Length[states], {sp, states}]],
    bnew / norm
  ]
];

b1 = updateBelief[b0, pull, good]
      

Mathematica's symbolic capabilities can also be used to verify properties such as value function convexity or to automatically derive alpha-vectors for tiny POMDPs.

Problem 1 (Belief Update for a Deformable State). Consider a POMDP with two states \( s_0 \) (cloth flat), \( s_1 \) (cloth wrinkled). There is a single action \( a \) (shake). The transition probabilities are \( T(s_0 \mid s_0,a) = 0.8 \), \( T(s_1 \mid s_0,a) = 0.2 \), \( T(s_0 \mid s_1,a) = 0.3 \), \( T(s_1 \mid s_1,a) = 0.7 \). Observations are \( o \in \{0,1\} \) ("flat" vs "wrinkled"), with \( Z(0 \mid s_0,a) = 0.9 \), \( Z(0 \mid s_1,a) = 0.2 \). Suppose the current belief is \( b_t(s_0) = 0.5, b_t(s_1) = 0.5 \) and the observation is \( o=0 \). Compute \( b_{t+1} \).

Solution. First compute the predictive belief:

\[ \tilde{b}_{t+1}(s_0) = 0.8 \cdot 0.5 + 0.3 \cdot 0.5 = 0.55, \quad \tilde{b}_{t+1}(s_1) = 0.2 \cdot 0.5 + 0.7 \cdot 0.5 = 0.45. \]

Now multiply by the observation likelihood and renormalize:

\[ \begin{aligned} \hat{b}_{t+1}(s_0) &= Z(0 \mid s_0,a)\,\tilde{b}_{t+1}(s_0) = 0.9 \cdot 0.55 = 0.495,\\ \hat{b}_{t+1}(s_1) &= Z(0 \mid s_1,a)\,\tilde{b}_{t+1}(s_1) = 0.2 \cdot 0.45 = 0.09. \end{aligned} \]

The normalizer is \( \eta^{-1} = 0.495 + 0.09 = 0.585 \), so

\[ b_{t+1}(s_0) = \frac{0.495}{0.585} \approx 0.846, \quad b_{t+1}(s_1) = \frac{0.09}{0.585} \approx 0.154. \]

The observation "flat" strongly increases the probability that the cloth is truly flat.

Problem 2 (Contraction of Belief Bellman Operator). Let \( \mathcal{B} \) denote the set of beliefs and define the optimality operator \( (TV)(b) = \max_a \{ R(b,a) + \gamma \sum_o p(o \mid b,a) V(\tau(b,a,o)) \} \). Show that \( T \) is a contraction mapping in the sup norm, i.e., \( \lVert TV - TW \rVert_\infty \le \gamma \lVert V - W \rVert_\infty \).

Solution. For any beliefs \( b \):

\[ \begin{aligned} |(TV)(b) - (TW)(b)| &\le \max_{a} \left| \gamma \sum_{o} p(o \mid b,a) \big[ V(\tau(b,a,o)) - W(\tau(b,a,o)) \big] \right| \\ &\le \gamma \max_{a} \sum_{o} p(o \mid b,a) \big| V(\tau(b,a,o)) - W(\tau(b,a,o)) \big| \\ &\le \gamma \lVert V - W \rVert_\infty \sum_{o} p(o \mid b,a) \\ &= \gamma \lVert V - W \rVert_\infty. \end{aligned} \]

Taking the supremum over \( b \) yields \( \lVert TV - TW \rVert_\infty \le \gamma \lVert V - W \rVert_\infty \). Thus value iteration in belief space converges to a unique fixed point.

Problem 3 (QMDP Approximation Quality). Consider a deformable manipulation POMDP in which the observation function is highly informative: after at most one action, the state is revealed almost exactly. Argue qualitatively why the QMDP approximation becomes near-optimal in this regime.

Solution. QMDP assumes that after the next step the state becomes fully observable, so it optimizes the one-step look-ahead reward plus the fully observable MDP continuation. If in the true problem the observation kernel \( Z \) is such that after one step the belief collapses to a near-delta distribution (i.e., the state is almost known), then the future evolution is effectively that of the underlying MDP. Hence the QMDP value approximates the true POMDP value very well, and the action chosen by QMDP is close to optimal.

Problem 4 (Belief Dimension and Deformable Representations). Suppose the latent representation \( \phi(x_t) \in \mathbb{R}^d \) of the deformable configuration has dimension \( d \). Compare the complexity of (i) maintaining a Gaussian belief \( \mathcal{N}(\mu_t,\Sigma_t) \) vs. (ii) maintaining a particle belief with \( N \) samples. Discuss which is more appropriate as \( d \) grows.

Solution. A Gaussian belief requires storing \( \mu_t \in \mathbb{R}^d \) and a covariance \( \Sigma_t \in \mathbb{R}^{d \times d} \), which is \( O(d^2) \) memory and update cost. A particle belief requires \( N \) samples, each of dimension \( d \), so memory and update cost is \( O(Nd) \). For moderate \( d \) and non-Gaussian posteriors (e.g., multi-modal folds of a cloth), particles may be more expressive, but \( N \) must grow rapidly with \( d \) to avoid degeneracy (curse of dimensionality). For very high-dimensional latent features but approximately unimodal uncertainty, Gaussian beliefs or low-rank covariance approximations may be more computationally feasible.

Problem 5 (Reward Modeling for Deformable Tasks). For a cloth-folding task, propose a reward function \( R(s,a) \) that encourages both progress towards a target folded shape and safety (avoiding excessive stretching). Express your reward as a function of a feature vector \( f(s) = (d_{\text{shape}}(s), \sigma_{\text{stretch}}(s)) \) measuring shape error and maximum stretch.

Solution. One simple choice is a linear reward:

\[ R(s,a) = -\lambda_1\, d_{\text{shape}}(s) - \lambda_2\, \sigma_{\text{stretch}}(s), \]

with \( \lambda_1, \lambda_2 > 0 \). Here \( d_{\text{shape}}(s) \) measures distance to the target folded configuration in feature space, and \( \sigma_{\text{stretch}}(s) \) penalizes large elastic strain. To regularize actions, we can add a small action cost

\[ R(s,a) = -\lambda_1\, d_{\text{shape}}(s) - \lambda_2\, \sigma_{\text{stretch}}(s) - \lambda_3 \lVert u(a) \rVert_2^2, \]

where \( u(a) \) is the commanded velocity or force associated with action \( a \). This reward can be evaluated on each latent state and extended to beliefs via \( R(b,a) = \sum_s b(s) R(s,a) \).

In this lesson we framed deformable object manipulation as a POMDP, making explicit the role of beliefs in coping with occlusions, noisy sensing, and latent high-dimensional configurations. We derived belief updates, reviewed the structure of value functions in belief space, and discussed why exact planning is intractable for realistic deformable tasks. We then focused on approximate methods such as QMDP and online Monte Carlo tree search, and illustrated how to implement belief updates and simple planners across multiple programming languages commonly used in robotics. These concepts prepare us to study learning-based deformable manipulation, where the transition and observation models are themselves learned from data.