Consider a rigid robot manipulator interacting with a deformable object. Let \( \mathbf{q}_t \in \mathbb{R}^{n_q} \) be the robot joint configuration, \( \dot{\mathbf{q}}_t \) its velocities, and \( \mathbf{y}_t \) a low-dimensional state representation of the deformable object (e.g., keypoints or latent coordinates introduced in Lesson 2). We define the system state as

\[ \mathbf{x}_t = \begin{bmatrix} \mathbf{q}_t \\ \dot{\mathbf{q}}_t \\ \mathbf{y}_t \end{bmatrix} \in \mathbb{R}^{n_x}. \]

The robot executes control inputs \( \mathbf{u}_t \in \mathbb{R}^{n_u} \), such as joint velocity or Cartesian twist commands. The true dynamics (including complex contact and deformation physics) are unknown but can be modeled probabilistically as

\[ p(\mathbf{x}_{t+1} \mid \mathbf{x}_t, \mathbf{u}_t), \]

which defines an underlying Markov decision process (MDP). Because deformable states are often only partially observed (e.g., from RGB-D images), the robot receives observations \( \mathbf{o}_t = h(\mathbf{x}_t) \), leading to a POMDP view consistent with Chapter 6.

A parameterized policy \( \pi_\theta(\mathbf{u}_t \mid \mathbf{o}_t) \) maps observations to actions. The expected finite-horizon cost for a task such as cloth folding or cable routing is

\[ J(\theta) = \mathbb{E}\!\left[ \sum_{t=0}^{T-1} \gamma^t c(\mathbf{x}_t, \mathbf{u}_t) \right], \quad 0 < \gamma \le 1, \]

where \( c(\mathbf{x}_t,\mathbf{u}_t) \) encodes deformation quality (e.g., distance to desired folded configuration) and control effort.

Learning-based deformable manipulation typically relies on two complementary paradigms:

• Imitation learning (IL): Learn \( \pi_\theta \) from expert demonstrations \( \mathcal{D} = \{(\mathbf{o}_t^i, \mathbf{u}_t^{i,\star})\} \).
• Reinforcement learning (RL): Optimize \( J(\theta) \) by trial and error, often in a simulator with deformable objects.

Both IL and RL must handle high-dimensional, non-rigid dynamics, partial observability, and safety constraints for physical robots.

We assume a dataset of expert trajectories \( \mathcal{D} = \{(\mathbf{o}_t^i, \mathbf{u}_t^{i,\star})\} \) collected via teleoperation or kinesthetic teaching. A parametric policy \( \pi_\theta(\mathbf{o}) \in \mathbb{R}^{n_u} \) is trained to minimize a supervised loss

\[ L(\theta) = \frac{1}{N} \sum_{i=1}^N \ell\!\left(\mathbf{u}_i^{\star}, \pi_\theta(\mathbf{o}_i)\right), \]

where \( \ell \) is typically a squared Euclidean loss

\[ \ell\!\left(\mathbf{u}^{\star}, \hat{\mathbf{u}}\right) = \frac{1}{2} \left\| \mathbf{u}^{\star} - \hat{\mathbf{u}} \right\|_2^2. \]

For a linear policy with fixed feature map \( \boldsymbol{\phi}:\mathcal{O} \rightarrow \mathbb{R}^d \),

\[ \pi_\theta(\mathbf{o}) = W \boldsymbol{\phi}(\mathbf{o}), \quad W \in \mathbb{R}^{n_u \times d},\; \theta \equiv W, \]

we can express the total loss as

\[ L(W) = \frac{1}{2N} \sum_{i=1}^N \left\| \mathbf{u}_i^{\star} - W \boldsymbol{\phi}_i \right\|_2^2, \quad \boldsymbol{\phi}_i \equiv \boldsymbol{\phi}(\mathbf{o}_i). \]

In matrix form, let \( \Phi \in \mathbb{R}^{N \times d} \) stack \( \boldsymbol{\phi}_i^\top \) row-wise and \( U^\star \in \mathbb{R}^{N \times n_u} \) stack \( (\mathbf{u}_i^{\star})^\top \). Then

\[ L(W) = \frac{1}{2N} \left\| U^\star - \Phi W^\top \right\|_F^2. \]

Proposition 1 (Convexity). The loss \(L(W)\) is convex in \(W\).

Proof. Each term of the sum is a composition of an affine map in \(W\) with a squared Euclidean norm, which is convex. A nonnegative sum of convex functions is convex. Alternatively, vectorize \(W\) as \( \mathbf{w} = \mathrm{vec}(W) \). Then

\[ L(\mathbf{w}) = \frac{1}{2N} \left\| \mathbf{A}\mathbf{w} - \mathbf{b} \right\|_2^2, \]

for some matrix \( \mathbf{A} \) and vector \( \mathbf{b} \). The Hessian is \( \nabla^2 L(\mathbf{w}) = \frac{1}{N}\mathbf{A}^\top \mathbf{A} \), which is positive semidefinite, so \(L\) is convex.

If \( \mathbf{A}^\top \mathbf{A} \) is full-rank, the minimizer is unique and given by the normal equations:

\[ W^\star = \left( \Phi^\top \Phi \right)^{-1} \Phi^\top U^\star. \]

This gives a closed-form supervised controller, which can then be refined with interactive techniques such as dataset aggregation from Chapter 11 (Learning from Demonstration).

When expert demonstrations are incomplete or suboptimal, reinforcement learning (RL) can optimize \( J(\theta) \) directly, typically in a deformable-object simulator. For stochastic policies \( \pi_\theta(\mathbf{u}\mid\mathbf{o}) \), the trajectory distribution is

\[ p_\theta(\tau) = p(\mathbf{x}_0) \prod_{t=0}^{T-1} \pi_\theta(\mathbf{u}_t \mid \mathbf{o}_t) p(\mathbf{x}_{t+1} \mid \mathbf{x}_t, \mathbf{u}_t), \]

where \( \tau = (\mathbf{x}_0,\mathbf{u}_0,\dots,\mathbf{x}_{T-1},\mathbf{u}_{T-1}) \). The RL objective can be written as

\[ J(\theta) = \mathbb{E}_{\tau \sim p_\theta} \big[ R(\tau) \big], \quad R(\tau) = \sum_{t=0}^{T-1} \gamma^t r(\mathbf{x}_t,\mathbf{u}_t), \]

where \( r = -c \) is a reward function (e.g., negative distance between the current and desired cloth shape).

The policy gradient theorem (derived in Chapter 12) yields

\[ \nabla_\theta J(\theta) = \mathbb{E}_{\tau \sim p_\theta} \left[ \sum_{t=0}^{T-1} \nabla_\theta \log \pi_\theta(\mathbf{u}_t \mid \mathbf{o}_t) G_t \right], \]

where \( G_t \) is a return estimate, such as the Monte Carlo return from time \(t\). A common choice for deformable manipulation is a shaped reward:

\[ r(\mathbf{x}_t,\mathbf{u}_t) = -\alpha d_{\text{shape}}(\mathbf{y}_t,\mathbf{y}_{\text{goal}}) -\beta \left\|\mathbf{u}_t\right\|_2^2, \quad \alpha, \beta > 0, \]

where \( d_{\text{shape}} \) measures distance in the deformable state space (e.g., Chamfer distance between keypoints) and \( \beta \left\|\mathbf{u}_t\right\|_2^2 \) penalizes large actions that may damage the object.

In practice, RL for deformable manipulation is often combined with imitation: initialize \( \pi_\theta \) via supervised learning on demonstrations, then continue training with RL to improve robustness and performance.

A central challenge is mapping high-dimensional observations (RGB-D images or point clouds) to compact, task-relevant features. Let \( \boldsymbol{\psi}_\phi: \mathcal{O} \rightarrow \mathbb{R}^d \) be a learned encoder (e.g., a convolutional or graph neural network) with parameters \( \phi \). A policy then operates on \( \mathbf{z}_t = \boldsymbol{\psi}_\phi(\mathbf{o}_t) \):

\[ \pi_{\theta,\phi}(\mathbf{u}_t \mid \mathbf{o}_t) = \pi_\theta(\mathbf{u}_t \mid \mathbf{z}_t), \quad \mathbf{z}_t = \boldsymbol{\psi}_\phi(\mathbf{o}_t). \]

Joint training minimizes a combined objective

\[ \mathcal{L}(\theta,\phi) = \lambda_{\text{IL}} L_{\text{IL}}(\theta,\phi) - \lambda_{\text{RL}} J(\theta,\phi) + \lambda_{\text{reg}} \Omega(\theta,\phi), \quad \lambda_{\text{IL}}, \lambda_{\text{RL}}, \lambda_{\text{reg}} \ge 0, \]

where \( L_{\text{IL}} \) is an imitation loss, \( J \) an RL return, and \( \Omega \) a regularizer (e.g., weight decay).

A typical training pipeline for deformable manipulation in simulation is summarized below.

flowchart TD
  A["Simulate deformable scenarios"] --> B["Collect expert or random rollouts"]
  B --> C["Encode observations into features z_t"]
  C --> D["Optimize policy via IL and/or RL"]
  D --> E["Validate policy on held-out deformable tasks"]
  E --> F["Deploy on real robot with calibrated dynamics"]
        

The simulation step uses a deformable physics engine; the encoder and policy are optimized using gradient-based methods, and the final policy is transferred to the real robot with domain randomization (Chapter 13) to mitigate the reality gap.

For the supervised loss \( L(\theta,\phi) \), gradient-based optimization uses

\[ \nabla_\theta L(\theta,\phi) = \frac{1}{N} \sum_{i=1}^N \nabla_\theta \ell\!\left( \mathbf{u}_i^{\star}, \pi_{\theta,\phi}(\mathbf{o}_i) \right), \]

and similarly for \( \nabla_\phi L(\theta,\phi) \) by the chain rule. For squared loss and a differentiable policy,

\[ \nabla_\theta \ell = -\left( \mathbf{u}^{\star} - \pi_{\theta,\phi}(\mathbf{o}) \right)^\top \nabla_\theta \pi_{\theta,\phi}(\mathbf{o}). \]

For the RL component \( J(\theta,\phi) \), stochastic gradient ascent uses sampled trajectories. Given a batch of trajectories \( \{\tau^{(k)}\}_{k=1}^K \), an unbiased estimator of the policy gradient (ignoring baselines) is

\[ \widehat{\nabla_{\theta,\phi} J} = \frac{1}{K} \sum_{k=1}^K \sum_{t=0}^{T-1} \nabla_{\theta,\phi} \log \pi_{\theta,\phi} \big( \mathbf{u}_t^{(k)} \mid \mathbf{o}_t^{(k)} \big) G_t^{(k)}. \]

Proposition 2 (Unbiasedness of Monte Carlo Policy Gradient). Suppose all trajectories are sampled independently from \( p_{\theta,\phi} \) and \( G_t^{(k)} \) is any return whose expectation equals the true state-action value. Then

\[ \mathbb{E}\left[ \widehat{\nabla_{\theta,\phi} J} \right] = \nabla_{\theta,\phi} J(\theta,\phi). \]

The proof follows from linearity of expectation and the log-derivative trick, as introduced in Chapter 12.

We implement a simple supervised imitation policy in Python using PyTorch. Observations are preprocessed into feature vectors (e.g., cloth keypoints or CNN embeddings), and the network outputs Cartesian end-effector velocity commands.

flowchart TD
  O["Observation o_t (image, depth, keypoints)"]
    --> FEAT["Feature extractor (precomputed or CNN)"]
  FEAT --> POL["MLP policy pi_theta"]
  POL --> ACT["End-effector velocity command u_t"]
  ACT --> ENV["Simulated deformable environment"]
        

import torch
import torch.nn as nn
import torch.optim as optim

# Hyperparameters
obs_dim = 128   # e.g., encoded deformable state
act_dim = 6     # e.g., Cartesian velocity (vx, vy, vz, wx, wy, wz)
hidden_dim = 256
lr = 1e-3

class DeformablePolicy(nn.Module):
    def __init__(self, obs_dim, act_dim, hidden_dim):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(obs_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, act_dim)
        )

    def forward(self, obs):
        return self.net(obs)

policy = DeformablePolicy(obs_dim, act_dim, hidden_dim)

# Dummy dataset: replace with real features and expert actions
# observations: Tensor of shape (N, obs_dim)
# actions_star: Tensor of shape (N, act_dim)
observations = torch.randn(1024, obs_dim)
actions_star = torch.randn(1024, act_dim)

dataset = torch.utils.data.TensorDataset(observations, actions_star)
loader = torch.utils.data.DataLoader(dataset, batch_size=64, shuffle=True)

optimizer = optim.Adam(policy.parameters(), lr=lr)
mse_loss = nn.MSELoss()

for epoch in range(50):
    running_loss = 0.0
    for obs_batch, act_batch in loader:
        optimizer.zero_grad()
        act_pred = policy(obs_batch)
        loss = mse_loss(act_pred, act_batch)
        loss.backward()
        optimizer.step()
        running_loss += loss.item() * obs_batch.size(0)
    avg_loss = running_loss / len(dataset)
    print(f"Epoch {epoch:03d}, Loss = {avg_loss:.6f}")

# Example of deploying the policy given a new observation feature vector:
with torch.no_grad():
    o_new = torch.randn(1, obs_dim)  # replace with real encoded observation
    u_cmd = policy(o_new)
    print("Commanded velocity:", u_cmd.cpu().numpy())
      

This implementation can be extended with convolutional encoders, recurrent layers for partial observability, or integrated into an RL algorithm by replacing the loss with a policy gradient estimate.

In lower-level control loops, a linear policy trained offline can be deployed efficiently in C++. The following snippet uses Eigen to implement a linear mapping between encoded deformable state features and velocity commands.

#include <iostream>
#include <Eigen/Dense>

int main() {
    const int obs_dim = 128;
    const int act_dim = 6;

    // Policy matrix W of size (act_dim x obs_dim), learned offline
    Eigen::MatrixXd W(act_dim, obs_dim);
    W.setRandom(); // replace with loaded weights

    // Example observation feature vector (encoded deformable state)
    Eigen::VectorXd obs(obs_dim);
    obs.setRandom(); // replace with real features

    // Compute action u = W * obs
    Eigen::VectorXd u = W * obs;

    std::cout << "Commanded velocity:" << std::endl;
    std::cout << u.transpose() << std::endl;

    return 0;
}
      

The matrix W can be learned in Python or MATLAB with least squares (Section 2) and then exported to C++ (e.g., as a header or binary file) for real-time execution on the robot controller.

8.1 Java — Gradient Descent for a Linear Policy

The following Java snippet demonstrates learning a linear policy via gradient descent from a small dataset of features and expert actions.

public class LinearDeformablePolicy {
    private final int obsDim;
    private final int actDim;
    private final double[][] W; // [actDim][obsDim]
    private final double lr;

    public LinearDeformablePolicy(int obsDim, int actDim, double lr) {
        this.obsDim = obsDim;
        this.actDim = actDim;
        this.lr = lr;
        this.W = new double[actDim][obsDim];
        // initialize small random weights
        java.util.Random rnd = new java.util.Random(0);
        for (int i = 0; i < actDim; ++i) {
            for (int j = 0; j < obsDim; ++j) {
                W[i][j] = 0.01 * rnd.nextGaussian();
            }
        }
    }

    // Forward pass: u = W * obs
    public double[] act(double[] obs) {
        double[] u = new double[actDim];
        for (int i = 0; i < actDim; ++i) {
            double sum = 0.0;
            for (int j = 0; j < obsDim; ++j) {
                sum += W[i][j] * obs[j];
            }
            u[i] = sum;
        }
        return u;
    }

    // Single gradient descent step on squared loss
    public void trainStep(double[][] obsBatch, double[][] uStarBatch) {
        int batchSize = obsBatch.length;
        // accumulate gradients dL/dW
        double[][] gradW = new double[actDim][obsDim];

        for (int k = 0; k < batchSize; ++k) {
            double[] obs = obsBatch[k];
            double[] uStar = uStarBatch[k];
            double[] uPred = act(obs);

            for (int i = 0; i < actDim; ++i) {
                double error = uPred[i] - uStar[i];
                for (int j = 0; j < obsDim; ++j) {
                    gradW[i][j] += error * obs[j];
                }
            }
        }

        // update W
        double scale = lr / batchSize;
        for (int i = 0; i < actDim; ++i) {
            for (int j = 0; j < obsDim; ++j) {
                W[i][j] -= scale * gradW[i][j];
            }
        }
    }
}
      

8.2 MATLAB/Simulink — Least-Squares Policy and Simulink Block

In MATLAB, we can compute a least-squares policy and then realize it as a Gain block in Simulink that maps deformable state features to velocity commands.

% obs: N x d feature matrix (encoded deformable states)
% u_star: N x m expert actions (e.g., Cartesian velocities)

% Solve W in least squares sense: minimize ||u_star - obs * W'||_F^2
W = (obs' * obs) \ (obs' * u_star);  % d x m

% Save weights for Simulink
save("policy_weights.mat", "W");

% In Simulink:
%  1. Create an input block that outputs obs_t (1 x d).
%  2. Use a "MATLAB Function" or "Gain" block that implements u_t = obs_t * W.
%  3. Connect u_t to the robot dynamics block (or joint-level controller).
      

8.3 Wolfram Mathematica — Training a Small Network

Mathematica can be used to experiment with small policies and differentiable losses for deformable manipulation.

(* obsData: list of observation feature vectors *)
(* uStarData: corresponding expert action vectors *)

obsDim = 128;
actDim = 6;

net = NetChain[{
    LinearLayer[256],
    ElementwiseLayer["ReLU"],
    LinearLayer[256],
    ElementwiseLayer["ReLU"],
    LinearLayer[actDim]
},
"Input" -> obsDim];

lossFun[net_, obsBatch_, uStarBatch_] := 
  Mean[Norm[net[obsBatch][[i]] - uStarBatch[[i]], 2]^2 & /@ 
    Range[Length[obsBatch]]];

trainedNet = NetTrain[
  net,
  Thread[obsData -> uStarData],
  LossFunction -> lossFun,
  BatchSize -> 64,
  MaxTrainingRounds -> 50
];

uCmd = trainedNet[obsData[[1]]];
      

The trained network can be exported and integrated into a C++ or Java control stack, similar to the linear policy case, using Mathematica's code generation capabilities.

Problem 1 (Convexity and Closed-Form Solution). Consider the linear imitation learning setup of Section 2 with squared loss. Show that the empirical risk minimizer is unique when \( \Phi^\top \Phi \) is invertible and derive its closed-form.

Solution. The loss is \( L(W) = \frac{1}{2N}\lVert U^\star - \Phi W^\top\rVert_F^2 \). Taking the derivative with respect to \( W \) and setting it to zero yields

\[ \frac{\partial L}{\partial W} = -\frac{1}{N} \left( U^\star - \Phi W^\top \right)^\top \Phi = 0. \]

Rearranging gives \( (\Phi^\top \Phi) W^\top = \Phi^\top U^\star \). If \( \Phi^\top \Phi \) is invertible, then

\[ W^\top = (\Phi^\top \Phi)^{-1} \Phi^\top U^\star, \quad W = U^{\star\top} \Phi (\Phi^\top \Phi)^{-1}, \]

which is unique because the inverse is unique. This matches the least-squares solution derived previously.

Problem 2 (Unbiased Monte Carlo Policy Gradient). Prove Proposition 2: that the estimator \( \widehat{\nabla_{\theta,\phi} J} \) in Section 5 is unbiased, assuming the returns \( G_t^{(k)} \) satisfy \( \mathbb{E}[G_t^{(k)} \mid \mathbf{x}_t,\mathbf{u}_t] = Q^{\pi_{\theta,\phi}}(\mathbf{x}_t,\mathbf{u}_t) \).

Solution. By definition,

\[ J(\theta,\phi) = \mathbb{E}_{\tau \sim p_{\theta,\phi}} \Big[ R(\tau) \Big]. \]

Using the log-derivative trick,

\[ \nabla_{\theta,\phi} J = \mathbb{E}_{\tau} \Big[ R(\tau) \nabla_{\theta,\phi} \log p_{\theta,\phi}(\tau) \Big]. \]

Since the dynamics do not depend on \( \theta,\phi \) by assumption, \( \nabla_{\theta,\phi} \log p_{\theta,\phi}(\tau) = \sum_t \nabla_{\theta,\phi} \log \pi_{\theta,\phi}(\mathbf{u}_t\mid\mathbf{o}_t) \). Replacing \( R(\tau) \) by a sum of returns from time \( t \) and using the conditional expectation property yields the expression for \( \nabla_{\theta,\phi} J \). The Monte Carlo estimator is the sample mean of this quantity, so its expectation equals the true gradient.

Problem 3 (Reward Shaping with Deformable State Distance). Let \( \mathbf{y}_t \) and \( \mathbf{y}_{\text{goal}} \) be deformable state descriptors (e.g., keypoints). Consider the reward \( r_t = -\alpha \lVert \mathbf{y}_t - \mathbf{y}_{\text{goal}}\rVert_2 \) with \( \alpha > 0 \). Show that for fixed dynamics, any policy that maximizes the discounted return also minimizes the expected discounted sum of distances to the goal.

Solution. The discounted return is

\[ J(\pi) = \mathbb{E} \left[ \sum_{t=0}^{T-1} \gamma^t r_t \right] = -\alpha \mathbb{E} \left[ \sum_{t=0}^{T-1} \gamma^t \left\| \mathbf{y}_t - \mathbf{y}_{\text{goal}} \right\|_2 \right]. \]

Since \( -\alpha < 0 \) is a constant scaling factor, maximizing \( J(\pi) \) is equivalent to minimizing

\[ \mathbb{E} \left[ \sum_{t=0}^{T-1} \gamma^t \left\| \mathbf{y}_t - \mathbf{y}_{\text{goal}} \right\|_2 \right]. \]

The dynamics are fixed, so the expectation depends only on the policy; thus optimizing either objective yields the same optimal policy.

Problem 4 (Stability of a Linear Deformable Policy). Consider a simplified linearized dynamics for a deformable feature vector \( \mathbf{z}_t \in \mathbb{R}^d \): \( \mathbf{z}_{t+1} = A\mathbf{z}_t + B\mathbf{u}_t \) with constant matrices \( A,B \). Let the policy be \( \mathbf{u}_t = K \mathbf{z}_t \). Show that if the spectral radius of \( A + BK \) is strictly less than 1, then \( \mathbf{z}_t \) converges to zero as \( t \rightarrow \infty \).

Solution. Substituting the policy into the dynamics gives \( \mathbf{z}_{t+1} = (A + BK)\mathbf{z}_t \). Thus \( \mathbf{z}_t = (A + BK)^t \mathbf{z}_0 \). If the spectral radius \( \rho(A + BK) < 1 \), then \( \lim_{t \rightarrow \infty} (A + BK)^t = 0 \), so \( \mathbf{z}_t \rightarrow 0 \) for all initial states \( \mathbf{z}_0 \). This is classical linear-systems stability, showing that a learned linear policy that places eigenvalues inside the unit disk stabilizes the linearized deformable dynamics.

Problem 5 (Sample Complexity Intuition). Suppose that in the linear imitation learning setting, features are bounded as \( \lVert \boldsymbol{\phi}(\mathbf{o})\rVert_2 \le R \) and actions are bounded as \( \lVert \mathbf{u}^{\star}\rVert_2 \le U \). Give an informal argument why the generalization error of the learned policy decreases on the order of \( \mathcal{O}(1/\sqrt{N}) \) with the number of demonstrations \( N \).

Solution. A linear policy class has finite pseudo-dimension proportional to \( d \). Standard results from statistical learning theory imply that, for bounded inputs and outputs, the excess risk over the best linear policy scales as \( \mathcal{O}(\sqrt{d/N}) \). Intuitively, each new demonstration reduces the uncertainty about the optimal weights by an amount proportional to \( 1/\sqrt{N} \) due to concentration of measure (e.g., Hoeffding or Bernstein bounds). Thus, for a fixed feature dimension, the mean-squared prediction error and induced control error shrink like \( \mathcal{O}(1/\sqrt{N}) \).

In this lesson we formalized learning-based deformable manipulation as an optimization problem over policies that map partial observations to control commands. We developed supervised imitation learning objectives, showed convexity and closed-form solutions for linear policies, and recalled policy gradient formulations for RL in deformable environments. Representation learning via encoders allows policies to operate on compact deformable state descriptors derived from high-dimensional sensory data. Finally, we provided multi-language implementations illustrating how learned controllers can be realized in Python, C++, Java, MATLAB/Simulink, and Mathematica, preparing for case studies on cloth folding and cable routing in the next lesson.