In previous lessons, you saw Markov Decision Processes (MDPs) and actor–critic methods for continuous control. Robotic manipulators and mobile platforms typically have continuous action spaces (joint torques, joint velocities, or Cartesian twists), making value-based tabular algorithms inapplicable. Off-policy actor–critic algorithms enable sample reuse via replay buffers and separate behavior policies from the target policy being optimized.

Consider an MDP \( (\mathcal{S},\mathcal{A},P,r,\gamma) \). For a target (possibly deterministic) policy \( \pi \), the action-value function is

\[ Q^{\pi}(s,a) = \mathbb{E}\!\left[ \sum_{t=0}^{\infty} \gamma^{t} r(s_t,a_t) \,\bigg|\, s_0 = s,\ a_0 = a,\ a_t \sim \pi(\cdot|s_t) \right]. \]

An off-policy method collects transitions using a behavior policy \( \mu \) (e.g., exploratory actor with added noise), stores them in a replay buffer, and updates \( Q^{\pi} \) and \( \pi \) using this experience even when \( \mu \neq \pi \). This decoupling is crucial in robotics, where simulator runs are expensive but can be reused many times.

flowchart TD
  ENV["Robot + Environment"] --> EXP["Transitions (s,a,r,s_next)"]
  EXP --> BUF["Replay buffer"]
  BUF --> CRIT["Critic update: fit Q(s,a)"]
  CRIT --> ACT["Actor update: improve policy pi(a|s)"]
  ACT --> EXPL["Behavior mu = pi + exploration noise"]
  EXPL --> ENV
        

The three algorithms we study share this overall structure, but differ in: (i) policy class (deterministic vs stochastic), (ii) how they mitigate overestimation bias in the critic, and (iii) whether they directly optimize expected return or an entropy-augmented objective.

For a fixed target policy \( \pi \), define the Bellman operator \( \mathcal{T}^{\pi} \) on action-value functions:

\[ (\mathcal{T}^{\pi} Q)(s,a) = r(s,a) + \gamma\, \mathbb{E}_{s' \sim P(\cdot|s,a),\, a' \sim \pi(\cdot|s')} \big[ Q(s',a') \big]. \]

The fixed point of \( \mathcal{T}^{\pi} \) is \( Q^{\pi} \), and for any two functions \( Q_1, Q_2 \) we have the contraction property

\[ \big\| \mathcal{T}^{\pi} Q_1 - \mathcal{T}^{\pi} Q_2 \big\|_{\infty} \leq \gamma \big\| Q_1 - Q_2 \big\|_{\infty}. \]

In on-policy algorithms, samples are drawn from state–action visitation of the current policy, matching the distribution used in theoretical convergence proofs. In off-policy deep RL, however, we approximate expectations using transitions \( (s,a,r,s') \sim \rho^{\mu} \), where \( \rho^{\mu} \) is the stationary distribution under behavior policy \( \mu \). The critic is trained by minimizing (stochastic) mean-squared Bellman error

\[ L(\theta_Q) = \mathbb{E}_{(s,a,r,s') \sim \rho^{\mu}}\! \left[ \big( Q_{\theta_Q}(s,a) - y(s,a,r,s') \big)^2 \right], \]

where the target \( y \) involves the target policy (and possibly target networks). This off-policy approximation is biased but dramatically improves sample efficiency, which is a key constraint in robotics.

Deterministic policies are particularly convenient for continuous control: an actor outputs a single action per state, \( \mu_{\theta} : \mathcal{S} \to \mathcal{A} \). Let the performance objective be

\[ J(\theta) = \mathbb{E}_{s \sim \rho^{\mu_{\theta}}} \big[ Q^{\mu_{\theta}}(s, \mu_{\theta}(s)) \big], \]

where \( \rho^{\mu_{\theta}} \) is the discounted state distribution induced by the deterministic policy.

The deterministic policy gradient theorem states that under mild regularity conditions,

\[ \nabla_{\theta} J(\theta) = \mathbb{E}_{s \sim \rho^{\mu_{\theta}}} \Big[ \nabla_{\theta}\mu_{\theta}(s)\, \nabla_{a} Q^{\mu_{\theta}}(s,a)\big|_{a = \mu_{\theta}(s)} \Big]. \]

Sketch of proof.

1. Write \( J(\theta) = \sum_{s} \rho^{\mu_{\theta}}(s)\, Q^{\mu_{\theta}}(s,\mu_{\theta}(s)) \) in discrete form.
2. Differentiate using the product rule. The derivative of \( \rho^{\mu_{\theta}} \) w.r.t. \( \theta \) can be expressed via the policy gradient theorem; its contribution cancels with the explicit dependence of \( Q^{\mu_{\theta}} \) on \( \theta \) through future states when the Bellman equation holds.
3. The remaining term involves only the explicit dependence of the current action on \( \theta \), yielding the expectation above.

In practice, DDPG approximates \( Q^{\mu_{\theta}} \) by a neural critic \( Q_{\theta_Q}(s,a) \), and replaces \( \rho^{\mu_{\theta}} \) with the empirical replay distribution. The resulting gradient estimator is biased but effective for high-DOF robotic arms and legged robots.

DDPG is an off-policy actor–critic algorithm that combines deterministic policy gradients with deep Q-learning ideas (replay buffer and target networks). It maintains:

• An actor \( \mu_{\theta_{\mu}}(s) \) (e.g., MLP or CNN) that outputs continuous actions.
• A critic \( Q_{\theta_Q}(s,a) \) approximating \( Q^{\mu} \).
• Slowly updated target networks \( \mu_{\theta_{\mu}'} \) and \( Q_{\theta_Q'} \) for stable targets.

For transitions \( (s,a,r,s') \) from the replay buffer, the critic target is

\[ y = r + \gamma\, Q_{\theta_Q'}\!\big( s', \mu_{\theta_{\mu}'}(s') \big), \]

and the critic loss is

\[ L(\theta_Q) = \mathbb{E}_{(s,a,r,s')} \left[ \big( Q_{\theta_Q}(s,a) - y \big)^2 \right]. \]

The actor is updated by ascending an estimate of the deterministic policy gradient:

\[ \nabla_{\theta_{\mu}} J(\theta_{\mu}) \approx \mathbb{E}_{s \sim \rho^{\mu}} \Big[ \nabla_{\theta_{\mu}}\mu_{\theta_{\mu}}(s)\, \nabla_{a} Q_{\theta_Q}(s,a)\big|_{a = \mu_{\theta_{\mu}}(s)} \Big]. \]

Target networks are updated by Polyak averaging with parameter \( \tau \in (0,1) \):

\[ \theta_Q' \leftarrow (1 - \tau)\,\theta_Q' + \tau\,\theta_Q, \quad \theta_{\mu}' \leftarrow (1 - \tau)\,\theta_{\mu}' + \tau\,\theta_{\mu}. \]

For robotics, the actor usually outputs a bounded action via a \( \tanh \) output layer, then linearly maps it to torque or velocity limits. Exploration is implemented with additive noise: Ornstein–Uhlenbeck (OU) or Gaussian noise to encourage temporally correlated control signals.

DDPG suffers from overestimation bias in the critic due to bootstrapping with a max-like structure over continuous actions embedded in the actor. TD3 introduces three key modifications:

1. Clipped double Q-learning.
2. Delayed policy updates.
3. Target policy smoothing.

TD3 uses two critics \( Q_{\theta_{Q_1}} \) and \( Q_{\theta_{Q_2}} \). The target employs their minimum:

\[ y = r + \gamma\, \min_{i \in \{1,2\}} Q_{\theta_{Q_i}'}\!\big( s', \tilde{a}' \big), \]

where \( \tilde{a}' \) is a smoothed target action:

\[ \tilde{a}' = \mathrm{clip}\!\big( \mu_{\theta_{\mu}'}(s') + \epsilon, a_{\min}, a_{\max} \big), \quad \epsilon \sim \mathcal{N}(0,\sigma_{\text{targ}}^2), \ \ |\epsilon| \leq c. \]

The smoothing noise prevents the critic from exploiting sharp peaks in the Q-function that correspond to narrow, unrealistic action choices. Delayed policy updates mean that the actor and target networks are updated only every \( d \) critic updates, allowing critics to track the value landscape more accurately before changing the policy.

In high-DOF robotics, TD3 tends to be more stable than DDPG because:

• The minimum of two critics reduces optimistic bias, especially in regions of sparse data.
• Smoothing roughly corresponds to injecting robustness over a neighborhood of actions, beneficial when torque commands are quantized or noisy.

SAC departs from deterministic policies and optimizes a maximum-entropy objective. Instead of maximizing expected return alone, we consider

\[ J(\pi) = \mathbb{E}_{\pi}\!\left[ \sum_{t=0}^{\infty} \gamma^{t}\big( r(s_t,a_t) + \alpha\, \mathcal{H}(\pi(\cdot|s_t)) \big) \right], \]

where \( \mathcal{H}(\pi(\cdot|s)) \) is the Shannon entropy and \( \alpha > 0 \) trades off reward and exploration. High entropy encourages stochastic, exploratory policies, which can help manipulators escape local minima in complex contact-rich tasks.

Define the soft Q- and V-functions:

\[ Q^{\pi}(s,a) = \mathbb{E}_{\pi}\!\left[ \sum_{t=0}^{\infty} \gamma^{t}\big( r(s_t,a_t) + \alpha\, \mathcal{H}(\pi(\cdot|s_t)) \big) \,\bigg|\, s_0 = s,\ a_0 = a \right], \]

\[ V^{\pi}(s) = \mathbb{E}_{a \sim \pi(\cdot|s)}\! \big[ Q^{\pi}(s,a) - \alpha \log \pi(a|s) \big]. \]

The soft Bellman backup for \( Q \) is

\[ Q(s,a) = r(s,a) + \gamma\, \mathbb{E}_{s' \sim P(\cdot|s,a)}[ V(s') ]. \]

For a parameterized stochastic policy \( \pi_{\theta}(a|s) \) (e.g. Gaussian with mean and log-variance from a neural network), SAC uses the following losses:

• Critic loss (often two critics as in TD3):

  \[ L_Q(\theta_Q) = \mathbb{E}_{(s,a,r,s')} \left[ \big( Q_{\theta_Q}(s,a) - y \big)^2 \right], \]

  where

  \[ y = r + \gamma\, \mathbb{E}_{a' \sim \pi_{\theta}(\cdot|s')} \big[ \min_i Q_{\theta_{Q_i}'}(s',a') - \alpha \log \pi_{\theta}(a'|s') \big]. \]

• Policy loss (reparameterization trick):

  \[ L_{\pi}(\theta) = \mathbb{E}_{s \sim \rho^{\mu},\, \epsilon \sim \mathcal{N}(0,I)} \Big[ \alpha \log \pi_{\theta}(f_{\theta}(s,\epsilon)|s) - Q_{\bar{\theta}_Q}\big(s, f_{\theta}(s,\epsilon)\big) \Big], \]

  where \( f_{\theta} \) is the differentiable mapping from noise \( \epsilon \) to actions (e.g. squashed Gaussian).

In robotics, SAC is often preferred because its stochasticity leads to robust behaviors that tolerate model mismatch between simulator and real robot, and automatic temperature tuning can adapt exploration to the difficulty of the task.

flowchart LR
  ALG["Off-policy RL algorithms"] --> DDPG["DDPG: deterministic actor, \n1 critic, target nets"]
  ALG --> TD3["TD3: twin critics, \ndelayed actor, smoothing"]
  ALG --> SAC["SAC: stochastic actor, \nentropy bonus, twin critics"]
  DDPG --> PROP1["Pros: simple, \nsample-efficient"]
  DDPG --> CONS1["Cons: brittle, \noverestimation"]
  TD3 --> PROP2["Pros: more stable, \nless bias"]
  TD3 --> CONS2["Cons: more parameters, \nslower updates"]
  SAC --> PROP3["Pros: robustness via entropy, \ngood in robotics"]
  SAC --> CONS3["Cons: temperature tuning, \nstochastic actions"]
        

When applying DDPG/TD3/SAC to robots, one must respect the kinematics and dynamics you already know from earlier courses. Some important design choices:

• Action parameterization. Actions should correspond to feasible low-level commands: joint torques, desired joint velocities, or short-horizon position increments. Saturation limits should match physical actuator bounds.
• State normalization. Normalize joint angles, velocities, and end-effector positions to comparable scales to avoid ill-conditioned gradients in the critic.
• Reward shaping. Typical rewards include negative distance to a target pose, control effort penalties \( \|a\|^2 \), and terminal bonuses for successful grasping or reaching. Careful shaping is required to avoid local minima.
• Safety in real robots. During sim-to-real transfer, clip actions and use safety monitors at the control layer to prevent joint limit violations or self-collisions.

The underlying gradient-based updates are agnostic to robot dynamics modeling, but training is greatly improved when the observation space encodes relevant dynamic quantities (e.g. joint velocities and contact flags) rather than purely geometric states.

Below is a compact Python sketch of a TD3-like agent controlling a 6-DOF manipulator in a Gym-style environment. In practice you would use libraries such as gymnasium, pybullet, or mujoco for simulation.

import numpy as np
import torch
import torch.nn as nn
import torch.optim as optim

# Simple MLP networks for actor and critic
class Actor(nn.Module):
    def __init__(self, state_dim, action_dim, max_action):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(state_dim, 256), nn.ReLU(),
            nn.Linear(256, 256), nn.ReLU(),
            nn.Linear(256, action_dim), nn.Tanh()
        )
        self.max_action = max_action

    def forward(self, state):
        return self.max_action * self.net(state)

class Critic(nn.Module):
    # Twin critics Q1 and Q2
    def __init__(self, state_dim, action_dim):
        super().__init__()
        self.q1 = nn.Sequential(
            nn.Linear(state_dim + action_dim, 256), nn.ReLU(),
            nn.Linear(256, 256), nn.ReLU(),
            nn.Linear(256, 1)
        )
        self.q2 = nn.Sequential(
            nn.Linear(state_dim + action_dim, 256), nn.ReLU(),
            nn.Linear(256, 256), nn.ReLU(),
            nn.Linear(256, 1)
        )

    def forward(self, state, action):
        x = torch.cat([state, action], dim=-1)
        q1 = self.q1(x)
        q2 = self.q2(x)
        return q1, q2

    def q1_only(self, state, action):
        x = torch.cat([state, action], dim=-1)
        return self.q1(x)

class ReplayBuffer:
    def __init__(self, max_size, state_dim, action_dim):
        self.max_size = max_size
        self.ptr = 0
        self.size = 0
        self.state = np.zeros((max_size, state_dim), dtype=np.float32)
        self.action = np.zeros((max_size, action_dim), dtype=np.float32)
        self.next_state = np.zeros((max_size, state_dim), dtype=np.float32)
        self.reward = np.zeros((max_size, 1), dtype=np.float32)
        self.done = np.zeros((max_size, 1), dtype=np.float32)

    def add(self, s, a, r, s_next, d):
        self.state[self.ptr] = s
        self.action[self.ptr] = a
        self.next_state[self.ptr] = s_next
        self.reward[self.ptr] = r
        self.done[self.ptr] = d
        self.ptr = (self.ptr + 1) % self.max_size
        self.size = min(self.size + 1, self.max_size)

    def sample(self, batch_size):
        idx = np.random.randint(0, self.size, size=batch_size)
        return (
            torch.tensor(self.state[idx]),
            torch.tensor(self.action[idx]),
            torch.tensor(self.reward[idx]),
            torch.tensor(self.next_state[idx]),
            torch.tensor(self.done[idx])
        )

class TD3Agent:
    def __init__(self, state_dim, action_dim, max_action,
                 gamma=0.99, tau=0.005,
                 policy_noise=0.2, noise_clip=0.5,
                 policy_delay=2, lr=3e-4):
        self.actor = Actor(state_dim, action_dim, max_action)
        self.actor_target = Actor(state_dim, action_dim, max_action)
        self.actor_target.load_state_dict(self.actor.state_dict())

        self.critic = Critic(state_dim, action_dim)
        self.critic_target = Critic(state_dim, action_dim)
        self.critic_target.load_state_dict(self.critic.state_dict())

        self.actor_opt = optim.Adam(self.actor.parameters(), lr=lr)
        self.critic_opt = optim.Adam(self.critic.parameters(), lr=lr)

        self.gamma = gamma
        self.tau = tau
        self.policy_noise = policy_noise
        self.noise_clip = noise_clip
        self.policy_delay = policy_delay
        self.total_it = 0
        self.max_action = max_action

    def select_action(self, state, noise_std=0.1):
        state_t = torch.tensor(state, dtype=torch.float32).unsqueeze(0)
        with torch.no_grad():
            action = self.actor(state_t).cpu().numpy()[0]
        if noise_std is not None:
            action += np.random.normal(0.0, noise_std, size=action.shape)
        return np.clip(action, -self.max_action, self.max_action)

    def train(self, replay, batch_size=256):
        if replay.size < batch_size:
            return

        self.total_it += 1
        state, action, reward, next_state, done = replay.sample(batch_size)

        with torch.no_grad():
            # Target policy smoothing
            noise = torch.normal(0.0, self.policy_noise,
                                 size=action.shape)
            noise = torch.clamp(noise, -self.noise_clip, self.noise_clip)

            next_action = self.actor_target(next_state)
            next_action = torch.clamp(
                next_action + noise,
                -self.max_action,
                self.max_action
            )

            q1_targ, q2_targ = self.critic_target(next_state, next_action)
            q_min = torch.min(q1_targ, q2_targ)
            target_q = reward + self.gamma * (1.0 - done) * q_min

        # Critic update
        q1, q2 = self.critic(state, action)
        critic_loss = nn.MSELoss()(q1, target_q) + nn.MSELoss()(q2, target_q)

        self.critic_opt.zero_grad()
        critic_loss.backward()
        self.critic_opt.step()

        # Delayed policy update
        if self.total_it % self.policy_delay == 0:
            actor_loss = -self.critic.q1_only(state, self.actor(state)).mean()
            self.actor_opt.zero_grad()
            actor_loss.backward()
            self.actor_opt.step()

            # Polyak averaging for target networks
            with torch.no_grad():
                for p, p_targ in zip(self.critic.parameters(),
                                     self.critic_target.parameters()):
                    p_targ.data.mul_(1.0 - self.tau)
                    p_targ.data.add_(self.tau * p.data)
                for p, p_targ in zip(self.actor.parameters(),
                                     self.actor_target.parameters()):
                    p_targ.data.mul_(1.0 - self.tau)
                    p_targ.data.add_(self.tau * p.data)
      

You can integrate this agent with a robotic simulator environment whose state is composed of joint positions and velocities, and whose action is a vector of joint torque commands. The same structure can be adapted to DDPG (single critic, no smoothing) or SAC (stochastic actor, entropy regularization).

In C++, we often use Eigen for linear algebra and roscpp to communicate with the robot or a Gazebo simulation. Below is a minimal skeleton of a DDPG style critic and training step (without full neural network details).

#include <iostream>
#include <Eigen/Dense>

// Simple fully-connected layer
struct LinearLayer {
    Eigen::MatrixXf W;
    Eigen::VectorXf b;

    LinearLayer(int in_dim, int out_dim) {
        W.setRandom(out_dim, in_dim);
        b.setZero(out_dim);
    }

    Eigen::MatrixXf forward(const Eigen::MatrixXf& x) const {
        return (W * x.transpose()).transpose().rowwise() + b.transpose();
    }
};

// Very simplified critic: Q(s,a) = f([s,a])
struct Critic {
    LinearLayer l1, l2, l3;

    Critic(int state_dim, int action_dim)
        : l1(state_dim + action_dim, 256),
          l2(256, 256),
          l3(256, 1) {}

    Eigen::VectorXf forward(const Eigen::MatrixXf& state,
                            const Eigen::MatrixXf& action) const {
        Eigen::MatrixXf x(state.rows(), state.cols() + action.cols());
        x << state, action;
        auto h1 = l1.forward(x).unaryExpr([](float v){ return std::max(0.0f, v); });
        auto h2 = l2.forward(h1).unaryExpr([](float v){ return std::max(0.0f, v); });
        auto q  = l3.forward(h2);
        return q.col(0); // shape: batch_size
    }
};

// Example update step (pseudo-code, no autodiff)
void updateCritic(Critic& critic,
                  const Eigen::MatrixXf& s_batch,
                  const Eigen::MatrixXf& a_batch,
                  const Eigen::VectorXf& y_target,
                  float lr) {
    // In practice use a deep learning library (PyTorch C++ frontend,
    // TensorFlow C API, or custom autodiff) to compute gradients.
    // Here we only outline the interface.
    // 1. Compute q_pred = critic.forward(s_batch, a_batch);
    // 2. Compute loss = mean((q_pred - y_target)^2);
    // 3. Backpropagate to get dW, db for each layer.
    // 4. Apply gradient descent: W -= lr * dW; b -= lr * db;
}

// ROS node skeleton to interact with a robot
int main(int argc, char** argv) {
    // ros::init(argc, argv, "ddpg_controller");
    // ros::NodeHandle nh;
    int state_dim = 14;  // e.g. 7 joint angles + 7 velocities
    int action_dim = 7;  // torques for 7 joints
    Critic critic(state_dim, action_dim);

    // In a training loop:
    //  - subscribe to joint states
    //  - compute action from actor network
    //  - publish torque commands
    //  - store (s,a,r,s_next,done) in replay buffer
    //  - sample minibatches and call updateCritic(...)
    std::cout << "DDPG critic initialized." << std::endl;
    return 0;
}
      

For a full implementation, you would replace the manual neural network and backpropagation with a C++ deep learning framework such as the PyTorch C++ frontend, and use ROS topics or services for interaction with the robot hardware or simulator.

In Java, the deeplearning4j ecosystem provides RL4J for RL. Below is a conceptual outline for building a continuous control agent for a robotic environment exposed through a Java interface or a ROS bridge.

import org.deeplearning4j.rl4j.mdp.MDP;
import org.deeplearning4j.rl4j.space.ObservationSpace;
import org.deeplearning4j.rl4j.space.ActionSpace;
import org.deeplearning4j.rl4j.observation.Observation;
import org.nd4j.linalg.api.ndarray.INDArray;

// Custom MDP for a robotic arm
public class ArmContinuousMDP implements MDP<Observation, double[], ArmState> {
    private boolean done = false;

    @Override
    public Observation reset() {
        // Reset simulator or real robot to a safe home configuration
        // Read joint states, end-effector pose, etc.
        ArmState state = readStateFromRobot();
        return new Observation(state.toINDArray());
    }

    @Override
    public void close() {
        // Cleanup connections
    }

    @Override
    public StepReply<Observation> step(double[] action) {
        // Send torque or velocity command to robot
        sendActionToRobot(action);
        ArmState nextState = readStateFromRobot();
        double reward = computeReward(nextState);
        done = checkTermination(nextState);
        return new StepReply<>(
            new Observation(nextState.toINDArray()),
            reward,
            done,
            null
        );
    }

    @Override
    public boolean isDone() {
        return done;
    }

    @Override
    public ObservationSpace<Observation> getObservationSpace() {
        // Define dimension and bounds of state vector
        return null; // implement
    }

    @Override
    public ActionSpace<double[]> getActionSpace() {
        // Define action dimension and bounds for torques
        return null; // implement
    }

    // ... other required methods ...
}

// High-level: configure a continuous actor-critic agent (DDPG/TD3 style)
// through RL4J's policy interfaces, and train in the above MDP.
      

While RL4J focuses more on discrete control out of the box, the general pattern is the same: wrap the robotics interface as an MDP and plug in a continuous control agent based on DDPG/TD3/SAC principles.

11.1 MATLAB/Simulink with Reinforcement Learning Toolbox

MATLAB's Reinforcement Learning Toolbox provides ready-made agents such as rlDDPGAgent and rlSACAgent. You can connect them to a Simulink model of a robot arm built with Simscape Multibody.

% Define observation and action specifications
obsDim = 14;  % joint angles + velocities
actDim = 7;   % joint torques

obsInfo = rlNumericSpec([obsDim 1], ...
    'LowerLimit', -ones(obsDim,1), ...
    'UpperLimit',  ones(obsDim,1));
obsInfo.Name = 'observations';

actInfo = rlNumericSpec([actDim 1], ...
    'LowerLimit', -ones(actDim,1), ...
    'UpperLimit',  ones(actDim,1));
actInfo.Name = 'torques';

% Create Simulink environment
mdl = 'RobotArmTD3Model';
load_system(mdl);
env = rlSimulinkEnv(mdl, 'RobotArmTD3Model/RL Agent', obsInfo, actInfo);

% Actor and critic networks (deep neural networks)
statePath = featureInputLayer(obsDim, 'Normalization', 'none', 'Name', 'state');
actionPath = featureInputLayer(actDim, 'Normalization', 'none', 'Name', 'action');

criticNetwork = [
    concatenationLayer(1, 2, 'Name', 'concat')
    fullyConnectedLayer(256, 'Name', 'fc1')
    reluLayer('Name', 'relu1')
    fullyConnectedLayer(256, 'Name', 'fc2')
    reluLayer('Name', 'relu2')
    fullyConnectedLayer(1, 'Name', 'q')];

criticGraph = layerGraph(criticNetwork);
criticGraph = addLayers(criticGraph, statePath);
criticGraph = addLayers(criticGraph, actionPath);
criticGraph = connectLayers(criticGraph, 'state', 'concat/in1');
criticGraph = connectLayers(criticGraph, 'action', 'concat/in2');

criticOpts = rlRepresentationOptions('LearnRate',1e-3,'GradientThreshold',1);
critic = rlQValueRepresentation(criticGraph, obsInfo, actInfo, ...
    'Observation', {'state'}, 'Action', {'action'}, criticOpts);

actorNetwork = [
    featureInputLayer(obsDim, 'Normalization', 'none', 'Name', 'state')
    fullyConnectedLayer(256, 'Name', 'fc1')
    reluLayer('Name', 'relu1')
    fullyConnectedLayer(256, 'Name', 'fc2')
    reluLayer('Name', 'relu2')
    fullyConnectedLayer(actDim, 'Name', 'fc3')
    tanhLayer('Name', 'tanh1')];

actorOpts = rlRepresentationOptions('LearnRate',1e-4,'GradientThreshold',1);
actor = rlDeterministicActorRepresentation(actorNetwork, obsInfo, actInfo, ...
    'Observation', {'state'}, 'Action', {'tanh1'}, actorOpts);

agentOpts = rlDDPGAgentOptions(...
    'SampleTime', 0.02, ...
    'TargetSmoothFactor', 5e-3, ...
    'DiscountFactor', 0.99, ...
    'MiniBatchSize', 256);

agent = rlDDPGAgent(actor, critic, agentOpts);

% Train the agent
trainOpts = rlTrainingOptions(...
    'MaxEpisodes', 1000, ...
    'MaxStepsPerEpisode', 1000, ...
    'StopTrainingCriteria', 'AverageReward', ...
    'StopTrainingValue', 200);

trainingStats = train(agent, env, trainOpts);
      

The same environment can be used to train SAC agents by replacing rlDDPGAgent with rlSACAgent and adjusting the options to include entropy temperature and stochastic actor structures.

11.2 Wolfram Mathematica (Symbolic Soft Bellman Backup)

Mathematica can be used to manipulate symbolic Bellman equations and verify properties of the soft value operator used in SAC.

(* Define symbolic soft Q and V for a finite state-action set *)
Clear[gamma, alpha, Q, V, P, r, pi];
gamma = Symbol["\[Gamma]"];
alpha = Symbol["\[Alpha]"];

(* Soft Bellman backup for V given Q and policy pi *)
softV[s_] := Sum[
  pi[s, a] * (Q[s, a] - alpha * Log[pi[s, a]]),
  {a, Actions}
];

(* Soft Bellman backup for Q given V *)
softTQ[s_, a_] := r[s, a] + gamma * Sum[
  P[s, a, sNext] * softV[sNext],
  {sNext, States}
];

(* Verify contraction property numerically for a simple MDP *)
States = {s1, s2};
Actions = {a1, a2};

(* Example numeric instantiation *)
P[s1, a1, s1] = 0.5; P[s1, a1, s2] = 0.5;
P[s1, a2, s1] = 0.3; P[s1, a2, s2] = 0.7;
P[s2, a1, s1] = 0.4; P[s2, a1, s2] = 0.6;
P[s2, a2, s1] = 0.2; P[s2, a2, s2] = 0.8;

r[s_, a_] := 0.0;
pi[s_, a_] := 0.5;
gammaVal = 0.9;
alphaVal = 0.1;

(* Represent Q as a vector and iterate softTQ *)
qVec0 = {0.0, 0.0, 0.0, 0.0}; (* Q(s1,a1), Q(s1,a2), Q(s2,a1), Q(s2,a2) *)

SoftBackup[q_List] := Module[
  {qs, v1, v2, qNew},
  qs[s1, a1] = q[[1]];
  qs[s1, a2] = q[[2]];
  qs[s2, a1] = q[[3]];
  qs[s2, a2] = q[[4]];

  softVNum[s_] := Sum[
    pi[s, a] * (qs[s, a] - alphaVal * Log[pi[s, a]]),
    {a, Actions}
  ];

  softTQNum[s_, a_] := gammaVal * Sum[
    P[s, a, sNext] * softVNum[sNext],
    {sNext, States}
  ];

  qNew = {
    softTQNum[s1, a1],
    softTQNum[s1, a2],
    softTQNum[s2, a1],
    softTQNum[s2, a2]
  };
  qNew
];

NestList[SoftBackup, qVec0, 10] // MatrixForm
      

This code shows how to construct and iterate a soft Bellman backup in a toy MDP, illustrating convergence numerically and providing a symbolic playground to understand maximum-entropy RL before deploying SAC in high-dimensional robotic tasks.

Problem 1 (Deterministic Policy Gradient Theorem): Starting from the performance objective \( J(\theta) = \mathbb{E}_{s \sim \rho^{\mu_{\theta}}} [ Q^{\mu_{\theta}}(s,\mu_{\theta}(s)) ] \), derive the deterministic policy gradient expression

\[ \nabla_{\theta} J(\theta) = \mathbb{E}_{s \sim \rho^{\mu_{\theta}}} \Big[ \nabla_{\theta}\mu_{\theta}(s)\, \nabla_{a} Q^{\mu_{\theta}}(s,a)\big|_{a = \mu_{\theta}(s)} \Big]. \]

Solution:

1. Expand the objective: \( J(\theta) = \sum_{s} \rho^{\mu_{\theta}}(s)\, Q^{\mu_{\theta}}(s,\mu_{\theta}(s)) \).
2. Differentiate: \( \nabla_{\theta} J = \sum_{s} \nabla_{\theta}\rho^{\mu_{\theta}}(s)\, Q^{\mu_{\theta}}(s,\mu_{\theta}(s)) + \rho^{\mu_{\theta}}(s)\, \nabla_{\theta} Q^{\mu_{\theta}}(s,\mu_{\theta}(s)) \).
3. Use the Bellman equation to express \( \nabla_{\theta} Q^{\mu_{\theta}} \) recursively and collect terms; the contribution from \( \nabla_{\theta}\rho^{\mu_{\theta}} \) cancels with future-state contributions, exactly as in the standard policy gradient theorem.
4. The remaining term is the immediate dependence of the action on \( \theta \), giving \( \rho^{\mu_{\theta}}(s)\, \nabla_{\theta}\mu_{\theta}(s)\, \nabla_{a} Q^{\mu_{\theta}}(s,a) \big|_{a = \mu_{\theta}(s)} \), and taking expectation over \( s \sim \rho^{\mu_{\theta}} \) yields the result.

Problem 2 (Overestimation Bias and Double Critics): Assume a critic approximation error \( \epsilon(s,a) \) with zero mean but nonzero variance. Show heuristically why the single-critic target in DDPG, \( y = r + \gamma Q_{\theta'}(s',\mu_{\theta_{\mu}'}(s')) \), tends to overestimate the true value, and why TD3's target using \( \min(Q_1,Q_2) \) reduces this bias.

Solution:

• Let the true value be \( Q^{\ast}(s',a') \) and the critic be \( Q_{\theta'}(s',a') = Q^{\ast}(s',a') + \epsilon(s',a') \), with \( \mathbb{E}[\epsilon] = 0 \) but \( \epsilon \) varying with \( a' \).
• The actor \( \mu_{\theta_{\mu}'} \) is trained to maximize the approximate critic, so at convergence \( a' = \mu_{\theta_{\mu}'}(s') \) tends to select actions where \( \epsilon(s',a') \) is positive. Hence \( \mathbb{E}[Q_{\theta'}(s',a')] \geq Q^{\ast}(s',a') \).
• With two independent critics \( Q_i = Q^{\ast} + \epsilon_i \), the expected minimum satisfies \( \mathbb{E}[\min(Q_1,Q_2)] \leq Q^{\ast} \), but the negative bias is smaller in magnitude than the positive bias of the maximum. Empirically, the minimum of two critics provides a conservative but less biased target than a single optimistic critic.
• Thus TD3's target \( y = r + \gamma \min(Q_1',Q_2') \) mitigates the overestimation that destabilizes DDPG in continuous robotic tasks.

Problem 3 (Soft Bellman Optimality and Policy Form): Consider a single-state MDP with finite actions and soft Q-function \( Q(a) \). Show that the policy that maximizes \( V = \mathbb{E}_{a \sim \pi} [ Q(a) - \alpha \log \pi(a) ] \) over distributions \( \pi \) on actions has the Boltzmann form

\[ \pi^{\ast}(a) = \frac{\exp\big( Q(a) / \alpha \big)} {\sum_{a'} \exp\big( Q(a') / \alpha \big)}. \]

Solution:

1. Form the Lagrangian with normalization constraint \( \sum_{a} \pi(a) = 1 \): \( \mathcal{L}(\pi,\lambda) = \sum_{a} \pi(a)\big( Q(a) - \alpha \log \pi(a) \big) + \lambda\big(1 - \sum_{a} \pi(a)\big). \)
2. Take partial derivatives: \( \frac{\partial \mathcal{L}}{\partial \pi(a)} = Q(a) - \alpha(1 + \log \pi(a)) - \lambda = 0. \)
3. Solve for \( \log \pi(a) \): \( \log \pi(a) = \frac{Q(a)}{\alpha} - 1 - \frac{\lambda}{\alpha}. \) Thus \( \pi(a) \propto \exp(Q(a)/\alpha). \)
4. Enforcing normalization yields the Boltzmann distribution stated.

Problem 4 (Off-Policy Bellman Operator Contraction): Let \( \mathcal{T}^{\pi} \) be the Bellman operator for a fixed target policy \( \pi \). Prove that for any functions \( Q_1, Q_2 \), \( \| \mathcal{T}^{\pi} Q_1 - \mathcal{T}^{\pi} Q_2 \|_{\infty} \leq \gamma \| Q_1 - Q_2 \|_{\infty} \), independent of the behavior policy.

Solution:

1. For any \( (s,a) \), \( (\mathcal{T}^{\pi} Q_1)(s,a) - (\mathcal{T}^{\pi} Q_2)(s,a) = \gamma \mathbb{E}_{s',a'}\big[ Q_1(s',a') - Q_2(s',a') \big]. \)
2. Take absolute value and apply Jensen's inequality: \( \big| (\mathcal{T}^{\pi} Q_1)(s,a) - (\mathcal{T}^{\pi} Q_2)(s,a) \big| \leq \gamma \mathbb{E}_{s',a'}\big[ | Q_1(s',a') - Q_2(s',a') | \big]. \)
3. Bound by the sup norm: \( \mathbb{E}[ | Q_1 - Q_2 | ] \leq \| Q_1 - Q_2 \|_{\infty}. \)
4. Taking the supremum over \( (s,a) \) yields the desired contraction. Note that the expectation is over \( P \) and \( \pi \) only; the behavior policy does not appear, so the contraction holds in the off-policy case where samples are drawn from another distribution.

Problem 5 (Action Squashing and Torque Limits): A DDPG actor outputs \( u(s) \in \mathbb{R}^n \) which is squashed through a \( \tanh \) and scaled to torque limits \( a_{\max} \in \mathbb{R}^n \): \( a(s) = a_{\max} \odot \tanh(u(s)) \). Derive \( \nabla_{\theta} a(s) \) in terms of \( \nabla_{\theta} u(s) \), and explain why very large \( \|u(s)\| \) can slow learning.

Solution:

1. For each component, \( a_i(s) = a_{\max,i} \tanh(u_i(s)) \), so \( \frac{\partial a_i}{\partial u_i} = a_{\max,i} (1 - \tanh^2(u_i(s))). \)
2. By the chain rule, \( \nabla_{\theta} a_i(s) = a_{\max,i} (1 - \tanh^2(u_i(s))) \nabla_{\theta} u_i(s). \)
3. When \( |u_i(s)| \) is large, \( \tanh(u_i(s)) \) saturates near \( \pm 1 \), so \( 1 - \tanh^2(u_i(s)) \approx 0 \), and the gradient \( \nabla_{\theta} a_i(s) \) becomes small even if \( \nabla_{\theta} u_i(s) \) is large.
4. This gradient saturation slows learning, especially for outer regions of the torque space. In practice one keeps network outputs in a moderate range and uses appropriate initialization to avoid immediate saturation.

In this lesson, we formalized off-policy actor–critic methods for continuous robotic control. Starting from the Bellman operator and the deterministic policy gradient theorem, we derived the core updates of DDPG, TD3, and SAC. We highlighted how double critics, delayed policy updates, and entropy regularization address overestimation, instability, and exploration in high-DOF robotic systems. Finally, we sketched implementations across Python, C++, Java, MATLAB/Simulink, and Mathematica, providing a bridge from theoretical constructs to practical controllers for real robots.