In reinforcement learning (RL) for robotics, the robot is modeled as an MDP \( (\mathcal{S}, \mathcal{A}, P, r, \gamma) \), where the state \( s_t \in \mathcal{S} \) encodes joint positions, velocities, and task variables; the action \( a_t \in \mathcal{A} \) denotes torques or commanded velocities; and the reward \( r(s_t, a_t, s_{t+1}) \) encodes task performance. For a stationary stochastic policy \( \pi_\theta(a \mid s) \), the standard discounted objective is

\[ J(\pi_\theta) = \mathbb{E}_{\pi_\theta}\!\left[ \sum_{t=0}^{\infty} \gamma^{t} r(s_t, a_t, s_{t+1}) \right], \quad 0 \le \gamma < 1. \]

Reward design is the process of choosing \( r \) so that maximizing \( J(\pi_\theta) \) corresponds to solving the robotics task we actually care about (e.g., reaching, manipulation, locomotion), while maintaining a learning signal that is informative and numerically well-conditioned.

In robotics, naive sparse rewards (e.g., \( r=1 \) for task success, \( r=0 \) otherwise) cause credit assignment problems and high-variance gradients. Dense rewards make learning faster but can distort optimal behavior if poorly designed. A central theme of this lesson is that some reward transformations are behavior-preserving, while others fundamentally change the optimal policy.

flowchart TD
  T["Robot task specification"] --> Rspec["Raw objective (success, constraints)"]
  Rspec --> Design["Design reward r(s,a,s')"]
  Design --> Analyze["Check shaping theory (potential-based?)"]
  Analyze --> Train["Train RL policy with chosen algorithm"]
  Train --> Eval["Evaluate task success and side-effects"]
  Eval --> Adjust["Adjust reward / shaping / tricks"]
  Adjust --> Design
        

Let the original reward be \( r(s,a,s') \). A shaped reward has the form

\[ r'(s,a,s') = r(s,a,s') + F(s,s') \]

for some function \( F : \mathcal{S}\times\mathcal{S} \to \mathbb{R} \). In general, changing the reward changes the optimal policy. However, a special class of shaping functions preserves optimality: potential-based shaping.

A shaping function is potential-based if there exists a scalar potential \( \Phi : \mathcal{S} \to \mathbb{R} \) such that

\[ F(s,s') = \gamma \,\Phi(s') - \Phi(s), \quad 0 \le \gamma < 1. \]

The shaped reward is then

\[ r'(s,a,s') = r(s,a,s') + \gamma \,\Phi(s') - \Phi(s). \]

Intuitively, \( \Phi \) measures “how good” a state is; we add reward for moving to higher-potential states and subtract reward for leaving high-potential states. In robotics, natural choices for \( \Phi \) include negative task-space distance to the goal or a Lyapunov-like function measuring deviation from a desired configuration.

2.1. Telescoping of Shaped Returns

Consider the (undiscounted) return from time \( t \) in the shaped MDP:

\[ \begin{aligned} G'_t &= \sum_{k=t}^{\infty} \gamma^{k-t} \Big(r(s_k, a_k, s_{k+1}) + \gamma \Phi(s_{k+1}) - \Phi(s_k)\Big) \\ &= \underbrace{ \sum_{k=t}^{\infty} \gamma^{k-t} r(s_k,a_k,s_{k+1}) }_{G_t} + \sum_{k=t}^{\infty} \gamma^{k-t+1}\Phi(s_{k+1}) - \sum_{k=t}^{\infty} \gamma^{k-t}\Phi(s_k). \end{aligned} \]

Re-indexing the middle sum with \( j=k+1 \) gives

\[ \sum_{k=t}^{\infty} \gamma^{k-t+1}\Phi(s_{k+1}) = \sum_{j=t+1}^{\infty} \gamma^{j-t}\Phi(s_j). \]

Therefore,

\[ G'_t = G_t - \Phi(s_t) + \lim_{K \rightarrow \infty} \gamma^{K-t+1}\Phi(s_{K+1}). \]

If \( \Phi \) is bounded and \( 0 \le \gamma < 1 \), the limit term vanishes and

\[ G'_t = G_t - \Phi(s_t). \]

Thus, for any fixed policy \( \pi \), \( G'_t \) differs from \( G_t \) only by a state-dependent baseline \( -\Phi(s_t) \). This implies a simple relation between value functions:

\[ V'^{\pi}(s) = V^{\pi}(s) - \Phi(s), \quad Q'^{\pi}(s,a) = Q^{\pi}(s,a) - \Phi(s). \]

Consequently, for each state \( s \),

\[ \arg\max_{a} Q'^{\pi}(s,a) = \arg\max_{a} Q^{\pi}(s,a), \]

i.e., the action ordering is unchanged. At optimality this yields the classic policy invariance under potential-based shaping result.

In robotics, this guarantees that we can add well-designed distance-based shaping rewards without changing the optimal motion plan, while significantly improving the gradient signal observed by actor–critic methods.

Let the robot state be \( s_t = (q_t, \dot{q}_t, x_t) \) where \( q_t \) are joint positions, \( \dot{q}_t \) joint velocities and \( x_t \) the end-effector pose in task space. Let the goal pose be \( x^{\star} \) and the action (torque or command) be \( u_t \). A common dense reward structure is

\[ r_t = r(s_t, u_t, s_{t+1}) = r_{\text{task}}(s_t) + r_{\text{control}}(u_t) + r_{\text{terminal}}(s_{t+1}). \]

Typical choices include:

• Task distance term:

  \[ r_{\text{task}}(s_t) = - \| x_t - x^{\star} \|_Q^2 = - (x_t - x^{\star})^{\top} Q (x_t - x^{\star}), \]

  where \( Q \succ 0 \) weights position vs orientation error.
• Control-effort penalty:

  \[ r_{\text{control}}(u_t) = - \lambda_u \, \|u_t\|_2^2, \quad \lambda_u > 0, \]

  penalizing aggressive torques and indirectly regularizing the learned policy.
• Terminal success bonus:

  \[ r_{\text{terminal}}(s_{t+1}) = \begin{cases} R_{\text{succ}} & \text{if } d(x_{t+1}, x^{\star}) \le \epsilon, \\ 0 & \text{otherwise,} \end{cases} \]

  capturing discrete success/failure at the end of an episode or when a goal tolerance is met.

A potential-based shaping term can be added by choosing \( \Phi(s) = -\alpha \, d(x, x^{\star}) \) for some \( \alpha > 0 \), giving

\[ r'(s_t, u_t, s_{t+1}) = r(s_t, u_t, s_{t+1}) + \gamma (-\alpha d(x_{t+1}, x^{\star})) + \alpha d(x_t, x^{\star}). \]

This encourages sequences of states that monotonically reduce task-space distance, without changing the optimal solution in theory. Care is still needed: if the dynamics or time-discretization used in simulation differ from reality, policies that exploit shaping terms can overfit to simulation artifacts, which is why Chapter 13 explores sim-to-real issues.

Many robotic tasks (e.g., opening a door, inserting a plug) admit a natural sparse reward: the agent receives \( +1 \) when the task is successfully completed, and \( 0 \) otherwise. Formally, with horizon \( T \) and (random) success time \( T_{\text{succ}} \), we may define

\[ r_t = \begin{cases} 1 & \text{if } t = T_{\text{succ}} \le T, \\ 0 & \text{otherwise.} \end{cases} \]

Let \( p_{\pi} = \mathbb{P}_{\pi}(T_{\text{succ}} \le T) \) be the success probability under policy \( \pi \). The expected return is

\[ J(\pi) = \mathbb{E}[G_0] = \mathbb{E}\big[\gamma^{T_{\text{succ}}} \mathbf{1}\{T_{\text{succ}} \le T\}\big] = \sum_{t=0}^{T} \gamma^t \, \mathbb{P}_{\pi}(T_{\text{succ}} = t). \]

When \( p_{\pi} \) is small (e.g., random exploration of a complex manipulation task), most trajectories yield zero reward, causing the Monte Carlo gradient estimator (for policy gradients)

\[ \hat{g} = \frac{1}{N}\sum_{i=1}^{N}\sum_{t=0}^{T^{(i)}} \nabla_{\theta} \log \pi_{\theta}(a_t^{(i)} \mid s_t^{(i)}) \, (G_t^{(i)} - b_t) \]

to have very high variance: most \( G_t^{(i)} \) are zero, while a tiny fraction are non-zero and dominate the gradient estimate. The variance grows roughly on the order of \( \mathcal{O}(p_{\pi}(1-p_{\pi})) \) for Bernoulli-like returns, and when \( p_{\pi} \) is exponentially small in the task dimension, naive RL becomes impractical.

Reward shaping in robotics is therefore not just a convenience; it is often the difference between feasible learning and hopeless exploration, especially when real-world interaction is expensive and safety constraints limit exploration.

5.1. Distance-Based Dense Rewards

The simplest sparse-reward trick is to add a dense shaping term proportional to the negative distance to the goal, as described in Section 3. For a goal-reaching task, define \( d_t = d(x_t, x^{\star}) \) and

\[ r_t = -\alpha d_t + \beta \mathbf{1}\{d_t \le \epsilon\}, \quad \alpha > 0, \; \beta > 0. \]

If we choose \( \Phi(s) = -\alpha d(x, x^{\star}) \), then the dense term is closely related to potential-based shaping, and for small time-steps the theoretical policy-invariance arguments approximately hold.

5.2. Hindsight Experience Replay (HER)

For goal-conditioned robotic tasks, we augment the state with a desired goal \( g \) and define a goal-conditioned reward \( r(s,a,s',g) \). A trajectory \( \tau = (s_0, a_0, \dots, s_T) \) that fails to achieve \( g \) is often successful for alternative goals, e.g., “where the end-effector actually ended up”.

HER constructs additional transitions by relabelling the goal with a function \( g' = h(s_k) \) of a later state along the same trajectory and recomputing rewards \( r(s_t, a_t, s_{t+1}, g') \). In manipulation, this drastically increases the density of non-zero rewards without modifying the underlying physical task.

5.3. Exploration Bonuses and Entropy Regularization

Sparse rewards can be complemented with intrinsic rewards that encourage exploration. For example, a simple count-based bonus in a discretized state-space is

\[ r^{\text{int}}_t = \frac{\kappa}{\sqrt{N(s_t)}}, \]

where \( N(s_t) \) is the visitation count of state \( s_t \) and \( \kappa > 0 \) controls the strength of the bonus. In continuous robotic state spaces, approximations using density models or pseudo-counts are employed. Entropy regularization, already present in algorithms like SAC (Lesson 3), can be interpreted as another way to maintain exploratory policies under sparse feedback.

flowchart TD
  S0["Collected trajectories with sparse rewards"] --> Relabel["Relabel goals / compute shaping"]
  Relabel --> Replay["Store (s,a,r',s',g') in replay buffer"]
  Replay --> Update["Off-policy update (e.g. DDPG/TD3/SAC)"]
  Update --> Policy["Improved policy with better exploration"]
        

We illustrate potential-based shaping and a minimal HER-style relabelling loop for a reaching task. We assume a Gym-like environment with observation obs that contains end-effector position ee_pos and goal position goal.

import numpy as np

# Base dense reward for a reaching task
def base_reward(ee_pos, goal, action, success, ctrl_coeff=0.01):
    dist = np.linalg.norm(ee_pos - goal)
    r_task = -dist
    r_ctrl = -ctrl_coeff * np.dot(action, action)
    r_term = 1.0 if success else 0.0
    return r_task + r_ctrl + r_term

# Potential-based shaping with Phi(s) = -alpha * ||ee_pos - goal||
def potential(ee_pos, goal, alpha=0.5):
    return -alpha * np.linalg.norm(ee_pos - goal)

def shaped_reward(prev_obs, obs, action, gamma=0.99, alpha=0.5):
    ee_prev = prev_obs["ee_pos"]
    goal = prev_obs["goal"]
    ee = obs["ee_pos"]
    success = obs["success"]

    r_base = base_reward(ee_prev, goal, action, success)
    phi_prev = potential(ee_prev, goal, alpha=alpha)
    phi = potential(ee, goal, alpha=alpha)
    r_shape = gamma * phi - phi_prev
    return r_base + r_shape

# Minimal HER-style relabelling for a batch of transitions
def her_relabel_episode(episode, k_future=4):
    """
    episode: list of dicts with keys
      "obs", "action", "next_obs", "reward", "done"
    """
    her_transitions = []
    T = len(episode)
    for t in range(T):
        transition = episode[t]
        obs = transition["obs"]
        next_obs = transition["next_obs"]

        # sample alternative goals from future states
        future_idxs = np.random.randint(t, T, size=k_future)
        for idx in future_idxs:
            alt_goal = episode[idx]["next_obs"]["ee_pos"].copy()
            obs_her = dict(obs)
            next_obs_her = dict(next_obs)
            obs_her["goal"] = alt_goal
            next_obs_her["goal"] = alt_goal

            # recompute success flag under alternative goal
            d = np.linalg.norm(next_obs_her["ee_pos"] - alt_goal)
            success = d <= 0.02
            next_obs_her["success"] = success

            r_her = base_reward(
                obs_her["ee_pos"], obs_her["goal"], transition["action"], success
            )
            her_transitions.append(
                {
                    "obs": obs_her,
                    "action": transition["action"],
                    "next_obs": next_obs_her,
                    "reward": r_her,
                    "done": transition["done"],
                }
            )
    return her_transitions
      

In a full actor–critic implementation (e.g., SAC from Lesson 3), the replay buffer would store both original and HER-relabeled transitions. The shaping term can be toggled for ablation to empirically verify its impact on sample efficiency.

In C++, a typical RL environment for robotics would expose a method computing the reward given the current and next state. Using Eigen for vector operations, a potential-based shaped reward could be implemented as:

#include <Eigen/Dense>

struct RobotState {
    Eigen::Vector3d ee_pos;
    Eigen::Vector3d goal;
    bool success;
};

double baseReward(const RobotState& s,
                  const RobotState& s_next,
                  const Eigen::VectorXd& action,
                  double ctrl_coeff = 0.01)
{
    double dist = (s.ee_pos - s.goal).norm();
    double r_task = -dist;
    double r_ctrl = -ctrl_coeff * action.squaredNorm();
    double r_term = s_next.success ? 1.0 : 0.0;
    return r_task + r_ctrl + r_term;
}

double potential(const RobotState& s, double alpha = 0.5)
{
    return -alpha * (s.ee_pos - s.goal).norm();
}

double shapedReward(const RobotState& s,
                    const RobotState& s_next,
                    const Eigen::VectorXd& action,
                    double gamma = 0.99,
                    double alpha = 0.5)
{
    double r_base = baseReward(s, s_next, action);
    double phi_s = potential(s, alpha);
    double phi_next = potential(s_next, alpha);
    double r_shape = gamma * phi_next - phi_s;
    return r_base + r_shape;
}
      

This function can be integrated into C++ RL frameworks or custom actor–critic implementations used for high-performance simulation or embedded control loops, as long as the chosen shaping potential is consistent across training and evaluation.

In Java-based RL frameworks (e.g., RL4J on top of DeepLearning4J), the environment typically exposes a step method that returns the next observation and reward. A goal-conditioned reward with simple dense shaping can be implemented as:

public class ReachTaskEnv {

    public static class Obs {
        public double[] eePos;   // length 3
        public double[] goal;    // length 3
        public boolean success;
    }

    private double alpha = 1.0;
    private double ctrlCoeff = 0.01;

    private double distance(double[] a, double[] b) {
        double dx = a[0] - b[0];
        double dy = a[1] - b[1];
        double dz = a[2] - b[2];
        return Math.sqrt(dx * dx + dy * dy + dz * dz);
    }

    private double baseReward(Obs s, Obs sNext, double[] action) {
        double dist = distance(s.eePos, s.goal);
        double rTask = -alpha * dist;

        double normSq = 0.0;
        for (double v : action) {
            normSq += v * v;
        }
        double rCtrl = -ctrlCoeff * normSq;

        double rTerm = sNext.success ? 1.0 : 0.0;
        return rTask + rCtrl + rTerm;
    }

    public double stepReward(Obs s, Obs sNext, double[] action) {
        return baseReward(s, sNext, action);
    }
}
      

More advanced shaping (e.g., using a potential function) can be added analogously. The key point is that the reward is encapsulated in a single place and does not leak implementation details of the policy or function approximator.

MATLAB's Reinforcement Learning Toolbox represents environments via MATLAB functions or Simulink models. A function-based environment can implement a shaped reward as follows:

function [NextObs, Reward, IsDone, LoggedSignals] = step(this, Action)
% Extract current state from this.State
eePos = this.State.eePos;   % 3x1
goal  = this.State.goal;    % 3x1

% Apply action through robot dynamics (not shown)
[nextEePos, successFlag] = robotDynamicsStep(eePos, Action);

% Build next observation
NextObs.eePos = nextEePos;
NextObs.goal  = goal;
NextObs.success = successFlag;

% Base reward components
dist = norm(eePos - goal);
alpha = 1.0;
ctrlCoeff = 0.01;
rTask = -alpha * dist;
rCtrl = -ctrlCoeff * (Action'*Action);
rTerm = 1.0 * double(successFlag);

% Potential-based shaping (Phi = -beta * ||eePos - goal||)
beta = 0.5;
gamma = 0.99;
phiPrev = -beta * norm(eePos - goal);
phiNext = -beta * norm(nextEePos - goal);
rShape = gamma * phiNext - phiPrev;

Reward = rTask + rCtrl + rTerm + rShape;

% Termination condition
IsDone = successFlag;

LoggedSignals = [];
end
      

In Simulink, the same computation can be implemented inside a MATLAB Function block connected to the RL Agent block. The state and action signals enter the block, and the block outputs the scalar reward signal used by the agent during training.

In Wolfram Mathematica, we can define symbolic reward and potential functions and verify the telescoping property of potential-based shaping for a finite-horizon trajectory.

ClearAll[r, phi, gamma, states, actions];

(* Base reward for step t *)
r[st_, at_, st1_] := -Norm[st["eePos"] - st["goal"]] +
                     1.0*Boole[st1["success"] == True];

(* Potential function Phi(s) = -alpha * distance to goal *)
alpha = 0.5;
phi[st_] := -alpha*Norm[st["eePos"] - st["goal"]];

gamma = 0.99;

(* Shaped reward r'(s,a,s') *)
rShaped[st_, at_, st1_] :=
  r[st, at, st1] + gamma*phi[st1] - phi[st];

(* Verify telescoping for a finite trajectory *)
states = Table[Association[
    "eePos" -> RandomReal[{-1, 1}, 3],
    "goal" -> {0., 0., 0.},
    "success" -> False
  ], {t, 0, 4}];

actions = Table[RandomReal[{-1, 1}, 3], {t, 0, 3}];

gBase = Sum[gamma^t*r[states[[t + 1]], actions[[t + 1]], states[[t + 2]]],
            {t, 0, 3}];

gShaped = Sum[gamma^t*rShaped[states[[t + 1]], actions[[t + 1]],
                               states[[t + 2]]],
              {t, 0, 3}];

Simplify[gShaped - gBase + phi[states[[1]]] - gamma^4*phi[states[[5]]]]
      

For bounded potentials and large horizons, the last term vanishes, and the simplification shows that the difference between shaped and base returns is a state-dependent baseline, consistent with the theory in Section 2.

Problem 1 (Potential-Based Shaping on a Line): A point-mass robot moves on the real line with state \( s_t = x_t \in \mathbb{R} \) and control \( u_t \in \mathbb{R} \). The dynamics are \( x_{t+1} = x_t + u_t \). The base reward is \( r_t = -x_t^2 \). Let \( \Phi(x) = -\alpha x^2 \) with \( \alpha > 0 \). Write down the shaped reward \( r'_t \) and show explicitly that for any fixed policy, the difference between shaped and base return from time 0 is independent of the actions.

Solution: The shaped reward is

\[ r'_t = r_t + \gamma \Phi(x_{t+1}) - \Phi(x_t) = -x_t^2 + \gamma(-\alpha x_{t+1}^2) + \alpha x_t^2. \]

The shaped return is

\[ G'_0 = \sum_{t=0}^{\infty} \gamma^t r'_t = \sum_{t=0}^{\infty} \gamma^t r_t + \sum_{t=0}^{\infty} \gamma^{t+1} \Phi(x_{t+1}) - \sum_{t=0}^{\infty} \gamma^{t} \Phi(x_t). \]

As in Section 2, the two sums in \( \Phi \) telescope, giving \( G'_0 = G_0 - \Phi(x_0) \) up to a vanishing tail term. This difference depends only on the initial state and not on the actions, hence optimal policies are unaffected.

Problem 2 (Action-Ordering Invariance): Let \( Q^{\pi}(s,a) \) and \( Q'^{\pi}(s,a) \) be the state-action value functions under base and shaped rewards with potential-based shaping. Prove that for any state \( s \) and any two actions \( a_1, a_2 \), we have \( Q^{\pi}(s,a_1) \ge Q^{\pi}(s,a_2) \) if and only if \( Q'^{\pi}(s,a_1) \ge Q'^{\pi}(s,a_2) \).

Solution: From Section 2, we have

\[ Q'^{\pi}(s,a) = Q^{\pi}(s,a) - \Phi(s). \]

Thus,

\[ Q'^{\pi}(s,a_1) - Q'^{\pi}(s,a_2) = \big(Q^{\pi}(s,a_1) - \Phi(s)\big) - \big(Q^{\pi}(s,a_2) - \Phi(s)\big) = Q^{\pi}(s,a_1) - Q^{\pi}(s,a_2). \]

The difference is identical, so the sign of the difference is preserved. Hence \( Q^{\pi}(s,a_1) \ge Q^{\pi}(s,a_2) \) if and only if \( Q'^{\pi}(s,a_1) \ge Q'^{\pi}(s,a_2) \).

Problem 3 (Probability of Ever Seeing Reward): Consider a sparse-reward robotic task where, under the initial random policy, the probability of success in a single episode is \( p \). Assume episodes are independent. Compute the probability that after \( N \) episodes, the agent has never observed a non-zero reward. For \( p \ll 1 \), approximate \( N \) required so that this probability is at most \( \delta \).

Solution: The probability of zero reward in a single episode is \( 1-p \). Independence gives

\[ \mathbb{P}(\text{no non-zero reward in } N \text{ episodes}) = (1-p)^N. \]

We require \( (1-p)^N \le \delta \). Taking logs and using \( \log(1-p) \approx -p \) for small \( p \),

\[ N \ge \frac{\log \delta}{\log(1-p)} \approx \frac{\log \delta}{-p}. \]

Since \( \log \delta < 0 \), this is \( N \approx \frac{|\log \delta|}{p} \), which can be enormous when \( p \) is small, illustrating why purely sparse rewards are impractical in complex robotic tasks.

Problem 4 (Designing a Reward for Pick-and-Place): Consider a 6-DOF manipulator tasked with picking up an object from the table and placing it into a box. Propose a dense reward decomposition \( r_t = r_{\text{reach}} + r_{\text{grasp}} + r_{\text{place}} + r_{\text{ctrl}} \). Explain how each term should depend on the state and action, and sketch how you would progressively adjust the weights during training.

Solution: A possible design is:

• Reaching term: \( r_{\text{reach}} = -\alpha_1 d(x_t^{\text{ee}}, x^{\text{obj}}) \), where \( x_t^{\text{ee}} \) is end-effector pose and \( x^{\text{obj}} \) the object pose.
• Grasp term: \( r_{\text{grasp}} = \beta_1 \mathbf{1}\{\text{object lifted}\} \).
• Placing term: \( r_{\text{place}} = \beta_2 \mathbf{1}\{\text{object in box}\} \).
• Control term: \( r_{\text{ctrl}} = -\lambda \|u_t\|_2^2 \).

Initially, \( \alpha_1 \) is set relatively large to force reaching behavior, while \( \beta_2 \) is moderate. As the policy becomes competent at reaching, \( \alpha_1 \) can be reduced and \( \beta_2 \) increased to emphasize successful placement. This manual curriculum approximates a shaping potential that favours states closer to the complete pick-and-place sequence.

flowchart TD
  Start["Start state (arm at rest)"] --> Reach["Reaching phase: maximize -dist(ee,obj)"]
  Reach --> Grasp["Grasp phase: close gripper, lift object"]
  Grasp --> Move["Move phase: move object towards box region"]
  Move --> Place["Place phase: release object in box"]
  Place --> Done["Done: large terminal bonus r_place"]
        

Problem 5 (Is This Shaping Potential-Based?): Let the base reward be \( r(s,a,s') \). Consider the transformed reward \( \tilde{r}(s,a,s') = r(s,a,s') + \kappa \) where \( \kappa \) is a constant. Is this transformation potential-based? Does it preserve optimal policies?

Solution: We seek a potential \( \Phi \) such that \( \kappa = \gamma \Phi(s') - \Phi(s) \) for all \( s,s' \). No bounded \( \Phi \) can satisfy this for all transitions in a general MDP, so \( \tilde{r} \) is not potential-based in the sense of Section 2. However, adding a constant to all rewards simply shifts \( J(\pi) \) by a constant and preserves optimal policies (because all policies are shifted equally). So \( \tilde{r} \) is behavior-preserving but not expressible as potential-based shaping with bounded \( \Phi \).

In this lesson we formalized reward design for robotic reinforcement learning, emphasizing the role of the reward function as the interface between task specification and learning dynamics. We introduced potential-based reward shaping and proved its policy-invariance property, allowing us to add dense shaping signals without changing the optimal policy. We analyzed why sparse, terminal rewards lead to high-variance gradients and extremely slow learning, and we surveyed practical tricks such as distance-based shaping, hindsight experience replay, and intrinsic exploration bonuses. Finally, we demonstrated how to implement these ideas across multiple programming environments commonly used in robotics, preparing the ground for safety-constrained RL in the next lesson.