In Chapters 11 and 12 we represented control tasks as Markov decision processes (MDPs) and learned policies \( \pi(a\mid s) \) that maximize expected cumulative reward in a given environment. In sim-to-real settings we have two environments:

\[ \mathcal{M}^{\text{sim}} = (\mathcal{S}, \mathcal{A}, P^{\text{sim}}, r^{\text{sim}}, \gamma),\quad \mathcal{M}^{\text{real}} = (\mathcal{S}, \mathcal{A}, P^{\text{real}}, r^{\text{real}}, \gamma) \]

A policy \( \pi \) is trained to be (approximately) optimal in \( \mathcal{M}^{\text{sim}} \), but is executed in \( \mathcal{M}^{\text{real}} \). The expected discounted returns are

\[ J^{\text{sim}}(\pi) = \mathbb{E}\Big[\sum_{t=0}^{\infty} \gamma^{t} r^{\text{sim}}(s_t,a_t)\Big],\quad J^{\text{real}}(\pi) = \mathbb{E}\Big[\sum_{t=0}^{\infty} \gamma^{t} r^{\text{real}}(s_t,a_t)\Big], \]

where trajectories are generated using \( P^{\text{sim}} \) or \( P^{\text{real}} \), respectively. The sim-to-real gap for policy \( \pi \) is

\[ \Delta(\pi) := J^{\text{real}}(\pi) - J^{\text{sim}}(\pi). \]

Policies that work perfectly in simulation but fail on the robot exhibit a large negative \( \Delta(\pi) \). Our goal is to understand the causes of large \( \Delta(\pi) \), and how to predict and reduce it.

flowchart TD
  A["Train policy in simulation"] --> B["Evaluate in simulation (high return)"]
  B --> C["Deploy on real robot"]
  C --> D{"Does behavior \nmatch simulation?"}
  D -->|no| E["Identify mismatch: dynamics, \nsensing, actuation, task spec"]
  D -->|yes| F["Successful transfer \n(small sim-to-real gap)"]
        

From a control perspective, the key point is that the controller was synthesized assuming a model (the simulator). If this model is inaccurate, closed-loop guarantees do not automatically carry over to the true plant.

Let \( x_t \in \mathbb{R}^n \) denote the robot state, and \( u_t \in \mathbb{R}^m \) the control input at time \( t \). Conceptually, the discrete-time dynamics in simulation and reality can be written as

\[ x_{t+1}^{\text{sim}} = f^{\text{sim}}(x_t^{\text{sim}}, u_t^{\text{sim}}),\qquad x_{t+1}^{\text{real}} = f^{\text{real}}(x_t^{\text{real}}, u_t^{\text{real}}). \]

For a policy described as a state-feedback law \( u_t = \pi(x_t) \), the closed-loop systems become

\[ x_{t+1}^{\text{sim}} = f^{\text{sim}}(x_t^{\text{sim}}, \pi(x_t^{\text{sim}})),\quad x_{t+1}^{\text{real}} = f^{\text{real}}(x_t^{\text{real}}, \pi(x_t^{\text{real}})). \]

Differences in the following components cause closed-loop trajectories to diverge:

• Dynamics modeling error: rigid-body parameters (masses, inertias, friction, compliance) are simplified in the simulator.
• Contact and friction models: Coulomb or viscous friction and rigid contact models poorly approximate real contact mechanics.
• Sensing mismatch: delay, bias, noise, and calibration errors in encoders, cameras, and force sensors.
• Actuation mismatch: motor saturation, backlash, dead zones, and unmodeled dynamics in drivers and gearboxes.
• Task specification mismatch: reward or cost functions in simulation that do not align with the real task (e.g., ignoring safety margins or actuator limits).

Formally, many of these are distribution shifts between the joint distributions over state–action trajectories in simulation and in reality.

We now quantify how differences in transition kernels and rewards affect \( \Delta(\pi) \). Let \( d_t^{\text{sim}} \) and \( d_t^{\text{real}} \) denote the state–action distributions at time \( t \) under the same policy \( \pi \) when interacting with \( \mathcal{M}^{\text{sim}} \) and \( \mathcal{M}^{\text{real}} \), respectively.

Assume that rewards are uniformly bounded: \( |r^{\text{sim}}(s,a)| \le R_{\max} \) and \( |r^{\text{real}}(s,a)| \le R_{\max} \). Define the reward mismatch

\[ \delta_r := \sup_{s,a} \big|r^{\text{real}}(s,a) - r^{\text{sim}}(s,a)\big|. \]

Let the total variation distance between transition kernels for any fixed state–action pair be

\[ \delta_P := \sup_{s,a} \left\|P^{\text{real}}(\cdot \mid s,a) - P^{\text{sim}}(\cdot \mid s,a)\right\|_{\text{TV}}. \]

A classical simulation lemma yields a bound of the form

\[ \big|J^{\text{real}}(\pi) - J^{\text{sim}}(\pi)\big| \le \frac{\delta_r}{1-\gamma} + \frac{2\gamma R_{\max}}{(1-\gamma)^2} \,\delta_P. \]

This shows that even if the reward is modeled perfectly (\( \delta_r=0 \)), small but systematic transition errors (\( \delta_P>0 \)) are amplified by the factor \( (1-\gamma)^{-2} \). Highly long-horizon tasks (large \( \gamma \)) are thus particularly sensitive to modeling errors.

Proof sketch. Express both returns as expectations over trajectories. First bound the difference in rewards directly by \( \delta_r \). Then use a coupling argument for Markov chains to bound the deviation in trajectory distributions due to transition differences by a geometric series in \( \gamma \delta_P \). The factor \( (1-\gamma)^{-2} \) arises from summing two nested geometric series.

For an \( n \)-DOF rigid manipulator, the standard joint-space model is

\[ \mathbf{M}(\mathbf{q}) \ddot{\mathbf{q}} + \mathbf{C}(\mathbf{q},\dot{\mathbf{q}})\dot{\mathbf{q}} + \mathbf{g}(\mathbf{q}) + \mathbf{\tau}_{\text{fric}}(\dot{\mathbf{q}}) = \mathbf{\tau}, \]

where \( \mathbf{q} \) is the joint configuration, \( \mathbf{\tau} \) is the actuator torque, and \( \mathbf{M}, \mathbf{C}, \mathbf{g}, \mathbf{\tau}_{\text{fric}} \) are the inertia, Coriolis/centrifugal, gravity, and friction terms, respectively.

In simulation we typically use nominal models \( \widehat{\mathbf{M}}, \widehat{\mathbf{C}}, \widehat{\mathbf{g}}, \widehat{\mathbf{\tau}}_{\text{fric}} \). Define the modeling error torque as

\[ \mathbf{\Delta}(\mathbf{q},\dot{\mathbf{q}},\ddot{\mathbf{q}}) := \big(\mathbf{M}-\widehat{\mathbf{M}}\big)\ddot{\mathbf{q}} + \big(\mathbf{C}-\widehat{\mathbf{C}}\big)\dot{\mathbf{q}} + \big(\mathbf{g}-\widehat{\mathbf{g}}\big) + \big(\mathbf{\tau}_{\text{fric}}-\widehat{\mathbf{\tau}}_{\text{fric}}\big). \]

If a policy \( \pi \) was trained assuming the nominal model, the torque command \( \mathbf{\tau} = \pi(\mathbf{q},\dot{\mathbf{q}}) \) will be systematically misapplied on the real robot by \( -\mathbf{\Delta} \). Under mild Lipschitz assumptions on the closed-loop dynamics, one can show that the state tracking error satisfies

\[ \|\mathbf{q}^{\text{real}}(t) - \mathbf{q}^{\text{sim}}(t)\| \le K \sup_{0\le \tau \le t} \|\mathbf{\Delta}(\tau)\|, \]

for some constant \( K \) depending on the closed-loop system matrices. Thus apparently small torque errors can accumulate into large pose deviations, especially in underdamped or contact-rich tasks.

Policies in Chapters 11–12 observe states (or estimated states) and output actions instantaneously. Real robots, however, experience sensing noise and delays. Let \( y_t \) denote the measurement, with

\[ y_t = h(x_{t-k}) + \eta_t, \]

where \( k \) is the sensor delay and \( \eta_t \) is noise. A policy trained in simulation using noiseless, delay-free observations \( x_t \) implicitly assumes \( k=0 \) and \( \eta_t=0 \). The deployed closed loop is then

\[ x_{t+1} = f\big(x_t, \pi(y_t)\big) = f\big(x_t, \pi\big(h(x_{t-k}) + \eta_t\big)\big), \]

which is qualitatively different from the simulated closed loop \( x_{t+1}^{\text{sim}} = f(x_t^{\text{sim}}, \pi(x_t^{\text{sim}})) \). If \( \pi \) is highly state-dependent and non-robust, small sensing errors \( \eta_t \) can move the system into regions of the state space that were rarely visited during training, causing unexpected failures.

Contact modeling introduces further discontinuities: rigid contacts are approximated by stiff springs and penalty forces, while real contacts involve compliance, stiction, and micro-slip. As a result, the true transition kernel \( P^{\text{real}} \) can differ from \( P^{\text{sim}} \) in a highly state-dependent way, especially at impact events, making \( \delta_P \) large exactly where precise control is needed.

Consider a 1-DOF rotary system (e.g., a single joint) with discrete-time dynamics

\[ \theta_{t+1} = \theta_t + \Delta t \,\omega_t,\qquad \omega_{t+1} = \omega_t + \Delta t \big(u_t - c \,\omega_t\big), \]

where \( c \) is a damping coefficient. Suppose a linear state-feedback policy \( u_t = -k_p \theta_t - k_d \omega_t \) is tuned in simulation using \( c = c_{\text{sim}} \), but the real system has \( c = c_{\text{real}} \). The closed-loop update for the velocity in simulation is

\[ \omega_{t+1}^{\text{sim}} = \omega_t^{\text{sim}} + \Delta t \big(-k_p \theta_t^{\text{sim}} - (k_d + c_{\text{sim}})\omega_t^{\text{sim}}\big), \]

while in reality

\[ \omega_{t+1}^{\text{real}} = \omega_t^{\text{real}} + \Delta t \big(-k_p \theta_t^{\text{real}} - (k_d + c_{\text{real}})\omega_t^{\text{real}}\big). \]

Writing the dynamics in matrix form,

\[ x_{t+1}^{\text{sim}} = A_{\text{sim}} x_t^{\text{sim}},\qquad x_{t+1}^{\text{real}} = A_{\text{real}} x_t^{\text{real}}, \]

the eigenvalues of \( A_{\text{sim}} \) and \( A_{\text{real}} \) determine whether each system is stable. If \( A_{\text{sim}} \) is stable but a different \( c_{\text{real}} \) pushes the eigenvalues of \( A_{\text{real}} \) outside the unit disk, then the policy will stabilize the simulated joint but destabilize the real one, even though the policy was obtained from a correct control design procedure applied to the wrong model.

We implement the 1-DOF example above and compare returns of a fixed policy under two models with different damping coefficients. This illustrates how a small model change causes a large performance drop.

import numpy as np

# Discrete-time 1-DOF dynamics parameters
dt = 0.02
c_sim = 0.1      # damping used during "training"
c_real = 0.3     # true damping on the robot (unknown to the designer)

# Linear feedback policy u = -K x, where x = [theta, omega]
Kp, Kd = 5.0, 1.0

def policy(x):
    theta, omega = x
    return -Kp * theta - Kd * omega

def step(x, u, c):
    theta, omega = x
    theta_next = theta + dt * omega
    omega_next = omega + dt * (u - c * omega)
    return np.array([theta_next, omega_next])

def rollout(c, horizon=500):
    x = np.array([0.5, 0.0])  # start with a position error
    total_reward = 0.0
    gamma = 0.99
    for t in range(horizon):
        u = policy(x)
        # quadratic cost; reward = -cost
        cost = x[0]**2 + 0.1 * x[1]**2 + 0.01 * u**2
        total_reward += (gamma**t) * (-cost)
        x = step(x, u, c)
    return total_reward

J_sim = rollout(c_sim)
J_real = rollout(c_real)

print("Discounted return in simulation:", J_sim)
print("Discounted return on 'real' model:", J_real)
print("Sim-to-real gap:", J_real - J_sim)
      

Increasing c_real or changing other parameters (e.g., adding actuator saturation) typically makes J_real much smaller than J_sim, even though the closed-loop policy is identical.

The following C++ snippet demonstrates the same 1-DOF dynamics and policy. In a larger project, the "sim" model could come from a physics engine (e.g., MuJoCo, Bullet), while the "real" model corresponds to parameters identified from experiments.

#include <iostream>
#include <cmath>

struct State {
    double theta;
    double omega;
};

double Kp = 5.0;
double Kd = 1.0;
double dt = 0.02;

double policy(const State& x) {
    return -Kp * x.theta - Kd * x.omega;
}

State step(const State& x, double u, double c) {
    State xn;
    xn.theta = x.theta + dt * x.omega;
    xn.omega = x.omega + dt * (u - c * x.omega);
    return xn;
}

double rollout(double c, int horizon = 500) {
    State x{0.5, 0.0};
    double gamma = 0.99;
    double total_reward = 0.0;
    double pow_gamma = 1.0;
    for (int t = 0; t < horizon; ++t) {
        double u = policy(x);
        double cost = x.theta * x.theta
                      + 0.1 * x.omega * x.omega
                      + 0.01 * u * u;
        total_reward += pow_gamma * (-cost);
        pow_gamma *= gamma;
        x = step(x, u, c);
    }
    return total_reward;
}

int main() {
    double J_sim  = rollout(0.1);
    double J_real = rollout(0.3);
    std::cout << "J_sim  = " << J_sim  << std::endl;
    std::cout << "J_real = " << J_real << std::endl;
    std::cout << "Gap   = " << (J_real - J_sim) << std::endl;
    return 0;
}
      

In a robotics software stack, this logic can be integrated with a middleware such as ROS, where the "real" model is replaced by actual sensor readings and actuator commands.

Similarly, we can implement the toy system in Java. This is useful when building control modules on top of JVM-based robotics frameworks or simulation environments.

public class SimToRealGap1DOF {

    static class State {
        double theta;
        double omega;
        State(double theta, double omega) {
            this.theta = theta;
            this.omega = omega;
        }
    }

    static double Kp = 5.0;
    static double Kd = 1.0;
    static double dt = 0.02;

    static double policy(State x) {
        return -Kp * x.theta - Kd * x.omega;
    }

    static State step(State x, double u, double c) {
        double thetaNext = x.theta + dt * x.omega;
        double omegaNext = x.omega + dt * (u - c * x.omega);
        return new State(thetaNext, omegaNext);
    }

    static double rollout(double c, int horizon) {
        State x = new State(0.5, 0.0);
        double gamma = 0.99;
        double totalReward = 0.0;
        double powGamma = 1.0;
        for (int t = 0; t < horizon; t++) {
            double u = policy(x);
            double cost = x.theta * x.theta
                          + 0.1 * x.omega * x.omega
                          + 0.01 * u * u;
            totalReward += powGamma * (-cost);
            powGamma *= gamma;
            x = step(x, u, c);
        }
        return totalReward;
    }

    public static void main(String[] args) {
        double Jsim = rollout(0.1, 500);
        double Jreal = rollout(0.3, 500);
        System.out.println("J_sim  = " + Jsim);
        System.out.println("J_real = " + Jreal);
        System.out.println("Gap    = " + (Jreal - Jsim));
    }
}
      

While the example is simple, the pattern of defining a step function and a rollout computation extends naturally to more complex robotic systems.

In MATLAB/Simulink, it is common to represent the plant as a Simulink model and synthesize a controller using Simulink blocks or MATLAB code. Here we sketch a simple experiment:

1. Build a Simulink model sim2real_1dof.slx with a continuous-time plant block implementing the rotary dynamics and a feedback controller.
2. Use a damping parameter c_sim in the block mask when tuning the controller.
3. After tuning, change the plant block parameter to c_real and re-run the simulation to observe closed-loop degradation.

% Parameters used for controller tuning
Kp = 5.0;
Kd = 1.0;
c_sim  = 0.1;   % nominal damping
c_real = 0.3;   % true damping to emulate the robot

% Open the Simulink model (assumes it exists on path)
open_system('sim2real_1dof');

% Configure controller gains (e.g., in a "Gain" block)
set_param('sim2real_1dof/Controller/Kp', 'Gain', num2str(Kp));
set_param('sim2real_1dof/Controller/Kd', 'Gain', num2str(Kd));

% First simulate with nominal damping
set_param('sim2real_1dof/Plant', 'c', num2str(c_sim));
simOutSim = sim('sim2real_1dof', 'StopTime', '5.0');
thetaSim = simOutSim.logsout.get('theta').Values.Data;

% Now simulate with "real" damping
set_param('sim2real_1dof/Plant', 'c', num2str(c_real));
simOutReal = sim('sim2real_1dof', 'StopTime', '5.0');
thetaReal = simOutReal.logsout.get('theta').Values.Data;

% Compare trajectories
t = simOutSim.tout;
figure;
plot(t, thetaSim, 'LineWidth', 1.5); hold on;
plot(t, thetaReal, '--', 'LineWidth', 1.5);
xlabel('Time (s)');
ylabel('\theta (rad)');
legend('Nominal model', 'Mismatched model');
title('Closed-loop response with and without damping mismatch');
grid on;
      

The difference between thetaSim and thetaReal visualizes the trajectory discrepancy induced purely by parameter mismatch.

Wolfram Mathematica is well suited for symbolic analysis of sim-to-real effects and numerical evaluation of policies under multiple models.

(* Discrete-time 1-DOF dynamics with damping c *)
dt = 0.02;
gamma = 0.99;
Kp = 5.0;
Kd = 1.0;

policy[{theta_, omega_}] := -Kp*theta - Kd*omega

step[{theta_, omega_}, u_, c_] := Module[
  {thetaNext, omegaNext},
  thetaNext = theta + dt*omega;
  omegaNext = omega + dt*(u - c*omega);
  {thetaNext, omegaNext}
];

rollout[c_, horizon_Integer] := Module[
  {x, t, u, cost, totalReward = 0.0, powGamma = 1.0},
  x = {0.5, 0.0};
  For[t = 0, t < horizon, t++,
    u = policy[x];
    cost = x[[1]]^2 + 0.1*x[[2]]^2 + 0.01*u^2;
    totalReward += powGamma*(-cost);
    powGamma *= gamma;
    x = step[x, u, c];
  ];
  totalReward
];

Jsim  = rollout[0.1, 500];
Jreal = rollout[0.3, 500];
simToRealGap = Jreal - Jsim

(* Optionally, visualize trajectories for both models *)
simulateTraj[c_, horizon_Integer] := 
  NestList[Function[x, step[x, policy[x], c]], {0.5, 0.0}, horizon];

trajSim  = simulateTraj[0.1, 300];
trajReal = simulateTraj[0.3, 300];

ListLinePlot[
 {trajSim[[All, 1]], trajReal[[All, 1]]},
 PlotLegends -> {"theta sim", "theta real"},
 AxesLabel -> {"time step", "theta"},
 PlotRange -> All
]
      

Mathematica can also be used to symbolically compute eigenvalues of the closed-loop matrices \( A_{\text{sim}} \) and \( A_{\text{real}} \) and to study stability regions as a function of control gains and plant parameters.

Problem 1 (Performance difference under reward mismatch only). Assume \( P^{\text{sim}} = P^{\text{real}} \) but \( r^{\text{sim}} \neq r^{\text{real}} \). Show that for any policy \( \pi \) we have

\[ \big|J^{\text{real}}(\pi) - J^{\text{sim}}(\pi)\big| \le \frac{\delta_r}{1-\gamma}, \quad \delta_r := \sup_{s,a} \big|r^{\text{real}}(s,a) - r^{\text{sim}}(s,a)\big|. \]

Solution. Because the transition kernels are identical, the state–action distributions \( d_t^{\text{sim}} \) and \( d_t^{\text{real}} \) coincide for all \( t \). Therefore,

\[ J^{\text{real}}(\pi) - J^{\text{sim}}(\pi) = \sum_{t=0}^{\infty} \gamma^{t} \mathbb{E}_{(s_t,a_t)\sim d_t}\big[r^{\text{real}}(s_t,a_t) - r^{\text{sim}}(s_t,a_t)\big]. \]

Taking absolute values and using the definition of \( \delta_r \) gives

\[ \big|J^{\text{real}}(\pi) - J^{\text{sim}}(\pi)\big| \le \sum_{t=0}^{\infty} \gamma^{t} \delta_r = \frac{\delta_r}{1-\gamma}. \]

This shows that errors in task specification (reward design) are linearly amplified by the effective horizon \( (1-\gamma)^{-1} \).

Problem 2 (Torque error due to inertia mismatch). Consider a single-joint robot with inertial parameter \( I \) and scalar dynamics \( I \ddot{\theta} = \tau \). The simulator uses \( I_{\text{sim}} \) and the real robot has \( I_{\text{real}} \). If the controller is designed as \( \tau = I_{\text{sim}} \alpha(t) \), where \( \alpha(t) \) is a desired acceleration profile, compute the acceleration tracking error on the real robot.

Solution. In simulation, the acceleration is \( \ddot{\theta}^{\text{sim}} = \tau / I_{\text{sim}} = \alpha(t) \) by construction. On the real robot,

\[ \ddot{\theta}^{\text{real}} = \frac{\tau}{I_{\text{real}}} = \frac{I_{\text{sim}}}{I_{\text{real}}}\alpha(t). \]

Therefore, the acceleration error is

\[ e_{\ddot{\theta}}(t) := \ddot{\theta}^{\text{real}}(t) - \ddot{\theta}^{\text{sim}}(t) = \left(\frac{I_{\text{sim}}}{I_{\text{real}}} - 1\right)\alpha(t). \]

If \( I_{\text{sim}} < I_{\text{real}} \), the true acceleration magnitude is smaller than expected, causing sluggish responses and tracking errors.

Problem 3 (Effect of observation noise on a deterministic policy). Let \( \pi: \mathcal{S} \to \mathcal{A} \) be a deterministic policy that stabilizes a deterministic system \( x_{t+1} = f(x_t,\pi(x_t)) \). Now consider a noisy observation \( y_t = x_t + \eta_t \), and a policy implementation \( \tilde{\pi}(y_t) := \pi(y_t) \). Argue qualitatively why the closed-loop system \( x_{t+1} = f(x_t,\tilde{\pi}(x_t+\eta_t)) \) can be unstable even if \( \pi \) stabilizes the nominal dynamics.

Solution. The closed-loop map with noise is \( x_{t+1} = f(x_t,\pi(x_t+\eta_t)) \). If \( \pi \) is highly sensitive to its input (large local Lipschitz constant), then small deviations in \( y_t \) produce large changes in \( \pi(y_t) \). As a consequence, the effective feedback law is random and can push the state outside the region of attraction of the nominal closed loop. In particular, if \( x_t \) approaches the equilibrium but \( \eta_t \) remains nonzero, then \( \pi(x_t+\eta_t) \) may not converge to the stabilizing action required at \( x_t \), causing oscillations or divergence. This is a typical failure mode for policies trained only with noise-free states.

Problem 4 (Qualitative decomposition of sim-to-real failure). Suppose a policy works well in simulation but fails on the robot. Propose a high-level diagnostic flow to separate three causes: dynamics mismatch, sensing mismatch, and reward mismatch.

Solution (flow).

flowchart TD
  S["Collect real robot rollouts with policy"] --> C["Compare real vs sim trajectories"]
  C --> D1{"Large state \ndeviation early?"}
  D1 -->|yes| DM["Dynamics or contact \nmismatch suspected"]
  D1 -->|no| D2{"Actions differ \ndespite similar states?"}
  D2 -->|yes| SM["Sensing mismatch \nor delays suspected"]
  D2 -->|no| RM["Reward or task specification \nmismatch suspected"]
  RM --> F["Redesign cost, constraints, \nand safety terms"]
        

By logging states, actions, and rewards in both simulation and reality, one can systematically identify at which stage the trajectories diverge, and whether the encoder measurements or task definition need to be revised.

Problem 5 (Bound with transition mismatch only). Assume \( r^{\text{real}} = r^{\text{sim}} \) and the transition mismatch parameter \( \delta_P \) defined in Section 3. Sketch how to obtain a bound of the form

\[ \big|J^{\text{real}}(\pi) - J^{\text{sim}}(\pi)\big| \le C \frac{R_{\max}}{(1-\gamma)^2} \delta_P, \]

for some universal constant \( C \).

Solution (sketch). Express both returns as \( \sum_{t} \gamma^{t} \mathbb{E}_{d_t}[r(s_t,a_t)] \) and note that the difference arises solely from the different state–action distributions \( d_t^{\text{real}} \) and \( d_t^{\text{sim}} \). Use a coupling argument for Markov chains: at each step, the probability that the two chains differ is bounded by \( \delta_P \), and once they diverge, the maximum reward difference at each subsequent step is \( 2 R_{\max} \). Summing the probabilities over time yields a geometric series, producing the factor \( (1-\gamma)^{-2} \).

In this lesson we modeled simulation and reality as distinct MDPs, and defined the sim-to-real gap \( \Delta(\pi) \) for a policy. We saw that differences in dynamics, sensing, actuation, and reward specification can be formalized as transition and reward mismatches, and that classical simulation lemmas provide bounds on the resulting performance loss. For robotic manipulators, even modest errors in inertial or friction parameters can drastically alter closed-loop stability and tracking quality. Through analytic bounds and multi-language toy examples, we built intuition for why controllers that are optimal in inaccurate simulators often fail on real hardware. Subsequent lessons will focus on methods such as domain randomization and residual learning to reduce this gap.