Assume we already have a trained policy \( \pi \) (from Chapter 12 and Lesson 2 of this chapter), obtained by domain randomization in simulation. The environment is parameterized by a vector of physical and sensor parameters \( \phi \in \Phi \subset \mathbb{R}^d \) (e.g., link masses, friction coefficients, joint backlash, latency).

In this lab we:

• Define training and evaluation parameter distributions \( q_{\text{train}}(\phi) \) and \( q_{\text{eval}}(\phi) \), plus an unknown real-world distribution \( p^\ast(\phi) \).
• Introduce transfer metrics: expected return, success probability, and transfer gap between simulation and reality.
• Derive Monte Carlo estimators and sample-complexity guarantees for these metrics.
• Implement evaluation loops that:
  1. Sample randomized parameters.
  2. Run rollouts (simulation and real/high-fidelity).
  3. Aggregate statistics and confidence intervals.

Throughout, we treat the underlying robot dynamics and kinematics as known (from prior courses); the focus is on distributional aspects introduced by domain randomization.

Let each parameter vector \( \phi \) define an MDP \( \mathcal{M}(\phi) = (\mathcal{S}, \mathcal{A}, P_\phi, r_\phi, \gamma) \), where \( P_\phi \) and \( r_\phi \) encode the dynamics and rewards under those parameters. For a stationary policy \( \pi \), define the episodic return

\[ R(\tau; \phi, \pi) = \sum_{t=0}^{T-1} \gamma^t r_\phi(s_t, a_t), \quad a_t \sim \pi(\cdot \mid s_t),\; s_{t+1} \sim P_\phi(\cdot \mid s_t, a_t). \]

For a distribution \( d(\phi) \) over parameters (e.g., training or evaluation distribution), we define the distributional performance:

\[ J_d(\pi) \;=\; \mathbb{E}_{\phi \sim d}\Big[\,\mathbb{E}_{\tau \sim \mathcal{M}(\phi),\pi} \big[ R(\tau; \phi, \pi) \big] \Big]. \]

Domain randomization training typically maximizes \( J_{q_{\text{train}}}(\pi) \). Real-world performance corresponds to an unknown distribution \( p^\ast(\phi) \), possibly concentrated near a nominal parameter vector \( \phi^\ast \):

\[ J_{\text{real}}(\pi) := J_{p^\ast}(\pi), \qquad J_{\text{sim}}(\pi) := J_{q_{\text{eval}}}(\pi). \]

The transfer gap and robust performance are then:

\[ \Delta(\pi) \;=\; J_{\text{real}}(\pi) - J_{\text{sim}}(\pi), \qquad J_{\min}(\pi) \;=\; \inf_{\phi \in \Phi_{\text{real}}} J(\pi; \phi), \]

where \( \Phi_{\text{real}} \subseteq \Phi \) is a set of physically plausible parameters consistent with calibration (Lesson 3 of this chapter).

In practice, we cannot compute these expectations analytically, so we estimate them via Monte Carlo, using a finite number of rollouts in simulation and on the real robot (or a high-fidelity simulator).

Suppose we have i.i.d. rollouts \( R_1, \dots, R_N \) from distribution \( R_i \sim R(\tau; \phi_i, \pi) \) where \( \phi_i \sim d \). The natural estimator of \( J_d(\pi) \) is

\[ \widehat{J}_d(\pi) \;=\; \frac{1}{N} \sum_{i=1}^N R_i. \]

Under mild conditions (finite variance), this estimator is unbiased and its variance is

\[ \mathbb{E}[\widehat{J}_d(\pi)] = J_d(\pi), \qquad \operatorname{Var}(\widehat{J}_d(\pi)) = \frac{\sigma_R^2}{N}, \]

where \( \sigma_R^2 = \operatorname{Var}(R(\tau; \phi, \pi)) \). An unbiased estimator of the variance is the usual sample variance

\[ \widehat{\sigma}_R^2 = \frac{1}{N-1} \sum_{i=1}^N \big(R_i - \widehat{J}_d(\pi)\big)^2. \]

For large \( N \), the central limit theorem yields approximate \( (1-\alpha) \) confidence intervals:

\[ \widehat{J}_d(\pi) \;\pm\; z_{1-\alpha/2}\; \sqrt{\frac{\widehat{\sigma}_R^2}{N}}, \]

where \( z_{1-\alpha/2} \) is the standard normal quantile (e.g., \( z_{0.975} \approx 1.96 \) for a 95% interval).

Many robotics tasks are evaluated by success probability instead of return. Let \( Z_i \in \{0,1\} \) indicate success of rollout \( i \), and let \( p_d(\pi) := \mathbb{P}_d(Z=1) \). Then the estimator

\[ \widehat{p}_d(\pi) = \frac{1}{N} \sum_{i=1}^N Z_i \]

is unbiased with variance \( p_d(\pi)\big(1-p_d(\pi)\big)/N \). A distribution-free concentration bound is given by Hoeffding's inequality:

\[ \mathbb{P}\Big(\big|\widehat{p}_d(\pi) - p_d(\pi)\big| \ge \varepsilon\Big) \le 2 \exp(-2 N \varepsilon^2). \]

Solving the inequality for \( N \) yields a sufficient number of rollouts for tolerance \( \varepsilon \) and failure probability \( \delta \):

\[ N \;\ge\; \frac{1}{2 \varepsilon^2} \log\!\Big(\frac{2}{\delta}\Big). \]

This expression is crucial when budgeting expensive hardware experiments in this lab.

The following flowchart summarizes the evaluation protocol implemented in the code sections below. Note that for real hardware we usually replace \( q_{\text{eval}}(\phi) \) by a measured or calibrated distribution around the actual robot parameters.

flowchart TD
  A["Train policy pi with parameters phi ~ q_train"] --> B["Freeze pi and select q_eval and real robot"]
  B --> C["Sample parameter set {phi_i} from q_eval"]
  C --> D["For each phi_i: run K episodes in sim and on robot"]
  D --> E["Aggregate returns and success rates"]
  E --> F["Compute transfer gap and confidence intervals"]
  F --> G["Decide: adjust randomization, retrain, or accept policy"]
        

We now implement a Python evaluation loop. We assume:

• A simulation interface RandomizedArmEnv wrapping a physics engine (e.g., PyBullet, MuJoCo).
• A trained policy object policy with a method act(observation).
• A function real_robot_rollout(params) that performs one episode on the real robot given a parameter sample (e.g., target mass/friction estimates).

import numpy as np

class RandomizationConfig:
    def __init__(self, mass_range=(0.8, 1.2), friction_range=(0.5, 1.5),
                 latency_range=(0.0, 0.04)):
        self.mass_range = mass_range
        self.friction_range = friction_range
        self.latency_range = latency_range

    def sample(self, rng: np.random.Generator) -> dict:
        return {
            "mass_scale": rng.uniform(*self.mass_range),
            "friction_scale": rng.uniform(*self.friction_range),
            "latency": rng.uniform(*self.latency_range),
        }

def run_episode_sim(env, policy, params, max_steps=200):
    obs = env.reset(randomized_params=params)
    total_reward = 0.0
    success = False
    for t in range(max_steps):
        action = policy.act(obs)
        obs, reward, done, info = env.step(action)
        total_reward += reward
        if info.get("is_success", False):
            success = True
        if done:
            break
    return total_reward, success

def run_episode_real(policy, params, max_steps=200):
    # Thin wrapper around low-level real-robot interface.
    # Must enforce safety checks and emergency stop externally.
    result = real_robot_rollout(policy=policy,
                                params=params,
                                max_steps=max_steps)
    return result["return"], result["success"]

def evaluate_transfer(env, policy, rand_cfg_eval, n_eval=50, seed=0):
    rng = np.random.default_rng(seed)
    sim_returns, sim_success = [], []
    real_returns, real_success = [], []

    for i in range(n_eval):
        params = rand_cfg_eval.sample(rng)

        # Simulation episode
        R_sim, S_sim = run_episode_sim(env, policy, params)
        sim_returns.append(R_sim)
        sim_success.append(1 if S_sim else 0)

        # Real or high-fidelity episode
        R_real, S_real = run_episode_real(policy, params)
        real_returns.append(R_real)
        real_success.append(1 if S_real else 0)

    sim_returns = np.asarray(sim_returns, dtype=float)
    sim_success = np.asarray(sim_success, dtype=float)
    real_returns = np.asarray(real_returns, dtype=float)
    real_success = np.asarray(real_success, dtype=float)

    def summarize(vec):
        mean = vec.mean()
        std = vec.std(ddof=1)
        return mean, std

    J_sim, std_sim = summarize(sim_returns)
    J_real, std_real = summarize(real_returns)

    p_sim, std_p_sim = summarize(sim_success)
    p_real, std_p_real = summarize(real_success)

    transfer_gap_return = J_real - J_sim
    transfer_gap_success = p_real - p_sim

    return {
        "J_sim": J_sim,
        "J_real": J_real,
        "std_sim": std_sim,
        "std_real": std_real,
        "p_sim": p_sim,
        "p_real": p_real,
        "std_p_sim": std_p_sim,
        "std_p_real": std_p_real,
        "gap_return": transfer_gap_return,
        "gap_success": transfer_gap_success,
    }

# Example usage (env and policy assumed to be constructed elsewhere)
if __name__ == "__main__":
    rand_eval = RandomizationConfig(
        mass_range=(0.9, 1.1),
        friction_range=(0.8, 1.2),
        latency_range=(0.0, 0.03),
    )
    stats = evaluate_transfer(env, policy, rand_eval, n_eval=40, seed=42)
    print("Sim vs Real expected return:", stats["J_sim"], stats["J_real"])
    print("Sim vs Real success prob.:", stats["p_sim"], stats["p_real"])
    print("Transfer gaps:", stats["gap_return"], stats["gap_success"])
      

In a real experiment, you would log the sampled parameter vectors \( \phi_i \) together with returns and success flags, then perform additional statistical analysis (regression, sensitivity analysis) offline.

Next, we sketch a C++ evaluation loop that assumes a Bullet-based simulation wrapper RandomizedArmEnv and a policy class Policy. The structure mirrors the Python code, but here we emphasize type safety and deterministic seeding.

#include <random>
#include <vector>
#include <iostream>

struct Params {
    double mass_scale;
    double friction_scale;
    double latency;
};

struct EpisodeStats {
    double ret;
    bool success;
};

class RandomizationConfig {
public:
    RandomizationConfig(double m_min, double m_max,
                        double f_min, double f_max,
                        double l_min, double l_max)
        : mass_min_(m_min), mass_max_(m_max),
          fric_min_(f_min), fric_max_(f_max),
          lat_min_(l_min), lat_max_(l_max) {}

    Params sample(std::mt19937& gen) const {
        std::uniform_real_distribution<double> mass_dist(mass_min_, mass_max_);
        std::uniform_real_distribution<double> fric_dist(fric_min_, fric_max_);
        std::uniform_real_distribution<double> lat_dist(lat_min_, lat_max_);
        Params p;
        p.mass_scale = mass_dist(gen);
        p.friction_scale = fric_dist(gen);
        p.latency = lat_dist(gen);
        return p;
    }

private:
    double mass_min_, mass_max_;
    double fric_min_, fric_max_;
    double lat_min_, lat_max_;
};

EpisodeStats run_episode_sim(RandomizedArmEnv& env,
                             Policy& policy,
                             const Params& params,
                             int max_steps) {
    env.reset(params);
    double total_reward = 0.0;
    bool success = false;
    for (int t = 0; t < max_steps; ++t) {
        Eigen::VectorXd obs = env.observe();
        Eigen::VectorXd action = policy.act(obs);
        StepResult step = env.step(action);
        total_reward += step.reward;
        if (step.info.is_success) {
            success = true;
        }
        if (step.done) {
            break;
        }
    }
    return {total_reward, success};
}

// On real hardware, use an analogous function that calls low-level controllers.
EpisodeStats run_episode_real(Policy& policy,
                              const Params& params,
                              int max_steps);

int main() {
    RandomizedArmEnv env;
    Policy policy = load_trained_policy("policy.bin");

    RandomizationConfig eval_cfg(0.9, 1.1,   // mass_range
                                 0.8, 1.2,   // friction_range
                                 0.0, 0.03); // latency_range

    std::mt19937 gen(42);
    const int N = 40;
    const int max_steps = 200;

    std::vector<double> sim_returns, real_returns;
    std::vector<int> sim_success, real_success;

    sim_returns.reserve(N);
    real_returns.reserve(N);
    sim_success.reserve(N);
    real_success.reserve(N);

    for (int i = 0; i < N; ++i) {
        Params p = eval_cfg.sample(gen);
        EpisodeStats es_sim = run_episode_sim(env, policy, p, max_steps);
        EpisodeStats es_real = run_episode_real(policy, p, max_steps);

        sim_returns.push_back(es_sim.ret);
        real_returns.push_back(es_real.ret);
        sim_success.push_back(es_sim.success ? 1 : 0);
        real_success.push_back(es_real.success ? 1 : 0);
    }

    auto mean = [](const std::vector<double>& v) {
        double s = 0.0;
        for (double x : v) s += x;
        return s / static_cast<double>(v.size());
    };

    auto mean_int = [](const std::vector<int>& v) {
        double s = 0.0;
        for (int x : v) s += static_cast<double>(x);
        return s / static_cast<double>(v.size());
    };

    double J_sim = mean(sim_returns);
    double J_real = mean(real_returns);
    double p_sim = mean_int(sim_success);
    double p_real = mean_int(real_success);

    std::cout << "Expected return (sim, real): "
              << J_sim << ", " << J_real << std::endl;
    std::cout << "Success prob. (sim, real): "
              << p_sim << ", " << p_real << std::endl;
    std::cout << "Transfer gap (return): " << (J_real - J_sim) << std::endl;
    std::cout << "Transfer gap (success): " << (p_real - p_sim) << std::endl;

    return 0;
}
      

In a real project, the RandomizedArmEnv wraps Bullet or another physics engine, exposing methods reset, observe, and step that internally modify masses, friction coefficients, and controller latency according to Params.

Java is less common in robotics control loops, but it is useful for tooling (e.g., evaluation dashboards). The snippet below assumes a Java wrapper around a simulator (possibly jBullet or a custom JNI binding) with interfaces RandomizedEnv and Policy.

import java.util.Random;
import java.util.ArrayList;
import java.util.List;

public class TransferEvaluator {

    public static class Params {
        public double massScale;
        public double frictionScale;
        public double latency;
    }

    public static class EpisodeStats {
        public double ret;
        public boolean success;
        public EpisodeStats(double r, boolean s) {
            this.ret = r;
            this.success = s;
        }
    }

    public static class RandomizationConfig {
        private double mMin, mMax, fMin, fMax, lMin, lMax;
        private Random rng;

        public RandomizationConfig(double mMin, double mMax,
                                   double fMin, double fMax,
                                   double lMin, double lMax,
                                   long seed) {
            this.mMin = mMin;
            this.mMax = mMax;
            this.fMin = fMin;
            this.fMax = fMax;
            this.lMin = lMin;
            this.lMax = lMax;
            this.rng = new Random(seed);
        }

        public Params sample() {
            Params p = new Params();
            p.massScale = mMin + rng.nextDouble() * (mMax - mMin);
            p.frictionScale = fMin + rng.nextDouble() * (fMax - fMin);
            p.latency = lMin + rng.nextDouble() * (lMax - lMin);
            return p;
        }
    }

    public static EpisodeStats runEpisodeSim(RandomizedEnv env,
                                             Policy policy,
                                             Params params,
                                             int maxSteps) {
        env.reset(params);
        double totalReward = 0.0;
        boolean success = false;

        for (int t = 0; t < maxSteps; ++t) {
            double[] obs = env.observe();
            double[] action = policy.act(obs);
            StepResult step = env.step(action);
            totalReward += step.getReward();
            if (step.getInfo().isSuccess()) {
                success = true;
            }
            if (step.isDone()) {
                break;
            }
        }
        return new EpisodeStats(totalReward, success);
    }

    public static EpisodeStats runEpisodeReal(Policy policy,
                                              Params params,
                                              int maxSteps) {
        // Wrap hardware control; safety constraints must be handled externally.
        return RealRobot.runEpisode(policy, params, maxSteps);
    }

    public static void main(String[] args) {
        RandomizedEnv env = new BulletRandomizedEnv();
        Policy policy = PolicyLoader.load("policy.bin");

        RandomizationConfig evalCfg = new RandomizationConfig(
                0.9, 1.1,
                0.8, 1.2,
                0.0, 0.03,
                42L
        );

        final int N = 40;
        final int maxSteps = 200;

        List<Double> simReturns = new ArrayList<>();
        List<Double> realReturns = new ArrayList<>();
        List<Integer> simSuccess = new ArrayList<>();
        List<Integer> realSuccess = new ArrayList<>();

        for (int i = 0; i < N; ++i) {
            Params p = evalCfg.sample();
            EpisodeStats esSim = runEpisodeSim(env, policy, p, maxSteps);
            EpisodeStats esReal = runEpisodeReal(policy, p, maxSteps);

            simReturns.add(esSim.ret);
            realReturns.add(esReal.ret);
            simSuccess.add(esSim.success ? 1 : 0);
            realSuccess.add(esReal.success ? 1 : 0);
        }

        double JSim = mean(simReturns);
        double JReal = mean(realReturns);
        double pSim = meanInt(simSuccess);
        double pReal = meanInt(realSuccess);

        System.out.println("Expected return (sim, real): " + JSim + ", " + JReal);
        System.out.println("Success prob. (sim, real): " + pSim + ", " + pReal);
        System.out.println("Transfer gap (return): " + (JReal - JSim));
        System.out.println("Transfer gap (success): " + (pReal - pSim));
    }

    private static double mean(List<Double> v) {
        double s = 0.0;
        for (double x : v) s += x;
        return s / v.size();
    }

    private static double meanInt(List<Integer> v) {
        double s = 0.0;
        for (int x : v) s += x;
        return s / v.size();
    }
}
      

Java-side evaluation is especially useful when integrating with visualization dashboards or large-scale parameter sweeps (e.g., using a cluster scheduler), while Python or C++ handle low-level simulation and control.

MATLAB/Simulink is frequently used for high-fidelity modeling of manipulators. The script below shows how to:

• Sample randomized parameter sets.
• Inject them into a Simulink model (e.g., a 2-link arm).
• Run batch simulations and compute transfer metrics.

% Randomized transfer evaluation for a Simulink model "RandomizedArm"
% The model should read parameters: massScale, frictionScale, latency

rng(42);

N = 40;               % number of evaluation parameter sets
maxTime = 5.0;        % seconds per simulation
successThreshold = 0.95;  % fraction of time within target tolerance

massRange = [0.9, 1.1];
fricRange = [0.8, 1.2];
latRange  = [0.0, 0.03];

J_sim  = zeros(N,1);
J_real = zeros(N,1);
S_sim  = zeros(N,1);
S_real = zeros(N,1);

for i = 1:N
    massScale = massRange(1) + rand() * (massRange(2) - massRange(1));
    frictionScale = fricRange(1) + rand() * (fricRange(2) - fricRange(1));
    latency = latRange(1) + rand() * (latRange(2) - latRange(1));

    % Simulated environment
    assignin('base', 'massScale', massScale);
    assignin('base', 'frictionScale', frictionScale);
    assignin('base', 'latency', latency);

    simOut = sim('RandomizedArm', 'StopTime', num2str(maxTime), ...
        'SaveOutput', 'on', 'SaveState', 'off');

    rewardSignal = simOut.logsout.get('reward').Values.Data;
    successSignal = simOut.logsout.get('is_success').Values.Data;

    J_sim(i) = trapz(rewardSignal);
    S_sim(i) = max(successSignal);   % 1 if success at any time

    % Real robot experiment handler (returns same fields)
    realOut = runRealRobotExperiment(massScale, frictionScale, latency, maxTime);
    J_real(i) = trapz(realOut.reward);
    S_real(i) = max(realOut.is_success);
end

% Compute means and transfer gaps
J_sim_mean  = mean(J_sim);
J_real_mean = mean(J_real);
p_sim       = mean(S_sim);
p_real      = mean(S_real);

gap_return  = J_real_mean - J_sim_mean;
gap_success = p_real - p_sim;

fprintf('Expected return (sim, real): %.3f, %.3f\n', J_sim_mean, J_real_mean);
fprintf('Success prob. (sim, real): %.3f, %.3f\n', p_sim, p_real);
fprintf('Transfer gaps (return, success): %.3f, %.3f\n', gap_return, gap_success);
      

The function runRealRobotExperiment abstracts away low-level hardware control and should enforce safety interlocks (workspace limits, torque bounds, emergency stop). The same evaluation design (parameter sampling, logging, aggregation) is shared across simulation and hardware.

In Wolfram Mathematica we can treat the transfer evaluation as a stochastic experiment, derive sample-complexity curves, and symbolically manipulate confidence bounds. Below, we construct Hoeffding-style bounds and visualize how many rollouts are needed to obtain a given resolution on success probability.

(* Parameters *)
ClearAll["Global`*"];
ε = 0.05; (* desired absolute accuracy on success probability *)
δ = 0.05; (* failure probability *)

(* Hoeffding sample complexity: N >= (1 / (2 ε^2)) log(2 / δ) *)
NRequired[eps_, del_] := Ceiling[(1.0/(2.0 eps^2)) Log[2.0/del]];

NRequired[ε, δ]

(* Plot N as a function of ε for fixed δ *)
Plot[
  NRequired[eps, δ],
  {eps, 0.02, 0.2},
  AxesLabel -> {"eps", "N"},
  PlotLabel -> "Sample complexity for success-probability estimation"
]

(* Monte Carlo estimate of transfer gap from stored log data *)
simReturns = Import["sim_returns.csv"];
realReturns = Import["real_returns.csv"];

JSim  = Mean[simReturns];
JReal = Mean[realReturns];
gapReturn = JReal - JSim;

(* Asymptotic confidence interval for the gap using pooled variance *)
σ2Sim  = Variance[simReturns];
σ2Real = Variance[realReturns];

nSim  = Length[simReturns];
nReal = Length[realReturns];

σGap = Sqrt[σ2Sim/nSim + σ2Real/nReal];
z975 = InverseCDF[NormalDistribution[0, 1], 0.975];

confInt = {gapReturn - z975 σGap, gapReturn + z975 σGap};
confInt
      

Mathematica is particularly convenient for deriving symbolic expressions for bounds and quickly exploring how design choices (e.g., \( \varepsilon \), \( \delta \), ratios of simulation to real rollouts) affect the required number of hardware experiments.

Problem 1 (Unbiasedness and Variance of Monte Carlo Estimator): Let \( R_1, \dots, R_N \) be i.i.d. episodic returns under a fixed policy \( \pi \) and distribution \( d(\phi) \). Show that the estimator \( \widehat{J}_d(\pi) = \frac{1}{N} \sum_{i=1}^N R_i \) is unbiased and compute its variance.

Solution: Since each \( R_i \) has mean \( \mu = J_d(\pi) \),

\[ \mathbb{E}\big[\widehat{J}_d(\pi)\big] = \mathbb{E}\Big[\frac{1}{N} \sum_{i=1}^N R_i\Big] = \frac{1}{N} \sum_{i=1}^N \mathbb{E}[R_i] = \frac{1}{N} \cdot N \mu = \mu = J_d(\pi). \]

Thus the estimator is unbiased. For variance, using independence and the fact that \( \operatorname{Var}(R_i) = \sigma_R^2 \) for all \( i \),

\[ \operatorname{Var}\big(\widehat{J}_d(\pi)\big) = \operatorname{Var}\Big(\frac{1}{N} \sum_{i=1}^N R_i\Big) = \frac{1}{N^2} \sum_{i=1}^N \operatorname{Var}(R_i) = \frac{1}{N^2} \cdot N \sigma_R^2 = \frac{\sigma_R^2}{N}. \]

Problem 2 (Sample Complexity for Success Probability): Suppose we wish to estimate the real-world success probability \( p_{\text{real}}(\pi) \) of a policy, and we have Bernoulli outcomes \( Z_1,\dots,Z_N \). Using Hoeffding's inequality, derive a bound on the required \( N \) to guarantee \( \big|\widehat{p}_{\text{real}}(\pi) - p_{\text{real}}(\pi)\big| \le \varepsilon \) with probability at least \( 1-\delta \).

Solution: Hoeffding's inequality for Bernoulli variables states

\[ \mathbb{P}\Big(\big|\widehat{p}_{\text{real}}(\pi) - p_{\text{real}}(\pi)\big| \ge \varepsilon\Big) \le 2 \exp(-2 N \varepsilon^2). \]

We require the right-hand side to be at most \( \delta \). Hence

\[ 2 \exp(-2 N \varepsilon^2) \le \delta \quad\Longrightarrow\quad -2 N \varepsilon^2 \le \log(\delta/2) \quad\Longrightarrow\quad N \ge \frac{1}{2 \varepsilon^2} \log\!\Big(\frac{2}{\delta}\Big). \]

Any integer \( N \) satisfying this inequality suffices.

Problem 3 (Bounding the Transfer Gap via Distribution Shift): Assume for all \( \phi \in \Phi \) we have \( |J(\pi; \phi)| \le B \). Let \( p^\ast \) and \( q_{\text{eval}} \) be two distributions over \( \Phi \). Show that

\[ \big| J_{p^\ast}(\pi) - J_{q_{\text{eval}}}(\pi) \big| \le 2 B \cdot \operatorname{TV}(p^\ast, q_{\text{eval}}), \]

where \( \operatorname{TV} \) denotes the total variation distance.

Solution: By definition of \( J_d(\pi) \),

\[ J_{p^\ast}(\pi) - J_{q_{\text{eval}}}(\pi) = \int_{\Phi} J(\pi; \phi)\,\big(p^\ast(\phi) - q_{\text{eval}}(\phi)\big)\,d\phi. \]

Taking absolute values and using \( |J(\pi; \phi)| \le B \),

\[ \big|J_{p^\ast}(\pi) - J_{q_{\text{eval}}}(\pi)\big| \le \int_{\Phi} \big|J(\pi; \phi)\big|\,\big|p^\ast(\phi) - q_{\text{eval}}(\phi)\big|\,d\phi \le B \int_{\Phi} \big|p^\ast(\phi) - q_{\text{eval}}(\phi)\big|\,d\phi. \]

By the definition of total variation distance \( \operatorname{TV}(p^\ast, q_{\text{eval}}) = \tfrac{1}{2} \int_{\Phi} |p^\ast(\phi) - q_{\text{eval}}(\phi)|\,d\phi \), we obtain the stated bound.

Problem 4 (Designing Evaluation Parameters): Suppose the real robot's mass and friction uncertainties are believed to lie within intervals \( [m_{\min}, m_{\max}] \) and \( [\mu_{\min}, \mu_{\max}] \). Explain how to design a grid of parameter samples for evaluation and discuss the trade-off between resolution and number of required episodes.

Solution: A simple design is a tensor grid with \( n_m \) mass levels and \( n_\mu \) friction levels: \( m_j = m_{\min} + \frac{j-1}{n_m-1}(m_{\max} - m_{\min}) \), \( \mu_k = \mu_{\min} + \frac{k-1}{n_\mu-1}(\mu_{\max} - \mu_{\min}) \). This yields \( n_m n_\mu \) parameter pairs. Running \( K \) episodes per pair requires \( N = K n_m n_\mu \) episodes. Increasing \( n_m \) or \( n_\mu \) improves resolution of the performance surface \( J(\pi; m, \mu) \) but scales linearly in cost. Alternatives include random sampling or Latin hypercube sampling, which cover the space more uniformly with fewer points.

Problem 5 (Evaluation Pipeline Design): Sketch a high-level decision process for choosing between (i) evaluating only at a nominal parameter \( \phi^\ast \), (ii) evaluating on a coarse grid, or (iii) performing full randomized evaluation as in this lab.

Solution (flow):

flowchart TD
  S["Start"] --> U["Is param uncertainty small and well calibrated?"]
  U -->|"yes"| N["Evaluate at nominal phi* only"]
  U -->|"no"| R["Is evaluation budget very limited?"]
  R -->|"yes"| G["Use coarse grid with few points"]
  R -->|"no"| F["Use full randomized evaluation \nwith many phi samples"]
  G --> D["Refine grid or randomization \nif large transfer gap observed"]
  F --> D
        

In this lab we formalized the evaluation of domain-randomized policies via parameterized MDPs and distributions over environment parameters. We introduced transfer metrics \( J_{\text{sim}}(\pi) \), \( J_{\text{real}}(\pi) \), their gap, and success probabilities, then derived Monte Carlo estimators and confidence bounds (including Hoeffding-based sample-complexity guarantees).

We implemented multi-language evaluation pipelines (Python, C++, Java, MATLAB/Simulink, Mathematica) that share a common structure: sample parameters, run rollouts in both simulation and on hardware, log returns and success indicators, and aggregate statistics. These tools allow principled decisions about whether a randomized training scheme provides sufficient robustness, or whether the randomization distribution and policy training must be revisited before deploying an advanced robotic system.