A robotic experiment typically consists of an algorithm (controller or planner), an environment (simulation or physical setup), and a protocol determining how trials are executed. Formally, let \( \mathcal{C} \) denote the space of configurations (code version, parameters, robot model, environment description), \( \Omega \) a probability space of randomness (sensor noise, initial conditions, random seeds), and \( f: \mathcal{C} \times \Omega \rightarrow \mathcal{Y} \) a measurable map that assigns outcomes \( y \in \mathcal{Y} \) (e.g., task success, trajectory cost).

\[ Y = f(C,\Omega), \quad C \in \mathcal{C},\; \Omega \sim P_{\Omega}. \]

For a fixed configuration \( C=c \), the distribution of outcomes is \( P_{Y|c} \). A basic performance quantity is the expected outcome \( \mu(c) \):

\[ \mu(c) = \mathbb{E}[Y \mid C=c] = \int_{\Omega} f(c,\omega)\, \mathrm{d}P_{\Omega}(\omega). \]

We distinguish:

• Bitwise reproducibility: repeated execution with the same configuration \( c \) and random seed \( \omega \) yields identical outcome \( y = f(c,\omega) \) up to floating point round-off at the machine level.
• Statistical reproducibility: independent repetitions under configuration \( c \) and randomness \( \Omega_1,\dots,\Omega_m \) produce empirical distributions whose discrepancy from \( P_{Y|c} \) is small in some metric (e.g. total variation, Wasserstein distance) with high probability.

A simple measure of statistical reproducibility compares the expectations of two implementations, \( c_1 \) and \( c_2 \):

\[ \Delta_{\mu}(c_1,c_2) = \big|\mu(c_1) - \mu(c_2)\big| = \big|\mathbb{E}[Y \mid C=c_1] - \mathbb{E}[Y \mid C=c_2]\big|. \]

If \( \Delta_{\mu}(c_1,c_2) \) is small relative to the statistical uncertainty of estimators (confidence intervals), the two configurations are considered statistically reproducible.

Consider a binary success metric for a robotic task: each episode either succeeds (\( Y=1 \)) or fails (\( Y=0 \)). Under a fixed configuration \( c \), assume \( Y_1,\dots,Y_n \) are i.i.d. Bernoulli with success probability \( p(c) \). The empirical success rate is

\[ \hat{p}_n(c) = \frac{1}{n}\sum_{i=1}^n Y_i. \]

Proposition 1 (Unbiasedness). \( \hat{p}_n(c) \) is an unbiased estimator of \( p(c) \): \( \mathbb{E}[\hat{p}_n(c)] = p(c) \).

Proof. Using linearity of expectation,

\[ \mathbb{E}[\hat{p}_n(c)] = \mathbb{E}\!\left[\frac{1}{n}\sum_{i=1}^n Y_i\right] = \frac{1}{n}\sum_{i=1}^n \mathbb{E}[Y_i] = \frac{1}{n} \cdot n \, p(c) = p(c). \quad \square \]

Proposition 2 (Variance). The variance of \( \hat{p}_n(c) \) is \( \operatorname{Var}(\hat{p}_n(c)) = \frac{p(c)(1-p(c))}{n} \).

Proof. Since the \( Y_i \) are i.i.d.,

\[ \operatorname{Var}(\hat{p}_n(c)) = \operatorname{Var}\!\left(\frac{1}{n}\sum_{i=1}^n Y_i\right) = \frac{1}{n^2} \sum_{i=1}^n \operatorname{Var}(Y_i) = \frac{1}{n^2} \cdot n \, p(c)(1-p(c)) = \frac{p(c)(1-p(c))}{n}. \]

Thus, to halve the standard deviation of the estimated success rate, one must quadruple the number of episodes \( n \). This quantifies how many trials are needed for reproducible statistics.

For a real-valued trajectory cost \( J_i \) (e.g., integrated torque or tracking error) with mean \( \mu_J(c) \) and variance \( \sigma_J^2(c) \), the sample mean \( \bar{J}_n = \frac{1}{n}\sum_{i=1}^n J_i \) satisfies

\[ \mathbb{E}[\bar{J}_n] = \mu_J(c), \quad \operatorname{Var}(\bar{J}_n) = \frac{\sigma_J^2(c)}{n}. \]

By the central limit theorem, for large \( n \), \( \bar{J}_n \) is approximately normal, enabling confidence intervals and hypothesis tests for reproducibility across independent implementations or laboratories.

In experimental robotics, variability arises at multiple levels: different laboratories, different runs on the same setup, and trial-to-trial noise. For a continuous metric \( Y_{ijk} \) (e.g., grasp success score), indexed by algorithm \( i \), lab \( j \), and run \( k \), a simple random-effects model is

\[ Y_{ijk} = \mu + \alpha_i + L_j + R_{k(j)} + \varepsilon_{ijk}, \]

where:

• \( \mu \): global mean performance,
• \( \alpha_i \): fixed effect of algorithm \( i \),
• \( L_j \sim \mathcal{N}(0,\sigma_L^2) \): random effect of lab \( j \) (hardware, calibration),
• \( R_{k(j)} \sim \mathcal{N}(0,\sigma_R^2) \): run-to-run variation within lab \( j \),
• \( \varepsilon_{ijk} \sim \mathcal{N}(0,\sigma^2) \): trial noise (contact randomness, sensor noise).

Under independence assumptions, the marginal variance decomposes as

\[ \operatorname{Var}(Y_{ijk}) = \sigma_L^2 + \sigma_R^2 + \sigma^2. \]

A useful quantity is the intra-class correlation (ICC) at lab level, measuring how strongly outcomes cluster within labs:

\[ \text{ICC}_{\text{lab}} = \frac{\sigma_L^2}{\sigma_L^2 + \sigma_R^2 + \sigma^2}. \]

High \( \text{ICC}_{\text{lab}} \) indicates that lab-specific factors significantly affect results; reproducible evaluation then requires either normalizing across labs or including multiple labs in the benchmark.

To make robotics experiments reproducible, we must treat the entire pipeline as an object that can be versioned, executed, and inspected. This includes:

• Code (planner/controller, simulation, logging, analysis),
• Configuration (hyperparameters, robot model, environment assets),
• Random seeds for all stochastic components,
• Data and logs (trajectories, sensor streams, metrics),
• Execution environment (OS, libraries, hardware description).

The following flow summarizes the typical lifecycle:

flowchart TD
  H["Define hypothesis and metrics"] --> C["Freeze configuration (code, params, robot model)"]
  C --> E["Build environment (sim or hardware protocol)"]
  E --> S["Choose global seeds for all PRNGs"]
  S --> R["Run experiment harness (multiple trials)"]
  R --> L["Log raw data and metadata"]
  L --> A["Analyze metrics and confidence intervals"]
  A --> P["Publish artifact (code, config, logs)"]
        

A reproducible artifact is one for which a third party can reconstruct the configuration \( c \), regenerate the same randomness \( \Omega \) (for bitwise reproducibility) or i.i.d. draws from the same distribution (for statistical reproducibility), and verify that summary statistics match within expected sampling error.

Most robotic simulators implement dynamics deterministically given initial conditions, control inputs, and pseudo-random numbers. Let \( x_t \) denote the system state, \( u_t \) the control, and \( w_t \) a disturbance generated by a pseudo-random number generator (PRNG). A typical discrete-time model is

\[ x_{t+1} = f(x_t, u_t, w_t), \quad t \ge 0. \]

The PRNG maintains an internal state \( s_t \), updated deterministically:

\[ s_{t+1} = h(s_t), \quad w_t = g(s_t), \]

where \( h \) and \( g \) are deterministic functions (e.g., linear congruential transformation followed by scaling to \( [0,1] \)).

Assume a deterministic policy \( u_t = \pi_t(x_t) \), possibly with internal controller state that is also deterministic given its initialization. Let \( z_t = (x_t, s_t) \) be the joint state of physical dynamics and PRNG. The evolution is

\[ z_{t+1} = F(z_t) = \big(f(x_t,\pi_t(x_t),g(s_t)),\, h(s_t)\big). \]

Proposition 3. Given initial joint state \( z_0 = (x_0, s_0) \), the trajectory \( (z_t)_{t\ge 0} \) is uniquely determined.

Proof. This follows by induction on \( t \):

1. For \( t=0 \), \( z_0 \) is given.
2. Suppose \( z_t \) is uniquely determined. Then \( z_{t+1} = F(z_t) \) is uniquely determined because \( F \) is a deterministic function.

By induction, all \( z_t \) are uniquely determined. The projected state trajectory \( (x_t)_{t\ge 0} \) is therefore uniquely determined. \( \square \)

This implies that in simulators, bitwise reproducibility is attainable if we fix:

• initial robot and environment state \( x_0 \),
• controller initialization,
• PRNG seed \( s_0 \),
• time step and numerical integration scheme.

Any divergence between repeated runs with the same seeds reveals hidden non-determinism (parallelism without fixed scheduling, uninitialized memory, or external timing dependencies), which must be eliminated for strict reproducibility.

We construct a minimal Python experiment harness for a manipulator in simulation. The harness ensures that all randomness is controlled by a global seed and that configurations are recorded. We assume a Gym-like API with a PyBullet-based environment (e.g., KukaBulletEnv), but the pattern applies to any simulator.

import os
import json
import random
from dataclasses import dataclass, asdict
from datetime import datetime

import numpy as np

# Optional robotics libraries
import gymnasium as gym          # modern Gym API
import pybullet_envs  # noqa: F401  # registers PyBullet envs with Gym

try:
    import torch
except ImportError:
    torch = None


@dataclass
class ExperimentConfig:
    env_id: str = "KukaBulletEnv-v0"
    policy_name: str = "scripted_push"
    num_episodes: int = 50
    max_steps: int = 200
    global_seed: int = 1234


def set_global_seeds(seed: int) -> None:
    """
    Set all relevant seeds to make simulation as deterministic as possible.
    """
    random.seed(seed)
    np.random.seed(seed)
    if torch is not None:
        torch.manual_seed(seed)
        if torch.cuda.is_available():
            torch.cuda.manual_seed_all(seed)


def scripted_policy(observation) -> np.ndarray:
    """
    Placeholder for a deterministic policy using the observation.
    In practice, this might be a fixed MPC controller or a frozen neural network.
    """
    # Example: zero action for all joints
    # Replace with your controller logic.
    return np.zeros(7, dtype=np.float32)


def run_single_episode(env, max_steps: int) -> dict:
    obs, info = env.reset()
    total_reward = 0.0
    success = False

    for _ in range(max_steps):
        action = scripted_policy(obs)
        obs, reward, terminated, truncated, info = env.step(action)
        total_reward += float(reward)
        if info.get("is_success", False):
            success = True
        if terminated or truncated:
            break

    return {
        "total_reward": total_reward,
        "success": success,
    }


def run_experiment(cfg: ExperimentConfig, output_dir: str) -> None:
    os.makedirs(output_dir, exist_ok=True)

    # Save configuration for reproducibility
    with open(os.path.join(output_dir, "config.json"), "w") as f:
        json.dump(asdict(cfg), f, indent=2)

    set_global_seeds(cfg.global_seed)

    env = gym.make(cfg.env_id)
    # Some environments accept seed in reset
    env.reset(seed=cfg.global_seed)

    results = []
    for episode in range(cfg.num_episodes):
        # To get independent but reproducible episodes, you can offset the seed
        env.reset(seed=cfg.global_seed + episode)
        ep_res = run_single_episode(env, cfg.max_steps)
        ep_res["episode"] = episode
        results.append(ep_res)

    env.close()

    # Save raw metrics
    with open(os.path.join(output_dir, "results.json"), "w") as f:
        json.dump(results, f, indent=2)

    # Compute summary statistics
    rewards = np.array([r["total_reward"] for r in results], dtype=np.float64)
    successes = np.array([r["success"] for r in results], dtype=np.float64)

    summary = {
        "mean_reward": float(np.mean(rewards)),
        "std_reward": float(np.std(rewards, ddof=1)),
        "success_rate": float(np.mean(successes)),
    }
    with open(os.path.join(output_dir, "summary.json"), "w") as f:
        json.dump(summary, f, indent=2)


if __name__ == "__main__":
    cfg = ExperimentConfig()
    timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
    out_dir = os.path.join("logs", f"{cfg.policy_name}_{timestamp}")
    run_experiment(cfg, out_dir)
      

This script:

• Centralizes experiment configuration in a dataclass.
• Sets seeds consistently for Python, NumPy, and optionally PyTorch.
• Saves configuration and raw per-episode results.
• Computes summary statistics that can be compared across runs or labs.

For physical or high-fidelity simulated robots, ROS 2 is a common middleware. Below is a sketch of a ROS 2 node that:

• Accepts a seed for a std::mt19937 engine,
• Executes scripted trajectories perturbed by reproducible noise,
• Logs episode-level metrics to a CSV file.

#include <chrono>
#include <fstream>
#include <memory>
#include <random>
#include <string>
#include <vector>

#include "rclcpp/rclcpp.hpp"
#include "std_msgs/msg/float64.hpp"

using std::placeholders::_1;
using rclcpp::NodeOptions;

class ReproExperimentNode : public rclcpp::Node {
public:
  explicit ReproExperimentNode(const NodeOptions & options = NodeOptions())
      : Node("repro_experiment_node", options),
        rng_seed_(declare_parameter<int>("global_seed", 1234)),
        num_episodes_(declare_parameter<int>("num_episodes", 50)),
        max_steps_(declare_parameter<int>("max_steps", 200)),
        rng_(static_cast<unsigned int>(rng_seed_)),
        noise_dist_(0.0, 0.01) {

    log_path_ = declare_parameter<std::string>("log_path", "repro_results.csv");
    log_file_.open(log_path_, std::ios::out | std::ios::trunc);
    log_file_ << "episode,total_cost\n";

    command_pub_ = create_publisher<std_msgs::msg::Float64>(
        "joint1/command", 10);

    timer_ = create_wall_timer(
        std::chrono::milliseconds(10),
        std::bind(&ReproExperimentNode::control_loop, this));
  }

private:
  void control_loop() {
    if (episode_ >= num_episodes_) {
      RCLCPP_INFO(get_logger(), "All episodes finished, shutting down.");
      rclcpp::shutdown();
      return;
    }
    if (step_ == 0) {
      RCLCPP_INFO(get_logger(), "Starting episode %d", episode_);
      episode_cost_ = 0.0;
    }

    double nominal_cmd = scripted_command(step_);
    double noisy_cmd = nominal_cmd + noise_dist_(rng_);

    std_msgs::msg::Float64 msg;
    msg.data = noisy_cmd;
    command_pub_->publish(msg);

    episode_cost_ += std::abs(noisy_cmd);

    ++step_;
    if (step_ >= max_steps_) {
      log_file_ << episode_ << "," << episode_cost_ << "\n";
      ++episode_;
      step_ = 0;
    }
  }

  double scripted_command(int step) const {
    // Placeholder: simple sinusoidal profile
    return 0.5 * std::sin(0.01 * static_cast<double>(step));
  }

  int rng_seed_;
  int num_episodes_;
  int max_steps_;

  int episode_ = 0;
  int step_ = 0;
  double episode_cost_ = 0.0;

  std::mt19937 rng_;
  std::normal_distribution<double> noise_dist_;

  std::string log_path_;
  std::ofstream log_file_;

  rclcpp::Publisher<std_msgs::msg::Float64>::SharedPtr command_pub_;
  rclcpp::TimerBase::SharedPtr timer_;
};

int main(int argc, char ** argv) {
  rclcpp::init(argc, argv);
  auto node = std::make_shared<ReproExperimentNode>();
  rclcpp::spin(node);
  rclcpp::shutdown();
  return 0;
}
      

Compiling and running this node with the same global_seed, number of episodes, and max_steps guarantees that the commanded sequence and costs are reproducible, assuming deterministic hardware drivers and scheduling.

Java is less common in industrial manipulators but appears in mobile and educational robotics (e.g., WPILib for FRC or ROSJava). The pattern for reproducibility is similar: centralize configuration and random seeds, and ensure all sources of randomness use the same seeded PRNG.

import java.io.FileWriter;
import java.io.IOException;
import java.util.Properties;
import java.util.Random;

public class ReproducibleEvaluator {

    private final RobotEnvironment env;
    private final Random rng;
    private final int numEpisodes;
    private final int maxSteps;

    public ReproducibleEvaluator(RobotEnvironment env,
                                 long globalSeed,
                                 int numEpisodes,
                                 int maxSteps) {
        this.env = env;
        this.rng = new Random(globalSeed);
        this.numEpisodes = numEpisodes;
        this.maxSteps = maxSteps;
    }

    public void run(String logPath) throws IOException {
        try (FileWriter writer = new FileWriter(logPath)) {
            writer.write("episode,totalCost,success\n");
            for (int episode = 0; episode < numEpisodes; episode++) {
                EpisodeResult res = runEpisode(episode);
                writer.write(episode + "," + res.totalCost + "," + res.success + "\n");
            }
        }
    }

    private EpisodeResult runEpisode(int episodeIndex) {
        env.reset(rng, episodeIndex); // deterministic given rng state and index
        double totalCost = 0.0;
        boolean success = false;

        for (int t = 0; t < maxSteps; t++) {
            double[] obs = env.getObservation();
            double[] action = scriptedPolicy(obs);
            // Inject small, reproducible perturbation
            for (int i = 0; i < action.length; i++) {
                action[i] += 0.01 * rng.nextGaussian();
            }
            StepResult step = env.step(action);
            totalCost += step.cost;
            if (step.success) {
                success = true;
            }
            if (step.terminated) {
                break;
            }
        }
        return new EpisodeResult(totalCost, success);
    }

    private double[] scriptedPolicy(double[] obs) {
        // Deterministic policy; fill with your control logic
        double[] action = new double[env.getActionDim()];
        for (int i = 0; i < action.length; i++) {
            action[i] = 0.0;
        }
        return action;
    }

    private static class EpisodeResult {
        final double totalCost;
        final boolean success;
        EpisodeResult(double totalCost, boolean success) {
            this.totalCost = totalCost;
            this.success = success;
        }
    }

    public static void main(String[] args) throws IOException {
        // RobotEnvironment would be implemented using ROSJava, WPILib, or a custom simulator.
        RobotEnvironment env = new MyRobotEnvironment();
        ReproducibleEvaluator evaluator =
                new ReproducibleEvaluator(env, 1234L, 50, 200);
        evaluator.run("java_repro_results.csv");
    }
}
      

Here, RobotEnvironment is an abstraction over the robot or simulator. Its implementation must avoid hidden randomness (e.g., system-level random calls) outside the provided Random rng to maintain reproducibility.

Matlab and Simulink are widely used in robotics for modeling and control. The Robotics System Toolbox and Simscape Multibody provide manipulator models and sensor blocks. Reproducible simulations require:

• Setting the global RNG seed,
• Configuring noise blocks to use the global stream,
• Logging configuration and results to MAT files.

function run_repro_sim()
    % Configuration
    cfg.modelName   = 'two_link_manipulator_model';
    cfg.stopTime    = 5.0;
    cfg.globalSeed  = 1234;
    cfg.numRuns     = 30;
    cfg.logFile     = 'matlab_repro_results.mat';

    % Set global RNG for reproducibility
    rng(cfg.globalSeed, 'twister');

    % Configure model parameters (e.g., friction, payload)
    set_param(cfg.modelName, 'StopTime', num2str(cfg.stopTime));

    % Preallocate logs
    episodeCost = zeros(cfg.numRuns, 1);
    successFlag = false(cfg.numRuns, 1);

    for k = 1:cfg.numRuns
        % Optionally vary initial condition deterministically
        baseSeed = cfg.globalSeed + k - 1;
        rng(baseSeed, 'twister');

        % Set workspace variables used by the model
        assignin('base', 'initSeed', baseSeed);

        simOut = sim(cfg.modelName, ...
            'SimulationMode', 'normal', ...
            'SrcWorkspace', 'current', ...
            'SaveOutput', 'on', ...
            'SaveState', 'off');

        % Extract metrics (assuming logsout or out structure)
        logs = simOut.logsout;
        costSig = logs.get('cost');
        successSig = logs.get('success');

        episodeCost(k) = costSig.Values.Data(end);
        successFlag(k) = logical(successSig.Values.Data(end));
    end

    summary.meanCost    = mean(episodeCost);
    summary.stdCost     = std(episodeCost, 1);
    summary.successRate = mean(successFlag);

    save(cfg.logFile, 'cfg', 'episodeCost', 'successFlag', 'summary');
end
      

In Simulink, noise blocks (e.g., random number or noise sources) should be configured to use the global random stream or an explicitly seeded stream based on initSeed, so that each run is deterministic given the configuration.

Wolfram Mathematica can be used to analyze dynamical models of manipulators and to perform Monte Carlo robustness studies. Reproducibility relies on SeedRandom and explicit storage of parameters and results.

(* Simple 1-DOF joint with viscous friction and torque control *)
ClearAll["Global`*"];

params = <|
   "m" -> 1.0,
   "l" -> 1.0,
   "b" -> 0.1
|>;

torquePolicy[t_, q_, qdot_] := -5.0 q - 1.0 qdot;

simulateOnce[seed_Integer] := Module[
  {m, l, b, q, qdot, tmax, sol, noise},
  {m, l, b} = ({"m", "l", "b"} /. params);

  SeedRandom[seed];

  noise[t_] := RandomVariate[NormalDistribution[0, 0.01]];

  tmax = 5.0;
  sol = NDSolve[
    {
      q''[t] == (torquePolicy[t, q[t], q'[t]] - b q'[t] + noise[t])/(m l^2),
      q[0] == 0.5,
      q'[0] == 0.0
    },
    {q},
    {t, 0, tmax}
  ][[1]];

  (* Cost: integral of squared position error *)
  NIntegrate[Evaluate[q[t]^2 /. sol], {t, 0, tmax}]
];

runs = Table[simulateOnce[1234 + k], {k, 0, 19}];
meanCost = Mean[runs];
stdCost  = StandardDeviation[runs];

Export["mma_repro_results.mx", <|"params" -> params,
   "runs" -> runs, "meanCost" -> meanCost, "stdCost" -> stdCost|>];
      

Re-running this notebook with the same seeds and parameters yields identical numeric trajectories (up to local numerical tolerances), allowing it to serve as an independent verification of results obtained in other languages or toolchains.

Problem 1 (Variance under Multi-Run Averaging): A lab evaluates a grasping policy with success probability \( p \). In each of \( m \) independent runs, it executes \( n \) episodes and records the per-run empirical success rate \( \hat{p}^{(r)} = \frac{1}{n}\sum_{i=1}^n Y_{ri} \), where \( Y_{ri} \) are i.i.d. Bernoulli(\( p \)). The overall estimate is the average across runs, \( \bar{p} = \frac{1}{m}\sum_{r=1}^m \hat{p}^{(r)} \). Show that \( \operatorname{Var}(\bar{p}) = \frac{p(1-p)}{mn} \).

Solution.

\[ \operatorname{Var}(\bar{p}) = \operatorname{Var}\!\left(\frac{1}{m}\sum_{r=1}^m \hat{p}^{(r)}\right) = \frac{1}{m^2} \sum_{r=1}^m \operatorname{Var}(\hat{p}^{(r)}), \]

because runs are independent. From Section 2, \( \operatorname{Var}(\hat{p}^{(r)}) = \frac{p(1-p)}{n} \). Thus

\[ \operatorname{Var}(\bar{p}) = \frac{1}{m^2} \cdot m \cdot \frac{p(1-p)}{n} = \frac{p(1-p)}{mn}. \]

Hence, increasing either the number of runs \( m \) or episodes per run \( n \) reduces variance; for a fixed computational budget, one can trade off \( m \) vs \( n \).

Problem 2 (Lab-Level ICC): In the hierarchical model of Section 3, assume \( \alpha_i = 0 \) for a single fixed algorithm. Suppose estimated variances are \( \hat{\sigma}_L^2 = 0.04 \), \( \hat{\sigma}_R^2 = 0.01 \), \( \hat{\sigma}^2 = 0.05 \). Compute the lab-level intra-class correlation \( \widehat{\text{ICC}}_{\text{lab}} \) and interpret it.

Solution.

\[ \widehat{\text{ICC}}_{\text{lab}} = \frac{\hat{\sigma}_L^2}{\hat{\sigma}_L^2 + \hat{\sigma}_R^2 + \hat{\sigma}^2} = \frac{0.04}{0.04 + 0.01 + 0.05} = \frac{0.04}{0.10} = 0.4. \]

Thus, 40% of the observed variance in the metric is attributable to between-lab differences. This indicates that reproducing the experiment in multiple labs is crucial; single-lab results are not representative of a hypothetical population of labs.

Problem 3 (Hypothesis Test for Reproducibility): Two research groups implement the same motion planner and report empirical success rates \( \hat{p}_1 = 0.84 \) based on \( n_1 = 400 \) episodes and \( \hat{p}_2 = 0.78 \) based on \( n_2 = 300 \) episodes. Under the null hypothesis that both implementations have the same success probability, derive the large-sample \( z \)-test statistic and state the rejection rule at significance level \( \alpha \).

Solution.

The pooled success rate under the null is

\[ \hat{p}_{\text{pool}} = \frac{n_1 \hat{p}_1 + n_2 \hat{p}_2}{n_1 + n_2}. \]

The standard error of the difference is

\[ \text{SE} = \sqrt{ \hat{p}_{\text{pool}}(1-\hat{p}_{\text{pool}}) \left( \frac{1}{n_1} + \frac{1}{n_2} \right) }. \]

The test statistic is

\[ z = \frac{\hat{p}_1 - \hat{p}_2}{\text{SE}}. \]

For a two-sided test at level \( \alpha \), we reject the null of reproducible success probability if \( |z| > z_{1-\alpha/2} \), where \( z_{1-\alpha/2} \) is the corresponding normal quantile. A significant difference indicates that at least one implementation or protocol differs enough to affect performance.

Problem 4 (Designing a Reproducible Pipeline): Sketch a process for ensuring that a new benchmark for mobile robot navigation is reproducible across laboratories, including code, configuration, and data.

Solution (one possible process):

flowchart TD
  S["Specify tasks and metrics"] --> R["Release canonical maps and robot models"]
  R --> V["Version control repository for code and configs"]
  V --> D["Define standard experiment script with seed arguments"]
  D --> G["Provide example runs and expected summaries"]
  G --> L["Require labs to publish logs and config hashes"]
  L --> C["Cross-check results across labs"]
        

The key components are: (i) public, versioned assets; (ii) a standardized script that accepts seeds and outputs logs in a fixed format; (iii) expected summary statistics to validate new installations; and (iv) cross-lab comparisons to detect systematic deviations.

We formalized reproducible experimental robotics in probabilistic terms, distinguishing bitwise from statistical reproducibility and deriving how variance shrinks with the number of runs and episodes. A hierarchical model decomposed total variability into lab, run, and trial components, leading to the intra-class correlation as a measure of sensitivity to lab conditions. We then mapped these ideas to concrete implementations in Python (Gym/PyBullet), C++ (ROS 2), Java (abstract robot environments), Matlab/Simulink (Robotics System Toolbox), and Wolfram Mathematica, emphasizing the consistent use of random seeds, configuration freezing, and logging. Finally, we formulated hypothesis tests and design patterns for multi-lab benchmarks that ensure robust and trustworthy evaluation of advanced robotic systems.