In a capstone project, an autonomous system (planner, controller, or policy) is specified by a mapping \( \pi \) from states to actions. Its performance is not a single number but a random variable induced by task variability, model uncertainty, and sensing noise. Let \( \mathcal{D} \) denote a distribution over task instances and disturbances, and let \( J(\pi,\omega) \in \mathbb{R} \) be a scalar cost (or negative reward) obtained when the system is run under realization \( \omega \sim \mathcal{D} \). The quantity of interest is the expected performance:

\[ \mu(\pi) := \mathbb{E}_{\omega \sim \mathcal{D}}[J(\pi,\omega)]. \]

In practice, we only observe finitely many runs \( \omega_1,\dots,\omega_N \), yielding samples \( J_1,\dots,J_N \) with \( J_i = J(\pi,\omega_i) \). Experimental validation is about:

• Designing how to draw \( \omega_i \) (task distributions, seeds).
• Choosing metrics derived from \( J_i \) (success, constraint violations, energy).
• Building estimators and tests for claims such as “method A outperforms method B”.

For a performance metric \( X \) (e.g., success indicator, path length), we model \( X_1,\dots,X_N \) as i.i.d. samples of a random variable \( X \). The empirical mean

\[ \hat{\mu}_N = \frac{1}{N}\sum_{i=1}^N X_i \]

is an unbiased estimator of \( \mu = \mathbb{E}[X] \). The central limit theorem (CLT), under mild conditions, implies asymptotic normality:

\[ \sqrt{N}\,(\hat{\mu}_N - \mu) \xrightarrow{d} \mathcal{N}(0,\sigma^2), \quad \sigma^2 := \operatorname{Var}(X). \]

This asymptotic normality underpins confidence intervals and hypothesis tests for your capstone experiments.

flowchart TD
  H["Define hypothesis: e.g., new planner better than baseline"] --> M["Choose metric: success rate, cost, time"]
  M --> D["Specify task distribution: scenes, initial states, disturbances"]
  D --> E["Execute trials in simulation and on hardware"]
  E --> L["Log outcomes and trajectories"]
  L --> S["Compute estimators and confidence intervals"]
  S --> T["Perform hypothesis tests"]
  T --> C["Conclude: accept or reject claimed improvement"]
        

Two fundamental types of metrics appear in robotics experiments: binary outcomes (success/failure) and real-valued costs (path length, energy, time-to-completion). We model them as follows.

2.1 Binary success metrics

Let \( S_i \in \{0,1\} \) denote success on trial \( i \). Assume \( S_i \overset{iid}{\sim} \mathrm{Bernoulli}(p_s) \), where \( p_s \) is the unknown success probability over the task distribution. The natural estimator is

\[ \hat{p}_s = \frac{1}{N}\sum_{i=1}^N S_i. \]

Then \( \mathbb{E}[\hat{p}_s] = p_s \) and \( \operatorname{Var}(\hat{p}_s) = \frac{p_s(1-p_s)}{N} \). Replacing \( p_s \) with \( \hat{p}_s \) yields the estimated standard deviation:

\[ \widehat{\operatorname{sd}}(\hat{p}_s) = \sqrt{\frac{\hat{p}_s(1-\hat{p}_s)}{N}}. \]

2.2 Real-valued cost metrics

Let \( C_i \in \mathbb{R} \) denote a cost (e.g., path length) for trial \( i \), with \( C_i \) i.i.d. draws of a random variable \( C \). The sample mean and variance are

\[ \bar{C}_N = \frac{1}{N}\sum_{i=1}^N C_i, \qquad s_C^2 = \frac{1}{N-1}\sum_{i=1}^N (C_i - \bar{C}_N)^2. \]

For sufficiently large \( N \), CLT gives \( \bar{C}_N \approx \mathcal{N}(\mu_C, s_C^2/N) \), where \( \mu_C = \mathbb{E}[C] \).

3.1 Confidence interval for success probability

A symmetric \( (1-\alpha) \) confidence interval for the success probability \( p_s \) based on the normal approximation is

\[ \hat{p}_s \pm z_{1-\alpha/2} \sqrt{\frac{\hat{p}_s(1-\hat{p}_s)}{N}}, \]

where \( z_{1-\alpha/2} \) is the corresponding Gaussian quantile (e.g., approximately \( 1.96 \) for \( 95\% \) confidence).

3.2 Confidence interval for mean cost

For a real-valued cost metric, if \( N \) is moderate and we are willing to assume approximate normality of \( C \), we use the \( t \)-distribution:

\[ \bar{C}_N \pm t_{1-\alpha/2,\,N-1}\,\frac{s_C}{\sqrt{N}}, \]

where \( t_{1-\alpha/2,\,N-1} \) is the \( (1-\alpha/2) \) quantile of the Student \( t \)-distribution with \( N-1 \) degrees of freedom. These intervals should accompany any reported success rates or costs in your capstone.

Suppose you compare a new planner \( \pi_A \) against a baseline \( \pi_B \). Let \( p_A \) and \( p_B \) denote their success probabilities. A typical null hypothesis is

\[ H_0: p_A = p_B \quad \text{vs.} \quad H_1: p_A > p_B. \]

4.1 Two-proportion z-test for success rates

Let \( \hat{p}_A \) and \( \hat{p}_B \) be empirical success rates over \( N_A \) and \( N_B \) trials:

\[ \hat{p}_A = \frac{1}{N_A}\sum_{i=1}^{N_A} S_i^{(A)}, \qquad \hat{p}_B = \frac{1}{N_B}\sum_{i=1}^{N_B} S_i^{(B)}. \]

Under \( H_0 \), the pooled estimator is

\[ \hat{p} = \frac{N_A \hat{p}_A + N_B \hat{p}_B}{N_A + N_B}, \]

and the test statistic is

\[ Z = \frac{\hat{p}_A - \hat{p}_B} {\sqrt{\hat{p}(1-\hat{p})\left(\frac{1}{N_A} + \frac{1}{N_B}\right)}}. \]

For large \( N_A,N_B \), \( Z \) is approximately standard normal under \( H_0 \). The one-sided \( p \)-value for \( H_1: p_A > p_B \) is \( 1 - \Phi(Z) \).

4.2 Difference-of-means test for costs

Let \( \bar{C}_A, s_A^2 \) and \( \bar{C}_B, s_B^2 \) denote sample means and variances for a cost metric under methods A and B. The Welch-type statistic is

\[ T = \frac{\bar{C}_A - \bar{C}_B} {\sqrt{\frac{s_A^2}{N_A} + \frac{s_B^2}{N_B}}}. \]

When cost distributions are not too skewed, \( T \) is close to a \( t \)-distribution with an effective number of degrees of freedom (the Welch–Satterthwaite approximation). This is the appropriate test for statements such as “our trajectory optimizer yields significantly shorter paths than CHOMP or RRT*”.

Beyond point estimates and tests, it is useful to know how many trials are needed to estimate success probabilities or costs to a desired accuracy. For bounded metrics, concentration inequalities provide non-asymptotic guarantees.

5.1 Hoeffding inequality for success rate

For \( S_i \in \{0,1\} \) with success probability \( p_s \), Hoeffding's inequality states

\[ \mathbb{P}\left( \bigl|\hat{p}_s - p_s\bigr| \ge \varepsilon \right) \le 2 \exp(-2N\varepsilon^2). \]

To ensure that \( \bigl|\hat{p}_s - p_s\bigr| \le \varepsilon \) with probability at least \( 1-\delta \), it suffices to choose \( N \) such that

\[ 2 \exp(-2N\varepsilon^2) \le \delta \quad \Longrightarrow \quad N \ge \frac{1}{2\varepsilon^2}\,\log\frac{2}{\delta}. \]

This provides a principled way to decide, for example, that \( N \approx 500 \) randomized scenes are needed to estimate a success probability within \( \varepsilon = 0.05 \) with \( 95\% \) confidence.

Earlier in the course, sim-to-real transfer and benchmarking were introduced. In a capstone setting, you typically:

1. Prototype and stress-test algorithms in simulation (e.g., OMPL, MoveIt, PyBullet).
2. Deploy to hardware with carefully controlled task distributions.
3. Aggregate data across both domains to estimate and compare performance.

A simplified pipeline is illustrated below.

flowchart TD
  S["Simulation experiments: N_sim trials"] --> F1["Filter logs, compute metrics"]
  H["Hardware experiments: N_real trials"] --> F1
  F1 --> A1["Aggregate metrics: success, cost, constraint violations"]
  A1 --> CI["Compute confidence intervals and test statistics"]
  CI --> DEC["Decide: is new method reliably better and safe?"]
        

We now implement the above estimators in Python. Assume a CSV log file with columns: method (e.g., "baseline", "new"), success (0 or 1), and cost (path length or time). The logs are produced, for example, by a MoveIt-based evaluation node or a PyBullet simulation loop.

import numpy as np
import pandas as pd
from scipy import stats

# Load experimental log
df = pd.read_csv("planner_logs.csv")  # columns: method, success, cost

def summarize_method(df_method, alpha=0.05):
    """Return success rate CI and cost CI for one method."""
    successes = df_method["success"].to_numpy()
    costs = df_method["cost"].to_numpy()

    N = len(successes)
    p_hat = successes.mean()
    # Normal approximation CI for Bernoulli
    z = stats.norm.ppf(1 - alpha / 2.0)
    se_p = np.sqrt(p_hat * (1.0 - p_hat) / N)
    ci_p = (p_hat - z * se_p, p_hat + z * se_p)

    # t-based CI for mean cost
    N_cost = len(costs)
    c_bar = costs.mean()
    s_c = costs.std(ddof=1)
    t_val = stats.t.ppf(1 - alpha / 2.0, df=N_cost - 1)
    se_c = s_c / np.sqrt(N_cost)
    ci_c = (c_bar - t_val * se_c, c_bar + t_val * se_c)

    return {
        "N": N,
        "p_hat": p_hat,
        "ci_p": ci_p,
        "c_bar": c_bar,
        "ci_c": ci_c,
    }

# Separate methods
methods = df["method"].unique()
summary = {m: summarize_method(df[df["method"] == m]) for m in methods}

for m, stats_m in summary.items():
    print(f"Method {m}:")
    print(f"  N trials       = {stats_m['N']}")
    print(f"  success rate   = {stats_m['p_hat']:.3f}, "
          f"CI = ({stats_m['ci_p'][0]:.3f}, {stats_m['ci_p'][1]:.3f})")
    print(f"  mean cost      = {stats_m['c_bar']:.3f}, "
          f"CI = ({stats_m['ci_c'][0]:.3f}, {stats_m['ci_c'][1]:.3f})")

# Two-proportion z-test for success rate difference (new vs baseline)
baseline = "baseline"
new = "new"

df_A = df[df["method"] == new]
df_B = df[df["method"] == baseline]

S_A = df_A["success"].to_numpy()
S_B = df_B["success"].to_numpy()
N_A, N_B = len(S_A), len(S_B)
p_A, p_B = S_A.mean(), S_B.mean()

p_pool = (N_A * p_A + N_B * p_B) / (N_A + N_B)
se_diff = np.sqrt(p_pool * (1.0 - p_pool) * (1.0 / N_A + 1.0 / N_B))
Z = (p_A - p_B) / se_diff
p_value_one_sided = 1.0 - stats.norm.cdf(Z)

print("\nTwo-proportion z-test (success rate):")
print(f"  p_A = {p_A:.3f}, p_B = {p_B:.3f}")
print(f"  Z   = {Z:.3f}, one-sided p-value = {p_value_one_sided:.4f}")

# t-test for mean cost (lower is better)
cost_A = df_A["cost"].to_numpy()
cost_B = df_B["cost"].to_numpy()

t_stat, p_val_cost = stats.ttest_ind(cost_B, cost_A, equal_var=False)
print("\nWelch t-test on cost (H1: new method has lower cost):")
print(f"  t = {t_stat:.3f}, two-sided p-value = {p_val_cost:.4f}")
      

This script assumes that the robotic stack (e.g., MoveIt, OMPL, PyBullet) has already produced structured logs. The statistical analysis is decoupled from the control stack, which is a good engineering practice.

In C++, log analysis can be integrated into an OMPL- or MoveIt-based evaluation node. The snippet below assumes a simple text log with one trial per line: method success cost.

#include <iostream>
#include <fstream>
#include <string>
#include <unordered_map>
#include <vector>
#include <cmath>

struct Stats {
    int N = 0;
    int successes = 0;
    double sumCost = 0.0;
    double sumCostSq = 0.0;
};

int main() {
    std::ifstream in("planner_logs.txt");
    if (!in) {
        std::cerr << "Could not open log file\n";
        return 1;
    }

    std::unordered_map<std::string, Stats> statsByMethod;
    std::string method;
    int success;
    double cost;

    while (in >> method >> success >> cost) {
        Stats &st = statsByMethod[method];
        st.N += 1;
        st.successes += success;
        st.sumCost += cost;
        st.sumCostSq += cost * cost;
    }

    double alpha = 0.05;
    double z = 1.96; // approx for 95% CI

    for (const auto &kv : statsByMethod) {
        const std::string &m = kv.first;
        const Stats &st = kv.second;
        double p_hat = static_cast<double>(st.successes) / st.N;
        double se_p = std::sqrt(p_hat * (1.0 - p_hat) / st.N);
        double ci_p_low = p_hat - z * se_p;
        double ci_p_high = p_hat + z * se_p;

        double meanCost = st.sumCost / st.N;
        double varCost = (st.sumCostSq - st.sumCost * st.sumCost / st.N)
                         / (st.N - 1);
        double se_c = std::sqrt(varCost / st.N);
        double ci_c_low = meanCost - z * se_c;
        double ci_c_high = meanCost + z * se_c;

        std::cout << "Method " << m << ":\n";
        std::cout << "  N           = " << st.N << "\n";
        std::cout << "  success     = " << p_hat
                  << "  CI = [" << ci_p_low << ", " << ci_p_high << "]\n";
        std::cout << "  mean cost   = " << meanCost
                  << "  CI = [" << ci_c_low << ", " << ci_c_high << "]\n";
    }

    return 0;
}
      

In a full capstone project, this computation could run inside a ROS node that subscribes to trial results and periodically prints updated statistics during large-scale experiments.

Java is often used in robotics middleware (e.g., ROSJava). The following utility class computes success rate confidence intervals given arrays of booleans per method.

import java.util.List;
import java.util.ArrayList;

public class CIUtils {

    public static class Summary {
        public int N;
        public double pHat;
        public double ciLow;
        public double ciHigh;
    }

    public static Summary summarizeSuccess(List<Boolean> successes, double alpha) {
        int N = successes.size();
        int count = 0;
        for (boolean s : successes) {
            if (s) {
                count++;
            }
        }
        double pHat = (double) count / (double) N;
        double z = 1.96; // for ~95% CI
        double se = Math.sqrt(pHat * (1.0 - pHat) / (double) N);

        Summary s = new Summary();
        s.N = N;
        s.pHat = pHat;
        s.ciLow = pHat - z * se;
        s.ciHigh = pHat + z * se;
        return s;
    }

    public static void main(String[] args) {
        // Example: successes for a planner in 10 trials
        List<Boolean> successes = new ArrayList<>();
        boolean[] arr = {true, true, false, true, true, true, false, true, true, true};
        for (boolean b : arr) {
            successes.add(b);
        }

        Summary s = summarizeSuccess(successes, 0.05);
        System.out.println("N      = " + s.N);
        System.out.println("p_hat  = " + s.pHat);
        System.out.println("CI     = [" + s.ciLow + ", " + s.ciHigh + "]");
    }
}
      

This logic can be embedded in a ROSJava-based monitoring node that listens to experiment topics and prints performance intervals online.

MATLAB and Simulink are widely used for model-based design. The following MATLAB script assumes that a Simulink model performs one trial per call to sim, and logs success and cost signals for each run.

% Monte Carlo evaluation of a controller or planner
N = 200;                % number of trials
success = false(N, 1);
cost = zeros(N, 1);

for k = 1:N
    % Set random seed or task parameters here, e.g.:
    % set_param('myModel', 'SimulationCommand', 'update');
    % assignin('base', 'initial_state', rand(3,1));

    simOut = sim('myAutonomyModel', 'ReturnWorkspaceOutputs', 'on');
    success(k) = simOut.success(end) > 0.5;
    cost(k) = simOut.cost(end);
end

p_hat = mean(success);
N_s = N;
alpha = 0.05;
z = 1.96;
se_p = sqrt(p_hat * (1 - p_hat) / N_s);
ci_p = [p_hat - z * se_p, p_hat + z * se_p];

c_bar = mean(cost);
s_c = std(cost, 1);
ci_c = [c_bar - z * s_c / sqrt(N), c_bar + z * s_c / sqrt(N)];

fprintf('Success rate: %.3f, CI = [%.3f, %.3f]\n', p_hat, ci_p(1), ci_p(2));
fprintf('Mean cost   : %.3f, CI = [%.3f, %.3f]\n', c_bar, ci_c(1), ci_c(2));
      

In Simulink, this corresponds to a Monte Carlo harness that repeatedly simulates the closed-loop autonomous system under randomized initial conditions and disturbances.

Wolfram Mathematica offers high-level primitives for statistical tests. The code below performs a proportion test and a difference-of-means test for two planners.

(* Binary success outcomes for two planners *)
successA = {1, 1, 0, 1, 1, 1, 1, 0, 1, 1};
successB = {1, 0, 0, 1, 0, 1, 0, 0, 1, 0};

nA = Length[successA];
nB = Length[successB];
pHatA = N[Mean[successA]];
pHatB = N[Mean[successB]];

Print["p_A = ", pHatA, "  p_B = ", pHatB];

(* One-sided test H0: p_A == p_B vs H1: p_A > p_B *)
pValueProp =
  ProportionTest[{successA, successB},
    "TestType" -> "Right", "Method" -> "Asymptotic"
  ];

Print["Proportion test p-value (H1: p_A > p_B) = ", pValueProp];

(* Continuous-cost comparison *)
costA = {10.2, 9.8, 10.0, 9.9, 9.7};
costB = {11.5, 11.0, 12.1, 11.7, 11.3};

pValueMean =
  LocationTest[{costA, costB},
    "TestType" -> "Left", "Method" -> "WelchTTest"
  ];

Print["Mean cost test p-value (H1: meanA < meanB) = ", pValueMean];
      

These commands internally implement the same statistics derived earlier in this lesson, providing an additional tool for validating your capstone results.

Problem 1 (Success Probability Interval): A motion planner is evaluated on \( N = 200 \) random scenes. It succeeds on \( 170 \) of them. Compute an approximate \( 95\% \) confidence interval for the success probability \( p_s \).

Solution: We have \( \hat{p}_s = 170/200 = 0.85 \). The standard error is

\[ \widehat{\operatorname{se}}(\hat{p}_s) = \sqrt{\frac{\hat{p}_s(1-\hat{p}_s)}{N}} = \sqrt{\frac{0.85 \cdot 0.15}{200}} \approx \sqrt{\frac{0.1275}{200}} \approx \sqrt{0.0006375} \approx 0.02525. \]

Using \( z_{0.975} \approx 1.96 \), the confidence interval is

\[ 0.85 \pm 1.96 \cdot 0.02525 \approx 0.85 \pm 0.0495 = [0.8005, 0.8995]. \]

Thus the planner's success probability is approximately between \( 0.80 \) and \( 0.90 \) with \( 95\% \) confidence.

Problem 2 (Sample Complexity via Hoeffding): You want to estimate a grasp success probability \( p_s \) with absolute error at most \( \varepsilon = 0.03 \) and confidence level \( 1-\delta = 0.95 \). Using Hoeffding's inequality, find a sufficient number of trials \( N \).

Solution: Hoeffding's inequality gives

\[ \mathbb{P}\bigl(\bigl|\hat{p}_s - p_s\bigr| \ge \varepsilon\bigr) \le 2 \exp(-2N\varepsilon^2). \]

We require \( 2 \exp(-2N\varepsilon^2) \le \delta \). With \( \delta = 0.05 \) and \( \varepsilon = 0.03 \),

\[ N \ge \frac{1}{2\varepsilon^2}\log\frac{2}{\delta} = \frac{1}{2 \cdot 0.03^2} \log\left(\frac{2}{0.05}\right) = \frac{1}{0.0018} \log(40) \approx 555.56 \cdot 3.6889 \approx 2051. \]

Hence \( N \approx 2051 \) trials are sufficient by this conservative bound.

Problem 3 (Two-Proportion Test for Planners): Planner A succeeds on \( 82 \) out of \( 100 \) scenes; planner B succeeds on \( 70 \) out of \( 100 \) scenes. Test \( H_0: p_A = p_B \) against \( H_1: p_A > p_B \) at significance level \( \alpha = 0.05 \) using the two-proportion z-test.

Solution: We have \( \hat{p}_A = 0.82 \), \( \hat{p}_B = 0.70 \), and \( N_A = N_B = 100 \). The pooled estimator is

\[ \hat{p} = \frac{N_A\hat{p}_A + N_B\hat{p}_B}{N_A + N_B} = \frac{100 \cdot 0.82 + 100 \cdot 0.70}{200} = \frac{82 + 70}{200} = 0.76. \]

The standard error for the difference is

\[ \operatorname{se} = \sqrt{\hat{p}(1-\hat{p})\left(\frac{1}{N_A} + \frac{1}{N_B}\right)} = \sqrt{0.76 \cdot 0.24 \cdot \left(\frac{1}{100} + \frac{1}{100}\right)} = \\ \sqrt{0.1824 \cdot 0.02} = \sqrt{0.003648} \approx 0.0604. \]

Thus

\[ Z = \frac{\hat{p}_A - \hat{p}_B}{\operatorname{se}} = \frac{0.82 - 0.70}{0.0604} \approx \frac{0.12}{0.0604} \approx 1.987. \]

The one-sided \( p \)-value is \( 1 - \Phi(1.987) \approx 0.0235 \), which is less than \( \alpha = 0.05 \). We therefore reject \( H_0 \) and conclude that planner A has significantly higher success probability than planner B at the 5% level.

Problem 4 (Cost Comparison with Welch's Test): An optimization-based trajectory planner and a sampling-based planner are evaluated on \( N_A = 20 \) and \( N_B = 20 \) tasks, respectively. The optimization-based planner has mean cost \( \bar{C}_A = 9.1 \) with sample standard deviation \( s_A = 0.8 \); the sampling-based planner has mean cost \( \bar{C}_B = 10.2 \) with sample standard deviation \( s_B = 1.0 \). Compute the Welch test statistic for \( H_0: \mu_A = \mu_B \) vs \( H_1: \mu_A < \mu_B \).

Solution: The test statistic is

\[ T = \frac{\bar{C}_A - \bar{C}_B} {\sqrt{\frac{s_A^2}{N_A} + \frac{s_B^2}{N_B}}} = \frac{9.1 - 10.2}{\sqrt{\frac{0.8^2}{20} + \frac{1.0^2}{20}}} = \frac{-1.1}{\sqrt{\frac{0.64}{20} + \frac{1.0}{20}}} = \frac{-1.1}{\sqrt{\frac{1.64}{20}}} = \frac{-1.1}{\sqrt{0.082}} \approx \frac{-1.1}{0.286} \approx -3.85. \]

A \( t \)-statistic of approximately \( -3.85 \) with around \( 30 \) degrees of freedom corresponds to a very small one-sided \( p \)-value, so we would reject \( H_0 \) and conclude that the optimization-based planner yields significantly lower cost.

In this lesson, we modeled robotic performance as random variables over task distributions and derived estimators, confidence intervals, and hypothesis tests for success probabilities and costs. Concentration inequalities such as Hoeffding's bound provided sample complexity guarantees for planning Monte Carlo experiments. We embedded these ideas into practical pipelines for sim-to-real validation and implemented multi-language analysis tools in Python, C++, Java, MATLAB/Simulink, and Wolfram Mathematica. These tools and concepts are essential for drawing defensible conclusions in your capstone project and for writing research-grade experimental sections.