In a robotics paper, the evaluation section connects an algorithmic claim (e.g., “this planner finds shorter paths more reliably”) to empirical evidence. At a mathematical level, we model performance metrics as random variables induced by randomness in environments, initial conditions, sensor noise, and algorithmic seeds.

Let \( \mathcal{A} \) be a set of algorithms (planners, controllers, policies), \( \mathcal{W} \) a space of task instances or worlds (scenes, start-goal pairs, object configurations), and \( \Xi \) a space of random seeds or disturbances. An evaluation metric vector is a measurable map

\[ \mathbf{M} : \mathcal{A} \times \mathcal{W} \times \Xi \to \mathbb{R}^d, \quad (a,w,\xi) \mapsto \mathbf{M}(a,w,\xi), \]

where components might include:

• \( M_1 \): success indicator (0/1);
• \( M_2 \): path cost (length, energy, time);
• \( M_3 \): constraint violations, etc.

Assume a distribution \( P_W \) on tasks and \( P_\Xi \) on seeds. The true population performance of algorithm \( a \) is

\[ \boldsymbol{\mu}(a) = \mathbb{E}_{W \sim P_W,\,\Xi \sim P_\Xi} [\,\mathbf{M}(a,W,\Xi)\,] \in \mathbb{R}^d. \]

A research-grade evaluation section describes:

1. the distributions or sampling procedures that approximate \( P_W \) and \( P_\Xi \),
2. the estimators used for \( \boldsymbol{\mu}(a) \),
3. uncertainty quantification (confidence intervals, tests), and
4. comparisons between algorithms as statistical statements.

2.1 Success rates

Let \( S_{a,j} \in \{0,1\} \) be the success indicator of algorithm \( a \) on trial \( j=1,\dots,n \), with \( S_{a,j} \sim \text{Bernoulli}(p_a) \) i.i.d. The empirical success rate is

\[ \hat{p}_a = \frac{1}{n}\sum_{j=1}^n S_{a,j}. \]

Unbiasedness. Using linearity of expectation,

\[ \mathbb{E}[\hat{p}_a] = \frac{1}{n}\sum_{j=1}^n \mathbb{E}[S_{a,j}] = \frac{1}{n}\sum_{j=1}^n p_a = p_a. \]

Variance. Since the trials are independent,

\[ \operatorname{Var}(\hat{p}_a) = \frac{1}{n^2}\sum_{j=1}^n \operatorname{Var}(S_{a,j}) = \frac{1}{n^2}\sum_{j=1}^n p_a(1-p_a) = \frac{p_a(1-p_a)}{n}. \]

For large \( n \), the central limit theorem gives approximate normality:

\[ \hat{p}_a \approx \mathcal{N}\!\left( p_a,\, \frac{p_a(1-p_a)}{n} \right). \]

Replacing unknown \( p_a \) by \( \hat{p}_a \) yields an approximate \( (1-\alpha) \)-confidence interval

\[ \hat{p}_a \pm z_{1-\alpha/2} \sqrt{ \frac{\hat{p}_a(1-\hat{p}_a)}{n} }, \]

where \( z_{1-\alpha/2} \) is the \( (1-\alpha/2) \)-quantile of the standard normal distribution.

2.2 Continuous metrics: path cost and execution time

Let \( C_{a,j} \) be the path cost or execution time of algorithm \( a \) on trial \( j \), with mean \( \mu_a \) and variance \( \sigma_a^2 \). The empirical mean and variance are

\[ \hat{\mu}_a = \frac{1}{n}\sum_{j=1}^n C_{a,j}, \quad \hat{\sigma}_a^2 = \frac{1}{n-1}\sum_{j=1}^n (C_{a,j} - \hat{\mu}_a)^2. \]

As in classical statistics, one can show \( \mathbb{E}[\hat{\mu}_a] = \mu_a \) and \( \mathbb{E}[\hat{\sigma}_a^2] = \sigma_a^2 \). A research-grade evaluation should therefore report at least \( \hat{\mu}_a \) and a measure of dispersion (standard deviation or standard error).

Assuming approximate normality of the sample mean for moderate \( n \), an approximate \( (1-\alpha) \)-confidence interval is

\[ \hat{\mu}_a \pm t_{n-1,1-\alpha/2}\, \frac{\hat{\sigma}_a}{\sqrt{n}}, \]

where \( t_{n-1,1-\alpha/2} \) is the \( (1-\alpha/2) \)-quantile of the Student \( t \)-distribution with \( n-1 \) degrees of freedom.

3.1 Paired design for robotics benchmarks

In robotics benchmarks, different algorithms are often evaluated on the same set of tasks and seeds. This yields paired samples which reduce variance. Let \( C_{1,j} \) and \( C_{2,j} \) be costs for algorithms 1 and 2 on task \( j \), \( j=1,\dots,n \). Define the per-task difference

\[ D_j = C_{1,j} - C_{2,j}, \quad \bar{D} = \frac{1}{n}\sum_{j=1}^n D_j, \quad s_D^2 = \frac{1}{n-1}\sum_{j=1}^n (D_j-\bar{D})^2. \]

The null hypothesis “no difference in mean cost” is \( H_0: \mu_D = 0 \), with \( \mu_D = \mathbb{E}[D_j] \). The usual \( t \)-statistic is

\[ t = \frac{\bar{D}}{s_D/\sqrt{n}}, \]

which under mild assumptions is approximately \( t_{n-1} \) under \( H_0 \). A research-grade evaluation reports:

• estimated difference \( \bar{D} \),
• its confidence interval, and
• the \( p \)-value or at least whether \( H_0 \) is rejected at pre-specified \( \alpha \).

3.2 Effect sizes and sample size planning

Let \( \Delta = \mu_1 - \mu_2 \) denote the true mean difference (e.g., expected path length). A dimensionless effect size is

\[ d = \frac{\Delta}{\sigma}, \]

where \( \sigma \) is a characteristic standard deviation (e.g., pooled across algorithms). For a two-sided test with significance level \( \alpha \) and desired power \( 1-\beta \), the approximate required per-algorithm sample size in an unpaired design is

\[ n \approx 2\left( \frac{ z_{1-\alpha/2} + z_{1-\beta} }{ d } \right)^2. \]

This formula, derived from normal approximations, should guide the number of seeds or task instances in a robotics evaluation. A paper claiming “better performance” with \( n \) extremely small but no power analysis is methodologically weak.

3.3 Multiple tasks and average ranks

Suppose algorithms \( a=1,\dots,A \) are evaluated on tasks \( t=1,\dots,T \), producing performance scores \( Y_{a,t} \) (e.g., lower is better). A common protocol ranks algorithms per task:

\[ R_{a,t} = 1 + \#\{\,a' : Y_{a',t} < Y_{a,t}\,\}, \quad \bar{R}_a = \frac{1}{T}\sum_{t=1}^T R_{a,t}. \]

Lower average ranks \( \bar{R}_a \) indicate better performance. Tests such as Friedman or aligned-rank procedures are then used to assess whether rank differences are statistically significant. In a robotics paper, average ranks are preferable to cherry-picked plots on a subset of tasks.

A research-grade reader parses an evaluation section as a pipeline from hypotheses to conclusions, checking for hidden assumptions and missing details. The following diagram describes a mental checklist:

flowchart TD
  H["State the claim (e.g. 'planner A dominates planner B')"] --> D["Understand data: tasks, worlds, seeds"]
  D --> M["Inspect metrics: success, cost, robustness"]
  M --> P["Check protocol: train/test split, randomization, hyperparameters"]
  P --> S["Examine statistics: estimators, confidence intervals, tests"]
  S --> B["Compare baselines: strength, tuning, fairness"]
  B --> R["Look for robustness: ablations, stress tests, failure modes"]
  R --> C["Assess conclusions: are they supported by the evidence?"]
        

When reading a robotics paper, insist on:

• explicit description of task distributions and environment randomness;
• clear metric definitions with units (meters, seconds, Joules);
• number of trials and seeds, and justification of that choice;
• full reporting (not only “best of 3 runs”);
• transparent statistical tests and effect sizes;
• links to code and configurations for reproducibility (seen in Lesson 4).

A well-written evaluation section has a predictable structure. For advanced robotics, a good pattern is:

1. Setup. Describe robot models, simulators, and task generation. Formally, specify \( P_W \) (task distribution) and controller/sensor noise models.
2. Metrics. Define each metric as a function on trajectories. For instance, if \( \gamma \) is a trajectory and \( \ell(\gamma) \) its length,

\[ M_{\text{path}}(a,w,\xi) = \ell(\gamma_{a,w,\xi}), \quad M_{\text{succ}}(a,w,\xi) = \mathbf{1}\{\gamma_{a,w,\xi} \text{ reaches goal}\}. \]

3. Protocol. Describe how many tasks, seeds, and repeats; training vs evaluation splits; early stopping rules; collision margin settings, etc.
4. Results. Present tables and plots with aggregated metrics (means, medians, percentiles) plus confidence intervals or error bars.
5. Statistical analysis. Explicitly state hypotheses, test statistics, and whether results are significant at chosen \( \alpha \).
6. Robustness and ablations. Evaluate sensitivity to hyperparameters, noise levels, and key design choices.

In prose, avoid vague statements such as “A is much better than B”. Instead, write:

“On 100 randomly sampled manipulation scenes, method A achieves \( \hat{p}_A = 0.92 \) success (95% CI \([0.86, 0.96]\)), whereas method B achieves \( \hat{p}_B = 0.81 \) (95% CI \([0.73, 0.88]\)). A paired test across scenes yields \( p = 0.004 \), indicating a statistically significant improvement at \( \alpha=0.05 \).”

Suppose you logged OMPL- or MoveIt-based planning runs into a CSV file planning_benchmark.csv with columns: planner, scene_id, seed, success, path_length, and solve_time. The snippet below computes empirical metrics, confidence intervals, and a paired test between two planners.

import pandas as pd
import numpy as np
from scipy import stats

# Load benchmark log produced by e.g. OMPL or MoveIt
df = pd.read_csv("planning_benchmark.csv")

# Basic utility: confidence interval for the mean (continuous metric)
def mean_ci(x, alpha=0.05):
    x = np.asarray(x, dtype=float)
    n = x.size
    mean = x.mean()
    std = x.std(ddof=1)
    tval = stats.t.ppf(1.0 - alpha / 2.0, df=n - 1)
    half_width = tval * std / np.sqrt(n)
    return mean, mean - half_width, mean + half_width

# Success rate CI (normal approximation)
def success_rate_ci(successes, alpha=0.05):
    x = np.asarray(successes, dtype=float)
    n = x.size
    p_hat = x.mean()
    z = stats.norm.ppf(1.0 - alpha / 2.0)
    half_width = z * np.sqrt(p_hat * (1.0 - p_hat) / n)
    return p_hat, p_hat - half_width, p_hat + half_width

planners = df["planner"].unique()
print("Planners:", planners)

# Aggregate metrics per planner
for planner in planners:
    sub = df[df["planner"] == planner]
    succ, succ_lo, succ_hi = success_rate_ci(sub["success"])
    plen, plen_lo, plen_hi = mean_ci(sub[sub["success"] == 1]["path_length"])
    tmean, t_lo, t_hi = mean_ci(sub[sub["success"] == 1]["solve_time"])
    print(f"Planner {planner}")
    print(f"  Success rate: {succ:.3f} [{succ_lo:.3f}, {succ_hi:.3f}]")
    print(f"  Path length (success only): {plen:.3f} [{plen_lo:.3f}, {plen_hi:.3f}]")
    print(f"  Solve time (success only): {tmean:.3f} [{t_lo:.3f}, {t_hi:.3f}]")

# Paired comparison between two planners on the same (scene_id, seed)
planner_a, planner_b = planners[:2]

pivot = df.pivot_table(
    index=["scene_id", "seed"],
    columns="planner",
    values=["success", "path_length"]
)

# Paired differences for path length where both succeeded
mask_both = (pivot["success"][planner_a] == 1) & (pivot["success"][planner_b] == 1)
c1 = pivot["path_length"][planner_a][mask_both]
c2 = pivot["path_length"][planner_b][mask_both]
d = c1.values - c2.values
t_stat, p_val = stats.ttest_1samp(d, popmean=0.0)
print(f"Paired t-test on path length ({planner_a} - {planner_b}):")
print(f"  n_pairs = {d.size}, t = {t_stat:.3f}, p = {p_val:.3g}")
      

This analysis pipeline can be integrated into ROS-based benchmarking scripts, with rosbag recording trajectories and a Python layer converting them to CSV for analysis.

In C++, especially when using OMPL or custom planners in ROS, it is common to accumulate metrics online. The following snippet shows a simple accumulator for success rate and path cost statistics over repeated runs.

#include <vector>
#include <cmath>
#include <iostream>

struct MetricAccumulator {
    int nTrials = 0;
    int nSuccess = 0;
    double sumCost = 0.0;
    double sumSqCost = 0.0;

    void addTrial(bool success, double costIfSuccess) {
        ++nTrials;
        if (success) {
            ++nSuccess;
            sumCost += costIfSuccess;
            sumSqCost += costIfSuccess * costIfSuccess;
        }
    }

    double successRate() const {
        if (nTrials == 0) return 0.0;
        return static_cast<double>(nSuccess) / static_cast<double>(nTrials);
    }

    int nSuccessSamples() const { return nSuccess; }

    double meanCost() const {
        if (nSuccess == 0) return std::numeric_limits<double>::quiet_NaN();
        return sumCost / static_cast<double>(nSuccess);
    }

    double varCost() const {
        if (nSuccess <= 1) return std::numeric_limits<double>::quiet_NaN();
        double mu = meanCost();
        return (sumSqCost - static_cast<double>(nSuccess) * mu * mu)
               / static_cast<double>(nSuccess - 1);
    }

    // Approximate 95% CI for mean cost using normal approximation
    std::pair<double,double> meanCostCI95() const {
        if (nSuccess <= 1) {
            return {std::numeric_limits<double>::quiet_NaN(),
                    std::numeric_limits<double>::quiet_NaN()};
        }
        double mu = meanCost();
        double var = varCost();
        double se = std::sqrt(var / static_cast<double>(nSuccess));
        double z = 1.96; // approx for 95%
        return {mu - z * se, mu + z * se};
    }
};

int main() {
    MetricAccumulator acc;
    // Example loop over benchmark runs (e.g. OMPL's PlannerTerminationCondition)
    for (int i = 0; i < 100; ++i) {
        bool success = /* run planner and check if goal reached */;
        double pathLength = /* compute path length if success */;
        acc.addTrial(success, pathLength);
    }

    std::cout << "Success rate: " << acc.successRate() << std::endl;
    std::cout << "Mean cost (success only): " << acc.meanCost() << std::endl;
    auto ci = acc.meanCostCI95();
    std::cout << "95% CI for mean cost: [" << ci.first
              << ", " << ci.second << "]" << std::endl;
    return 0;
}
      

In an OMPL benchmarking harness, you would instantiate MetricAccumulator per planner and fill it inside the loop that calls solve with different random seeds.

Java is less common in robotics than C++ or Python, but rosjava and simulation front-ends sometimes produce logs that need statistical processing. The snippet below illustrates how to compute mean and confidence intervals for solve times from a CSV file.

import java.io.BufferedReader;
import java.io.FileReader;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;

public class BenchmarkStats {

    public static class Summary {
        public final double mean;
        public final double lower;
        public final double upper;

        public Summary(double mean, double lower, double upper) {
            this.mean = mean;
            this.lower = lower;
            this.upper = upper;
        }
    }

    public static Summary meanCI95(List<Double> xs) {
        int n = xs.size();
        if (n < 2) {
            return new Summary(Double.NaN, Double.NaN, Double.NaN);
        }
        double sum = 0.0;
        for (double x : xs) sum += x;
        double mean = sum / n;
        double sumSq = 0.0;
        for (double x : xs) {
            double d = x - mean;
            sumSq += d * d;
        }
        double var = sumSq / (n - 1);
        double se = Math.sqrt(var / n);
        double z = 1.96; // approx 95% normal
        return new Summary(mean, mean - z * se, mean + z * se);
    }

    public static void main(String[] args) throws IOException {
        String path = "planning_benchmark.csv";
        List<Double> solveTimesPlannerA = new ArrayList<>();

        try (BufferedReader br = new BufferedReader(new FileReader(path))) {
            String header = br.readLine(); // skip header
            String line;
            while ((line = br.readLine()) != null) {
                String[] tokens = line.split(",");
                String planner = tokens[0];
                boolean success = Integer.parseInt(tokens[3]) == 1;
                double solveTime = Double.parseDouble(tokens[5]);
                if (planner.equals("RRTstar") && success) {
                    solveTimesPlannerA.add(solveTime);
                }
            }
        }

        Summary s = meanCI95(solveTimesPlannerA);
        System.out.println("RRT*: mean solve time = " + s.mean
                + " [" + s.lower + ", " + s.upper + "]");
    }
}
      

This pattern can be extended to multiple planners and metrics, then used to automatically generate evaluation tables for a robotics paper.

MATLAB and Simulink are widely used for simulating robot manipulators and mobile robots. The following script sketches a Monte Carlo evaluation of a motion planner implemented via Robotics System Toolbox and Simulink.

% Assume you have a Simulink model "planner_benchmark"
% that, for each run, writes results to the workspace variables:
%   success (logical), pathLength (double), solveTime (double).

nTrials = 200;
success = false(nTrials,1);
pathLength = nan(nTrials,1);
solveTime = nan(nTrials,1);

for k = 1:nTrials
    % Randomize initial condition and goal
    q0 = rand(1,6) * 2*pi - pi;
    qg = rand(1,6) * 2*pi - pi;
    assignin('base','q0',q0);
    assignin('base','qg',qg);

    % Run Simulink model once
    simOut = sim('planner_benchmark','StopTime','10');

    success(k)    = simOut.success(end);
    pathLength(k) = simOut.pathLength(end);
    solveTime(k)  = simOut.solveTime(end);
end

% Success rate and CI
pHat = mean(success);
alpha = 0.05;
z = norminv(1 - alpha/2);
n = nTrials;
halfWidthSucc = z * sqrt(pHat * (1 - pHat) / n);
succCI = [pHat - halfWidthSucc, pHat + halfWidthSucc];

% Mean path length and CI over successful runs
idx = success == 1;
L = pathLength(idx);
muL = mean(L);
sigmaL = std(L,1);          % population std
seL = sigmaL / sqrt(numel(L));
halfWidthL = z * seL;
lenCI = [muL - halfWidthL, muL + halfWidthL];

fprintf('Success rate: %.3f [%.3f, %.3f]\n', pHat, succCI(1), succCI(2));
fprintf('Path length (success): %.3f [%.3f, %.3f]\n', muL, lenCI(1), lenCI(2));
      

The same pattern applies when evaluating closed-loop controllers in Simulink: record success and performance metrics in a To Workspace block and post-process them as above.

Mathematica is useful for symbolic manipulation and probabilistic reasoning about evaluation metrics. The example below defines symbolic success rates and uses Monte Carlo simulation to approximate the distribution of the difference in success rates between two planners.

(* Symbolic mean and variance of the empirical success rate *)
Clear[p, n];
pHat = 1/n Sum[Subscript[S, i], {i, 1, n}];
(* Assume S_i are i.i.d. Bernoulli(p) *)
meanPHat = Expectation[pHat, Array[Subscript[S, #] &, n] \[Distributed] 
   ProductDistribution @@ Table[BernoulliDistribution[p], {n}]];
varPHat = Variance[pHat, Array[Subscript[S, #] &, n] \[Distributed] 
   ProductDistribution @@ Table[BernoulliDistribution[p], {n}]];

(* Monte Carlo comparison of two planners with pA and pB *)
pA = 0.9; pB = 0.8; nTrials = 200;
sim = Table[
   Module[{sa, sb},
    sa = RandomVariate[BernoulliDistribution[pA], nTrials];
    sb = RandomVariate[BernoulliDistribution[pB], nTrials];
    Mean[sa] - Mean[sb]
   ],
   {10000}
];

Histogram[sim, 50, "Probability",
 PlotLabel -> "Distribution of empirical success-rate difference (A - B)"];

N@Mean[sim]
N@StandardDeviation[sim]
      

Symbolic expressions such as meanPHat and varPHat help verify textbook formulas before implementing them in robotics benchmarking code.

Problem 1 (Unbiasedness of success-rate estimator). Consider an algorithm evaluated on \( n \) tasks. Let \( S_j \in \{0,1\} \) be the success indicator on task \( j \), with \( \mathbb{P}(S_j=1)=p \) and i.i.d. across \( j \). Show that the empirical success rate \( \hat{p} = \frac{1}{n}\sum_{j=1}^n S_j \) is an unbiased estimator of \( p \) and compute its variance.

Solution. Linearity of expectation gives

\[ \mathbb{E}[\hat{p}] = \frac{1}{n}\sum_{j=1}^n \mathbb{E}[S_j] = \frac{1}{n}\sum_{j=1}^n p = p. \]

For the variance, using independence and \( \operatorname{Var}(S_j) = p(1-p) \):

\[ \operatorname{Var}(\hat{p}) = \frac{1}{n^2}\sum_{j=1}^n \operatorname{Var}(S_j) = \frac{1}{n^2}\sum_{j=1}^n p(1-p) = \frac{p(1-p)}{n}. \]

Problem 2 (Sample size for path-length comparison). Two planners are compared on independent tasks. Suppose the path length per trial is approximately normal with common standard deviation \( \sigma = 0.3 \) meters. You want to detect a mean difference \( |\Delta| = 0.1 \) meters with power \( 1-\beta = 0.8 \) at \( \alpha = 0.05 \) (two-sided). Using the approximation

\[ n \approx 2\left( \frac{ z_{1-\alpha/2} + z_{1-\beta} }{ \Delta / \sigma } \right)^2, \]

compute the required per-algorithm sample size \( n \).

Solution. For \( \alpha=0.05 \), \( z_{1-\alpha/2} \approx 1.96 \). For \( 1-\beta = 0.8 \), \( z_{1-\beta} \approx 0.84 \). Thus \( \Delta/\sigma = 0.1/0.3 \approx 0.333\overline{3} \) and

\[ n \approx 2\left( \frac{1.96 + 0.84}{0.333\overline{3}} \right)^2 = 2\left( \frac{2.8}{0.333\overline{3}} \right)^2 \approx 2\cdot(8.4)^2 \approx 2\cdot70.56 \approx 141.12. \]

Hence you need approximately \( n \approx 142 \) tasks per algorithm.

Problem 3 (Paired test interpretation in a manipulation benchmark). Two grasp planners A and B are evaluated on \( n=50 \) objects. The empirical per-object success differences \( D_j = S_{A,j} - S_{B,j} \) yield \( \bar{D}=0.14 \) and \( s_D = 0.40 \). Compute the paired \( t \)-statistic and determine whether the improvement is significant at \( \alpha=0.05 \).

Solution. The test statistic is

\[ t = \frac{\bar{D}}{s_D/\sqrt{n}} = \frac{0.14}{0.40/\sqrt{50}} \approx \frac{0.14}{0.05657} \approx 2.476. \]

For \( n-1 = 49 \) degrees of freedom, the two-sided critical value at \( \alpha=0.05 \) is approximately \( t_{49,0.975} \approx 2.01 \). Since \( 2.476 > 2.01 \), the null hypothesis of equal mean success is rejected, and the improvement is significant at \( \alpha=0.05 \).

Problem 4 (Average ranks across tasks). Three planners are evaluated on \( T=4 \) tasks. Lower cost is better. The normalized costs are:

• Task 1: A=1.0, B=1.2, C=1.5
• Task 2: A=2.0, B=1.9, C=2.2
• Task 3: A=0.9, B=1.0, C=0.95
• Task 4: A=1.1, B=1.1, C=1.3

Assign ranks per task (1 is best) and compute the average rank \( \bar{R}_a \) for each planner. Which planner is best by average rank?

Solution.

• Task 1: A(1.0)→rank 1, B(1.2)→rank 2, C(1.5)→rank 3.
• Task 2: B(1.9)→rank 1, A(2.0)→rank 2, C(2.2)→rank 3.
• Task 3: A(0.9)→rank 1, C(0.95)→rank 2, B(1.0)→rank 3.
• Task 4: A(1.1), B(1.1) tie; assign average rank 1.5 each; C(1.3)→rank 3.

Summing over tasks:

• A: ranks \( 1,2,1,1.5 \) → \( \bar{R}_A = (1+2+1+1.5)/4 = 1.375 \).
• B: ranks \( 2,1,3,1.5 \) → \( \bar{R}_B = (2+1+3+1.5)/4 = 1.875 \).
• C: ranks \( 3,3,2,3 \) → \( \bar{R}_C = (3+3+2+3)/4 = 2.75 \).

Thus planner A is best by average rank.

Problem 5 (Designing an evaluation section). Sketch the logical structure of an evaluation section for a new kinodynamic planner that claims improved robustness to actuation limits.

Solution (high-level flow).

flowchart TD
  C["Claim: new kinodynamic planner is more robust to actuation limits"] --> T["Define tasks: random start-goal with dynamic constraints"]
  T --> M["Specify metrics: success, time-to-goal, control effort"]
  M --> P["Describe protocol: number of tasks, seeds, simulators, noise"]
  P --> B["Choose baselines: existing kinodynamic planners, same tuning budget"]
  B --> S["Plan statistics: estimators, confidence intervals, hypothesis tests"]
  S --> R["Report results: tables, plots, ablations, failure cases"]
        

An actual paper should instantiate each node of this flow with precise details, numbers, and statistical statements.

In this lesson, we formalized evaluation in advanced robotics as estimating population performance functionals of random metrics (success, cost, time) and built the statistical tools required for research-grade evaluation: estimators, confidence intervals, hypothesis tests, and effect sizes. We discussed how to critically read evaluation sections and how to write them in a way that clearly connects algorithmic claims to empirical evidence. Finally, we demonstrated concrete implementations in Python, C++, Java, MATLAB/Simulink, and Wolfram Mathematica that integrate naturally with common robotics libraries and simulators.