Robotics experiments couple algorithms, electromechanical systems, sensors, and human choices (parameter tuning, environment selection). As a result, a qualitative statement like “the new controller works better” is meaningless without:

• A precisely defined performance metric.
• A clear experimental protocol that could be repeated by others.
• Quantification of uncertainty (variation across trials, noise).
• A structured report that separates facts (measurements) from interpretation.

Let \( J \) denote a scalar performance metric for a single run of a robot experiment, for example the root-mean-square tracking error over a trajectory. We model \( J \) as a random variable with (unknown) mean \( \mu_J \) and variance \( \sigma_J^2 \). The fundamental goal of experimental analysis is to estimate \( \mu_J \) and compare it between different controllers, environments, or hardware configurations.

\[ \mu_J = \mathbb{E}[J], \qquad \sigma_J^2 = \mathbb{E}\big[(J-\mu_J)^2\big]. \]

An experiment is rigorous if it provides enough information and mathematical justification that another competent robotics group could:

1. Recreate the robot setup and experimental conditions.
2. Recompute the metrics from raw logs.
3. Check whether your statistical conclusions still hold.

Suppose we execute the same experiment \( N \) times under nominally identical conditions (same path, controller, environment). Let \( J_1, J_2, \dots, J_N \) be the resulting scalar metrics (e.g., tracking RMS error for each trial). A simple additive noise model is

\[ J_k = \mu_J + \varepsilon_k, \qquad k = 1,\dots,N, \]

where \( \varepsilon_k \) are independent random variables with \( \mathbb{E}[\varepsilon_k]=0 \) and \( \operatorname{Var}(\varepsilon_k)=\sigma_J^2 \). The sample mean is

\[ \bar{J} = \frac{1}{N}\sum_{k=1}^{N} J_k, \]

and the unbiased sample variance is

\[ s_J^2 = \frac{1}{N-1} \sum_{k=1}^{N} (J_k - \bar{J})^2. \]

Under the model above, we can prove two key properties that should be quoted in a rigorous robotics report.

Property 1 (Unbiasedness of the mean).

\[ \mathbb{E}[\bar{J}] = \mu_J. \]

Proof. Using linearity of expectation, \( \mathbb{E}[\bar{J}] = \frac{1}{N} \sum_{k=1}^{N} \mathbb{E}[J_k] = \frac{1}{N} \sum_{k=1}^{N} \mu_J = \mu_J. \)

Property 2 (Variance reduction by repeated trials).

\[ \operatorname{Var}(\bar{J}) = \frac{\sigma_J^2}{N}. \]

Proof. Independence implies \( \operatorname{Var}\big(\sum_{k} J_k\big) = \sum_{k}\operatorname{Var}(J_k) = N\sigma_J^2 \). Therefore \( \operatorname{Var}(\bar{J}) = \operatorname{Var}(\frac{1}{N}\sum_k J_k) = \frac{1}{N^2} N \sigma_J^2 = \sigma_J^2 / N. \)

Intuitively, doubling the number of trials reduces the variance of the mean by a factor of two. This is why, in robotics experiments, a single “nice” run is never enough for a scientific claim.

If the noise is approximately Gaussian, a \( (1-\alpha) \) confidence interval for \( \mu_J \) is often reported as

\[ \bar{J} \pm t_{1-\alpha/2,\,N-1}\,\frac{s_J}{\sqrt{N}}, \]

where \( t_{1-\alpha/2,\,N-1} \) is the quantile of the Student t distribution with \( N-1 \) degrees of freedom. In a report you must specify both \( N \) and \( \alpha \) (e.g., a 95% interval).

In a control-theoretic robotics experiment, the robot has a reference output trajectory \( y_{\text{ref}}(t) \) (e.g., a desired path) and a measured output \( y(t) \). Define the error

\[ e(t) = y_{\text{ref}}(t) - y(t). \]

With a sampling period \( T_s \) and samples \( e_1, \dots, e_K \), typical discrete-time metrics reported in robot experiments are:

• Root-mean-square (RMS) tracking error

  \[ J_{\mathrm{RMS}} = \sqrt{\frac{1}{K} \sum_{k=1}^{K} e_k^2}. \]

• Maximum absolute deviation

  \[ J_{\infty} = \max_{1 \leq k \leq K} \lvert e_k \rvert. \]

• Control effort using the control input samples \( u_1,\dots,u_K \)

  \[ J_{u} = \sum_{k=1}^{K} u_k^2\,T_s. \]

When describing your metrics, a rigorous report must:

1. Provide the exact mathematical definition (as above).
2. State the units and physical meaning.
3. Explain whether lower or higher values are better.

For example, in a mobile robot path-following experiment, you might state: “We use \( J_{\mathrm{RMS}} \) (m) and \( J_{\infty} \) (m) as primary accuracy metrics, and \( J_u \) (rad\(^2\)/s\(^2\)) to quantify control effort. For each controller, the reported values are means over \( N \) repeated trials with 95% confidence intervals.”

Consider two controllers, A and B, evaluated on the same robot and task. Let \( J_1^{(A)},\dots,J_{N_A}^{(A)} \) and \( J_1^{(B)},\dots,J_{N_B}^{(B)} \) be their RMS errors over repeated trials. Define the sample means and variances

\[ \bar{J}_A = \frac{1}{N_A}\sum_{k=1}^{N_A} J_k^{(A)}, \quad s_A^2 = \frac{1}{N_A-1}\sum_{k=1}^{N_A} \big(J_k^{(A)}-\bar{J}_A\big)^2, \]

\[ \bar{J}_B = \frac{1}{N_B}\sum_{k=1}^{N_B} J_k^{(B)}, \quad s_B^2 = \frac{1}{N_B-1}\sum_{k=1}^{N_B} \big(J_k^{(B)}-\bar{J}_B\big)^2. \]

To make a rigorous statement “controller A has smaller mean error than B”, one can formulate the hypotheses

\[ H_0 : \mu_A = \mu_B, \qquad H_1 : \mu_A < \mu_B, \]

where \( \mu_A \) and \( \mu_B \) are the (unknown) true mean errors for A and B. A commonly used test statistic (under approximate normality and independent trials) is

\[ t = \frac{\bar{J}_A - \bar{J}_B}{\sqrt{\dfrac{s_A^2}{N_A} + \dfrac{s_B^2}{N_B}}}. \]

The denominator is the estimated standard deviation of \( \bar{J}_A - \bar{J}_B \). A detailed report does not need to re-derive the distribution theory, but it should:

• Specify which statistical test was used (e.g., two-sample t-test).
• State the significance level (e.g., \( \alpha = 0.05 \)).
• Report the test statistic and p-value, not only a qualitative verdict.

Even if your course does not pursue full statistical theory, using these formulas to compute and quote uncertainty makes your robotics report much more convincing and scientifically grounded.

A rigorous experimental protocol structures the entire lifecycle from research question to written report. The following flow chart represents a typical process for a robotics integration lab experiment.

flowchart TD
  Q["Formulate precise question"] --> H["Define hypotheses and metrics"]
  H --> D["Design experiment and choose variables"]
  D --> R["Run repeated trials and log raw data"]
  R --> A["Analyze data and compute statistics"]
  A --> C["Check assumptions and identify limitations"]
  C --> W["Write structured report and archive artifacts"]
        

Each block must be reflected in your written report:

• Question: clearly states what is being tested.
• Metrics: defined mathematically (Section 3).
• Design: describes environments, number of trials, and controlled variables (e.g., same initial pose, same reference).
• Trials and data: specify how logs were stored, with time stamps and units.
• Analysis: details formulas, averages, confidence intervals, and comparisons (Sections 2–4).
• Limitations: discusses repeatability, sensor drift, modeling assumptions, and possible biases.

A well-written robotics report typically follows a structure similar to the IMRaD style (Introduction–Methods–Results–Discussion), extended with system-specific descriptions and experimental details. A high-level organization is illustrated below.

flowchart TD
  T["Title, authors, abstract"] --> I["Introduction and objectives"]
  I --> S["System and task description"]
  S --> M["Experimental methods and metrics"]
  M --> R["Results with tables and plots"]
  R --> D["Discussion and limitations"]
  D --> C["Conclusion and future work"]
  C --> A["Appendices and code or data links"]
        

In more detail:

• Abstract: 5–7 sentences summarizing the task, methods, key quantitative results (with numbers and units), and main conclusion.
• Introduction: motivation, problem formulation, and a concise statement of contributions (what is new or insightful).
• System and task description: robot hardware, sensors, controller architecture, and experimental environment. Include diagrams or references to earlier lessons for the sense–think–act loop.
• Experimental methods: what you varied (e.g., controller gains), what you held constant, how many trials you ran, sampling rates, and formal metric definitions.
• Results: numerical tables and plots of \( \bar{J} \), confidence intervals, and representative trajectories. Avoid raw screenshots without explanation.
• Discussion: interpret trends, relate back to control concepts (overshoot, settling time), and discuss plausible physical or algorithmic reasons for observed behavior.
• Conclusion: concise bullet points summarizing what was learned and what should be improved next.

Assume you exported your robot logs (from a simulator or real robot) to a CSV file with columns time, y_ref, y_meas, u, and an integer trial_id. The code below shows, in several languages, how to compute the RMS tracking error \( J_{\mathrm{RMS}} \) for each trial and the mean and standard deviation across trials. These computations implement the formulas in Sections 2 and 3.

7.1 Python (NumPy / Pandas)

import pandas as pd
import numpy as np

# Load CSV exported from simulator or robot logger
df = pd.read_csv("robot_logs.csv")

# Compute error signal
df["e"] = df["y_ref"] - df["y_meas"]

# Group by trial and compute RMS error
def rms(x):
    return np.sqrt(np.mean(np.square(x)))

rms_per_trial = df.groupby("trial_id")["e"].apply(rms)

J_values = rms_per_trial.values
N = len(J_values)

J_mean = np.mean(J_values)
J_std = np.std(J_values, ddof=1)

print("RMS error per trial:")
print(rms_per_trial)
print(f"Mean RMS error: {J_mean:.4f}")
print(f"Std of RMS error: {J_std:.4f}")

# 95% confidence interval for the mean (approximate, using normal quantile)
alpha = 0.05
z = 1.96  # approximate for large N
se = J_std / np.sqrt(N)
print(f"95% CI for mean RMS error: [{J_mean - z * se:.4f}, {J_mean + z * se:.4f}]")
      

7.2 C++ (standard library)

#include <iostream>
#include <fstream>
#include <sstream>
#include <vector>
#include <unordered_map>
#include <cmath>

struct Sample {
    double e;
};

int main() {
    std::ifstream in("robot_logs.csv");
    if (!in) {
        std::cerr << "Cannot open file\n";
        return 1;
    }

    std::string header;
    std::getline(in, header); // skip header

    // Map trial_id to list of error samples
    std::unordered_map<int, std::vector<Sample>> data;

    std::string line;
    while (std::getline(in, line)) {
        std::stringstream ss(line);
        std::string token;
        double time, y_ref, y_meas, u;
        int trial_id;

        std::getline(ss, token, ','); time = std::stod(token);
        std::getline(ss, token, ','); y_ref = std::stod(token);
        std::getline(ss, token, ','); y_meas = std::stod(token);
        std::getline(ss, token, ','); u = std::stod(token);
        std::getline(ss, token, ','); trial_id = std::stoi(token);

        Sample s;
        s.e = y_ref - y_meas;
        data[trial_id].push_back(s);
    }

    std::vector<double> J_values;
    for (auto &kv : data) {
        const auto &samples = kv.second;
        double sum_sq = 0.0;
        for (const auto &s : samples) {
            sum_sq += s.e * s.e;
        }
        double J_rms = std::sqrt(sum_sq / samples.size());
        J_values.push_back(J_rms);
        std::cout << "Trial " << kv.first << " RMS error = " << J_rms << "\n";
    }

    int N = static_cast<int>(J_values.size());
    double mean = 0.0;
    for (double v : J_values) mean += v;
    mean /= N;

    double var = 0.0;
    for (double v : J_values) {
        double d = v - mean;
        var += d * d;
    }
    var /= (N - 1);
    double stddev = std::sqrt(var);

    std::cout << "Mean RMS error = " << mean << "\n";
    std::cout << "Std of RMS error = " << stddev << "\n";

    return 0;
}
      

7.3 Java (standard library)

import java.io.BufferedReader;
import java.io.FileReader;
import java.io.IOException;
import java.util.*;

public class RobotStats {
    public static void main(String[] args) throws IOException {
        Map<Integer, List<Double>> data = new HashMap<>();

        try (BufferedReader br = new BufferedReader(new FileReader("robot_logs.csv"))) {
            String line = br.readLine(); // header
            while ((line = br.readLine()) != null) {
                String[] tokens = line.split(",");
                double yRef = Double.parseDouble(tokens[1]);
                double yMeas = Double.parseDouble(tokens[2]);
                int trialId = Integer.parseInt(tokens[4]);
                double e = yRef - yMeas;

                data.computeIfAbsent(trialId, k -> new ArrayList<>()).add(e);
            }
        }

        List<Double> Jvalues = new ArrayList<>();
        for (Map.Entry<Integer, List<Double>> entry : data.entrySet()) {
            int trialId = entry.getKey();
            List<Double> samples = entry.getValue();
            double sumSq = 0.0;
            for (double e : samples) {
                sumSq += e * e;
            }
            double J = Math.sqrt(sumSq / samples.size());
            Jvalues.add(J);
            System.out.println("Trial " + trialId + " RMS error = " + J);
        }

        int N = Jvalues.size();
        double mean = 0.0;
        for (double v : Jvalues) mean += v;
        mean /= N;

        double var = 0.0;
        for (double v : Jvalues) {
            double d = v - mean;
            var += d * d;
        }
        var /= (N - 1);
        double stddev = Math.sqrt(var);

        System.out.println("Mean RMS error = " + mean);
        System.out.println("Std of RMS error = " + stddev);
    }
}
      

7.4 MATLAB / Simulink

% Load logs exported from the experiment (e.g., from Simulink Data Inspector)
data = readtable("robot_logs.csv");

% Error signal
data.e = data.y_ref - data.y_meas;

trial_ids = unique(data.trial_id);
J_values = zeros(numel(trial_ids), 1);

for i = 1:numel(trial_ids)
    tid = trial_ids(i);
    idx = data.trial_id == tid;
    e = data.e(idx);
    J_values(i) = sqrt(mean(e.^2));
    fprintf("Trial %d RMS error = %.4f\n", tid, J_values(i));
end

J_mean = mean(J_values);
J_std  = std(J_values, 1);   % population std; for sample use std(J_values, 0)

fprintf("Mean RMS error = %.4f\n", J_mean);
fprintf("Std of RMS error = %.4f\n", J_std);

% Example: if J_values come from a Simulink model run multiple times
% for different seeds, this script can be placed in a Live Script that
% also embeds plots and report text.
      

In a full integration lab, you would connect these computations back to the report by including tables of \( \bar{J} \) and \( s_J \), plots of trajectories, and a textual interpretation of the numerical differences between controllers or setups.

In this lesson we connected mathematical models of experimental uncertainty (means, variances, confidence intervals) with practical choices in robotics experiments and report writing. By defining clear metrics such as RMS tracking error, modeling repeated trials as random variables, and applying elementary statistical reasoning, we obtain reproducible and interpretable evaluations of robot performance. A structured report then documents the entire process from question to conclusion, enabling other researchers to verify, critique, and build on your work.