In a robotics capstone, it is not enough that the robot works once. We need evidence that the system reliably satisfies explicit requirements such as tracking accuracy, safety margins, and task completion time under realistic disturbances and sensor noise. This is the purpose of validation: checking whether the implemented robot actually meets its performance specifications in the real (or simulated) environment.

We assume that, from linear control, you are familiar with modeling a closed-loop system, defining a reference signal \( r(t) \), and measuring the output \( y(t) \). Validation makes these ideas concrete for your robot by:

• choosing metrics on signals such as tracking error or control effort,
• designing test scenarios and repetitions, and
• checking whether metrics satisfy acceptance thresholds.

At a high level, the validation flow for your capstone robot can be summarized as:

flowchart TD
  R["Specify requirements (accuracy, safety, time)"] --> M["Select quantitative metrics"]
  M --> T["Design test cases and scenarios"]
  T --> S["Run batch of tests in simulation"]
  S --> H["Run corresponding tests on hardware"]
  H --> L["Log data from sensors and controller"]
  L --> C["Compute performance metrics"]
  C --> J["Compare metrics with thresholds"]
  J --> DONE["Pass: ready for final demo"]
  J --> IMP["Fail: refine design and repeat tests"]
  

The rest of this lesson develops the mathematical definitions behind these metrics and shows how to compute and interpret them rigorously.

Consider a single-input single-output loop with reference \( r(t) \), measured output \( y(t) \), and tracking error

\[ e(t) = r(t) - y(t), \quad 0 \leq t \leq T. \]

On a finite horizon \( [0,T] \), common scalar performance indices based on \( e(t) \) are:

• Integral of absolute error (IAE):

  \[ \text{IAE} = \int_{0}^{T} |e(t)| \,\mathrm{d}t. \]

  This penalizes persistent error regardless of its sign.
• Integral of squared error (ISE):

  \[ \text{ISE} = \int_{0}^{T} e^{2}(t) \,\mathrm{d}t. \]

  Large excursions are heavily penalized.
• Integral of time-weighted absolute error (ITAE):

  \[ \text{ITAE} = \int_{0}^{T} t\,|e(t)| \,\mathrm{d}t. \]

  This emphasizes errors that occur late in the response.
• Maximum error (infinity norm):

  \[ \|e\|_{\infty,[0,T]} = \max_{0 \leq t \leq T} |e(t)|. \]

  This is often related to peak overshoot or a safety margin.

In practice, your capstone robot logs data at discrete times \( t_{k} \) with sampling period \( \Delta t \) and error samples \( e_{k} = e(t_{k}) \). The continuous-time indices are approximated by Riemann sums:

\[ \text{IAE}_{N} = \sum_{k=0}^{N-1} |e_{k}|\,\Delta t,\quad \text{ISE}_{N} = \sum_{k=0}^{N-1} e_{k}^{2}\,\Delta t \]

\[ \text{ITAE}_{N} = \sum_{k=0}^{N-1} t_{k}\,|e_{k}|\,\Delta t,\quad \|e\|_{\infty,N} = \max_{0 \leq k \leq N-1} |e_{k}|. \]

Another widely used index is the root-mean-square error (RMSE) over \(N\) samples:

\[ \text{RMSE} = \sqrt{\frac{1}{N}\sum_{k=0}^{N-1} e_{k}^{2}}. \]

Note that \(\text{ISE}_{N}\) and \(\text{RMSE}\) are closely related: over a fixed horizon \(T = N\,\Delta t\), we have \(\text{ISE}_{N} = N\,\Delta t\,\text{RMSE}^{2}\).

In your capstone, you can specify quantitative requirements such as "RMSE of end-effector position below 1 cm over a 10 s tracking task" or "IAE of heading error below a given bound".

While signal-based metrics capture low-level loop performance, a robot must also satisfy task-level goals: following a geometric path, reaching waypoints, or manipulating objects. Let \( p(t) \in \mathbb{R}^{n} \) denote the robot pose (e.g., position and yaw), and let \( p_{\text{ref}}(t) \) be the desired trajectory. The path-tracking error is

\[ e_{p}(t) = p(t) - p_{\text{ref}}(t). \]

A typical task-level tracking metric is the RMS path error:

\[ \text{RMSE}_{\text{path}} = \sqrt{\frac{1}{T} \int_{0}^{T} \|e_{p}(t)\|_{2}^{2} \,\mathrm{d}t} \approx \sqrt{\frac{1}{N} \sum_{k=0}^{N-1} \|e_{p,k}\|_{2}^{2}}, \]

where \( e_{p,k} = p_{k} - p_{\text{ref},k} \). For a waypoint navigation task, you might also record:

• Time to completion: each trial \(i\) yields a random completion time \( T_{c,i} \). The empirical mean and variance are

\[ \hat{T}_{c} = \frac{1}{N} \sum_{i=1}^{N} T_{c,i},\quad s_{T}^{2} = \frac{1}{N-1}\sum_{i=1}^{N} (T_{c,i} - \hat{T}_{c})^{2}. \]

• Success probability: for each trial \(i\), define a Bernoulli variable \(\,X_{i} = 1\,\) if the robot completes the task in time, and \(X_{i} = 0\) otherwise. The empirical success rate is

\[ \hat{p}_{\text{succ}} = \frac{1}{N}\sum_{i=1}^{N} X_{i}. \]

If your capstone robot uses simple perception (for example, thresholding a depth image to detect obstacles), you can also track classification metrics. Let TP, FP, TN, FN denote true positives, false positives, true negatives, and false negatives for obstacle detection. Then:

\[ \text{Accuracy} = \frac{\text{TP} + \text{TN}}{\text{TP} + \text{FP} + \text{TN} + \text{FN}}, \quad \text{Precision} = \frac{\text{TP}}{\text{TP} + \text{FP}}, \quad \text{Recall} = \frac{\text{TP}}{\text{TP} + \text{FN}}. \]

These quantities summarize how informative the perception module is for the rest of the robot stack, and they can be treated as additional performance metrics during validation.

Due to sensor noise, unmodeled disturbances, and changes in initial conditions, metrics are random variables. Suppose a given performance index \(J\) (for example, RMSE) is measured over \(N\) independent trials, yielding samples \( J_{1},\dots,J_{N} \). We model \(J_{i}\) as i.i.d. with mean \( \mu \) and variance \( \sigma^{2} \).

The sample mean and sample variance are

\[ \bar{J} = \frac{1}{N}\sum_{i=1}^{N} J_{i}, \quad s^{2} = \frac{1}{N-1}\sum_{i=1}^{N} (J_{i} - \bar{J})^{2}. \]

By linearity of expectation, \( \mathbb{E}[\bar{J}] = \mu \), so \(\bar{J}\) is an unbiased estimator of the true average performance. Under mild conditions, the central limit theorem implies that for large \(N\), \(\bar{J}\) is approximately Gaussian, which allows us to form confidence intervals.

For example, an approximate \( (1-\alpha) \) confidence interval for \( \mu \) is

\[ \bar{J} \,\pm\, z_{1-\alpha/2}\, \frac{s}{\sqrt{N}}, \]

where \( z_{1-\alpha/2} \) is the quantile of the standard normal distribution. In the context of the capstone, a reasonable validation statement could be:

"Based on \(N=20\) trials, the estimated mean path RMSE is \(0.012\,\mathrm{m}\) with a 95% confidence interval of \([0.010, 0.014]\ \mathrm{m}\), which lies below the 0.02 m requirement."

For Bernoulli metrics such as the success rate \( \hat{p}_{\text{succ}} \), the sample mean is again unbiased, and a large-sample confidence interval is

\[ \hat{p}_{\text{succ}} \,\pm\, z_{1-\alpha/2}\, \sqrt{\frac{\hat{p}_{\text{succ}} \left(1-\hat{p}_{\text{succ}}\right)}{N}}. \]

This allows you to argue that your robot achieves, for example, at least a 90% success rate with high confidence.

Most control-theoretic performance indices are defined as integrals over continuous time, but in both simulation and hardware you only observe sampled data. Consider again the ISE:

\[ \text{ISE} = \int_{0}^{T} e^{2}(t)\,\mathrm{d}t. \]

With uniform sampling \( t_{k} = k\,\Delta t \) and \( N\,\Delta t = T \), the discrete approximation is

\[ \text{ISE}_{N} = \sum_{k=0}^{N-1} e_{k}^{2}\,\Delta t. \]

From calculus, this is a Riemann sum. If the error signal \(e(t)\) is continuous and bounded on \([0,T]\), then as \( \Delta t \to 0 \) (and hence \(N \to \infty\) with \(N\,\Delta t = T\)), we have \( \text{ISE}_{N} \to \text{ISE} \). Therefore, when your simulator produces sufficiently dense samples, using the discrete sums as approximations of the continuous indices is mathematically justified.

In practice, you will use exactly the same discrete formulas for both simulation and hardware logs. This makes it straightforward to compare your design in the idealized model with its behavior on the physical robot.

Because failures are cheaper in simulation, a typical capstone workflow validates first in a physics engine and only then on the real robot. Using the same metrics in both domains lets you quantify the simulation-to-hardware gap.

flowchart TD
  SPEC["Design controller and planner"] --> SIM["Run many scenarios in simulation"]
  SIM --> METSIM["Compute metrics from simulated logs"]
  METSIM --> THR["Check metrics against requirements"]
  THR --> HW["OK - deploy configuration to hardware"]
  THR --> RET["Not OK - adjust design or gains"]
  HW --> METHW["Compute metrics from hardware logs"]
  METHW --> COMP["Compare simulation and hardware metrics"]
  COMP --> DEC["Accept design or refine models"]
  

Quantitatively, you might report, for example:

• Simulated path RMSE: \( \text{RMSE}_{\text{sim}} = 0.009\,\mathrm{m} \),
• Hardware path RMSE: \( \text{RMSE}_{\text{hw}} = 0.013\,\mathrm{m} \),
• Relative increase: \( (\text{RMSE}_{\text{hw}} - \text{RMSE}_{\text{sim}}) / \text{RMSE}_{\text{sim}} \approx 0.44 \).

Such quantitative comparisons help you argue whether the simulator is sufficiently faithful for design purposes, or whether additional modeling of friction, delays, or sensor noise is required.

Python, together with numpy and matplotlib, is widely used in robotics for quick analysis of log files. The following example assumes that you have arrays of timestamps t, reference positions r, and measured positions y from your capstone robot (possibly extracted from a ROS bag as discussed in the ROS lessons):

import numpy as np

# Example: step response tracking
# t: time stamps [s], r: reference signal, y: measured output
T = 10.0
N = 1001
t = np.linspace(0.0, T, N)
r = np.ones_like(t)                  # unit step reference
y = 1.0 - np.exp(-t) * np.cos(2.0*t) # example response
e = r - y

dt = t[1] - t[0]

# Signal-based metrics
iae = np.sum(np.abs(e)) * dt
ise = np.sum(e**2) * dt
itae = np.sum(t * np.abs(e)) * dt
rmse = np.sqrt(np.mean(e**2))
max_err = np.max(np.abs(e))

print(f"IAE  = {iae:.4f}")
print(f"ISE  = {ise:.4f}")
print(f"ITAE = {itae:.4f}")
print(f"RMSE = {rmse:.4f}")
print(f"max |e| = {max_err:.4f}")

# Suppose we have multiple trials for a path-tracking experiment:
# errors_trials: list of numpy arrays of per-sample errors
errors_trials = [e + 0.01*np.random.randn(N) for _ in range(10)]
rmse_trials = [np.sqrt(np.mean(err**2)) for err in errors_trials]

rmse_mean = np.mean(rmse_trials)
rmse_std = np.std(rmse_trials, ddof=1)

print(f"Mean RMSE over trials = {rmse_mean:.4f}")
print(f"Std dev of RMSE       = {rmse_std:.4f}")
      

In your project repository, you can package such analysis into a reusable script (for example, analyze_logs.py) that your team runs whenever a new batch of experiments is recorded.

Many robot controllers are implemented in C++, especially within ROS2 or other real-time frameworks. The following C++ snippet shows how to accumulate IAE, ISE, and maximum error over a trajectory, using std::vector:

#include <vector>
#include <cmath>
#include <iostream>

struct Metrics {
    double iae;
    double ise;
    double itae;
    double rmse;
    double max_abs_err;
};

Metrics compute_metrics(const std::vector<double>& t,
                        const std::vector<double>& r,
                        const std::vector<double>& y)
{
    const std::size_t N = t.size();
    if (N == 0 || r.size() != N || y.size() != N) {
        throw std::runtime_error("Input vectors must have same nonzero size.");
    }

    const double dt = (N > 1) ? (t[1] - t[0]) : 0.0;

    double iae = 0.0;
    double ise = 0.0;
    double itae = 0.0;
    double sum_e2 = 0.0;
    double max_abs_err = 0.0;

    for (std::size_t k = 0; k < N; ++k) {
        double e = r[k] - y[k];
        double ae = std::fabs(e);

        iae += ae * dt;
        ise += e * e * dt;
        itae += t[k] * ae * dt;
        sum_e2 += e * e;

        if (ae > max_abs_err) {
            max_abs_err = ae;
        }
    }

    Metrics m;
    m.iae = iae;
    m.ise = ise;
    m.itae = itae;
    m.rmse = std::sqrt(sum_e2 / static_cast<double>(N));
    m.max_abs_err = max_abs_err;
    return m;
}

int main()
{
    // Example usage with synthetic data
    std::vector<double> t;
    std::vector<double> r;
    std::vector<double> y;
    const std::size_t N = 1001;
    const double T = 10.0;
    t.reserve(N);
    r.reserve(N);
    y.reserve(N);

    for (std::size_t k = 0; k < N; ++k) {
        double tk = T * static_cast<double>(k) / static_cast<double>(N - 1);
        t.push_back(tk);
        r.push_back(1.0);
        double yk = 1.0 - std::exp(-tk) * std::cos(2.0 * tk);
        y.push_back(yk);
    }

    Metrics m = compute_metrics(t, r, y);
    std::cout << "RMSE = " << m.rmse << std::endl;
    std::cout << "max |e| = " << m.max_abs_err << std::endl;
    return 0;
}
      

In a ROS2 node, the vectors t, r, and y would typically be filled from incoming messages or recorded in a log during each experiment.

Although less common than C++ and Python in robotics, Java can be used for offline analysis tools, for example in Android-based robots or graphical user interfaces. The following example computes RMSE and maximum error given arrays of samples:

public final class Metrics {

    public static class Result {
        public double iae;
        public double ise;
        public double itae;
        public double rmse;
        public double maxAbsErr;
    }

    public static Result compute(double[] t, double[] r, double[] y) {
        if (t.length == 0 || r.length != t.length || y.length != t.length) {
            throw new IllegalArgumentException("Arrays must have same nonzero length.");
        }

        int N = t.length;
        double dt = (N > 1) ? (t[1] - t[0]) : 0.0;

        double iae = 0.0;
        double ise = 0.0;
        double itae = 0.0;
        double sumE2 = 0.0;
        double maxAbsErr = 0.0;

        for (int k = 0; k < N; ++k) {
            double e = r[k] - y[k];
            double ae = Math.abs(e);

            iae += ae * dt;
            ise += e * e * dt;
            itae += t[k] * ae * dt;
            sumE2 += e * e;

            if (ae > maxAbsErr) {
                maxAbsErr = ae;
            }
        }

        Result res = new Result();
        res.iae = iae;
        res.ise = ise;
        res.itae = itae;
        res.rmse = Math.sqrt(sumE2 / (double) N);
        res.maxAbsErr = maxAbsErr;
        return res;
    }

    public static void main(String[] args) {
        int N = 1001;
        double T = 10.0;
        double[] t = new double[N];
        double[] r = new double[N];
        double[] y = new double[N];

        for (int k = 0; k < N; ++k) {
            double tk = T * (double) k / (double) (N - 1);
            t[k] = tk;
            r[k] = 1.0;
            y[k] = 1.0 - Math.exp(-tk) * Math.cos(2.0 * tk);
        }

        Result res = compute(t, r, y);
        System.out.println("RMSE = " + res.rmse);
        System.out.println("max |e| = " + res.maxAbsErr);
    }
}
      

The same logic can be integrated into a Java-based dashboard that reads log files and summarizes robot performance after each experiment.

In a linear control course you have already encountered MATLAB and Simulink. For your capstone, you can reuse these tools to prototype and validate your controller before implementing it on the robot. The following MATLAB script computes IAE, ISE, ITAE, RMSE, and maximum error from simulation data:

% Time vector and signals (for example, exported from Simulink)
T = 10.0;
N = 1001;
t = linspace(0.0, T, N);
r = ones(size(t));                         % step reference
y = 1.0 - exp(-t) .* cos(2.0 * t);        % example response
e = r - y;

dt = t(2) - t(1);

IAE  = sum(abs(e))      * dt;
ISE  = sum(e.^2)        * dt;
ITAE = sum(t .* abs(e)) * dt;
RMSE = sqrt(mean(e.^2));
maxErr = max(abs(e));

fprintf('IAE  = %.4f\n', IAE);
fprintf('ISE  = %.4f\n', ISE);
fprintf('ITAE = %.4f\n', ITAE);
fprintf('RMSE = %.4f\n', RMSE);
fprintf('max |e| = %.4f\n', maxErr);

% For multiple runs, store metrics in a vector and compute statistics
numRuns = 10;
rmseRuns = zeros(numRuns,1);
for i = 1:numRuns
    noise = 0.01 * randn(size(e));
    ei = e + noise;
    rmseRuns(i) = sqrt(mean(ei.^2));
end

rmseMean = mean(rmseRuns);
rmseStd  = std(rmseRuns, 1);  % population std

fprintf('Mean RMSE over runs = %.4f\n', rmseMean);
fprintf('Std of RMSE         = %.4f\n', rmseStd);
      

In Simulink, you can compute such indices either by exporting signals to the workspace or by using dedicated blocks that integrate error signals over time.

In this lesson you learned how to formalize validation of your capstone robot using quantitative performance metrics. Starting from the tracking error \(e(t)\), we defined classical indices such as IAE, ISE, ITAE, RMSE, and maximum error, and extended these ideas to task-level metrics like path-tracking accuracy, completion time, and success probability. We treated metrics as random variables over repeated experiments and introduced basic statistical tools (sample mean, variance, confidence intervals) for reporting performance with uncertainty.

You also saw how the continuous-time indices known from linear control are approximated by discrete sums over sampled data, and how to use the same formulas consistently in simulation and on hardware. Finally, you implemented these metrics in Python, C++, Java, and MATLAB, providing a practical toolkit that you can integrate directly into your capstone project to produce rigorous validation plots and tables.