Chapter 16: Robot System Design Process

Lesson 4: Prototyping Cycles and Testing

This lesson presents a rigorous view of prototyping cycles and testing in robot system design. We formalize iterative improvement as a discrete-time process, define quantitative test metrics, and relate prototype evaluation to familiar concepts from linear control. Simple software and hardware testing harnesses are illustrated in Python, C++, Java, and MATLAB/Simulink.

1. Conceptual Overview of Prototyping Cycles

A prototype is any early implementation of a robot subsystem or system created to explore design choices, validate requirements, or reduce uncertainty. For a robot with requirements vector \( \mathbf{r} \in \mathbb{R}^m \), we can think of a prototype at iteration \( k \) as having a performance vector \( \mathbf{p}_k \in \mathbb{R}^m \) measured by tests. The error vector is

\[ \mathbf{e}_k = \mathbf{p}_k - \mathbf{r}. \]

Prototyping cycles aim to drive \( \|\mathbf{e}_k\| \) below an acceptable threshold while respecting time and cost constraints. A typical design–build–test–learn loop is shown below.

flowchart TD
  R["Define requirements r"] --> C["Create concept & model"]
  C --> P["Build prototype k"]
  P --> T["Execute test plan"]
  T --> A["Analyze metrics e_k"]
  A --> D["Decide: iterate or freeze?"]
  D -->|iterate| U["Update design parameters"]
  U --> C
  D -->|freeze| F["Freeze design for next phase"]
        

This loop occurs at multiple levels: component (motor driver), subsystem (mobile base), and full robot. Mathematically, we can treat each loop as an iteration of a discrete-time map that updates design parameters from \( \boldsymbol{\theta}_k \) to \( \boldsymbol{\theta}_{k+1} \) based on test feedback.

2. A Simple Mathematical Model of Iterative Improvement

Let \( \mathbf{e}_k \in \mathbb{R}^m \) denote the requirement error after the \( k \)-th prototype. A simple linear model of iterative improvement is

\[ \mathbf{e}_{k+1} = \mathbf{A}\,\mathbf{e}_k + \mathbf{w}_k, \]

where \( \mathbf{A} \in \mathbb{R}^{m \times m} \) is a contraction matrix representing how much the team reduces error per iteration, and \( \mathbf{w}_k \) captures modeling errors, newly discovered issues, or measurement noise.

If we assume \( \mathbf{w}_k = \mathbf{0} \) for analysis, the recursion has solution

\[ \mathbf{e}_k = \mathbf{A}^k \mathbf{e}_0. \]

Using the induced 2-norm \( \|\cdot\|_2 \), we have

\[ \|\mathbf{e}_k\|_2 \leq \|\mathbf{A}^k\|_2 \,\|\mathbf{e}_0\|_2 \leq \|\mathbf{A}\|_2^k \,\|\mathbf{e}_0\|_2. \]

Thus a sufficient condition for convergence \( \lim_{k\to\infty} \mathbf{e}_k = \mathbf{0} \) is \( \|\mathbf{A}\|_2 < 1 \). A stronger result uses the spectral radius \( \rho(\mathbf{A}) \):

\[ \rho(\mathbf{A}) < 1 \;\Longrightarrow\; \mathbf{A}^k \to \mathbf{0}, \quad k\to\infty. \]

Proof sketch. The result follows from the spectral radius formula and Jordan decomposition: if all eigenvalues satisfy \( |\lambda_i| < 1 \), then each Jordan block \( \mathbf{J}_i^k \) decays to \( \mathbf{0} \), and hence so does \( \mathbf{A}^k \).

When disturbance terms are present, i.e. \( \mathbf{w}_k \neq \mathbf{0} \), we obtain

\[ \mathbf{e}_k = \mathbf{A}^k \mathbf{e}_0 + \sum_{j=0}^{k-1} \mathbf{A}^{k-1-j}\mathbf{w}_j. \]

If \( \rho(\mathbf{A}) < 1 \) and \( \|\mathbf{w}_k\|_2 \leq \bar{w} \), then \( \|\mathbf{e}_k\|_2 \) is bounded by a geometric series, which implies the prototyping process reaches an error “tube” of radius proportional to \( \bar{w} \). This formalizes the idea that beyond a certain iteration count, noise and uncertainty dominate further improvements.

3. Test Metrics and Acceptance Criteria

To evaluate prototypes rigorously, we define scalar metrics derived from time-domain or frequency-domain responses. For example, suppose a single-input, single-output (SISO) closed-loop robot subsystem (e.g., wheel speed control) has step response \( y(t) \) for reference \( r(t) \equiv 1 \).

Typical control-derived metrics include:

  • Steady-state error:

    \[ e_{\infty} = \lim_{t\to\infty} (r(t)-y(t)). \]

  • Percent overshoot:

    \[ M_p = 100 \cdot \frac{\max_{t \geq 0} y(t) - y_{\infty}}{y_{\infty}} \;\;[\%], \]

    where \( y_{\infty} = \lim_{t\to\infty} y(t) \).
  • Settling time (2% band):

    \[ T_s = \inf\left\{ t \geq 0 \;:\; |y(\tau)-y_{\infty}| \leq 0.02\,|y_{\infty}| \;\; \forall \tau \geq t \right\}. \]

For a prototype, we may define a composite cost

\[ J = w_1 |e_{\infty}| + w_2 M_p + w_3 T_s + w_4 C_{\text{energy}}, \]

with positive weights \( w_i > 0 \) chosen according to design priorities, and \( C_{\text{energy}} \) an energy consumption metric. A prototype \( k \) is acceptable if \( J_k \leq J_{\max} \) and all safety constraints are satisfied (e.g. torque, current, temperature limits).

Over multiple test runs, we can also consider statistical metrics. For a scalar test result \( z \) (e.g., measured settling time) observed over \( N \) independent trials \( z_1,\dots,z_N \), the empirical mean and variance are

\[ \bar{z} = \frac{1}{N}\sum_{i=1}^N z_i, \qquad s_z^2 = \frac{1}{N-1}\sum_{i=1}^N (z_i-\bar{z})^2. \]

4. Reliability and Pass/Fail Testing

Many robot tests are binary: a test passes (no fault, requirements met) or fails (fault, safety violation, or performance outside limits). Let the probability that a randomly selected operation cycle passes be \( p \). We approximate this using repeated tests.

Suppose we run \( N \) independent cycles and observe \( X \) passes. Under a Bernoulli model,

\[ X \sim \operatorname{Binomial}(N,p). \]

The maximum-likelihood estimator (MLE) of \( p \) is

\[ \hat{p} = \frac{X}{N}. \]

For large \( N \), an approximate \( 100(1-\alpha)\% \) confidence interval is

\[ \hat{p} \pm z_{1-\alpha/2} \sqrt{\frac{\hat{p}(1-\hat{p})}{N}}, \]

where \( z_{1-\alpha/2} \) is the quantile of the standard normal distribution. This allows us to answer questions like: “How many cycles must we run to be 95% confident that the pass probability is at least 0.99?”

Setting the lower confidence bound \( \hat{p} - z_{0.975}\sqrt{\hat{p}(1-\hat{p})/N} \geq p_{\min} \) and solving for \( N \) gives a conservative estimate of the required test count.

flowchart TD
  S["Specify required reliability p_min"] --> N["Choose test count N"]
  N --> R["Run N cycles, record pass/fail"]
  R --> E["Estimate p_hat = X/N"]
  E --> C["Compute confidence interval"]
  C --> D["Decide: sufficient or more testing"]
        

5. Bench Testing and Simulation-Based Testing

Before full-system tests, we typically use bench tests and simulation-based tests:

  • Bench tests use real hardware in a controlled environment, often with the mechanical load simplified or replaced (e.g. driving a wheel in the air).
  • Simulation-based tests use physics engines and models (Chapter 13) to explore conditions that are hard or unsafe to realize physically.

For a first-order plant representing a velocity loop,

\[ G(s) = \frac{K}{\tau s + 1}, \]

and a proportional controller

\[ C(s) = K_p, \]

the closed-loop transfer function (unity feedback) is

\[ T(s) = \frac{K_p K}{\tau s + 1 + K_p K}. \]

The step response is again first order with effective time constant

\[ \tau_{\text{eff}} = \frac{\tau}{1 + K_p K}, \]

and steady-state gain 1. Bench tests can estimate \( \tau_{\text{eff}} \) and compare against design targets. Simulation can predict how different values of \( K_p \) change \( T_s \) and \( e_{\infty} \) before hardware is modified.

6. Python Example — Simple Prototyping Test Harness

The following Python script constructs a discrete-time approximation of the first-order closed-loop system above and evaluates step response metrics for several candidate proportional gains \( K_p \). This mimics a simulation-based prototyping cycle.


import numpy as np

# Closed-loop first-order model:
# y_{k+1} = a * y_k + b * r_k, with r_k = 1 (unit step)
def simulate_step(Kp, K=1.0, tau=0.5, dt=0.01, T=5.0):
    a = np.exp(-dt * (1.0 + Kp * K) / tau)
    b = (1.0 - a)  # ensures steady-state gain 1
    n = int(T / dt)
    y = np.zeros(n)
    r = 1.0
    for k in range(n - 1):
        y[k+1] = a * y[k] + b * r
    t = np.linspace(0.0, T, n)
    return t, y

def metrics(t, y, tol=0.02):
    y_inf = y[-1]
    e_inf = 1.0 - y_inf
    Mp = 100.0 * (np.max(y) - y_inf) / max(y_inf, 1e-9)
    # settling time: first time from which we remain in 2% band
    idx = np.where(np.abs(y - y_inf) > tol * abs(y_inf))[0]
    Ts = t[idx[-1]] if idx.size > 0 else 0.0
    return e_inf, Mp, Ts

Kp_candidates = [0.5, 1.0, 2.0, 4.0]
for Kp in Kp_candidates:
    t, y = simulate_step(Kp)
    e_inf, Mp, Ts = metrics(t, y)
    print(f"Kp={Kp:.2f}: e_inf={e_inf:.3f}, Mp={Mp:.1f}%, Ts={Ts:.2f}s")
      

In a real robot project, this type of script would be paired with hardware-in-the-loop tests where simulate_step is replaced by calls to the actual robot controller via ROS topics or a low-level communication interface.

7. C++ Example — Embedded Test Loop Skeleton

The next code fragment sketches a firmware-style prototyping loop for testing a velocity controller. It assumes low-level functions to read encoders and send motor commands already exist from earlier hardware work.


#include <stdint.h>
#include <stdio.h>

// Hardware abstraction (stubs)
double read_velocity();               // rad/s
void set_motor_command(double u);     // normalized command [-1, 1]
void delay_ms(uint32_t ms);           // blocking delay

struct Metrics {
    double e_inf;
    double Ts;
};

Metrics run_step_test(double Kp, double v_ref, double tol, double T_max) {
    const double dt = 0.01; // 10 ms control period
    uint32_t steps = static_cast<uint32_t>(T_max / dt);
    double v = 0.0;
    double v_inf = v_ref;
    double t = 0.0;

    uint32_t last_out_of_band = 0;

    for (uint32_t k = 0; k < steps; ++k) {
        v = read_velocity();
        double e = v_ref - v;
        double u = Kp * e;
        if (u > 1.0) u = 1.0;
        if (u < -1.0) u = -1.0;
        set_motor_command(u);

        double err_band = (v_inf != 0.0) ?
                          (v - v_inf) / v_inf : 0.0;
        if (fabs(err_band) > tol) {
            last_out_of_band = k;
        }

        delay_ms(static_cast<uint32_t>(dt * 1000.0));
        t += dt;
    }

    Metrics m;
    m.e_inf = v_inf - v;
    m.Ts = last_out_of_band * 0.01;
    return m;
}

int main() {
    double Kp_list[] = {0.5, 1.0, 2.0};
    for (double Kp : Kp_list) {
        Metrics m = run_step_test(Kp, 5.0, 0.02, 5.0);
        printf("Kp=%.2f, e_inf=%.3f, Ts=%.2fs\n",
               Kp, m.e_inf, m.Ts);
    }
    return 0;
}
      

This style of code structures repeatable experiments, which is crucial when comparing successive prototypes or validating that a firmware change has not regressed system performance.

8. Java Example — High-Level Test Orchestration

Java is often used on higher-level control PCs or mobile devices that send commands to the robot and log data. The following skeleton coordinates multiple test runs and aggregates simple statistics.


public class PrototypeTester {

    public static class RunResult {
        public final double settlingTime;
        public final boolean pass;
        public RunResult(double Ts, boolean pass) {
            this.settlingTime = Ts;
            this.pass = pass;
        }
    }

    // Stub: would call into ROS, gRPC, or a custom protocol
    private static RunResult executeTest(double Kp) {
        // Send configuration to robot, start a step test, wait for log
        double TsMeasured = 1.2; // placeholder
        boolean pass = TsMeasured <= 1.5;
        return new RunResult(TsMeasured, pass);
    }

    public static void main(String[] args) {
        double[] KpValues = {0.5, 1.0, 2.0};
        int N = 10; // repetitions per gain

        for (double Kp : KpValues) {
            double sumTs = 0.0;
            int passes = 0;
            for (int i = 0; i < N; i++) {
                RunResult r = executeTest(Kp);
                sumTs += r.settlingTime;
                if (r.pass) {
                    passes++;
                }
            }
            double meanTs = sumTs / N;
            double pHat = (double) passes / N;
            System.out.printf("Kp=%.2f, mean Ts=%.3fs, pass rate=%.2f%n",
                              Kp, meanTs, pHat);
        }
    }
}
      

This pattern separates low-level control (often in C/C++ on a microcontroller) from high-level experimentation logic (Java, Python, etc.), which simplifies rapid prototyping and testing.

9. MATLAB/Simulink Example — Model-Based Prototyping

In MATLAB, the first-order closed-loop model can be prototyped symbolically and then simulated. Simulink can then be used to represent the same structure with blocks (Step, Gain, Transfer Fcn, Scope) for graphical tuning.


K  = 1.0;
tau = 0.5;

% Plant and proportional controller
s = tf('s');
G = K / (tau * s + 1);
Kp_values = [0.5 1.0 2.0 4.0];

for Kp = Kp_values
    C = Kp;
    T = feedback(C * G, 1);  % closed-loop transfer

    % Step response and metrics
    [y, t] = step(T, 5);
    y_inf = y(end);
    e_inf = 1 - y_inf;
    Mp = 100 * (max(y) - y_inf) / y_inf;

    idx = find(abs(y - y_inf) > 0.02 * abs(y_inf), 1, 'last');
    if isempty(idx)
        Ts = 0;
    else
        Ts = t(idx);
    end

    fprintf('Kp=%.2f, e_inf=%.3f, Mp=%.1f, Ts=%.2f\n', Kp, e_inf, Mp, Ts);
end

% Simulink notes:
% - Create a model with:
%   Step block --> Gain (Kp) --> Transfer Fcn (K / (tau s + 1)) --> Scope.
% - Use Simulation -> Model Explorer to sweep Kp and log response data.
      

This model-based approach supports early “virtual prototyping” before hardware exists, and the same controller component can later be auto-coded for embedded deployment in more advanced courses.

10. Problems and Solutions

Problem 1 (Error decay per prototype): Consider a scalar error model \( e_{k+1} = \alpha e_k \) with \( |\alpha| < 1 \) and initial error \( e_0 \). Derive a closed-form expression for \( e_k \). How many iterations \( k \) are needed to ensure \( |e_k| \leq \varepsilon \) for a given tolerance \( \varepsilon > 0 \)?

Solution:

The recursion is a geometric progression. By induction:

\[ e_k = \alpha^k e_0. \]

We require \( |\alpha|^k |e_0| \leq \varepsilon \), or

\[ |\alpha|^k \leq \frac{\varepsilon}{|e_0|}. \]

Taking natural logarithms (and using \( 0 < |\alpha| < 1 \) so that \( \ln|\alpha| < 0 \)):

\[ k \geq \frac{\ln(\varepsilon/|e_0|)}{\ln|\alpha|}. \]

Because the denominator is negative, the inequality direction reverses when dividing; we thus take the ceiling of the right-hand side to obtain the smallest integer \( k \).

Problem 2 (Error with constant disturbance): Now consider \( e_{k+1} = \alpha e_k + b \) with \( |\alpha| < 1 \) and constant disturbance \( b \). Derive \( e_k \) and its limit as \( k\to\infty \).

Solution:

This is a non-homogeneous linear difference equation. A particular solution is constant: \( e_k^{(p)} = e^{\star} \) such that \( e^{\star} = \alpha e^{\star} + b \), hence

\[ e^{\star} = \frac{b}{1-\alpha}. \]

The homogeneous solution is \( e_k^{(h)} = \alpha^k c \). The general solution is

\[ e_k = \alpha^k c + \frac{b}{1-\alpha}. \]

Using \( e_0 \) to determine \( c \):

\[ e_0 = c + \frac{b}{1-\alpha} \;\Rightarrow\; c = e_0 - \frac{b}{1-\alpha}. \]

Therefore

\[ e_k = \alpha^k\left(e_0 - \frac{b}{1-\alpha}\right) + \frac{b}{1-\alpha}. \]

As \( k\to\infty \), \( \alpha^k \to 0 \), so

\[ \lim_{k\to\infty} e_k = \frac{b}{1-\alpha}, \]

which is the steady-state error caused by the disturbance \( b \).

Problem 3 (Required number of reliability tests): A robot team requires that the probability of a successful operation cycle be at least \( p_{\min} = 0.99 \). They use the estimator \( \hat{p} = X/N \) with the normal-approximation confidence interval from Section 4. Find an approximate formula for the number of tests \( N \) required such that, if \( \hat{p} = p_{\min} \), the lower 95% confidence bound is at least 0.98.

Solution:

The lower bound is

\[ p_{\min} - z_{0.975} \sqrt{\frac{p_{\min}(1-p_{\min})}{N}} \geq 0.98. \]

Rearranging:

\[ z_{0.975} \sqrt{\frac{p_{\min}(1-p_{\min})}{N}} \leq p_{\min} - 0.98. \]

Squaring both sides:

\[ \frac{p_{\min}(1-p_{\min})}{N} \leq \frac{(p_{\min}-0.98)^2}{z_{0.975}^2}, \]

so

\[ N \geq \frac{p_{\min}(1-p_{\min})\,z_{0.975}^2}{(p_{\min}-0.98)^2}. \]

Substituting \( p_{\min}=0.99 \) and \( z_{0.975}\approx 1.96 \) yields a numerical lower bound for \( N \).

Problem 4 (Selecting gains via simulation): For the first-order closed-loop model of Section 5, assume \( K=1 \), \( \tau=0.5 \), and proportional control. Show that the time constant scales as \( \tau_{\text{eff}} = \tau/(1+K_p) \). If the requirement is \( T_s \leq 1 \) second with the 4 time-constant rule \( T_s \approx 4 \tau_{\text{eff}} \), find the minimum \( K_p \) satisfying the requirement.

Solution:

The transfer function in Section 5 gives

\[ T(s) = \frac{K_p}{\tau s + 1 + K_p}. \]

This is equivalent to

\[ T(s) = \frac{1}{\tau_{\text{eff}} s + 1}, \quad \tau_{\text{eff}} = \frac{\tau}{1+K_p}. \]

Using the approximation \( T_s \approx 4 \tau_{\text{eff}} \) and \( \tau = 0.5 \), the requirement \( T_s \leq 1 \) implies

\[ 4 \frac{0.5}{1+K_p} \leq 1 \;\Rightarrow\; \frac{2}{1+K_p} \leq 1 \;\Rightarrow\; 1+K_p \geq 2 \;\Rightarrow\; K_p \geq 1. \]

Thus any \( K_p \geq 1 \) should satisfy the settling-time requirement in this idealized model.

Problem 5 (Test matrix size): A mobile robot must be tested over three floor friction conditions (low, medium, high) and three payload levels (empty, half, full). The team wants to estimate the effect of both factors on settling time. How many combinations are in the full-factorial test matrix? Comment on how you would reduce the test count while still gaining useful insight.

Solution:

There are 3 levels for friction and 3 levels for payload, giving \( 3 \times 3 = 9 \) factor combinations in a full factorial. If each combination is repeated \( r \) times, the total number of runs is \( 9r \). To reduce test count, we could:

  • Test all three payloads at only two friction levels.
  • Use corner points (low/high friction and empty/full payload) plus a few center-point runs.
  • Leverage simulation to explore the full grid, and validate only a subset on hardware.

These strategies trade test count against resolution of interaction effects between friction and payload.

11. Summary

In this lesson we formalized the robot prototyping cycle as an iterative, discrete-time error-reduction process, characterized convergence properties of simple models, and connected test metrics to familiar control concepts such as steady-state error and settling time. We introduced reliability estimation for pass/fail testing and illustrated how simulation (Python, MATLAB/Simulink) and hardware test harnesses (C++, Java) support principled iteration on robot prototypes. These ideas underpin the transition from informal experimentation to systematic, repeatable testing that is essential in later stages of robot product development.

12. References

  1. Ljung, L. (1999). System identification: Theory for the user (2nd ed.). Prentice Hall.
  2. Astrom, K.J., & Murray, R.M. (2008). Feedback systems: An introduction for scientists and engineers. Princeton University Press.
  3. Ziegler, J.G., & Nichols, N.B. (1942). Optimum settings for automatic controllers. Transactions of the ASME, 64(11), 759–768.
  4. Broyden, C.G. (1965). A class of methods for solving nonlinear simultaneous equations. Mathematics of Computation, 19(92), 577–593.
  5. Henzinger, T.A. (1996). The theory of hybrid automata. Proceedings of the 11th Annual IEEE Symposium on Logic in Computer Science, 278–292.
  6. Clarke, E.M., Grumberg, O., & Peled, D. (1999). Model checking. MIT Press.
  7. Box, G.E.P., & Draper, N.R. (1959). A basis for the selection of a response surface design. Journal of the American Statistical Association, 54(287), 622–654.
  8. Wald, A. (1945). Sequential tests of statistical hypotheses. Annals of Mathematical Statistics, 16(2), 117–186.