Chapter 10: Robot Computing and Embedded Systems

Lesson 6: System Bring-Up and Debugging Basics

This lesson provides a rigorous, step-by-step methodology for powering on a robot system for the first time, validating subsystems, and debugging failures with a principled approach. We connect practical bring-up procedures to mathematical models of timing, sampling, communication integrity, and fault detection. By the end, you will be able to design a reproducible bring-up checklist, analyze real-time and I/O correctness, and implement minimal diagnostic firmware and host-side test tools.

1. Conceptual Overview: What “Bring-Up” Means

System bring-up is the controlled process of turning a newly assembled robot on, verifying that power, compute, sensing, actuation, and communication subsystems behave within specifications, and establishing a baseline for future debugging. Unlike routine debugging of a mature platform, bring-up assumes unknown unknowns: wiring mistakes, swapped devices, firmware misconfiguration, timing violations, and electrical noise.

We model the robot platform as interconnected layers: (i) Power/electrical, (ii) Compute/RT firmware, (iii) I/O (sensors/actuators), (iv) Communication links, (v) Application behaviors. Bring-up proceeds bottom-up because higher layers depend on lower layers being correct.

flowchart TD
  S["Start: new robot hardware"] --> P["Power checks + current limit"]
  P --> C["Compute boots?"]
  C -->|no| D1["Debug boot/clock/reset"]
  C -->|yes| IO["I/O validation \n(sensors, actuators)"]
  IO -->|fail| D2["Isolate device + minimal test"]
  IO -->|pass| COM["Comms validation \n(UART/I2C/SPI/CAN/EtherCAT)"]
  COM -->|fail| D3["Check physical + protocol layers"]
  COM -->|pass| APP["Run minimal app + safety stop"]
  APP --> LOG["Log baseline + freeze configuration"]
        

A key design goal is reproducibility. Let a bring-up procedure be a sequence of tests \( T_1, T_2, \dots, T_m \). Each test outputs a pass/fail random variable \( X_i \in \{0,1\} \). We want the empirical pass probability \( \hat{p}_i = \frac{1}{N}\sum_{k=1}^N X_i^{(k)} \) to be close to 1 under correct hardware/firmware. If not, the fault is localized to the earliest failing stage.

2. Power-On and Boot Correctness

From Lesson 5, you know how power management and interlocks are designed. Here we formalize the first bring-up checks. Let \( V(t) \) be the supply voltage and \( I(t) \) the current draw. A stable boot requires:

\[ \forall t \in [0, t_b]:\quad V_{\min} \le V(t) \le V_{\max}, \qquad 0 \le I(t) \le I_{\text{limit}}. \]

We interpret deviations using basic circuit laws. Suppose a rail supplies a load with equivalent resistance \( R_L \). Then Ohm’s law gives instantaneous current: \( I(t) = \frac{V(t)}{R_L} \). If during boot we measure \( I(t) > I_{\text{limit}} \), then

\[ R_L < \frac{V(t)}{I_{\text{limit}}} \quad \Rightarrow \quad \text{possible short, wrong component, or wiring fault.} \]

Boot-time constraint. If the MCU clock is \( f_c \) and the boot firmware executes \( N_b \) cycles before enabling safety outputs, the theoretical minimum boot time is \( t_b^{\min} = \frac{N_b}{f_c} \). A measured boot time \( t_b \) much larger than this indicates waiting on peripherals (e.g., bus timeouts) or clock misconfiguration.

We can model boot variability as \( t_b = t_b^{\min} + \sum_{j=1}^q \delta_j \), where each \( \delta_j \ge 0 \) is a wait or retry delay. A bring-up log should estimate \( \delta_j \) by timestamping milestones.

3. Timing Verification, Jitter, and Watchdogs

Real-time firmware (Lesson 2) must meet timing constraints. Consider a periodic control/IO loop with nominal period \( T \). Let actual loop start times be \( t_k \), and define jitter: \( J_k = t_{k+1} - t_k - T \). We require bounded jitter: \( |J_k| \le J_{\max} \).

If worst-case execution time (WCET) is \( C \), schedulability requires

\[ C \le T - J_{\max}. \]

Proof sketch. Each iteration must finish before the next starts. The shortest time available is a nominal period reduced by maximal late start: \( T - J_{\max} \). If \( C \) exceeds that, overlap occurs, violating determinism. Hence the inequality is necessary.

Watchdog sizing. A watchdog resets the system if no “kick” occurs within timeout \( T_w \). If kicks are issued once per loop, safety requires

\[ T_w \ge T + J_{\max} + \epsilon, \]

where \( \epsilon \) is a margin for interrupt latency. Too small \( T_w \) causes false resets; too large delays fault recovery.

4. I/O Bring-Up: Sensor Sanity and Actuator Safety

A new robot must validate each sensor and actuator in isolation. Use minimal excitation and hard safety limits.

Sensor quantization check. Suppose an ADC has \( n \) bits and reference \( V_{\text{ref}} \). The quantization step is \( \Delta = \frac{V_{\text{ref}}}{2^n} \). If the true analog signal is \( v(t) \), the digital reading is \( \hat{v}(t) = Q(v(t)) = v(t) + e_q(t) \), where quantization error satisfies \( |e_q(t)| \le \frac{\Delta}{2} \).

In bring-up, a drift-free stationary sensor should yield sample variance dominated by noise and quantization:

\[ \operatorname{Var}[\hat{v}] \approx \sigma_n^2 + \sigma_q^2, \qquad \sigma_q^2 = \frac{\Delta^2}{12}. \]

If the observed variance is much larger, check grounding, shielding, or incorrect sampling configuration.

Actuator step test (safe micro-step). Apply a small command \( u(t)=u_0 \mathbf{1}_{t\ge 0} \). If the actuator output (e.g., velocity) is roughly first-order with time constant \( \tau \) , then

\[ y(t) = K u_0\left(1 - e^{-t/\tau}\right), \quad t \ge 0. \]

Estimate \( \tau \) from data by solving \( y(t_{63\%}) = 0.63\, K u_0 \Rightarrow t_{63\%}\approx \tau \). Large deviations indicate incorrect polarity, saturation, or mechanical binding.

5. Communication Bring-Up and CRC Guarantees

Bring-up of UART/I2C/SPI/CAN links (Lesson 3) starts from physical layer, then protocol layer, then payload semantics. A mathematically grounded integrity tool is the cyclic redundancy check (CRC).

Represent a bitstring message \( \mathbf{m} \) as a polynomial over GF(2): \( M(x)=\sum_{i=0}^{L-1} m_i x^i \). Given generator polynomial \( G(x) \) of degree \( r \), the transmitted codeword is \( C(x)=x^r M(x) + R(x) \), where remainder \( R(x)=x^r M(x)\bmod G(x) \).

The receiver checks that \( C(x)\bmod G(x)=0 \). Let an error polynomial be \( E(x) \). The received polynomial is \( C'(x)=C(x)+E(x) \). An undetected error occurs iff \( E(x)\bmod G(x)=0 \iff G(x)\mid E(x). \)

Theorem (burst detection). If \( G(x) \) has degree \( r \), then all burst errors of length \( \ell \le r \) are detected.

Proof. A burst of length \( \ell \) has form \( E(x)=x^k B(x) \) where \( B(x) \) is a nonzero polynomial with degree \( \ell-1 \). If \( \ell \le r \), then \( \deg(B) < r = \deg(G) \), so \( G(x) \) cannot divide \( B(x) \). Multiplication by \( x^k \) does not change divisibility, hence \( G(x) \nmid E(x) \). Therefore the receiver detects the error. ∎

During bring-up, compute empirical bit error rate (BER) by sending \( N \) known frames and counting errors \( n_e \): \( \widehat{\text{BER}} = n_e / (N L) \). A sudden BER increase when actuators switch on suggests EMI coupling.

flowchart TD
  PHY["Physical layer check: wires/termination/levels"] --> SIG["Signal integrity: scope/logic analyzer"]
  SIG --> PROT["Protocol check: IDs, baud, timing"]
  PROT --> CRC["Payload integrity: CRC pass rate"]
  CRC --> SEM["Semantic check: units, scaling, ranges"]
  SEM --> OK["Link accepted baseline"]
        

6. Debugging Strategy and Instrumentation

Debugging is controlled hypothesis testing. Let \( H_0 \) be “subsystem is correct.” Each diagnostic test yields evidence \( D \). We reject \( H_0 \) if a test statistic exceeds a threshold.

Example: sensor range check. Let expected mean and variance be \( \mu_0 \), \( \sigma_0^2 \), and gather \( N \) samples \( \{x_i\} \). Use \( z = \frac{\bar{x}-\mu_0}{\sigma_0/\sqrt{N}} \) and flag fault if \( |z| > z_\alpha \).

Ring-buffer logging bound. Suppose logs arrive at rate \( \lambda \) entries/s, buffer size \( B \) entries, and dump interval \( T_d \). No data loss requires

\[ \lambda T_d \le B. \]

This inequality guides how much you can log at runtime without overflow. Always log with timestamps and monotonically increasing sequence numbers.

Instrumentation ladder: (1) LEDs/GPIO toggles for phase markers, (2) Serial prints, (3) Logic analyzer / oscilloscope, (4) SWD/JTAG breakpoints, (5) Hardware-in-the-loop replay. Bring-up prefers (1–3) first because intrusive debugging can change timing.

7. Python Implementation: Host-Side Diagnostic Tool

A common bring-up pattern is a PC script that pings embedded nodes, requests sensor snapshots, and plots statistics. Below uses pyserial and numpy.


import serial
import struct
import time
import numpy as np

PORT = "/dev/ttyUSB0"
BAUD = 115200

# Simple packet: [0xAA][cmd][len][payload...][crc16]
def crc16_ccitt(data: bytes, poly=0x1021, init=0xFFFF):
    crc = init
    for b in data:
        crc ^= (b << 8)
        for _ in range(8):
            if crc & 0x8000:
                crc = ((crc << 1) ^ poly) & 0xFFFF
            else:
                crc = (crc << 1) & 0xFFFF
    return crc

def make_pkt(cmd, payload=b""):
    header = bytes([0xAA, cmd, len(payload)])
    body = header + payload
    crc = crc16_ccitt(body)
    return body + struct.pack(">H", crc)

def read_pkt(ser):
    # naive sync on 0xAA
    while True:
        b = ser.read(1)
        if not b:
            return None
        if b[0] == 0xAA:
            break
    cmd = ser.read(1)[0]
    ln  = ser.read(1)[0]
    payload = ser.read(ln)
    crc_rx = struct.unpack(">H", ser.read(2))[0]
    crc_ok = (crc_rx == crc16_ccitt(bytes([0xAA, cmd, ln]) + payload))
    return cmd, payload, crc_ok

with serial.Serial(PORT, BAUD, timeout=0.2) as ser:
    # 1) ping
    ser.write(make_pkt(0x01))
    resp = read_pkt(ser)
    assert resp and resp[2], "Ping CRC failed"
    print("Ping OK")

    # 2) read 200 sensor samples
    samples = []
    for _ in range(200):
        ser.write(make_pkt(0x10))
        cmd, payload, ok = read_pkt(ser)
        assert ok and cmd == 0x10
        x = struct.unpack(">f", payload)[0]
        samples.append(x)
        time.sleep(0.01)

    samples = np.array(samples)
    print("mean =", samples.mean(), "std =", samples.std())

    # z-score fault check with expected bounds
    mu0, sigma0 = 0.0, 0.02
    z = (samples.mean() - mu0) / (sigma0 / np.sqrt(len(samples)))
    print("z =", z)
    if abs(z) > 3.0:
        print("WARNING: sensor bias/drift suspected")
      

This script embodies bring-up principles: synchronized framing, CRC verification, repeated sampling, and statistical sanity checks.

8. C++ Implementation: Minimal Embedded Self-Test

On the MCU, implement a tiny “safe bring-up” firmware: toggle a GPIO heartbeat, verify a sensor, and expose a ping/snapshot protocol.


// Bare-metal style pseudo-C++ (platform-specific HAL omitted)
#include <cstdint>
#include <cmath>

static volatile uint32_t ms_ticks = 0;

extern "C" void SysTick_Handler() { ms_ticks++; }

uint32_t millis() { return ms_ticks; }

// Heartbeat pin toggle every T ms
void heartbeat_task(uint32_t T_ms) {
    static uint32_t last = 0;
    if (millis() - last >= T_ms) {
        last = millis();
        gpio_toggle(LED1);
    }
}

// First-order low-pass filter: y_k = a y_{k-1} + (1-a) x_k
struct LPF {
    float a, y;
    LPF(float tau, float Ts): y(0.0f) {
        a = std::exp(-Ts / tau);
    }
    float step(float x) {
        y = a * y + (1.0f - a) * x;
        return y;
    }
};

int main() {
    clock_init();
    systick_init(1000); // 1 kHz tick
    uart_init(115200);

    LPF filt(0.05f, 0.001f); // tau=50ms, Ts=1ms

    while (1) {
        heartbeat_task(200);

        float raw = adc_read_channel(0);
        float v = filt.step(raw);

        // Sanity: expected range [0.2, 3.0] volts
        if (v < 0.2f || v > 3.0f) {
            fault_flag_set(SENSOR_RANGE_FAULT);
        }

        protocol_poll(); // handles ping and snapshot commands

        watchdog_kick();
    }
}
      

The low-pass filter coefficient matches Section 4: \( a = e^{-T_s/\tau} \), linking firmware constants to physical timing.

9. Java Implementation: Simple Serial Diagnostic Client

Java is often used for offboard GUIs or factory tests. Below uses the cross-platform jSerialComm library.


import com.fazecast.jSerialComm.SerialPort;
import java.nio.ByteBuffer;
import java.util.ArrayList;

public class BringUpClient {
    static int crc16(byte[] data) {
        int poly = 0x1021, crc = 0xFFFF;
        for (byte b : data) {
            crc ^= (b & 0xFF) << 8;
            for (int i = 0; i < 8; i++) {
                if ((crc & 0x8000) != 0) crc = ((crc << 1) ^ poly) & 0xFFFF;
                else crc = (crc << 1) & 0xFFFF;
            }
        }
        return crc;
    }

    static byte[] makePkt(int cmd, byte[] payload) {
        int len = payload.length;
        byte[] hdr = new byte[]{(byte)0xAA, (byte)cmd, (byte)len};
        byte[] body = new byte[3 + len];
        System.arraycopy(hdr, 0, body, 0, 3);
        System.arraycopy(payload, 0, body, 3, len);

        int crc = crc16(body);
        ByteBuffer bb = ByteBuffer.allocate(body.length + 2);
        bb.put(body);
        bb.putShort((short)crc);
        return bb.array();
    }

    public static void main(String[] args) throws Exception {
        SerialPort sp = SerialPort.getCommPort("/dev/ttyUSB0");
        sp.setBaudRate(115200);
        if (!sp.openPort()) throw new RuntimeException("Port open failed");

        // Ping
        sp.writeBytes(makePkt(0x01, new byte[]{}), 5);

        ArrayList<Float> xs = new ArrayList<>();
        for (int i = 0; i < 200; i++) {
            sp.writeBytes(makePkt(0x10, new byte[]{}), 5);
            byte[] buf = new byte[3 + 4 + 2];
            sp.readBytes(buf, buf.length);

            // Parse float payload (big-endian)
            float x = ByteBuffer.wrap(buf, 3, 4).getFloat();
            xs.add(x);
            Thread.sleep(10);
        }

        double mean = xs.stream().mapToDouble(f -> f).average().orElse(0.0);
        System.out.println("mean=" + mean);

        sp.closePort();
    }
}
      

This mirrors the Python test but can be embedded into a larger test framework.

10. MATLAB / Simulink: Log-Based Bring-Up Verification

MATLAB is ideal for quick statistical validation of logged bring-up data. Suppose your MCU streams samples \( x_k \). We estimate bias and variance, then compare against expected specs.


% samples.csv contains a column x
x = readmatrix("samples.csv");
N = length(x);

mu_hat = mean(x);
sigma_hat = std(x);

% expected values
mu0 = 0.0;
sigma0 = 0.02;

% z-score for bias test
z = (mu_hat - mu0) / (sigma0 / sqrt(N));

fprintf("mu_hat=%f, sigma_hat=%f, z=%f\n", mu_hat, sigma_hat, z);

if abs(z) > 3
    warning("Bias suspected beyond 3-sigma bound.");
end

% Simple step-response fit: y(t)=K(1-exp(-t/tau))
t = (0:N-1)' * 0.01;  % Ts=10ms
y = x;
K_hat = max(y);
idx63 = find(y >= 0.63*K_hat, 1);
tau_hat = t(idx63);

fprintf("Estimated tau ~ %f s\n", tau_hat);
      

Simulink hint (no diagram): Build a model with blocks: “From Workspace” (test input), “Discrete Transfer Fcn” to represent first-order actuator dynamics, and “Scope” to overlay measured vs simulated output during bring-up.

11. Problems and Solutions

Problem 1 (Boot-Time Budget): A microcontroller runs at \( f_c = 120\,\text{MHz} \). Before enabling motor drivers, the boot code executes \( N_b = 9.6 \times 10^6 \) cycles plus waits on three peripherals, each contributing delays \( \delta_1=4\,\text{ms} \), \( \delta_2=7\,\text{ms} \), \( \delta_3=5\,\text{ms} \). Compute theoretical and observed boot times.

Solution:

\[ t_b^{\min} = \frac{N_b}{f_c} = \frac{9.6\times 10^6}{120\times 10^6} = 0.08\,\text{s} = 80\,\text{ms}. \]

\[ t_b = t_b^{\min} + \sum_{j=1}^3 \delta_j = 80\,\text{ms} + (4+7+5)\,\text{ms} = 96\,\text{ms}. \]

An observed boot time significantly larger than 96 ms indicates extra hidden waits or clock mismatch.

Problem 2 (Jitter and Schedulability): A periodic loop has nominal period \( T=5\,\text{ms} \). Measured jitter bound is \( J_{\max}=0.6\,\text{ms} \). (a) What is the maximum allowable WCET \( C \)? (b) If the loop WCET is 4.7 ms, is it schedulable?

Solution:

\[ C_{\max} = T - J_{\max} = 5 - 0.6 = 4.4\,\text{ms}. \]

(b) Since \( 4.7\,\text{ms} > 4.4\,\text{ms} \), the loop is not schedulable, so timing overrun faults may occur.

Problem 3 (ADC Variance Sanity): A sensor is read by a 12-bit ADC with \( V_{\text{ref}}=3.3\,\text{V} \). The analog noise standard deviation is \( \sigma_n = 2\,\text{mV} \). Estimate expected digital variance and compare to an observed \( \operatorname{Var}[\hat{v}] = 5.0\times 10^{-5}\,\text{V}^2 \).

Solution:

\[ \Delta = \frac{3.3}{2^{12}} \approx 8.06\times 10^{-4}\,\text{V}. \quad \sigma_q^2 = \frac{\Delta^2}{12} \approx \frac{(8.06\times 10^{-4})^2}{12} \approx 5.41\times 10^{-8}. \]

\[ \sigma_n^2 = (2\times 10^{-3})^2 = 4\times 10^{-6}. \]

\[ \operatorname{Var}[\hat{v}]_{\text{exp}} \approx 4\times 10^{-6} + 5.41\times 10^{-8} \approx 4.05\times 10^{-6}\,\text{V}^2. \]

Observed variance \( 5.0\times 10^{-5} \) is about 12× larger, suggesting EMI, grounding issues, or sampling misconfiguration.

Problem 4 (CRC Burst Detection): A communication bus uses a CRC generator polynomial of degree \( r=8 \). What is the maximum burst error length guaranteed to be detected?

Solution:

From Section 5, all bursts with \( \ell \le r \) are detected. Thus any burst of length up to 8 bits is guaranteed detected.

12. Summary

We formalized robot bring-up as a bottom-up, reproducible validation pipeline. Using mathematical models, we analyzed power integrity, boot-time budgets, real-time jitter constraints, watchdog sizing, I/O quantization and step tests, and CRC-based communication guarantees. Practical debugging was tied to statistical hypothesis checks and logging bounds, and we implemented minimal host-side and embedded diagnostic tools in Python, C++, Java, and MATLAB/Simulink.

13. References

  1. Kopetz, H. (1997). Real-Time Systems: Design Principles for Distributed Embedded Applications. Addison-Wesley. (Foundational timing and determinism theory.)
  2. Buttazzo, G.C. (2005). Rate monotonic vs. EDF: Judgment day. Real-Time Systems, 29(1), 5–26.
  3. Wakerly, J.F. (1976). Error detecting codes, self-checking circuits and applications. North-Holland Computer Science Press.
  4. Castagnoli, G., Brauer, S., & Herrmann, M. (1993). Optimization of cyclic redundancy-check codes. IEEE Transactions on Communications, 41(6), 883–892.
  5. Laprie, J.-C. (1992). Dependability: Basic concepts and terminology. Springer Lecture Notes in Computer Science, 564, 3–29.
  6. Lee, E.A. (2008). Cyber physical systems: Design challenges. 11th IEEE International Symposium on Object/Component/Service-Oriented Real-Time Distributed Computing, 363–369.
  7. Henzinger, T.A. (2006). The theory of hybrid automata. Proceedings of the 11th Annual IEEE Symposium on Logic in Computer Science, 278–292.