Chapter 10: Robot Computing and Embedded Systems

Lesson 6: System Bring-Up and Debugging Basics

This lesson provides a rigorous, step-by-step methodology for powering on a robot system for the first time, validating subsystems, and debugging failures with a principled approach. We connect practical bring-up procedures to mathematical models of timing, sampling, communication integrity, and fault detection. By the end, you will be able to design a reproducible bring-up checklist, analyze real-time and I/O correctness, and implement minimal diagnostic firmware and host-side test tools.

1. Conceptual Overview: What “Bring-Up” Means

System bring-up is the controlled process of turning a newly assembled robot on, verifying that power, compute, sensing, actuation, and communication subsystems behave within specifications, and establishing a baseline for future debugging. Unlike routine debugging of a mature platform, bring-up assumes unknown unknowns: wiring mistakes, swapped devices, firmware misconfiguration, timing violations, and electrical noise.

We model the robot platform as interconnected layers: (i) Power/electrical, (ii) Compute/RT firmware, (iii) I/O (sensors/actuators), (iv) Communication links, (v) Application behaviors. Bring-up proceeds bottom-up because higher layers depend on lower layers being correct.

flowchart TD
  S["Start: new robot hardware"] --> P["Power checks + current limit"]
  P --> C["Compute boots?"]
  C -->|no| D1["Debug boot/clock/reset"]
  C -->|yes| IO["I/O validation \n(sensors, actuators)"]
  IO -->|fail| D2["Isolate device + minimal test"]
  IO -->|pass| COM["Comms validation \n(UART/I2C/SPI/CAN/EtherCAT)"]
  COM -->|fail| D3["Check physical + protocol layers"]
  COM -->|pass| APP["Run minimal app + safety stop"]
  APP --> LOG["Log baseline + freeze configuration"]
        

A key design goal is reproducibility. Let a bring-up procedure be a sequence of tests \( T_1, T_2, \dots, T_m \). Each test outputs a pass/fail random variable \( X_i \in \{0,1\} \). We want the empirical pass probability \( \hat{p}_i = \frac{1}{N}\sum_{k=1}^N X_i^{(k)} \) to be close to 1 under correct hardware/firmware. If not, the fault is localized to the earliest failing stage.

2. Power-On and Boot Correctness

From Lesson 5, you know how power management and interlocks are designed. Here we formalize the first bring-up checks. Let \( V(t) \) be the supply voltage and \( I(t) \) the current draw. A stable boot requires:

\[ \forall t \in [0, t_b]:\quad V_{\min} \le V(t) \le V_{\max}, \qquad 0 \le I(t) \le I_{\text{limit}}. \]

We interpret deviations using basic circuit laws. Suppose a rail supplies a load with equivalent resistance \( R_L \). Then Ohm’s law gives instantaneous current: \( I(t) = \frac{V(t)}{R_L} \). If during boot we measure \( I(t) > I_{\text{limit}} \), then

\[ R_L < \frac{V(t)}{I_{\text{limit}}} \quad \Rightarrow \quad \text{possible short, wrong component, or wiring fault.} \]

Boot-time constraint. If the MCU clock is \( f_c \) and the boot firmware executes \( N_b \) cycles before enabling safety outputs, the theoretical minimum boot time is \( t_b^{\min} = \frac{N_b}{f_c} \). A measured boot time \( t_b \) much larger than this indicates waiting on peripherals (e.g., bus timeouts) or clock misconfiguration.

We can model boot variability as \( t_b = t_b^{\min} + \sum_{j=1}^q \delta_j \), where each \( \delta_j \ge 0 \) is a wait or retry delay. A bring-up log should estimate \( \delta_j \) by timestamping milestones.

3. Timing Verification, Jitter, and Watchdogs

Real-time firmware (Lesson 2) must meet timing constraints. Consider a periodic control/IO loop with nominal period \( T \). Let actual loop start times be \( t_k \), and define jitter: \( J_k = t_{k+1} - t_k - T \). We require bounded jitter: \( |J_k| \le J_{\max} \).

If worst-case execution time (WCET) is \( C \), schedulability requires

\[ C \le T - J_{\max}. \]

Proof sketch. Each iteration must finish before the next starts. The shortest time available is a nominal period reduced by maximal late start: \( T - J_{\max} \). If \( C \) exceeds that, overlap occurs, violating determinism. Hence the inequality is necessary.

Watchdog sizing. A watchdog resets the system if no “kick” occurs within timeout \( T_w \). If kicks are issued once per loop, safety requires

\[ T_w \ge T + J_{\max} + \epsilon, \]

where \( \epsilon \) is a margin for interrupt latency. Too small \( T_w \) causes false resets; too large delays fault recovery.

4. I/O Bring-Up: Sensor Sanity and Actuator Safety

A new robot must validate each sensor and actuator in isolation. Use minimal excitation and hard safety limits.

Sensor quantization check. Suppose an ADC has \( n \) bits and reference \( V_{\text{ref}} \). The quantization step is \( \Delta = \frac{V_{\text{ref}}}{2^n} \). If the true analog signal is \( v(t) \), the digital reading is \( \hat{v}(t) = Q(v(t)) = v(t) + e_q(t) \), where quantization error satisfies \( |e_q(t)| \le \frac{\Delta}{2} \).

In bring-up, a drift-free stationary sensor should yield sample variance dominated by noise and quantization:

\[ \operatorname{Var}[\hat{v}] \approx \sigma_n^2 + \sigma_q^2, \qquad \sigma_q^2 = \frac{\Delta^2}{12}. \]

If the observed variance is much larger, check grounding, shielding, or incorrect sampling configuration.

Actuator step test (safe micro-step). Apply a small command \( u(t)=u_0 \mathbf{1}_{t\ge 0} \). If the actuator output (e.g., velocity) is roughly first-order with time constant \( \tau \) , then

\[ y(t) = K u_0\left(1 - e^{-t/\tau}\right), \quad t \ge 0. \]

Estimate \( \tau \) from data by solving \( y(t_{63\%}) = 0.63\, K u_0 \Rightarrow t_{63\%}\approx \tau \). Large deviations indicate incorrect polarity, saturation, or mechanical binding.

5. Communication Bring-Up and CRC Guarantees

Bring-up of UART/I2C/SPI/CAN links (Lesson 3) starts from physical layer, then protocol layer, then payload semantics. A mathematically grounded integrity tool is the cyclic redundancy check (CRC).

Represent a bitstring message \( \mathbf{m} \) as a polynomial over GF(2): \( M(x)=\sum_{i=0}^{L-1} m_i x^i \). Given generator polynomial \( G(x) \) of degree \( r \), the transmitted codeword is \( C(x)=x^r M(x) + R(x) \), where remainder \( R(x)=x^r M(x)\bmod G(x) \).

The receiver checks that \( C(x)\bmod G(x)=0 \). Let an error polynomial be \( E(x) \). The received polynomial is \( C'(x)=C(x)+E(x) \). An undetected error occurs iff \( E(x)\bmod G(x)=0 \iff G(x)\mid E(x). \)

Theorem (burst detection). If \( G(x) \) has degree \( r \), then all burst errors of length \( \ell \le r \) are detected.

Proof. A burst of length \( \ell \) has form \( E(x)=x^k B(x) \) where \( B(x) \) is a nonzero polynomial with degree \( \ell-1 \). If \( \ell \le r \), then \( \deg(B) < r = \deg(G) \), so \( G(x) \) cannot divide \( B(x) \). Multiplication by \( x^k \) does not change divisibility, hence \( G(x) \nmid E(x) \). Therefore the receiver detects the error. ∎

During bring-up, compute empirical bit error rate (BER) by sending \( N \) known frames and counting errors \( n_e \): \( \widehat{\text{BER}} = n_e / (N L) \). A sudden BER increase when actuators switch on suggests EMI coupling.

flowchart TD
  PHY["Physical layer check: wires/termination/levels"] --> SIG["Signal integrity: scope/logic analyzer"]
  SIG --> PROT["Protocol check: IDs, baud, timing"]
  PROT --> CRC["Payload integrity: CRC pass rate"]
  CRC --> SEM["Semantic check: units, scaling, ranges"]
  SEM --> OK["Link accepted baseline"]
        

6. Debugging Strategy and Instrumentation

Debugging is controlled hypothesis testing. Let \( H_0 \) be “subsystem is correct.” Each diagnostic test yields evidence \( D \). We reject \( H_0 \) if a test statistic exceeds a threshold.

Example: sensor range check. Let expected mean and variance be \( \mu_0 \), \( \sigma_0^2 \), and gather \( N \) samples \( \{x_i\} \). Use \( z = \frac{\bar{x}-\mu_0}{\sigma_0/\sqrt{N}} \) and flag fault if \( |z| > z_\alpha \).

Ring-buffer logging bound. Suppose logs arrive at rate \( \lambda \) entries/s, buffer size \( B \) entries, and dump interval \( T_d \). No data loss requires

\[ \lambda T_d \le B. \]

This inequality guides how much you can log at runtime without overflow. Always log with timestamps and monotonically increasing sequence numbers.

Instrumentation ladder: (1) LEDs/GPIO toggles for phase markers, (2) Serial prints, (3) Logic analyzer / oscilloscope, (4) SWD/JTAG breakpoints, (5) Hardware-in-the-loop replay. Bring-up prefers (1–3) first because intrusive debugging can change timing.

7. Python Implementation: Host-Side Diagnostic Tool

A common bring-up pattern is a PC script that pings embedded nodes, requests sensor snapshots, and plots statistics. Below uses pyserial and numpy.


import serial
import struct
import time
import numpy as np

PORT = "/dev/ttyUSB0"
BAUD = 115200

# Simple packet: [0xAA][cmd][len][payload...][crc16]
def crc16_ccitt(data: bytes, poly=0x1021, init=0xFFFF):
    crc = init
    for b in data:
        crc ^= (b << 8)
        for _ in range(8):
            if crc & 0x8000:
                crc = ((crc << 1) ^ poly) & 0xFFFF
            else:
                crc = (crc << 1) & 0xFFFF
    return crc

def make_pkt(cmd, payload=b""):
    header = bytes([0xAA, cmd, len(payload)])
    body = header + payload
    crc = crc16_ccitt(body)
    return body + struct.pack(">H", crc)

def read_pkt(ser):
    # naive sync on 0xAA
    while True:
        b = ser.read(1)
        if not b:
            return None
        if b[0] == 0xAA:
            break
    cmd = ser.read(1)[0]
    ln  = ser.read(1)[0]
    payload = ser.read(ln)
    crc_rx = struct.unpack(">H", ser.read(2))[0]
    crc_ok = (crc_rx == crc16_ccitt(bytes([0xAA, cmd, ln]) + payload))
    return cmd, payload, crc_ok

with serial.Serial(PORT, BAUD, timeout=0.2) as ser:
    # 1) ping
    ser.write(make_pkt(0x01))
    resp = read_pkt(ser)
    assert resp and resp[2], "Ping CRC failed"
    print("Ping OK")

    # 2) read 200 sensor samples
    samples = []
    for _ in range(200):
        ser.write(make_pkt(0x10))
        cmd, payload, ok = read_pkt(ser)
        assert ok and cmd == 0x10
        x = struct.unpack(">f", payload)[0]
        samples.append(x)
        time.sleep(0.01)

    samples = np.array(samples)
    print("mean =", samples.mean(), "std =", samples.std())

    # z-score fault check with expected bounds
    mu0, sigma0 = 0.0, 0.02
    z = (samples.mean() - mu0) / (sigma0 / np.sqrt(len(samples)))
    print("z =", z)
    if abs(z) > 3.0:
        print("WARNING: sensor bias/drift suspected")
      

This script embodies bring-up principles: synchronized framing, CRC verification, repeated sampling, and statistical sanity checks.

8. C++ Implementation: Minimal Embedded Self-Test

On the MCU, implement a tiny “safe bring-up” firmware: toggle a GPIO heartbeat, verify a sensor, and expose a ping/snapshot protocol.


// Bare-metal style pseudo-C++ (platform-specific HAL omitted)
#include <cstdint>
#include <cmath>

static volatile uint32_t ms_ticks = 0;

extern "C" void SysTick_Handler() { ms_ticks++; }

uint32_t millis() { return ms_ticks; }

// Heartbeat pin toggle every T ms
void heartbeat_task(uint32_t T_ms) {
    static uint32_t last = 0;
    if (millis() - last >= T_ms) {
        last = millis();
        gpio_toggle(LED1);
    }
}

// First-order low-pass filter: y_k = a y_{k-1} + (1-a) x_k
struct LPF {
    float a, y;
    LPF(float tau, float Ts): y(0.0f) {
        a = std::exp(-Ts / tau);
    }
    float step(float x) {
        y = a * y + (1.0f - a) * x;
        return y;
    }
};

int main() {
    clock_init();
    systick_init(1000); // 1 kHz tick
    uart_init(115200);

    LPF filt(0.05f, 0.001f); // tau=50ms, Ts=1ms

    while (1) {
        heartbeat_task(200);

        float raw = adc_read_channel(0);
        float v = filt.step(raw);

        // Sanity: expected range [0.2, 3.0] volts
        if (v < 0.2f || v > 3.0f) {
            fault_flag_set(SENSOR_RANGE_FAULT);
        }

        protocol_poll(); // handles ping and snapshot commands

        watchdog_kick();
    }
}
      

The low-pass filter coefficient matches Section 4: \( a = e^{-T_s/\tau} \), linking firmware constants to physical timing.

9. Java Implementation: Simple Serial Diagnostic Client

Java is often used for offboard GUIs or factory tests. Below uses the cross-platform jSerialComm library.


import com.fazecast.jSerialComm.SerialPort;
import java.nio.ByteBuffer;
import java.util.ArrayList;

public class BringUpClient {
    static int crc16(byte[] data) {
        int poly = 0x1021, crc = 0xFFFF;
        for (byte b : data) {
            crc ^= (b & 0xFF) << 8;
            for (int i = 0; i < 8; i++) {
                if ((crc & 0x8000) != 0) crc = ((crc << 1) ^ poly) & 0xFFFF;
                else crc = (crc << 1) & 0xFFFF;
            }
        }
        return crc;
    }

    static byte[] makePkt(int cmd, byte[] payload) {
        int len = payload.length;
        byte[] hdr = new byte[]{(byte)0xAA, (byte)cmd, (byte)len};
        byte[] body = new byte[3 + len];
        System.arraycopy(hdr, 0, body, 0, 3);
        System.arraycopy(payload, 0, body, 3, len);

        int crc = crc16(body);
        ByteBuffer bb = ByteBuffer.allocate(body.length + 2);
        bb.put(body);
        bb.putShort((short)crc);
        return bb.array();
    }

    public static void main(String[] args) throws Exception {
        SerialPort sp = SerialPort.getCommPort("/dev/ttyUSB0");
        sp.setBaudRate(115200);
        if (!sp.openPort()) throw new RuntimeException("Port open failed");

        // Ping
        sp.writeBytes(makePkt(0x01, new byte[]{}), 5);

        ArrayList<Float> xs = new ArrayList<>();
        for (int i = 0; i < 200; i++) {
            sp.writeBytes(makePkt(0x10, new byte[]{}), 5);
            byte[] buf = new byte[3 + 4 + 2];
            sp.readBytes(buf, buf.length);

            // Parse float payload (big-endian)
            float x = ByteBuffer.wrap(buf, 3, 4).getFloat();
            xs.add(x);
            Thread.sleep(10);
        }

        double mean = xs.stream().mapToDouble(f -> f).average().orElse(0.0);
        System.out.println("mean=" + mean);

        sp.closePort();
    }
}
      

This mirrors the Python test but can be embedded into a larger test framework.

10. MATLAB / Simulink: Log-Based Bring-Up Verification

MATLAB is ideal for quick statistical validation of logged bring-up data. Suppose your MCU streams samples \( x_k \). We estimate bias and variance, then compare against expected specs.


% samples.csv contains a column x
x = readmatrix("samples.csv");
N = length(x);

mu_hat = mean(x);
sigma_hat = std(x);

% expected values
mu0 = 0.0;
sigma0 = 0.02;

% z-score for bias test
z = (mu_hat - mu0) / (sigma0 / sqrt(N));

fprintf("mu_hat=%f, sigma_hat=%f, z=%f\n", mu_hat, sigma_hat, z);

if abs(z) > 3
    warning("Bias suspected beyond 3-sigma bound.");
end

% Simple step-response fit: y(t)=K(1-exp(-t/tau))
t = (0:N-1)' * 0.01;  % Ts=10ms
y = x;
K_hat = max(y);
idx63 = find(y >= 0.63*K_hat, 1);
tau_hat = t(idx63);

fprintf("Estimated tau ~ %f s\n", tau_hat);
      

Simulink hint (no diagram): Build a model with blocks: “From Workspace” (test input), “Discrete Transfer Fcn” to represent first-order actuator dynamics, and “Scope” to overlay measured vs simulated output during bring-up.

11. Problems and Solutions

Problem 1 (Boot-Time Budget): A microcontroller runs at \( f_c = 120\,\text{MHz} \). Before enabling motor drivers, the boot code executes \( N_b = 9.6 \times 10^6 \) cycles plus waits on three peripherals, each contributing delays \( \delta_1=4\,\text{ms} \), \( \delta_2=7\,\text{ms} \), \( \delta_3=5\,\text{ms} \). Compute theoretical and observed boot times.

Solution:

\[ t_b^{\min} = \frac{N_b}{f_c} = \frac{9.6\times 10^6}{120\times 10^6} = 0.08\,\text{s} = 80\,\text{ms}. \]

\[ t_b = t_b^{\min} + \sum_{j=1}^3 \delta_j = 80\,\text{ms} + (4+7+5)\,\text{ms} = 96\,\text{ms}. \]

An observed boot time significantly larger than 96 ms indicates extra hidden waits or clock mismatch.

Problem 2 (Jitter and Schedulability): A periodic loop has nominal period \( T=5\,\text{ms} \). Measured jitter bound is \( J_{\max}=0.6\,\text{ms} \). (a) What is the maximum allowable WCET \( C \)? (b) If the loop WCET is 4.7 ms, is it schedulable?

Solution:

\[ C_{\max} = T - J_{\max} = 5 - 0.6 = 4.4\,\text{ms}. \]

(b) Since \( 4.7\,\text{ms} > 4.4\,\text{ms} \), the loop is not schedulable, so timing overrun faults may occur.

Problem 3 (ADC Variance Sanity): A sensor is read by a 12-bit ADC with \( V_{\text{ref}}=3.3\,\text{V} \). The analog noise standard deviation is \( \sigma_n = 2\,\text{mV} \). Estimate expected digital variance and compare to an observed \( \operatorname{Var}[\hat{v}] = 5.0\times 10^{-5}\,\text{V}^2 \).

Solution:

\[ \Delta = \frac{3.3}{2^{12}} \approx 8.06\times 10^{-4}\,\text{V}. \quad \sigma_q^2 = \frac{\Delta^2}{12} \approx \frac{(8.06\times 10^{-4})^2}{12} \approx 5.41\times 10^{-8}. \]

\[ \sigma_n^2 = (2\times 10^{-3})^2 = 4\times 10^{-6}. \]

\[ \operatorname{Var}[\hat{v}]_{\text{exp}} \approx 4\times 10^{-6} + 5.41\times 10^{-8} \approx 4.05\times 10^{-6}\,\text{V}^2. \]

Observed variance \( 5.0\times 10^{-5} \) is about 12× larger, suggesting EMI, grounding issues, or sampling misconfiguration.

Problem 4 (CRC Burst Detection): A communication bus uses a CRC generator polynomial of degree \( r=8 \). What is the maximum burst error length guaranteed to be detected?

Solution:

From Section 5, all bursts with \( \ell \le r \) are detected. Thus any burst of length up to 8 bits is guaranteed detected.

12. Summary

We formalized robot bring-up as a bottom-up, reproducible validation pipeline. Using mathematical models, we analyzed power integrity, boot-time budgets, real-time jitter constraints, watchdog sizing, I/O quantization and step tests, and CRC-based communication guarantees. Practical debugging was tied to statistical hypothesis checks and logging bounds, and we implemented minimal host-side and embedded diagnostic tools in Python, C++, Java, and MATLAB/Simulink.

13. References

  1. Kopetz, H. (1997). Real-Time Systems: Design Principles for Distributed Embedded Applications. Addison-Wesley. (Foundational timing and determinism theory.)
  2. Buttazzo, G.C. (2005). Rate monotonic vs. EDF: Judgment day. Real-Time Systems, 29(1), 5–26.
  3. Wakerly, J.F. (1976). Error detecting codes, self-checking circuits and applications. North-Holland Computer Science Press.
  4. Castagnoli, G., Brauer, S., & Herrmann, M. (1993). Optimization of cyclic redundancy-check codes. IEEE Transactions on Communications, 41(6), 883–892.
  5. Laprie, J.-C. (1992). Dependability: Basic concepts and terminology. Springer Lecture Notes in Computer Science, 564, 3–29.
  6. Lee, E.A. (2008). Cyber physical systems: Design challenges. 11th IEEE International Symposium on Object/Component/Service-Oriented Real-Time Distributed Computing, 363–369.
  7. Henzinger, T.A. (2006). The theory of hybrid automata. Proceedings of the 11th Annual IEEE Symposium on Logic in Computer Science, 278–292.
Support CaaT Academy

Help keep these engineering tutorials free and growing

If these lessons, examples, and project pages help you, a small donation supports the continued creation and improvement of free control, robotics, software, and engineering education resources.

Created and maintained by Abolfazl Mohammadijoo.