Chapter 10: Robot Computing and Embedded Systems
Lesson 6: System Bring-Up and Debugging Basics
This lesson provides a rigorous, step-by-step methodology for powering on a robot system for the first time, validating subsystems, and debugging failures with a principled approach. We connect practical bring-up procedures to mathematical models of timing, sampling, communication integrity, and fault detection. By the end, you will be able to design a reproducible bring-up checklist, analyze real-time and I/O correctness, and implement minimal diagnostic firmware and host-side test tools.
1. Conceptual Overview: What “Bring-Up” Means
System bring-up is the controlled process of turning a newly assembled robot on, verifying that power, compute, sensing, actuation, and communication subsystems behave within specifications, and establishing a baseline for future debugging. Unlike routine debugging of a mature platform, bring-up assumes unknown unknowns: wiring mistakes, swapped devices, firmware misconfiguration, timing violations, and electrical noise.
We model the robot platform as interconnected layers: (i) Power/electrical, (ii) Compute/RT firmware, (iii) I/O (sensors/actuators), (iv) Communication links, (v) Application behaviors. Bring-up proceeds bottom-up because higher layers depend on lower layers being correct.
flowchart TD
S["Start: new robot hardware"] --> P["Power checks + current limit"]
P --> C["Compute boots?"]
C -->|no| D1["Debug boot/clock/reset"]
C -->|yes| IO["I/O validation \n(sensors, actuators)"]
IO -->|fail| D2["Isolate device + minimal test"]
IO -->|pass| COM["Comms validation \n(UART/I2C/SPI/CAN/EtherCAT)"]
COM -->|fail| D3["Check physical + protocol layers"]
COM -->|pass| APP["Run minimal app + safety stop"]
APP --> LOG["Log baseline + freeze configuration"]
A key design goal is reproducibility. Let a bring-up procedure be a sequence of tests \( T_1, T_2, \dots, T_m \). Each test outputs a pass/fail random variable \( X_i \in \{0,1\} \). We want the empirical pass probability \( \hat{p}_i = \frac{1}{N}\sum_{k=1}^N X_i^{(k)} \) to be close to 1 under correct hardware/firmware. If not, the fault is localized to the earliest failing stage.
2. Power-On and Boot Correctness
From Lesson 5, you know how power management and interlocks are designed. Here we formalize the first bring-up checks. Let \( V(t) \) be the supply voltage and \( I(t) \) the current draw. A stable boot requires:
\[ \forall t \in [0, t_b]:\quad V_{\min} \le V(t) \le V_{\max}, \qquad 0 \le I(t) \le I_{\text{limit}}. \]
We interpret deviations using basic circuit laws. Suppose a rail supplies a load with equivalent resistance \( R_L \). Then Ohm’s law gives instantaneous current: \( I(t) = \frac{V(t)}{R_L} \). If during boot we measure \( I(t) > I_{\text{limit}} \), then
\[ R_L < \frac{V(t)}{I_{\text{limit}}} \quad \Rightarrow \quad \text{possible short, wrong component, or wiring fault.} \]
Boot-time constraint. If the MCU clock is \( f_c \) and the boot firmware executes \( N_b \) cycles before enabling safety outputs, the theoretical minimum boot time is \( t_b^{\min} = \frac{N_b}{f_c} \). A measured boot time \( t_b \) much larger than this indicates waiting on peripherals (e.g., bus timeouts) or clock misconfiguration.
We can model boot variability as \( t_b = t_b^{\min} + \sum_{j=1}^q \delta_j \), where each \( \delta_j \ge 0 \) is a wait or retry delay. A bring-up log should estimate \( \delta_j \) by timestamping milestones.
3. Timing Verification, Jitter, and Watchdogs
Real-time firmware (Lesson 2) must meet timing constraints. Consider a periodic control/IO loop with nominal period \( T \). Let actual loop start times be \( t_k \), and define jitter: \( J_k = t_{k+1} - t_k - T \). We require bounded jitter: \( |J_k| \le J_{\max} \).
If worst-case execution time (WCET) is \( C \), schedulability requires
\[ C \le T - J_{\max}. \]
Proof sketch. Each iteration must finish before the next starts. The shortest time available is a nominal period reduced by maximal late start: \( T - J_{\max} \). If \( C \) exceeds that, overlap occurs, violating determinism. Hence the inequality is necessary.
Watchdog sizing. A watchdog resets the system if no “kick” occurs within timeout \( T_w \). If kicks are issued once per loop, safety requires
\[ T_w \ge T + J_{\max} + \epsilon, \]
where \( \epsilon \) is a margin for interrupt latency. Too small \( T_w \) causes false resets; too large delays fault recovery.
4. I/O Bring-Up: Sensor Sanity and Actuator Safety
A new robot must validate each sensor and actuator in isolation. Use minimal excitation and hard safety limits.
Sensor quantization check. Suppose an ADC has \( n \) bits and reference \( V_{\text{ref}} \). The quantization step is \( \Delta = \frac{V_{\text{ref}}}{2^n} \). If the true analog signal is \( v(t) \), the digital reading is \( \hat{v}(t) = Q(v(t)) = v(t) + e_q(t) \), where quantization error satisfies \( |e_q(t)| \le \frac{\Delta}{2} \).
In bring-up, a drift-free stationary sensor should yield sample variance dominated by noise and quantization:
\[ \operatorname{Var}[\hat{v}] \approx \sigma_n^2 + \sigma_q^2, \qquad \sigma_q^2 = \frac{\Delta^2}{12}. \]
If the observed variance is much larger, check grounding, shielding, or incorrect sampling configuration.
Actuator step test (safe micro-step). Apply a small command \( u(t)=u_0 \mathbf{1}_{t\ge 0} \). If the actuator output (e.g., velocity) is roughly first-order with time constant \( \tau \) , then
\[ y(t) = K u_0\left(1 - e^{-t/\tau}\right), \quad t \ge 0. \]
Estimate \( \tau \) from data by solving \( y(t_{63\%}) = 0.63\, K u_0 \Rightarrow t_{63\%}\approx \tau \). Large deviations indicate incorrect polarity, saturation, or mechanical binding.
5. Communication Bring-Up and CRC Guarantees
Bring-up of UART/I2C/SPI/CAN links (Lesson 3) starts from physical layer, then protocol layer, then payload semantics. A mathematically grounded integrity tool is the cyclic redundancy check (CRC).
Represent a bitstring message \( \mathbf{m} \) as a polynomial over GF(2): \( M(x)=\sum_{i=0}^{L-1} m_i x^i \). Given generator polynomial \( G(x) \) of degree \( r \), the transmitted codeword is \( C(x)=x^r M(x) + R(x) \), where remainder \( R(x)=x^r M(x)\bmod G(x) \).
The receiver checks that \( C(x)\bmod G(x)=0 \). Let an error polynomial be \( E(x) \). The received polynomial is \( C'(x)=C(x)+E(x) \). An undetected error occurs iff \( E(x)\bmod G(x)=0 \iff G(x)\mid E(x). \)
Theorem (burst detection). If \( G(x) \) has degree \( r \), then all burst errors of length \( \ell \le r \) are detected.
Proof. A burst of length \( \ell \) has form \( E(x)=x^k B(x) \) where \( B(x) \) is a nonzero polynomial with degree \( \ell-1 \). If \( \ell \le r \), then \( \deg(B) < r = \deg(G) \), so \( G(x) \) cannot divide \( B(x) \). Multiplication by \( x^k \) does not change divisibility, hence \( G(x) \nmid E(x) \). Therefore the receiver detects the error. ∎
During bring-up, compute empirical bit error rate (BER) by sending \( N \) known frames and counting errors \( n_e \): \( \widehat{\text{BER}} = n_e / (N L) \). A sudden BER increase when actuators switch on suggests EMI coupling.
flowchart TD
PHY["Physical layer check: wires/termination/levels"] --> SIG["Signal integrity: scope/logic analyzer"]
SIG --> PROT["Protocol check: IDs, baud, timing"]
PROT --> CRC["Payload integrity: CRC pass rate"]
CRC --> SEM["Semantic check: units, scaling, ranges"]
SEM --> OK["Link accepted baseline"]
6. Debugging Strategy and Instrumentation
Debugging is controlled hypothesis testing. Let \( H_0 \) be “subsystem is correct.” Each diagnostic test yields evidence \( D \). We reject \( H_0 \) if a test statistic exceeds a threshold.
Example: sensor range check. Let expected mean and variance be \( \mu_0 \), \( \sigma_0^2 \), and gather \( N \) samples \( \{x_i\} \). Use \( z = \frac{\bar{x}-\mu_0}{\sigma_0/\sqrt{N}} \) and flag fault if \( |z| > z_\alpha \).
Ring-buffer logging bound. Suppose logs arrive at rate \( \lambda \) entries/s, buffer size \( B \) entries, and dump interval \( T_d \). No data loss requires
\[ \lambda T_d \le B. \]
This inequality guides how much you can log at runtime without overflow. Always log with timestamps and monotonically increasing sequence numbers.
Instrumentation ladder: (1) LEDs/GPIO toggles for phase markers, (2) Serial prints, (3) Logic analyzer / oscilloscope, (4) SWD/JTAG breakpoints, (5) Hardware-in-the-loop replay. Bring-up prefers (1–3) first because intrusive debugging can change timing.
7. Python Implementation: Host-Side Diagnostic Tool
A common bring-up pattern is a PC script that pings embedded nodes,
requests sensor snapshots, and plots statistics. Below uses
pyserial and numpy.
import serial
import struct
import time
import numpy as np
PORT = "/dev/ttyUSB0"
BAUD = 115200
# Simple packet: [0xAA][cmd][len][payload...][crc16]
def crc16_ccitt(data: bytes, poly=0x1021, init=0xFFFF):
crc = init
for b in data:
crc ^= (b << 8)
for _ in range(8):
if crc & 0x8000:
crc = ((crc << 1) ^ poly) & 0xFFFF
else:
crc = (crc << 1) & 0xFFFF
return crc
def make_pkt(cmd, payload=b""):
header = bytes([0xAA, cmd, len(payload)])
body = header + payload
crc = crc16_ccitt(body)
return body + struct.pack(">H", crc)
def read_pkt(ser):
# naive sync on 0xAA
while True:
b = ser.read(1)
if not b:
return None
if b[0] == 0xAA:
break
cmd = ser.read(1)[0]
ln = ser.read(1)[0]
payload = ser.read(ln)
crc_rx = struct.unpack(">H", ser.read(2))[0]
crc_ok = (crc_rx == crc16_ccitt(bytes([0xAA, cmd, ln]) + payload))
return cmd, payload, crc_ok
with serial.Serial(PORT, BAUD, timeout=0.2) as ser:
# 1) ping
ser.write(make_pkt(0x01))
resp = read_pkt(ser)
assert resp and resp[2], "Ping CRC failed"
print("Ping OK")
# 2) read 200 sensor samples
samples = []
for _ in range(200):
ser.write(make_pkt(0x10))
cmd, payload, ok = read_pkt(ser)
assert ok and cmd == 0x10
x = struct.unpack(">f", payload)[0]
samples.append(x)
time.sleep(0.01)
samples = np.array(samples)
print("mean =", samples.mean(), "std =", samples.std())
# z-score fault check with expected bounds
mu0, sigma0 = 0.0, 0.02
z = (samples.mean() - mu0) / (sigma0 / np.sqrt(len(samples)))
print("z =", z)
if abs(z) > 3.0:
print("WARNING: sensor bias/drift suspected")
This script embodies bring-up principles: synchronized framing, CRC verification, repeated sampling, and statistical sanity checks.
8. C++ Implementation: Minimal Embedded Self-Test
On the MCU, implement a tiny “safe bring-up” firmware: toggle a GPIO heartbeat, verify a sensor, and expose a ping/snapshot protocol.
// Bare-metal style pseudo-C++ (platform-specific HAL omitted)
#include <cstdint>
#include <cmath>
static volatile uint32_t ms_ticks = 0;
extern "C" void SysTick_Handler() { ms_ticks++; }
uint32_t millis() { return ms_ticks; }
// Heartbeat pin toggle every T ms
void heartbeat_task(uint32_t T_ms) {
static uint32_t last = 0;
if (millis() - last >= T_ms) {
last = millis();
gpio_toggle(LED1);
}
}
// First-order low-pass filter: y_k = a y_{k-1} + (1-a) x_k
struct LPF {
float a, y;
LPF(float tau, float Ts): y(0.0f) {
a = std::exp(-Ts / tau);
}
float step(float x) {
y = a * y + (1.0f - a) * x;
return y;
}
};
int main() {
clock_init();
systick_init(1000); // 1 kHz tick
uart_init(115200);
LPF filt(0.05f, 0.001f); // tau=50ms, Ts=1ms
while (1) {
heartbeat_task(200);
float raw = adc_read_channel(0);
float v = filt.step(raw);
// Sanity: expected range [0.2, 3.0] volts
if (v < 0.2f || v > 3.0f) {
fault_flag_set(SENSOR_RANGE_FAULT);
}
protocol_poll(); // handles ping and snapshot commands
watchdog_kick();
}
}
The low-pass filter coefficient matches Section 4: \( a = e^{-T_s/\tau} \), linking firmware constants to physical timing.
9. Java Implementation: Simple Serial Diagnostic Client
Java is often used for offboard GUIs or factory tests. Below uses the
cross-platform jSerialComm library.
import com.fazecast.jSerialComm.SerialPort;
import java.nio.ByteBuffer;
import java.util.ArrayList;
public class BringUpClient {
static int crc16(byte[] data) {
int poly = 0x1021, crc = 0xFFFF;
for (byte b : data) {
crc ^= (b & 0xFF) << 8;
for (int i = 0; i < 8; i++) {
if ((crc & 0x8000) != 0) crc = ((crc << 1) ^ poly) & 0xFFFF;
else crc = (crc << 1) & 0xFFFF;
}
}
return crc;
}
static byte[] makePkt(int cmd, byte[] payload) {
int len = payload.length;
byte[] hdr = new byte[]{(byte)0xAA, (byte)cmd, (byte)len};
byte[] body = new byte[3 + len];
System.arraycopy(hdr, 0, body, 0, 3);
System.arraycopy(payload, 0, body, 3, len);
int crc = crc16(body);
ByteBuffer bb = ByteBuffer.allocate(body.length + 2);
bb.put(body);
bb.putShort((short)crc);
return bb.array();
}
public static void main(String[] args) throws Exception {
SerialPort sp = SerialPort.getCommPort("/dev/ttyUSB0");
sp.setBaudRate(115200);
if (!sp.openPort()) throw new RuntimeException("Port open failed");
// Ping
sp.writeBytes(makePkt(0x01, new byte[]{}), 5);
ArrayList<Float> xs = new ArrayList<>();
for (int i = 0; i < 200; i++) {
sp.writeBytes(makePkt(0x10, new byte[]{}), 5);
byte[] buf = new byte[3 + 4 + 2];
sp.readBytes(buf, buf.length);
// Parse float payload (big-endian)
float x = ByteBuffer.wrap(buf, 3, 4).getFloat();
xs.add(x);
Thread.sleep(10);
}
double mean = xs.stream().mapToDouble(f -> f).average().orElse(0.0);
System.out.println("mean=" + mean);
sp.closePort();
}
}
This mirrors the Python test but can be embedded into a larger test framework.
10. MATLAB / Simulink: Log-Based Bring-Up Verification
MATLAB is ideal for quick statistical validation of logged bring-up data. Suppose your MCU streams samples \( x_k \). We estimate bias and variance, then compare against expected specs.
% samples.csv contains a column x
x = readmatrix("samples.csv");
N = length(x);
mu_hat = mean(x);
sigma_hat = std(x);
% expected values
mu0 = 0.0;
sigma0 = 0.02;
% z-score for bias test
z = (mu_hat - mu0) / (sigma0 / sqrt(N));
fprintf("mu_hat=%f, sigma_hat=%f, z=%f\n", mu_hat, sigma_hat, z);
if abs(z) > 3
warning("Bias suspected beyond 3-sigma bound.");
end
% Simple step-response fit: y(t)=K(1-exp(-t/tau))
t = (0:N-1)' * 0.01; % Ts=10ms
y = x;
K_hat = max(y);
idx63 = find(y >= 0.63*K_hat, 1);
tau_hat = t(idx63);
fprintf("Estimated tau ~ %f s\n", tau_hat);
Simulink hint (no diagram): Build a model with blocks: “From Workspace” (test input), “Discrete Transfer Fcn” to represent first-order actuator dynamics, and “Scope” to overlay measured vs simulated output during bring-up.
11. Problems and Solutions
Problem 1 (Boot-Time Budget): A microcontroller runs at \( f_c = 120\,\text{MHz} \). Before enabling motor drivers, the boot code executes \( N_b = 9.6 \times 10^6 \) cycles plus waits on three peripherals, each contributing delays \( \delta_1=4\,\text{ms} \), \( \delta_2=7\,\text{ms} \), \( \delta_3=5\,\text{ms} \). Compute theoretical and observed boot times.
Solution:
\[ t_b^{\min} = \frac{N_b}{f_c} = \frac{9.6\times 10^6}{120\times 10^6} = 0.08\,\text{s} = 80\,\text{ms}. \]
\[ t_b = t_b^{\min} + \sum_{j=1}^3 \delta_j = 80\,\text{ms} + (4+7+5)\,\text{ms} = 96\,\text{ms}. \]
An observed boot time significantly larger than 96 ms indicates extra hidden waits or clock mismatch.
Problem 2 (Jitter and Schedulability): A periodic loop has nominal period \( T=5\,\text{ms} \). Measured jitter bound is \( J_{\max}=0.6\,\text{ms} \). (a) What is the maximum allowable WCET \( C \)? (b) If the loop WCET is 4.7 ms, is it schedulable?
Solution:
\[ C_{\max} = T - J_{\max} = 5 - 0.6 = 4.4\,\text{ms}. \]
(b) Since \( 4.7\,\text{ms} > 4.4\,\text{ms} \), the loop is not schedulable, so timing overrun faults may occur.
Problem 3 (ADC Variance Sanity): A sensor is read by a 12-bit ADC with \( V_{\text{ref}}=3.3\,\text{V} \). The analog noise standard deviation is \( \sigma_n = 2\,\text{mV} \). Estimate expected digital variance and compare to an observed \( \operatorname{Var}[\hat{v}] = 5.0\times 10^{-5}\,\text{V}^2 \).
Solution:
\[ \Delta = \frac{3.3}{2^{12}} \approx 8.06\times 10^{-4}\,\text{V}. \quad \sigma_q^2 = \frac{\Delta^2}{12} \approx \frac{(8.06\times 10^{-4})^2}{12} \approx 5.41\times 10^{-8}. \]
\[ \sigma_n^2 = (2\times 10^{-3})^2 = 4\times 10^{-6}. \]
\[ \operatorname{Var}[\hat{v}]_{\text{exp}} \approx 4\times 10^{-6} + 5.41\times 10^{-8} \approx 4.05\times 10^{-6}\,\text{V}^2. \]
Observed variance \( 5.0\times 10^{-5} \) is about 12× larger, suggesting EMI, grounding issues, or sampling misconfiguration.
Problem 4 (CRC Burst Detection): A communication bus uses a CRC generator polynomial of degree \( r=8 \). What is the maximum burst error length guaranteed to be detected?
Solution:
From Section 5, all bursts with \( \ell \le r \) are detected. Thus any burst of length up to 8 bits is guaranteed detected.
12. Summary
We formalized robot bring-up as a bottom-up, reproducible validation pipeline. Using mathematical models, we analyzed power integrity, boot-time budgets, real-time jitter constraints, watchdog sizing, I/O quantization and step tests, and CRC-based communication guarantees. Practical debugging was tied to statistical hypothesis checks and logging bounds, and we implemented minimal host-side and embedded diagnostic tools in Python, C++, Java, and MATLAB/Simulink.
13. References
- Kopetz, H. (1997). Real-Time Systems: Design Principles for Distributed Embedded Applications. Addison-Wesley. (Foundational timing and determinism theory.)
- Buttazzo, G.C. (2005). Rate monotonic vs. EDF: Judgment day. Real-Time Systems, 29(1), 5–26.
- Wakerly, J.F. (1976). Error detecting codes, self-checking circuits and applications. North-Holland Computer Science Press.
- Castagnoli, G., Brauer, S., & Herrmann, M. (1993). Optimization of cyclic redundancy-check codes. IEEE Transactions on Communications, 41(6), 883–892.
- Laprie, J.-C. (1992). Dependability: Basic concepts and terminology. Springer Lecture Notes in Computer Science, 564, 3–29.
- Lee, E.A. (2008). Cyber physical systems: Design challenges. 11th IEEE International Symposium on Object/Component/Service-Oriented Real-Time Distributed Computing, 363–369.
- Henzinger, T.A. (2006). The theory of hybrid automata. Proceedings of the 11th Annual IEEE Symposium on Logic in Computer Science, 278–292.