This lesson is the capstone of Chapter 19. We do not introduce a new localization or planning algorithm; instead, we evaluate the complete AMR stack designed in previous chapters (odometry, probabilistic localization, SLAM, navigation stack, local planner, dynamic obstacle handling, and outdoor adaptations). The objective is to produce a full-system report that is statistically valid and reproducible.

Let \( \mathcal{S} \) denote the set of test scenarios and \( \mathcal{K}_s \) the set of seeds (or repeated trials) for scenario \( s \in \mathcal{S} \). For each run, we record a metric vector \( \mathbf{m}_{s,k} \in \mathbb{R}^p \) containing localization, safety, efficiency, and computational measurements.

\[ \mathbf{m}_{s,k} = \begin{bmatrix} m^{(\mathrm{ATE})}_{s,k} & m^{(\mathrm{RPE})}_{s,k} & m^{(\mathrm{succ})}_{s,k} & m^{(\mathrm{coll})}_{s,k} & m^{(\mathrm{time})}_{s,k} & m^{(\mathrm{SPL})}_{s,k} & m^{(\mathrm{energy})}_{s,k} & m^{(\mathrm{cpu})}_{s,k} \end{bmatrix}^\top \]

A full-system report compares at least two configurations: a baseline \( A \) and a candidate \( B \), using paired trials (same scenario and seed) so that environmental randomness cancels in the comparison.

The recommended workflow is: define protocol, run paired experiments, validate logs, compute metrics, run uncertainty analysis, then write the report. The pipeline below is intentionally aligned with the benchmarking concepts introduced in Lessons 1–4.

flowchart TD
  A["Define scenarios and seeds"] --> B["Freeze software and parameters"]
  B --> C["Run baseline and candidate (paired runs)"]
  C --> D["Collect logs: pose, events, CPU, energy"]
  D --> E["Parse metrics and quality checks"]
  E --> F["Compute paired statistics and confidence intervals"]
  F --> G["Generate tables, figures, and conclusions"]
  G --> H["Archive artifacts: configs, commit hash, raw logs"]
        

Pairing is essential. If run \( (s,k) \) exists for the baseline but not for the candidate, that pair should be excluded from paired inference (or explicitly handled as missing data in a pre-specified way).

A complete report should contain metrics from four layers: (i) localization/mapping accuracy, (ii) mission success and safety, (iii) efficiency, and (iv) computation. We use the following canonical definitions.

3.1 Absolute trajectory error (ATE)

With aligned estimated pose translation \( \hat{\mathbf{p} }_t \) and reference translation \( \mathbf{p}_t \), define translational error \( \mathbf{e}_t = \hat{\mathbf{p} }_t - \mathbf{p}_t \). The RMSE is

\[ \mathrm{ATE}_{\mathrm{RMSE} } = \sqrt{\frac{1}{T}\sum_{t=1}^T \|\mathbf{e}_t\|_2^2} \]

3.2 Relative pose error (RPE)

For a fixed horizon \( \Delta \), let the relative transform error be \( \mathbf{E}_{t,\Delta} \). A translational RPE summary is

\[ \mathrm{RPE}_{\mathrm{RMSE} }(\Delta) = \sqrt{\frac{1}{T-\Delta}\sum_{t=1}^{T-\Delta} \|\mathrm{trans}(\mathbf{E}_{t,\Delta})\|_2^2} \]

3.3 Mission success, collision rate, and safety margin

For run indicator \( y_{s,k} \in \{0,1\} \) (success/failure), the empirical success rate is

\[ \hat{p}_{\mathrm{succ} } = \frac{1}{N}\sum_{(s,k)} y_{s,k} \]

If \( c_{s,k} \in \{0,1\} \) indicates at least one collision event, then

\[ \hat{p}_{\mathrm{coll} } = \frac{1}{N}\sum_{(s,k)} c_{s,k} \]

Let \( d_{\min,s,k} \) be the minimum obstacle clearance during run \( (s,k) \). A robust safety summary is the lower-tail quantile \( Q_{0.10}(d_{\min}) \) instead of only the mean.

3.4 Path efficiency and SPL

Using shortest-path distance \( L^*_{s,k} \) and executed path length \( L_{s,k} \), define the per-run score

\[ \mathrm{spl}_{s,k} = y_{s,k} \cdot \frac{L^*_{s,k} }{\max(L_{s,k}, L^*_{s,k})} \]

The dataset-level SPL is \( \mathrm{SPL} = \frac{1}{N}\sum_{(s,k)} \mathrm{spl}_{s,k} \).

Proposition: For every run, \( 0 \le \mathrm{spl}_{s,k} \le 1 \).

Proof: Because \( y_{s,k} \in \{0,1\} \) and \( \max(L_{s,k},L^*_{s,k}) \ge L^*_{s,k} \ge 0 \), the fraction lies in \( [0,1] \). Multiplication by \( y_{s,k} \) preserves this interval. Averaging preserves bounds, hence \( 0 \le \mathrm{SPL} \le 1 \). □

3.5 Energy and compute efficiency

To decouple mission length from power usage, report energy per meter:

\[ e_{s,k}^{(\mathrm{pm})} = \frac{E_{s,k} }{\max(L_{s,k},\epsilon)} \]

and computational load statistics such as mean CPU cycle time and tail latency \( \mathrm{CPU}_{95\%} \).

The report should never rely on point estimates alone. For any metric where “higher is better,” define the paired difference \( d_{s,k} = m^B_{s,k} - m^A_{s,k} \). For metrics where “lower is better” (e.g., ATE, time), evaluate the transformed metric \( -m \) so that interpretation remains consistent.

The paired mean improvement is

\[ \bar{d} = \frac{1}{N}\sum_{(s,k)} d_{s,k} \]

with sample variance

\[ s_d^2 = \frac{1}{N-1}\sum_{(s,k)} (d_{s,k}-\bar{d})^2 \]

and approximate Gaussian confidence interval

\[ \bar{d} \pm t_{1-\alpha/2,\,N-1}\frac{s_d}{\sqrt{N} } \]

For non-Gaussian metrics (success, collisions, tail-latency), a paired bootstrap interval is often more robust. Resample the paired differences with replacement:

\[ \bar{d}^{*(b)} = \frac{1}{N}\sum_{i=1}^N d_{I_i^{(b)} }, \quad I_i^{(b)} \sim \mathrm{Unif}\{1,\dots,N\} \]

Then use empirical quantiles of \( \{\bar{d}^{*(b)}\}_{b=1}^B \) for the interval.

Blocked analysis across scenarios

Since scenario difficulty differs (indoor clutter, outdoor terrain, moving obstacles), a mixed-effects view is useful even in a lab report. Let \( i \) index the paired run and \( s(i) \) its scenario:

\[ d_i = \mu_d + u_{s(i)} + \varepsilon_i, \quad u_s \sim \mathcal{N}(0,\sigma_s^2), \quad \varepsilon_i \sim \mathcal{N}(0,\sigma_\varepsilon^2) \]

Here \( \mu_d \) is the global improvement and \( u_s \) captures scenario-specific difficulty shifts. This avoids overconfidence when many seeds come from a small number of maps.

Multiple metrics and family-wise risk

If you test many metrics, p-values should be adjusted. A practical option is the Holm correction. Sort p-values \( p_{(1)} \le \dots \le p_{(M)} \) and compare sequentially with \( \alpha/(M-j+1) \).

A single scalar score is useful for executive summaries, but it must never replace the metric table. We recommend reporting both: (i) the full metric vector and (ii) a normalized composite score.

Let \( z_j \) be the normalized score of metric \( j \) after direction correction (higher is better), for example:

\[ z_j = \mathrm{clip}\!\left(\frac{m_j - a_j}{b_j-a_j}, 0, 1\right) \]

where \( [a_j,b_j] \) is a pre-specified operating range from prior benchmarks. With nonnegative weights \( w_j \) and \( \sum_j w_j = 1 \), define

\[ J = \sum_{j=1}^p w_j z_j \]

Theorem (boundedness): If \( 0 \le z_j \le 1 \), \( w_j \ge 0 \), and \( \sum_j w_j = 1 \), then \( 0 \le J \le 1 \).

Proof: Since each \( z_j \ge 0 \) and each \( w_j \ge 0 \), the sum is nonnegative. Also, \( w_j z_j \le w_j \) for all \( j \), so \( J = \sum_j w_j z_j \le \sum_j w_j = 1 \). □

Even with a composite score, retain a Pareto view: if system B improves success and safety but worsens CPU tail latency, that trade-off must be explicitly stated.

A research-style AMR evaluation report should contain the following sections:

1. System definition: hardware, sensors, software version, calibration, and planner/localizer settings.
2. Protocol: maps/scenarios, seeds, safety-stop criteria, timeout, and reset rules.
3. Metrics: exact formulas and units (as in Section 3).
4. Results: aggregate table + per-scenario breakdown + paired intervals.
5. Ablation/stress links: connect the conclusions to Lessons 3 and 4 of this chapter.
6. Reproducibility package: raw logs, parsing code, report scripts, commit hashes.

For scenario-level reporting, define weighted aggregation by scenario importance weights \( \omega_s \ge 0 \), \( \sum_s \omega_s = 1 \):

\[ \bar{m} = \sum_{s \in \mathcal{S} } \omega_s \left( \frac{1}{|\mathcal{K}_s|} \sum_{k \in \mathcal{K}_s} m_{s,k} \right) \]

This prevents a scenario with many repeated seeds from dominating the final summary if your protocol intentionally oversampled one map.

The final deliverable is not only a PDF table: it is a reproducible artifact bundle. Every reported number should be traceable to raw logs and a deterministic parsing script.

Keep the artifact bundle together: raw logs, scenario/seed manifest, configuration files, commit hashes, parser scripts, and the final report. In grading and in research, a result that cannot be regenerated from archived artifacts is considered incomplete.

In the code below, the Python script generates a markdown report from CSV logs, while the C++, Java, MATLAB, and Mathematica implementations provide cross-language evaluators for the same log schema.

Code: Chapter19_Lesson5.py

# Chapter19_Lesson5.py
# Full-System Evaluation Report Lab (Autonomous Mobile Robots)
# Usage:
#   python Chapter19_Lesson5.py --runs baseline.csv candidate.csv --labels baseline candidate
# If no files are provided, synthetic data will be generated for demonstration.

import argparse
import csv
import math
import random
from dataclasses import dataclass
from typing import Dict, List, Tuple


@dataclass
class RunRecord:
    scenario: str
    seed: int
    success: int
    time_s: float
    path_length_m: float
    shortest_path_m: float
    collisions: int
    min_clearance_m: float
    energy_j: float
    ate_rmse_m: float
    rpe_rmse_m: float
    cpu_mean_ms: float
    cpu_p95_ms: float


def read_runs_csv(path: str) -> List[RunRecord]:
    rows: List[RunRecord] = []
    with open(path, "r", newline="", encoding="utf-8") as f:
        reader = csv.DictReader(f)
        for r in reader:
            rows.append(
                RunRecord(
                    scenario=r["scenario"],
                    seed=int(r["seed"]),
                    success=int(r["success"]),
                    time_s=float(r["time_s"]),
                    path_length_m=float(r["path_length_m"]),
                    shortest_path_m=float(r["shortest_path_m"]),
                    collisions=int(r["collisions"]),
                    min_clearance_m=float(r["min_clearance_m"]),
                    energy_j=float(r["energy_j"]),
                    ate_rmse_m=float(r["ate_rmse_m"]),
                    rpe_rmse_m=float(r["rpe_rmse_m"]),
                    cpu_mean_ms=float(r["cpu_mean_ms"]),
                    cpu_p95_ms=float(r["cpu_p95_ms"]),
                )
            )
    return rows


def synthetic_runs(label: str, n_scenarios: int = 4, seeds_per_scenario: int = 15) -> List[RunRecord]:
    random.seed(7 if label == "baseline" else 11)
    rows: List[RunRecord] = []
    for s in range(n_scenarios):
        scenario = f"S{s+1}"
        hardness = 1.0 + 0.25 * s
        for seed in range(seeds_per_scenario):
            if label == "baseline":
                p_success = max(0.55, 0.92 - 0.08 * s)
                time_scale = 1.0
                loc_scale = 1.0
                cpu_scale = 1.0
            else:
                p_success = max(0.60, 0.95 - 0.06 * s)
                time_scale = 0.92
                loc_scale = 0.85
                cpu_scale = 1.08

            success = 1 if random.random() < p_success else 0
            shortest = 18.0 + 3.0 * s + random.random() * 2.0
            path = shortest * (1.05 + 0.22 * hardness * random.random())
            time_s = time_scale * (path / (0.55 + 0.1 * random.random())) + 6.0 * (1 - success)
            collisions = 0 if success and random.random() < 0.85 else (1 if random.random() < 0.35 else 0)
            min_clearance = max(0.03, 0.55 - 0.18 * hardness + 0.2 * random.random())
            energy = 18.0 * path + 120.0 * (1 if collisions > 0 else 0) + 25.0 * random.random()
            ate = loc_scale * (0.08 + 0.05 * hardness + 0.05 * random.random())
            rpe = loc_scale * (0.03 + 0.02 * hardness + 0.02 * random.random())
            cpu_mean = cpu_scale * (18 + 6 * hardness + 5 * random.random())
            cpu_p95 = cpu_mean + cpu_scale * (8 + 6 * random.random())
            rows.append(
                RunRecord(
                    scenario=scenario,
                    seed=seed,
                    success=success,
                    time_s=time_s,
                    path_length_m=path,
                    shortest_path_m=shortest,
                    collisions=collisions,
                    min_clearance_m=min_clearance,
                    energy_j=energy,
                    ate_rmse_m=ate,
                    rpe_rmse_m=rpe,
                    cpu_mean_ms=cpu_mean,
                    cpu_p95_ms=cpu_p95,
                )
            )
    return rows


def mean(xs: List[float]) -> float:
    return sum(xs) / len(xs) if xs else float("nan")


def stdev(xs: List[float]) -> float:
    if len(xs) < 2:
        return 0.0
    m = mean(xs)
    return math.sqrt(sum((x - m) ** 2 for x in xs) / (len(xs) - 1))


def quantile(xs: List[float], q: float) -> float:
    if not xs:
        return float("nan")
    ys = sorted(xs)
    k = (len(ys) - 1) * q
    i = int(math.floor(k))
    j = int(math.ceil(k))
    if i == j:
        return ys[i]
    return ys[i] * (j - k) + ys[j] * (k - i)


def success_weighted_path_length(rows: List[RunRecord]) -> float:
    vals = []
    for r in rows:
        if r.success:
            vals.append(r.shortest_path_m / max(r.path_length_m, 1e-9))
        else:
            vals.append(0.0)
    return mean(vals)


def aggregate(rows: List[RunRecord]) -> Dict[str, float]:
    succ = [r.success for r in rows]
    times = [r.time_s for r in rows]
    paths = [r.path_length_m for r in rows]
    cols = [1 if r.collisions > 0 else 0 for r in rows]
    clear = [r.min_clearance_m for r in rows]
    eper = [r.energy_j / max(r.path_length_m, 1e-9) for r in rows]
    ate = [r.ate_rmse_m for r in rows]
    rpe = [r.rpe_rmse_m for r in rows]
    cpu_m = [r.cpu_mean_ms for r in rows]
    cpu_p = [r.cpu_p95_ms for r in rows]

    return {
        "n_runs": float(len(rows)),
        "success_rate": mean(succ),
        "success_rate_sd": stdev([float(x) for x in succ]),
        "time_mean_s": mean(times),
        "time_sd_s": stdev(times),
        "path_mean_m": mean(paths),
        "collision_rate": mean(cols),
        "min_clearance_p10_m": quantile(clear, 0.10),
        "energy_per_m_mean": mean(eper),
        "ate_rmse_mean_m": mean(ate),
        "rpe_rmse_mean_m": mean(rpe),
        "cpu_mean_ms": mean(cpu_m),
        "cpu_p95_ms": mean(cpu_p),
        "spl": success_weighted_path_length(rows),
    }


def paired_by_scenario_seed(
    base_rows: List[RunRecord], cand_rows: List[RunRecord], metric_fn
) -> List[Tuple[str, int, float]]:
    bmap = {(r.scenario, r.seed): r for r in base_rows}
    cmap = {(r.scenario, r.seed): r for r in cand_rows}
    keys = sorted(set(bmap.keys()) & set(cmap.keys()))
    diffs = []
    for k in keys:
        diffs.append((k[0], k[1], metric_fn(cmap[k]) - metric_fn(bmap[k])))
    return diffs


def bootstrap_ci(diffs: List[float], n_boot: int = 3000, alpha: float = 0.05) -> Tuple[float, float, float]:
    if not diffs:
        return float("nan"), float("nan"), float("nan")
    random.seed(123)
    n = len(diffs)
    mhat = mean(diffs)
    boots = []
    for _ in range(n_boot):
        sample = [diffs[random.randrange(n)] for _ in range(n)]
        boots.append(mean(sample))
    lo = quantile(boots, alpha / 2.0)
    hi = quantile(boots, 1.0 - alpha / 2.0)
    return mhat, lo, hi


def make_markdown_report(
    baseline_label: str,
    candidate_label: str,
    base_rows: List[RunRecord],
    cand_rows: List[RunRecord],
    out_path: str,
) -> None:
    A = aggregate(base_rows)
    B = aggregate(cand_rows)

    def fmt(x: float, k: int = 4) -> str:
        return f"{x:.{k}f}"

    pairs_success = paired_by_scenario_seed(base_rows, cand_rows, lambda r: float(r.success))
    pairs_time = paired_by_scenario_seed(base_rows, cand_rows, lambda r: -r.time_s)  # negative time => higher is better
    pairs_ate = paired_by_scenario_seed(base_rows, cand_rows, lambda r: -r.ate_rmse_m)  # lower is better
    pairs_spl = paired_by_scenario_seed(
        base_rows,
        cand_rows,
        lambda r: (r.shortest_path_m / max(r.path_length_m, 1e-9)) if r.success else 0.0,
    )

    stats = {
        "delta_success": bootstrap_ci([d for _, _, d in pairs_success]),
        "delta_neg_time": bootstrap_ci([d for _, _, d in pairs_time]),
        "delta_neg_ate": bootstrap_ci([d for _, _, d in pairs_ate]),
        "delta_spl": bootstrap_ci([d for _, _, d in pairs_spl]),
    }

    with open(out_path, "w", encoding="utf-8") as f:
        f.write("# Full-System Evaluation Report\\n\\n")
        f.write(f"**Baseline:** {baseline_label}\\n\\n")
        f.write(f"**Candidate:** {candidate_label}\\n\\n")
        f.write("## 1. Aggregate Metrics\\n\\n")
        f.write("| Metric | Baseline | Candidate |\\n")
        f.write("|---|---:|---:|\\n")
        rows = [
            ("Success rate", A["success_rate"], B["success_rate"]),
            ("SPL", A["spl"], B["spl"]),
            ("Collision rate", A["collision_rate"], B["collision_rate"]),
            ("Mean time [s]", A["time_mean_s"], B["time_mean_s"]),
            ("ATE RMSE [m]", A["ate_rmse_mean_m"], B["ate_rmse_mean_m"]),
            ("RPE RMSE [m]", A["rpe_rmse_mean_m"], B["rpe_rmse_mean_m"]),
            ("Min clearance P10 [m]", A["min_clearance_p10_m"], B["min_clearance_p10_m"]),
            ("Energy / m [J/m]", A["energy_per_m_mean"], B["energy_per_m_mean"]),
            ("CPU mean [ms]", A["cpu_mean_ms"], B["cpu_mean_ms"]),
            ("CPU p95 [ms]", A["cpu_p95_ms"], B["cpu_p95_ms"]),
        ]
        for name, x, y in rows:
            f.write(f"| {name} | {fmt(x,3)} | {fmt(y,3)} |\\n")

        f.write("\\n## 2. Paired Bootstrap Differences (candidate - baseline)\\n\\n")
        f.write("| Metric (higher is better) | Mean diff | 95% CI |\\n")
        f.write("|---|---:|---:|\\n")
        f.write(
            f"| Success rate | {fmt(stats['delta_success'][0],4)} | "
            f"[{fmt(stats['delta_success'][1],4)}, {fmt(stats['delta_success'][2],4)}] |\\n"
        )
        f.write(
            f"| -Time (faster is better) | {fmt(stats['delta_neg_time'][0],4)} | "
            f"[{fmt(stats['delta_neg_time'][1],4)}, {fmt(stats['delta_neg_time'][2],4)}] |\\n"
        )
        f.write(
            f"| -ATE (lower is better) | {fmt(stats['delta_neg_ate'][0],4)} | "
            f"[{fmt(stats['delta_neg_ate'][1],4)}, {fmt(stats['delta_neg_ate'][2],4)}] |\\n"
        )
        f.write(
            f"| SPL | {fmt(stats['delta_spl'][0],4)} | "
            f"[{fmt(stats['delta_spl'][1],4)}, {fmt(stats['delta_spl'][2],4)}] |\\n"
        )

        f.write("\\n## 3. Reproducibility Checklist\\n\\n")
        f.write("- Fixed scenario list and maps\\n")
        f.write("- Shared random seeds for paired comparisons\\n")
        f.write("- Versioned software commit hashes\\n")
        f.write("- Sensor calibration and hardware metadata\\n")
        f.write("- Runtime limits and safety-stop policy\\n")
        f.write("- Log schema and parsing scripts archived\\n")
        f.write("\\n## 4. Interpretation Template\\n\\n")
        f.write("Report primary metric first (mission success and safety), then efficiency (time/SPL/energy), ")
        f.write("then localization quality (ATE/RPE), then computational load. Emphasize confidence intervals, ")
        f.write("not only point estimates.\\n")


def write_example_csv(path: str, rows: List[RunRecord]) -> None:
    with open(path, "w", newline="", encoding="utf-8") as f:
        writer = csv.writer(f)
        writer.writerow([
            "scenario","seed","success","time_s","path_length_m","shortest_path_m",
            "collisions","min_clearance_m","energy_j","ate_rmse_m","rpe_rmse_m",
            "cpu_mean_ms","cpu_p95_ms"
        ])
        for r in rows:
            writer.writerow([
                r.scenario, r.seed, r.success, f"{r.time_s:.6f}", f"{r.path_length_m:.6f}",
                f"{r.shortest_path_m:.6f}", r.collisions, f"{r.min_clearance_m:.6f}",
                f"{r.energy_j:.6f}", f"{r.ate_rmse_m:.6f}", f"{r.rpe_rmse_m:.6f}",
                f"{r.cpu_mean_ms:.6f}", f"{r.cpu_p95_ms:.6f}"
            ])


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--runs", nargs=2, metavar=("BASELINE_CSV", "CANDIDATE_CSV"))
    parser.add_argument("--labels", nargs=2, default=["baseline", "candidate"])
    parser.add_argument("--out", default="Chapter19_Lesson5_Report.md")
    args = parser.parse_args()

    if args.runs:
        base_rows = read_runs_csv(args.runs[0])
        cand_rows = read_runs_csv(args.runs[1])
    else:
        base_rows = synthetic_runs("baseline")
        cand_rows = synthetic_runs("candidate")
        write_example_csv("baseline_example.csv", base_rows)
        write_example_csv("candidate_example.csv", cand_rows)

    make_markdown_report(args.labels[0], args.labels[1], base_rows, cand_rows, args.out)
    print(f"Report written to {args.out}")


if __name__ == "__main__":
    main()

      

Code: Chapter19_Lesson5.cpp

// Chapter19_Lesson5.cpp
// Full-System Evaluation Report Lab (Autonomous Mobile Robots)
// Minimal C++17 evaluator for CSV run logs.

#include <algorithm>
#include <cmath>
#include <fstream>
#include <iomanip>
#include <iostream>
#include <map>
#include <numeric>
#include <sstream>
#include <string>
#include <tuple>
#include <vector>

struct RunRecord {
  std::string scenario;
  int seed{ {} };
  int success{ {} };
  double time_s{ {} };
  double path_length_m{ {} };
  double shortest_path_m{ {} };
  int collisions{ {} };
  double min_clearance_m{ {} };
  double energy_j{ {} };
  double ate_rmse_m{ {} };
  double rpe_rmse_m{ {} };
  double cpu_mean_ms{ {} };
  double cpu_p95_ms{ {} };
};

std::vector<std::string> SplitCSV(const std::string& line) {
  std::vector<std::string> out;
  std::stringstream ss(line);
  std::string item;
  while (std::getline(ss, item, ',')) out.push_back(item);
  return out;
}

std::vector<RunRecord> ReadCSV(const std::string& path) {
  std::ifstream fin(path);
  if (!fin) throw std::runtime_error("Cannot open file: " + path);
  std::string line;
  std::getline(fin, line); // header
  std::vector<RunRecord> rows;
  while (std::getline(fin, line)) {
    if (line.empty()) continue;
    auto t = SplitCSV(line);
    if (t.size() < 13) continue;
    RunRecord r;
    r.scenario = t[0];
    r.seed = std::stoi(t[1]);
    r.success = std::stoi(t[2]);
    r.time_s = std::stod(t[3]);
    r.path_length_m = std::stod(t[4]);
    r.shortest_path_m = std::stod(t[5]);
    r.collisions = std::stoi(t[6]);
    r.min_clearance_m = std::stod(t[7]);
    r.energy_j = std::stod(t[8]);
    r.ate_rmse_m = std::stod(t[9]);
    r.rpe_rmse_m = std::stod(t[10]);
    r.cpu_mean_ms = std::stod(t[11]);
    r.cpu_p95_ms = std::stod(t[12]);
    rows.push_back(r);
  }
  return rows;
}

double Mean(const std::vector<double>& x) {
  if (x.empty()) return std::nan("");
  return std::accumulate(x.begin(), x.end(), 0.0) / static_cast<double>(x.size());
}

double Quantile(std::vector<double> x, double q) {
  if (x.empty()) return std::nan("");
  std::sort(x.begin(), x.end());
  double k = (x.size() - 1) * q;
  size_t i = static_cast<size_t>(std::floor(k));
  size_t j = static_cast<size_t>(std::ceil(k));
  if (i == j) return x[i];
  return x[i] * (j - k) + x[j] * (k - i);
}

struct Aggregate {
  double success_rate{ {} };
  double spl{ {} };
  double collision_rate{ {} };
  double time_mean_s{ {} };
  double ate_rmse_mean_m{ {} };
  double rpe_rmse_mean_m{ {} };
  double clearance_p10_m{ {} };
  double energy_per_m_mean{ {} };
  double cpu_mean_ms{ {} };
  double cpu_p95_ms{ {} };
};

Aggregate AggregateRuns(const std::vector<RunRecord>& rows) {
  std::vector<double> succ, spls, cols, times, ate, rpe, clear, eper, cpum, cpup;
  for (const auto& r : rows) {
    succ.push_back(static_cast<double>(r.success));
    spls.push_back(r.success ? r.shortest_path_m / std::max(r.path_length_m, 1e-9) : 0.0);
    cols.push_back(r.collisions > 0 ? 1.0 : 0.0);
    times.push_back(r.time_s);
    ate.push_back(r.ate_rmse_m);
    rpe.push_back(r.rpe_rmse_m);
    clear.push_back(r.min_clearance_m);
    eper.push_back(r.energy_j / std::max(r.path_length_m, 1e-9));
    cpum.push_back(r.cpu_mean_ms);
    cpup.push_back(r.cpu_p95_ms);
  }
  Aggregate a;
  a.success_rate = Mean(succ);
  a.spl = Mean(spls);
  a.collision_rate = Mean(cols);
  a.time_mean_s = Mean(times);
  a.ate_rmse_mean_m = Mean(ate);
  a.rpe_rmse_mean_m = Mean(rpe);
  a.clearance_p10_m = Quantile(clear, 0.10);
  a.energy_per_m_mean = Mean(eper);
  a.cpu_mean_ms = Mean(cpum);
  a.cpu_p95_ms = Mean(cpup);
  return a;
}

int main(int argc, char** argv) {
  try {
    if (argc < 3) {
      std::cerr << "Usage: " << argv[0] << " baseline.csv candidate.csv\\n";
      return 1;
    }
    auto baseline = ReadCSV(argv[1]);
    auto candidate = ReadCSV(argv[2]);
    auto A = AggregateRuns(baseline);
    auto B = AggregateRuns(candidate);

    std::cout << std::fixed << std::setprecision(4);
    std::cout << "=== Chapter19 Lesson5: Full-System Evaluation Summary ===\\n";
    std::cout << "Metric, Baseline, Candidate\\n";
    std::cout << "SuccessRate, " << A.success_rate << ", " << B.success_rate << "\\n";
    std::cout << "SPL, " << A.spl << ", " << B.spl << "\\n";
    std::cout << "CollisionRate, " << A.collision_rate << ", " << B.collision_rate << "\\n";
    std::cout << "TimeMean_s, " << A.time_mean_s << ", " << B.time_mean_s << "\\n";
    std::cout << "ATE_RMSE_m, " << A.ate_rmse_mean_m << ", " << B.ate_rmse_mean_m << "\\n";
    std::cout << "RPE_RMSE_m, " << A.rpe_rmse_mean_m << ", " << B.rpe_rmse_mean_m << "\\n";
    std::cout << "Clearance_P10_m, " << A.clearance_p10_m << ", " << B.clearance_p10_m << "\\n";
    std::cout << "EnergyPerM_Jpm, " << A.energy_per_m_mean << ", " << B.energy_per_m_mean << "\\n";
    std::cout << "CPU_Mean_ms, " << A.cpu_mean_ms << ", " << B.cpu_mean_ms << "\\n";
    std::cout << "CPU_P95_ms, " << A.cpu_p95_ms << ", " << B.cpu_p95_ms << "\\n";
  } catch (const std::exception& e) {
    std::cerr << "Error: " << e.what() << "\\n";
    return 2;
  }
  return 0;
}

      

Code: Chapter19_Lesson5.java

// Chapter19_Lesson5.java
// Full-System Evaluation Report Lab (Autonomous Mobile Robots)
// Java 17+ CSV aggregator with paired-difference statistics.

import java.io.BufferedReader;
import java.io.FileReader;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Comparator;
import java.util.HashMap;
import java.util.List;
import java.util.Locale;
import java.util.Map;
import java.util.Random;

public class Chapter19_Lesson5 {
    static class RunRecord {
        String scenario;
        int seed;
        int success;
        double timeS;
        double pathLengthM;
        double shortestPathM;
        int collisions;
        double minClearanceM;
        double energyJ;
        double ateRmseM;
        double rpeRmseM;
        double cpuMeanMs;
        double cpuP95Ms;
    }

    static List<RunRecord> readCsv(String path) throws IOException {
        List<RunRecord> rows = new ArrayList<>();
        try (BufferedReader br = new BufferedReader(new FileReader(path))) {
            String line = br.readLine(); // header
            while ((line = br.readLine()) != null) {
                if (line.isBlank()) continue;
                String[] t = line.split(",");
                if (t.length < 13) continue;
                RunRecord r = new RunRecord();
                r.scenario = t[0];
                r.seed = Integer.parseInt(t[1]);
                r.success = Integer.parseInt(t[2]);
                r.timeS = Double.parseDouble(t[3]);
                r.pathLengthM = Double.parseDouble(t[4]);
                r.shortestPathM = Double.parseDouble(t[5]);
                r.collisions = Integer.parseInt(t[6]);
                r.minClearanceM = Double.parseDouble(t[7]);
                r.energyJ = Double.parseDouble(t[8]);
                r.ateRmseM = Double.parseDouble(t[9]);
                r.rpeRmseM = Double.parseDouble(t[10]);
                r.cpuMeanMs = Double.parseDouble(t[11]);
                r.cpuP95Ms = Double.parseDouble(t[12]);
                rows.add(r);
            }
        }
        return rows;
    }

    static double mean(List<Double> x) {
        if (x.isEmpty()) return Double.NaN;
        double s = 0.0;
        for (double v : x) s += v;
        return s / x.size();
    }

    static double quantile(List<Double> x, double q) {
        if (x.isEmpty()) return Double.NaN;
        x = new ArrayList<>(x);
        x.sort(Comparator.naturalOrder());
        double k = (x.size() - 1) * q;
        int i = (int) Math.floor(k);
        int j = (int) Math.ceil(k);
        if (i == j) return x.get(i);
        return x.get(i) * (j - k) + x.get(j) * (k - i);
    }

    static double spl(List<RunRecord> rows) {
        List<Double> vals = new ArrayList<>();
        for (RunRecord r : rows) {
            vals.add((r.success == 1) ? r.shortestPathM / Math.max(r.pathLengthM, 1e-9) : 0.0);
        }
        return mean(vals);
    }

    static class Aggregate {
        double successRate, spl, collisionRate, timeMean, ateMean, rpeMean, clrP10, ePerM, cpuMean, cpuP95;
    }

    static Aggregate aggregate(List<RunRecord> rows) {
        List<Double> succ = new ArrayList<>(), col = new ArrayList<>(), time = new ArrayList<>();
        List<Double> ate = new ArrayList<>(), rpe = new ArrayList<>(), clr = new ArrayList<>();
        List<Double> eper = new ArrayList<>(), cm = new ArrayList<>(), cp = new ArrayList<>();
        for (RunRecord r : rows) {
            succ.add((double) r.success);
            col.add((r.collisions > 0) ? 1.0 : 0.0);
            time.add(r.timeS);
            ate.add(r.ateRmseM);
            rpe.add(r.rpeRmseM);
            clr.add(r.minClearanceM);
            eper.add(r.energyJ / Math.max(r.pathLengthM, 1e-9));
            cm.add(r.cpuMeanMs);
            cp.add(r.cpuP95Ms);
        }
        Aggregate a = new Aggregate();
        a.successRate = mean(succ);
        a.spl = spl(rows);
        a.collisionRate = mean(col);
        a.timeMean = mean(time);
        a.ateMean = mean(ate);
        a.rpeMean = mean(rpe);
        a.clrP10 = quantile(clr, 0.10);
        a.ePerM = mean(eper);
        a.cpuMean = mean(cm);
        a.cpuP95 = mean(cp);
        return a;
    }

    static List<Double> pairedDiff(List<RunRecord> base, List<RunRecord> cand, String metric) {
        Map<String, RunRecord> b = new HashMap<>();
        Map<String, RunRecord> c = new HashMap<>();
        for (RunRecord r : base) b.put(r.scenario + "#" + r.seed, r);
        for (RunRecord r : cand) c.put(r.scenario + "#" + r.seed, r);

        List<Double> out = new ArrayList<>();
        for (String k : b.keySet()) {
            if (!c.containsKey(k)) continue;
            RunRecord rb = b.get(k), rc = c.get(k);
            switch (metric) {
                case "success" -> out.add((double) (rc.success - rb.success));
                case "negTime" -> out.add((-rc.timeS) - (-rb.timeS));
                case "negATE" -> out.add((-rc.ateRmseM) - (-rb.ateRmseM));
                case "spl" -> {
                    double sb = (rb.success == 1) ? rb.shortestPathM / Math.max(rb.pathLengthM, 1e-9) : 0.0;
                    double sc = (rc.success == 1) ? rc.shortestPathM / Math.max(rc.pathLengthM, 1e-9) : 0.0;
                    out.add(sc - sb);
                }
                default -> throw new IllegalArgumentException("Unknown metric: " + metric);
            }
        }
        return out;
    }

    static double[] bootstrapCI(List<Double> diffs, int nBoot, long seed) {
        Random rng = new Random(seed);
        int n = diffs.size();
        List<Double> boots = new ArrayList<>();
        for (int b = 0; b < nBoot; b++) {
            double s = 0.0;
            for (int i = 0; i < n; i++) {
                s += diffs.get(rng.nextInt(n));
            }
            boots.add(s / n);
        }
        double m = mean(diffs);
        return new double[]{m, quantile(boots, 0.025), quantile(boots, 0.975)};
    }

    static void printAggregate(String label, Aggregate a) {
        System.out.printf(Locale.US,
                "%s: success=%.3f, SPL=%.3f, collision=%.3f, time=%.2f s, ATE=%.3f m, RPE=%.3f m, clrP10=%.3f m, E/m=%.2f J/m, CPUmean=%.2f ms, CPUp95=%.2f ms%n",
                label, a.successRate, a.spl, a.collisionRate, a.timeMean, a.ateMean, a.rpeMean, a.clrP10, a.ePerM, a.cpuMean, a.cpuP95);
    }

    public static void main(String[] args) throws Exception {
        if (args.length < 2) {
            System.err.println("Usage: java Chapter19_Lesson5 baseline.csv candidate.csv");
            return;
        }
        List<RunRecord> baseline = readCsv(args[0]);
        List<RunRecord> candidate = readCsv(args[1]);

        Aggregate A = aggregate(baseline);
        Aggregate B = aggregate(candidate);

        printAggregate("Baseline", A);
        printAggregate("Candidate", B);

        for (String metric : List.of("success", "negTime", "negATE", "spl")) {
            double[] ci = bootstrapCI(pairedDiff(baseline, candidate, metric), 2000, 42L + metric.hashCode());
            System.out.printf(Locale.US, "Paired delta %-7s = %.4f  [%.4f, %.4f]%n", metric, ci[0], ci[1], ci[2]);
        }
    }
}

      

The MATLAB script computes the same aggregate metrics and paired bootstrap intervals. In Simulink-based labs, the logged signals can be exported to the same CSV schema and parsed by this script.

Code: Chapter19_Lesson5.m

% Chapter19_Lesson5.m
% Full-System Evaluation Report Lab (Autonomous Mobile Robots)
% MATLAB script for aggregate metrics and paired confidence intervals.
%
% Expected CSV columns:
% scenario,seed,success,time_s,path_length_m,shortest_path_m,collisions,min_clearance_m,
% energy_j,ate_rmse_m,rpe_rmse_m,cpu_mean_ms,cpu_p95_ms

function Chapter19_Lesson5()
    if ~isfile('baseline_example.csv') || ~isfile('candidate_example.csv')
        error('Provide baseline_example.csv and candidate_example.csv (e.g., generated by Python script).');
    end

    base = readtable('baseline_example.csv', 'TextType', 'string');
    cand = readtable('candidate_example.csv', 'TextType', 'string');

    A = aggregateRuns(base);
    B = aggregateRuns(cand);

    fprintf('=== Chapter19 Lesson5: Full-System Evaluation Report ===\n');
    disp(struct2table(A));
    disp(struct2table(B));

    [dSucc, dTime, dATE, dSPL] = pairedDiffs(base, cand);

    ciSucc = bootstrapCI(dSucc, 2000);
    ciTime = bootstrapCI(-dTime, 2000); % faster is better
    ciATE  = bootstrapCI(-dATE, 2000);  % lower is better
    ciSPL  = bootstrapCI(dSPL, 2000);

    fprintf('Delta success  : %.4f [%.4f, %.4f]\n', ciSucc(1), ciSucc(2), ciSucc(3));
    fprintf('Delta -time    : %.4f [%.4f, %.4f]\n', ciTime(1), ciTime(2), ciTime(3));
    fprintf('Delta -ATE     : %.4f [%.4f, %.4f]\n', ciATE(1),  ciATE(2),  ciATE(3));
    fprintf('Delta SPL      : %.4f [%.4f, %.4f]\n', ciSPL(1),  ciSPL(2),  ciSPL(3));

    % Optional figure for report appendix
    figure;
    histogram(dSPL);
    xlabel('Paired SPL difference (candidate - baseline)');
    ylabel('Count');
    title('Chapter19_Lesson5: SPL Paired Differences');
    grid on;
end

function S = aggregateRuns(T)
    spl = zeros(height(T),1);
    for i = 1:height(T)
        if T.success(i) == 1
            spl(i) = T.shortest_path_m(i) / max(T.path_length_m(i), 1e-9);
        else
            spl(i) = 0;
        end
    end

    S = struct();
    S.success_rate = mean(T.success);
    S.spl = mean(spl);
    S.collision_rate = mean(T.collisions > 0);
    S.time_mean_s = mean(T.time_s);
    S.ate_rmse_mean_m = mean(T.ate_rmse_m);
    S.rpe_rmse_mean_m = mean(T.rpe_rmse_m);
    S.clearance_p10_m = prctile(T.min_clearance_m, 10);
    S.energy_per_m_mean = mean(T.energy_j ./ max(T.path_length_m, 1e-9));
    S.cpu_mean_ms = mean(T.cpu_mean_ms);
    S.cpu_p95_ms = mean(T.cpu_p95_ms);
end

function [dSucc, dTime, dATE, dSPL] = pairedDiffs(B, C)
    % Inner join by scenario and seed
    keyB = B.scenario + "#" + string(B.seed);
    keyC = C.scenario + "#" + string(C.seed);
    [commonKeys, idxB, idxC] = intersect(keyB, keyC, 'stable');

    %#ok<NASGU> % commonKeys retained for reproducibility logs if needed
    dSucc = C.success(idxC) - B.success(idxB);
    dTime = C.time_s(idxC) - B.time_s(idxB);
    dATE  = C.ate_rmse_m(idxC) - B.ate_rmse_m(idxB);

    splB = zeros(numel(idxB),1);
    splC = zeros(numel(idxC),1);
    for i = 1:numel(idxB)
        if B.success(idxB(i)) == 1
            splB(i) = B.shortest_path_m(idxB(i)) / max(B.path_length_m(idxB(i)), 1e-9);
        end
        if C.success(idxC(i)) == 1
            splC(i) = C.shortest_path_m(idxC(i)) / max(C.path_length_m(idxC(i)), 1e-9);
        end
    end
    dSPL = splC - splB;
end

function ci = bootstrapCI(x, nBoot)
    x = x(:);
    n = numel(x);
    mu = mean(x);
    boots = zeros(nBoot,1);
    rng(42);
    for b = 1:nBoot
        idx = randi(n, [n,1]);
        boots(b) = mean(x(idx));
    end
    q = prctile(boots, [2.5, 97.5]);
    ci = [mu, q(1), q(2)];
end

      

Code: Chapter19_Lesson5.nb

(* Chapter19_Lesson5.nb *)
(* Full-System Evaluation Report Lab (Autonomous Mobile Robots) *)
(* Wolfram Language script content to place in a notebook. *)

ClearAll["Global`*"];

readRuns[file_] := Module[{raw, header, rows},
  raw = Import[file, "CSV"];
  header = First[raw];
  rows = Rest[raw];
  Association /@ (Thread[header -> #] & /@ rows)
];

toNum[x_] := Quiet@Check[ToExpression[x], x];

normalizeRow[r_] := <|
  "scenario" -> r["scenario"],
  "seed" -> toNum[r["seed"]],
  "success" -> toNum[r["success"]],
  "time_s" -> toNum[r["time_s"]],
  "path_length_m" -> toNum[r["path_length_m"]],
  "shortest_path_m" -> toNum[r["shortest_path_m"]],
  "collisions" -> toNum[r["collisions"]],
  "min_clearance_m" -> toNum[r["min_clearance_m"]],
  "energy_j" -> toNum[r["energy_j"]],
  "ate_rmse_m" -> toNum[r["ate_rmse_m"]],
  "rpe_rmse_m" -> toNum[r["rpe_rmse_m"]],
  "cpu_mean_ms" -> toNum[r["cpu_mean_ms"]],
  "cpu_p95_ms" -> toNum[r["cpu_p95_ms"]]
|>;

splValue[r_] := If[r["success"] == 1,
  r["shortest_path_m"]/Max[r["path_length_m"], 10^-9],
  0.0
];

aggregateRuns[data_] := Module[
  {d = normalizeRow /@ data, succ, col, t, ate, rpe, clr, eper, cm, cp, spl},
  succ = d[[All, "success"]];
  col = UnitStep[(d[[All, "collisions"]] /. x_ /; x > 0 -> 1) - 0.5];
  t = d[[All, "time_s"]];
  ate = d[[All, "ate_rmse_m"]];
  rpe = d[[All, "rpe_rmse_m"]];
  clr = d[[All, "min_clearance_m"]];
  eper = Map[#["energy_j"]/Max[#["path_length_m"], 10^-9] &, d];
  cm = d[[All, "cpu_mean_ms"]];
  cp = d[[All, "cpu_p95_ms"]];
  spl = splValue /@ d;
  <|
    "success_rate" -> Mean[succ],
    "spl" -> Mean[spl],
    "collision_rate" -> Mean[col],
    "time_mean_s" -> Mean[t],
    "ate_rmse_mean_m" -> Mean[ate],
    "rpe_rmse_mean_m" -> Mean[rpe],
    "clearance_p10_m" -> Quantile[clr, 0.10],
    "energy_per_m_mean" -> Mean[eper],
    "cpu_mean_ms" -> Mean[cm],
    "cpu_p95_ms" -> Mean[cp]
  |>
];

pairedDiffs[base_, cand_, keyMetric_] := Module[
  {b = normalizeRow /@ base, c = normalizeRow /@ cand, bAssoc, cAssoc, keys, f},
  bAssoc = Association@Table[(r["scenario"] <> "#" <> ToString[r["seed"]]) -> r, {r, b}];
  cAssoc = Association@Table[(r["scenario"] <> "#" <> ToString[r["seed"]]) -> r, {r, c}];
  keys = Intersection[Keys[bAssoc], Keys[cAssoc]];
  f = Switch[keyMetric,
    "success", (#[["success"]]) &,
    "negTime", (-#[["time_s"]]) &,
    "negATE", (-#[["ate_rmse_m"]]) &,
    "spl", splValue,
    _, (0 &)
  ];
  Table[f[cAssoc[k]] - f[bAssoc[k]], {k, keys}]
];

bootstrapCI[x_List, nBoot_:2000] := Module[{n = Length[x], boots},
  SeedRandom[42];
  boots = Table[Mean[RandomChoice[x, n]], {nBoot}];
  <|
    "meanDiff" -> Mean[x],
    "ci95" -> Quantile[boots, {0.025, 0.975}]
  |>
];

baseline = readRuns["baseline_example.csv"];
candidate = readRuns["candidate_example.csv"];

a = aggregateRuns[baseline];
b = aggregateRuns[candidate];

Print["=== Chapter19 Lesson5 Report Summary ==="];
Print[Dataset[{<|"System" -> "Baseline"|> ~Join~ a, <|"System" -> "Candidate"|> ~Join~ b}]];

deltas = <|
  "success" -> bootstrapCI[pairedDiffs[baseline, candidate, "success"]],
  "negTime" -> bootstrapCI[pairedDiffs[baseline, candidate, "negTime"]],
  "negATE" -> bootstrapCI[pairedDiffs[baseline, candidate, "negATE"]],
  "spl" -> bootstrapCI[pairedDiffs[baseline, candidate, "spl"]]
|>;

Print[Dataset@KeyValueMap[<|"Metric" -> #1, "MeanDiff" -> #2["meanDiff"], "CI95" -> #2["ci95"]|> &, deltas]];

      

Problem 1 (Paired confidence interval): For a metric transformed so that “higher is better,” suppose the paired differences are \( d_1,\dots,d_N \). Derive the classical t-based confidence interval for the mean improvement.

Solution: The estimator is \( \bar{d} = \frac{1}{N}\sum_i d_i \). Under independent paired differences with finite variance, estimate the variance using \( s_d^2 = \frac{1}{N-1}\sum_i (d_i-\bar{d})^2 \). Then

\[ T = \frac{\bar{d}-\mu_d}{s_d/\sqrt{N} } \]

follows a Student t distribution under Gaussian assumptions, so the \( (1-\alpha) \) confidence interval is

\[ \bar{d} \pm t_{1-\alpha/2,\,N-1}\frac{s_d}{\sqrt{N} } \]

This interval is preferred over unpaired analysis because the scenario/seed pairing removes shared disturbance variation.

Problem 2 (SPL bounds): Show that dataset-level SPL satisfies \( 0 \le \mathrm{SPL} \le 1 \).

Solution: For each run, \( \mathrm{spl}_i = y_i L_i^*/\max(L_i,L_i^*) \) with \( y_i \in \{0,1\} \). Since \( 0 \le L_i^*/\max(L_i,L_i^*) \le 1 \), each per-run score lies in \( [0,1] \). Averaging over \( N \) runs preserves the interval:

\[ 0 \le \frac{1}{N}\sum_{i=1}^N \mathrm{spl}_i \le 1 \]

Problem 3 (Composite score monotonicity): Let \( J = \sum_j w_j z_j \) with nonnegative weights summing to one. Prove that if system B has \( z_j^B \ge z_j^A \) for all \( j \), then \( J_B \ge J_A \).

Solution: Subtract:

\[ J_B - J_A = \sum_j w_j (z_j^B - z_j^A) \]

Every term is nonnegative because \( w_j \ge 0 \) and \( z_j^B - z_j^A \ge 0 \). Therefore \( J_B - J_A \ge 0 \), hence \( J_B \ge J_A \). □

Problem 4 (Scenario-weighted aggregation): Why can naive averaging over all runs be misleading when one scenario is repeated many more times than others?

Solution: Naive averaging weights scenarios proportionally to the number of seeds. If a single easy scenario has many repeats, it can dominate the final mean and hide failures in harder environments. The remedy is to aggregate within each scenario first and then apply explicit scenario weights \( \omega_s \), as in Section 6.

Problem 5 (Report logic flow): Draw a compact logic flow for writing the conclusion section of the report so that safety and mission success are prioritized before efficiency claims.

Solution (flow):

flowchart TD
  A["Check mission success and collisions"] --> B["If unsafe: reject candidate or retune"]
  B --> C["If safe: compare localization and path quality"]
  C --> D["Then compare time, energy, and CPU tail latency"]
  D --> E["State confidence intervals and scenario caveats"]
  E --> F["Write final recommendation"]
        

This lab formalized a full-system AMR evaluation report as a reproducible statistical artifact. We defined a complete metric set, justified paired comparisons, introduced blocked scenario-aware inference, and packaged cross-language code for parsing logs and generating report tables. In the capstone chapter, this report style becomes the template for the final project write-up.