By this point in the course, you have seen:

• Sampling- and optimization-based motion planning in high-dimensional configuration spaces.
• Planning under uncertainty and reinforcement learning for continuous control.
• Task and motion planning (TAMP) and long-horizon manipulation.
• Sim-to-real transfer, multi-robot coordination, swarm robotics, and formal methods.
• Foundation models, generalist robot policies, and world models (previous lessons of Chapter 19).

An open problem in advanced robotics is a question about robot capabilities or algorithms that is:

• Well-posed: can be stated in precise mathematical or algorithmic terms.
• Non-trivial: not directly solvable by composition of known results.
• Falsifiable: admits experiments or proofs that could refute a hypothesis.

A common abstraction for many open problems in this course is a distribution over tasks. Let \( \mathcal{T} \) denote a set of tasks (e.g., pick-and-place, cabinet opening, tool manipulation), and let \( p(t) \) be a probability distribution over \( t \in \mathcal{T} \). Each task is modeled as an MDP \( M_t = (\mathcal{S}, \mathcal{A}, P_t, r_t, \gamma) \). A generalist policy \( \pi_{\theta} \) receives state or observation \( s \) and a task description (symbolic or language-based) and outputs an action distribution.

\[ \pi_{\theta}(a \mid s, x_t), \quad t \sim p(t), \]

where \( x_t \) encodes the task (e.g. a PDDL goal, natural language prompt, or demonstration embedding, all discussed in previous chapters).

The meta-objective for a generalist robot policy is then

\[ J(\theta) = \mathbb{E}_{t \sim p(t)} \left[ \mathbb{E}_{\tau \sim P_t, \pi_{\theta}} \Bigg[ \sum_{k=0}^{H-1} \gamma^k r_t(s_k, a_k) \Bigg] \right], \]

where \( H \) is a (possibly large) planning horizon. Many open problems in advanced robotics can be phrased as:

• Representation: How to parameterize \( \pi_{\theta} \) so that it generalizes across \( \mathcal{T} \) with reasonable sample complexity?
• World models: How accurate must a learned dynamics model be to guarantee that planning in the model approximately optimizes \( J(\theta) \)?
• Long-horizon reasoning: How to control compounding errors in planning and perception across hundreds or thousands of steps?

2.1 World-Model Error vs. Policy Performance

Suppose we learn a task-conditioned world model for each task \( t \):

\[ p_{\phi}(s_{k+1} \mid s_k, a_k, t) \approx P_t(s_{k+1} \mid s_k, a_k), \]

and plan with this model (Chapter 4 and Chapter 19, Lesson 4). A core open question is: how does model error degrade performance? Consider a single task, drop the subscript \( t \), and assume rewards are bounded:

\[ |r(s,a)| \le R_{\max}, \quad \forall (s,a). \]

Suppose the true and learned dynamics differ in total-variation distance by at most \( \varepsilon \):

\[ \sup_{s,a} \frac{1}{2} \left\| P(\cdot \mid s,a) - p_{\phi}(\cdot \mid s,a) \right\|_{1} \le \varepsilon. \]

Let \( V^{\pi}_P(s_0) \) be the expected return of policy \( \pi \) in the true MDP (horizon \( H \)), and \( V^{\pi}_{\phi}(s_0) \) the expected return when the trajectory is generated by the learned model. We can bound the difference between the two values:

\[ V^{\pi}_P(s_0) = \mathbb{E}_{\tau \sim P,\pi} \Bigg[ \sum_{k=0}^{H-1} r(s_k,a_k) \Bigg], \quad V^{\pi}_{\phi}(s_0) = \mathbb{E}_{\tau \sim p_{\phi},\pi} \Bigg[ \sum_{k=0}^{H-1} r(s_k,a_k) \Bigg]. \]

Proposition (Simple simulation bound). If the per-step dynamics error is at most \( \varepsilon \) in total variation, then

\[ \left| V^{\pi}_P(s_0) - V^{\pi}_{\phi}(s_0) \right| \le 2 H^2 R_{\max} \varepsilon. \]

Proof sketch.

1. Consider the distribution over length-\( H \) trajectories under the true and learned models: \( \mathbb{P}_P(\tau) \) and \( \mathbb{P}_{\phi}(\tau) \).
2. Using a coupling argument, one can show that if each one-step transition differs in total variation by at most \( \varepsilon \), then the total variation distance between trajectory distributions is bounded by \( H \varepsilon \).
3. The cumulative reward is a bounded function of the trajectory: \( f(\tau) = \sum_{k=0}^{H-1} r(s_k,a_k) \) with \( |f(\tau)| \le H R_{\max} \).
4. For any function \( f \) bounded by \( |f|\le B \), the difference in expectation between two distributions satisfies

\[ \left| \mathbb{E}_{\mathbb{P}_P}[f] - \mathbb{E}_{\mathbb{P}_{\phi}}[f] \right| \le 2 B \, \mathrm{TV}(\mathbb{P}_P, \mathbb{P}_{\phi}), \]

where \( \mathrm{TV} \) denotes total variation. Plugging \( B = H R_{\max} \) and \( \mathrm{TV}(\mathbb{P}_P, \mathbb{P}_{\phi}) \le H \varepsilon \) yields the bound.

This bound suggests one class of open problems: design training objectives or model classes that minimize the induced trajectory-level value error, not just per-step prediction error.

2.2 Generalization Gap Across Tasks

Another key theme in open problems is the generalization gap between train and test tasks. For a generalist policy \( \pi_{\theta} \), define empirical performance on a finite set of training tasks \( \{t_i\}_{i=1}^{N_{\mathrm{train}}} \) as

\[ \hat{J}_{\mathrm{train}}(\theta) = \frac{1}{N_{\mathrm{train}}} \sum_{i=1}^{N_{\mathrm{train}}} \mathbb{E}_{\tau \sim P_{t_i},\pi_{\theta}} \Bigg[ \sum_{k=0}^{H-1} \gamma^k r_{t_i}(s_k,a_k) \Bigg]. \]

The population objective \( J(\theta) \) from Section 1 is the expected value over the true task distribution \( p(t) \). The generalization gap is

\[ \mathcal{G}(\theta) = J(\theta) - \hat{J}_{\mathrm{train}}(\theta). \]

Open problems include:

• Establishing \( \mathcal{G}(\theta) \) bounds for specific policy classes (e.g., transformer-based vision-language-action policies) under realistic task distributions.
• Designing curricula or task sampling schemes that provably reduce \( |\mathcal{G}(\theta)| \) for long-horizon robotics tasks.

A strong research proposal in advanced robotics connects a high-level frontier question to a concrete mathematical formulation and executable experimental plan. A minimal structure is:

1. Context: which chapter/area (planning, RL, TAMP, world models, etc.)?
2. Problem statement: a formal definition in terms of MDPs, C-spaces, or task sets.
3. Hypothesis: a testable claim formulated as an inequality or property.
4. Method: algorithmic idea, with equations and pseudocode.
5. Evaluation: metrics, baselines, and ablation plan.

The flow below summarizes how an initial idea becomes a research-grade proposal.

flowchart TD
  I["Idea (frontier question)"] --> L["Map to prior chapters (planning, RL, TAMP, world models)"]
  L --> F["Formalize as tasks, mdp, objectives"]
  F --> H["State hypothesis: e.g. new policy reduces regret or increases success rate"]
  H --> M["Design method: algorithm, architecture, training procedure"]
  M --> E["Plan evaluation: metrics, baselines, ablations"]
  E --> R["Risk analysis: failure modes, safety constraints"]
  R --> P["Write proposal and implementation roadmap"]
        

3.1 Example: Open Problem Template

Consider the following template for a proposal around generalist manipulation policies:

• Context: Chapters 9, 12, 13, 19 (TAMP, RL, sim-to-real, world models).
• Problem: Learn a single policy \( \pi_{\theta} \) that solves a distribution of long-horizon pick-and-place tasks with minimal environment interactions.
• Hypothesis (formal): there exists a representation \( \theta \) such that, after \( N \) environment steps,

\[ J(\theta_N) \ge J(\theta_0) + \Delta, \]

for some target improvement \( \Delta > 0 \), with high probability over sampled tasks. Here \( \theta_0 \) is a pre-trained foundation policy (Chapter 19, Lesson 1), and \( \theta_N \) is the adapted policy.

The proposal must then specify:

• How to initialize \( \theta_0 \) (e.g., from a dataset of offline trajectories).
• How to update \( \theta \) (e.g., off-policy RL, policy gradients, or imitation).
• Which metrics capture \( J(\theta_N) \) in practice (success rate, path cost, constraint violations).

4.1 Success Rate and Confidence Intervals

Most experimental robotics papers report success rate for a given task distribution. Let \( p \) be the true probability that a policy succeeds on a randomly sampled task instance. We run \( N \) independent trials and record binary outcomes \( X_i \in \{0,1\} \) (success or failure). The empirical success rate is

\[ \hat{p}_N = \frac{1}{N} \sum_{i=1}^{N} X_i. \]

We often want \( |\hat{p}_N - p| \le \alpha \) with probability at least \( 1 - \delta \). Using Hoeffding's inequality for bounded random variables \( X_i \in [0,1] \), one obtains

\[ \mathbb{P}\Big( |\hat{p}_N - p| \ge \alpha \Big) \le 2 \exp\!\big( -2 N \alpha^2 \big). \]

To guarantee \( \mathbb{P}(|\hat{p}_N - p| \ge \alpha) \le \delta \), it suffices to choose

\[ N \ge \frac{1}{2 \alpha^2} \log\!\left( \frac{2}{\delta} \right). \]

This simple formula should appear in your proposal whenever you justify the number of real-robot or simulation trials.

4.2 Regret as a Long-Horizon Metric

For planning and RL problems, a more informative metric than raw success/failure is regret relative to an oracle policy. For task \( t \), define optimal value \( V_t^{\star}(s_0) \) and policy value \( V_t^{\pi_{\theta}}(s_0) \). The regret on a task is

\[ \mathrm{Regret}_t(\theta) = V_t^{\star}(s_0) - V_t^{\pi_{\theta}}(s_0) \ge 0. \]

Averaging over train/test tasks yields train and test regret:

\[ \overline{\mathrm{Regret}}_{\mathrm{train}}(\theta) = \frac{1}{N_{\mathrm{train}}} \sum_{i=1}^{N_{\mathrm{train}}} \mathrm{Regret}_{t_i}(\theta), \quad \overline{\mathrm{Regret}}_{\mathrm{test}}(\theta) = \frac{1}{N_{\mathrm{test}}} \sum_{j=1}^{N_{\mathrm{test}}} \mathrm{Regret}_{t_j}^{\mathrm{test}}(\theta). \]

An interesting open problem is to connect regret bounds to the structure of the configuration space and task set—for example, whether certain topological properties of C-space (Chapter 1) or symbolic structure in TAMP (Chapter 9) imply smaller worst-case regret for a given planner.

In this section we provide minimal prototyping skeletons in Python, C++, Java, MATLAB/Simulink, and Wolfram Mathematica that:

• Interface with typical robotics libraries in that language, and
• Compute basic metrics such as success rate and average cost.

5.1 Python (e.g., Gymnasium, MuJoCo, PyBullet, MoveIt)

import numpy as np
from typing import Callable, Dict, Any, List

# Example: general evaluation loop for a policy on a set of Gym-like environments.
# You can wrap MuJoCo, PyBullet, Isaac Gym, or MoveIt-based simulations in this interface.

class TaskEnv:
    def __init__(self, env_id: str, make_fn: Callable[[], Any]):
        self.env_id = env_id
        self.make_fn = make_fn

    def new_env(self):
        return self.make_fn()

def evaluate_policy_on_tasks(policy_fn: Callable[[np.ndarray, Dict], np.ndarray],
                             tasks: List[TaskEnv],
                             episodes_per_task: int = 10,
                             max_horizon: int = 200,
                             success_threshold: float = 0.0) -> Dict[str, float]:
    """
    policy_fn(obs, info) -> action
    """
    total_episodes = 0
    total_success = 0
    total_cost = 0.0

    for task in tasks:
        for _ in range(episodes_per_task):
            env = task.new_env()
            obs, info = env.reset()
            done = False
            step = 0
            episode_cost = 0.0
            while not done and step < max_horizon:
                action = policy_fn(obs, info)
                obs, reward, terminated, truncated, info = env.step(action)
                done = terminated or truncated
                # Assume "cost" is negative reward
                episode_cost += -float(reward)
                step += 1

            success = float(info.get("is_success", reward >= success_threshold))
            total_success += success
            total_cost += episode_cost
            total_episodes += 1

    success_rate = total_success / max(total_episodes, 1)
    avg_cost = total_cost / max(total_episodes, 1)
    return {"success_rate": success_rate, "avg_cost": avg_cost}

# Example usage stub:
# tasks = [
#     TaskEnv("PickPlace-v1", lambda: gym.make("PickPlace-v1")),
#     TaskEnv("DrawerOpen-v1", lambda: gym.make("DrawerOpen-v1")),
# ]
# results = evaluate_policy_on_tasks(my_generalist_policy, tasks)
# print(results)
      

5.2 C++ (e.g., ROS 2, OMPL-based Evaluation)

In C++, OMPL and ROS 2 are common platforms for motion planning and manipulation experiments. The following skeleton shows how to wrap a planner and compute success statistics over multiple random start/goal pairs in configuration space.

#include <iostream>
#include <vector>
#include <random>

// OMPL-style includes (assuming it is available in your environment)
#include <ompl/base/SpaceInformation.h>
#include <ompl/base/ProblemDefinition.h>
#include <ompl/base/ScopedState.h>
#include <ompl/geometric/SimpleSetup.h>
#include <ompl/geometric/planners/rrt/RRTstar.h>

namespace ob = ompl::base;
namespace og = ompl::geometric;

struct EvalResult {
    double successRate;
    double avgPathLength;
};

EvalResult evaluatePlanner(og::SimpleSetup &ss,
                           std::size_t numEpisodes,
                           double planningTime)
{
    std::mt19937 rng(42);
    std::uniform_real_distribution<double> uni(0.0, 1.0);

    std::size_t successes = 0;
    double totalPathLength = 0.0;

    auto si = ss.getSpaceInformation();
    auto space = ss.getStateSpace();

    for (std::size_t i = 0; i < numEpisodes; ++i) {
        ob::ScopedState<> start(space), goal(space);
        space->as<ob::RealVectorStateSpace>()->setToRandomState(start.get());
        space->as<ob::RealVectorStateSpace>()->setToRandomState(goal.get());

        ss.clear();
        ss.setStartAndGoalStates(start, goal);
        ss.solve(planningTime);

        if (ss.haveExactSolutionPath()) {
            successes++;
            auto path = ss.getSolutionPath();
            totalPathLength += path.length();
        }
    }

    EvalResult result;
    result.successRate = static_cast<double>(successes) /
                         static_cast<double>(numEpisodes);
    result.avgPathLength = (successes > 0)
                           ? (totalPathLength / static_cast<double>(successes))
                           : 0.0;
    return result;
}

int main()
{
    auto space(std::make_shared<ob::RealVectorStateSpace>(7)); // 7-DOF arm
    ob::RealVectorBounds bounds(7);
    bounds.setLow(-3.14);
    bounds.setHigh(3.14);
    space->setBounds(bounds);

    og::SimpleSetup ss(space);
    ss.setStateValidityChecker(
        [&](const ob::State *state) {
            // TODO: call your collision checker here
            (void) state;
            return true;
        });

    auto planner = std::make_shared<og::RRTstar>(ss.getSpaceInformation());
    ss.setPlanner(planner);

    EvalResult result = evaluatePlanner(ss, 50, 1.0 /*seconds*/);
    std::cout << "Success rate = " << result.successRate
              << ", avg path length = " << result.avgPathLength << std::endl;
    return 0;
}
      

5.3 Java (e.g., ROSJava, Simulation Wrappers)

In Java, robotics support often comes through ROSJava or custom simulation wrappers. The following sketch assumes a generic interface TaskEnvironment with reset() and step() methods, similar to the Python example.

public interface TaskEnvironment {
    Observation reset();
    StepResult step(Action action);
}

public class StepResult {
    public final Observation observation;
    public final double reward;
    public final boolean done;
    public final boolean isSuccess;

    public StepResult(Observation obs, double reward, boolean done, boolean isSuccess) {
        this.observation = obs;
        this.reward = reward;
        this.done = done;
        this.isSuccess = isSuccess;
    }
}

public interface Policy {
    Action act(Observation observation);
}

public class EvalStats {
    public double successRate;
    public double avgCost;
}

public class Evaluator {
    public static EvalStats evaluate(Policy policy,
                                     java.util.List<TaskEnvironment> tasks,
                                     int episodesPerTask,
                                     int maxHorizon) {
        int totalEpisodes = 0;
        int totalSuccess = 0;
        double totalCost = 0.0;

        for (TaskEnvironment task : tasks) {
            for (int i = 0; i < episodesPerTask; i++) {
                Observation obs = task.reset();
                boolean done = false;
                int step = 0;
                double episodeCost = 0.0;

                while (!done && step < maxHorizon) {
                    Action action = policy.act(obs);
                    StepResult result = task.step(action);
                    obs = result.observation;
                    done = result.done;
                    episodeCost += -result.reward;
                    step++;
                    if (done && result.isSuccess) {
                        totalSuccess++;
                    }
                }
                totalCost += episodeCost;
                totalEpisodes++;
            }
        }
        EvalStats stats = new EvalStats();
        stats.successRate = (double) totalSuccess / Math.max(totalEpisodes, 1);
        stats.avgCost = totalCost / Math.max(totalEpisodes, 1);
        return stats;
    }
}
      

5.4 MATLAB / Simulink (Robotics and RL Toolboxes)

MATLAB provides Robotics System Toolbox and Reinforcement Learning Toolbox, with Simulink models for robot dynamics. Below is a script that repeatedly calls a Simulink model and aggregates success statistics, assuming the model exposes a variable is_success in the base workspace.

function stats = evaluate_policy_simulink(modelName, numEpisodes)
% modelName: Simulink model implementing a task (e.g. long-horizon manipulation)
% The model should read policy parameters from the workspace and write:
%   is_success (logical), episode_cost (double)

if nargin < 2
    numEpisodes = 20;
end

totalSuccess = 0;
totalCost = 0.0;

for ep = 1:numEpisodes
    % Randomize initial conditions, goals, etc.
    % For example:
    % goalPose = randomSE3Pose();
    % assignin('base', 'goalPose', goalPose);

    % Run one episode
    simOut = sim(modelName, 'StopTime', 'Tfinal', ...
        'SaveOutput', 'on', 'SrcWorkspace', 'base');

    % Extract logged variables
    is_success = simOut.get('is_success');
    episode_cost = simOut.get('episode_cost');

    totalSuccess = totalSuccess + double(is_success(end) ~= 0);
    totalCost = totalCost + episode_cost(end);
end

stats.successRate = totalSuccess / numEpisodes;
stats.avgCost = totalCost / numEpisodes;
end
      

5.5 Wolfram Mathematica (Symbolic Objectives and Simple Simulations)

Mathematica is useful for symbolic manipulations (e.g., gradients of objectives) and quick Monte Carlo simulations of simplified models.

(* Define symbolic reward function for a parameterized policy theta *)
ClearAll["Global`*"];
H = 10;
gamma = 0.99;

(* Toy one-dimensional dynamics: s_{k+1} = s_k + a_k + noise *)
r[s_, a_] := - (s^2 + 0.1 a^2);

(* Parameterized linear policy a = theta s *)
J[theta_] := Sum[
  gamma^k * Expectation[
    r[s, theta s],
    s \[Distributed] NormalDistribution[0, 1]
  ],
  {k, 0, H - 1}
];

gradJ = D[J[theta], theta] // Simplify

(* Monte Carlo approximation to J(theta) and gradJ for a nonlinear policy *)
simulateReturn[theta_, nEpisodes_Integer : 100] := Module[
  {total = 0.0},
  Do[
    s = 0.0;
    ret = 0.0;
    Do[
      a = theta*s; (* replace by NN policy if desired *)
      rStep = r[s, a];
      ret = ret + gamma^k*rStep;
      s = s + a + RandomVariate[NormalDistribution[0, 0.1]];
      ,
      {k, 0, H - 1}
    ];
    total = total + ret;
    ,
    {nEpisodes}
  ];
  total/nEpisodes
];

Table[{th, simulateReturn[th]}, {th, -1.0, 1.0, 0.5}]
      

Such symbolic and numerical tools can be used in the proposal phase to analyze simplified versions of your open problem before committing to heavy simulations.

The following diagram can serve as a checklist while you refine your own project idea based on the open problems discussed above.

flowchart TD
  S["Start from an open problem (e.g. world model error, task generalization)"]
    --> Q["Formulate as precise question on tasks, mdp, objectives"]
  Q --> B["Check literature: what do Chapters 1-18 and 19 (1-4) already solve?"]
  B --> G["Identify gap: missing guarantee, missing metric, missing algorithm"]
  G --> D["Design method and evaluation loop (Section 5 templates)"]
  D --> C["Compute sample size and metrics (Section 4)"]
  C --> W["Write proposal: problem, hypothesis, method, experiments, risks"]
        

The following problems are designed to help you practice moving from informal research questions to precise mathematical formulations and evaluation plans.

Problem 1 (Bounding Model-Based Planning Error). Consider the setting of Section 2.1 with finite horizon \( H \) and bounded rewards \( |r(s,a)| \le R_{\max} \). Suppose the learned dynamics \( p_{\phi} \) satisfy

\[ \sup_{s,a} \left\| P(\cdot \mid s,a) - p_{\phi}(\cdot \mid s,a) \right\|_{1} \le \delta. \]

Show that the total variation distance between the trajectory distributions \( \mathbb{P}_P \) and \( \mathbb{P}_{\phi} \) corresponding to a fixed policy \( \pi \) and horizon \( H \) satisfies \( \mathrm{TV}(\mathbb{P}_P, \mathbb{P}_{\phi}) \le \tfrac{H \delta}{2} \).

Solution. We only sketch the key steps.

1. For each step \( k \), consider the conditional distributions over \( s_{k+1} \) given \( (s_k,a_k) \) under \( P \) and \( p_{\phi} \). Their \( \ell_1 \) distance is bounded by \( \delta \).
2. Using a coupling argument, construct a joint process \( (s_k, \tilde{s}_k) \) such that the probability that \( s_{k+1} \neq \tilde{s}_{k+1} \) at each step is at most \( \tfrac{\delta}{2} \). (This is a standard property relating \( \ell_1 \) distance and coupling.)
3. By a union bound over \( H \) steps, the probability that the two trajectories differ at some time is at most \( H \cdot \tfrac{\delta}{2} \).
4. The total variation distance between trajectory distributions is at most the probability that coupled trajectories differ, hence

\[ \mathrm{TV}(\mathbb{P}_P, \mathbb{P}_{\phi}) \le \frac{H \delta}{2}. \]

Plugging this into the bound in Section 2.1 yields a more explicit dependence on \( \delta \).

Problem 2 (Sample Size for Success-Rate Estimation). You plan to evaluate a long-horizon manipulation policy on real hardware. Each trial is costly, so you must justify the number of trials \( N \) in your proposal. You want to estimate the success probability \( p \) within \( \alpha = 0.05 \) with confidence \( 1-\delta = 0.95 \). Using the bound in Section 4.1, compute a sufficient value of \( N \).

Solution. From Section 4.1, we require

\[ N \ge \frac{1}{2 \alpha^2} \log\!\left( \frac{2}{\delta} \right). \]

Plugging \( \alpha = 0.05 \) and \( \delta = 0.05 \),

\[ N \ge \frac{1}{2 \cdot 0.05^2} \log\!\left( \frac{2}{0.05} \right) = \frac{1}{0.005} \log(40) \approx 200 \cdot 3.688 \approx 738. \]

So \( N = 750 \) trials is a reasonable rounded choice to report in a proposal. In practice, resource constraints may force you to accept larger \( \alpha \) or smaller \( 1-\delta \).

Problem 3 (Regret-Based Hypothesis Statement). You are comparing a new TAMP algorithm \( \mathcal{A}_{\text{new}} \) with a baseline \( \mathcal{A}_{\text{base}} \) on tasks \( t \). Let \( \mathrm{Regret}_t^{\text{new}} \) and \( \mathrm{Regret}_t^{\text{base}} \) denote their per-task regrets. Formulate a precise statistical hypothesis that your method improves regret, and describe how you would test it using the success-rate evaluation loops from Section 5.

Solution.

• The null hypothesis can be written as

  \[ H_0: \mathbb{E}_{t \sim p(t)}\big[ \mathrm{Regret}_t^{\text{new}} \big] \ge \mathbb{E}_{t \sim p(t)}\big[ \mathrm{Regret}_t^{\text{base}} \big]. \]

• The alternative hypothesis is

  \[ H_1: \mathbb{E}_{t \sim p(t)}\big[ \mathrm{Regret}_t^{\text{new}} \big] < \mathbb{E}_{t \sim p(t)}\big[ \mathrm{Regret}_t^{\text{base}} \big]. \]

• In practice, you approximate expectations with empirical means over a finite test set \( \{t_j\}_{j=1}^{N_{\mathrm{test}}} \) and compute \( \overline{\mathrm{Regret}}^{\text{new}} \) and \( \overline{\mathrm{Regret}}^{\text{base}} \) as in Section 4.2.
• Using the evaluation loops from Section 5, you implement both algorithms as policies and compute per-task costs and regrets, then apply a paired statistical test (e.g., paired t-test) across \( t_j \) to decide whether to reject \( H_0 \).

Problem 4 (From Informal Idea to Formal Proposal Bullet Points). Your informal idea: “Use a language-conditioned world model to make a mobile manipulator reliably execute multi-step household instructions.” Provide three bullet points that turn this into a more formal proposal, each mentioning a precise object (MDP, task set, objective, or metric).

Solution (one possible answer).

• Model each instruction as a task \( t \) with MDP \( M_t = (\mathcal{S}, \mathcal{A}, P_t, r_t, \gamma) \), where the reward \( r_t \) is sparse and only given when the full sequence of sub-goals is completed.
• Learn a language-conditioned world model \( p_{\phi}(s_{k+1} \mid s_k, a_k, x_t) \) and use it in a planning loop to maximize the meta-objective \( J(\theta) \) defined in Section 1.
• Evaluate on a held-out test set of instruction tasks by measuring success rate, average path cost, and empirical regret relative to a TAMP baseline, using the sampling and evaluation schemas of Sections 4 and 5.

In this lesson, you translated frontier topics in advanced robotics into precise research problems. We introduced mathematical abstractions for generalist robot policies and world models, derived basic bounds relating model error and trajectory-value error, and discussed success-rate and regret-based evaluation metrics with explicit sample-complexity guarantees. Multi-language evaluation skeletons (Python, C++, Java, MATLAB/Simulink, and Mathematica) provide a starting point for implementing your own proposals. You should now be ready to formulate a research-grade problem statement and experimental plan for the capstone in Chapter 20.