Consider a robot with continuous-time state \( x(t)\in\mathbb{R}^n \) and control input \( u(t)\in\mathbb{R}^m \). A common state-space model is

\[ \dot{x}(t) = f(x(t), u(t)) + d(x(t), u(t)), \quad y(t) = h(x(t)), \]

where \( f \) is a nominal physics-based model (e.g., rigid-body dynamics) and \( d \) collects unmodeled effects: friction, flexibilities, sensor bias, environment interaction, etc.

In Chapter 3 and Chapter 4 you designed controllers based on \( f \) only, for example:

\[ u(t) = u_{\text{nom}}(x(t), r(t)) \quad \text{designed for} \quad \dot{x}(t) = f(x(t), u(t)), \]

where \( r(t) \) is a reference trajectory. In reality, the closed loop sees the disturbance term \( d \), and the true error dynamics are

\[ \dot{e}(t) = \dot{x}(t) - \dot{r}(t) = F_c(e(t), r(t)) + \Delta(e(t), r(t)), \]

where \( F_c \) is the error dynamics under the nominal model and \( \Delta \) aggregates the effect of \( d \) and model mismatch. Classical robust and adaptive control try to bound or adapt to this disturbance.

The key idea of learning-augmented control is:

Instead of only bounding unknown effects, we learn a predictive model of them from data and use this model to improve tracking, energy usage, and robustness, while keeping a proven nominal controller in the loop.

For many robot controllers, a discrete-time description is convenient. With sampling period \( T_s > 0 \), write

\[ x_{k+1} = f_d(x_k, u_k) + d_k, \quad y_k = h_d(x_k), \]

where \( d_k \) is the discrete-time disturbance (including modeling error). Suppose a state-feedback controller \( u_k = \kappa(x_k, r_k) \) has been designed (e.g., PD, computed torque, LQR, or MPC from earlier chapters). The closed-loop state-error dynamics can be written as

\[ e_{k+1} = A_c e_k + \phi(e_k, r_k) + w_k, \]

where \( A_c \) is the linear part of the closed-loop dynamics around a trajectory, \( \phi \) collects higher-order terms, and \( w_k \) is induced by \( d_k \). Under standard design assumptions (local linearization, PD/LQR, etc.) we often ensure:

\[ \rho(A_c) < 1 \quad\Rightarrow\quad \exists K\ge 1,\, 0<\lambda<1: \;\|A_c^k\|\le K\lambda^k, \;\forall k\in\mathbb{N}, \]

and for bounded disturbances \( \|w_k\|\le \bar{w} \), we get the classical ISS-type bound

\[ \|e_k\| \le K\lambda^k\|e_0\| + \frac{K}{1-\lambda}\bar{w}. \]

A learning module aims to reduce \( \bar{w} \) by predicting and compensating \( d_k \). Let a learned model \( \hat{d}_k = \hat{d}(x_k, u_k; \theta) \) be trained from data \( (x_k, u_k, d_k) \). We define the residual

\[ r_k = d_k - \hat{d}_k, \quad \|r_k\|\le \alpha \bar{w}, \quad 0\le\alpha<1, \]

which measures how much uncertainty remains after learning. If the controller is augmented with

\[ u_k = \kappa(x_k, r_k) + u_{\text{learn}}(x_k, r_k), \quad u_{\text{learn}}(x_k, r_k)\approx -\hat{d}_k, \]

then the new disturbance entering the error dynamics is \( r_k \) instead of \( d_k \). The same ISS argument now yields

\[ \|e_k\| \le K\lambda^k\|e_0\| + \frac{K}{1-\lambda}\alpha\bar{w}, \]

which shows that, if the learning component reduces the disturbance bound (small &strong;\( \alpha \)), the asymptotic tracking error shrinks proportionally, without changing the nominal stability margin determined by \( A_c \).

flowchart TD
  P["Robot + unknown dynamics d(x,u)"] --> C0["Nominal controller kappa(x,r)"]
  C0 --> CL0["Closed-loop error e_k (baseline)"]
  CL0 --> DATA["Collect data (x_k, u_k, d_k)"]
  DATA --> LEARN["Train residual model hat_d(x,u;theta)"]
  LEARN --> AUG["Augmented control u = kappa + u_learn"]
  AUG --> CL1["New closed loop with smaller disturbance r_k"]
  CL1 --> PERF["Improved tracking / lower energy / robustness"]
        

There are several generic ways to combine learning with control. Three important non-RL patterns are:

1. Model Refinement: learn a better dynamics model \( \hat{f} \) or disturbance model \( \hat{d} \) and use it inside a model-based controller (computed torque, MPC, etc.).
2. Residual / Add-on Controller: keep a provably stable nominal controller \( \kappa \) and add a learned term \( u_{\text{learn}} \) to reduce error.
3. Gain / Parameter Scheduling: learn how controller gains or parameters should change with operating condition, e.g., \( K(x) \) instead of a constant gain \( K \).

A generic residual architecture can be written as

\[ u(t) = u_{\text{nom}}(x(t), r(t)) + u_{\text{res}}(x(t), \xi(t)), \quad u_{\text{res}}(x,\xi) = \mathcal{L}(z(x,\xi);\theta), \]

where \( \mathcal{L} \) is a learned function (e.g., regressor, Gaussian process, neural network) with input \( z(x,\xi) \) (features such as velocity, acceleration, estimated disturbance, etc.) and parameters \( \theta \). The vector \( \xi \) may contain internal controller or estimator states.

To keep stability, one designs the nominal controller so that \( u_{\text{res}} \) acts as a bounded perturbation. A common requirement is a gain bound

\[ \|\mathcal{L}(z;\theta)\| \le \gamma\|z\| + \delta, \quad \text{with } 0\le\gamma < \gamma_{\max}, \]

where \( \gamma_{\max} \) is determined by small-gain or Lyapunov arguments for the nominal closed loop. Later lessons will show concrete constructions for residual learning and Gaussian-process error compensation.

We now formalize the intuition from Section 2 in a simple result for linearized discrete-time error dynamics with additive uncertainty.

Theorem 1 (Effect of Learned Disturbance Reduction).

Consider the error system \( e_{k+1} = A_c e_k + w_k \) with \( \rho(A_c) < 1 \) and disturbances satisfying \( \|w_k\|\le \bar{w} \). Then:

1. There exist constants \( K\ge 1 \), \( 0<\lambda<1 \) such that \( \|e_k\|\le K\lambda^k\|e_0\| + \frac{K}{1-\lambda}\bar{w} \).
2. Suppose we construct a learned compensator so that the new disturbance is \( r_k = w_k - \hat{w}_k \) with \( \|r_k\|\le \alpha\bar{w} \) for some \( 0\le\alpha<1 \). Then the augmented closed loop satisfies \( \|e_k\|\le K\lambda^k\|e_0\| + \frac{K}{1-\lambda}\alpha\bar{w} \).

Sketch of proof.

The solution of \( e_{k+1} = A_c e_k + w_k \) is

\[ e_k = A_c^k e_0 + \sum_{i=0}^{k-1} A_c^{k-1-i} w_i. \]

Using submultiplicativity of the norm,

\[ \|e_k\| \le \|A_c^k\|\|e_0\| + \sum_{i=0}^{k-1}\|A_c^{k-1-i}\|\|w_i\| \le K\lambda^k\|e_0\| + K\bar{w}\sum_{j=0}^{k-1}\lambda^j \le K\lambda^k\|e_0\| + \frac{K}{1-\lambda}\bar{w}. \]

For the learned case, the same argument applies with \( \|r_k\|\le\alpha\bar{w} \), simply replacing \( \bar{w} \) by \( \alpha\bar{w} \), which proves the second claim.

The theorem shows that any improvement in disturbance prediction (smaller \( \alpha \)) directly improves the asymptotic error bound, while the stability (through \( A_c \)) is unchanged as long as the learned term is treated as a bounded disturbance.

flowchart TD
  S["Start from stable nominal closed loop"] --> D1["Model unknown w_k as bounded disturbance"]
  D1 --> L1["Fit predictor hat_w_k from data"]
  L1 --> B1["Guarantee error ||w_k - hat_w_k|| <= alpha * w_bar"]
  B1 --> SG["Check small-gain / Lyapunov with alpha"]
  SG --> RES["New ISS bound with reduced asymptotic error"]
        

Consider a single revolute joint with generalized coordinate \( q \). The dynamics are

\[ M(q)\ddot{q} + C(q,\dot{q})\dot{q} + g(q) + \tau_f(\dot{q}) = \tau, \]

where \( M > 0 \) is inertia, \( C(q,\dot{q}) \) Coriolis/centripetal term, \( g(q) \) gravity, and \( \tau_f(\dot{q}) \) friction torque. A computed-torque controller ignoring friction uses

\[ \tau_{\text{nom}} = \hat{M}(q)v + \hat{C}(q,\dot{q})\dot{q} + \hat{g}(q), \quad v = \ddot{q}_r - K_d\tilde{\dot{q}} - K_p\tilde{q}, \]

where \( \tilde{q} = q - q_r \) and \( \tilde{\dot{q}} = \dot{q} - \dot{q}_r \). The resulting error dynamics (ignoring parameter mismatch) are

\[ M(q)\ddot{\tilde{q}} + K_d\dot{\tilde{q}} + K_p\tilde{q} = -\tau_f(\dot{q}), \]

so friction acts as a disturbance. Suppose we collect data \( (\dot{q}^{(i)}, \tau_f^{(i)}) \) and fit a model \( \hat{\tau}_f(\dot{q}) \). We then define the augmented torque

\[ \tau = \tau_{\text{nom}} + \tau_{\text{learn}}, \quad \tau_{\text{learn}}(\dot{q}) = \hat{\tau}_f(\dot{q}), \]

which yields the (idealized) error dynamics

\[ M(q)\ddot{\tilde{q}} + K_d\dot{\tilde{q}} + K_p\tilde{q} = -\bigl(\tau_f(\dot{q}) - \hat{\tau}_f(\dot{q})\bigr). \]

If \( \|\tau_f(\dot{q}) - \hat{\tau}_f(\dot{q})\|\le \alpha\bar{\tau}_f \) with \( \alpha<1 \), the same analysis as in Section 4 shows reduced tracking error bounds. Here, learning effectively replaces a crude friction bound by a data-driven approximation.

Below is a Python-style sketch combining a nominal PD/computed-torque controller with a learned friction model. For robotics dynamics one can use, e.g., roboticstoolbox or pinocchio; for learning, common choices are scikit-learn or torch.

import numpy as np
# Nominal robot dynamics (e.g., from roboticstoolbox or your own model)
def M(q):
    return 1.2  # scalar inertia for a simple example

def C(q, qd):
    return 0.0

def g(q):
    return 0.0

# PD gains
Kp = 50.0
Kd = 10.0

# Placeholder for a trained friction model (e.g., sklearn or torch)
class LearnedFrictionModel:
    def __init__(self):
        # In practice, load trained parameters here
        self.w = np.array([0.2, 0.05])  # dummy linear model params

    def predict(self, qd):
        # Features: [qd, sign(qd)]
        phi = np.array([qd, np.sign(qd)])
        return float(self.w @ phi)

fric_model = LearnedFrictionModel()

def joint_controller(q, qd, q_ref, qd_ref, qdd_ref):
    # Nominal computed torque ignoring friction
    q_tilde = q - q_ref
    qd_tilde = qd - qd_ref
    v = qdd_ref - Kd * qd_tilde - Kp * q_tilde
    tau_nom = M(q) * v + C(q, qd) * qd + g(q)

    # Learned friction compensation
    tau_learn = fric_model.predict(qd)

    tau = tau_nom + tau_learn
    return tau

# Example closed-loop simulation step
def step(q, qd, q_ref, qd_ref, qdd_ref, dt=0.001):
    tau = joint_controller(q, qd, q_ref, qd_ref, qdd_ref)
    # True dynamics with unknown friction tau_f(qd) = 0.3*qd + 0.1*sign(qd)
    tau_f_true = 0.3 * qd + 0.1 * np.sign(qd)
    qdd = (tau - tau_f_true) / M(q)
    qd_next = qd + dt * qdd
    q_next = q + dt * qd_next
    return q_next, qd_next
      

In a full implementation you would:

• Log data (qd, tau, qdd) from a baseline controller.
• Regress tau_f as a function of qd using a flexible model (e.g., Gaussian process, neural network).
• Re-run the controller with the learned model, monitoring tracking error and ensuring that the added term remains bounded.

7.1 C++ (Eigen + simple linear friction model)

#include <iostream>
#include <Eigen/Dense>

// Simple 1-DOF joint; in practice you may integrate with ROS control and Pinocchio/RBDL
double M(double q) { return 1.2; }
double C(double q, double qd) { return 0.0; }
double g(double q) { return 0.0; }

// Learned friction model: tau_f_hat(qd) = w0 * qd + w1 * sign(qd)
struct LearnedFriction {
    Eigen::Vector2d w;
    LearnedFriction() {
        w << 0.25, 0.08; // trained offline
    }
    double predict(double qd) const {
        double s = (qd > 0.0) ? 1.0 : (qd < 0.0 ? -1.0 : 0.0);
        Eigen::Vector2d phi(qd, s);
        return w.dot(phi);
    }
};

double jointController(double q, double qd,
                       double q_ref, double qd_ref, double qdd_ref,
                       const LearnedFriction& model)
{
    const double Kp = 50.0;
    const double Kd = 10.0;
    double q_tilde = q - q_ref;
    double qd_tilde = qd - qd_ref;
    double v = qdd_ref - Kd * qd_tilde - Kp * q_tilde;
    double tau_nom = M(q) * v + C(q, qd) * qd + g(q);
    double tau_learn = model.predict(qd);
    return tau_nom + tau_learn;
}
      

7.2 Java (simple gain-scheduled PD based on learned table)

public class LearnedPdJointController {
    // Example: gain scheduling table learned from data
    // (e.g., Map from velocity bin to extra damping term)
    private final double[] velBins = {-5.0, -1.0, 1.0, 5.0};
    private final double[] extraKd = {4.0, 1.0, 1.0, 4.0};

    private double baseKp = 50.0;
    private double baseKd = 10.0;

    private double lookupExtraKd(double qd) {
        // naive piecewise-constant schedule
        for (int i = 0; i < velBins.length - 1; ++i) {
            if (qd >= velBins[i] && qd < velBins[i + 1]) {
                return extraKd[i];
            }
        }
        return extraKd[extraKd.length - 1];
    }

    public double computeTorque(double q, double qd,
                                double qRef, double qdRef, double qddRef)
    {
        double qTilde = q - qRef;
        double qdTilde = qd - qdRef;
        double KdEff = baseKd + lookupExtraKd(qd); // learned scheduling
        double v = qddRef - KdEff * qdTilde - baseKp * qTilde;
        double M = 1.2; // nominal
        return M * v;   // ignoring Coriolis and gravity in this sketch
    }
}
      

7.3 MATLAB/Simulink (data-driven friction fit and use in control)

% Suppose we collected data from a baseline controller:
% qd_log: joint velocity samples, tau_log: measured actuator torques
% We estimate a simple polynomial friction model tau_f(qd) = a1*qd + a3*qd^3

coeffs = polyfit(qd_log, tau_log, 3);  % [a3 a2 a1 a0]

% Controller function (e.g., in a MATLAB Function block in Simulink)
function tau = joint_controller(q, qd, q_ref, qd_ref, qdd_ref, coeffs)
    Kp = 50; Kd = 10;
    q_tilde  = q  - q_ref;
    qd_tilde = qd - qd_ref;
    v = qdd_ref - Kd * qd_tilde - Kp * q_tilde;
    M = 1.2; C = 0; G = 0;

    tau_nom = M * v + C * qd + G;
    % Learned friction compensation
    a3 = coeffs(1); a2 = coeffs(2); a1 = coeffs(3); a0 = coeffs(4);
    tau_f_hat = a3*qd.^3 + a2*qd.^2 + a1*qd + a0;

    tau = tau_nom + tau_f_hat;
end
      

In Simulink, the nominal dynamics can be modeled with standard blocks or the Robotics System Toolbox, while the friction model appears as a subsystem implementing the polynomial.

7.4 Wolfram Mathematica (symbolic Lyapunov check with learned bound)

(* PD-controlled mass with bounded learned disturbance *)
Clear["Global`*"];
m = 1.2;
Kp = 50; Kd = 10;
q[t_] := q[t];
qr[t_] := 0;

eq = m*q''[t] + Kd*q'[t] + Kp*q[t] == -w[t];
(* Assume learned model reduces disturbance bound: |w[t]| <= alpha * wBar *)
alpha = 0.3; wBar = Symbol["wBar"];

(* Define Lyapunov candidate V = 1/2*(Kp*q^2 + m*qdot^2) *)
V[q_, qdot_] := 1/2 (Kp*q^2 + m*qdot^2);
dVdt = D[V[q[t], q'[t]], t] /. eq // Simplify

(* dV/dt = -Kd*qdot^2 + qdot*w[t] *)
(* Use bound |w[t]| <= alpha * wBar to derive inequality-based conditions *)
      

Mathematica is well-suited to manipulate Lyapunov functions and derive symbolic bounds that include learned disturbance bounds (here encoded in alpha).

Problem 1 (Effect of Residual Error). Consider the discrete-time system \( e_{k+1} = A_c e_k + w_k \) with \( \rho(A_c) < 1 \) and \( \|w_k\|\le \bar{w} \). A learned compensator yields \( r_k = w_k - \hat{w}_k \) with \( \|r_k\|\le \alpha\bar{w} \), \( 0\le\alpha<1 \). Show that the ratio of asymptotic error bounds with and without learning is at most \( \alpha \).

Solution.

From Section 4, the closed-loop bound without learning is

\[ \limsup_{k\to\infty}\|e_k\| \le \frac{K}{1-\lambda}\bar{w} =: B_{\text{no learn}}. \]

With learning, the disturbance is replaced by \( r_k \) with \( \|r_k\|\le \alpha\bar{w} \), so

\[ \limsup_{k\to\infty}\|e_k\| \le \frac{K}{1-\lambda}\alpha\bar{w} = \alpha B_{\text{no learn}}. \]

Therefore the ratio of asymptotic bounds is \( B_{\text{learn}} / B_{\text{no learn}} \le \alpha \), so any \( \alpha<1 \) strictly improves the worst-case error bound.

Problem 2 (Nominal Stability Requirement). Consider the 1-DOF joint in Section 5 with error dynamics

\[ M(q)\ddot{\tilde{q}} + K_d\dot{\tilde{q}} + K_p\tilde{q} = -\eta(t), \]

where \( \eta(t) = \tau_f(\dot{q}) - \hat{\tau}_f(\dot{q}) \). Assume \( M(q)\ge m_{\min}>0 \) and \( K_p, K_d > 0 \). Show that the nominal system (\( \eta(t)=0 \)) is globally asymptotically stable and that the perturbed system is input-to-state stable (ISS) with respect to \( \eta \).

Solution.

For the nominal system \( \eta(t)=0 \), choose Lyapunov function

\[ V(\tilde{q},\dot{\tilde{q}}) = \tfrac{1}{2}M(q)\dot{\tilde{q}}^2 + \tfrac{1}{2}K_p\tilde{q}^2. \]

Differentiating and using the dynamics with \( \eta=0 \) and standard rigid-body properties (skew-symmetry of \( \dot{M} - 2C \)) yields

\[ \dot{V} = -K_d\dot{\tilde{q}}^2 \le 0, \]

and the largest invariant set in \( \{\dot{V}=0\} \) is \( \tilde{q} = \dot{\tilde{q}} = 0 \). By LaSalle, the origin is globally asymptotically stable. For the perturbed system,

\[ \dot{V} = -K_d\dot{\tilde{q}}^2 + \dot{\tilde{q}}\eta(t), \]

and Young's inequality gives, for any \( \varepsilon > 0 \),

\[ \dot{V} \le -(K_d-\varepsilon)\dot{\tilde{q}}^2 + \frac{1}{4\varepsilon}\eta(t)^2. \]

Taking \( \varepsilon < K_d \) shows ISS with respect to \( \eta \). If learning reduces \( \|\eta\|_{\infty} \), the ISS gain decreases accordingly.

Problem 3 (Design Flow for Learning-Augmented Control). Sketch a design flow for adding a learned residual controller to an existing stabilizing controller, ensuring you do not invalidate stability.

Solution (conceptual flow).

flowchart TD
  S["Specify task and performance metrics"] --> N["Design stabilizing nominal controller"]
  N --> B["Analyze stability margin and disturbance bounds"]
  B --> D["Decide what to learn (residual, friction, disturbance)"]
  D --> E["Collect data under safe nominal controller"]
  E --> L["Train and validate predictor with error bounds"]
  L --> C["Check small-gain / Lyapunov conditions with learned term"]
  C --> DEP["Deploy with online monitoring and fallback to nominal"]
        

Problem 4 (Trade-off Between Robustness and Learning). Suppose a robust controller is designed for disturbance bound \( \bar{w}_{\max} \). Learning reduces the estimated disturbance to \( \bar{w}_{\text{learn}} \) but there is risk of extrapolation error if the robot leaves the training region. Explain why it is wise to preserve the original robust margin rather than redesign the nominal controller under the smaller bound \( \bar{w}_{\text{learn}} \).

Solution.

The robust controller guarantees stability for all disturbances with \( \|w_k\|\le \bar{w}_{\max} \). Learning reduces the typical disturbance magnitude but may fail under unmodeled conditions, leading to larger-than-expected residuals. If we redesign the nominal controller using the smaller bound \( \bar{w}_{\text{learn}} \), the closed loop might lose stability when the learned model extrapolates. Keeping the original robust design and viewing learning as a performance enhancer preserves global robustness while still benefiting from smaller average disturbances.

In this lesson we motivated learning-augmented control from the perspective of model mismatch in robotics, formalized nominal closed-loop error dynamics under bounded disturbances, and showed how learned disturbance or residual models can provably reduce error bounds without compromising stability, provided the nominal controller remains in charge of stability. We examined a 1-DOF joint example where friction is learned and compensated and sketched implementations in Python, C++, Java, MATLAB/Simulink, and Mathematica. Subsequent lessons will instantiate these ideas with concrete residual learning and Gaussian process models, and provide more detailed stability-preserving wrappers around learned components.