In classical linear control, we design a feedback law \( u_k = \mathbf{K}\mathbf{x}_k \) (or PID in SISO) assuming a reasonably accurate model of the robot dynamics. In practice, robotic systems often suffer from:

• Unmodeled friction and backlash,
• Changing payloads and wear over time,
• Approximate or linearized models of inherently nonlinear dynamics.

Learning-enabled robotics augments the classical control law with a parametric module that is tuned from data. A basic structure is

\[ u_k = u_k^{\text{fb}} + u_k^{\text{ff}}(\mathbf{x}_k, \mathbf{r}_k; \boldsymbol{\theta}), \]

where \( u_k^{\text{fb}} \) is a stabilizing feedback term (e.g., a linear controller that you already know how to design), and \( u_k^{\text{ff}} \) is a learned feedforward term parameterized by \( \boldsymbol{\theta} \). The parameter vector is updated from data so as to reduce tracking error, energy, or some other cost.

A common architecture is summarized in the following data flow:

flowchart TD
  S["Sensors: x_k, y_k"] --> E["Estimator / Preprocessing"]
  E --> L["Learning module: theta_k"]
  L --> P["Control law: u_k = u_fb + u_ff(theta_k)"]
  P --> A["Actuators"]
  A --> R["Robot dynamics"]
  R --> S
        

In this lesson we restrict attention to shallow, parametric models (e.g., linear regression) and simple update rules (least squares, gradient descent). Deep neural networks and deep reinforcement learning are deferred to advanced courses.

Many learning problems in robotics can be phrased as approximating an unknown function from data. Examples include:

• Inverse dynamics: approximate \( \boldsymbol{\tau} = f(\mathbf{q}, \dot{\mathbf{q}}, \ddot{\mathbf{q}}) \) from measured joint positions, velocities, accelerations and torques.
• Feedforward policy: approximate \( u = \pi(\mathbf{x}, \mathbf{r}) \) that maps state and reference to a nominal control input.
• Residual modeling: approximate the model mismatch \( \Delta(\mathbf{x}, u) \) between a nominal linear model and the true robot dynamics.

Mathematically, suppose we want to learn a scalar function \( f^\star : \mathbb{R}^m \to \mathbb{R} \) from examples \( (\boldsymbol{\phi}_i, y_i) \), where \( \boldsymbol{\phi}_i \in \mathbb{R}^m \) is a feature vector derived from the robot state and reference (e.g., velocities, desired accelerations) and \( y_i \) is a measured quantity (e.g., torque).

A simple linear model approximates

\[ \hat{y}_i = f(\boldsymbol{\phi}_i; \mathbf{w}) = \mathbf{w}^\top \boldsymbol{\phi}_i, \quad \mathbf{w} \in \mathbb{R}^m. \]

Given a dataset of \( N \) samples, we choose \( \mathbf{w} \) to minimize the empirical mean squared error

\[ J(\mathbf{w}) = \frac{1}{N} \sum_{i=1}^N \left( \mathbf{w}^\top \boldsymbol{\phi}_i - y_i \right)^2. \]

Defining the design matrix \( \boldsymbol{\Phi} \in \mathbb{R}^{N \times m} \) whose \( i \)-th row is \( \boldsymbol{\phi}_i^\top \), and \( \mathbf{y} = [y_1,\dots,y_N]^\top \), we can rewrite

\[ J(\mathbf{w}) = \frac{1}{N} \left\| \boldsymbol{\Phi}\mathbf{w} - \mathbf{y} \right\|_2^2. \]

Minimizing this convex quadratic cost leads to closed-form and iterative solutions, which we explore next.

To solve the supervised learning problem, compute the gradient of \( J(\mathbf{w}) \) with respect to \( \mathbf{w} \). Using matrix calculus,

\[ J(\mathbf{w}) = \frac{1}{N} (\boldsymbol{\Phi}\mathbf{w} - \mathbf{y})^\top (\boldsymbol{\Phi}\mathbf{w} - \mathbf{y}). \]

Differentiating gives

\[ \nabla_{\mathbf{w}} J(\mathbf{w}) = \frac{2}{N} \boldsymbol{\Phi}^\top(\boldsymbol{\Phi}\mathbf{w} - \mathbf{y}). \]

Setting the gradient to zero yields the normal equations:

\[ \boldsymbol{\Phi}^\top \boldsymbol{\Phi} \, \mathbf{w}^\star = \boldsymbol{\Phi}^\top \mathbf{y}. \]

If the Gram matrix \( \boldsymbol{\Phi}^\top \boldsymbol{\Phi} \) is invertible (full column rank), the unique minimizer is

\[ \mathbf{w}^\star = \left(\boldsymbol{\Phi}^\top \boldsymbol{\Phi}\right)^{-1} \boldsymbol{\Phi}^\top \mathbf{y}. \]

In a robotics context, this formula gives a batch identification of unknown parameters from collected trajectories. For example, let \( y_i \) be a measured torque and \( \boldsymbol{\phi}_i \) functions of joint position and velocity; then \( \mathbf{w}^\star \) provides estimates of inertial and friction coefficients in a regressor model.

Robots often operate continuously while the environment or payload slowly changes. Instead of recomputing \( \mathbf{w}^\star \) in batch, one may use an online gradient descent update. For each new sample \( (\boldsymbol{\phi}_k, y_k) \), define the instantaneous cost

\[ J_k(\mathbf{w}) = \frac{1}{2} \left( \mathbf{w}^\top \boldsymbol{\phi}_k - y_k \right)^2. \]

The gradient at time step \( k \) is

\[ \nabla_{\mathbf{w}} J_k(\mathbf{w}) = \left( \mathbf{w}^\top \boldsymbol{\phi}_k - y_k \right) \boldsymbol{\phi}_k. \]

The stochastic gradient descent (SGD) update is then

\[ \mathbf{w}_{k+1} = \mathbf{w}_k - \alpha_k \left( \mathbf{w}_k^\top \boldsymbol{\phi}_k - y_k \right) \boldsymbol{\phi}_k, \]

where \( \alpha_k \) is a learning rate (step size). Under standard assumptions (e.g., persistent excitation of \( \boldsymbol{\phi}_k \), decreasing step sizes satisfying \( \sum_k \alpha_k = \infty \) and \( \sum_k \alpha_k^2 < \infty \)), the SGD iterates converge in mean square to the least-squares solution.

For a scalar unknown parameter \( \theta \), the update simplifies to

\[ \theta_{k+1} = \theta_k - \alpha_k \, \phi_k \left( \theta_k \phi_k - y_k \right), \]

which coincides with the classical gradient adaptive law in adaptive control for linear-in-parameters models.

Consider a simple discrete-time model of a 1-DOF joint with viscous friction:

\[ x_{k+1} = x_k + \Delta t \left( u_k - a x_k \right), \]

where \( x_k \) is the joint velocity, \( u_k \) the commanded torque-like input, \( a > 0 \) an unknown friction coefficient, and \( \Delta t \) the sampling period.

Suppose we design a proportional controller to track a reference velocity \( x_k^{\text{ref}} \):

\[ u_k^{\text{fb}} = -K \left( x_k - x_k^{\text{ref}} \right), \quad K > 0. \]

We augment this with a learned feedforward term \( u_k^{\text{ff}} = \hat{a}_k x_k \), where \( \hat{a}_k \) approximates the friction coefficient \( a \). The total control law becomes

\[ u_k = -K \left( x_k - x_k^{\text{ref}} \right) + \hat{a}_k x_k. \]

To adapt \( \hat{a}_k \), define the tracking error \( e_k = x_k - x_k^{\text{ref}} \) and consider the cost

\[ J_k(\hat{a}) = \frac{1}{2} e_k^2. \]

Assuming \( x_k \) and \( x_k^{\text{ref}} \) are measured, we can compute \( e_k \) and adapt \( \hat{a}_k \) by gradient descent. Treating \( e_k \) as depending on \( \hat{a}_k \) through \( u_k \), and using a simple approximation, one often uses an update proportional to \( e_k x_k \):

\[ \hat{a}_{k+1} = \hat{a}_k - \gamma e_k x_k, \quad \gamma > 0. \]

Here \( \gamma \) is a learning gain. For sufficiently small \( \gamma \) and a stabilizing feedback gain \( K \), one can show, using standard adaptive control arguments, that the tracking error \( e_k \) remains bounded, and often converges toward zero as \( \hat{a}_k \) approaches the true friction coefficient.

Conceptually, the control architecture is:

flowchart TD
  R["Reference x_ref"] --> SUM["Error e_k"]
  X["Measured x_k"] --> SUM
  SUM --> C["Feedback controller u_fb"]
  X --> L["Learning block: a_hat_k"]
  L --> FF["Feedforward u_ff = a_hat_k * x_k"]
  C --> U["Sum u_k"]
  FF --> U
  U --> PLANT["Joint dynamics"]
  PLANT --> X
        

The following Python script simulates the adaptive velocity controller described above. It uses a simple Euler integration of the dynamics and online gradient adaptation of \( \hat{a}_k \).

import numpy as np

# Simulation parameters
dt = 0.001
T = 5.0
steps = int(T / dt)

# True friction coefficient
a_true = 2.0

# Controller and learning gains
K = 10.0           # feedback gain
gamma = 5.0        # learning rate for a_hat

# Desired reference velocity (step)
def x_ref(t):
    return 1.0 if t > 0.5 else 0.0

x = 0.0            # initial velocity
a_hat = 0.0        # initial estimate of friction

xs = []
xrefs = []
ahat_hist = []

for k in range(steps):
    t = k * dt
    xr = x_ref(t)
    e = x - xr

    # Control law: u = -K e + a_hat * x
    u_fb = -K * e
    u_ff = a_hat * x
    u = u_fb + u_ff

    # True dynamics: x_{k+1} = x_k + dt * (u - a_true * x_k)
    x_dot = u - a_true * x
    x = x + dt * x_dot

    # Parameter adaptation: a_hat_{k+1} = a_hat_k - gamma * e * x
    a_hat = a_hat - gamma * e * x * dt  # dt can be absorbed into gamma

    xs.append(x)
    xrefs.append(xr)
    ahat_hist.append(a_hat)

# Optional: plot results (requires matplotlib)
if __name__ == "__main__":
    import matplotlib.pyplot as plt

    tgrid = np.arange(steps) * dt
    plt.figure()
    plt.plot(tgrid, xs, label="x")
    plt.plot(tgrid, xrefs, "--", label="x_ref")
    plt.xlabel("time [s]")
    plt.ylabel("velocity")
    plt.legend()

    plt.figure()
    plt.plot(tgrid, ahat_hist, label="a_hat")
    plt.axhline(a_true, linestyle="--", label="a_true")
    plt.xlabel("time [s]")
    plt.ylabel("friction estimate")
    plt.legend()
    plt.show()
      

This illustrates how a simple learning rule embedded in a feedback loop can compensate for unknown friction. The structure remains shallow and interpretable, in contrast to deep neural networks.

Below is a C++-style pseudocode implementation of the same adaptive controller. In a real robot, this update would be executed inside a hard real-time control loop.

#include <iostream>

int main() {
    double dt = 0.001;
    double T  = 5.0;
    int steps = static_cast<int>(T / dt);

    double a_true = 2.0;
    double K      = 10.0;
    double gamma  = 5.0;

    auto x_ref = [](double t) {
        return (t > 0.5) ? 1.0 : 0.0;
    };

    double x     = 0.0;
    double a_hat = 0.0;

    for (int k = 0; k < steps; ++k) {
        double t  = k * dt;
        double xr = x_ref(t);
        double e  = x - xr;

        double u_fb = -K * e;
        double u_ff = a_hat * x;
        double u    = u_fb + u_ff;

        // Plant dynamics
        double x_dot = u - a_true * x;
        x += dt * x_dot;

        // Adaptive update
        a_hat -= gamma * e * x * dt;
    }

    std::cout << "Final a_hat = " << a_hat << std::endl;
    return 0;
}
      

To interface this with an actual robot, one would replace the simple plant integration by sending u to the actuator and reading the measured velocity from sensors (e.g., encoders), while preserving the same adaptation law.

8.1 Java (Simulation Loop)

public class AdaptiveServo {
    public static void main(String[] args) {
        double dt = 0.001;
        double T = 5.0;
        int steps = (int)(T / dt);

        double aTrue = 2.0;
        double K = 10.0;
        double gamma = 5.0;

        double x = 0.0;
        double aHat = 0.0;

        for (int k = 0; k < steps; ++k) {
            double t = k * dt;
            double xRef = (t > 0.5) ? 1.0 : 0.0;
            double e = x - xRef;

            double uFb = -K * e;
            double uFf = aHat * x;
            double u = uFb + uFf;

            double xDot = u - aTrue * x;
            x += dt * xDot;

            aHat -= gamma * e * x * dt;
        }

        System.out.println("Final aHat = " + aHat);
    }
}
      

8.2 MATLAB Script with Simulink-Oriented Structure

In MATLAB/Simulink, the dynamics and controller can be implemented either as blocks or using an S-function. Below is a pure MATLAB script that mirrors a single-step update that one could place in a MATLAB Function block within Simulink.

% Parameters
dt = 0.001;
T  = 5.0;
steps = floor(T/dt);

a_true = 2.0;
K      = 10.0;
gamma  = 5.0;

x     = 0.0;
a_hat = 0.0;

x_hist = zeros(steps,1);
a_hist = zeros(steps,1);

for k = 1:steps
    t  = (k-1)*dt;
    if t > 0.5
        x_ref = 1.0;
    else
        x_ref = 0.0;
    end

    e   = x - x_ref;
    u_fb = -K * e;
    u_ff = a_hat * x;
    u    = u_fb + u_ff;

    x_dot = u - a_true * x;
    x = x + dt * x_dot;

    a_hat = a_hat - gamma * e * x * dt;

    x_hist(k) = x;
    a_hist(k) = a_hat;
end

% Plot (optional)
figure; plot((0:steps-1)*dt, x_hist, (0:steps-1)*dt, x_hist*0 + 1.0, '--');
xlabel('time [s]'); ylabel('velocity'); legend('x','x\_ref');

figure; plot((0:steps-1)*dt, a_hist); hold on;
yline(a_true,'--','a\_true');
xlabel('time [s]'); ylabel('a\_hat');
legend('a\_hat','a\_true');
      

In Simulink, one could realize this scheme by:

1. A continuous or discrete-time integrator block implementing the plant dynamics,
2. Gain blocks for K and gamma,
3. A unit delay or memory block storing a_hat,
4. A MATLAB Function block implementing the adaptation law a_hat_next = a_hat - gamma * e * x * dt;.

In this lesson, we framed learning-enabled robotics as the embedding of shallow, parametric learning modules into otherwise classical feedback loops. We derived least-squares and gradient descent solutions to basic function approximation problems relevant to inverse dynamics and feedforward control, and we illustrated an adaptive compensator for unknown viscous friction. Code examples in Python, C++, Java, and MATLAB/Simulink demonstrated that such learning laws are easy to implement within existing control infrastructures, without resorting to deep reinforcement learning.

In the next lesson, we will examine how to reason about validation, generalization, and failure modes when deploying such learned components on physical robots.