Up to the industrial era (Lesson 2), most robots were controlled by explicit programs and engineered models. Let the robot state be \( x(t) \in \mathbb{R}^n \) and control input \( u(t) \in \mathbb{R}^m \). Classical control assumes a known model \( \dot{x}=f(x,u) \) (or linearized \( \dot{x} = A x + B u \)).

The modern boom began when robots gained the ability to learn unknown mappings from data. In AI-driven robotics, we introduce a parameterized function \( \hat{f}_\theta \) and adjust parameters \( \theta \in \mathbb{R}^p \) from examples rather than only from physics.

flowchart TD
  A["Industrial robots (pre-1990s): fixed programs"] --> B["Sensors become cheaper & richer"]
  B --> C["Large datasets from robots & simulation"]
  C --> D["Scalable compute (GPUs, clusters)"]
  D --> E["Learning algorithms mature"]
  E --> F["Modern robotics boom: perception + autonomy + adaptation"]
        

Historically, three AI capabilities changed what robots could do:

• Perception from high-dimensional sensors (vision, depth, LiDAR).
• Decision-making under uncertainty (task selection, navigation, manipulation).
• Adaptation and skill learning (learning from demonstrations, trial and error).

We now formalize these capabilities mathematically.

Many robotics problems can be written as learning an input–output map. Given training data \( \mathcal{D}=\{(x_i,y_i)\}_{i=1}^N \), the goal is to approximate \( y \approx g(x) \) with a model \( g_\theta(x) \). The standard empirical risk minimization problem is

\[ \theta^\star = \arg\min_{\theta} \; J(\theta), \quad J(\theta) = \frac{1}{N}\sum_{i=1}^N \ell\!\left(g_\theta(x_i),y_i\right) \]

where \( \ell \) is a loss function. For regression, a common choice is squared loss:

\[ \ell(g_\theta(x_i),y_i)=\|g_\theta(x_i)-y_i\|_2^2 \]

Example (linear model). Suppose \( g_\theta(x)=\theta^\top x \), with \( x \in \mathbb{R}^d \). Then

\[ J(\theta)=\frac{1}{N}\sum_{i=1}^N (\theta^\top x_i - y_i)^2 \]

Closed-form solution. Let \( X=[x_1^\top;\dots;x_N^\top] \in \mathbb{R}^{N\times d} \), \( y=[y_1,\dots,y_N]^\top \). Then

\[ J(\theta)=\frac{1}{N}\|X\theta-y\|_2^2 \]

Setting the gradient to zero:

\[ \nabla_\theta J(\theta) = \frac{2}{N}X^\top(X\theta-y)=0 \;\Rightarrow\; X^\top X\theta = X^\top y \]

If \( X^\top X \) is invertible, the minimizer is

\[ \theta^\star = (X^\top X)^{-1}X^\top y \]

When closed forms are unavailable, we use gradient descent:

\[ \theta_{k+1}=\theta_k-\eta \nabla_\theta J(\theta_k), \quad \eta > 0 \]

Efficient large-scale optimization is a major driver of AI-enabled robotics.

Deep learning provides a hierarchical approximation \( g_\theta = g_L \circ \cdots \circ g_1 \):

\[ h_0=x,\quad h_{\ell}= \sigma(W_\ell h_{\ell-1}+b_\ell), \quad g_\theta(x)=h_L \]

Convolutional structure (conceptual math). For an image \( I \in \mathbb{R}^{H\times W} \) and kernel \( K\in\mathbb{R}^{r\times r} \), a convolution layer computes

\[ (I*K)(i,j)=\sum_{a=0}^{r-1}\sum_{b=0}^{r-1} K(a,b)\,I(i+a,j+b) \]

Weight sharing reduces parameters from \( O(HW) \) to \( O(r^2) \), enabling learning on limited robot data.

Reinforcement learning (RL) learns control laws when accurate models are unavailable. Consider discrete-time dynamics \( x_{t+1}=f(x_t,u_t) \). The robot maximizes expected return:

\[ G_t = \sum_{k=0}^{\infty} \gamma^k r_{t+k+1}, \quad 0 < \gamma < 1 \]

A policy \( \pi(u|x) \) induces the value function \( V^\pi(x)=\mathbb{E}_\pi[G_t \,|\, x_t=x] \). The optimal value satisfies the Bellman optimality equation:

\[ V^\star(x)=\max_{u}\Big( r(x,u)+\gamma V^\star(f(x,u)) \Big) \]

Proof sketch (contraction). Define the Bellman operator \( (TV)(x)=\max_u\{r(x,u)+\gamma V(f(x,u))\} \). For any bounded \( V_1,V_2 \),

\[ \|TV_1-TV_2\|_\infty \le \gamma \|V_1-V_2\|_\infty \]

because the max cannot increase differences and \( \gamma < 1 \). Thus \( T \) is a contraction mapping, and by Banach’s fixed-point theorem value iteration converges to the unique fixed point \( V^\star \).

flowchart TD
  S["State x_t"] --> P["Policy pi(u|x)"]
  P --> A["Action u_t"]
  A --> E["Environment / Robot body f"]
  E --> S2["Next state x_{t+1}"]
  E --> R["Reward r(x_t,u_t)"]
  R --> U["Update value/policy"]
  U --> P
        

The boom is a systems phenomenon. Consider a nominal linear controller \( u = Kx \) for dynamics with modeling error:

\[ x_{t+1}=Ax_t+Bu_t+d(x_t) \]

A learned compensator \( \hat{d}_\theta(x) \) yields

\[ u_t = Kx_t - B^\dagger \hat{d}_\theta(x_t) \]

If learning achieves \( \|\hat{d}_\theta(x)-d(x)\| \le \varepsilon \), then the closed-loop behaves like the nominal stable system plus bounded disturbance. This hybrid “model + learning” view scales across tasks.

We simulate a simple classification problem: a robot uses two sensors \( x \in \mathbb{R}^2 \) to decide whether an object is “graspable.” We fit logistic regression by gradient descent.

import numpy as np

np.random.seed(1)
N = 200
X = np.random.randn(N, 2)

w_true = np.array([1.2, -0.8])
b_true = -0.1
y = (1/(1+np.exp(-(X @ w_true + b_true))) > 0.5).astype(int)

w = np.zeros(2); b = 0.0
eta = 0.1

def sigmoid(z):
    return 1/(1+np.exp(-z))

for k in range(2000):
    p = sigmoid(X @ w + b)
    grad_w = (1/N) * X.T @ (p - y)
    grad_b = (1/N) * np.sum(p - y)
    w -= eta * grad_w
    b -= eta * grad_b

acc = np.mean((sigmoid(X @ w + b) > 0.5) == y)
print("learned w:", w, "learned b:", b)
print("training accuracy:", acc)
      

Deep robotics stacks use the same optimization idea but with large networks (PyTorch/TensorFlow) and sensor toolkits (OpenCV, ROS2 perception packages).

#include <iostream>
#include <vector>
#include <random>
#include <algorithm>
using namespace std;

int main() {
    const int S = 21;                 // positions -10,...,10
    const int A = 3;                  // actions: 0 left, 1 stay, 2 right
    vector<vector<double>> Q(S, vector<double>(A, 0.0));

    double alpha = 0.1, gamma = 0.95, eps = 0.2;
    mt19937 gen(0);
    uniform_real_distribution<double> uni(0.0,1.0);
    uniform_int_distribution<int> a_rand(0,A-1);

    auto step = [&](int s, int a){
        int pos = s - 10;
        int next_pos = pos + (a-1); // left=-1, stay=0, right=+1
        next_pos = max(-10, min(10, next_pos));
        int s2 = next_pos + 10;
        double r = (next_pos == 0) ? 0.0 : -1.0;
        return pair<int,double>(s2,r);
    };

    for(int ep=0; ep<5000; ++ep){
        int s = 10+5; // start at pos=5
        for(int t=0; t<50; ++t){
            int a;
            if(uni(gen) < eps) a = a_rand(gen);
            else a = int(max_element(Q[s].begin(), Q[s].end()) - Q[s].begin());

            auto sr = step(s,a);
            int s2 = sr.first;
            double r = sr.second;

            double qmax = *max_element(Q[s2].begin(), Q[s2].end());
            Q[s][a] += alpha * (r + gamma*qmax - Q[s][a]);
            s = s2;
            if(s==10) break; // reached origin
        }
    }

    cout << "Greedy policy (pos -> action):\n";
    for(int s=0; s<S; ++s){
        int pos = s-10;
        int a = int(max_element(Q[s].begin(), Q[s].end()) - Q[s].begin());
        cout << pos << " -> " << (a-1) << "\n";
    }
}
      

Real RL robotics in C++ often combines Eigen, ROS2, and libtorch for deep value functions.

import java.util.Random;

public class LinearGD {
    public static void main(String[] args) {
        int N = 200;
        double[][] X = new double[N][2];
        double[] y = new double[N];
        Random rng = new Random(1);

        double[] wTrue = {1.2, -0.8};
        double bTrue = -0.1;
        for(int i=0; i<N; i++){
            X[i][0] = rng.nextGaussian();
            X[i][1] = rng.nextGaussian();
            y[i] = wTrue[0]*X[i][0] + wTrue[1]*X[i][1] + bTrue
                   + 0.1*rng.nextGaussian();
        }

        double[] w = {0.0, 0.0};
        double b = 0.0, eta = 0.05;
        for(int k=0; k<3000; k++){
            double gw0=0, gw1=0, gb=0;
            for(int i=0; i<N; i++){
                double pred = w[0]*X[i][0] + w[1]*X[i][1] + b;
                double err = pred - y[i];
                gw0 += err*X[i][0];
                gw1 += err*X[i][1];
                gb  += err;
            }
            gw0 *= (2.0/N); gw1 *= (2.0/N); gb *= (2.0/N);
            w[0] -= eta*gw0; w[1] -= eta*gw1; b -= eta*gb;
        }

        System.out.println("learned w0=" + w[0] + " w1=" + w[1] + " b=" + b);
    }
}
      

Java robotics ecosystems include ROSJava and deep-learning libraries such as DL4J.

states = -10:10;
actions = [-1 0 1];
gamma = 0.95;

V = zeros(size(states));
for iter = 1:200
    Vnew = V;
    for s = 1:length(states)
        pos = states(s);
        Qvals = zeros(size(actions));
        for a = 1:length(actions)
            next_pos = min(10, max(-10, pos + actions(a)));
            r = (next_pos==0)*0 + (next_pos~=0)*(-1);
            s2 = find(states==next_pos);
            Qvals(a) = r + gamma*V(s2);
        end
        Vnew(s) = max(Qvals);
    end
    V = Vnew;
end

policy = zeros(size(states));
for s = 1:length(states)
   pos = states(s);
   Qvals = zeros(size(actions));
   for a = 1:length(actions)
      next_pos = min(10, max(-10, pos + actions(a)));
      r = (next_pos==0)*0 + (next_pos~=0)*(-1);
      s2 = find(states==next_pos);
      Qvals(a) = r + gamma*V(s2);
   end
   [~, idx] = max(Qvals);
   policy(s) = actions(idx);
end

disp([states' policy'])
      

Problem 1 (Closed-form linear learning): Let \( g_\theta(x)=\theta^\top x \) and squared loss. Show that the minimizer of \( J(\theta)=\frac{1}{N}\|X\theta-y\|_2^2 \) satisfies the normal equations.

Solution: Expand and differentiate:

\[ J(\theta)=\frac{1}{N}(X\theta-y)^\top(X\theta-y) \]

\[ \nabla_\theta J(\theta)=\frac{2}{N}X^\top(X\theta-y) \]

Setting \( \nabla_\theta J(\theta)=0 \) gives \( X^\top X\theta=X^\top y \), hence \( \theta^\star=(X^\top X)^{-1}X^\top y \) if invertible.

Problem 2 (Bellman contraction): For the Bellman operator \( (TV)(x)=\max_u\{r(x,u)+\gamma V(f(x,u))\} \), prove it is a contraction in the sup norm when \( 0 < \gamma < 1 \).

Solution: For any state \( x \):

\[ \begin{aligned} |(TV_1)(x)-(TV_2)(x)| &\le \max_u \gamma |V_1(f(x,u))-V_2(f(x,u))| \\ &\le \gamma \|V_1-V_2\|_\infty. \end{aligned} \]

Taking the supremum over \( x \) yields \( \|TV_1-TV_2\|_\infty \le \gamma\|V_1-V_2\|_\infty \).

Problem 3 (Model + learning bound): Consider \( x_{t+1}=Ax_t+Bu_t+d(x_t) \) and controller \( u_t=Kx_t-B^\dagger \hat{d}_\theta(x_t) \). If \( \|\hat{d}_\theta(x)-d(x)\|\le\varepsilon \), derive the closed-loop error dynamics.

Solution: Substitute the control law:

\[ \begin{aligned} x_{t+1} &=Ax_t+B\left(Kx_t-B^\dagger \hat{d}_\theta(x_t)\right)+d(x_t) \\ &=(A+BK)x_t + \left(d(x_t)-\hat{d}_\theta(x_t)\right). \end{aligned} \]

Thus the learning mismatch appears as a bounded disturbance \( e_t=d(x_t)-\hat{d}_\theta(x_t) \) with \( \|e_t\|\le\varepsilon \).

Problem 4 (Gradient for logistic regression): For logistic regression with cross-entropy loss

\[ J(\theta)= -\frac{1}{N}\sum_{i=1}^N \left[y_i\log p_i+(1-y_i)\log(1-p_i)\right], \quad p_i=\sigma(\theta^\top x_i), \]

derive \( \nabla_\theta J(\theta) \).

Solution: Using \( \sigma'(z)=\sigma(z)(1-\sigma(z)) \) and the chain rule:

\[ \nabla_\theta J(\theta)=\frac{1}{N}\sum_{i=1}^N (p_i-y_i)x_i. \]

Therefore the gradient descent update is \( \theta_{k+1}=\theta_k-\eta \frac{1}{N}\sum_i(p_i-y_i)x_i \).

The modern robotics boom came from AI enabling robots to learn perception, decisions, and control from data. We formalized supervised learning, deep perception as layered approximation, and RL via Bellman equations and contraction. These advances transformed robots from fixed industrial machines into adaptive systems.