In ecological psychology, an affordance characterizes what actions the environment offers an agent. In robotics, this idea is made quantitative: given a physical state of the world and a parameterized action (e.g., a grasp pose), we assign a value indicating whether the action is feasible or how likely it is to succeed.

Let the robot's perceptual state space be \( \mathcal{X} \subset \mathbb{R}^n \) and the action space be \( \mathcal{A} \subset \mathbb{R}^m \). A binary affordance for manipulation is modeled as a Bernoulli random variable \( Y \in \{0,1\} \) encoding success (\( Y=1 \)) or failure (\( Y=0 \)) of a given action at a given state. The affordance function is then

\[ f(x,a) \triangleq \mathbb{P}(Y = 1 \mid X = x, A = a), \quad x \in \mathcal{X},\; a \in \mathcal{A}. \]

For manipulation, typical examples include:

• Grasp affordance: \( f(x,a) \) is the probability that a grasp specified by pose and gripper aperture will lift the object.
• Push affordance: \( f(x,a) \) is the probability that a pushing action moves the object into a desired region without toppling.

Importantly, \( f(x,a) \) depends on both perception (\( x \)) and the candidate action (\( a \)). This explicitly couples sensing and control, forming the basis for action-conditioned perception.

flowchart TD
  S["Sensor data (image / point cloud)"] --> F["Feature extraction phi(x)"]
  AGEN["Generate candidate actions a"] --> FE["Combine features with action: phi(x,a)"]
  F --> FE
  FE --> AFF["Affordance model f(x,a) approx P(success|x,a)"]
  AFF --> SEL["Select best action(s) for execution"]
  SEL --> EXE["Execute action on robot"]
  EXE --> LOG["Log (x,a,y) for learning"]
        

Suppose we have a dataset of experiences \( \mathcal{D} = \{(x_i, a_i, y_i)\}_{i=1}^N \), where \( y_i \in \{0,1\} \) encodes success of executing action \( a_i \) in state \( x_i \). We choose a feature map \( \phi : \mathcal{X} \times \mathcal{A} \to \mathbb{R}^d \) and parametrize the affordance as a logistic model:

\[ f_\theta(x,a) \triangleq \sigma\big(\theta^\top \phi(x,a)\big), \quad \sigma(z) \triangleq \frac{1}{1 + e^{-z}}, \quad \theta \in \mathbb{R}^d. \]

The log-likelihood of the parameters \( \theta \) under independent Bernoulli observations is

\[ \ell(\theta) = \sum_{i=1}^N \Big( y_i \log f_\theta(x_i,a_i) + (1 - y_i) \log\big(1 - f_\theta(x_i,a_i)\big) \Big). \]

Writing \( \phi_i = \phi(x_i,a_i) \) and \( p_i = f_\theta(x_i,a_i) \), the gradient is

\[ \nabla_\theta \ell(\theta) = \sum_{i=1}^N (y_i - p_i)\,\phi_i. \]

The Hessian is

\[ \nabla_\theta^2 \ell(\theta) = -\sum_{i=1}^N p_i (1 - p_i) \, \phi_i \phi_i^\top, \]

which is negative semi-definite because \( p_i(1-p_i) \ge 0 \) and \( \phi_i \phi_i^\top \) is positive semi-definite. Therefore the log-likelihood is concave and has no spurious local maxima. This property is useful when training affordance models from large datasets of robot interactions.

To control overfitting we may add an \( \ell_2 \)-regularizer \( \tfrac{\lambda}{2}\|\theta\|_2^2 \), yielding the penalized objective

\[ J(\theta) = -\ell(\theta) + \frac{\lambda}{2}\|\theta\|_2^2, \quad \nabla_\theta J(\theta) = -\sum_{i=1}^N (y_i - p_i)\,\phi_i + \lambda \theta. \]

For a fixed observation (e.g., a depth image or point cloud) \( o \), a manipulation system typically evaluates multiple candidate actions \( \{a_1,\dots,a_K\} \subset \mathcal{A} \). The affordance model produces a finite-dimensional vector

\[ \mathbf{q}(o) = \big( f_\theta(o,a_1),\dots,f_\theta(o,a_K) \big)^\top \in [0,1]^K. \]

In pixel-wise affordance maps, each grid location \( u \in \Omega \subset \mathbb{Z}^2 \) (pixel coordinates) and local orientation \( \alpha \) are treated as an action parameter:

\[ a = (u,\alpha),\quad f_\theta(o,a) = f_\theta\big(o,(u,\alpha)\big), \]

yielding a 3D tensor of affordance values (height × width × orientations). Unlike semantic segmentation, which predicts category labels for each pixel, affordance maps predict task-specific quantities such as grasp success probability or pushability.

In continuous action spaces, the robot often selects \( a^\star \) by solving

\[ a^\star \in \arg\max_{a \in \mathcal{A}} \; f_\theta(x,a). \]

When \( \mathcal{A} \) is high-dimensional, this maximization is approximated via sampling (e.g., sampling gripper poses and scores) or gradient-based optimization through a differentiable affordance network.

A key idea in action-conditioned perception is to define perceptual features \( \phi(x,a) \) in a coordinate frame attached to the candidate action rather than to the world or camera. This uses only kinematics knowledge, which you already have from earlier courses.

Let the world be represented by a point cloud \( \mathcal{P} = \{p_k\}_{k=1}^M \subset \mathbb{R}^3 \). Each action \( a \) (e.g., a 6-DOF gripper pose) is associated with a homogeneous transform \( T(a) \in SE(3) \) mapping points from the gripper frame to the world frame. We construct a local point cloud in the gripper frame as

\[ \mathcal{P}_a = \big\{ T(a)^{-1} p_k \;:\; p_k \in \mathcal{P} \big\}. \]

Features are then computed from this local geometry, e.g.,

\[ \phi(x,a) = \rho\big(\mathcal{P}_a\big), \]

where \( \rho \) may compute distances to the gripper fingers, principal directions, local curvature, or a voxel grid encoding.

This construction is invariant to rigid motions of the entire scene. Suppose a global transform \( S \in SE(3) \) changes both the environment and the action:

\[ p_k' = S p_k, \quad T'(a) = S T(a). \]

The new local cloud becomes

\[ \mathcal{P}_a' = \big\{ T'(a)^{-1} p_k' \big\} = \big\{ T(a)^{-1} S^{-1} S p_k \big\} = \big\{ T(a)^{-1} p_k \big\} = \mathcal{P}_a. \]

Therefore, \( \phi(x,a) \) is invariant to global rigid motions, which is a desirable property: affordance should depend on the relative geometry between gripper and object, not on the absolute pose in the workspace.

Deep networks for manipulation often use action-conditioned encoders, which can be abstractly written as

\[ z = e_\theta(o,a), \quad f_\theta(o,a) = g_\theta\big(e_\theta(o,a)\big), \]

where \( o \) is the raw observation (image, depth, or point cloud), \( z \) is a latent representation conditioned on the candidate action, and \( g_\theta \) maps this latent vector to a scalar affordance value.

Training proceeds by minimizing a supervised loss over experience:

\[ \mathcal{L}(\theta) = -\sum_{i=1}^N \Big( y_i \log f_\theta(o_i,a_i) + (1 - y_i) \log \big(1 - f_\theta(o_i,a_i)\big) \Big) + \lambda \,\mathcal{R}(\theta), \]

where \( \mathcal{R}(\theta) \) encodes regularization (e.g., weight decay or smoothness across neighboring actions). The gradient w.r.t. \( \theta \) is computed by backpropagation through both the encoder and the action-conditioned transformations (e.g., transforms that map the point cloud into the gripper frame).

In practice, the encoder \( e_\theta \) may mix:

• convolutions over images,
• point convolutions or voxel networks over point clouds,
• explicit kinematic transforms as in Section 4.

The crucial point is that, unlike classical perception pipelines that compute a state representation once and reuse it for many actions, action-conditioned perception recomputes (or at least transforms) the representation for each candidate action, aligning perception with the intended control.

The following diagram summarizes a typical loop for learning and using affordance models in manipulation, including data collection and model updates.

flowchart TD
  O["Observe scene o (image / point cloud)"] --> CAND["Sample candidate actions a_1,...,a_K"]
  CAND --> SCORE["Compute features phi(o,a_k) and scores f_theta(o,a_k)"]
  SCORE --> CHOOSE["Select top actions for execution"]
  CHOOSE --> EXEC["Execute action on real or simulated robot"]
  EXEC --> OUT["Record outcome y in {0,1}"]
  OUT --> DATA["Append (o,a_k,y) to dataset D"]
  DATA --> UPDATE["Update theta by gradient descent on L(theta)"]
  UPDATE --> CAND
        

This loop can run entirely in simulation at first and then be continued or fine-tuned on real robots to reduce the simulation-to-reality gap.

We now implement a simple logistic affordance model in Python. The feature vector \( \phi(x,a) \) is abstract; in a real system it would include quantities such as distances to the object, local surface normals, or voxelized occupancy in the gripper frame.

import numpy as np

# Synthetic feature generator for (x,a) pairs.
# In practice, phi(x,a) would use open3d, ROS, or MoveIt! to compute
# geometric features in the gripper frame.
def phi(x, a):
    """
    x: state features, shape (n_state,)
    a: action parameters, shape (n_action,)
    returns: feature vector, shape (d,)
    """
    # Concatenate and add simple interactions as a placeholder
    xa = np.concatenate([x, a])
    interactions = np.outer(x, a).flatten()
    return np.concatenate([xa, interactions])

def sigmoid(z):
    return 1.0 / (1.0 + np.exp(-z))

class LogisticAffordance:
    def __init__(self, d, lr=1e-2, reg=1e-3):
        self.theta = np.zeros(d)
        self.lr = lr
        self.reg = reg

    def predict_proba(self, phi_batch):
        z = phi_batch @ self.theta
        return sigmoid(z)

    def loss_and_grad(self, phi_batch, y_batch):
        p = self.predict_proba(phi_batch)
        # Binary cross-entropy with L2 regularization
        eps = 1e-8
        loss = -np.mean(y_batch * np.log(p + eps)
                        + (1.0 - y_batch) * np.log(1.0 - p + eps))
        loss += 0.5 * self.reg * np.dot(self.theta, self.theta)

        grad = -(phi_batch.T @ (y_batch - p)) / y_batch.shape[0]
        grad += self.reg * self.theta
        return loss, grad

    def step(self, phi_batch, y_batch):
        loss, grad = self.loss_and_grad(phi_batch, y_batch)
        self.theta -= self.lr * grad
        return loss

# Example usage with synthetic data
rng = np.random.default_rng(0)
n_state = 3
n_action = 3
d = (n_state + n_action) + n_state * n_action

model = LogisticAffordance(d=d, lr=5e-2, reg=1e-3)

N = 2000
X = rng.normal(size=(N, n_state))
A = rng.normal(size=(N, n_action))

# Ground-truth theta_star (unknown in practice)
theta_star = rng.normal(size=d)
Phi = np.stack([phi(X[i], A[i]) for i in range(N)], axis=0)
logits = Phi @ theta_star
p_true = sigmoid(logits)
Y = rng.binomial(1, p_true)  # Bernoulli labels (success/failure)

for epoch in range(50):
    # Mini-batch SGD
    idx = rng.permutation(N)
    Phi_shuf = Phi[idx]
    Y_shuf = Y[idx]

    for start in range(0, N, 128):
        end = min(start + 128, N)
        loss = model.step(Phi_shuf[start:end], Y_shuf[start:end])

    if (epoch % 10) == 0:
        print(f"Epoch {epoch}, loss {loss:.4f}")

# Now model.predict_proba can be used as an affordance score.
      

This snippet can be integrated into a ROS-based manipulation pipeline by computing phi(x,a) from live sensor data and candidate gripper poses (e.g., extracted from MoveIt! planners).

Next, we show how to implement a lightweight affordance scorer in C++ using Eigen. This can be embedded in a ROS node that subscribes to point clouds and publishes scores for candidate grasps.

#include <Eigen/Dense>
#include <vector>
#include <cmath>

using Feature = Eigen::VectorXf;

float sigmoid(float z) {
    return 1.0f / (1.0f + std::exp(-z));
}

class LogisticAffordance {
public:
    explicit LogisticAffordance(int d)
        : theta_(d)
    {
        theta_.setZero();
    }

    float score(const Feature& phi) const {
        float z = theta_.dot(phi);
        return sigmoid(z);  // probability of success in (0,1)
    }

    void trainBatch(const std::vector<Feature>& features,
                    const std::vector<int>& labels,
                    float lr,
                    float reg)
    {
        Eigen::VectorXf grad(theta_.size());
        grad.setZero();
        const std::size_t N = features.size();

        for (std::size_t i = 0; i != N; ++i) {
            float p = score(features[i]);
            float diff = static_cast<float>(labels[i]) - p;
            grad += -diff * features[i];  // negative log-likelihood
        }

        grad /= static_cast<float>(N);
        grad += reg * theta_;

        theta_ -= lr * grad;
    }

    Eigen::VectorXf& theta() { return theta_; }
    const Eigen::VectorXf& theta() const { return theta_; }

private:
    Eigen::VectorXf theta_;
};

// In a ROS node, features would be computed from a point cloud
// in the gripper coordinate frame obtained via TF and robot kinematics.
      

By using Eigen, the same class can be compiled into a ROS package and linked with PCL-based perception and MoveIt!-based planning components.

For Java-based robotics stacks (e.g., ROSJava or custom middleware), we can implement a similar logistic affordance model. Here we assume that phi(x,a) has already been computed using upstream perception modules.

public class LogisticAffordance {
    private final double[] theta;
    private final double lr;
    private final double reg;

    public LogisticAffordance(int d, double lr, double reg) {
        this.theta = new double[d];
        this.lr = lr;
        this.reg = reg;
    }

    private double sigmoid(double z) {
        return 1.0 / (1.0 + Math.exp(-z));
    }

    public double predictProba(double[] phi) {
        double z = 0.0;
        for (int i = 0; i != phi.length; ++i) {
            z += theta[i] * phi[i];
        }
        return sigmoid(z);
    }

    public void trainBatch(double[][] Phi, int[] y) {
        int N = Phi.length;
        int d = theta.length;
        double[] grad = new double[d];

        for (int i = 0; i != N; ++i) {
            double p = predictProba(Phi[i]);
            double diff = y[i] - p;
            for (int j = 0; j != d; ++j) {
                grad[j] += -diff * Phi[i][j];
            }
        }

        for (int j = 0; j != d; ++j) {
            grad[j] /= (double) N;
            grad[j] += reg * theta[j];
            theta[j] -= lr * grad[j];
        }
    }

    public double[] getTheta() {
        return theta;
    }
}
      

This class can be wrapped in ROSJava subscribers and services to expose affordance scores for downstream task and motion planners.

MATLAB can be used to prototype affordance learning and then export models to Simulink for integration with robot controllers. Below is a simple vectorized implementation of logistic regression for affordance prediction.

function model = trainAffordanceLogistic(Phi, y, lr, reg, nEpoch)
% Phi: N-by-d feature matrix
% y:   N-by-1 labels in {0,1}
% lr:  learning rate
% reg: L2 regularization
% nEpoch: number of passes over data

[N, d] = size(Phi);
theta = zeros(d, 1);

for epoch = 1:nEpoch
    % Shuffle indices
    idx = randperm(N);
    Phi_shuf = Phi(idx, :);
    y_shuf = y(idx);

    for i = 1:N
        phi_i = Phi_shuf(i, :).';
        z = theta.' * phi_i;
        p = 1.0 / (1.0 + exp(-z));

        % Gradient of negative log-likelihood with L2
        grad = -(y_shuf(i) - p) * phi_i + reg * theta;
        theta = theta - lr * grad;
    end
end

model.theta = theta;
end

function p = predictAffordance(model, Phi_query)
% Phi_query: M-by-d feature matrix
theta = model.theta;
z = Phi_query * theta;
p = 1.0 ./ (1.0 + exp(-z));
end
      

To embed this in Simulink, one can wrap predictAffordance in a MATLAB Function block. The block then outputs affordance scores for each candidate action at every control cycle.

Wolfram Mathematica includes built-in functionality for logistic regression that can be directly used for affordance modeling. Here is an example using symbolic and numeric features.

(* Synthetic data for affordances *)
SeedRandom[0];
nState = 3;
nAction = 3;
d = (nState + nAction) + nState*nAction;

phi[x_List, a_List] := Module[{xa, inter},
  xa = Join[x, a];
  inter = Flatten[Outer[Times, x, a]];
  Join[xa, inter]
];

Ndata = 1000;
X = RandomVariate[NormalDistribution[0, 1], {Ndata, nState}];
A = RandomVariate[NormalDistribution[0, 1], {Ndata, nAction}];

Phi = MapThread[phi, {X, A}];

thetaStar = RandomVariate[NormalDistribution[0, 1], d];
logits = Phi.thetaStar;
pTrue = 1.0/(1.0 + Exp[-logits]);
Y = RandomVariate[BernoulliDistribution /@ pTrue];

data = Table[
   {Phi[[i]], Y[[i]]},
   {i, 1, Ndata}
];

(* Fit a logistic regression model *)
logisticModel = LogisticRegression[data];

(* Affordance prediction for a new (x,a) pair *)
xQuery = {0.0, 0.5, -0.2};
aQuery = {0.3, -0.1, 0.7};
phiQuery = phi[xQuery, aQuery];
pAfford = logisticModel[phiQuery];

pAfford
      

This higher-level interface is convenient for theoretical exploration and for validating the behavior of lower-level implementations in other languages.

Problem 1 (Concavity of the Logistic Affordance Likelihood). Consider the logistic affordance model of Section 2 with features \( \phi_i = \phi(x_i,a_i) \) and \( p_i = \sigma(\theta^\top \phi_i) \). Prove that the log-likelihood \( \ell(\theta) \) is concave in \( \theta \).

Solution. The gradient is \( \nabla_\theta \ell(\theta) = \sum_i (y_i - p_i)\phi_i \). Differentiating again yields

\[ \nabla_\theta^2 \ell(\theta) = -\sum_{i=1}^N p_i(1 - p_i)\,\phi_i \phi_i^\top. \]

For any vector \( v \),

\[ v^\top \big(\nabla_\theta^2 \ell(\theta)\big) v = -\sum_{i=1}^N p_i(1 - p_i)\,(v^\top \phi_i)^2 \le 0, \]

since \( p_i(1 - p_i) \ge 0 \) and squares are nonnegative. Thus the Hessian is negative semi-definite and \( \ell(\theta) \) is concave. Hence any local maximum is global.

Problem 2 (Invariance of Local Features). Let \( \mathcal{P} \subset \mathbb{R}^3 \) be a point cloud and \( T(a) \in SE(3) \) the transform from gripper to world frame for a given action. We define \( \mathcal{P}_a = \{T(a)^{-1}p : p \in \mathcal{P}\} \). Show that if the entire scene and the gripper are transformed by the same global transform \( S \in SE(3) \), then \( \mathcal{P}_a \) is unchanged.

Solution. Under a global transform \( S \), points become \( p' = S p \) and the gripper pose becomes \( T'(a) = S T(a) \). Then

\[ \mathcal{P}_a' = \big\{ T'(a)^{-1} p' : p' \in \mathcal{P}' \big\} = \big\{ T(a)^{-1} S^{-1} S p : p \in \mathcal{P} \big\} = \big\{ T(a)^{-1} p : p \in \mathcal{P} \big\} = \mathcal{P}_a. \]

Therefore any feature function \( \phi(x,a) = \rho(\mathcal{P}_a) \) is invariant to global rigid motions, provided \( \rho \) itself is rigid-motion invariant in the gripper frame.

Problem 3 (Expected Success of a Greedy Policy). Suppose at a given state \( x \) we evaluate \( K \) independent candidate actions \( a_1,\dots,a_K \) with affordance probabilities \( p_k = f_\theta(x,a_k) \). If the robot executes the single best action \( a^\star \) with the highest \( p_k \), what is the expected success probability of this greedy policy?

Solution. By definition of the model, \( f_\theta(x,a_k) = \mathbb{P}(Y_k = 1 \mid x,a_k) \). If the robot chooses \( a^\star = \arg\max_k p_k \), the success probability of its executed action is simply

\[ \mathbb{P}(\text{success} \mid x) = \max_{1 \le k \le K} p_k. \]

This follows because only one action is executed; modeling independence between actions is relevant only if we consider executing multiple actions sequentially.

Problem 4 (Multi-Step Logistic Gradient). Consider a sequence of affordance-labeled samples \( \{(o_t, a_t, y_t)\}_{t=1}^T \) collected along a trajectory. Show that the gradient of the cumulative loss

\[ \mathcal{L}(\theta) = -\sum_{t=1}^T \Big( y_t \log f_\theta(o_t,a_t) + (1 - y_t) \log \big(1 - f_\theta(o_t,a_t)\big) \Big) \]

is the sum of per-time-step gradients of the same form as in the i.i.d. case.

Solution. Treat each time step as a separate Bernoulli observation. Using the chain rule,

\[ \nabla_\theta \mathcal{L}(\theta) = -\sum_{t=1}^T \Big( \frac{y_t}{f_\theta(o_t,a_t)} - \frac{1 - y_t}{1 - f_\theta(o_t,a_t)} \Big) \nabla_\theta f_\theta(o_t,a_t). \]

For a logistic model \( f_\theta = \sigma(z_t) \) with \( z_t = \theta^\top \phi_t \), we have \( \nabla_\theta f_\theta(o_t,a_t) = f_\theta(1 - f_\theta)\phi_t \). Substituting and simplifying yields

\[ \nabla_\theta \mathcal{L}(\theta) = \sum_{t=1}^T \big(f_\theta(o_t,a_t) - y_t\big)\,\phi_t, \]

which is the same structure as the gradient for independent samples.

Problem 5 (Affine Transform Equivariance). Let \( T(a) \) be the homogeneous transform from gripper to world and let \( \phi(x,a) \) be any differentiable function of the local cloud \( \mathcal{P}_a \). Show that the derivative of \( \phi(x,a) \) with respect to the translation component of \( T(a) \) depends only on the local coordinates of points in \( \mathcal{P}_a \), not on their absolute world coordinates.

Solution. Let \( T(a) = (R(a),t(a)) \) with rotation \( R(a) \) and translation \( t(a) \). Then each local point is \( p_k^{\text{local}} = R(a)^\top (p_k - t(a)) \). The local cloud \( \mathcal{P}_a \) depends on \( t(a) \) only via these differences \( p_k - t(a) \). The derivative of any function \( \phi(\mathcal{P}_a) \) w.r.t. \( t(a) \) is therefore a sum over terms of the form \( \partial \phi / \partial p_k^{\text{local}} \) multiplied by \( \partial p_k^{\text{local}} / \partial t(a) \), which depends only on \( p_k^{\text{local}} \) (and \( R(a) \)), not on absolute world coordinates of \( p_k \). Hence the derivative is determined by the local configuration in the gripper frame.

In this lesson we introduced affordances as probabilistic, action-conditioned properties of the robot–environment system and formalized them as functions \( f(x,a) \) that map state–action pairs to success probabilities. We developed a logistic formulation with concave log-likelihood, described pixel-wise and continuous affordance maps, and emphasized action-conditioned representations based on transforms into gripper-centric frames. We also examined invariance properties, learning objectives, and provided multi-language implementations that can be integrated into advanced manipulation pipelines. These concepts form the basis for the next lesson on active perception, where the robot will choose actions not only to manipulate but also to gather information.