In HRI, a modality is a physical communication channel from the human to the robot (and often back from robot to human). In this lesson we consider:

• Speech: acoustic signals from a microphone.
• Gesture: body or hand motion captured by a camera or motion sensors.
• Haptics: forces, torques, and tactile feedback between the human and robot.
• AR/VR: headsets and controllers that couple a virtual environment to the real robot.

At a control-theoretic level, all modalities can be seen as providing measurements of a latent human intent variable that we denote by the feature vector \( \mathbf{z}(k) \in \mathbb{R}^{d} \) at discrete time index \( k \). A robot controller maps this to a control input \( \mathbf{u}(k) \in \mathbb{R}^{m} \):

\[ \mathbf{u}(k) = \pi\!\big(\mathbf{z}(k)\big), \]

where \( \pi \) is an (often nonlinear) interaction policy. In an introductory course we often approximate \( \pi \) as linear around a nominal operating point, consistent with linear control:

\[ \mathbf{u}(k) \approx \mathbf{K} \,\mathbf{z}(k), \]

with gain matrix \( \mathbf{K} \in \mathbb{R}^{m \times d} \).

flowchart TD
  H["Human intention"] --> C1["Interaction channels: speech, gesture, haptics, AR/VR"]
  C1 --> FE["Feature extraction per channel"]
  FE --> FUS["Multimodal fusion and intent estimation"]
  FUS --> CTRL["Robot controller (linear or nonlinear policy)"]
  CTRL --> ACT["Robot actuators and motion"]
  ACT --> FB["Robot state feedback to human (audio, visual, haptic)"]
  FB --> H
        

Different modalities differ mainly in: (i) signal type and dimensionality, (ii) noise characteristics, (iii) bandwidth and delay, and (iv) how comfortable and natural they are for humans.

Let there be \( M \) interaction modalities, indexed by \( m = 1,\dots,M \). For each modality we define a discrete-time raw signal \( \mathbf{s}_m[k] \in \mathbb{R}^{n_m} \) sampled with period \( T_s \). A simple additive-noise model is:

\[ \mathbf{s}_m[k] = \mathbf{x}_m[k] + \mathbf{n}_m[k], \]

where \( \mathbf{x}_m[k] \) encodes the ideal (noise-free) human signal and \( \mathbf{n}_m[k] \) is measurement noise or disturbance. A feature-extraction function \( \phi_m \) maps the raw signal to a lower-dimensional feature vector \( \mathbf{z}_m[k] \in \mathbb{R}^{d_m} \):

\[ \mathbf{z}_m[k] = \phi_m\!\big(\mathbf{s}_m[k]\big) \in \mathbb{R}^{d_m}. \]

Collecting all modality features yields the stacked feature vector \( \mathbf{z}[k] \in \mathbb{R}^{d} \) with \( d = \sum_{m=1}^M d_m \):

\[ \mathbf{z}[k] = \begin{bmatrix} \mathbf{z}_1[k]^\top & \dots & \mathbf{z}_M[k]^\top \end{bmatrix}^\top. \]

A linear interaction policy has the form:

\[ \mathbf{u}[k] = \mathbf{K} \,\mathbf{z}[k], \]

which fits nicely into standard state-feedback or output-feedback control structures you have seen in linear control.

2.1 Reliability-weighted fusion (scalar case)

Consider a scalar latent intent variable \( x[k] \) (e.g., a desired velocity) observed through \( M \) independent, unbiased estimators \( \hat{x}_m[k] \) from each modality. Assume:

• \( \mathbb{E}[\hat{x}_m[k]] = x[k] \) (unbiased),
• \( \operatorname{Var}(\hat{x}_m[k]) = \sigma_m^2 \),
• estimation errors are mutually independent across modalities.

A general linear fusion rule is:

\[ \hat{x}[k] = \sum_{m=1}^M w_m \hat{x}_m[k], \quad \sum_{m=1}^M w_m = 1. \]

Under the independence assumption, the variance of the fused estimate is:

\[ J(\mathbf{w}) = \operatorname{Var}(\hat{x}[k]) = \sum_{m=1}^M w_m^2 \sigma_m^2. \]

We want to choose weights \( w_m \) that minimize \( J(\mathbf{w}) \) subject to \( \sum_{m=1}^M w_m = 1 \). Using a Lagrange multiplier \( \lambda \), define

\[ \mathcal{L}(\mathbf{w},\lambda) = \sum_{m=1}^M w_m^2 \sigma_m^2 + \lambda\!\left( \sum_{m=1}^M w_m - 1 \right). \]

Taking partial derivatives and setting them to zero gives

\[ \frac{\partial \mathcal{L}}{\partial w_m} = 2 w_m \sigma_m^2 + \lambda = 0 \quad\Rightarrow\quad w_m = -\frac{\lambda}{2\sigma_m^2}. \]

Summing over \( m \) and using the constraint \( \sum_{m=1}^M w_m = 1 \):

\[ \sum_{m=1}^M w_m = -\frac{\lambda}{2} \sum_{m=1}^M \frac{1}{\sigma_m^2} = 1 \quad\Rightarrow\quad \lambda = -\frac{2}{\sum_{m=1}^M 1/\sigma_m^2}. \]

Substituting back:

\[ w_m^{\star} = \frac{\dfrac{1}{\sigma_m^2}}{\sum_{j=1}^M \dfrac{1}{\sigma_j^2}}. \]

Thus, the optimal linear fusion weights are proportional to the inverse variance, i.e., more reliable modalities (smaller \( \sigma_m^2 \)) receive larger weights in the fusion.

Speech is an acoustic pressure signal \( x(t) \) measured by a microphone and converted to a discrete-time signal \( x[n] = x(nT_s) \) via sampling period \( T_s \) (sampling frequency \( f_s = 1/T_s \)). For human speech, \( f_s \) is typically in the range 8–16 kHz.

3.1 Short-time energy and command detection

A simple low-level measure for detecting speech segments is the short-time energy:

\[ E[k] = \sum_{n=k-N+1}^{k} x[n]^2, \]

where \( N \) is the window length in samples. A basic decision rule for detecting the presence of speech in frame \( k \) is:

\[ \text{``speech present'' at frame } k \quad\Longleftrightarrow\quad E[k] \ge \theta_E, \]

where \( \theta_E \) is an energy threshold chosen based on background noise statistics.

3.2 Linear command classification from speech features

Modern speech-based HRI often uses higher level features such as MFCCs (mel-frequency cepstral coefficients). Let \( \mathbf{z}_s[k] \in \mathbb{R}^{d_s} \) be a vector of speech features at time \( k \), and let there be \( C \) discrete robot commands \( \mathcal{C} = \{1,\dots,C\} \). A linear classifier computes a score \( g_j[k] \) for each class:

\[ g_j[k] = \mathbf{w}_j^\top \mathbf{z}_s[k] + b_j, \quad j = 1,\dots,C, \]

and selects the winning command

\[ \hat{c}[k] = \arg\max_{j \in \{1,\dots,C\}} g_j[k]. \]

The chosen command index \( \hat{c}[k] \) can then be mapped to a reference for a low-level controller, e.g., a desired linear velocity \( v_d \) or joint reference \( q_d \).

Gesture interactions use visual or inertial sensors to track joint positions or hand poses. For this introductory course we consider a simple point-mass model of a tracked hand in 3D: \( \mathbf{p}_h[k] \in \mathbb{R}^3 \). Let the components be \( \mathbf{p}_h[k] = (x_h[k], y_h[k], z_h[k])^\top \).

4.1 Velocity-based swipe gesture

In many interfaces, a “swipe” gesture can be interpreted as a command to move the robot in a particular direction. The discrete-time velocity along the camera x-axis is approximated by

\[ v_x[k] = \frac{x_h[k] - x_h[k-1]}{T_s}. \]

Over a window of \( K \) frames, the average velocity is

\[ \bar{v}_x = \frac{1}{K} \sum_{k=1}^{K} v_x[k]. \]

A simple decision rule for a “swipe right” gesture is:

\[ \bar{v}_x \ge v_{\min}, \]

where \( v_{\min} > 0 \) is a design parameter. This threshold plays a role similar to a control gain: too small and accidental motion triggers a swipe; too large and the robot will ignore many deliberate gestures.

More sophisticated systems represent gestures as sequences in a high-dimensional feature space and classify them using sequence models, but the basic idea still reduces to extracting features \( \mathbf{z}_g[k] \) and feeding them into a policy \( \pi \).

Haptic interaction involves forces and motions exchanged between human and robot through physical contact, often via a joystick, handle, or exoskeleton. Consider a simple 1-DOF translational interaction along the coordinate \( x(t) \) with robot mass \( M \):

\[ M \ddot{x}(t) = u(t) + F_h(t), \]

where \( u(t) \) is the actuator force and \( F_h(t) \) is the force applied by the human.

5.1 Impedance control law

A standard approach is impedance control, in which the robot behaves as a virtual mass–spring–damper with respect to a desired position \( x_d(t) \):

\[ u(t) = -K\big(x(t) - x_d(t)\big) - B\big(\dot{x}(t) - \dot{x}_d(t)\big), \]

where \( K > 0 \) and \( B > 0 \) are stiffness and damping gains. Substituting into the dynamics gives the closed-loop equation

\[ M \ddot{x}(t) + B \dot{x}(t) + K x(t) = B \dot{x}_d(t) + K x_d(t) + F_h(t). \]

In the Laplace domain, and for constant desired position \( x_d \), the mechanical impedance seen by the human is

\[ Z(s) = \frac{F_h(s)}{X(s) - X_d(s)} = M s^2 + B s + K. \]

5.2 Damping ratio and natural frequency

Ignoring the human input for stability analysis, the homogeneous equation

\[ M \ddot{x}(t) + B \dot{x}(t) + K x(t) = 0 \]

is a standard second-order system with

\[ \omega_n = \sqrt{\frac{K}{M}}, \quad \zeta = \frac{B}{2\sqrt{K M}}, \]

where \( \omega_n \) is the natural frequency and \( \zeta \) is the damping ratio. Choosing \( \zeta \) between about \( 0.7 \) and \( 1 \) yields a responsive but well-damped haptic feel.

AR/VR systems provide 6D pose data (position and orientation) of the user's head and controllers. Suppose \( ^V\mathbf{p} \in \mathbb{R}^3 \) is the position of a controller expressed in a virtual world frame \( V \). To command a robot with base frame \( B \), we need a rigid-body transform from \( V \) to \( B \):

\[ ^B\mathbf{p} = \mathbf{R}_{BV} \, ^V\mathbf{p} + \mathbf{t}_{BV}, \]

where \( \mathbf{R}_{BV} \in SO(3) \) is a rotation matrix and \( \mathbf{t}_{BV} \in \mathbb{R}^3 \) is a translation vector. These concepts were introduced qualitatively in the coordinate frames chapter; in AR/VR we use them operationally.

6.1 Low-pass filtering of noisy AR/VR poses

AR/VR tracking can be noisy. A common way to stabilize the perceived pointer is a discrete-time first-order low-pass filter:

\[ \mathbf{p}_f[k] = \alpha \,\mathbf{p}_f[k-1] + (1-\alpha)\,\mathbf{p}[k], \quad 0 \le \alpha \le 1, \]

where \( \mathbf{p}[k] \) is the raw position at time \( k \) and \( \mathbf{p}_f[k] \) is the filtered position. In the z-domain, this filter has transfer function

\[ H(z) = \frac{1-\alpha}{1 - \alpha z^{-1}}, \]

which corresponds to a stable pole at \( z = \alpha \). Larger \( \alpha \) yields smoother but more delayed tracking.

Suppose speech, gesture, haptics, and AR/VR all produce feature vectors \( \mathbf{z}_s[k], \mathbf{z}_g[k], \mathbf{z}_h[k], \mathbf{z}_a[k] \) and per-command scalar scores \( s_{m,c}[k] \), where \( m \in \{s,g,h,a\} \) indexes the modality and \( c \in \mathcal{C} \) indexes the command. Reliability-weighted fusion yields a combined score

\[ S_c[k] = \sum_{m} w_m s_{m,c}[k], \]

where \( w_m \) are the weights derived in Section 2. The selected command is

\[ \hat{c}[k] = \arg\max_{c \in \mathcal{C}} S_c[k], \]

and the corresponding control input is \( \mathbf{u}[k] = \mathbf{u}_{\hat{c}[k]} \) (for example, a discrete set of velocities or pose increments). This high-level decision is then passed to a lower-level controller that ensures stability and trajectory tracking.

flowchart TD
  SS["Speech features \nz_s[k]"] --> FUS["Fusion node (reliability weights)"]
  GS["Gesture features \nz_g[k]"] --> FUS
  HS["Haptic features \nz_h[k]"] --> FUS
  AS["AR/VR features \nz_a[k]"] --> FUS
  FUS --> CMD["Discrete command index c_hat[k]"]
  CMD --> CTRL["Low-level controller (e.g. PID, state feedback)"]
  CTRL --> ROB["Robot motion and state"]
        

In Python, a typical robotics setup combines a speech recognizer (possibly external) with ROS 2 middleware. The recognizer publishes transcribed text on a topic (e.g. /speech_cmd), and a node converts text into velocity commands on /cmd_vel.

import rclpy
from rclpy.node import Node
from std_msgs.msg import String
from geometry_msgs.msg import Twist

class SpeechCommandNode(Node):
    def __init__(self):
        super().__init__("speech_command_node")
        self.sub = self.create_subscription(
            String, "/speech_cmd", self.speech_callback, 10
        )
        self.pub = self.create_publisher(Twist, "/cmd_vel", 10)
        # simple mapping from keywords to linear velocity (m/s)
        self.command_table = {
            "forward": 0.3,
            "back": -0.3,
            "stop": 0.0,
        }

    def speech_callback(self, msg: String):
        text = msg.data.lower()
        v = 0.0
        for keyword, v_cmd in self.command_table.items():
            if keyword in text:
                v = v_cmd
                break

        twist = Twist()
        twist.linear.x = float(v)
        twist.angular.z = 0.0
        self.pub.publish(twist)
        self.get_logger().info(f"Speech: '{text}' - cmd v = {v:.2f} m/s")

def main(args=None):
    rclpy.init(args=args)
    node = SpeechCommandNode()
    try:
        rclpy.spin(node)
    finally:
        node.destroy_node()
        rclpy.shutdown()

if __name__ == "__main__":
    main()
      

This node implements a simple interaction policy \( \pi \) that maps text-derived features to velocity commands. More advanced versions would use confidence scores and multimodal fusion as in Section 7.

In C++ with ROS 2, we can implement a simple safety mechanism: if the measured haptic interaction force exceeds a threshold, we send a stop command. Assume a force sensor publishes on /haptic_force and the robot listens to /cmd_vel.

#include <rclcpp/rclcpp.hpp>
#include <std_msgs/msg/float64.hpp>
#include <geometry_msgs/msg/twist.hpp>

class HapticStopNode : public rclcpp::Node {
public:
  HapticStopNode()
  : Node("haptic_stop_node"), force_threshold_(20.0) {  // newton
    using std::placeholders::_1;
    sub_ = this->create_subscription<std_msgs::msg::Float64>(
      "/haptic_force", 10,
      std::bind(&HapticStopNode::force_callback, this, _1)
    );
    pub_ = this->create_publisher<geometry_msgs::msg::Twist>("/cmd_vel", 10);
  }

private:
  void force_callback(const std_msgs::msg::Float64::SharedPtr msg) {
    double Fh = msg->data;
    if (Fh > force_threshold_) {
      geometry_msgs::msg::Twist stop;
      stop.linear.x = 0.0;
      stop.angular.z = 0.0;
      pub_->publish(stop);
      RCLCPP_WARN(this->get_logger(),
                  "High haptic force %.2f N, sending STOP", Fh);
    }
  }

  double force_threshold_;
  rclcpp::Subscription<std_msgs::msg::Float64>::SharedPtr sub_;
  rclcpp::Publisher<geometry_msgs::msg::Twist>::SharedPtr pub_;
};

int main(int argc, char** argv) {
  rclcpp::init(argc, argv);
  auto node = std::make_shared<HapticStopNode>();
  rclcpp::spin(node);
  rclcpp::shutdown();
  return 0;
}
      

This node directly enforces a safety constraint on \( F_h \), the measured human force. In terms of the impedance model, the constraint prevents large deviations in \( x(t) \) by ensuring that the controller output \( u(t) \) is set to zero when interaction forces are excessive.

Java is often used for user-interface code. With rosjava, Java applications can participate in a ROS ecosystem and publish commands. Below is a simplified sketch that maps GUI button presses (or key codes) to a Twist message:

import org.ros.namespace.GraphName;
import org.ros.node.AbstractNodeMain;
import org.ros.node.ConnectedNode;
import org.ros.node.topic.Publisher;
import geometry_msgs.Twist;

public class GuiTeleopNode extends AbstractNodeMain {

    private Publisher<Twist> cmdPub;

    @Override
    public GraphName getDefaultNodeName() {
        return GraphName.of("gui_teleop_node");
    }

    @Override
    public void onStart(final ConnectedNode connectedNode) {
        cmdPub = connectedNode.newPublisher("/cmd_vel", Twist._TYPE);
    }

    // Example handler for a key press (e.g. from a Swing GUI)
    public void handleKeyCommand(String key) {
        Twist twist = cmdPub.newMessage();
        if ("W".equals(key)) {
            twist.getLinear().setX(0.3);
        } else if ("S".equals(key)) {
            twist.getLinear().setX(-0.3);
        } else {
            twist.getLinear().setX(0.0);
        }
        twist.getAngular().setZ(0.0);
        cmdPub.publish(twist);
    }
}
      

Here the GUI itself can represent an AR-style or VR-style interface, with buttons or gestures mapped to discrete commands, again implementing a policy \( \pi \) from user inputs into robot velocities.

In MATLAB with Robotics System Toolbox, we can implement a simple reliability-weighted fusion of two channels, e.g. speech and gesture, and publish a fused velocity command. A script-level implementation could look like:

% Assume ROS master is already running and a robot listens on /cmd_vel
rosinit;  % connect to ROS network (only once)

subSpeech  = rossubscriber("/speech_velocity", "std_msgs/Float64");
subGesture = rossubscriber("/gesture_velocity", "std_msgs/Float64");
pubCmd     = rospublisher("/cmd_vel", "geometry_msgs/Twist");

% Variance estimates for the two channels (speech, gesture)
sigma_s = 0.15;
sigma_g = 0.05;

w_s = (1/sigma_s^2) / (1/sigma_s^2 + 1/sigma_g^2);
w_g = (1/sigma_g^2) / (1/sigma_s^2 + 1/sigma_g^2);

rate = rosrate(50);  % 50 Hz fusion loop

while true
    speechMsg  = receive(subSpeech, 0.1);
    gestureMsg = receive(subGesture, 0.1);

    v_s = speechMsg.Data;
    v_g = gestureMsg.Data;

    % fused velocity command: v = w_s * v_s + w_g * v_g
    v_fused = w_s * v_s + w_g * v_g;

    twistMsg = rosmessage(pubCmd);
    twistMsg.Linear.X  = v_fused;
    twistMsg.Angular.Z = 0.0;
    send(pubCmd, twistMsg);

    waitfor(rate);
end

rosshutdown;
      

In Simulink, the same logic can be implemented graphically using:

• Two Subscribe blocks for /speech_velocity and /gesture_velocity.
• Gain blocks implementing w_s and w_g.
• Sum blocks to compute v_fused.
• A Publish block for /cmd_vel.

This directly instantiates the scalar fusion formula with \( M = 2 \) modalities.

In this lesson we modeled human–robot interaction modalities as signal channels feeding into a control policy. We introduced mathematical descriptions of speech, gesture, haptic, and AR/VR interaction, and showed how linear control concepts such as gains, second-order dynamics, and discrete-time filtering apply directly to HRI. We also derived reliability-weighted fusion rules, connecting multimodal interaction to estimator design, and illustrated basic implementations in Python, C++, Java, and MATLAB/Simulink using robotics middleware such as ROS 2 and rosjava. These foundations will be used in later lessons on teleoperation, shared autonomy, and user-centered design.