Overview of RoboFS. Given a word, a text-conditioned generator synthesizes human fingerspelling landmarks. Relation-preserving retargeting constructs robot reference landmarks by preserving bone directions and fingertip relations. Conditioned on the simulated hand state and upcoming motion references, RoboFS-Tracker produces 18-dimensional actions and executes them in physics simulation. Recording its actions with the input word yields training pairs for RoboFS-Data.
The generated hand differs from the Shadow Hand in its proportions and kinematic structure, so its landmarks cannot be used directly as robot targets. Each generated hand is first aligned to a palm coordinate system (wrist at the origin, \(z\)-axis toward the middle-finger base) and scaled so that the median wrist-to-middle-finger-base distance is 0.10 m, giving normalized landmarks \(\hat{\mathbf{h}}_k\). Rather than fitting the human skeleton to the robot and matching landmark positions, RoboFS preserves the relations between landmarks. Fingertip relations alone do not fully specify how the fingers bend, so the bone directions of all 20 bones \(\mathcal{B}\) are matched as well, together with the vectors between 7 fingertip pairs \(\mathcal{F}\), rescaled to the robot hand by \(s\):
$$\mathcal{L}_{\text{dir}}(\mathbf{q}_k) = \frac{1}{|\mathcal{B}|} \sum_{(i,j) \in \mathcal{B}} \big\lVert \mathbf{u}(\mathbf{x}^{ij}_k) - \mathbf{u}(\hat{\mathbf{h}}^{ij}_k) \big\rVert^2, \qquad \mathcal{L}_{\text{tip}}(\mathbf{q}_k) = \frac{1}{|\mathcal{F}|} \sum_{(i,j) \in \mathcal{F}} \big\lVert \mathbf{x}^{ij}_k - s\, \hat{\mathbf{h}}^{ij}_k \big\rVert^2,$$
where \(\mathbf{x}_k = \mathrm{FK}(\mathbf{q}_k)\) are the robot landmarks of the reference commands \(\mathbf{q}_k\) and \(\mathbf{u}(\mathbf{v}) = \mathbf{v}/\lVert\mathbf{v}\rVert\). With a penalty on changes between consecutive normalized commands \(\bar{\mathbf{q}}_k\), the full objective is
$$\min_{\mathbf{q}_{1:T}} \; \frac{1}{T} \sum_{k=1}^{T} \Big( \lambda_{\text{dir}} \mathcal{L}_{\text{dir}}(\mathbf{q}_k) + \lambda_{\text{tip}} \mathcal{L}_{\text{tip}}(\mathbf{q}_k) \Big) + \frac{\lambda_{\text{smooth}}}{T-1} \sum_{k=1}^{T-1} \lVert \bar{\mathbf{q}}_{k+1} - \bar{\mathbf{q}}_k \rVert^2 .$$
A sigmoid parameterization keeps each command within its joint or tendon range, and all frames of a clip are solved jointly with Adam through differentiable forward kinematics.
The retargeted references are kinematically valid, but replaying them in physics simulation does not reproduce the target motion, since the actuators lag behind the commands and the fingers make contact with one another. RoboFS-Tracker is a single PPO policy that tracks the references in a physics simulation of the Shadow Hand at 30 Hz. At each control step it observes the current hand state (positions, orientations and velocities of the robot landmarks), the next-step target with its errors, and look-ahead targets 0.1, 0.2 and 0.33 s ahead, and it outputs an 18-dimensional action that commands 14 joints and 4 coupled tendons. The reward consists of a tracking term and two penalties:
$$r_t = r^{\text{track}}_t - w_{\text{pen}}\, r^{\text{pen}}_t - w_{\text{act}}\, r^{\text{act}}_t .$$
The tracking term applies Gaussian kernels to the landmark and fingertip errors,
$$r^{\text{track}}_t = w_{\text{lm}} \exp\!\big( -e_{\text{lm},t}^2 / \sigma_{\text{lm}}^2 \big) + w_{\text{tip}} \exp\!\big( -e_{\text{tip},t}^2 / \sigma_{\text{tip}}^2 \big),$$
where \(e_{\text{lm},t}^2\) and \(e_{\text{tip},t}^2\) are the mean squared distances between the executed landmarks and the next-step target over the 20 non-wrist landmarks and over the 5 fingertips. The self-penetration penalty and the action-rate penalty are
$$r^{\text{pen}}_t = \sum_{c} \Big( \frac{\max(0,\, d_{c,t} - \delta)}{\kappa} \Big)^2, \qquad r^{\text{act}}_t = \lVert \mathbf{a}_t - \mathbf{a}_{t-1} \rVert^2,$$
where \(d_{c,t}\) is the depth of a contact \(c\) between fingers reported by the simulator, \(\delta\) is the contact-depth tolerance and \(\kappa\) is a normalization scale. The penetration penalty reduces finger penetration, and the action-rate penalty discourages abrupt changes between consecutive actions. Conditioning on the references lets one policy execute variable-length clips for any word.
Solving the retargeting optimization for every clip becomes the bottleneck at scale, so a temporal 1D convolutional network with dilated residual blocks learns to predict the optimized reference commands (RoboFS†). Running this pipeline on 50k words drawn from the training labels of FSboard, ChicagoFSWild and ChicagoFSWild+, excluding the validation and test words, and recording the actions that RoboFS-Tracker executes yields RoboFS-Data. RoboFS-T2A, a Transformer encoder–decoder, is trained on these pairs to map input character sequences to robot actions; it generates four-frame action chunks autoregressively and predicts when to stop. At inference it needs no human-motion generator, retargeting network, or tracker.
