Dexterity from Supervised Play

Learning dexterity from 15 minutes of goal-free play, with no task demonstrations or sim2real.

Botao He, Yiming Dou, Yiannis Aloimonos, Jitendra Malik, Akash Sharma, Ruoshi Liu

Amazon Frontier AI & Robotics (FAR) · University of Maryland

Paper and arXiv are coming soon.

Code BibTeX

From minutes of goal-free human play

To goal-conditioned dexterity

The five stages of DSP

Stages 1 to 4 form DSP; stage 5 optionally distills it into a faster policy.

Pipeline diagram from the paper. 1, supervised play: minutes of goal-free teleoperation train a play policy. 2, autonomous play: the policy collects a play dataset. 3, world modeling: context states and actions go into a Diffusion Forcing model that predicts future states. 4, model predictive control: the play policy proposes actions, the world model predicts their outcomes, and the closest prediction to the goal is executed. 5, policy distillation: self-play in the world model trains a goal-conditioned closed-loop policy.

Fifteen minutes of human play

Fifteen minutes of goal-free glove teleoperation train a diffusion play policy to produce 22-joint motion chunks.

data
~15 min goal-free teleoperation
policy
diffusion transformer, behavior cloning
output
chunks of 22 joint-position targets

Autonomous play on the robot

The play policy runs autonomously for unlimited data collection.

logged
(ot, at, ot+1) transitions
scaling study
0.54 to 5.11 h

A world model trained on play data

A Diffusion Forcing world model predicts future cube poses from recent observations and each candidate action chunk.

input
last C observations + action chunk
output
next K cube poses
training
Diffusion Forcing

tracked cube prediction

rotation error of the prediction

+0.00 s ahead

Use the slider to compare tracked and predicted cube poses and see rotation error across the prediction horizon.

Planning with play proposals

At each step, DSP predicts 1,024 play-policy proposals, executes the one that gets closest to the goal, and replans; see one replan below.

proposals
N = 1,024 per replan
horizon
50 steps predicted, 25 executed (sim)

Optional policy distillation

DSP can distill model-based planning into a 21.8 ms goal-conditioned policy that retains over 90% of MPC success at 0.8 rad.

  1. 54K recorded states × 24 goals
  2. MPC inside the world model
  3. 1.3M (history, goal, action) labels
  4. policy πθ(a | h, g) at 21.8 ms per step

One planning step in simulation

Three of the 1,024 proposals from a single replan, from sampling to execution.

Current state: the simulated hand holding the cube.
Current statereplan from here
Sampled chunk a 
Sampled chunk bclosest to the goal
Sampled chunk c 
Goal orientation: the cube held with its red face toward the camera.
Goal 

The play policy samples 1,024 goal-agnostic action chunks from the current state.

Try the loop

How DSP compares

Emerged atomic dexterous skills from play

We see this as one form of dexterity.

Yaw rotation, thumb tip
Yaw rotation, thumb root
Edge flip
Corner flip
Fine adjustment
Composite

Simulation: imitation and RL baselines

Success rate on the real hand

Push-T: planar pushing from a 10k-frame teleoperation budget

With the same 10K-frame budget, DSP reaches 98% success at 0.90 overlap versus 74% for hindsight BC.

Play data
Play-policy proposals in orange
Predicted block motion in blue, selected plan in yellow
Executed plan

Planning from pixels

Object state

Object pose and hand state predict future object pose.

RGB pixels

RGB and hand state predict future frames.

Analysis

World-model error falls as autonomous play grows

Increasing autonomous play from 0.54 to 5.11 hours lowers 50-step rotation error by 18.8%, ending 26.4% below a static baseline.

A diffusion play policy covers more transitions

With similar imitation error, the diffusion policy covers 84.2% of transitions versus 55.4% for the deterministic policy.

Replanning takes 134 ms on one GPU

One replan takes 133.6 ms on an RTX 5090 while overlapping execution keeps control at 15 to 18 Hz.

Real-world cube reorientation demos

Run 1118° → 10° in 36 s
Run 2136° → 20° in 53 s
Run 3139° → 20° in 26 s
Run 497° → 22° in 25 s
Run 5162° → 21° in 48 s
Run 6129° → 22° in 36 sstarts after the cube is placed in the hand
Run 7141° → 17° in 18 s
Run 8155° → 22° in 56 spartly sped up in the source

All clips play at 2×, with goal error and orientation trajectory shown on the right.

Questions about DSP

What counts as play here?

Play is goal-free interaction that keeps the object moving, letting behavior cloning learn useful motions without rewards or targets.

Why not demonstrate the task directly?

Play is easier to demonstrate, and with the same 15 minutes of human time DSP reaches 91.4% at 0.6 rad versus 34.4% for goal-conditioned BC.

Why not reinforcement learning?

PPO needs 344M simulator frames to reach 86.7% at 0.6 rad, while DSP reaches 91.4% with 701K frames and avoids unsafe hardware exploration.

Why use the play policy to propose actions?

The play policy supplies feasible contact motions, outperforming an equally fast orientation random walk at 0.6 rad (38.3% versus 11.4%).

Does DSP use simulation?

Not on hardware; Isaac Lab is used only to evaluate baselines in repeatable simulation.

What are the limitations?

DSP is limited by its play repertoire, requires per-step planning compute and initial human play, and has only been shown on cube reorientation and Push-T.

Could a large pretrained behavior model replace the human play?

Possibly: a pretrained behavior model could replace initial human play before autonomous data collection, but we have not tested it.

Will the code and data be released?

Yes; code, data and training details will be linked at the top of this page.

BibTeX

@misc{he2026dsp,  title  = {Dexterity from Supervised Play},  author = {He, Botao and Dou, Yiming and Aloimonos, Yiannis and            Malik, Jitendra and Sharma, Akash and Liu, Ruoshi},  year   = {2026}}