toy-wm policy viewer
A small learned controller moves a gripper in a tiny 2D world, running live in your browser. You pick which object goes where; it pushes, carries and places objects to carry that out, one instruction or two in a row. It learned by imitating demonstrations. Nudge an object to see how it copes.
Put the red object on the pedestal, then the green object on the yellow tile.
The policy is given the instruction as tokens (which object, which destination), not as a picture: no goal marker is drawn, so it has to find the destination in the scene itself.
Example scene, picked because the model solves it here. Press New scene for a random one. Outcomes vary from scene to scene: over the lab’s held-out scenes, two instructions on pairings seen in training succeed 822 times out of 864.
The two instructions use different objects and different destinations; picking one the other row already uses swaps them.
Moves an object 4 to 7 units off its destination (success needs it within 3), then the model carries on from there with a fresh time budget. The model is not yet robust to being knocked about, so a nudge can defeat it.
What you are looking at
A side view of a tiny world, 64 units wide; gravity pulls things down.
- Gripper: a white circle when open, an orange circle when closed (gripping).
- Objects: one red and one green, each a ball (circle) or a block (square).
- Pedestal: a raised grey platform; an object sent there must rest on top of it.
- Tiles: a yellow and a magenta strip painted on the ground. With the pedestal they are the possible destinations; nothing in the scene marks which one the instruction means.
- Ground and side walls: grey. The background is near-black.
Advanced
Details for the curious
How often it succeeds. The lab’s held-out scenes, three training seeds pooled, 10-step sampler. One instruction, on pairings seen in training: 527 of 540; from a near-miss start: 470 of 516. Two instructions in order, on pairings seen in training: 822 of 864.
The never-demonstrated pairing (green object to the magenta tile): 68 of 108 as a single instruction, 71 of 108 from a near miss. Inside a two-instruction task: 140 of 432 — 135 of 216 when it is the second instruction, 5 of 216 when it is the first.
Not yet robust. Under deliberate disturbance (objects shaken and the gripper’s actions briefly overridden) the single-instruction score falls from 527 to 396 of 540: the model is not yet robust to being knocked about. That is why “Nudge it” can defeat it.
This page’s model. The page shows one training seed (seed 0, the median of the three). Its own totals: one instruction 171 of 180 on trained pairings and 11 of 36 on the never-demonstrated one; from a near miss 158 of 172 and 19 of 36; two instructions 277 of 288 and 54 of 144.
The instruction reaches the policy only as four token ids (first object, first destination, second object, second destination; the second pair is empty for a single instruction). They steer the image encoder and the action decoder; nothing about the goal is drawn in the image.
Scenes, rendering and physics are bit-exact with the Python lab (owned engine, PCG32 seed contract, world version 3), and the success check is a tested twin of the lab’s. Policy rollouts can diverge from the lab at NN-inference float level (~1e-6 per call) at contact bifurcations — physics replay of identical actions cannot.
Browser compute: the policy replans every 8 steps with 10 denoising passes (expect a short pause on phones). The sampling noise uses the browser’s own random stream, so per-scene outcomes differ from the lab census; the numbers above are the lab’s.
“Nudge it” and the near-miss start are interactive features of this page. Near-miss scenes use the lab’s recovery-start construction (twinned and tested), but nudges and this page’s random seeds are not part of any lab census.
Lab wording of the task: carry out two bound instructions in order (different objects, different destinations), given as instruction tokens [obj1, goal1, obj2, goal2]; success = both conditions at once, held for 10 steps (completion order is not scored); budget 300 steps.
Live readout
t=0/0 dist=–u 0ms/step