<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Sim-to-Real | Alexander Nettekoven</title><link>https://www.alexkoven.com/tag/sim-to-real/</link><atom:link href="https://www.alexkoven.com/tag/sim-to-real/index.xml" rel="self" type="application/rss+xml"/><description>Sim-to-Real</description><generator>Wowchemy (https://wowchemy.com)</generator><language>en-us</language><lastBuildDate>Mon, 05 Oct 2026 00:00:00 +0000</lastBuildDate><image><url>https://www.alexkoven.com/media/sharing_2026.jpg</url><title>Sim-to-Real</title><link>https://www.alexkoven.com/tag/sim-to-real/</link></image><item><title>From a Phone Scan to a Real Pick: Training a Conveyor Policy on Simulated Data Alone</title><link>https://www.alexkoven.com/project/real_to_sim_conveyor/</link><pubDate>Mon, 05 Oct 2026 00:00:00 +0000</pubDate><guid>https://www.alexkoven.com/project/real_to_sim_conveyor/</guid><description>&lt;figure class="post-clip" style="max-width: 100%; margin: 1.5rem auto;">
&lt;video controls muted playsinline loop preload="metadata" autoplay poster="lead.jpg" style="width: 100%; height: auto; border-radius: 6px;">
&lt;source src="lead.mp4" type="video/mp4">
&lt;/video>&lt;figcaption>Figure 1: The real UR5e runs ACT v8, a policy trained only on simulated episodes, on 2026-10-02. It picks a cup off the moving belt and drops it into the carton. Phone video, cropped.&lt;/figcaption>
&lt;/figure>
&lt;p>On the evening of 2026-10-02, a robot arm at Formic picked cups off a moving conveyor belt and dropped them into a cardboard carton (Figure 1). The policy that drove the arm had never seen a real robot episode. It learned from 965 episodes recorded in a twin of the cell in MuJoCo, a fast physics simulator, and the twin started as a scan from my phone. The policy is ACT, short for Action Chunking with Transformers, an imitation learning method.&lt;/p>
&lt;p>The policy picked short cups off the moving belt well and struggled with tall tubs, which tipped as the gripper came down on them. No teleoperation episodes went into training; real recordings served only as a measuring stick for the twin. From the first phone scan of the cell on 2026-09-13 to the first real placement took 19 days.&lt;/p>
&lt;p>These were early tests of one question. Can AI coding agents, here OpenAI&amp;rsquo;s Astra and Anthropic&amp;rsquo;s Fable, build a simulation whose demonstrations train a policy that works on the real robot, with no teleoperation data in training? This post walks through the steps in the order I took them, with what worked, what did not, and what the two models did along the way.&lt;/p>
&lt;details class="toc-collapsible">
&lt;summary class="toc-collapsible__summary">Table of Contents&lt;/summary>
&lt;div class="toc-collapsible__content">
&lt;nav id="TableOfContents">
&lt;ul>
&lt;li>&lt;a href="#the-cell-and-the-task">The Cell and the Task&lt;/a>&lt;/li>
&lt;li>&lt;a href="#step-1-scanning-the-cell-with-a-phone">Step 1: Scanning the Cell With a Phone&lt;/a>&lt;/li>
&lt;li>&lt;a href="#step-2-from-scan-to-digital-twin">Step 2: From Scan to Digital Twin&lt;/a>
&lt;ul>
&lt;li>&lt;a href="#parts-and-contact">Parts and Contact&lt;/a>&lt;/li>
&lt;li>&lt;a href="#belt-objects-from-a-phone-clip">Belt Objects From a Phone Clip&lt;/a>&lt;/li>
&lt;li>&lt;a href="#one-specification-two-simulators">One Specification, Two Simulators&lt;/a>&lt;/li>
&lt;li>&lt;a href="#repairing-the-background-in-blender-with-astra">Repairing the Background in Blender With Astra&lt;/a>&lt;/li>
&lt;li>&lt;a href="#choosing-mujoco">Choosing MuJoCo&lt;/a>&lt;/li>
&lt;/ul>
&lt;/li>
&lt;li>&lt;a href="#step-3-calibrating-the-twin-against-the-real-cell">Step 3: Calibrating the Twin Against the Real Cell&lt;/a>
&lt;ul>
&lt;li>&lt;a href="#putting-the-real-cameras-into-the-twin">Putting the Real Cameras Into the Twin&lt;/a>&lt;/li>
&lt;li>&lt;a href="#the-robot-sat-6-degrees-off-the-scanned-cell">The Robot Sat 6 Degrees Off the Scanned Cell&lt;/a>&lt;/li>
&lt;/ul>
&lt;/li>
&lt;li>&lt;a href="#step-4-teleoperation-data-as-the-measuring-stick">Step 4: Teleoperation Data as the Measuring Stick&lt;/a>
&lt;ul>
&lt;li>&lt;a href="#belt-speed-as-an-observation">Belt Speed as an Observation&lt;/a>&lt;/li>
&lt;/ul>
&lt;/li>
&lt;li>&lt;a href="#step-5-a-scripted-expert-that-reads-the-true-state">Step 5: A Scripted Expert That Reads the True State&lt;/a>&lt;/li>
&lt;li>&lt;a href="#step-6-recovery-and-special-episodes">Step 6: Recovery and Special Episodes&lt;/a>
&lt;ul>
&lt;li>&lt;a href="#failures-the-arm-did-not-cause">Failures the Arm Did Not Cause&lt;/a>&lt;/li>
&lt;li>&lt;a href="#empty-belts-empty-grippers-and-going-home">Empty Belts, Empty Grippers and Going Home&lt;/a>&lt;/li>
&lt;/ul>
&lt;/li>
&lt;li>&lt;a href="#step-7-dataset-versions-and-randomization">Step 7: Dataset Versions and Randomization&lt;/a>
&lt;ul>
&lt;li>&lt;a href="#what-gets-randomized">What Gets Randomized&lt;/a>&lt;/li>
&lt;li>&lt;a href="#from-the-first-set-to-v8">From the First Set to v8&lt;/a>&lt;/li>
&lt;li>&lt;a href="#the-qa-gate-and-one-chained-script">The QA Gate and One Chained Script&lt;/a>&lt;/li>
&lt;/ul>
&lt;/li>
&lt;li>&lt;a href="#step-8-training-act-on-simulated-data-alone">Step 8: Training ACT on Simulated Data Alone&lt;/a>
&lt;ul>
&lt;li>&lt;a href="#act-in-one-paragraph">ACT in One Paragraph&lt;/a>&lt;/li>
&lt;li>&lt;a href="#the-state-copying-trap">The State-Copying Trap&lt;/a>&lt;/li>
&lt;li>&lt;a href="#evaluating-in-closed-loop">Evaluating in Closed Loop&lt;/a>&lt;/li>
&lt;/ul>
&lt;/li>
&lt;li>&lt;a href="#step-9-on-the-real-cell">Step 9: On the Real Cell&lt;/a>
&lt;ul>
&lt;li>&lt;a href="#act-v7-nothing-placed-and-the-wrist-camera-was-the-reason">ACT v7: Nothing Placed, and the Wrist Camera Was the Reason&lt;/a>&lt;/li>
&lt;li>&lt;a href="#act-v8-8-of-12-cups">ACT v8: 8 of 12 Cups&lt;/a>&lt;/li>
&lt;li>&lt;a href="#a-later-version-that-matched-in-sim-and-missed-for-real">A Later Version That Matched in Sim and Missed for Real&lt;/a>&lt;/li>
&lt;/ul>
&lt;/li>
&lt;li>&lt;a href="#who-did-what">Who Did What&lt;/a>&lt;/li>
&lt;li>&lt;a href="#what-broke">What Broke&lt;/a>&lt;/li>
&lt;li>&lt;a href="#lessons">Lessons&lt;/a>&lt;/li>
&lt;li>&lt;a href="#references">References&lt;/a>&lt;/li>
&lt;/ul>
&lt;/nav>
&lt;/div>
&lt;/details>
&lt;h2 id="the-cell-and-the-task">The Cell and the Task&lt;/h2>
&lt;p>The cell holds a UR5e arm from Universal Robots with a Robotiq 2F-85 two-finger gripper, a white conveyor belt on casters, and a cardboard carton on a desk beside the belt. Two cameras watch the work. A Logitech C920e scene camera looks at the whole cell, and the color stream of an Intel RealSense D435i depth camera looks out from the wrist.&lt;/p>
&lt;p>The task, which I set on 2026-09-21, is short. The arm picks one object off the moving belt and releases it inside the carton. An episode succeeds when the object rests inside the carton below the rim, the carton stands upright within 5 cm of its place, and the gripper is empty.&lt;/p>
&lt;p>A conveyor makes behavior cloning, which trains a policy to copy demonstrations, harder than a static table. The cup moves while the arm reaches for it, and one camera frame cannot tell the policy how fast the cup travels. The real belt runs from 0.10 m/s at dial setting 20 to 0.358 m/s at setting 80.&lt;/p>
&lt;h2 id="step-1-scanning-the-cell-with-a-phone">Step 1: Scanning the Cell With a Phone&lt;/h2>
&lt;p>Scaniverse, a phone app, builds a textured 3D mesh on the device, in metric units and aligned with gravity. One Scaniverse scan and two short phone clips, one of the empty workspace and one of objects riding the belt, were the only capture.&lt;/p>
&lt;p>A ChArUco board, a chessboard with small square ArUco codes that give every corner an ID, lay flat on the desk. The board appears in the scan, in both clips and later in the robot&amp;rsquo;s cameras, so it ties all of them into one world frame. Fitting the board&amp;rsquo;s grid of 32.1 mm squares to the mesh texture (Figure 2) found 15 corners with a 1.4 mm residual. Solving each phone frame&amp;rsquo;s pose from the board, a perspective-n-point (PnP) problem, gave reprojection errors of 0.2 to 0.3 px. A tight rescan on 2026-09-19 replaced the room-sized first scan of 2026-09-13.&lt;/p>
&lt;figure id="figure-figure-2-the-first-scaniverse-mesh-of-the-cell-seen-from-above-at-2-mm-per-pixel-the-texture-is-sharp-enough-to-read-the-charuco-board-on-the-desk">
&lt;a data-fancybox="" href="https://www.alexkoven.com/project/real_to_sim_conveyor/fig02_scan_mesh_topdown_hu2423a21bae197d5bdc53fc0bf4ac90eb_109286_2000x2000_fit_q90_lanczos.jpg" data-caption="Figure 2: The first Scaniverse mesh of the cell, seen from above at 2 mm per pixel. The texture is sharp enough to read the ChArUco board on the desk.">
&lt;img data-src="https://www.alexkoven.com/project/real_to_sim_conveyor/fig02_scan_mesh_topdown_hu2423a21bae197d5bdc53fc0bf4ac90eb_109286_2000x2000_fit_q90_lanczos.jpg" class="lazyload" alt="" width="60%" height="1006">
&lt;/a>
&lt;figcaption>
Figure 2: The first Scaniverse mesh of the cell, seen from above at 2 mm per pixel. The texture is sharp enough to read the ChArUco board on the desk.
&lt;/figcaption>
&lt;/figure>
&lt;p>My first plan used the scan&amp;rsquo;s Gaussian splat, a cloud of soft colored blobs, as the visual layer of the twin. Rendered from a solved phone pose, the textured mesh scored 91 percent coverage of the real frame against the splat&amp;rsquo;s 45 percent. The test ran on 2026-09-14, and I dropped the splat route.&lt;/p>
&lt;h2 id="step-2-from-scan-to-digital-twin">Step 2: From Scan to Digital Twin&lt;/h2>
&lt;h3 id="parts-and-contact">Parts and Contact&lt;/h3>
&lt;p>Hand-written boxes cut the scan into six parts, such as the belt, desk and gantry table, and a cylinder around the measured base cuts the robot out. One rule held from 2026-09-19 on. Nothing the robot touches comes from scanned geometry. Convex hulls over the scanned surfaces sat 8 to 22 mm too high and acted as a lid over the belt, so objects did not ride it. Contact now comes from hand-authored boxes, 33 at first and 44 after the repair, and the belt&amp;rsquo;s contact surface is a plane fitted to the scanned top.&lt;/p>
&lt;h3 id="belt-objects-from-a-phone-clip">Belt Objects From a Phone Clip&lt;/h3>
&lt;p>SimFoundry, a published pipeline that turns a video into a simulation scene, has stages that find, segment and rebuild objects. On the object clip, its first pass found three of the six items on the belt, and a second pass with lower segmentation thresholds found all six. In 18 minutes on a GPU machine it returned textured meshes with poses for five kinds of object, a white box, a red cup, a white jar, a wrapper and a green-label tub. Later dataset versions kept only the cups and the wrapper.&lt;/p>
&lt;h3 id="one-specification-two-simulators">One Specification, Two Simulators&lt;/h3>
&lt;p>One simulator-neutral JSON file holds the parts, contact boxes, belt, robot, objects and cameras. Two emitters read the file, one for MuJoCo and one for OmniGibson, a simulator built on NVIDIA Isaac Sim. Both simulators settled the same objects within 1 mm and carried them 0.198 to 0.200 m in 2 s on a 0.1 m/s belt. The robot and gripper models come from MuJoCo Menagerie, Google DeepMind&amp;rsquo;s library of robot models.&lt;/p>
&lt;h3 id="repairing-the-background-in-blender-with-astra">Repairing the Background in Blender With Astra&lt;/h3>
&lt;p>The raw scan smeared the gantry table and the robot mount into one blob (Figure 3, left). Astra, OpenAI&amp;rsquo;s model, rebuilt the background in Blender, the open-source 3D tool. I ran Astra through Codex, OpenAI&amp;rsquo;s coding agent, both as a plug-in inside Claude Code, Anthropic&amp;rsquo;s coding agent, and in Codex sessions I started myself. Astra worked in eight steps, each from a written brief that named the region, the reference view and the replacement:&lt;/p>
&lt;ol>
&lt;li>Gantry table and robot mount (Figure 3).&lt;/li>
&lt;li>Belt frame.&lt;/li>
&lt;li>Desk.&lt;/li>
&lt;li>Floor and far wall.&lt;/li>
&lt;li>73 regions I marked by hand.&lt;/li>
&lt;li>The belt surface, re-textured from five registered video frames.&lt;/li>
&lt;li>The carton, as five 5 mm panels with a mass of 0.45 kg, free to move.&lt;/li>
&lt;li>A flat desk top.&lt;/li>
&lt;/ol>
&lt;p>For step 5, I painted repair targets in red on fixed renders, and a script turned the paint into 3D regions. A brief that said only &amp;ldquo;fix the artifacts&amp;rdquo; returned 55 triangles, so briefs had to name the region and its replacement.&lt;/p>
&lt;figure id="figure-figure-3-blender-repair-step-1-the-scanned-gantry-table-and-robot-mount-before-left-and-after-astras-rebuild-right-rendered-from-the-same-phone-view">
&lt;a data-fancybox="" href="https://www.alexkoven.com/project/real_to_sim_conveyor/fig03_blender_step1_hu971c75caa07039daae082b92549080d2_119343_2000x2000_fit_q90_lanczos.jpg" data-caption="Figure 3: Blender repair step 1, the scanned gantry table and robot mount before (left) and after Astra&amp;amp;rsquo;s rebuild (right), rendered from the same phone view.">
&lt;img data-src="https://www.alexkoven.com/project/real_to_sim_conveyor/fig03_blender_step1_hu971c75caa07039daae082b92549080d2_119343_2000x2000_fit_q90_lanczos.jpg" class="lazyload" alt="" width="1600" height="466">
&lt;/a>
&lt;figcaption>
Figure 3: Blender repair step 1, the scanned gantry table and robot mount before (left) and after Astra&amp;rsquo;s rebuild (right), rendered from the same phone view.
&lt;/figcaption>
&lt;/figure>
&lt;h3 id="choosing-mujoco">Choosing MuJoCo&lt;/h3>
&lt;p>On 2026-09-22 I measured both simulators on my laptop. MuJoCo ran 414 control steps per second in one process and 5,750 over 24 threads, and it rendered a 640x480 frame in 0.3 ms on top. OmniGibson ran 24.8 steps per second with rendering. The MuJoCo numbers cover physics only, on the rescanned cell. A dataset needs thousands of rendered episodes, so I chose MuJoCo on 2026-09-23.&lt;/p>
&lt;h2 id="step-3-calibrating-the-twin-against-the-real-cell">Step 3: Calibrating the Twin Against the Real Cell&lt;/h2>
&lt;h3 id="putting-the-real-cameras-into-the-twin">Putting the Real Cameras Into the Twin&lt;/h3>
&lt;p>The scene camera&amp;rsquo;s intrinsics came from 28 board views, with 0.18 px reprojection error. For the wrist camera, a hand-eye calibration finds where the camera sits on the gripper. It used 15 arm poses, planned and checked for clearance in MuJoCo before the real arm moved, and four classic solvers (Tsai, Park, Horaud and Daniilidis) agreed within 1.0 mm and 0.52 degrees. Both cameras went into MuJoCo with their exact pinhole intrinsics, and I left lens distortion out of the renders.&lt;/p>
&lt;h3 id="the-robot-sat-6-degrees-off-the-scanned-cell">The Robot Sat 6 Degrees Off the Scanned Cell&lt;/h3>
&lt;p>To check the fit, I drew MuJoCo&amp;rsquo;s edges over the real scene image (Figure 4). The arm matched within 1.3 px at half resolution, but the desk sat 7.6 px off. Moving the camera alone made the arm match worse, so the robot, not the camera, sat in the wrong place in the cell. Yawing robot and camera together by 6.3 degrees about the robot base cut the desk error to 4.2 px, and I approved the fix on 2026-09-24.&lt;/p>
&lt;p>The belt stands on casters and moved after the scan. Fable&amp;rsquo;s belt check on 2026-09-25 put the real belt about 6.2 degrees off, 6 cm along, 4 cm across and 36 mm lower than the simulated belt. I had already decided to stop calibrating there. Instead, every episode draws the belt pose and the camera pose around the measured offset.&lt;/p>
&lt;figure id="figure-figure-4-mujoco-edges-magenta-over-the-real-scene-camera-image-before-left-and-after-right-yawing-robot-and-camera-together-by-63-degrees-about-the-robot-base-the-arm-matched-from-the-start-and-the-desk-and-gantry-edges-move-closer-to-the-real-ones-after-the-fix">
&lt;a data-fancybox="" href="https://www.alexkoven.com/project/real_to_sim_conveyor/fig04_yaw_fix_edges_hub6fe7682635f741d6801cbdeb0f2c8fb_164284_2000x2000_fit_q90_lanczos.jpg" data-caption="Figure 4: MuJoCo edges (magenta) over the real scene camera image, before (left) and after (right) yawing robot and camera together by 6.3 degrees about the robot base. The arm matched from the start, and the desk and gantry edges move closer to the real ones after the fix.">
&lt;img data-src="https://www.alexkoven.com/project/real_to_sim_conveyor/fig04_yaw_fix_edges_hub6fe7682635f741d6801cbdeb0f2c8fb_164284_2000x2000_fit_q90_lanczos.jpg" class="lazyload" alt="" width="1600" height="604">
&lt;/a>
&lt;figcaption>
Figure 4: MuJoCo edges (magenta) over the real scene camera image, before (left) and after (right) yawing robot and camera together by 6.3 degrees about the robot base. The arm matched from the start, and the desk and gantry edges move closer to the real ones after the fix.
&lt;/figcaption>
&lt;/figure>
&lt;h2 id="step-4-teleoperation-data-as-the-measuring-stick">Step 4: Teleoperation Data as the Measuring Stick&lt;/h2>
&lt;p>I had real teleoperated recordings from the same cell, 32 episodes from 2026-09-17, 98 from 2026-09-25, 6 from 2026-10-01 and 65 from 2026-10-02. None of them entered training. Instead, a sim-versus-real ledger compared them with the simulated data dimension by dimension, each row marked matched, covered or open. Four findings changed the simulation:&lt;/p>
&lt;ul>
&lt;li>The real driver caps each joint near 0.70 rad/s, keeps the command within 0.10 rad of the measured joints, and lags 4 frames behind. The simulation got the same limits plus a feedforward term for the lag.&lt;/li>
&lt;li>The operator reacts 0.4 to 1.2 s after a cup appears. The expert got a matching reaction delay.&lt;/li>
&lt;li>The real belt ran at about 0.22 m/s, against a first simulated range of 0.08 to 0.12 m/s. The range grew to 0.05 to 0.40 m/s.&lt;/li>
&lt;li>The real gripper reports at about 4 Hz in steps of 1/255, reads 0.90 fully closed and 0.84 to 0.89 on a cup, and needs 24 frames to close 90 percent against 7 frames in the first simulation (Figure 5). Dataset v8 modeled all three effects.&lt;/li>
&lt;/ul>
&lt;figure id="figure-figure-5-gripper-command-dashed-and-reading-solid-simulation-on-top-and-real-teleoperation-below-2026-09-27-on-the-right-25-episodes-aligned-at-the-close-command-the-real-gripper-reads-in-coarse-steps-and-closes-several-times-slower">
&lt;a data-fancybox="" href="https://www.alexkoven.com/project/real_to_sim_conveyor/fig05_gripper_sim_vs_real_hu6c6334d8f24be8891d09a1e9e9d10d6f_109763_2000x2000_fit_q90_lanczos.jpg" data-caption="Figure 5: Gripper command (dashed) and reading (solid), simulation on top and real teleoperation below, 2026-09-27. On the right, 25 episodes aligned at the close command. The real gripper reads in coarse steps and closes several times slower.">
&lt;img data-src="https://www.alexkoven.com/project/real_to_sim_conveyor/fig05_gripper_sim_vs_real_hu6c6334d8f24be8891d09a1e9e9d10d6f_109763_2000x2000_fit_q90_lanczos.jpg" class="lazyload" alt="" width="1170" height="720">
&lt;/a>
&lt;figcaption>
Figure 5: Gripper command (dashed) and reading (solid), simulation on top and real teleoperation below, 2026-09-27. On the right, 25 episodes aligned at the close command. The real gripper reads in coarse steps and closes several times slower.
&lt;/figcaption>
&lt;/figure>
&lt;p>I chose not to copy the operator&amp;rsquo;s grasp, which intercepts the cup about 0.5 m downstream with the gripper tilted about 35 degrees. On 2026-09-27 I picked a tracking grasp that follows the cup instead.&lt;/p>
&lt;h3 id="belt-speed-as-an-observation">Belt Speed as an Observation&lt;/h3>
&lt;p>One frame carries no velocity, so the policy also gets a belt-speed estimate from the scene camera. The estimator tracks corners on moving pixels with Lucas-Kanade optical flow, projects them onto the belt plane and keeps the most common value, plus a valid flag. On 10 of 10 real episodes it landed within 10 percent of the measured 0.219 m/s. The same estimator runs on the robot PC, and the simulation scales its output by a random factor of 0.95 to 1.05 per episode.&lt;/p>
&lt;h2 id="step-5-a-scripted-expert-that-reads-the-true-state">Step 5: A Scripted Expert That Reads the True State&lt;/h2>
&lt;p>A scripted expert is a hand-written controller that reads the true simulator state, such as the object pose and contacts, and produces the demonstrations. Its demonstrations are free, exactly labeled and repeatable, and the price is that the policy inherits the expert&amp;rsquo;s habits. Learning by Cheating, where a privileged agent teaches a camera-based one, is the closest published idea. On 2026-09-27 I set a rule that no real data enters training until a good simulated policy exists.&lt;/p>
&lt;p>The expert waits at the upstream edge of the arm&amp;rsquo;s reach, hovers above the cup, follows it at belt speed, then descends and closes while moving, all under the real driver limits. Figure 6 shows one v8 episode as the policy sees it.&lt;/p>
&lt;figure class="post-clip" style="max-width: 100%; margin: 1.5rem auto;">
&lt;video controls muted playsinline loop preload="metadata" poster="fig06_v8_pilot_tall_cup.jpg" style="width: 100%; height: auto; border-radius: 6px;">
&lt;source src="fig06_v8_pilot_tall_cup.mp4" type="video/mp4">
&lt;/video>&lt;figcaption>Figure 6: A v8 pilot episode as the policy sees it, scene camera on the left and wrist camera on the right, both at 640x360. The red cup is scaled to 1.7 times its height, and the belt runs at 0.38 m/s.&lt;/figcaption>
&lt;/figure>
&lt;p>Each expert version had to pass a check on a held-out seed set before it recorded a dataset, and v7 and v8 faced a gate of 80 percent. The main versions:&lt;/p>
&lt;ul>
&lt;li>The first expert (2026-09-22) placed 150 of 248 attempts, 60 percent.&lt;/li>
&lt;li>Expert v3 (Astra, 2026-09-28) added inverse kinematics (IK), which turns a gripper target into joint angles, bounded by joint, lead and speed limits together. It also added a lift monitor with up to three tries. It placed 221 of 240.&lt;/li>
&lt;li>Expert v5 (Astra, 2026-09-29) runs a physics forecast on a scratch copy of the simulation before it commits to a pick. It placed 212 of 240.&lt;/li>
&lt;li>Experts v6 and v7 (2026-09-30 and 2026-10-01) added empty-belt waits, start poses over the belt and new episode kinds, at 213 and 205 of 240.&lt;/li>
&lt;li>Expert v8 (2026-10-02) retries only on evidence the cameras can see, a 5 cm lift that leaves the cup on the belt. It never regrasps, uses the slow gripper and moves away from the carton without hitting it. It placed 208 of 240.&lt;/li>
&lt;/ul>
&lt;p>The v8 expert carries a cup for 10.5 s at the median, against 3 to 4.5 s for the human operator. A faster carry waited for a later version.&lt;/p>
&lt;h2 id="step-6-recovery-and-special-episodes">Step 6: Recovery and Special Episodes&lt;/h2>
&lt;h3 id="failures-the-arm-did-not-cause">Failures the Arm Did Not Cause&lt;/h3>
&lt;p>In a recovery episode, my idea from 2026-09-26, the first attempt fails without the arm doing anything wrong, then the expert recovers. The labels stay the expert&amp;rsquo;s clean commands, as in DART, which injects noise into demonstrations. Astra built a damped snag, which holds the object back on the belt for about 1.5 s after a lift of 1 to 3 mm (Figure 7). Held-out yields reached 83 percent for the box, 61 for the wrapper, 42 for the red cup and 91 for the green cup. Once the real driver limits slowed the simulated arm, recording kept 0 of 38 recovery episodes, until Astra&amp;rsquo;s expert v3 rebuilt the motion.&lt;/p>
&lt;figure class="post-clip" style="max-width: 100%; margin: 1.5rem auto;">
&lt;video controls muted playsinline loop preload="metadata" poster="fig07_recovery_snag.jpg" style="width: 100%; height: auto; border-radius: 6px;">
&lt;source src="fig07_recovery_snag.mp4" type="video/mp4">
&lt;/video>&lt;figcaption>Figure 7: A recovery episode from the v7 pilot. The cup catches on the belt during the lift, and the scripted expert grasps it again.&lt;/figcaption>
&lt;/figure>
&lt;p>Other recovery kinds drop the cup during the carry or let it slip before the release. With expert v3, strict recovery stayed near zero at belt speeds of 0.25 m/s or faster.&lt;/p>
&lt;h3 id="empty-belts-empty-grippers-and-going-home">Empty Belts, Empty Grippers and Going Home&lt;/h3>
&lt;p>ACT v5 failed its first real test on 2026-09-29. The arm moved along the belt before any cup arrived, because the data held no empty-belt wait longer than ACT&amp;rsquo;s look-ahead. Version v6 added empty-belt waits, and v7 added special episodes, 5 percent of the data, where the arm stays home on an empty belt, goes home from a nearby start, or turns back from an empty carry. ACT v8 passed 30 of 30 stay-home tests, 30 of 30 go-home tests and 26 of 30 empty-carry tests in simulation.&lt;/p>
&lt;h2 id="step-7-dataset-versions-and-randomization">Step 7: Dataset Versions and Randomization&lt;/h2>
&lt;h3 id="what-gets-randomized">What Gets Randomized&lt;/h3>
&lt;p>Every episode draws its conditions from a seed, so a recording can be reproduced and a held-out seed is never trained on. Each draw sets the belt speed from 0.05 to 0.40 m/s in four bands with quotas, and the belt pose around the measured offset (yaw 6.26 ± 3.5 degrees). It also sets the object, its mass and friction, the carton, the scene camera pose within ± 5 degrees and ± 3 cm, latencies, lighting and waits.&lt;/p>
&lt;p>Version v8 added what the real test of 2026-10-01 asked for. It modeled the slow gripper, the binary gripper command with hysteresis and the coarse gripper reading. It put 4 to 16 textured boxes behind the table in the scene camera (Figure 8), varied exposure and color per camera, and scaled cup heights from 0.75 to 1.75 times. The finger pads became rounded and ribbed like the real ones, with a refitted wrist camera. Fable&amp;rsquo;s gate review halved the planned wrist camera jitter to 1 cm and 3 degrees, because the wider draws put 17 percent of pad views outside the real range.&lt;/p>
&lt;figure id="figure-figure-8-the-same-simulated-episodes-with-the-v7-profile-left-and-with-v8s-textured-boxes-behind-the-table-right-the-boxes-stand-in-for-the-office-that-the-real-scene-camera-sees">
&lt;a data-fancybox="" href="https://www.alexkoven.com/project/real_to_sim_conveyor/fig08_background_v7_v8_huee9f27e50d7e2267ec4eed1d9e53c142_107333_2000x2000_fit_q90_lanczos.jpg" data-caption="Figure 8: The same simulated episodes with the v7 profile (left) and with v8&amp;amp;rsquo;s textured boxes behind the table (right). The boxes stand in for the office that the real scene camera sees.">
&lt;img data-src="https://www.alexkoven.com/project/real_to_sim_conveyor/fig08_background_v7_v8_huee9f27e50d7e2267ec4eed1d9e53c142_107333_2000x2000_fit_q90_lanczos.jpg" class="lazyload" alt="" width="1280" height="720">
&lt;/a>
&lt;figcaption>
Figure 8: The same simulated episodes with the v7 profile (left) and with v8&amp;rsquo;s textured boxes behind the table (right). The boxes stand in for the office that the real scene camera sees.
&lt;/figcaption>
&lt;/figure>
&lt;h3 id="from-the-first-set-to-v8">From the First Set to v8&lt;/h3>
&lt;p>Table 1 lists what each dataset version changed.&lt;/p>
&lt;table>
&lt;thead>
&lt;tr>
&lt;th>Version&lt;/th>
&lt;th>Date&lt;/th>
&lt;th>Episodes&lt;/th>
&lt;th>What changed&lt;/th>
&lt;/tr>
&lt;/thead>
&lt;tbody>
&lt;tr>
&lt;td>First set&lt;/td>
&lt;td>2026-09-22&lt;/td>
&lt;td>150&lt;/td>
&lt;td>First scripted expert&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>Phase 1&lt;/td>
&lt;td>2026-09-26&lt;/td>
&lt;td>1,804&lt;/td>
&lt;td>Full randomization, first recovery episodes&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>v3&lt;/td>
&lt;td>2026-09-28&lt;/td>
&lt;td>1,804&lt;/td>
&lt;td>Real driver limits, belt-speed input, 640x360 images&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>v5&lt;/td>
&lt;td>2026-09-29&lt;/td>
&lt;td>1,201&lt;/td>
&lt;td>Physics forecast in the expert&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>v6&lt;/td>
&lt;td>2026-09-30&lt;/td>
&lt;td>2,394&lt;/td>
&lt;td>Empty-belt waits, start poses over the belt, no box&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>v7&lt;/td>
&lt;td>2026-10-01&lt;/td>
&lt;td>1,970&lt;/td>
&lt;td>Episode kinds, belt-speed estimate kept between episodes&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>v8&lt;/td>
&lt;td>2026-10-02&lt;/td>
&lt;td>965&lt;/td>
&lt;td>Slow gripper, real pad shape, background boxes, cup sizes&lt;/td>
&lt;/tr>
&lt;/tbody>
&lt;/table>
&lt;p>Table 1: Dataset versions. Every episode in every row came from the MuJoCo twin.&lt;/p>
&lt;h3 id="the-qa-gate-and-one-chained-script">The QA Gate and One Chained Script&lt;/h3>
&lt;p>One chained script on a GPU server ran the expert check, a 100-episode pilot with quality checks (QA), the full recording, the upload, the ACT training and the evaluations, with no laptop in the loop. The v8 recording ran from 01:37 to 05:40 on 2026-10-02 and kept 672 pick, 243 recovery and 50 special episodes, plus 56 held-out episodes on a separate seed. The QA script checks image size, the 7-value state and action (six joints and the gripper), the belt-speed state, the success of every episode, the shares of each kind and speed band, and joint motion. It found zero hard failures and zero warnings. I halved v8 against v7 on purpose, to get a faster morning round.&lt;/p>
&lt;h2 id="step-8-training-act-on-simulated-data-alone">Step 8: Training ACT on Simulated Data Alone&lt;/h2>
&lt;h3 id="act-in-one-paragraph">ACT in One Paragraph&lt;/h3>
&lt;p>ACT predicts a chunk of 50 future joint targets from both camera images and the joint state, with a transformer and a conditional variational autoencoder (CVAE). At run time, temporal ensembling averages the overlapping chunks into one command per tick. I trained it in LeRobot, Hugging Face&amp;rsquo;s robot learning library, with batch 32, learning rate 2e-5, 640x360 images, image augmentation and 100,000 steps. State-token dropout of 0.2 randomly hides the joint-state input during training, but never the belt-speed input. Training ACT v8 on the 965-episode v8 set, and nothing else, took about 3 h 50 min.&lt;/p>
&lt;h3 id="the-state-copying-trap">The State-Copying Trap&lt;/h3>
&lt;p>The first ACT run, on 2026-09-27, placed 1 of 60 cups in closed loop, although its fit on held-out data looked good. The policy mostly copied its own joint state into its output and drifted off the expert&amp;rsquo;s path. The evaluation also lacked the recorder&amp;rsquo;s latencies. Fixing the evaluation alone doubled the measured success from 12 to 24 of 60. State-token dropout, smaller images and a larger batch followed.&lt;/p>
&lt;p>Dropout values of 0, 0.2 and 0.5 tied on 60 episodes, so I fixed 0.2.&lt;/p>
&lt;h3 id="evaluating-in-closed-loop">Evaluating in Closed Loop&lt;/h3>
&lt;p>Every checkpoint drove the simulated arm on 150 episodes of a fresh seed, with the recorder&amp;rsquo;s latencies, the same belt-speed estimator and the same ensembling as on the real cell. I report 95 percent Wilson intervals, a standard confidence interval for a success rate. The open-loop action error on held-out data did not track success across runs, so closed-loop success decided.&lt;/p>
&lt;p>ACT v8 placed 78, 89, 102 and 95 of 150 at 40,000, 60,000, 80,000 and 100,000 steps. The 80,000-step checkpoint reached 68 percent, with an interval of 60 to 75 percent. At 100,000 steps, 38 of the 55 failures were drops outside the carton.&lt;/p>
&lt;h2 id="step-9-on-the-real-cell">Step 9: On the Real Cell&lt;/h2>
&lt;p>On the robot PC, the rollout resizes both camera images to 640x360 as the recorder does, runs the belt-speed estimator, and thresholds ACT&amp;rsquo;s gripper output with hysteresis, closing above 0.5 and opening below 0.4. The loop runs at 30 Hz.&lt;/p>
&lt;h3 id="act-v7-nothing-placed-and-the-wrist-camera-was-the-reason">ACT v7: Nothing Placed, and the Wrist Camera Was the Reason&lt;/h3>
&lt;p>On 2026-10-01, ACT v7 failed twice. In the first run, the arm crept from home into the empty belt, pulled 4.1 degrees off home by the office behind the table in the scene image, and only 0.3 degrees with that region painted grey. In the second run, with the grey mask, ACT closed on cups and lifted them 4 to 5 cm, then let go 0.8 to 1.7 s after the close.&lt;/p>
&lt;p>Fable replayed the recorded run through the policy with one input swapped at a time, and the wrist view of a real grasp triggered the release. The real pads filled 21 to 24 percent of the lower wrist image against 10 to 14 percent in simulation, and pasting in a simulated wrist frame kept every carry. The root cause sat in the data. The v7 expert reopened 0.15 s into the lift when its pads touched in ways the cameras could not see, so ACT learned to guess failure from the wrist image. A continuous gripper command and a real gripper 3.3 times slower than the simulated one added to the problem, and all of these fixes went into v8. After the pad refit, the pads filled 22 percent of the simulated wrist image against 20 percent in real frames (Figure 9).&lt;/p>
&lt;figure id="figure-figure-9-the-policys-wrist-camera-input-at-three-moments-of-a-cup-pick-simulated-v8-left-against-a-real-teleoperated-pick-right-the-pads-now-look-alike-but-the-simulated-image-still-carries-far-more-fine-detail">
&lt;a data-fancybox="" href="https://www.alexkoven.com/project/real_to_sim_conveyor/fig09_wrist_sim_vs_real_hu21f66b165fa8412236b8ef528463203b_59894_2000x2000_fit_q90_lanczos.jpg" data-caption="Figure 9: The policy&amp;amp;rsquo;s wrist camera input at three moments of a cup pick, simulated v8 (left) against a real teleoperated pick (right). The pads now look alike, but the simulated image still carries far more fine detail.">
&lt;img data-src="https://www.alexkoven.com/project/real_to_sim_conveyor/fig09_wrist_sim_vs_real_hu21f66b165fa8412236b8ef528463203b_59894_2000x2000_fit_q90_lanczos.jpg" class="lazyload" alt="" width="70%" height="834">
&lt;/a>
&lt;figcaption>
Figure 9: The policy&amp;rsquo;s wrist camera input at three moments of a cup pick, simulated v8 (left) against a real teleoperated pick (right). The pads now look alike, but the simulated image still carries far more fine detail.
&lt;/figcaption>
&lt;/figure>
&lt;h3 id="act-v8-8-of-12-cups">ACT v8: 8 of 12 Cups&lt;/h3>
&lt;p>On 2026-10-02 between 19:46 and 19:56, ACT v8 at 80,000 steps ran on the cell without a scene mask. It placed 8 of 12 cups, 7 of 8 short cups at belt speeds of 0.23 to 0.35 m/s and 1 of 4 tall 24 oz tubs. No grasp that lifted a cup let go before the carton. After each placement the arm went home and took the next cup. The carry took 8.8 to 13.1 s, close to the expert&amp;rsquo;s 10.5 s median. My note that evening read, &amp;ldquo;It worked well! Policy had some issues with the larger cups, but was very good at picking up the smaller cups, even at different conveyor speeds.&amp;rdquo;&lt;/p>
&lt;p>The tall tubs tipped as the gripper came down onto them (Figure 10). The fingers went 46 to 55 mm below the rim, where the expert never goes deeper than about 28 mm, and the policy then carried an empty gripper on. On 2026-10-05 the same checkpoint placed 5 of 7 cups in one session and 3 of 3 in another.&lt;/p>
&lt;figure id="figure-figure-10-act-v8-on-a-tall-tub-on-2026-10-02-seen-by-the-scene-camera-with-the-belt-at-025-ms-the-fingers-close-low-on-the-tub-and-the-tub-tips-over">
&lt;a data-fancybox="" href="https://www.alexkoven.com/project/real_to_sim_conveyor/fig10_tall_tub_tips_hu352a3223aaa049b368eb32f6d0e34395_49033_2000x2000_fit_q90_lanczos.jpg" data-caption="Figure 10: ACT v8 on a tall tub on 2026-10-02, seen by the scene camera with the belt at 0.25 m/s. The fingers close low on the tub, and the tub tips over.">
&lt;img data-src="https://www.alexkoven.com/project/real_to_sim_conveyor/fig10_tall_tub_tips_hu352a3223aaa049b368eb32f6d0e34395_49033_2000x2000_fit_q90_lanczos.jpg" class="lazyload" alt="" width="1200" height="262">
&lt;/a>
&lt;figcaption>
Figure 10: ACT v8 on a tall tub on 2026-10-02, seen by the scene camera with the belt at 0.25 m/s. The fingers close low on the tub, and the tub tips over.
&lt;/figcaption>
&lt;/figure>
&lt;h3 id="a-later-version-that-matched-in-sim-and-missed-for-real">A Later Version That Matched in Sim and Missed for Real&lt;/h3>
&lt;p>Version v9c sped up the expert&amp;rsquo;s carry to 5.6 s, added 16 generated cups and made the rim grasp shallower. ACT v9c placed 100 of 150 in simulation, level with v8, but 0 of 2 on the real cell on 2026-10-05. On recorded real frames it aimed 59 to 78 mm further downstream than v8, while on simulated frames the two aimed within 2 mm of each other. The cause sits somewhere in the v9 training data, and I have not found it yet.&lt;/p>
&lt;h2 id="who-did-what">Who Did What&lt;/h2>
&lt;p>Since 2026-09-27 I have split the AI work by a standing rule. Astra takes scripted-policy problems, and Fable takes reviews and analyses. Claude Code, Anthropic&amp;rsquo;s coding agent, ran the day-to-day sessions that integrated everything.&lt;/p>
&lt;p>Astra, OpenAI&amp;rsquo;s model run through Codex:&lt;/p>
&lt;ul>
&lt;li>Rebuilt the scanned background in Blender in the eight briefed steps of Step 2.&lt;/li>
&lt;li>Wrote the MuJoCo emitter for the cell specification on 2026-09-19.&lt;/li>
&lt;li>Wrote experts v3, v4, v4b and v5 and the damped snag for recovery episodes.&lt;/li>
&lt;li>Compared real and simulated wrist views on 2026-10-02, which ranked the v8 fixes, then refit the v8 wrist camera and finger pads and wrote the retries that wait for visible evidence.&lt;/li>
&lt;/ul>
&lt;p>Fable, Anthropic&amp;rsquo;s strongest Claude model:&lt;/p>
&lt;ul>
&lt;li>Wrote the data plan of 2026-09-24, with one recorder format for simulated and real data.&lt;/li>
&lt;li>Ran the belt check of 2026-09-25 that found the real belt about 6.2 degrees off.&lt;/li>
&lt;li>Reviewed the project on 2026-09-29, beside a review by Astra, and the two reviews set the experimental plan.&lt;/li>
&lt;li>Made the machine plan of 2026-09-30.&lt;/li>
&lt;li>Replayed the failed real run of 2026-10-01 with one input swapped at a time and traced the release to the wrist view.&lt;/li>
&lt;li>Reviewed the v8 gates on 2026-10-02 and halved the wrist camera jitter.&lt;/li>
&lt;li>Compiled every iteration from v1 to v8 with its sources.&lt;/li>
&lt;/ul>
&lt;p>Claude Code sessions built my pipeline package (scan, repair, specification, calibration and simulation), the recorder, the belt-speed estimator, the evaluation, QA and training scripts, the calibration solve, the sim-versus-real ledger and the real-cell analyses.&lt;/p>
&lt;p>I scanned the cell, painted the repair targets, chose randomization over more calibration, recorded the teleoperation sets, set the project rules, proposed the recovery episodes, picked each checkpoint for the real cell, and ran every real test and the emergency stop.&lt;/p>
&lt;h2 id="what-broke">What Broke&lt;/h2>
&lt;ul>
&lt;li>In the twin, the splat lost to the mesh, scanned convex hulls formed a lid over the belt, and soft contact let objects sink through the carton.&lt;/li>
&lt;li>In training, the first ACT run copied its joint state and placed 1 of 60, and two recording nights died to out-of-memory kills.&lt;/li>
&lt;li>On the cell, ACT v5 moved before any cup arrived, ACT v7 crept off home and let go of every cup it lifted, and ACT v8 tipped 3 of 4 tall tubs by grasping too low.&lt;/li>
&lt;li>An offline gate built on real teleoperation grasps predicted that ACT v8 would let go. ACT v8 held every lifted cup, so I dropped that gate on 2026-10-04.&lt;/li>
&lt;li>ACT v9c matched v8 in simulation and missed on the real cell.&lt;/li>
&lt;/ul>
&lt;h2 id="lessons">Lessons&lt;/h2>
&lt;ol>
&lt;li>Let real data measure the twin before you let it train anything. Real recordings found the driver limits, the slow gripper and the true belt speed.&lt;/li>
&lt;li>Randomize around a measured offset instead of chasing the last millimeters of calibration.&lt;/li>
&lt;li>Make the simulated evaluation run the deployed control path, with the same latencies, estimator and gripper logic.&lt;/li>
&lt;li>Check what the labels teach. The v7 expert reopened for reasons no camera could see, and the policy learned to guess.&lt;/li>
&lt;li>Trust no offline score until it has met the real cell. The teleoperation gate was wrong about v8, and the simulated score was wrong about v9c. A replay check on recorded real frames now runs before any cell time, and it needs a pass threshold set before its first use.&lt;/li>
&lt;li>Brief AI models like contractors, with the region and the replacement named, and give each model the work that suits it.&lt;/li>
&lt;/ol>
&lt;p>Gaps stay open. The simulated wrist image carries about 7 times the fine detail of the real one, the renders skip lens distortion, and the operator tilts the gripper about 38 degrees where the policy grasps near vertical. The carry is slow, and the policy cannot yet learn beyond its expert. DAgger, which labels the states a learned policy visits, could help in simulation once the regular pipeline hits a wall. I have since written the steps up as a general recipe with gates, and the next cell will test it.&lt;/p>
&lt;h2 id="references">References&lt;/h2>
&lt;ul>
&lt;li>Zhao et al., Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware (ACT), &lt;a href="https://arxiv.org/abs/2304.13705" target="_blank" rel="noopener">arXiv:2304.13705&lt;/a>.&lt;/li>
&lt;li>Laskey et al., DART: Noise Injection for Robust Imitation Learning, &lt;a href="https://arxiv.org/abs/1703.09327" target="_blank" rel="noopener">arXiv:1703.09327&lt;/a>.&lt;/li>
&lt;li>Ross, Gordon and Bagnell, A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning (DAgger), &lt;a href="https://arxiv.org/abs/1011.0686" target="_blank" rel="noopener">arXiv:1011.0686&lt;/a>.&lt;/li>
&lt;li>Chen, Zhou, Koltun and Krähenbühl, Learning by Cheating, &lt;a href="https://arxiv.org/abs/1912.12294" target="_blank" rel="noopener">arXiv:1912.12294&lt;/a>.&lt;/li>
&lt;li>Ranawaka, Wong, Pai et al., SimFoundry: Modular and Automated Scene Generation for Policy Learning and Evaluation, &lt;a href="https://arxiv.org/abs/2606.28276" target="_blank" rel="noopener">arXiv:2606.28276&lt;/a>.&lt;/li>
&lt;li>&lt;a href="https://mujoco.org/" target="_blank" rel="noopener">MuJoCo&lt;/a>, &lt;a href="https://github.com/google-deepmind/mujoco_menagerie" target="_blank" rel="noopener">MuJoCo Menagerie&lt;/a> and &lt;a href="https://github.com/huggingface/lerobot" target="_blank" rel="noopener">LeRobot&lt;/a>.&lt;/li>
&lt;/ul></description></item></channel></rss>