<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Bimanual | Alexander Nettekoven</title><link>https://www.alexkoven.com/tag/bimanual/</link><atom:link href="https://www.alexkoven.com/tag/bimanual/index.xml" rel="self" type="application/rss+xml"/><description>Bimanual</description><generator>Wowchemy (https://wowchemy.com)</generator><language>en-us</language><lastBuildDate>Thu, 24 Sep 2026 00:00:00 +0000</lastBuildDate><image><url>https://www.alexkoven.com/media/sharing_2026.jpg</url><title>Bimanual</title><link>https://www.alexkoven.com/tag/bimanual/</link></image><item><title>Hands-Off Cycles on Two Robot Arms: From π0.5 Imitation to 50 Minutes of π0.7</title><link>https://www.alexkoven.com/project/bimanual_yam_cycles/</link><pubDate>Thu, 24 Sep 2026 00:00:00 +0000</pubDate><guid>https://www.alexkoven.com/project/bimanual_yam_cycles/</guid><description>&lt;figure class="post-clip" style="max-width: 100%; margin: 1.5rem auto;">
&lt;video controls muted playsinline loop preload="metadata" autoplay poster="lead.jpg" style="width: 100%; height: auto; border-radius: 6px;">
&lt;source src="lead.mp4" type="video/mp4">
&lt;/video>&lt;figcaption>Figure 1: Part of the 50-minute automatic run on 2026-09-24, sped up 16 times. The two YAM arms take snack bars out of a box and put them back, while a vision-language model chooses the next prompt.&lt;/figcaption>
&lt;/figure>
&lt;p>At Formic I spent July to September 2026 on a two-arm robot rig built from I2RT YAM arms, a low-cost arm with six joints and a gripper. The task sounds simple. Put snack bars into a box, take them out again, and repeat. The hard part is the repeat. A policy that packs one box makes a nice demo. A policy that fills and empties a box for most of an hour, while a model rather than a person picks each prompt, is something you can leave running.&lt;/p>
&lt;p>The robot policies in this post are vision-language-action models (VLAs), networks that take camera images, the arm joint angles and a text prompt and return motor commands. Both come from Physical Intelligence. π0.5 is open source, and I used π0.7 through Physical Intelligence&amp;rsquo;s partner program. In July a fine-tuned π0.5 packed the box reliably, where ACT (Action Chunking with Transformers, a small imitation-learning policy) had reached only about 20% success on a simpler block pickup on the same rig. In September π0.7, steered by a small vision-language model, ran fill-and-empty cycles hands-off for 50 minutes.&lt;/p>
&lt;p>Below I describe the hardware, the data, both policies, the supervisor, what broke, and what I would keep for the next rig.&lt;/p>
&lt;details class="toc-collapsible">
&lt;summary class="toc-collapsible__summary">Table of Contents&lt;/summary>
&lt;div class="toc-collapsible__content">
&lt;nav id="TableOfContents">
&lt;ul>
&lt;li>&lt;a href="#the-rig">The rig&lt;/a>
&lt;ul>
&lt;li>&lt;a href="#arms-and-teleoperation">Arms and teleoperation&lt;/a>&lt;/li>
&lt;li>&lt;a href="#cameras">Cameras&lt;/a>&lt;/li>
&lt;li>&lt;a href="#compute-and-software">Compute and software&lt;/a>&lt;/li>
&lt;/ul>
&lt;/li>
&lt;li>&lt;a href="#data-collection">Data collection&lt;/a>&lt;/li>
&lt;li>&lt;a href="#method">Method&lt;/a>
&lt;ul>
&lt;li>&lt;a href="#step-1-π05-fine-tuned-in-lerobot">Step 1, π0.5 fine-tuned in LeRobot&lt;/a>&lt;/li>
&lt;li>&lt;a href="#step-2-π07-on-physical-intelligences-platform">Step 2, π0.7 on Physical Intelligence&amp;rsquo;s platform&lt;/a>&lt;/li>
&lt;li>&lt;a href="#step-3-a-supervisor-that-presses-the-keys">Step 3, a supervisor that presses the keys&lt;/a>&lt;/li>
&lt;/ul>
&lt;/li>
&lt;li>&lt;a href="#results">Results&lt;/a>
&lt;ul>
&lt;li>&lt;a href="#2026-07-30-π05-packs-the-box">2026-07-30, π0.5 packs the box&lt;/a>&lt;/li>
&lt;li>&lt;a href="#2026-07-31-to-2026-08-12-stacking-where-more-data-did-not-help">2026-07-31 to 2026-08-12, stacking, where more data did not help&lt;/a>&lt;/li>
&lt;li>&lt;a href="#2026-09-21-π07-next-to-π05">2026-09-21, π0.7 next to π0.5&lt;/a>&lt;/li>
&lt;li>&lt;a href="#2026-09-24-50-minutes-in-automatic-mode">2026-09-24, 50 minutes in automatic mode&lt;/a>&lt;/li>
&lt;/ul>
&lt;/li>
&lt;li>&lt;a href="#what-broke">What broke&lt;/a>&lt;/li>
&lt;li>&lt;a href="#lessons">Lessons&lt;/a>&lt;/li>
&lt;li>&lt;a href="#links">Links&lt;/a>&lt;/li>
&lt;/ul>
&lt;/nav>
&lt;/div>
&lt;/details>
&lt;h2 id="the-rig">The rig&lt;/h2>
&lt;h3 id="arms-and-teleoperation">Arms and teleoperation&lt;/h3>
&lt;p>The rig has four YAM arms. Two follower arms sit on the edge of a wooden table and do the work. Two leader arms with teaching handles sit on an aluminum frame beside the operator&amp;rsquo;s chair. When the operator moves a leader arm, the matching follower copies each joint, a setup called leader-follower teleoperation (Figure 2). Each arm reports six joint angles plus a gripper value, so the robot state and the action are both 14 numbers.&lt;/p>
&lt;figure id="figure-figure-2-leader-follower-teleoperation-on-the-rig-the-operator-moves-the-two-leader-arms-on-the-frame-in-front-of-the-chair-and-the-two-follower-arms-over-the-table-copy-every-joint">
&lt;a data-fancybox="" href="https://www.alexkoven.com/project/bimanual_yam_cycles/teleop_hu728e405d12c4603fae613ac8c69fbdff_141862_2000x2000_fit_q90_lanczos.jpg" data-caption="Figure 2: Leader-follower teleoperation on the rig. The operator moves the two leader arms on the frame in front of the chair, and the two follower arms over the table copy every joint.">
&lt;img data-src="https://www.alexkoven.com/project/bimanual_yam_cycles/teleop_hu728e405d12c4603fae613ac8c69fbdff_141862_2000x2000_fit_q90_lanczos.jpg" class="lazyload" alt="" width="1400" height="604">
&lt;/a>
&lt;figcaption>
Figure 2: Leader-follower teleoperation on the rig. The operator moves the two leader arms on the frame in front of the chair, and the two follower arms over the table copy every joint.
&lt;/figcaption>
&lt;/figure>
&lt;p>Each arm talks over its own CAN bus, the two-wire field bus that many robot motors use, through a USB-to-CAN adapter at 1 Mbit/s. Linux numbers these adapters in the order they appear, which need not match the physical arms. A small script therefore names all four by serial number and brings them up at boot and on every replug.&lt;/p>
&lt;h3 id="cameras">Cameras&lt;/h3>
&lt;p>Three cameras feed the policies, one overhead and one on each wrist. In July the overhead camera was a global-shutter USB camera and the wrists carried Intel RealSense depth cameras. On 2026-08-13, after the stacking detour described under Results showed that the overhead camera could not see into the box, all three became Intel RealSense D405 cameras. The new overhead looks straight down at the whole table, and both new wrist views keep the gripper fingers in frame at all times. The datasets store color images at 640 by 480 pixels and 30 frames per second.&lt;/p>
&lt;h3 id="compute-and-software">Compute and software&lt;/h3>
&lt;p>One x86 Ubuntu workstation with an RTX 5080 graphics card (16 GB) runs the arms, the cameras, the recording and, for π0.5, the policy itself. An earlier plan used a separate NVIDIA Jetson, an embedded computer, for the robot. I dropped it so that one machine records data and runs the policy in the same process. Training happens elsewhere. π0.5 trained on a remote machine with one 96 GB GPU, while π0.7 trains and runs on Physical Intelligence&amp;rsquo;s servers.&lt;/p>
&lt;p>The software stack is LeRobot 0.6.0, Hugging Face&amp;rsquo;s open library for robot datasets, training and deployment, on Python 3.12, plus I2RT&amp;rsquo;s software development kit (SDK) for the arms. LeRobot&amp;rsquo;s &lt;code>lerobot-record&lt;/code>, &lt;code>lerobot-train&lt;/code> and &lt;code>lerobot-rollout&lt;/code> commands cover recording, training and running a policy on the robot.&lt;/p>
&lt;h2 id="data-collection">Data collection&lt;/h2>
&lt;p>All training data comes from teleoperation. In one episode the operator starts at a home pose, reaches for a bar, grasps it, places it in the box and returns home. One episode places one bar.&lt;/p>
&lt;p>By late July the rig had 14 recorded datasets of the bar task. Merged, they gave 507 episodes and 210,441 frames, the set behind the July π0.5 policy. After the camera swap in August, the old recordings no longer matched the new views, so collection started again. The corpus behind the September π0.7 runs holds 443 episodes and 374,914 frames across seven tasks:&lt;/p>
&lt;ul>
&lt;li>Filling the box, &amp;ldquo;Place a bar in the box, filling from left to right&amp;rdquo;, 401 episodes and 94.95% of all frames.&lt;/li>
&lt;li>Taking a bar out and placing it beside the box, once for each arm, 12 episodes per arm.&lt;/li>
&lt;li>Opening or closing the box lid with either arm, 18 episodes in total.&lt;/li>
&lt;/ul>
&lt;p>That imbalance matters later. The two bar-retrieval tasks hold 3.12% of the frames, and the models gave them about as little weight.&lt;/p>
&lt;h2 id="method">Method&lt;/h2>
&lt;h3 id="step-1-π05-fine-tuned-in-lerobot">Step 1, π0.5 fine-tuned in LeRobot&lt;/h3>
&lt;p>π0.5 is Physical Intelligence&amp;rsquo;s open VLA. I fine-tuned it from LeRobot&amp;rsquo;s &lt;code>pi05_base&lt;/code> checkpoint on the 507-episode set. The run used 20,000 steps at batch size 16, a peak learning rate of 2.5e-5 with cosine decay, weight decay of 1e-4 and action chunks of 50 steps. It took 6 h 24 min at 1.09 s per step and ended at a training loss of 0.011.&lt;/p>
&lt;p>I set the weight decay to 1e-4 instead of LeRobot&amp;rsquo;s default of 0.01. The default is a generic value for AdamW, the standard optimizer for this kind of training, and weight decay pulls the weights away from the pretrained model, which is the main asset when you fine-tune.&lt;/p>
&lt;p>π0.5 scales each state and action value by the 1st and 99th percentiles of the training data. LeRobot 0.6.0 got those percentiles wrong. It averaged per-episode percentiles instead of pooling all frames, and on one of my sets the stored ranges came out 2.0 to 4.6 times too narrow. I recomputed the pooled values over every frame and checked that the trained checkpoint carried them, to within 9.3e-08.&lt;/p>
&lt;p>Loss did not pick the checkpoint. On another robot at Formic, validation loss rose while the deployed policy kept working well, so I judged checkpoints by rollouts on the arms.&lt;/p>
&lt;h3 id="step-2-π07-on-physical-intelligences-platform">Step 2, π0.7 on Physical Intelligence&amp;rsquo;s platform&lt;/h3>
&lt;p>π0.7 is a 5-billion-parameter VLA. It pairs a 4-billion-parameter vision-language backbone, initialized from Google&amp;rsquo;s open Gemma 3 model, with an 860-million-parameter action expert. Unlike π0.5 it can look back in time. Its history variant sees up to six past frames per camera, one second apart.&lt;/p>
&lt;p>π0.7 trains and runs on Physical Intelligence&amp;rsquo;s servers. I uploaded the 443-episode corpus in 46 minutes with 10 upload workers, defined which episodes to train on, and started a fine-tune. The first run, on the plain π0.7 base, took 3 h 38 min at batch size 256 and saved checkpoints at steps 10,000 and 20,000, about 13.7 passes over the data.&lt;/p>
&lt;p>Inference is remote only. For each request the rig sends three JPEG images, the 14 joint values and the prompt over a WebSocket, a connection that stays open between requests, and gets back a chunk of 50 actions. From the rig, the median request took 154 ms, and 148 ms of that was waiting for the reply.&lt;/p>
&lt;p>The second π0.7 run used the history base model and changed the data mix. Two episode filters split every batch, 20% from the 24 bar-retrieval episodes and 80% from the rest, which is 6.4 times the natural share of the retrieval tasks. That run took 9 h 23 min. A test with one frame per request showed that the server stacks the six-frame history itself, so the rig still sends one frame per call.&lt;/p>
&lt;p>To run π0.7 inside my stack, I wrapped Physical Intelligence&amp;rsquo;s real-time chunking runner in a LeRobot inference engine with its own worker thread. Real-time chunking (RTC) requests the next chunk while the arm still plays the current one, and guides the new chunk to continue the old one, so the motion does not jump at the seams. At 30 Hz the runner asks for a new chunk every 25 actions, which leaves 833 ms for each reply. Thanks to the worker thread, a slow reply makes the arm hold its pose for a tick instead of blocking the control loop.&lt;/p>
&lt;p>I also put a safety clamp between the model and the motors. It caps each joint target at 1.2 times the 99.9th percentile of the gap between commanded and measured position seen in teleoperation. At that cap, at most 0.07% of teleoperation frames would clamp on any joint, so almost all normal motion passes untouched while a bad chunk cannot jerk an arm.&lt;/p>
&lt;h3 id="step-3-a-supervisor-that-presses-the-keys">Step 3, a supervisor that presses the keys&lt;/h3>
&lt;p>π0.7 needs a prompt, and the right prompt depends on the state of the box. In manual mode I pressed number keys to switch between filling the box and taking a bar out with the left or the right arm. For long runs, a vision-language model (VLM) took over the keyboard. Every 4 s, Gemini 3.1 Flash-Lite, a small VLM from Google, gets the last four overhead frames, spaced 1 s apart, and reads the box. The supervisor fills until four bars sit in the box, then empties it, and picks the arm again for each bar. A step closes only after two readings agree. Each call cost about $0.0013 and took 2.2 s at the median.&lt;/p>
&lt;p>The supervisor only switches between the prompts I would pick by key, and any key I press stops it at once. The sketch below shows the idea. I simplified it for this post, and it is not the code that ran on the rig.&lt;/p>
&lt;pre>&lt;code class="language-python"># Simplified sketch of the cycle supervisor (illustration only).
phase = &amp;quot;empty&amp;quot; if box_is_full(read_box()) else &amp;quot;fill&amp;quot;
while running:
reading = vlm.read_box(overhead.last_frames(n=4, spacing_s=1.0))
if confirmed(reading, n_agree=2):
if phase == &amp;quot;fill&amp;quot; and reading.bars_in_box == 4:
phase = &amp;quot;empty&amp;quot;
elif phase == &amp;quot;empty&amp;quot; and reading.bars_in_box == 0:
phase = &amp;quot;fill&amp;quot;
arm = choose_arm(reading) # picked again for every bar
policy.set_prompt(prompt_for(phase, arm))
sleep(4.0)
&lt;/code>&lt;/pre>
&lt;h2 id="results">Results&lt;/h2>
&lt;h3 id="2026-07-30-π05-packs-the-box">2026-07-30, π0.5 packs the box&lt;/h3>
&lt;p>The step-20,000 checkpoint ran on the rig and packed bars into the box at or near 100% success, counting retries (Figure 3). It was the first policy on the rig that clearly worked. The earlier block pickup had reached about 20% with ACT and 2 of 5 with Diffusion Policy, a policy that generates actions by step-by-step denoising. My notes from that day record 3 tries with 10 bars each. The policy placed all or almost all bars every time. In the first two tries, the 180 s time limit of the rollout command ran out with the last two bars still on the table, so I raised the limit to 5 minutes.&lt;/p>
&lt;figure class="post-clip" style="max-width: 100%; margin: 1.5rem auto;">
&lt;video controls muted playsinline loop preload="metadata" poster="pi05_packing.jpg" style="width: 100%; height: auto; border-radius: 6px;">
&lt;source src="pi05_packing.mp4" type="video/mp4">
&lt;/video>&lt;figcaption>Figure 3: A π0.5 rollout on 2026-07-30, shown at twice real speed. The far arm carries a bar into the box and goes back for the next one.&lt;/figcaption>
&lt;/figure>
&lt;h3 id="2026-07-31-to-2026-08-12-stacking-where-more-data-did-not-help">2026-07-31 to 2026-08-12, stacking, where more data did not help&lt;/h3>
&lt;p>On 2026-07-31 I made the task harder. The bars had to be stacked in two rows of four, with every wrapper facing the same way. Over the next week the stacking data went through four merged datasets of 253, 229, 330 and 262 episodes. The policies sometimes sat still and did nothing, and sometimes placed a bar with its wrapper flipped. The policy trained on 330 episodes did worse on the robot than the one trained on 229.&lt;/p>
&lt;p>My note from 2026-08-07 blamed the overhead camera, which could not see what the gripper did inside the box. A camera audit on 2026-08-10 confirmed it. The overhead camera wasted 44 percent of its frame, the wrist cameras had no fixed reference in view, and π0.5 received the camera images in a different order than in its pretraining. The audit led to the camera swap on 2026-08-13 and to a new dataset recorded with the new views. More episodes could not make up for cameras that did not show the task.&lt;/p>
&lt;h3 id="2026-09-21-π07-next-to-π05">2026-09-21, π0.7 next to π0.5&lt;/h3>
&lt;p>Before the long run, I ran π0.5 and π0.7 on the same prompts and cut the two runs side by side (Figure 4). Each test starts with the fill prompt and then asks for a bar out of the box with the right arm and with the left arm, twice each. π0.7 looked smoother and committed to a grasp sooner. Both policies sometimes used the wrong arm on the retrieval prompts. My note from the next day reads that π0.7 was not noticeably better than π0.5, maybe slightly.&lt;/p>
&lt;figure class="post-clip" style="max-width: 100%; margin: 1.5rem auto;">
&lt;video controls muted playsinline loop preload="metadata" poster="pi07_vs_pi05.jpg" style="width: 100%; height: auto; border-radius: 6px;">
&lt;source src="pi07_vs_pi05.mp4" type="video/mp4">
&lt;/video>&lt;figcaption>Figure 4: π0.5 running locally on the rig (left) and π0.7 hosted by Physical Intelligence (right) on the same fill prompt, 2026-09-21, shown at four times real speed.&lt;/figcaption>
&lt;/figure>
&lt;h3 id="2026-09-24-50-minutes-in-automatic-mode">2026-09-24, 50 minutes in automatic mode&lt;/h3>
&lt;p>With the supervisor in charge, π0.7 ran for 50 minutes on 2026-09-24. It completed 15 fill-and-empty cycles, and all 29 steps the supervisor started ended as done. A fill took 92 s at the median and an empty took 68 s. Figure 1 shows part of the run.&lt;/p>
&lt;p>The run stalled once. A bar lay left of the left gripper, hidden from every camera, while the box had crept to the left. The rig got out of the stall and kept cycling.&lt;/p>
&lt;p>A smooth 50 minutes took two fixes earlier that day. First, the supervisor&amp;rsquo;s scheduler spun in a busy loop and starved the control loop and the CAN threads of CPU time. Second, Physical Intelligence&amp;rsquo;s runner switches to blocking requests for about ten calls whenever one reply takes longer than its &lt;code>max_delay_ms&lt;/code> setting, 300 ms by default. That switch showed up as visible stop-and-go. Raising the setting to 700 ms kept the arms smooth for the whole run.&lt;/p>
&lt;h2 id="what-broke">What broke&lt;/h2>
&lt;p>Cables, not code. On 2026-08-03 the arms would not come up. Everyone suspected a USB-to-CAN adapter on the right arm, but the cause was a loose CAN cable on the left follower. A wrist camera also kept dropping from USB 3 to USB 2 mid-stream. When the fault came back on a different port and a different USB controller, it pointed at the cable, and a new cable fixed it. Every fault that day lived in a cable, a connector or a USB link, and each one surfaced as a software error that pointed somewhere else.&lt;/p>
&lt;p>Arms that idle between pickups. A π0.5 policy trained after the camera swap looked good, but its arms paused between pickups and sometimes swung back and forth between the home pose and the workspace. Every demonstration starts and ends at the same home pose. Near home, the demonstrated motion is about 9 times slower and stillness about 16 times more likely than in the rest of the data. In a rollout the episode never ends, so the arms sit exactly where the training data says to stay still. This explanation fits the data, but I have not tested it directly.&lt;/p>
&lt;p>Prompts the model barely saw. With 12 episodes each, the two bar-retrieval tasks were close to untrained. Given the same scene under three prompts, including the fill prompt, π0.7 moved the same arm every time. The scene picked the arm, not the prompt. On 2026-09-25 I also tried two recovery prompts it had never seen, &amp;ldquo;Push the box to the right with the left arm&amp;rdquo; and &amp;ldquo;Center the box&amp;rdquo;. It followed neither. So a smarter supervisor alone cannot fix a drifting box; the policy first needs demonstrations of the recovery.&lt;/p>
&lt;p>Latency. On 2026-09-25 I added artificial network delay with netem, the Linux network emulator, in steps up to 400 ms each way. Most of the extra wait came from Linux itself, which shrinks a TCP connection&amp;rsquo;s sending window whenever the connection sits idle between requests. One kernel setting, &lt;code>tcp_slow_start_after_idle=0&lt;/code>, kept runs clean up to 300 ms each way. At 400 ms each way the replies came too late, and the arm leaped and froze.&lt;/p>
&lt;p>A policy that parks. During the same sweep, π0.7 sometimes parked both arms at the start pose for 10 to 300 s while the network was healthy. Holding and releasing twice got it moving about 1 time in 4. The cause is still open.&lt;/p>
&lt;h2 id="lessons">Lessons&lt;/h2>
&lt;ol>
&lt;li>Check the physical layer first. Before you debug the policy, reseat cables, watch the USB link speed, and soak the cameras for a few minutes under load.&lt;/li>
&lt;li>Verify the statistics your policy normalizes with. A silent error in the 1st and 99th percentiles changes what the model sees, and no training curve will tell you.&lt;/li>
&lt;li>Demonstrations teach pauses as well as motions. If every episode ends at rest, expect the policy to rest. Two fixes I would try next are trimming the still frames at both ends and recording episodes that place several bars in a row.&lt;/li>
&lt;li>A rare task needs data, not a better prompt. A dozen episodes did not make the retrieval prompts work, and untrained recovery prompts did nothing.&lt;/li>
&lt;li>Keep the supervisor dumb and separate. Letting a cheap VLM press the same keys a person would press gave long runs for about $0.0013 per call, and the human stays in charge.&lt;/li>
&lt;li>Treat remote inference as a network problem too. Most of the trouble I hit came from TCP settings and runner timeouts, not from the model.&lt;/li>
&lt;/ol>
&lt;h2 id="links">Links&lt;/h2>
&lt;ul>
&lt;li>&lt;a href="https://arxiv.org/abs/2604.15483" target="_blank" rel="noopener">π0.7 paper&lt;/a> and &lt;a href="https://www.pi.website/blog/pi07" target="_blank" rel="noopener">blog post&lt;/a>, Physical Intelligence&lt;/li>
&lt;li>&lt;a href="https://arxiv.org/abs/2504.16054" target="_blank" rel="noopener">π0.5 paper&lt;/a>, Physical Intelligence&lt;/li>
&lt;li>&lt;a href="https://github.com/Physical-Intelligence/openpi" target="_blank" rel="noopener">openpi&lt;/a>, the open-source release of π0 and π0.5&lt;/li>
&lt;li>&lt;a href="https://github.com/huggingface/lerobot" target="_blank" rel="noopener">LeRobot&lt;/a>, Hugging Face&lt;/li>
&lt;li>&lt;a href="https://github.com/i2rt-robotics/i2rt" target="_blank" rel="noopener">i2rt&lt;/a>, the SDK for the YAM arms&lt;/li>
&lt;/ul>
&lt;p>Questions and comments are welcome on &lt;a href="https://www.linkedin.com/in/alexander-nettekoven/" target="_blank" rel="noopener">LinkedIn&lt;/a>.&lt;/p></description></item></channel></rss>