Teaching a Humanoid to Pack Boxes with π0.7 in Four Days

Four fine-tunes of Physical Intelligence’s π0.7 on an Astribot S1 at Formic, from the first demonstrations to a full pack-and-push cycle

Figure 1: The Astribot S1 runs my fine-tuned π0.7 policy on 2026-10-11. It sets a box on the table, packs four bags, pushes the full box onto the conveyor and fetches the next box. Played at 2x speed.

In early October 2026, I taught an Astribot S1 humanoid a three-stage packing task at Formic. Formic brings robot automation to manufacturers across the US, and humanoids are one way to reach lines where classic automation struggles. The robot runs π0.7 (pi0.7), the newest robot foundation model from Physical Intelligence (PI). I fine-tuned it on my own demonstrations through PI’s API. I recorded the first packing demonstrations on 2026-10-07, and on 2026-10-11 the robot ran the whole task on camera (Figure 1), after four fine-tunes in between.

In four days, a few hundred teleoperated demonstrations and four fine-tunes took π0.7 from a first test run to the full task: setting out a box, packing four bags into it and pushing the full box onto the conveyor. Each fine-tune trained on PI’s servers in four to nine hours, so every day held one round of data, training and robot tests.

Table of Contents

Setup

The robot

The Astribot S1 is a wheeled humanoid with two arms of seven joints each, a two-joint head, a four-joint torso and two grippers. Its policy interface has 25 values per step, namely 20 joint positions, 3 base velocities and 2 gripper positions. The base never moved in my demonstrations, so the base velocities are exactly zero in every training frame. On the robot, my code sets them to zero again, and the robot bridge clamps the base speed to zero as a second guard.

The S1 carries five cameras. I drop the head’s stereo pair and feed the policy the color streams of the other four, one each in the head, the torso and both wrists, at 640x360 pixels and 30 frames per second. Figure 2 shows what the head camera sees during a run.

Figure 2: The robot’s head camera during a run of the fourth fine-tune (v4) on 2026-10-11. The policy also sees the torso and both wrist cameras. Played at 2x speed.

Compute and software

No policy runs on the robot. π0.7 runs on PI’s servers, and the robot reaches it over a WebSocket, a standard two-way web connection. The S1 has two onboard computers, an x86 PC and an NVIDIA Jetson Orin, which run its drivers, cameras and web interface. My laptop sits on the robot’s network and runs three Docker containers, packaged programs that run side by side:

  • the bridge to Astribot’s software development kit (SDK), which takes control of the robot and executes joint commands at 100 Hz;
  • a bridge from the robot’s ROS 1 to ROS 2 (ROS, the Robot Operating System, is the standard robot middleware);
  • an inference kit with my model file, which sends observations to PI and turns the replies into commands.

Each request carries the latest camera frames, the 25-value state and a short text prompt. The server resizes every image to 448x448, keeps a history of six frames per camera, and returns a chunk of 50 future commands. PI’s real-time chunking runner, a client that requests the next chunk while the current one still plays, asks for a new chunk every 25 steps and joins it onto the motion in flight, so the arm does not stop while it waits for the network. A healthy round trip takes 175 to 190 ms.

What π0.7 is

π0.7 is a vision-language-action model (VLA), a model that maps camera images and a text instruction to robot actions. According to PI’s paper, it has about 5 billion parameters, a 4-billion-parameter vision-language backbone started from Gemma 3, Google’s open model family, and an 860-million-parameter action expert trained with flow matching, a training method for generative models. It predicts chunks of 50 actions and takes up to four cameras with up to six history frames each. I used the history variant of the base model and fine-tuned it through PI’s partner API, which opened as a beta on 2026-10-13.

Data collection

I recorded every demonstration by teleoperating the S1, steering it remotely, in Astribot’s own data-collection mode. The task copies a packing line in three stages, and each stage has its own episodes and its own text prompt:

  1. grab a box on the left and place it on the table, which takes an empty box from the side table on the robot’s left and sets it on the work table;
  2. put a red bag in the box until the box has 4, which picks bags from the side table one at a time;
  3. push the box onto the conveyor on the right, which slides the full box onto a roller conveyor.

The first 132 episodes, recorded on 2026-10-07, held 42,009 frames, about 23 minutes of robot time. The table below shows how the data grew over the four rounds.

Promptv1v2v3v4
Grab a box4576110126
Pack clear bags50103141141
Pack red bags57
Push onto the conveyor37476080
Center the box161616
Helper prompts (looks, idle, go home)161616
Total episodes132258343436

The first product came in clear plastic bags. On 2026-10-10 I switched to red snack bags. That switch took 57 new packing episodes, plus 36 new grab and push episodes with the red bags on the table.

Three data problems cost me time:

  • The capture tool never exports the task text the operator types. I checked 3,785 episodes in nine task folders, and every one stored only its task ID. So the prompt must go into the upload metadata, which PI prefers over the dataset’s own field.
  • Four episodes sat in the wrong task folder, because the operator kept recording under the old task ID. No metadata showed the error; only the video did.
  • The merge tool ran in a Docker container as root and wrote the merged videos as root-only files. The upload then failed on the first video, although the dry run passed, because the dry run reads only metadata.

Method

A test run first

Before the packing work, I ran one fine-tune on 100 episodes of a simpler flip-the-box task, about 15.5 minutes of data, to test the path from upload to robot. That run trained in about 2.5 hours. I then pushed three recorded frames through the hosted model from my laptop, with no robot involved, and compared the returned chunks with the recorded actions. The mean error over arms, head and torso was 0.003 to 0.005 rad, close to the 0.002 to 0.009 rad of an earlier model for the UR5e, a six-joint industrial arm, which went on to work on its robot.

The same check showed small base velocities of about ±0.006, although the training data held exact zeros. So the model was interpolating, and my model file zeroes the base on every row.

I also trained on absolute joint targets, PI’s default, rather than on deltas. The base action is zero in every frame while the base state carries odometry noise, the jitter in the base’s estimate of its own motion. With deltas, the model would learn to predict that noise with the sign flipped, and the server would add the state back at run time, which turns noise into real base commands.

The four fine-tunes

All runs made about 50 passes over their data at a batch size of 256, so the step count grew with the data. A run went from upload to a hosted checkpoint within hours, so I could try a change on the robot the same day or the next morning.

RunTrained (UTC)EpisodesPromptsStepsTraining timeWhat changed
v12026-10-0813238,2493 h 39 minThe three stages
v22026-10-09258915,6995 h 42 minBox centering, helper prompts, more stage data
v32026-10-10343919,8997 h 28 min85 more stage episodes
v42026-10-114361025,0999 h 04 minRed bags, renamed packing prompts, weighted mix

v2 added a prompt that centers the box and a few helper prompts, such as a scripted look to the left for the next box and a return to the working pose. It also renamed the push prompt. PI cannot rename a stored prompt, so all 258 episodes went up again.

For v3, PI does not skip episodes it already holds, so I uploaded only the 85 new ones and trained on the v2 set plus the new set.

For v4, I renamed the packing prompts so that each one names what the camera sees, put a clear bag in the box until the box has 4 and put a red bag in the box until the box has 4. Renaming meant uploading all 436 episodes again, which took 24 minutes. I split the data into two sets, red-bag packing and everything else, and weighted them 0.289 and 0.711 so that red and clear packing get equal shares of each batch.

PI reports a validation error on held-out episodes. v2 ended at a mean absolute error (MAE) of about 0.16. v3 and v4 each report two numbers, likely one per data set, and v4’s (0.49 and 0.47) sit well above v3’s (0.15 and 0.41). The sets, and so the held-out episodes, differ between runs, so these numbers do not rank the runs. Only robot trials can rank them.

Running the policy

A small menu on the laptop sends the stage prompts. Each number key sends one prompt, and a switch reaches the robot within about 1 s while it keeps moving, with no reset. Before a stage starts from rest, a warmup walks the robot to the stage’s start pose, the median first frame of its training episodes, if the robot is more than 0.3 rad away.

The menu also has an automatic mode. Gemini Flash-Lite, a small multimodal model from Google, looks at five recent head-camera frames and a reference picture of the station every 2 s and picks the next prompt. It switches only after two agreeing answers, it counts the bags in the box, and it costs about 4.3 USD per hour.

Two settings shape the motion. A playback speed of 1.0 plays PI’s actions at the 30 Hz of the training data, and 1.2 plays them 20 percent faster. A smoothing filter, an exponential moving average (EMA) on the 20 joint commands, removes a small tick-to-tick jitter in the hosted model’s output.

The rule that shapes my model file is simple. In the inference kit, an exception in the model makes the client return nothing, and the robot keeps executing the rest of the old chunk. So the model must never raise. The few lines below show the idea in my own simplified code.

def infer(self, obs: dict) -> np.ndarray:
    try:
        chunk = self.client.infer(obs)        # (50, 25) joint commands
        chunk[:, 20:23] = 0.0                 # the base never moves
        return chunk
    except Exception:
        return np.tile(self.last_command, (50, 1))  # hold in place instead of failing

Results

Figure 1 shows the v4 checkpoint on 2026-10-11, the run I filmed for the demo. In 92 seconds of real time, the robot packs two boxes and pushes both onto the conveyor. In the first cycle, the fourth bag lands at 37 s and the push starts at 40 s. Figure 3 shows a stretch of the same run at real speed. The demo ran with the playback speed at 1.3 and the smoothing at 0.5.

Figure 3: The same v4 run at real speed, from grabbing the box to packing the first bag and reaching for the second.

The robot is still slow, which is why Figure 1 runs at 2x.

What the demo does show is that a few hundred demonstrations and four short fine-tunes got π0.7 through a three-stage task on the humanoid. The switch to red bags took one round of data and one overnight run.

What broke

Most of the problems sat in the plumbing, not in the policy:

  • The arm jerked back at every replan. In its original mode, the inference kit stopped sending commands during each call to PI, which happened every 20 rows. I measured 51 stalls of 180 to 240 ms in 45 s. PI’s real-time chunking runner, the same one the UR5e and the YAM arms (a two-arm research setup) use, fixed the stalls. A second bug made PI’s replies arrive one step late, because the kit reset the connection while the policy still used it. Now one thread talks to PI, and a reset only raises a flag.
  • Before the packing work, the head and torso cameras reached the kit at 6 Hz. The bridge sent raw 2.76 MB frames over the network, and the messaging layer split each one into many packets. Switching to compressed images restored 30 Hz and cut outbound traffic from 117.74 to 0.28 MB/s.
  • The laptop failed silently twice. Its firewall dropped the messages between the containers, and a virtual private network (VPN) interface gave the messaging layer a second network to send on. A new box between the laptop and the office network also turned out to be a router, not a switch, and cut the laptop off.
  • From the standing home pose, the first chunk asked for a 0.40 rad torso move, because every stage starts bent forward, about 0.58 rad from home on that joint. The warmup to the start pose fixed it.
  • At the bridge’s old filter setting, the arm trailed the policy by up to 0.42 rad on elbow swings. A higher setting fixed the whipping.
  • The warmup first drove the grippers to 100, which is closed on the S1, and the policy then reopened them. Now the warmup opens them.
  • During one home move, the automatic mode pressed a prompt, the waist jumped 0.65 rad, and the drivers shut off. Now the kit refuses any prompt until the home move ends.
  • The hosted model’s joint commands step back and forth by about 1.3 mrad every tick, although the training actions are smooth. An EMA at 0.3 cuts that noise to about a quarter, at the cost of about 80 ms of delay.
  • On the demo day, v4 showed bursts in 4 of about 26 runs. After a pause of 70 to 900 ms in the command stream, 10 to 16 joints jumped at once by 0.6 to 0.8 rad of raw command for two ticks. Slow replies that drain the chunk queue are the likely cause, made worse by the faster playback, but I have not confirmed it.
  • Once the head camera simply stopped, and the automatic mode did nothing, because the policy had no full set of images. A restart of the camera stack fixed it.

Lessons

  • Test the hosted model offline before the robot moves. Three recorded frames through the server caught the drifting base velocities before any rollout.
  • Never let the model file raise. A crash should become a hold, not a silent replay of old commands.
  • Plumbing faults look like bad policies. A stale camera stream feeds a six-frame history model old images, which fails quietly and looks like a weak policy.
  • Put the prompt in the upload metadata, and check what the server stored. A prompt cannot be renamed later without uploading the data again.
  • Name what the camera sees in the prompt. Splitting “clear bag” from “red bag” gave the model a visual cue for each product.
  • Validation error does not rank runs that trained on different data.
  • Fast fine-tunes change how you work. With runs of a few hours, each day held one round of data, training and robot tests.

Since the demo, the S1 has moved on to whole-task demonstrations cut into stage spans, stage ends set by the inference software and a MuJoCo digital twin of the station. The follow-up post covers that work.

Related