What happens when a coding agent works on a robot the way it works on a software repository? It can inspect an observation, run a command, read the failure, and change the code. The important question is whether those changes become useful capabilities that survive the next attempt.
Our paper, HexaShell: A Thin Harness for Exploring the Limits of VLM Agents as Robot Coding Agents, studies this process across all 42 tasks in RoboDojo. An unchanged coding agent controls a simulated dual-arm robot through one command line. After a failed episode, an optimizer agent diagnoses the trace and repairs reusable perception or execution tools. The tools persist; the foundation model's weights stay fixed.
Our latest aggregate result is a 46.85% success rate and a progress score of 53.18. The study also examines what makes the approach work: calibrated RGB geometry, explicit execution feedback, and code that can be tested and repaired. Its harder failures reveal the next challenge: turning a correct diagnosis into reliable motion during contact.
01 · A small interface to the physical world
HexaShell gives the acting agent one executable, robo, a short interface manual, and observations from three cameras: one head camera and two wrist cameras. Each RGB image has a resolution of 640 × 480. The observations also include camera calibration, joint angles, and both tool-center-point poses. The standard observation contains no depth image or object poses.
The base commands move and rotate an arm, set a gripper orientation, open or close the gripper, return home, wait, request observations, and finish the episode. Each command returns structured feedback: whether the motion was accepted, where the arm actually reached, and how much action budget remains. Task tools become additional commands with their own interface descriptions.
The harness validates requests, constructs a straight Cartesian path, solves inverse kinematics, and times the resulting joint trajectory under the robot's limits. Execution uses RoboDojo's official action interface at 25 Hz. The harness contains no task strategy, and its collision checks cover the robot and table rather than a complete model of surrounding objects.
This division makes responsibility visible. The model decides what to attempt. Tools implement measurements and motion routines. The harness translates requests into robot actions and reports what happened. The acting agent runs in an isolated container without simulator internals, scoring code, or direct access to ground-truth object state.
02 · From pixels to positions
A visual model can identify a block or a cup, but a robot command needs a position in meters. A plausible-looking estimate can still miss a grasp by several centimeters. HexaShell therefore separates selecting the relevant visual evidence from computing its geometry.
A calibrated image pixel defines a ray in the world. To locate a point along that ray, the tools add another constraint. If the point lies on a measured plane, they intersect the ray with that plane. If two cameras see the same point, they triangulate its position from the two rays.
The shared table_plane tool estimates table height by testing candidate planes and comparing images warped between calibrated cameras. Other tools measure surfaces such as a block's top face or a container's rim. Corresponding points can come from pixels selected by the agent, or from image operations such as template matching, contours, and circle fitting.
When a useful second view is missing, the robot moves a wrist camera. This makes perception an action with a cost: moving the camera consumes the same limited robot steps as manipulating an object. Measurements that require no movement do not consume action steps.
The model identifies what to measure; executable geometry determines where it is. This makes errors easier to inspect. A failed measurement can expose an incorrect assumption about a surface, a bad correspondence, or an unsuitable camera view, each of which can be addressed in code.
03 · Keeping improvements as code
The acting and optimizer agents use the same unmodified Codex command-line agent with GPT-6-Astra at medium reasoning effort. Their roles operate on different timescales.
During an episode, the acting agent selects commands from observations and execution feedback. Between episodes, the optimizer reads the command log, camera frames, acting-agent transcript, and current tool code. It records a diagnosis, edits a tool, and writes or updates unit tests. The simulator then restarts the same layout so the revised tools can be tried again.
A delivery check verifies that the required files exist, tools load, and interface descriptions stay within the permitted format. An invalid delivery is reverted. These checks establish that a tool can be used; success still comes from executing the task and passing the benchmark's own checker.
The optimizer can modify task tools and notes, but cannot change the harness, shared manual, or shared tool pool. A successful attempt adds a short account of what worked to a playbook. Development allows up to five edits per layout, then eventually freezes the tool set for evaluation without an optimizer.
In the paper's RGB development runs, the process produced 327 task commands across 769 optimizer rounds. A repair round following failure took a median of 6.3 minutes, and 92% of those rounds ran the task's tests. These are development costs, separate from the robot's action budget. What accumulates is executable knowledge and supporting notes, without task-specific updates to the foundation model.
04 · How we evaluate the tools
RoboDojo contains 42 simulated tasks for a dual-arm ARX X5 in Isaac Sim 5.1. They cover Generalization, Memory, Long-Horizon, Precision, and Open tasks. Twelve Generalization tasks also have cluttered variants, adding 13 to 18 distractor objects and bringing the total to 54 variants.
In paper v1, tools are developed on ten layouts of evaluation seed 0 per task. Generalization tasks divide that allowance into five standard and five cluttered layouts sharing the same tools. Frozen evaluation then uses ten previously unused layouts from the same seed, again divided between the two variants where applicable. This produces 420 evaluation episodes across the 42 tasks.
Success is determined by the official checker. The overall metric weights the five task types equally; Generalization first averages its standard and cluttered variants. Consequently, the leaderboard-style score is different from a plain average across all 54 variants.
Development-layout retests answer a separate question: how different perception methods behave on matched layouts. Those comparisons use the same model and harness with triangulation, a monocular depth network, or simulator-rendered depth as a reference. Rendered depth is unavailable in the standard observation.
The paper's leaderboard comparisons also use different evaluation budgets: most policy entries report 50 episodes on each of three seeds. The HexaShell results establish performance under the stated frozen-tool protocol, not an identical evaluation of every system. Fresh layouts within one seed should also be distinguished from unseen task families, new robot embodiments, and deployment on physical hardware.
05 · Results and what explains them
| System | Success rate | Progress score |
|---|---|---|
| HexaShell | 46.85% | 53.18 |
The updated overall results above supersede the headline totals in paper v1. The ablations, capability breakdowns, and case studies below are from paper v1, dated October 16, 2026; they are not a breakdown of the updated aggregate score.
Calibrated geometry improves measurement
On matched development layouts, triangulation exceeds the Depth Anything V2 configuration by 8.7 points on the overall success metric. Across 420 paired episodes, triangulation succeeds on 57 layouts that the depth-network version fails, while the reverse occurs on 23. This isolates a practical benefit of geometric measurement within the tested system.
The depth-network analysis helps explain the difference. On 300 frames from eight tasks, median object-pixel errors were 41 mm at the head camera and approximately 100 mm at the wrists, even after using the table to fix scale. Many grasps and placements need accuracy closer to a centimeter.
A second view can earn back its cost
Explicit viewing moves account for at least 3.6% of executed steps in the triangulation runs. On the 180 layouts where the RGB agent moved to look, it succeeded 39 times, compared with 22 for the depth-network version on those same layouts. The accounting excludes viewing embedded inside larger tools, so the measured overhead is a lower bound.
Contact requirements separate strengths from weaknesses
The v1 capability map reports 92.5% success on pushing, sliding, and pressing, and 65.0% on memory tasks, but only 8.3% on tight insertion. Geometry and persistent tools help when a measured position can determine a useful action. Millimeter-scale alignment during contact remains much harder.
06 · Seven repairs, from diagnosis to execution
The paper follows seven failures through an optimizer edit and the next attempt on the same layout. These cases show concrete mechanisms behind improvement. A successful next attempt alone does not establish that the edit caused everything that changed, so the study checks that the revised code ran and addressed the diagnosed failure.
Push T: compare physical area, not pixel counts
A height check rejected matching views because a distant camera and a nearby wrist camera contributed different numbers of occupied image-grid cells. The agent then used the table height for the top of the block, corrupting its placement check. The optimizer changed the comparison to sum each pixel's physical footprint on the plane. The next attempt measured the block height correctly and succeeded in 354 of 600 action steps.
Swap T: separate the top from the side
A color mask mixed bright top faces with darker side faces of the same blocks, distorting height and heading estimates. The repair matched colors by hue and retained the bright top surface in each view. Outline-fit error fell from 2.6 mm to 0.4 mm, and the next swap passed. A synthetic test reproduced the visual conditions behind the failure.
Play Xylophone: measure the held tool from two views
A single-camera estimate misplaced the mallet head by 22.9 mm. The robot completed eight strikes, but the checker counted none. The optimizer revised the measurement tool to match the head in a second image and triangulate its center. After the agent moved the cameras to obtain a complete view, the error fell to 2.7 mm and the episode succeeded in 377 of 500 steps.
Conveyor picking: schedule what the arm can reach
The target was localized reasonably well, but the tool requested a closing schedule shorter than the arm's approach time. The repair estimated motion cost before committing. On the next attempt it delayed closure and moved the interception point farther along the belt, allowing the gripper to meet and lift the moving object.
Tic-tac-toe: combine motions to preserve the budget
Repeatedly turning the empty gripper at home consumed steps needed for the final ring. The optimizer added a route that turns while moving above the source, retaining clearance checks. The next attempt completed the board in 1,028 of 1,100 steps. The improvement came from execution efficiency rather than a new strategy for the game.
Fold clothes: require a grasp inside the fabric
The original check allowed a sleeve grasp too close to its edge. The revised tool required cloth around the point on every side and searched inward when necessary. The next attempt folded the shirt successfully, but the same layout failed in the later frozen retest. A useful local repair had not yet become reliable performance.
Insert tubes: distinguish uncertain depth from an obstacle
A fallback marked unmeasured space along camera rays as occupied, rejecting every route after a lift. The repair preserved confirmed obstacles while reconsidering the original path from the reached joints. The next episode succeeded, although one insertion still relied on dropping the tube and later tubes needed individual base commands chosen by the agent. The case fixes an unnecessary stop without solving precise contact control.
07 · What the current study does not resolve
Recognizing a failed action is different from being able to repair it physically. In the v1 development-layout analysis, 57% of non-tool-error failures used at least three quarters of the action budget, while 30% ended with the agent declaring completion despite the checker disagreeing. These signatures suggest both execution limitations and gaps in visual verification.
Precise contact is especially demanding. A head-camera pixel can cover several millimeters, and a helpful wrist view may need to be acquired while the gripper already holds the part. Cloth and fluids also change during manipulation. A tool that commits to one interpretation and a planned motion can miss changes that require correction during execution.
Latency is another limit. The paper reports a median retest duration of 739 seconds for 26 seconds of robot time, largely due to model latency. RoboDojo does not charge thinking time against the action budget. This evaluation therefore does not demonstrate a controller that can react at real-world control rates.
The tool library also has a development history. RGB runs started from earlier depth-based tools; 25 tasks included optimizer notes derived from public checker definitions, and some initial tools came from work by team members. An audit replaced fixed object geometry found in seven task tool sets with measurements made during the episode. The results should be understood as improvement of a developed tool system, not capability emerging from an empty library without human input.
Broader tests are still needed across additional seeds, unfamiliar objects, task families, and physical robots. They should account for the full cost of developing, verifying, and maintaining tools as well as the cost of using them.
08 · Toward tools that keep correcting
HexaShell connects physical execution to a familiar programming workflow: inspect evidence, diagnose a failure, repair a tool, test it, and keep the useful change. The small harness makes that process explicit, while calibrated RGB geometry gives the agent measurements it can reason about and code it can inspect.
The next step is to extend this feedback into contact itself. Tools need to keep measuring a held part, detect how the interaction is changing, and adjust the motion while it unfolds. More capable physical coding will depend on this connection between durable software improvements and reliable feedback during action.
Read the HexaShell paper for the full protocol, per-task results, and development audit, or explore the released tool code. Our earlier work on Physical Coding and URAI develops the broader direction.