HexaShell: A Universal Harness for Robot Coding Agents in Simulation and the Real World

1 Introduction

Given a task and camera images, a vision language model can write a program against a robot’s interface, execute it and observe the outcome. Used in this way, frontier models now rival learned policies on manipulation benchmarks without a single demonstration. Their performance depends to a large extent on the harness that connects the model to the robot. With the same model, GPT-6-Astra, the RoboDojo team’s agent reaches 22.5% success in its evaluation, while HexaShell reaches 69.9% without task skills on our public test layouts.

RoboCasa365 illustrates both the task and the evaluation. A mobile robot works in a simulated kitchen, where preparing food or operating an appliance can require a sequence of actions. Across 50 target tasks, HexaShell reaches 85.6% overall, 21.2 percentage points above Awomo-0.5I. The largest gap is on composite tasks in the unseen split, where the success rates are 80.0% and 45.2%. Here “unseen” describes the policy pretraining split. HexaShell has a skill for each task and runs in held-out kitchens on held-out seeds.

Building such a harness poses two challenges. The first is spatial grounding. The model perceives objects as pixels, yet a motion can require a position accurate to about a centimetre. The second is task knowledge. An instruction can leave open what counts as completion, how deep a plug must be seated or which arm should move first. Vision-language-action models learn from demonstrations collected for each task and robot. Code-as-policies systems expose hand-written primitives. Other agents accumulate tools, programs or memories over repeated attempts.

Our earlier system, HexaShell-TaskTools, shows the cost of keeping this knowledge in code. An optimizer agent wrote about 78,000 lines of task tools for RoboDojo over 769 rounds. Once frozen, the tools passed only 36% of the development episodes on which they had been built.

Most of what a coding agent needs is common to many tasks. It needs to locate objects, verify a motion before committing to it, and receive clear feedback when something goes wrong. We implement these capabilities once as shared, tested code. A short, reviewed skill supplies the task-specific remainder, such as a success condition, a parameter revealed by contact or an order of steps.

We study three aspects of this design. First, we evaluate the general tools on their own, including how often the agent still needs to write its own vision code. Second, we test a single API, agent loop and sandbox on four benchmarks spanning three simulators and two kinds of arm, and on real dual-arm PiPER-X robots. Third, we compare runs with and without the natural-language skills to identify which facts help and which physical failures persist.

2 HexaShell

A single robot API, agent loop and sandbox serve every task in every setting. Each simulator or robot has its own adapter below the API and a prompt describing its world. The only task-specific input is an optional skill, about a page of guidance on the success condition, the steps that work and a few parameters from earlier runs.

At each turn, the model writes one Python cell. The cell runs in a sandbox and calls the robot API. Perception and motion run on the host. The model makes the decisions, and the API executes one request at a time and reports the outcome. The agent observes the scene through camera images, depth where available, camera calibration and the state of the arms. In simulation, the benchmark’s own checker judges success.

2.1 Agent loop and robot API

Cells run in a persistent Python session. Each returns the printed output, the value of the last expression, any exception and the requested images. The model ends an episode by giving a reason. The simulation pauses while the model deliberates, and only motion consumes the benchmark’s step budget.

RoboCasa365 and LIBERO-PRO add calls for their robots. The real robots currently support a smaller interface that omits find, measure and things. Before moving, the agent can run a motion as a dry run to check reachability, path, step cost and proximity to the other arm or a measured object. It can flag a press or insertion as a contact motion so that expected resistance does not stop the motion early.

Each result prints a short summary with warnings first and remains available as a structured object in code. An unreachable pose raises a typed exception with a reachable alternative. A stalled motion or a lost grasp interrupts the rest of the cell, giving the model a fresh opportunity to observe the scene.

2.2 Grounded perception

HexaShell separates identifying the pixels that belong to an object from locating them in the world. find queries SAM 3 with a few words, points marked on or off the object, or a box. It returns masks and their scores. measure uses NumPy and OpenCV geometry to convert the masks into a 3D box.

With RGB cameras, each outline defines a cone of rays. Their intersection across calibrated views gives a visual hull. Its top and footprint give the object’s centre and size, and a rotated rectangle fitted to the footprint gives a grasp angle. The result includes an error estimate. If only one view is available, the tool flags the uncertain height and proposes a wrist pose that would add a view. Where depth is available, it back-projects the mask’s pixels. For a single pixel, to_world recovers a world point using a known plane, a second view or the depth of the surface.

In a RoboDojo episode, the head camera found four bottles. The left wrist camera also saw the third bottle, giving a two-view position estimate within 2 mm. Only the head camera saw the first bottle. The agent moved its wrist to the suggested pose and measured again. The estimated error fell from ±17 mm to ±3 mm. The agent grasped the bottle and completed the episode.

2.3 Adapters, sandbox and prompt

Below the API, an adapter translates each motion into the setting’s actions and reads its cameras and arm state. HexaShell solves inverse kinematics for the ARX X5 arms and drives robosuite’s operational-space controller with a feed-forward planner for the Franka Panda. On the real robots, URAI plans and executes each motion.

Landlock and seccomp confine the Python worker to its sandbox. Every robot call goes to the host process, which owns the arms, cameras and perception. The worker cannot access the simulator, task code, skill sources or network.

All tasks in a benchmark share one system prompt. Its six parts describe the job and rules, the world and robot, the API, seeing and measuring, moving and handling objects, and checking results and recovering from errors. A task card adds the instruction, step budget, starting arm poses and initial images. RoboDojo also supplies the task’s score table.

2.4 Task skills

A skill is a Markdown file for one task. On LIBERO-PRO, RoboCasa365 and RoboTwin 2.0, each has five sections. Judging records what counted as success in earlier runs. Flow gives the steps that worked and the calls to use. Verified parameters records measured grasp heights, offsets and gripper openings. Common failures and fixes describes what went wrong and how to recover. Typical step costs helps the agent plan its budget. RoboDojo skills add final checks and, for the twelve Generalization tasks, guidance on clutter.

An analysis agent drafts each skill from earlier runs of related tasks, reading the cells, outputs and images and marking the evidence behind each claim. A person reviews every draft. A checking script enforces the rule that skills use only observable information, excluding simulator positions, internal names and checker thresholds. Median skill length ranges from 39 to 61 lines across benchmarks. Only three of the 205 skills contain code.

3 Experiments

We ask how far the general tools go without task-specific input, whether the harness transfers to other simulators and real robots when only the adapter and prompt change, and what a short task skill adds. We compare skills both with runs without them and with an earlier system that wrote a toolset for each task.

3.1 Experimental setup

We evaluate on four simulated benchmarks and on real robots. RoboDojo has 42 tabletop tasks for two ARX X5 arms in Isaac Sim, grouped into Generalization, Precision, Long-Horizon, Memory and Open. Its eight Open tasks receive no skills. On RoboCasa365 we run the 50 leaderboard target tasks for a Franka Panda on a wheeled base. On LIBERO-PRO we run the four LIBERO suites under position, object, language and task perturbations. On RoboTwin 2.0 we run 50 dual-arm tasks in randomized scenes with unseen instructions.

We develop prompts, tools and skills on development episodes. Test episodes differ in their layouts, seeds, kitchens, initial states or instruction templates. GPT-6-Astra writes the code in every simulated run. The main RoboDojo and RoboCasa365 runs use xhigh reasoning effort, except for RoboDojo’s Open tasks, which use high. The other simulations also use high. RoboDojo averages both success and task-progress score first over tasks within each dimension and then over the five dimensions. The other benchmarks report the fraction of successful episodes. Our results are evaluations on public episodes, with fewer episodes than the official protocols.

3.2 Task-agnostic performance

On RoboDojo we run 30 test episodes per task, using layouts 10–19 for each of seeds 0–2. We report the three-seed mean. The standard deviation is below one percentage point. The official October 30, 2026 leaderboard uses 50 episodes per task and seed on private layouts. Without task-specific input, HexaShell reaches 69.9% success and a score of 75.7 on our layouts. Q0-Preview, the strongest entry on that leaderboard, reaches 68.9% and 74.2 respectively. The harness comparison is clearest with the model held fixed. Using GPT-6-Astra, the RoboDojo team’s agent reaches 22.5% success, RoboDawn reports 35.7% zero-shot and 47.2% one-shot, and HexaShell reaches 69.9% with its general tools, prompt and loop.

* Our own evaluation on public test layouts. As of November 5, 2026, this HexaShell version is unlisted. Q0-Preview is the official simulation leaderboard leader in overall success rate (68.87%). The other rows report official RoboDojo results. The Open tasks use the general harness alone in both HexaShell rows.

The agent operates almost entirely through the API. Across our two RoboDojo runs, goto accounts for 41–43% of robot calls, look for 19–21% and measure for 10–11%. The median episode measures 16 times without skills and 11 times with them. The agent often measures again after a hand moves an object or a single view leaves its height uncertain. Only about one episode in ten processes images in agent-written code. More than half of those episodes involve the two tasks with T-shaped blocks.

3.3 Transfer across simulators and real robots

The same API, loop and sandbox run on three simulators with two kinds of arm, fixed and wheeled bases, cameras with and without depth, and on real robots. Each setting supplies its own adapter and prompt.

The three groups use different episodes and protocols. Our result is listed on the agentic track as our own evaluation on 30 held-out seeds per task. The policy entries use 50 episodes per task.

On RoboCasa365, HexaShell reaches 59.3% without skills and 85.6% with them. The largest margins over Awomo-0.5I are on the composite tasks. On the same 1,500 episodes, 466 succeed only with skills and 72 only without (p < 0.001). Many gains are on tasks at the sink and on appliance doors and buttons. For example, washing lettuce goes from 1 to 26 successes out of 30 and loading the dishwasher from 3 to 27. The run without skills also retained a longer image history. “Composite unseen” refers to the policy pretraining split. Our agent has a skill for each of these tasks and is tested on held-out kitchens and seeds.

On LIBERO-PRO, HexaShell reaches 78.3% without skills. With skills available it succeeds in 1,509 of 1,560 episodes (96.7%) across all 16 suite–perturbation combinations. The two aggregate runs use different initial states. A paired control on the same initial states of 54 tasks raises successes from 252 to 531 out of 540. Other coding-agent results cover different subsets. PyRUA-Lean reports 94.0% on 40 tasks, HarnessPAI 96.5% on six suite–perturbation combinations, and ASPIRE 96.0% with GPT-6-Astra or 98.8% with Claude Opus 5.5 on eight.

On RoboTwin 2.0, HexaShell without skills passes 411 of 500 episodes (82.2%) on learning seeds with seen instructions. With skills it passes 726 of 750 (96.8%) on randomized test scenes with unseen instructions. These runs use different seeds and instructions. Our 96.8% is a self-evaluated result and is unlisted. As of November 5, 2026, GO-3 Nano Preview leads the official clean-to-random (hard) leaderboard at 81.48%, rounded to 81.5% in the chart. A separate published agent result reaches 90.4%.

Real dual-arm robots

The same loop, sandbox and API drive three dual-arm PiPER-X robots with RGB-D head and wrist cameras. The 907-line adapter passes each goto to URAI for planning, collision checking and execution. A refused plan is reported as an unreachable pose. On hardware the agent reads pixels from the images and obtains world coordinates with to_world and measured depth. GPT-6.1-Terra writes the action code. In the absence of a task checker, GPT-6-Astra supervises and judges each episode from fresh images and arm feedback.

The table reports ten episodes per task, drawn from the recorded trials. For cover blocks, the agent notes the colours of three blocks, covers them with cups from left to right, then uncovers them. All 53 finished episodes succeeded, and the table counts the last ten. The counts come from the supervisor’s records. Full manual verification of the episodes remains outstanding, and prompts and supervision rules changed between episodes.

3.4 Effect of task skills

HexaShell-TaskTools, listed on the RoboDojo leaderboard as HexaShell-Preview, stored task knowledge in a separate set of tools for each task. After a failure, an optimizer agent wrote or repaired those tools and their tests, then the acting agent retried the layout. Over 769 rounds the optimizer produced 327 commands in about 78,000 lines of Python. Once frozen, the tools passed 152 of the 420 development episodes on which they had been built. An audit found object sizes copied from the scenes into the tools for seven tasks. HexaShell instead stores its task guidance in 34 RoboDojo skills, about 2,700 lines in total. Only three contain code.

TaskTools’ result comes from RoboDojo’s official hidden-seed evaluation. The ordering also holds on 420 public episodes of seed 0, with the same layouts outside Generalization. The frozen task tools pass 157 episodes, the shared tools pass 284 without skills, and the shared tools with skills pass 351. The loop, perception, motion and prompt changed between the two systems. To isolate the skills, we compared the same 1,020 HexaShell episodes outside Open. Of these, 234 succeed only with skills and 37 only without (p < 0.001).

The largest gains are on tasks that the general tools alone find difficult. Out of 30 episodes per task, make kong rises from 0 successes to 30, fasten screws from 10 to 30, plug in charger from 9 to 28, pack objects into box from 8 to 26, and build tower from 14 to 29. No task loses more than two successes. In these tasks the agent can already perceive, grasp and carry the objects. What it lacks is task knowledge of three kinds.

• Success conditions left open by the instruction. In make kong, the tile to place is the one drawn from the wall. Ten development runs moved the opponent’s discard, and all scored zero. For packing objects, a box that shifted but stayed upright still counted. Without this guidance, the agent gripped the box in all 22 failed test episodes, often before handling an item.
• Parameters that only contact reveals. In fasten screws, the bolts are fixed to the table. The skill tells the agent to carry each nut to its bolt. A pushed turn down to a grasp height of 0.793–0.800 m seated eight of eleven first placements during development.
• The order of steps. The build-tower skill specifies one pillar from each side of the table for each level. The three development runs that placed both lower pillars with one arm never got past the lowest score.

In plug in charger, all 13 development attempts with at least 3° of tilt failed. None of the eight attempts that released the charger before it was seated counted. The 61-line skill tells the agent to check both arms with a dry run and move the power strip if neither can insert the charger level. In one test episode, the left arm moves the strip 37 cm, the wrist camera locates the socket, and the right hand seats and releases the charger. The episode succeeds at step 179 of 400. In press by number, the skill instead supplies a counting rule. The agent prints each tally after pressing a button and finishes in five Python cells. Across RoboDojo episodes outside Open, skills reduce model turns by 17%, robot steps by 19% and reasoning tokens by 44%.

3.5 Failure analysis

The agent ends most failed episodes itself, usually when the remaining steps cannot pay for another attempt. Most still earn part of the task score. The most frequent failure is an object slipping from the fingers. With skills, 153 of the 1,020 RoboDojo episodes outside Open contain a slip. Their success rate is 46%, against 91% for the others.

In the Open task of storing tools in a toolbox, no episode succeeds. As the arm lifts a long tool gripped near its middle, the tool rotates between the fingertips and slides out. Scripted tests outside the agent also failed to place the pliers. The failure originates in grasping and motion below the API. Improving grasp stability there would benefit every task that uses the shared tools.

4 Related Work

Code generation for robot control. Code as Policies gives a language model perception and control primitives that it composes into programs. ProgPrompt and Inner Monologue use feedback from the scene during planning, and SayCan grounds language in a library of skills. VoxPoser composes value maps for a motion planner. ReKep and MOKA use visual keypoints to guide motion. HexaShell also relies on hand-written primitives. The model uses them in a closed loop, choosing its next Python cell after reading the output and images from the last one.

VLM agents as robot policies. A frontier VLM can select actions directly through a fixed interface. The RoboDojo team evaluates GPT-6-Astra with an end-effector interface, and RoboDawn studies the same model in zero-shot and one-shot settings. Other systems have an agent steer a learned policy. In HexaShell, perception calls return measured 3D positions with error estimates, motion calls check a pose before it consumes steps, and errors tell the agent what it can change.

Self-improving coding agents. Reflexion retains verbal lessons from failed attempts. Voyager builds a library of code skills. In robotics, URAI pairs an agent that writes motion tools with an execution agent that calls them. EmbodiedEvo turns physical feedback into code skills and lessons. MEMENTO retains a memory of past programs. Our earlier HexaShell-TaskTools also kept and repaired code for each task. In the present system, the tools are shared and fixed. What we retain for an individual task is a short, reviewed skill in natural language.

Visual grounding for manipulation. The SAM family segments objects from points or boxes, and SAM 3 also accepts short phrases. Calibrated multiple-view geometry can recover the world point seen from different cameras. In HexaShell the model names an object or selects pixels, SAM 3 returns the outline, and geometry or measured depth yields its position and an error estimate. Asking the wrist camera for a further view lets the agent act on that uncertainty.

Learned policies and benchmarks. Vision-language-action policies learn manipulation from demonstrations. LIBERO-PRO perturbs objects, positions, instructions and tasks to test transfer. RoboDojo sets a step budget, varies seeds and pairs clean layouts with cluttered ones. Several comparisons in these benchmarks therefore involve systems with different sources of experience. HexaShell uses no demonstrations, while skills can draw on its own earlier runs of a task.

5 Limitations

Our results come from our own evaluations on public test episodes, with fewer episodes than the official protocols. A skill requires runs of its task, so a new task starts at the level of the general components. We have yet to measure how few runs suffice or to isolate the contributions of SAM 3 and the geometry by switching them off. Slips of held objects remain the most common failure. The real robots run without SAM 3, and the hardware study covers only a small number of episodes.

6 Conclusion

We presented HexaShell, a harness in which a VLM controls robots by writing one Python cell per turn against a small robot API. Task-agnostic tools carry the agent most of the way. SAM 3 segments objects, and NumPy and OpenCV geometry localizes them across four simulated benchmarks. The real robots use the same loop, sandbox and geometry, with the model choosing the pixels itself. About a page of guidance distilled from previous runs supplies the remaining task knowledge, including the success condition, parameters that only contact reveals and the order of steps.

Most remaining failures are slips of held objects and final placements that must be exact. Better grasps and motions below the API would benefit every task at once. They are the focus of our next steps, together with official evaluations, more real-robot episodes and an ablation without SAM 3.