LADDER: A Long-Horizon Robot Manipulation Benchmark

Petko Petkov1, Giuliano Albanese1, Nikolay Nikolov1, Luc Van Gool1, Jonathan Tremblay2,†, Danda Pani Paudel1,†
1Institute for Computer Science, Artificial Intelligence and Technology 2NVIDIA †Equal supervision
INSAIT NVIDIA
Paper Code Hugging Face
Overview of our benchmark and key findings

Overview of our benchmark and key findings. Across all suites, orchestrating policy execution with sequential subgoal decomposition and visual verification substantially outperforms issuing the full instruction directly to the policy (bottom right). See Tables 1 and 3.

Abstract

General-purpose robot manipulation policies learn short-horizon skills effectively, but degrade rapidly on long-horizon tasks. Existing benchmarks cannot isolate failures because task horizon is either too short or entangled with scene and object variations. We propose LADDER, a long-horizon robot manipulation benchmark scaling from 1 to 10 subgoals within the same environment over three suites (SimplerEnv-Long, RobotArena-∞-Long, and RoboLab-Long), 145 tasks, eight categories, two embodiments, and 2,024 simulated episodes per system. We evaluate two orchestration paradigms: text subgoals for learned low-level policies (VoLo) and tool selection orchestration (CaP-X, GPT-Policy, PSP-Agent, TiPToP). Given the full instruction directly, low-level policies achieve 85.4% on single-subgoal tasks, but drop to near zero on extended tasks. While orchestration remedies this, it remains bounded by physical execution errors of the low-level policy rather than high-level planning and achieves only ∼25%. Tool selection orchestration with GPT-Policy on GPT-6 Astra significantly outperforms learned-policy methods, but incurs significant time and cost budget and succeeds only ∼62%. Real-world experiments across two embodiments confirm the collapse of the low-level policy and the gain from orchestration. Project website: ladderrobotics.ai

Key findings

1 87.2% → 0.0% Robot policies fail on long instructions Given the whole instruction, the low-level policy completes most 1-subgoal tasks and none with 5 subgoals. See the data ↓ 2 13.1% → 25.0% Checking each subgoal helps Giving the same policy one subgoal at a time, and checking each one with a VLM, nearly doubles its success. See the data ↓ 3 63.3% Tool-selection agents lead On the full benchmark, GPT-Policy and PSP-Agent (42.8%) are the only systems above the privileged verifier (29.8%), which reads the true state from the simulator. See the data ↓ 4 $6.49 vs $0.33 The leader costs much more per success Cost per successful episode of GPT-Policy with GPT-6 Astra and of PSP-Agent with Gemini 3.7 Flash, 20 times apart. See the data ↓ 5 34.8% Long tasks are far from solved The best system completes about one in three tasks with 8 to 10 subgoals. See the data ↓ 6 Same order Simulation predicts the real world On a real WidowX 250 and Franka FR3, the systems rank in the same order as in simulation. See the data ↓

Benchmark

145 tasks in three simulators, each with 1 to 10 subgoals.

145tasks
3simulators
8task categories
1–10subgoals per task
10systems evaluated
2real robots
GPT-Policy · 7 subgoals2× speed

RoboLab-Long

44 tasks · Isaac Sim · Franka FR3

low-level policy: π0.5 (DROID)

GPT-Policy · 10 subgoals2× speed

SimplerEnv-Long

54 tasks · SAPIEN · WidowX 250

low-level policy: Bridge policy

GPT-Policy · 8 subgoals2× speed

RobotArena-∞-Long

47 tasks · Genesis · WidowX 250

low-level policy: Bridge policy

Eight task categories

Single destination

Single destination 44 tasks

Every object goes to the same destination.

“put both carrots, the spoon and the eggplant on the plate”

Transfer

Transfer 8 tasks

Every object starts on one destination and is moved to another.

“move the carrot and the spoon from the plate to the blue towel”

Split destination

Split destination 25 tasks

Different objects go to different destinations.

“put the lemon on the plate and put the orange in the bin”

Relay

Relay 20 tasks

Objects pass through an intermediate destination.

“first put the carrot on the plate, then put the carrot on the blue towel”

Revisit

Revisit 13 tasks

Objects return to a destination they have already left.

“first put the spoon on the cloth, then put the spoon in the pot, then put the spoon on the cloth”

Exclusion

Exclusion 13 tasks

Some objects must be left where they are.

“put everything except the eggplant on the plate”

Ordered

Ordered 14 tasks

The placements must occur in a stated order.

“first put the banana in the bin, then put the orange in the bin”

Duplicates

Duplicates 8 tasks

Several identical objects, so the instruction refers to them by count, not by name.

“put both corn cobs on the cloth”

The systems

Same task, ten ways to solve it. Pick one to see how it works.

Robotics-only model
Planner + robotics model
AI agent
Reference
Sees
camerainstruction
Thinks withOne robotics model
Sends to the robotArm movements, many times per second

A robotics model on its own, trained only on robot data, gets the whole instruction and the camera picture, and moves the arm directly. No plan, no checks.

Side by side

How each system plans, when it moves on, how it checks a step and what it does after a failure.

SystemVLMExecutionPlanTriggerVerificationRecovery
End-to-end
Low-level policy–Learned policyNoneNoneNoneNone
Orchestrated low-level policy
Periodic trigger–Learned policyText subgoalsPeriodicNoneAdvance
Proprioceptive trigger–Learned policyText subgoalsProprioceptiveNoneAdvance
Proprioceptive verifierGemini 3.7 FlashLearned policyText subgoalsProprioceptiveVisual (VLM)Retry
VoLoGemini 3.7 FlashLearned policyText subgoalsPeriodicVisual (VLM)Replan
Tool selection
CaP-XGemini 3.7 FlashMotion primitivesPython programProgram completionVisual (VLM)Replan
TiPToPGemini 3.7 FlashSymbolic TAMPGoal predicatesNoneNoneNone
PSP-AgentGemini 3.7 FlashMotion primitivesText subgoalsPrimitive completionGeometric stateRetry, replan
GPT-PolicyGPT-6 AstraMotion primitivesNext tool callPrimitive completionVisual (VLM)Replan
Privileged reference
Privileged verifier–Learned policyGround-truth subgoalsPrivilegedSimulator stateRetry

Results

25.0% vs 29.8%

A VLM that only looks at the camera gets the robotics model most of the way to perfect step checks. What is left is the robotics model itself.

30.7% vs 9.5%

On 5-subgoal tasks, PSP-Agent's ready-made skills beat even perfect checks on top of a robotics model. Today the learned policy, not the planner, is the bottleneck.

0.6% → 34.1%

Tasks where some objects must stay put ("everything except the eggplant"): the robotics-only model moves the wrong objects. One step at a time, it gets them right.

SystemRoboLab-LongSimplerEnv-LongRobotArena-∞-LongMeanSubgoals done
End-to-end
Low-level policy19.6±4.58.7±1.710.9±3.613.1±1.831.8
Orchestrated low-level policy
Periodic trigger25.6±4.817.4±2.215.2±4.019.4±2.141.8
Proprioceptive trigger28.1±4.921.5±2.316.2±4.121.9±2.144.5
Proprioceptive verifierGemini 3.7 Flash25.6±4.825.2±2.424.2±4.625.0±2.246.9
VoLoGemini 3.7 Flash29.0±4.925.5±2.421.3±4.425.3±2.247.4
Tool selection
CaP-XGemini 3.7 Flash20.7±4.528.7±2.516.2±4.121.9±2.139.3
TiPToPGemini 3.7 Flash17.3±4.36.7±1.511.2±3.611.7±1.814.0
PSP-AgentGemini 3.7 Flash51.4±5.237.1±2.739.9±5.042.8±2.663.3
GPT-PolicyGPT-6 Astra62.8±5.260.3±2.766.8±4.963.3±2.581.8
Privileged reference
Privileged verifier34.4±5.129.5±2.525.5±4.629.8±2.449.5
Scored on LADDER-Lite only
Gemini 4 Argonvia GPT-Policy, LADDER-Lite50.0±17.377.1±11.940.6±17.055.9±9.074.2
Claude Opus 5.5via GPT-Policy, LADDER-Lite59.4±17.062.5±13.725.0±15.049.0±8.865.6

Task success (%) with 95% Wilson intervals. Mean: the three suites weighed equally, as on the leaderboard. Subgoals done: the share of all subgoals completed. Bold: best system, the privileged reference aside. Systems scored on LADDER-Lite only are on its 10 tasks, so their numbers are not on the same tasks as the rows above.

Task success by number of subgoals

0%25%50%75%100%12–34–56–78–10subgoals in the task

Task success (%), the three simulators weighed equally. Subgoal counts are grouped so every point covers many tasks. Hover the chart to read the values. Click a name to show or hide it.

By task category

SystemSingle destinationTransferSplit destinationRelayRevisitExclusionOrderedDuplicates
End-to-end
Low-level policy33.530.14.91.10.00.623.858.4
Orchestrated low-level policy
Periodic trigger40.825.711.78.72.835.927.147.2
Proprioceptive trigger38.327.512.312.12.839.231.059.0
Proprioceptive verifier42.926.017.617.66.934.139.151.7
VoLo43.932.521.815.17.332.542.857.3
Tool selection
CaP-X31.743.918.721.37.341.831.542.7
TiPToP21.131.817.80.60.018.114.631.2
PSP-Agent52.469.065.427.84.256.562.071.9
GPT-Policy72.270.459.270.442.465.673.876.4
Privileged reference
Privileged verifier45.432.222.629.69.445.743.353.1

Task success (%) per task category, the three suites weighed equally.

Cost

$6.49 vs $0.33

Cost per successful episode of GPT-Policy with GPT-6 Astra and of PSP-Agent with Gemini 3.7 Flash. One full run of GPT-Policy on LADDER costs up to $8,134.

36×

The leader pays 36 times more per finished task than the Proprioceptive verifier, for 2.5 times its success. A small VLM that only checks steps is the cheapest way to add reliability.

42.1% → 49.5%

Subgoals done by the Proprioceptive verifier at half the time limit, and at the end. Failures are not a lack of time: more time would add little.

SystemVLMVLM calls per episodeCost per episodeCost per successTask success
GPT-Policy LADDER-LiteClaude Opus 5.563.1$3.575$7.0249.0%
GPT-PolicyGPT-6 Astra54.2$4.019$6.4963.3%
GPT-Policy LADDER-LiteGemini 4 Argon38.0$0.784$1.3355.9%
VoLoGemini 3.7 Flash23.5$0.166$0.6625.3%
PSP-AgentGemini 3.7 Flash23.3$0.133$0.3342.8%
Proprioceptive verifierGemini 3.7 Flash10.4$0.045$0.1825.0%
CaP-XGemini 3.7 Flash5.2$0.042$0.1721.9%
TiPToPGemini 3.7 Flash1.0$0.007$0.0711.7%

API cost at list prices on 25 September 2026, from each provider's token counts. It covers only the hosted VLM, not the GPU time of the low-level policy or the simulators. The usage reports do not say how much input was cached, so GPT-Policy's cost is an upper bound. Calls: mean of the three suites. Rows marked LADDER-Lite are scored on its 10 tasks only. For Gemini 4 Argon the gateway omits the output-token count on most calls; those are estimated from the calls of the same run that report it, about 5% of its cost.

Why subgoals fail

7.1% vs 43.4%

Subgoals lost to an orchestrator decision: Proprioceptive verifier against Periodic trigger, which never checks. Once subgoals are checked, most failures come from the policy.

31.0%

Subgoals never started in 10-subgoal tasks, the largest cause on long tasks. Systems that retry run out of time on earlier subgoals.

50.9%

Subgoals for which TiPToP finds no plan. Its planner fails more often as tasks get longer.

(a) Per system, all tasks

0%25%50%75%100%GPT-PolicyGPT-Policy, Orchestrator decision: 0.0%GPT-Policy, Grasping: 1.5%GPT-Policy, Carrying: 1.7%GPT-Policy, Placing: 1.5%GPT-Policy, Undone later: 3.7%GPT-Policy, Never started: 9.9%Grasping: 1.5% · Carrying: 1.7% · Placing: 1.5% · Undone later: 3.7% · Never started: 9.9%18PSP-AgentPSP-Agent, Orchestrator decision: 23.9%PSP-Agent, Grasping: 2.0%PSP-Agent, Carrying: 2.0%PSP-Agent, Placing: 1.5%PSP-Agent, Undone later: 3.0%PSP-Agent, Never started: 4.2%Orchestrator decision: 23.9% · Grasping: 2.0% · Carrying: 2.0% · Placing: 1.5% · Undone later: 3.0% · Never started: 4.2%37Privileged verifierPrivileged verifier, Orchestrator decision: 4.1%Privileged verifier, Grasping: 4.5%Privileged verifier, Carrying: 2.5%Privileged verifier, Placing: 5.5%Privileged verifier, Undone later: 6.5%Privileged verifier, Never started: 27.3%Orchestrator decision: 4.1% · Grasping: 4.5% · Carrying: 2.5% · Placing: 5.5% · Undone later: 6.5% · Never started: 27.3%51VoLoVoLo, Orchestrator decision: 13.8%VoLo, Grasping: 5.1%VoLo, Carrying: 2.4%VoLo, Placing: 4.2%VoLo, Undone later: 5.9%VoLo, Never started: 21.3%Orchestrator decision: 13.8% · Grasping: 5.1% · Carrying: 2.4% · Placing: 4.2% · Undone later: 5.9% · Never started: 21.3%53Proprioceptive verifierProprioceptive verifier, Orchestrator decision: 7.1%Proprioceptive verifier, Grasping: 4.6%Proprioceptive verifier, Carrying: 2.8%Proprioceptive verifier, Placing: 6.4%Proprioceptive verifier, Undone later: 6.4%Proprioceptive verifier, Never started: 25.7%Orchestrator decision: 7.1% · Grasping: 4.6% · Carrying: 2.8% · Placing: 6.4% · Undone later: 6.4% · Never started: 25.7%53Proprioceptive triggerProprioceptive trigger, Orchestrator decision: 33.0%Proprioceptive trigger, Grasping: 2.0%Proprioceptive trigger, Carrying: 1.2%Proprioceptive trigger, Placing: 1.8%Proprioceptive trigger, Undone later: 5.5%Proprioceptive trigger, Never started: 11.9%Orchestrator decision: 33.0% · Grasping: 2.0% · Carrying: 1.2% · Placing: 1.8% · Undone later: 5.5% · Never started: 11.9%55Periodic triggerPeriodic trigger, Orchestrator decision: 43.5%Periodic trigger, Grasping: 1.7%Periodic trigger, Carrying: 1.1%Periodic trigger, Placing: 1.4%Periodic trigger, Undone later: 3.9%Periodic trigger, Never started: 6.5%Orchestrator decision: 43.5% · Grasping: 1.7% · Carrying: 1.1% · Placing: 1.4% · Undone later: 3.9% · Never started: 6.5%58CaP-XCaP-X, Orchestrator decision: 0.0%CaP-X, Grasping: 6.3%CaP-X, Carrying: 3.9%CaP-X, Placing: 5.6%CaP-X, Undone later: 8.5%CaP-X, Never started: 36.4%Grasping: 6.3% · Carrying: 3.9% · Placing: 5.6% · Undone later: 8.5% · Never started: 36.4%61Low-level policyLow-level policy, Orchestrator decision: 0.0%Low-level policy, Grasping: 9.1%Low-level policy, Carrying: 3.9%Low-level policy, Placing: 12.6%Low-level policy, Undone later: 4.4%Low-level policy, Never started: 38.1%Grasping: 9.1% · Carrying: 3.9% · Placing: 12.6% · Undone later: 4.4% · Never started: 38.1%68TiPToPTiPToP, Orchestrator decision: 50.9%TiPToP, Grasping: 6.7%TiPToP, Carrying: 2.8%TiPToP, Placing: 4.5%TiPToP, Undone later: 1.8%TiPToP, Never started: 19.2%Orchestrator decision: 50.9% · Grasping: 6.7% · Carrying: 2.8% · Placing: 4.5% · Undone later: 1.8% · Never started: 19.2%86subgoals lost (%)0%25%50%75%100%GPT-PolicyGPT-Policy, Orchestrator decision: 0.0%GPT-Policy, Grasping: 1.5%GPT-Policy, Carrying: 1.7%GPT-Policy, Placing: 1.5%GPT-Policy, Undone later: 3.7%GPT-Policy, Never started: 9.9%Grasping: 1.5% · Carrying: 1.7% · Placing: 1.5% · Undone later: 3.7% · Never started: 9.9%18PSP-AgentPSP-Agent, Orchestrator decision: 23.9%PSP-Agent, Grasping: 2.0%PSP-Agent, Carrying: 2.0%PSP-Agent, Placing: 1.5%PSP-Agent, Undone later: 3.0%PSP-Agent, Never started: 4.2%Orchestrator decision: 23.9% · Grasping: 2.0% · Carrying: 2.0% · Placing: 1.5% · Undone later: 3.0% · Never started: 4.2%37Privileged verifierPrivileged verifier, Orchestrator decision: 4.1%Privileged verifier, Grasping: 4.5%Privileged verifier, Carrying: 2.5%Privileged verifier, Placing: 5.5%Privileged verifier, Undone later: 6.5%Privileged verifier, Never started: 27.3%Orchestrator decision: 4.1% · Grasping: 4.5% · Carrying: 2.5% · Placing: 5.5% · Undone later: 6.5% · Never started: 27.3%51VoLoVoLo, Orchestrator decision: 13.8%VoLo, Grasping: 5.1%VoLo, Carrying: 2.4%VoLo, Placing: 4.2%VoLo, Undone later: 5.9%VoLo, Never started: 21.3%Orchestrator decision: 13.8% · Grasping: 5.1% · Carrying: 2.4% · Placing: 4.2% · Undone later: 5.9% · Never started: 21.3%53Proprioceptive verifierProprioceptive verifier, Orchestrator decision: 7.1%Proprioceptive verifier, Grasping: 4.6%Proprioceptive verifier, Carrying: 2.8%Proprioceptive verifier, Placing: 6.4%Proprioceptive verifier, Undone later: 6.4%Proprioceptive verifier, Never started: 25.7%Orchestrator decision: 7.1% · Grasping: 4.6% · Carrying: 2.8% · Placing: 6.4% · Undone later: 6.4% · Never started: 25.7%53Proprioceptive triggerProprioceptive trigger, Orchestrator decision: 33.0%Proprioceptive trigger, Grasping: 2.0%Proprioceptive trigger, Carrying: 1.2%Proprioceptive trigger, Placing: 1.8%Proprioceptive trigger, Undone later: 5.5%Proprioceptive trigger, Never started: 11.9%Orchestrator decision: 33.0% · Grasping: 2.0% · Carrying: 1.2% · Placing: 1.8% · Undone later: 5.5% · Never started: 11.9%55Periodic triggerPeriodic trigger, Orchestrator decision: 43.5%Periodic trigger, Grasping: 1.7%Periodic trigger, Carrying: 1.1%Periodic trigger, Placing: 1.4%Periodic trigger, Undone later: 3.9%Periodic trigger, Never started: 6.5%Orchestrator decision: 43.5% · Grasping: 1.7% · Carrying: 1.1% · Placing: 1.4% · Undone later: 3.9% · Never started: 6.5%58CaP-XCaP-X, Orchestrator decision: 0.0%CaP-X, Grasping: 6.3%CaP-X, Carrying: 3.9%CaP-X, Placing: 5.6%CaP-X, Undone later: 8.5%CaP-X, Never started: 36.4%Grasping: 6.3% · Carrying: 3.9% · Placing: 5.6% · Undone later: 8.5% · Never started: 36.4%61Low-level policyLow-level policy, Orchestrator decision: 0.0%Low-level policy, Grasping: 9.1%Low-level policy, Carrying: 3.9%Low-level policy, Placing: 12.6%Low-level policy, Undone later: 4.4%Low-level policy, Never started: 38.1%Grasping: 9.1% · Carrying: 3.9% · Placing: 12.6% · Undone later: 4.4% · Never started: 38.1%68TiPToPTiPToP, Orchestrator decision: 50.9%TiPToP, Grasping: 6.7%TiPToP, Carrying: 2.8%TiPToP, Placing: 4.5%TiPToP, Undone later: 1.8%TiPToP, Never started: 19.2%Orchestrator decision: 50.9% · Grasping: 6.7% · Carrying: 2.8% · Placing: 4.5% · Undone later: 1.8% · Never started: 19.2%86subgoals lost (%)

(b) Per task length, all systems

0%10%20%30%40%50%60%70%1 subgoals, Orchestrator decision: 1.4%1 subgoals, Grasping: 13.5%1 subgoals, Carrying: 2.7%1 subgoals, Placing: 0.9%1 subgoals, Undone later: 0.0%1 subgoals, Never started: 1.8%12 subgoals, Orchestrator decision: 6.4%2 subgoals, Grasping: 6.1%2 subgoals, Carrying: 3.3%2 subgoals, Placing: 3.5%2 subgoals, Undone later: 3.8%2 subgoals, Never started: 8.7%23 subgoals, Orchestrator decision: 10.6%3 subgoals, Grasping: 5.6%3 subgoals, Carrying: 2.6%3 subgoals, Placing: 2.6%3 subgoals, Undone later: 5.5%3 subgoals, Never started: 13.9%34 subgoals, Orchestrator decision: 14.2%4 subgoals, Grasping: 5.3%4 subgoals, Carrying: 2.8%4 subgoals, Placing: 4.2%4 subgoals, Undone later: 5.5%4 subgoals, Never started: 16.2%45 subgoals, Orchestrator decision: 17.7%5 subgoals, Grasping: 5.0%5 subgoals, Carrying: 2.4%5 subgoals, Placing: 4.8%5 subgoals, Undone later: 8.8%5 subgoals, Never started: 15.6%56 subgoals, Orchestrator decision: 19.8%6 subgoals, Grasping: 3.4%6 subgoals, Carrying: 2.3%6 subgoals, Placing: 4.5%6 subgoals, Undone later: 6.4%6 subgoals, Never started: 20.8%67 subgoals, Orchestrator decision: 20.2%7 subgoals, Grasping: 3.2%7 subgoals, Carrying: 2.6%7 subgoals, Placing: 6.8%7 subgoals, Undone later: 4.4%7 subgoals, Never started: 24.2%78 subgoals, Orchestrator decision: 22.3%8 subgoals, Grasping: 2.7%8 subgoals, Carrying: 2.3%8 subgoals, Placing: 5.5%8 subgoals, Undone later: 3.3%8 subgoals, Never started: 28.4%89 subgoals, Orchestrator decision: 22.2%9 subgoals, Grasping: 2.2%9 subgoals, Carrying: 2.0%9 subgoals, Placing: 5.5%9 subgoals, Undone later: 2.3%9 subgoals, Never started: 30.6%910 subgoals, Orchestrator decision: 22.8%10 subgoals, Grasping: 2.3%10 subgoals, Carrying: 1.5%10 subgoals, Placing: 3.8%10 subgoals, Undone later: 2.4%10 subgoals, Never started: 31.0%10subgoals in the task0%10%20%30%40%50%60%70%1 subgoals, Orchestrator decision: 1.4%1 subgoals, Grasping: 13.5%1 subgoals, Carrying: 2.7%1 subgoals, Placing: 0.9%1 subgoals, Undone later: 0.0%1 subgoals, Never started: 1.8%12 subgoals, Orchestrator decision: 6.4%2 subgoals, Grasping: 6.1%2 subgoals, Carrying: 3.3%2 subgoals, Placing: 3.5%2 subgoals, Undone later: 3.8%2 subgoals, Never started: 8.7%23 subgoals, Orchestrator decision: 10.6%3 subgoals, Grasping: 5.6%3 subgoals, Carrying: 2.6%3 subgoals, Placing: 2.6%3 subgoals, Undone later: 5.5%3 subgoals, Never started: 13.9%34 subgoals, Orchestrator decision: 14.2%4 subgoals, Grasping: 5.3%4 subgoals, Carrying: 2.8%4 subgoals, Placing: 4.2%4 subgoals, Undone later: 5.5%4 subgoals, Never started: 16.2%45 subgoals, Orchestrator decision: 17.7%5 subgoals, Grasping: 5.0%5 subgoals, Carrying: 2.4%5 subgoals, Placing: 4.8%5 subgoals, Undone later: 8.8%5 subgoals, Never started: 15.6%56 subgoals, Orchestrator decision: 19.8%6 subgoals, Grasping: 3.4%6 subgoals, Carrying: 2.3%6 subgoals, Placing: 4.5%6 subgoals, Undone later: 6.4%6 subgoals, Never started: 20.8%67 subgoals, Orchestrator decision: 20.2%7 subgoals, Grasping: 3.2%7 subgoals, Carrying: 2.6%7 subgoals, Placing: 6.8%7 subgoals, Undone later: 4.4%7 subgoals, Never started: 24.2%78 subgoals, Orchestrator decision: 22.3%8 subgoals, Grasping: 2.7%8 subgoals, Carrying: 2.3%8 subgoals, Placing: 5.5%8 subgoals, Undone later: 3.3%8 subgoals, Never started: 28.4%89 subgoals, Orchestrator decision: 22.2%9 subgoals, Grasping: 2.2%9 subgoals, Carrying: 2.0%9 subgoals, Placing: 5.5%9 subgoals, Undone later: 2.3%9 subgoals, Never started: 30.6%910 subgoals, Orchestrator decision: 22.8%10 subgoals, Grasping: 2.3%10 subgoals, Carrying: 1.5%10 subgoals, Placing: 3.8%10 subgoals, Undone later: 2.4%10 subgoals, Never started: 31.0%10subgoals in the task
Orchestrator decisionGraspingCarryingPlacingUndone laterNever started

Share of subgoals not completed at the end of the episode, by the stage where they were lost, averaged over the three suites. Orchestrator decision: a planner or verifier mistake made the loss final (moved on too early, no plan found). The other stages describe what happened to the object. Hover a bar for its value.

Which VLM plans best

80.8%

Mean score of Gemini 3.7 Flash, the highest of the ten VLMs. Other models lead single columns. All orchestrators in the results use Gemini 3.7 Flash.

87–93%

Exact plans from every model except the smallest. Writing the plan is easy. Seeing from a picture whether a step is really done is where models differ: the best tracks every step only 70.8% of the time.

ModelPlan
exactly right		Sees a step done
when it is		Sees a step not done
when it is not		Tracks every step
all correct		Picks the next step
valid		Tokens
per question
Qwen3-VL-2B75.2 ±7.668.2 ±7.034.1 ±5.99.3 ±4.058.9 ±5.3–
Qwen3-VL-8B91.0 ±5.869.0 ±6.879.9 ±5.747.9 ±5.572.3 ±4.4–
Qwen3.5-9B86.9 ±6.544.4 ±6.396.9 ±2.511.7 ±3.763.2 ±4.85,173
Qwen3.8-27B91.0 ±5.876.2 ±6.384.8 ±4.358.2 ±4.879.3 ±4.11,904
Gemini 3.1 Pro89.7 ±6.054.8 ±6.696.0 ±2.968.7 ±4.375.2 ±4.5583
Gemini 3.7 Flash93.1 ±5.459.4 ±6.696.6 ±2.670.8 ±4.081.4 ±4.2412
Gemini Robotics-ER 290.3 ±5.954.0 ±6.597.8 ±2.369.6 ±4.276.6 ±4.1510
GPT-5.6 Sol93.1 ±5.472.8 ±6.386.4 ±4.356.4 ±4.782.0 ±4.3282
GPT-6 Astra91.7 ±5.771.1 ±6.286.4 ±4.453.2 ±5.185.4 ±4.0203
Claude Opus 590.3 ±5.968.6 ±6.687.9 ±4.161.0 ±4.779.8 ±4.4400

Each VLM answers the questions an orchestrator asks, from still pictures and without running the robot: 145 scenes and 562 labelled frames. Scores in % with 95% intervals.

Real robots

Same order

On both robots: robotics-only model, then Proprioceptive verifier, then PSP-Agent, as in simulation.

1.2% → 31.0% → 76.2%

Tasks with 2 to 5 subgoals finished on the real WidowX: the robotics-only model, the Proprioceptive verifier, and PSP-Agent. Breaking a task into checked steps helps on hardware too.

0%25%50%75%100%Robotics-only model, simulation: 19.6%Proprioceptive verifier, simulation: 25.6%Gemini 3.7 Flash via PSP-Agent, simulation: 51.4%SimulationRobotics-only model, real robot: 5.6%Proprioceptive verifier, real robot: 58.3%Gemini 3.7 Flash via PSP-Agent, real robot: 75.0%RealRobotics-only model, simulation: 9.8%Proprioceptive verifier, simulation: 24.7%Gemini 3.7 Flash via PSP-Agent, simulation: 38.5%SimulationRobotics-only model, real robot: 15.7%Proprioceptive verifier, real robot: 40.7%Gemini 3.7 Flash via PSP-Agent, real robot: 81.5%RealFrankaWidowX0%25%50%75%100%Robotics-only model, simulation: 19.6%Proprioceptive verifier, simulation: 25.6%Gemini 3.7 Flash via PSP-Agent, simulation: 51.4%SimulationRobotics-only model, real robot: 5.6%Proprioceptive verifier, real robot: 58.3%Gemini 3.7 Flash via PSP-Agent, real robot: 75.0%RealRobotics-only model, simulation: 9.8%Proprioceptive verifier, simulation: 24.7%Gemini 3.7 Flash via PSP-Agent, simulation: 38.5%SimulationRobotics-only model, real robot: 15.7%Proprioceptive verifier, real robot: 40.7%Gemini 3.7 Flash via PSP-Agent, real robot: 81.5%RealFrankaWidowX
Robotics-only modelProprioceptive verifierGemini 3.7 Flash via PSP-Agent
Task categorySubgoalsLow-level policyProp. verifierPSP-Agent
Franka
Single destination50.08.325.0
Exclusion20.083.3100.0
Ordered316.783.3100.0
WidowX
Single destination166.775.0100.0
Single destination2–30.033.387.5
Split destination2–30.029.283.3
Exclusion20.058.391.7
Ordered28.333.341.7
Duplicates20.00.058.3

Real-robot videos

Each video is typical for its system on that task: a success if it succeeds in at least half of its attempts.

2× speed

“Put the corn, the carrot, the eggplant, the blue cup and the banana in the bowl”

5 subgoalsfailure · 3 of 5 done

2× speed

“First put the blue cup in the bowl, then put the carrot in the bowl, then put the eggplant in the bowl”

3 subgoals

2× speed

“Put everything except the eggplant in the bowl”

2 subgoals

Videos

The label on each video gives its playback speed.

44 tasks · Isaac Sim · Franka FR3 · low-level policy: π0.5 (DROID)

Highlights: long tasks solved without a single mistake

2× speed

CaP-X

“first put the orange in the bin, then put the pumpkin in the bin, then put the lychee in the bin, then put the lemon in the bin, then put the lime in the bin, then put the banana in the bin, then put the onion in the bin, then put the avocado in the bin”

8 subgoals

2× speed

GPT-Policy

“first put the lemon on the plate, then put the lemon in the bin, then put the lime on the plate, then put the lime in the bin, then put the banana on the plate, then put the banana in the bin”

6 subgoals

2× speed

GPT-Policy

“put the avocado, the banana and the lemon on the plate and put the orange, the lime and the onion in the bin”

6 subgoals

One scene, growing horizon

2× speed

“put the banana in the bin”

1 subgoal

2× speed

“put the banana, the orange, the lemon, the lime, the avocado and the onion in the bin”

6 subgoals

2× speed

“put the banana, both oranges, the lemon, the lime, the avocado, the onion, the lychee, the pumpkin and the pomegranate in the bin”

10 subgoals

Task categories

2× speed

Proprioceptive trigger

“move the orange, the lemon and the lime from the plate to the bin”

3 subgoalsTransfer

2× speed

Privileged verifier

“put everything except the red block and the blue block in the bin”

4 subgoalsExclusion

Add your system

Policies, orchestrators and agents can all be evaluated on LADDER.

A policy

A VLA or any other end-to-end model. It runs as a small server and returns the next arm pose.

Example →

An orchestrator

A system that tells the low-level policy of each suite what to do next, in words. One Python class.

Example →

An agent

A system that moves the arm through tools, using depth and the camera calibration. One Python class.

Example →
  1. Start from the example for your kind of system and replace the model with yours.
  2. Run all tasks of the three suites, or the 10 tasks of LADDER-Lite for a quick first result.
  3. Open a pull request with your episode logs. We recompute every score from the logs.

Integration guide How to submit Request a model

Code and data

Code The three benchmark suites, every orchestrator and baseline adapter, and the evaluation harness.
Bridge policy The WidowX low-level policy of SimplerEnv-Long and RobotArena-∞-Long (Qwen3-VL backbone, BridgeData v2).
Evaluation episodes Every episode of the evaluation, with videos and step logs, including the real-robot runs.
Offline VLM benchmark 145 scenes and 562 labelled frames to score a VLM as an orchestrator.
Paper Code Hugging Face

BibTeX

@article{ladder2026,
  title   = {{LADDER}: A Long-Horizon Robot Manipulation Benchmark},
  author  = {Petkov, Petko and Albanese, Giuliano and Nikolov, Nikolay and Van Gool, Luc and Tremblay, Jonathan and Paudel, Danda Pani},
  journal = {arXiv preprint arXiv:2610.18053},
  year    = {2026},
  url     = {https://arxiv.org/abs/2610.18053}
}