Skip to content
Developer312

Mind2Act: Robots Reason at 52.7% but Collapse to 12% Under Composition

A new diagnostic benchmark across 30 real-world manipulation tasks and 180 configurations finds robots at 52.7% on reasoning but 12.0% on closed-loop composition, with end-to-end policies capping at 3.6%.

Published Report an error

A new diagnostic benchmark called Mind2Act, published at Hugging Face Papers (2611.26444), ran robots through 30 real-world manipulation tasks across 180 configurations, and the headline finding is a collapse. The strongest agent system scored 52.7% on the reasoning component (Mind) and 39.0% on the execution component (Act), but only 12.0% on the closed-loop Mind2Act composite, where an agent has to chain perception, planning and manipulation end to end. End-to-end vision-language-action (VLA) and world-action-model (WAM) policies did worse, capping at 3.6%.

The scale of the evaluation matters as much as the scores. Across 56,700 episodes, 8 of the 30 tasks were never completed by any system. That is not a distribution with a weak tail; that is a set of tasks where every architecture evaluated hit zero. If you are signing a robotics contract based on demo videos, these are the numbers to demand first.

Key Takeaways: What Matters Most?

  • The Mind2Act diagnostic covers 30 real-world manipulation tasks across 180 configurations.
  • The strongest agent system scored 52.7% on reasoning (Mind) and 39.0% on execution (Act), but only 12.0% closed-loop.
  • End-to-end VLA and WAM policies cap at 3.6% on the same composed tasks.
  • 8 of the tasks were never completed across 56,700 episodes run.
  • The gap from 52.7% to 12.0% is the composition penalty, the wall between robotics demos and deployable robots.

Takeaways like these, every weekday morning. Get the briefing

Composition is the wall

The single most important number in the benchmark is the distance between the parts and the whole. A system that reasons at 52.7% and executes at 39.0% should, if components composed cleanly, land somewhere between those figures on the chained task. Instead it lands at 12.0%. The act of coupling reasoning to execution to perception in a closed loop destroys most of the capability the components demonstrate in isolation.

This is a familiar failure shape to anyone who has shipped multi-stage systems. Each stage has an error rate; the chained system inherits all of them. But the benchmark's contribution is quantifying how severe the penalty is in physical tasks specifically, where recovery is expensive, the world does not reset between attempts, and an execution slip can invalidate everything the reasoning stage concluded. The 3.6% ceiling on end-to-end VLA/WAM policies says the alternative architecture, learning the whole loop as one policy, currently absorbs the composition penalty rather than avoiding it.

The 8 never-completed tasks sharpen the point. Whatever capability separates a task that some system occasionally finishes from a task that none does, it is not raw model scale or architecture choice, because 56,700 episodes across 180 configurations gave every approach room to succeed. The wall is structural: certain compositions of skill are simply not reachable by today's systems.

What the 12% number means for buyers

Robotics procurement has a demo-to-deployment gap that vendors price optimistically and buyers discover expensively. Mind2Act gives the gap a number. A robot platform whose reasoning module benchmarks at 52.7% sounds like a usable system; the same platform running composed tasks at 12.0% is a research prototype in a commercial costume. The benchmark's Mind/Act/closed-loop split is exactly the right instrument for due diligence, because vendors quote component scores and integration is where the money goes.

The comparison between architectures is also procurement-relevant. Modular agent systems (12.0% closed-loop) degrade more gracefully under composition than end-to-end VLA/WAM policies (3.6%). Neither number is deployable, but the fourfold difference tells you which approach fails more softly, and soft failure, where the system stalls or asks for help, is what you want in a warehouse. Hard failure, where an end-to-end policy confidently executes the wrong thing, is what breaks merchandise.

The week's broader context makes this benchmark timely rather than academic. On the model side, labs are claiming autonomous research competence, with Anthropic reporting unreviewed black hole physics results from Fable 5.2. On the hardware side, NVIDIA just made 128GB local inference boxes a line item. Both trends feed a robotics narrative where capability is assumed to be arriving. Mind2Act's data says the binding constraint is neither the model nor the silicon: it is composition, and composition does not improve on anyone's announced roadmap.

What it costs and where it breaks

The cost side of composition is multiplicative risk. In a chained task, a system with three stages each succeeding 50% of the time succeeds about 12.5% of the time, assuming independence, which is roughly the shape Mind2Act found. For a buyer, that means a composed robot workflow with five stages is a workflow that almost never completes, and the fix is human checkpoints, which erase the labor savings that justified the robot. The honest deployment math today is: robots for single tasks, humans for chains.

For builders, the benchmark identifies where engineering effort pays. Since reasoning (52.7%) is already the strongest component, the effort belongs in execution and handoffs: better state estimation at the perception-to-planning boundary, explicit recovery behaviors when an action slips, and task decompositions that minimize the number of stages that must succeed in one run. The 8 never-completed tasks are the sharpest research targets, because whatever capability they demand is absent from every current approach.

The benchmark's own limits should be stated: 30 tasks and 180 configurations is a diagnostic instrument, not a census of manipulation work, and closed-loop scores may shift with better tooling even at fixed model capability. But a diagnostic that separates components from composition is worth more to a buyer than a leaderboard that conflates them, and this one's spread is stark enough to plan around.

What To Do About It

  • Demand composite scores: when a vendor quotes a robotics benchmark, ask specifically for closed-loop or composed-task numbers, not component scores; the gap between 52.7% and 12.0% is where budgets die.
  • Scope to single tasks: design your first deployments around one capability at a time; every additional chained stage multiplies failure risk.
  • Favor recoverable failure: between a 12.0% modular system and a 3.6% end-to-end policy, pick the one that stalls instead of executing confidently wrong actions.
  • Pilot with a stop metric: define what fraction of episodes your operation needs to complete, and hold the vendor to it over a fixed episode budget like the benchmark's 56,700.
  • Budget human checkpoints: at current composition rates, plan human intervention into any chained workflow and price it before signing, not after.

The Bottom Line

Mind2Act quantifies the robotics industry's most inconvenient truth: components are getting good, systems are not. A 52.7% reasoner collapsing to 12.0% under composition, with end-to-end policies at 3.6% and 8 tasks never solved across 56,700 episodes, means the obstacle between the demo and the contract is structural, not incremental. Buyers should treat any robotics pitch that does not volunteer closed-loop composed-task numbers as a red flag by default, and builders should spend their effort exactly where the benchmark says capability leaks out: the seams between the stages.

Sources

  1. [1]Hugging Face Papers — Mind2Act benchmark (opens in a new tab)
  2. [2]AI Weekly — aggregated AI news coverage (opens in a new tab)

Frequently Asked Questions

Get the next briefing

The lead story each weekday morning, with the key takeaways and a link to the full analysis.

One lead story each weekday morning: what happened, the key takeaways, and a link to the full analysis.

New posts daily, briefing weekday mornings. We use your email only to send the briefing. Unsubscribe anytime. Privacy policy.

Related Articles