NVIDIA explains how to evaluate general-purpose robot policies for the real world

RoboLab is NVIDIA's simulation platform for robot benchmarking. Robotics foundation models have made an impressive leap, the company believes. But the more powerful these models become, the more acute the problem of their objective assessment becomes.

This is one of the most difficult unsolved problems in robotics. In this post we will analyze the key problems and tell you how we solve them.

Why current benchmarks don't work

Testing in the real world is expensive, slow, and poorly reproducible. To truly appreciate a robot's performance, you need an adequate surrogate. Simulation is a natural environment for mass testing. But most existing benchmarks have common critical flaws.

Visual intersection of learning and assessment

The data and environment for policy learning and evaluation almost always come from a single visual source. If a model is retrained on simulated data and tested in the same simulation, high performance only indicates that it has memorized the scenario, not its ability to generalize.

The image quality in the simulation is not yet up to par with real images. Real2sim approaches attempt to solve this problem by recreating photorealistic environments from real images. But setting up each scene takes more than an hour - this is unsuitable for mass testing.

Figure 1. Existing simulation benchmarks suffer from overlap between visual and target domains, low realism, and high costs for creating scenes and tasks.

Benchmark saturation

Generating tasks is a tedious task. In most benchmarks, the set of tasks is fixed and rarely updated. This quickly leads to saturation: models score maximum on static tasks, and it becomes impossible to understand which one is truly better.

When all systems report success rates above 90% on the same test, the numbers become meaningless.

Figure 2. Almost every article on models reports results on this benchmark, but saturation makes it nearly impossible to draw meaningful conclusions.

Diagnostic gap

There is also a deeper problem. A simple binary outcome of success/failure does not explain why the robot made a mistake. Was he confused by the color of the object? Team formulation? Moved camera?

Did he complete the task effectively according to the specific instructions? Without answers to these questions, researchers have nothing to go on.

Figure 3. Example of an evaluation episode for the task “Put the orange measuring cup and the blue measuring cup outside the plate” with policy pi0.5.

Statistical significance

Any physics engine and policy is subject to randomness. A single success rate over N runs says almost nothing about how much you can trust the actual performance of a policy. If a policy works nine times out of ten, is it considered “90 percent”?

To figure it out, let's turn to the Clopper-Pearson method. This is an "exact" way of constructing a binomial confidence interval around the success rate.

Figure 4. Clopper-Pearson interval - a method for estimating the boundaries of the binomial success rate.

Here's an example: With an observed success rate of 90% and only 70 runs, the 95% confidence interval covers a whopping 15.4 percentage points (80.5% to 95.9%). At 1030 runs, the error narrows to ±2 percentage points (88.0% to 91.8%). Most published benchmarks do not run enough times to reach statistical significance.

Figure 5. Clopper-Pearson 95% interval for 90% success rate. Narrowing the confidence interval from 10 to 2 percentage points requires approximately 15 times more runs (from 70 to 1030). | Source: NVIDIA

NVIDIA presents RoboLab

NVIDIA created a simulation benchmark platform called RoboLab to solve these problems. RoboLab is built on three principles:

Provide robot-independent task assessments that provide meaningful metrics.

Enable rapid creation of new tasks to avoid benchmark saturation, with support for agent-based AI workflows.

Provide a comprehensive set of analysis tools that provide a holistic picture of how well a policy works, when it fails, and why.

Figure 6: Benchmarks must adapt to new opportunities as the field evolves. | Source: NVIDIA

Benchmarking robots in the era of agent-based AI

Figure 7. Three-step process for generating a scene, task, and environment in RoboLab. | Source: NVIDIA

RoboLab follows the real-world setup process: place objects, add language instructions, and run the policy. With a library of objects, users can simply place them on the stage and specify a language instruction - or even three! - for a task, and the whole process takes only a few minutes.

RoboLab also comes with agent skills that the coding agent can use to generate new tasks right in the user's workflow. This efficiency makes the benchmark resistant to future changes: as models improve, new tasks can be added and obsolete ones removed.

Bring your robot

Creating a generalist robot policy requires solving a long tail of specific problems, and no team has abundant data on every type of instantiation. One lab may have thousands of hours of work with a Franka arm, but almost no data on a humanoid robot, or vice versa.

A benchmark tied to one specific robot forces every user into the same data gap. RoboLab's objectives are independent of both the robot and politics. This means that the same set of tasks can be evaluated regardless of the type of robot or policy architecture.

Users are free to make their own design decisions; RoboLab simply compiles the same scenes and tasks for the robot they brought. In the future, the number of embodiments of robots will increase; It becomes less important which robot was used to generate the data - only that it solved the problem.

Challenges targeting specific abilities

A useful benchmark should isolate individual abilities, rather than simply measure whether the robot completed a task. We observed that general purpose manipulations rely on at least three distinct competencies:

Visual competence tests whether a policy is able to recognize and act on perceptual attributes such as color, size, and semantic category.

Procedural competence assesses action-based reasoning: folding objects, reorienting them, or inferring how to interact with a tool.

Relational competence examines spatial and linguistic logic, including conjunctions, counting, and relative positions.

By designing tasks that each target one or more specific abilities, we can ensure broad coverage of the entire skill space. In RoboLab-120, our initial benchmark of 120 hand-selected tasks, each task is labeled with multiple required abilities.

Competence What is tested Example of a task

VisualColor, size, semantic recognition"Put the little red cup in the container"

ProceduralFolding, reorienting, affordances“Put all the mugs upside down and put the red ones on the shelf.”

RelationalSpatial logic, counting, conjunctions"Take an orange or a lime and put it in a bowl"

Table 1. Competence is the ability of a policymaker to perform tasks in a particular area.

Evaluation of robotic policies

What metrics show that a robot's policy is good? The success rate alone says almost nothing about how the robot performed the task - only whether it crossed the finish line. A policy that grabs the desired object but drops it prematurely may be recorded as a failure.

To fix this, RoboLab uses three additional assessment tools:

Points for Task Steps: Partial points are awarded for completing sub-tasks within a multi-step instruction.

Path quality: Measuring the efficiency of movement through path length and SPARC - spectral arc length.

Execution Speed: Measures the speed of the end effector.

How NVIDIA detects robot failures

Realizing that the task has failed is half the battle. It is much more important to know why exactly this happened. RoboLab doesn't stop at standard metrics: it digs deeper, finding out at what stage and why everything went wrong.

The system automatically records failure events: grasping the wrong object, falling object, collision of the gripper. It pinpoints the exact place where a task went off the rails.

Figure 8. Three separate failures during the “Remove all plastic bottles in the container” task - despite the overall completion of the task. | Source: NVIDIA

To sort out such cases, RoboLab has a built-in panel. It shows events right during the sequence, so the user can instantly jump to the frame where the failure occurred. Diagnostics ceases to be an afterthought - it turns into a debugger of the robot's behavior.

Figure 9. Built-in RoboLab panel for viewing events in episodes. | Source: NVIDIA

How resilient is your policy to complexity?

There are no clean laboratory conditions in the real world. Instructions are worded differently, scenes are littered with junk, and tasks stretch out over dozens of steps. To understand whether a robot policy is truly reliable, you need to test it on three fronts: language complexity, scene complexity, and task length.

Complexity of language

A robot that only understands clearly formulated commands is useless in reality - people speak ambiguously and differently. Testing on a variety of formulations shows how much a policy depends on the exact phrase rather than on the actual understanding of the problem.

RoboLab allows you to specify several options for instructions and select them during execution. In its first benchmark, NVIDIA used three options: vague, standard and specific. It turned out that vague instructions consistently lead to failures.

Figure 10. A demonstration of how politics stumbles on vague commands. | Source: NVIDIA

Scene complexity

Real rooms are rarely as neat as training stages. There are distracting objects, clutter, and visual noise that confuses recognition. Performance evaluation as scene complexity increases indicates whether the policy is able to identify the desired target from the noise.

Problem complexity: short and long horizon

Many real tasks are not one action, but a chain of dependent subtasks. A small mistake at the beginning can turn into a complete failure. By measuring how accuracy decreases as the chain lengthens, we can understand how accurate the policy is over long series of arguments.

Task developers can write an expected sequence of subtasks in RoboLab and track how the policy progresses along it. It turned out that most policies cope poorly with long tasks - none was able to complete more than four complex subtasks in a row.

How sensitive is your policy to variation?

Figure 11. Scene variations that can affect performance. | Source: NVIDIA

NVIDIA has implemented sensitivity analysis. It allows you to identify variables that affect performance without testing in isolation. Some changes in the environment may reduce performance. But at scale, testing each variable individually is an almost impossible task.

Instead, we run the estimation on many variations of the scene at once and apply sensitivity analysis. It determines which environmental parameters are most strongly associated with success or failure. This is how intuitions turn into quantitative conclusions.

NVIDIA estimated the posterior distribution using neural posterior estimation (NPE). This allowed the company to pinpoint exactly which environmental variable was causing the performance degradation.

Why is this important

Benchmarking in robotics is still far behind other areas of AI research. Without a common industry platform for testing, it is difficult to measure progress. As models become more powerful, success rates alone are not enough.

What is needed is an assessment that evolves as quickly as the models themselves. Benchmarks that expand rather than saturate. Metrics that diagnose, not just score. Analysis that tells the researcher not only how well the model works, but also how to improve it. RoboLab offers a scalable path to real-world diagnostic assessment