Did the robot finish?

RoboJev predicts whether a robot completed its task from a task description and frames sampled across the attempt.

Try a video Watch the film Read the paper

P(success) over time

First score at 0.5 s

P(success)—Waiting for frames

Task Insert the colourful ring into the central wooden peg.

Play the attempt to see the frames RoboJev uses.

These are recorded model outputs, queried every 0.5 s on a task held out from training. View trace.

95.2%Balanced accuracy · 268 SO-101 runs

0.26 sMedian answer time · one H100

$0.19Per 1,000 answers at the measured H100 rate.
See inputs, costs and all results.

Watch the launch

Watch the ring slip off and the score fall.

One task,
ten frames.

We sample ten frames from across the attempt and provide them to RoboJev with the task description. When available, we add the final image from each other camera. RoboJev outputs a probability of success for that attempt.

The two episodes below were held out from training. With all three camera views, the released checkpoint outputs 0.995 and 0.022.

Try an attempt yourself

Ten frames from each ring attempt, plus final overhead and wrist views, feed into RoboJev and return completion probabilities. — The diagram shows exactly the frames RoboJev sees. Each score belongs to the attempt shown.

Compare the answers

RoboJev, general-purpose models and decision APIs on the recorded benchmarks.

SO-101 balanced accuracy against measured cost per 1,000 answers. RoboJev: 95.2% and $0.19.

Scoreboard comparing RoboJev with frontier language models and decision APIs.

Median response times: RoboJev 0.26 seconds, frontier APIs roughly 5 to 7 seconds.

For the recorded frontier-model runs we used four frames from the front camera. RoboJev received ten front-camera frames and, when available, the final frame from each other camera. The explorer lists the protocol for every result.

On SO-101, the uncertainty range for the difference in balanced accuracy between RoboJev and Astra includes zero. With ten frames, base JEV scores 0.900 AUROC on the unseen-lab set; RoboJev scores 0.890.

Open the full comparison: all 29 model/protocol entries

Loading recorded results…

Inspect individual answers and compare any two models

Billions of
robot hours

If a million robots recorded eight hours a day, that would amount to 2.92 billion hours of video every year.

This chart shows how many people and GPUs it would take to label that video. The fleet sizes are assumptions for this calculation.

Illustrative robot fleet scenarios, annual video hours, people required to watch and H100s required to score.

Estimated cost to grade 68,555 sampled LeRobot episodes, totalling about 501 hours of video. — We sampled 68,555 episodes from 7,641 public datasets to estimate how expensive it would be to grade them all. Our released checkpoint hasn’t graded the whole sample. Explore the catalogue.

First page of the RoboJev paper with its accuracy and cost chart.

Read the research

The paper goes over the training and held-out tasks, including where RoboJev makes mistakes and the confidence intervals around its results.

We fine-tuned JEV-27B-VL with LoRA on 52,598 robot episodes to make RoboJev. We provide the weights and evaluation data; you can also use the LeRobot reward plugin or run the video overlay.

Read the paper

Try your own recording

Try a video Download the model