From Demonstrations to Rewards: Test-Time Prompt Optimization for VLM Reward Models
Demo2Reward tunes a VLM reward model's instruction from a few demonstrations to reduce false positives and improve robotic policy learning.
Intelligence analysis by GPT-5.4 Mini
The paper argues that VLMs can act as reward models for robotics, but only if their prompts are tuned carefully. Demo2Reward adapts the instruction at test time using 3-10 expert trajectories, aiming to keep true positives while cutting false positives.
A robot needs a score to know if it is doing the right thing. But writing that score by hand can be hard.
This paper says a smart computer model can be given a few example robot runs, then its instructions can be adjusted so it gives better scores. It is a bit like showing a coach a few good plays so the coach learns what to praise and what to ignore.
The goal is simple: fewer wrong scores, better robot learning, and less manual work for people building the robot system.
Analysis
What the paper claims
The paper starts from a practical problem in reinforcement learning: good rewards are often hard to define, and in real-world settings they may not exist at all. It focuses on vision-language models as zero-shot reward models, a direction that has attracted attention because the model can judge whether a robot action matches a goal from language and visual input.
The authors argue that these reward models are fragile when the prompt is not carefully written. In their framing, the biggest failure mode is false positives, where the model assigns reward to the wrong behavior. That matters because mistaken rewards can mislead policy learning and hurt the final robot policy.
Demo2Reward
To address that, the paper proposes Demo2Reward, a test-time adaptation method that uses a small set of expert demonstrations, described as 3 to 10 trajectories, to optimize the language instruction given to the reward model. The stated goal is to reduce false positives while preserving true positives. The key point is that this happens before policy training, so it does not require extra model training or added computation during policy learning.
Reported results
According to the abstract, Demo2Reward outperforms existing zero-shot and few-shot VLM reward models across multiple simulated robotics tasks and different policy backbones. The authors also say the method transfers to a real-world robotic learning setting, where it enables policy learning without manually engineering a reward function.
The paper is positioned as a bridge between demonstrations and rewards: instead of using demonstrations only to train a policy, it uses them to improve the reward model that guides learning. That makes it potentially useful anywhere reward design is expensive or brittle, especially in robotics.
Key points
- The paper addresses reward design for reinforcement learning, especially in robotics where rewards are often hard to hand-code.
- Demo2Reward adapts a VLM reward model's language instruction at test time using 3 to 10 expert trajectories.
- The method is meant to reduce false positives while keeping true positives.
- The authors say it needs no extra training and no added computation during policy learning.
- The abstract claims gains in simulated tasks and transfer to a real-world robotic learning scenario.
If the method works as described, it could make robot learning easier to start because teams would need less hand-built reward design. The paper also suggests the approach can transfer from simulation to a real robot setting, which is the main test that matters for practical use.
The method still depends on a small set of demonstrations, so it may be sensitive to how representative those trajectories are. If the optimized prompt misses edge cases or still produces false positives, policy learning could still drift in the wrong direction.
