Salesforce Introduces Interactive Visual Gym to Test Visual AI Assistants

Salesforce AI Research has introduced Interactive Visual Gym, a simulation framework for evaluating visual AI assistants that guide people through physical workflows. According to the company's blog post, the framework pairs simulated physical workflows with autonomous simulated users, and can vary user behavior and inject disruptions. Salesforce also created a benchmark called IVG-Bench. The company frames the work as an evaluation and training tool; practitioners should distinguish its simulation results from evidence of deployed productivity gains. Salesforce Blog report.

A Closed Loop Replaces Passive Video Playback

Salesforce argues that conventional evaluation frameworks rely on passive, pre-recorded video clips and grade an assistant's text outputs against a static answer key. In that setup, the company says, an assistant's warning and the recorded person's subsequent action are decoupled, so organizations cannot verify whether guidance changes execution.

Interactive Visual Gym is designed as an active feedback loop instead. Per the post, the assistant analyzes the simulated user's video feed and provides spoken intervention; the simulated user then alters its actions and decisions in response to that advice, changing the state of a digital twin in real time.

Disruptions and Behavioral Profiles Stress-Test Guidance

The harness operates at a high-level semantic layer rather than requiring micro-level physics controls such as joint velocities, per the company.

Salesforce also models user personas along four behavioral axes: domain expertise, ranging from novices to experienced users; AI trust, meaning how quickly a user accepts or questions recommendations; execution velocity, from cautious to rapid; and interruption tolerance, from step-by-step support to necessary-only interruptions. These profiles are simulated constructs, not observations of a deployed workforce.

What the Benchmark Reports

Salesforce says it created IVG-Bench to evaluate 14 leading visual AI assistants across 315 complex scenarios in three operational environments.

The company also reports that better task success does not necessarily mean better guidance during a task, and that for some open-weight models later responses become less accurate and less relevant as the activity proceeds. These are vendor-reported benchmark results from simulated environments; the post does not demonstrate equivalent outcomes in live frontline operations.

Human Ratings and a Training Proof of Concept

To check whether the evaluation reflects human experience, Salesforce says it invited participants to enter the simulator, complete cooking tasks with assistant help and rate their interactions. The company reports that participants' preferences matched the automated scores nearly 80% of the time, which it presents as confirmation that the benchmark predicts human satisfaction.

Salesforce further describes an early proof of concept in which it used the simulator to train an open-source model, incentivizing task completion and error prevention. It reports that the model's success rate rose from 33.3% to 42.2% on tasks it had not seen during training, and that the assistant became better at intervening before mistakes.

What This Means for Practitioners

For enterprise leaders weighing vision-enabled assistants in mission-critical operations, the practical takeaway is a testing method rather than a performance guarantee. Closed-loop simulation lets teams vary user expertise, trust, pace and tolerance for interruption, and inject disruptions at chosen milestones, so guidance quality can be examined as a task unfolds instead of against a static answer key.

Practitioners should treat the reported success rates, agreement with human ratings and training gains as vendor assertions from simulated settings. A prudent next step is to reproduce such scenarios against a team's own workflows and validate results with live users before converting simulated success into expectations about operational benefit. The post positions IVG as an evaluation and training tool; it does not establish deployed productivity gains.