NVIDIA outlines Cosmos tools for interactive simulation

NVIDIA describes Cosmos as a platform of world foundation models and development tools for physical AI.[1] Cosmos Framework is presented as an open, modular framework for the full world-model development lifecycle, with agent skills and recipes for training, multimodal data processing, optimization and inference.[1] NVIDIA says it supports models beyond Cosmos 3, naming Alpamayo, GR00T, Qwen, pi0.5 and compatible Hugging Face models.[1] Cosmos Framework is described as a development framework, while Cosmos is described as a platform of world foundation models and development tools.[1]

Developers can use either LoRA adaptation or full supervised fine-tuning, NVIDIA says.[1] For training data, the framework accepts JSONL, WebDataset and LeRobot, as well as custom data adapters.[1] NVIDIA also lists offline DPO and online GRPO pipelines, with rule-based or model-based rewards and vLLM rollouts.[1] Distillation and quantization are available to reduce model size, generation steps or memory use.[1] Together, these options give teams multiple routes to adapt and optimize models, though the post’s listed capabilities do not establish which approach will work best for a particular deployment.[1]

NVIDIA lists simulation recipes for robotics, scene generation, multi-camera driving and surgical policy evaluation.[1] Its new Cosmos 3 Nano recipes and example checkpoints are intended to let developers assess those examples and adapt them to custom sensors, action spaces and data domains.[1] A walkthrough uses released checkpoints to generate synchronized camera video and LiDAR observations for autonomous-vehicle, robotics and surgical scenarios.[1] The quoted specifications state support for 30-fps input and output video, with up to 400 frames at 480p or 720p.[1] NVIDIA reports one reference Transfer-Auto run generated seven 720p videos, each 297 frames at 30 fps, and 99 LiDAR sweeps.[1]

For driving scenarios, observations are generated incrementally as an autonomous-vehicle policy or human driving input updates the scenario through a WSM renderer.[1] Another workflow converts depth video and a prompt into RGB video, varying lighting, textures and backgrounds while preserving geometry and motion.[1] Robotics recipes generate the next visual observation from the scene and robot actions, and support interactive two-arm simulation for YAM and AgiBot embodiments controlled by a policy or human-operated joysticks.[1] Surgical recipes can generate video from robot actions, predict actions from video, or generate both together.[1] NVIDIA describes the model as causal and few-step, generating frames from live robot kinematics for real-time, closed-loop simulation.[1]

Cosmos NIM microservices support GPU-accelerated serving through application APIs.[1] NVIDIA reports mean speculative-decoding speedups of roughly two to three times on H200 GPUs, and says streaming video-language-model inference can enable sub-second alerts with 10-times-lower latency.[1] NVIDIA also says Cosmos NIM provides automatic configuration selection, validated performance and quality across supported GPUs, licensing support and quarterly updates.[1] Separately, NVIDIA says Sim-Bimanual achieved higher video quality and reference-frame PSNR than DreamDojo 2B and 14B on two AgiBot benchmarks.[2]