NVIDIA's New AI Can Predict The Future

TL;DR

DreamDojo trains robots using 44,000 hours of human video and four key techniques to accurately predict real-world physics and object interactions.

Key Points

• 1.Simulation-to-reality transfer fails because simulations aren't accurate enough. Robots trained in virtual environments often perform well in games but break down completely when deployed in the real world, a problem observed across AI and robotics labs globally.
• 2.DreamDojo uses 44,000 hours of human video with 4 billion+ frames. Despite humans and robots having different bodies and the footage containing no action labels, the AI is forced to self-interpret actions and compress data down to fundamental patterns.
• 3.Four genius ideas make unlabeled video useful for robot training. These include self-labeled action understanding, information compression, relative rather than absolute coordinates, and block-based frame prediction (4 frames at a time) to prevent the AI from 'cheating' by peeking ahead.
• 4.The new method dramatically outperforms previous techniques on physical predictions. Where older methods cause hands to clip through paper or fail to move lids, DreamDojo correctly simulates crumpling paper and lid displacement with realistic physics.
• 5.Distillation makes the model 4x faster, reaching ~10 frames per second interactively. The full model requires 35 denoising steps per prediction, but a student model trained via distillation matches quality at interactive speed, and all code and pre-trained models are freely available.