Nvidia releases DreamDojo, a robot ‘world model’ trained on 44,000 hours of human video

A team of researchers led by Nvidia has released DreamDojo, a new AI system designed to teach robots how to interact with the physical world by watching tens of thousands of hours of human video — a development that could significantly reduce the time and cost required to train the next generation of humanoid machines.

The research, published this month and involving collaborators from UC Berkeley, Stanford, the University of Texas at Austin, and several other institutions, introduces what the team calls "the first robot world model of its kind that demonstrates strong generalization to diverse objects and environments after post-training."

At the core of DreamDojo is what the researchers describe as "a large-scale video dataset" comprising "44k hours of diverse human egocentric videos, the largest dataset to date for world model pretraining." The dataset, called DreamDojo-HV, is a dramatic leap in scale — "15x longer duration, 96x more skills, and 2 ...