Hugging Face Papers ↗ · 2026-11-10
BirdDreams:联合视角与鸟瞰视频生成的驾驶世界模型
研究论文 Hugging Face Papers 原文语言:中文
NVIDIA 研究者提出 BirdDreams,一种自回归驾驶世界模型,能从单张相机图像同时生成透视视频和鸟瞰(BEV)视频,无需高精地图或车辆状态输入;在 PhysicalAI-AV 和 nuScenes 上,其自车轨迹跟随与周围车辆反应生成优于此前模型。
全文
Drivingworldmodelsaimtosimulatehowasceneunfoldsunderdifferentegoactions.Thisrequiresplausiblemotionofsurroundingvehiclesaswellaspreciseego-trajectorycontrol.Learningvehiclemotionfromperspectivevideoaloneischallengingbecausemotionisentangledwithappearanceandvehiclescanbeoccluded.Priorworkthereforeconditionsvideogenerationonlayoutsthatspecifyvehiclemotion,providingcontroloversurroundingvehiclesbutrequiringtheirfuturetrajectoriesasinputs.WeintroduceBirdDreams,anautoregressivedrivingworldmodelthatjointlygeneratesperspectivevideoandaschematicbird's-eye-view(BEV)video,makingscenelayoutpartofthepredictedfuture.TheBEV,renderedfromsceneannotationsfortraining,depictsroadsandvehiclesassimpleshapesatafixedmetricscale,providingatop-downviewoftheirpositionsandmotionwithreducedocclusionbetweenvehicles.Perspectivevideo,bycontrast,retainsscenedetailsthattheBEVomits,suchastrafficsignalsandobjectappearance.Jointattentionbetweentwovideodiffusiontransformers,oneperstream,letseachstreaminformtheotherduringgeneration.ThepredictedBEValsoprovidesageometricinterfaceforegocontrol:wetranslateandrotateitslatestframealonganinputegotrajectoryandusetheresultingframesasasoftconditionthatshowshowthescenewouldappeartomoveifonlytheegovehiclemoved.Atinference,BirdDreamsgeneratesbothvideosfromasinglecameraimagewithoutrequiringanHDmaporvehicle-stateinputs.OnPhysicalAI-AVandnuScenes,BirdDreamsfollowsinputegotrajectoriesmorefaithfullythanpriordrivingworldmodelsandgeneratesmoreplausibleegomotionwithoutego-actionconditioning.HumanevaluationshowsthatBirdDreamsoutperformsaperspective-onlybaselineingeneratingrealisticsurrounding-vehiclebehavior.
View arXiv pageView PDF Project page Add to collection
正文以来源语言显示。