BirdDreams:联合视角与鸟瞰视频生成的驾驶世界模型

Research Hugging Face Papers Source language: Chinese

NVIDIA 研究者提出 BirdDreams,一种自回归驾驶世界模型,能从单张相机图像同时生成透视视频和鸟瞰(BEV)视频,无需高精地图或车辆状态输入;在 PhysicalAI-AV 和 nuScenes 上,其自车轨迹跟随与周围车辆反应生成优于此前模型。

Full text

Drivingworldmodelsaimtosimulatehowasceneunfoldsunderdifferentegoactions.Thisrequiresplausiblemotionofsurroundingvehiclesaswellaspreciseego-trajectorycontrol.Learningvehiclemotionfromperspectivevideoaloneischallengingbecausemotionisentangledwithappearanceandvehiclescanbeoccluded.Priorworkthereforeconditionsvideogenerationonlayoutsthatspecifyvehiclemotion,providingcontroloversurroundingvehiclesbutrequiringtheirfuturetrajectoriesasinputs.WeintroduceBirdDreams,anautoregressivedrivingworldmodelthatjointlygeneratesperspectivevideoandaschematicbird's-eye-view(BEV)video,makingscenelayoutpartofthepredictedfuture.TheBEV,renderedfromsceneannotationsfortraining,depictsroadsandvehiclesassimpleshapesatafixedmetricscale,providingatop-downviewoftheirpositionsandmotionwithreducedocclusionbetweenvehicles.Perspectivevideo,bycontrast,retainsscenedetailsthattheBEVomits,suchastrafficsignalsandobjectappearance.Jointattentionbetweentwovideodiffusiontransformers,oneperstream,letseachstreaminformtheotherduringgeneration.ThepredictedBEValsoprovidesageometricinterfaceforegocontrol:wetranslateandrotateitslatestframealonganinputegotrajectoryandusetheresultingframesasasoftconditionthatshowshowthescenewouldappeartomoveifonlytheegovehiclemoved.Atinference,BirdDreamsgeneratesbothvideosfromasinglecameraimagewithoutrequiringanHDmaporvehicle-stateinputs.OnPhysicalAI-AVandnuScenes,BirdDreamsfollowsinputegotrajectoriesmorefaithfullythanpriordrivingworldmodelsandgeneratesmoreplausibleegomotionwithoutego-actionconditioning.HumanevaluationshowsthatBirdDreamsoutperformsaperspective-onlybaselineingeneratingrealisticsurrounding-vehiclebehavior.

View arXiv pageView PDF Project page Add to collection

Sources

← Back to the daily brief