Depth Pro 2: Sharp and Temporally Consistent Metric Monocular Depth

Media reports Apple 机器学习 Source language: English

Depth Pro 2 is an approach for estimating dense metric depth for images and video at full resolution, overcoming the traditional trade-off between temporal consistency and boundary sharpness.

Full text

Depth Pro 2: Sharp and Temporally Consistent Metric Monocular Depth

AuthorsPeiyun Hu, Marko Mihajlovic, Lyndon Duong, Lahav Lipson, Wei Dong, Amaël Delaunoy, Stephan R. Richter, Vladlen Koltun

View source code (GitHub)

We present Depth Pro 2, an approach to estimating dense metric depth for images and video at full resolution. By analyzing prior methods for monocular depth estimation, we find that they commonly trade off temporal consistency and boundary sharpness: multi-frame methods favor consistency, single-frame methods favor sharpness. To overcome this trade-off, we train a single, scalable network that processes both images and video and predicts depth maps at native resolution. Depth Pro 2 produces depth maps that are metrically accurate, sharp, and temporally consistent all at once, outperforming prior work in all of these dimensions. We release a family of models that define a new Pareto frontier in terms of metric accuracy, sharpness, temporal consistency, and runtime.

We release code and weights here.

Depth Pro 2 estimates the depth of every pixel, in meters, from images taken with an ordinary camera. One network handles a single photo, a whole video, or a live camera feed, at the input’s full resolution. Its depth is sharp at object edges and steady from frame to frame.

Sources

← Back to Daily AI Report