Implement Depth Estimation Pipeline

[Sohaib-Ahmed21]

Depth estimation is the process of determining the distance of objects from a viewpoint, which is crucial for various applications in computer vision, including autonomous driving, augmented reality, and 3D reconstruction..

Sample Input and Output:

Sample images (up) and depth annotation (down).

Applications of Depth Estimation:

1. Estimation of Volumetric Information: Depth estimation models help study the volumetric formation of objects in images, crucial for computer graphics.
2. 3D Representation: Depth estimation enables the development of 3D representations from 2D images.
3. Augmented Reality: Depth estimation ensures accurate object placement and perspective calibration in AR applications.
4. Robotics and Object Trajectory Estimation: Depth information helps estimate the motion trajectory of objects in 3D space.
5. Haze and Fog Removal: Depth estimation aids in removing haze and fog by understanding how they affect distant objects.
6. Portrait Mode: Depth-based blur in portrait mode enhances the focus on subjects and creates appealing background effects.

Depth Estimation Subtasks:

There are two depth estimation subtasks.

Absolute depth estimation: Absolute (or metric) depth estimation aims to provide exact depth measurements from the camera. Absolute depth estimation models output depth maps with real-world distances in meter or feet.

Relative depth estimation: Relative depth estimation aims to predict the depth order of objects or points in a scene without providing the precise measurements.

Depth Estimation Models

• Depth Anything
  Downloads last month on huggingface: 592,379
  Backbone: ResNet and Convolution based
  Cited by: 744
  Published on: April 7, 2024
  Paper: https://arxiv.org/pdf/2401.10891
  Model card: https://huggingface.co/depth-anything/Depth-Anything-V2-Small-hf

• DPT by Intel Labs research
  Downloads last month on huggingface: 162,385
  Backbone: Convolution and VIT based
  Cited By: 2151
  Released on: Mar 24, 2021
  Paper: https://arxiv.org/pdf/2103.13413
  Model card: https://huggingface.co/Intel/dpt-large

• Depth Crafter by Tencent AI Lab
  Downloads last month on huggingface: 362,047
  Backbone: UNet, CLIP and VAE based
  Cited By: 41
  Released on: Nov 27, 2024
  Paper: https://arxiv.org/pdf/2409.02095
  Model card: https://huggingface.co/tencent/DepthCrafter

• Depth Pro by Apple
  Downloads last month on huggingface: 51,653
  Backbone: VIT Based
  Cited by: 54
  Released on: Oct 2, 2024.
  Paper: https://arxiv.org/pdf/2410.02073
  Model card: https://huggingface.co/apple/DepthPro-hf

Why will it be a good addition?

The backbone of above models already exist in keras-hub and above models will also be compatible with keras-hub structure and API, so it will be easy to add them. The pipeline will also follow the same workflow as other pipelines of image classifcation, object detection, etc. Also, above applications are quite crucial in daily life and this pipeline will help users perform above tasks through keras-hub with greater ease.

I'd love to hear comments from the community. FYI @divyashreepathihalli @mattdangerw @sineeli

[divyashreepathihalli]

depthanything is great! would you like to contribute this to KerasHub?

[Sohaib-Ahmed21]

depthanything is great! would you like to contribute this to KerasHub?

Yes, I'd be happy to take that, starting from the depth estimation pipeline. Do we plan to add other famous depth estimation models(as listed above) in the future as well?