CIL 2026 depth estimation project.
Training infrastructure: Lightning Fabric + Hydra + WandB.
Left to right: original RGB · SAMv2 segmentation overlay · raw DINOv2 depth prediction rendered as a spinning 3D point cloud.
Plane fitting enforces geometric consistency within each SAMv2 segment via robust least-squares (MAD outlier rejection), improving scale-invariant SILog on the test set.
Authors: Harvi Seitaj, Cyril Moser, David Blickenstorfer, Yu-chuan Liao — Department of Computer Science, ETH Zurich
- Setup
- Project structure
- Running training
- Configuring experiments
- SAMv2 segmentation masks
- Adding a new model
- Adding a new loss
- Generating predictions
- WandB & checkpoints
- AI Usage Declaration
The cluster provides a shared conda environment with most dependencies pre-installed. Activate it first, then install this project and any missing packages on top:
# 1. Activate the shared conda env
conda activate monocular-depth-estimation
# 2. Clone and enter the repo
git clone https://github.com/CyrilFMoser/Seblimo
cd Seblimo
# 3. Install the project in editable mode.
# pip skips packages already satisfied by the conda env and only
# pulls in what is genuinely missing (e.g. hydra-core).
pip install --user -e .If you hit version conflicts, install only the project itself and then add the missing packages manually:
pip install --user --no-deps -e .
pip install --user hydra-core # add others as neededThe data lives at /cluster/courses/cil/monocular-depth-estimation/ and is
already mounted on the cluster — you don't need to download anything.
Seblimo/
├── configs/ # All experiment configuration (Hydra)
│ ├── config.yaml # Root config — edit defaults here
│ ├── model/ # One .yaml per model architecture
│ │ ├── dinov2_dpt.yaml # DINOv2-ViT-S + DPT decoder (default)
│ │ └── unet.yaml # UNet baseline
│ ├── data/ # Dataset paths, batch size, split settings
│ ├── training/ # Epochs, validation, logging, hardware
│ ├── optimizer/ # adam.yaml, adamw.yaml
│ ├── scheduler/ # cosine.yaml, none.yaml
│ └── loss/ # silog.yaml, combined.yaml
├── src/seblimo/
│ ├── data/dataset.py # DepthDataset, DepthTestDataset, make_splits()
│ ├── data/mask_generator.py # SAMv2 automatic mask generation + caching
│ ├── models/
│ │ ├── base.py # DepthModel ABC — all models must subclass this
│ │ ├── dinov2_dpt.py # DINOv2-ViT-S encoder + DPT decoder (default model)
│ │ ├── unet.py # UNet with batch norm baseline
│ │ └── dummy.py # Trivial model for smoke-testing
│ ├── losses/
│ │ ├── silog.py # Scale-invariant log RMSE loss
│ │ └── combined.py # WeightedCombinedLoss
│ ├── metrics/depth.py # SILogRMSEMeter for evaluation
│ ├── training/trainer.py # Fabric-based Trainer class
│ └── utils/
│ ├── visualization.py # depth → colourmap, composite preview panels
│ └── postprocess.py # segment-aware plane fitting (SAMv2 masks)
├── train.py # Entry point: python train.py [overrides]
├── eval.py # Entry point: python eval.py checkpoint=...
└── predict.py # Entry point: python predict.py checkpoint=...
# Basic run with all defaults (dummy model, SILog loss, AdamW, cosine LR)
python train.py
# Quick smoke-test on CPU to check everything connects
python train.py training.max_epochs=2 data.batch_size=2 training.accelerator=cpu
# Resume from a checkpoint
python train.py training.resume_from=outputs/2025-01-01/12-00-00/checkpoints/last.ptHydra creates a timestamped output directory for each run:
outputs/
└── 2025-01-01/
└── 12-00-00/
├── .hydra/config.yaml ← full config snapshot
└── checkpoints/
├── config.yaml ← copy of the config (self-contained)
├── epoch_0001.pt
├── best.pt ← best validation SILog RMSE
└── last.pt
Every value in configs/ can be overridden from the command line using Hydra's
dot-notation syntax. You can also swap entire sub-configs by name.
# Use a different optimiser
python train.py optimizer=adam
# Change learning rate
python train.py optimizer.lr=3e-4
# Train for more epochs with a larger batch
python train.py training.max_epochs=200 data.batch_size=16
# Use 20% of training data as validation
python train.py data.val_fraction=0.2
# Use combined loss (SILog + gradient matching) instead of SILog only
python train.py loss=combined
# Disable validation entirely
python train.py data.val_fraction=0.0
# Mixed precision on GPU
python train.py training.precision=bf16-mixed
# Disable LR scheduling
python train.py scheduler=none
The pipeline can optionally generate and load per-image segment label maps
using SAMv2 (Segment Anything
Model v2 from Meta). These are stored alongside the RGB/depth data and made
available as a mask key in every batch, intended for future mask-aware
losses such as depth-consistency regularization within segments.
facebook/sam2.1-hiera-large (default) — SAMv2.1 with a Hiera-Large
hierarchical vision encoder. ~224 M parameters, ~2.4 GB download from
HuggingFace (cached automatically on first use in ~/.cache/huggingface/).
Lighter alternatives if VRAM is tight (set data.masks.model=...):
| Model ID | Size | VRAM (approx.) |
|---|---|---|
facebook/sam2.1-hiera-large |
2.4 GB | ~8 GB |
facebook/sam2.1-hiera-base-plus |
800 MB | ~4 GB |
facebook/sam2.1-hiera-small |
185 MB | ~3 GB |
facebook/sam2.1-hiera-tiny |
155 MB | ~2.5 GB |
Mask generation runs the model over a grid of 32×32 point prompts per image (1024 forward passes). A CUDA GPU with at least 8 GB VRAM is required. CPU inference is not practical — it takes several minutes per image.
Mask generation only runs once per dataset; after that, every epoch just reads the cached PNGs from disk. Prefer a 2080 Ti over a 1080 Ti — the older card takes roughly twice as long (~25 h for the full set).
masks/
train/ ← masks for the training set (used by train.py)
test/ ← masks for the test set (used by predict.py)
Both directories are tracked with Git LFS (see .gitattributes).
The sam2 package is an optional dependency (not installed by default):
pip install --user -e ".[masks]"
# or simply:
pip install --user sam2If the masks have already been generated and pushed, teammates can pull them without ever touching a GPU:
# First-time setup (once per machine)
git lfs install
# Pull all mask files
git lfs pullThe script writes directly to masks/train/ and masks/test/ regardless
of where it is submitted from. The paths are anchored to the script's own
location in the repo.
# Quick smoke-test (20 images per split, ~1 min)
# Leave N_IMAGES=20 in the script, then:
sbatch scripts/generate_masks.sh
# Full generation (all images, ~12–25 h depending on GPU)
# Set N_IMAGES=0 in the script, then:
sbatch scripts/generate_masks.shAfter generation, push to share with teammates:
git add masks/train masks/test
git commit -m "Add SAMv2 masks"
git push # LFS pointers are pushed automatically# Training — masks are read from masks/train/ automatically
python train.py data.masks.enabled=true
# Prediction — masks are read from masks/test/ automatically
python predict.py checkpoint=.../best.pt data.masks.enabled=true
# Use a lighter model to save VRAM
python train.py data.masks.enabled=true data.masks.model=facebook/sam2.1-hiera-base-plusOn the first run (if masks are missing), training/prediction will pause to generate them before starting. Subsequent runs skip this entirely.
Each image produces one 16-bit grayscale PNG — a uint16 array of shape
(H, W) where each pixel value is a segment ID (1-indexed). Value 0 means
no segment (shouldn't occur in practice). Smaller foreground objects
overwrite larger background regions, so distinct objects always get distinct
IDs. uint16 supports up to 65535 IDs, well above SAMv2's typical output
of 50–300 masks per image.
PNG's delta filter + zlib compression reduces each mask from ~1.2 MB (raw int32) to roughly 50–200 KB, so the full training set fits comfortably in a HOME directory.
When masks are enabled, every batch contains an additional key:
batch["seg_mask"] # Tensor, shape (B, 1, H, W), dtype int64Shape mirrors batch["depth"]. Existing losses are unaffected — they only
read rgb and depth from the batch.
- Create
src/seblimo/models/my_model.py:
from seblimo.models.base import DepthModel
import torch.nn as nn
from torch import Tensor
class MyModel(DepthModel):
def __init__(self, some_param: int = 64):
super().__init__()
# build your network here
...
def forward(self, rgb: Tensor) -> Tensor:
# rgb: (B, 3, H, W) — ImageNet-normalised
# return: (B, 1, H, W) — predicted depth in metres (positive)
...The only contract is: input (B, 3, H, W), output (B, 1, H, W). Outputs
should be positive (depth in metres).
- Create
configs/model/my_model.yaml:
_target_: seblimo.models.my_model.MyModel
some_param: 128- Use it:
python train.py model=my_modelThat's it — no changes to the training code needed.
-
Create
src/seblimo/losses/my_loss.pywith ann.Modulethat takes(pred, gt)and returns a scalar tensor. -
Create
configs/loss/my_loss.yaml:
_target_: seblimo.losses.my_loss.MyLoss
some_param: 1.0- Use it:
python train.py loss=my_loss
To combine multiple losses, edit or copy configs/loss/combined.yaml and list
your losses with weights.
After training, run inference on the test set:
python predict.py checkpoint=outputs/2025-01-01/12-00-00/checkpoints/best.ptThis writes a ready-to-submit predictions/submission.csv directly — depths
are encoded as float16 → zlib-compressed → base64, matching the competition
format. No separate submission script needed.
Optional post-processing flags (all off by default):
# TTA + MAD plane fit (best test score)
python predict.py checkpoint=.../best.pt \
+predict.tta=true \
+predict.mad_plane_fit=true \
+predict.mad_k=2.0
# TTA + soft MAD plane fit (blended variant)
python predict.py checkpoint=.../best.pt \
+predict.tta=true \
+predict.soft_mad_plane_fit=true \
+predict.mad_k=2.0 \
+predict.alpha=0.5
# Override output directory
python predict.py checkpoint=.../best.pt output_dir=my_predictions/Benchmarks three modes against the validation split — all start from TTA predictions:
| Mode | Description |
|---|---|
tta |
Horizontal-flip TTA only, no post-processing |
tta_mad |
TTA + MAD-robust plane fit per segment |
tta_soft_mad |
TTA + blended MAD plane fit (alpha * plane + (1-alpha) * pred) |
python eval.py checkpoint=outputs/.../best.pt
# Tune MAD threshold, blend factor, or minimum segment size
python eval.py checkpoint=.../best.pt +eval.mad_k=2.0 +eval.alpha=0.5 +eval.min_pixels=300
# Run offline (no WandB account needed)
python eval.py checkpoint=.../best.pt wandb.mode=offlineResults are logged to WandB under eval/ keys alongside a depth preview
gallery showing raw vs post-processed predictions and SAMv2 segment masks.
Before spending GPU hours, verify your installation is fully working on the login node:
python scripts/smoke_test.pyThis creates a tiny synthetic dataset in /tmp, runs 2 training epochs on
CPU, generates predictions, and checks that checkpoints and logs were written.
Takes about 30 seconds. Expected output:
ALL CHECKS PASSED — your installation is working correctly.
Weights & Biases is the experiment tracker — it records loss curves, metrics, and depth preview images for every training run so the whole team can compare experiments in a shared dashboard.
| Term | What it means |
|---|---|
| account | Your personal WandB account (free at wandb.ai). Each person needs one. |
| entity | Either your personal username, or a team name. If you create a team on WandB and set entity to the team name, all runs from all teammates end up in one shared workspace. |
| project | A named bucket for related runs — ours is Monocular Depth Estimation. All runs in the same project appear together in the dashboard. |
| run | One execution of python train.py. Gets a unique name, stores all metrics and config. |
# 1. Create a free account at https://wandb.ai, then:
wandb login
# Paste your API key when prompted. It's saved to ~/.netrc — you won't need
# to do this again on this machine.The team and project are already configured in configs/config.yaml:
wandb:
entity: seblimo
project: Monocular Depth EstimationRuns will appear at: wandb.ai/seblimo/Monocular Depth Estimation
| Key | When | What |
|---|---|---|
train/loss |
every step | training loss |
train/lr |
every step | current learning rate |
val/loss |
every val epoch | validation loss |
val/silog_rmse |
every val epoch | scale-invariant RMSE (competition metric) |
val/depth_gallery |
every val epoch | composite panel: RGB | pred | GT | error |
N images per epoch is controlled by training.log_images_per_val (default 8).
# Offline: logs are saved locally, you can sync later
python train.py wandb.mode=offline
# Sync offline runs to WandB later
wandb sync outputs/<date>/<time>/wandb/
# Disable WandB entirely (nothing is logged or saved)
python train.py wandb.mode=disabledEach run saves checkpoints to its own timestamped folder:
outputs/
└── 2026-01-01/
└── 12-00-00/
├── .hydra/config.yaml ← full config (written by Hydra)
└── checkpoints/
├── config.yaml ← copy of the above (for safe-keeping)
├── epoch_0001.pt
├── epoch_0002.pt
├── best.pt ← lowest val/silog_rmse so far
└── last.pt ← always the most recent epoch
To resume training from a checkpoint:
python train.py training.resume_from=outputs/2026-01-01/12-00-00/checkpoints/last.ptThe config.yaml copy inside the checkpoint folder means you can fully
reproduce any run even if the outputs/ tree is cleaned up:
python train.py --config-path outputs/2026-01-01/12-00-00/checkpoints --config-name configThe competition uses scale-invariant RMSE (SILog RMSE):
d_i = log(pred_i) - log(gt_i)
SILog-RMSE = sqrt(mean(d²) - mean(d)²) × 100
Lower is better. Both the training loss and the val/silog_rmse metric use
λ=1.0 (fully scale-invariant), matching the Kaggle evaluation exactly.
Tool used: Claude Sonnet 4.5 / 4.6 (Claude Code CLI)
Files affected: train.py, eval.py, predict.py, src/seblimo/training/trainer.py, src/seblimo/data/dataset.py, src/seblimo/losses/, src/seblimo/metrics/, configs/, pyproject.toml, src/seblimo/models/dinov2_dpt.py, src/seblimo/models/DINOV2_DPT.md, scripts/generate_masks.sh, README.md
Purpose: The research direction — architecture selection (DINOv2+DPT), the SAMv2 mask integration strategy, loss design, and all experiment decisions — was entirely our own. The overall project infrastructure (Lightning Fabric training loop, Hydra config system, WandB integration, entry-point scripts, some documentation) was scaffolded with Claude Code. Having built equivalent training infrastructure from scratch in prior courses, we used AI to handle this standard boilerplate so we could focus our effort on the research contributions specific to this project. The DINOv2+DPT model (dinov2_dpt.py) was implemented with Claude's help after we read the DPT paper; Claude also generated the accompanying DINOV2_DPT.md summary to help us grasp it better. generate_masks.sh (SLURM job script for SAMv2 mask generation) and parts of this README were generated directly. The training infrastructure and losses are fully understood by the team. The DPT decoder implementation is understood at the architectural level: patch tokens are reassembled into spatial feature maps at multiple scales, fused via a feature pyramid with skip connections, and upsampled to the full resolution depth map. The low-level tensor operations were AI-assisted, but the architectural choices and their rationale are fully understood.