GPU VRAM exceeds 32GB with 4096 envs

[prem-chand]

Question

I am training an RL policy for quadrupeds in isaaclab on rough ground with depth camera sensor, however for 4096 envs the GPU VRAM exceeds beyond 32GB (which is what RTX 5090 provides), the image size i'm using is 64x64. how can i resolve this without compromising the training.

Here is the the depth cam sensor config:

depth_sensor = RayCasterCameraCfg(
        prim_path="/World/envs/env_.*/Robot/base",
        mesh_prim_paths=["/World/ground"],
        offset=RayCasterCameraCfg.OffsetCfg(pos=(0.5, 0, 0.05), rot=(0.389, 0.0, 0.921, 0.0)), 
        data_types=["distance_to_image_plane"],
        debug_vis=False,
        pattern_cfg=patterns.PinholeCameraPatternCfg(
            # focal_length=24.0,
            # horizontal_aperture=46.0,
            focal_length=1.3,
            horizontal_aperture=3.8,
            height=64,
            width=64,
        ),
        max_distance=2,
    )

[garylvov]

I've found similar VRAM usage at that resolution/number of environments. So far, I've found that the only viable solution is to decrease the number of environments or camera resolution