GradCAM question

[alexcbb]

Hi, quick question following the one asked in #7 . In GradCam you use a target to guide the grad process extraction which is by default a classification target. For your visualization did you keep the default targets or predicted actions ? Because I did not managed to get proper visualization on my own experiments with pre-trained MCR:

As you can see it does not really focus on the objects but rather only one the robot arm. Do you have any suggestions ? Thank you in advance

[guangnianyuji]

This is really quite strange. Could you send me the original image so I can have a try?

[alexcbb]

This is really quite strange. Could you send me the original image so I can have a try?

Hi, here is the image

[guangnianyuji]

Hi, this is the result I got, just used the sample code in https://github.com/jacobgil/pytorch-grad-cam.

[alexcbb]

What checkpoint are you using ? Can you provide the full code please ? I'm still not able to reproduce using the provided checkpoints

[guangnianyuji]

import mcr
import argparse
import os
import cv2
import numpy as np
import torch
from torchvision import models
from pytorch_grad_cam import (
GradCAM, FEM, HiResCAM, ScoreCAM, GradCAMPlusPlus,
AblationCAM, XGradCAM, EigenCAM, EigenGradCAM,
LayerCAM, FullGrad, GradCAMElementWise, KPCA_CAM, ShapleyCAM,
FinerCAM
)
from pytorch_grad_cam import GuidedBackpropReLUModel
from pytorch_grad_cam.utils.image import (
show_cam_on_image, deprocess_image, preprocess_image
)
model = mcr.load_mcr(ckpt_path="mcr_resnet50.pth")

print(model)

targets = None

Using the with statement ensures the context is freed, and you can
recreate different CAM objects in a loop.
cam_algorithm = GradCAM
target_layers = [model.layer4]

image_path = "issue10.png"
device="cpu"

aug_smooth = False
eigen_smooth = False

rgb_img = cv2.imread(image_path, 1)[:, :, ::-1]
rgb_img = np.float32(rgb_img) / 255
input_tensor = preprocess_image(rgb_img,
mean=[0.485, 0.456, 0.406],
std=[0.229, 0.224, 0.225]).to(device)

with cam_algorithm(model=model,
target_layers=target_layers) as cam:

# AblationCAM and ScoreCAM have batched implementations.
# You can override the internal batch size for faster computation.
cam.batch_size = 32
grayscale_cam = cam(input_tensor=input_tensor,
                    targets=targets,
                    aug_smooth=aug_smooth,
                    eigen_smooth=eigen_smooth)

grayscale_cam = grayscale_cam[0, :]

cam_image = show_cam_on_image(rgb_img, grayscale_cam, use_rgb=True)
cam_image = cv2.cvtColor(cam_image, cv2.COLOR_RGB2BGR)
gb_model = GuidedBackpropReLUModel(model=model, device=device)
gb = gb_model(input_tensor, target_category=None)

cam_mask = cv2.merge([grayscale_cam, grayscale_cam, grayscale_cam])
cam_gb = deprocess_image(cam_mask * gb)
gb = deprocess_image(gb)

output_path="outputs"
os.makedirs(output_path, exist_ok=True)
cam_output_path = os.path.join(output_path, f'cam.jpg')
gb_output_path = os.path.join(output_path, f'gb.jpg')
cam_gb_output_path = os.path.join(output_path, f'cam_gb.jpg')
cv2.imwrite(cam_output_path, cam_image)
cv2.imwrite(gb_output_path, gb)
cv2.imwrite(cam_gb_output_path, cam_gb)

And my ckpt is https://huggingface.co/GqJiang/robots-pretrain-robots/blob/main/mcr_resnet50.pth

[alexcbb]

Ok I was able to get the same result. But then strangely I discovered that the issue that I had was due to a resize operation I had in my code. My preprocess function is :

        transforms.ToTensor(),
        transforms.Resize((224, 224)),
        transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
    ])

With the resize the visualization gets the same as I had before. On what image shape was the model trained ? Do you have any clue on why this phenomenon is happening ? Thanks again for your time !