Wired results for predicting miF in HEMIT dataset

[HelloWorldLTY]

Hi, I try to download the checkpoint from MIPHEI-VIT trained with HEMIT, however, the testing results look very wired.

exp_mithvi = np.load("../../MIPHEI-ViT/notebooks/miphei_vit_pred_hemit_256.npy")

pcc,scc = [],[]
for i,j in zip(exp_mithvi, np.array(map_data)):
    i = i[:,[13,10,0],:,:]
    p1,s1 = calculate_correlations(torch.FloatTensor(i),torch.FloatTensor(j))
    pcc.append(p1)
    scc.append(s1)

np.array(map_data).shape

np.nanmean(np.array(pcc),axis=0)

My average PCC is only around:

array([0.01171147, 0.0020005 , 0.02392417], dtype=float32)

I follow the infernece code and compute the predicted miF as follows:

import tqdm
pred_list = []
for item in tqdm.tqdm(test_dataset):
    image_pil = item[0]
    image = np.asarray(image_pil)
    image = center_crop(image=image)["image"]
    x = transforms(image=image)["image"]

    nuclei_mask = inference_cellpose(image, cellpose_model, radius=radius)
    nuclei_mask_torch = torch.from_numpy(nuclei_mask).unsqueeze(0).unsqueeze(0).cuda().long()

    with torch.inference_mode():
        pred = generator(x.unsqueeze(0).half().cuda())
        pred = pred.clamp(-0.9, 0.9)
        pred = pred[:,:,:,:]
        pred = (pred + 0.9) / 1.8
        pred = (pred * 255).to(torch.uint8)
        
        pred_list.append(pred.cpu().numpy())
outimage = np.array(pred_list)
np.save("miphei_vit_pred_hemit_256.npy",outimage)

Therefore, I have some problems here. 1. Can we resize the original 1024 image for inference and testing (also resize the predicted mif file)? Also, why we do not use the original 1024 image size for prediction? 2. Is the prediction of generator same as the miF strength? I think so as there is no cell segmentation informaiton.

I cannot find the reported pcc info in the manuscript for this dataset, so any suggestions or sharing the original pcc/scc result will be helpful. Thanks!

[Guillaume-Balezo]

Hi, sorry for the late reply.

I just want to clarify one point first: are you evaluating MIPHEI-ViT on the HEMIT dataset ? Could you specify the checkpoint name you are using?

1. When testing MIPHEI-ViT on HEMIT, images are downsampled from 1024×1024 to 512×512 because MIPHEI is trained at 20×, while HEMIT images are provided at 40×.

   I would advise not to change the input resolution too much from the training one (256×256), as H-optimus-0 is a plain ViT with standard positional encoding, and this can degrade performance.

2. The generator directly predicts mIF intensities at the pixel level. If you want cell-level predictions, you need to apply a separate cell segmentation step (e.g. CellPose) and aggregate intensities per cell. There is an example in the inference notebook.

Using our benchmark code, we obtain the following Pearson correlations for MIPHEI-ViT on the HEMIT dataset:

• DAPI: 0.77
• CD3: 0.27
• Pan-CK: 0.58

If possible, could you share the code you use to compute PCC/SCC?

Hope this helps.

[HelloWorldLTY]

Thank you for your message. I guess it might come from a different evaluation approach. I evaluate it based on 512*512 transistion and I use the MIPHEI-ViT released in your ckpt library. The evaluation method of correlation is (same as gigapath):

def calculate_correlations(matrix1, matrix2):
    """
    Calculate Pearson and Spearman correlation coefficients between two matrices.

    Args:
        matrix1 (np.ndarray): The first matrix.
        matrix2 (np.ndarray): The second matrix.

    Returns:
        dict: A dictionary containing Pearson and Spearman correlation coefficients.
    """
    assert matrix1.shape == matrix2.shape, "Matrices must have the same shape"
    b, c, h, w = matrix1.shape

    pearson_correlations = []
    spearman_correlations = []

    for channel in range(c):
        pearson_corrs = []
        spearman_corrs = []

        for batch in range(b):
            flat_matrix1 = matrix1[batch, channel].flatten()
            flat_matrix2 = matrix2[batch, channel].flatten()

            # Remove NaN values
            valid_indices = ~np.isnan(flat_matrix1.cpu().numpy()) & ~np.isnan(flat_matrix2.cpu().numpy())
            flat_matrix1 = flat_matrix1[valid_indices]
            flat_matrix2 = flat_matrix2[valid_indices]

            if len(flat_matrix1) > 0 and len(flat_matrix2) > 0:
                pearson_corr, _ = pearsonr(flat_matrix1.cpu().numpy(), flat_matrix2.cpu().numpy())
                spearman_corr, _ = spearmanr(flat_matrix1.cpu().numpy(), flat_matrix2.cpu().numpy())
            else:
                pearson_corr = np.nan
                spearman_corr = np.nan

            pearson_corrs.append(pearson_corr)
            spearman_corrs.append(spearman_corr)

        # Average correlations across the batch dimension
        pearson_correlations.append(np.nanmean(pearson_corrs))
        spearman_correlations.append(np.nanmean(spearman_corrs))

    return pearson_correlations, spearman_correlations