Questions about unused modules in Model class for training implementation

[StupidAdam]

Hello,

Thank you for open-sourcing the implementation of your excellent work PartField! I'm currently studying the model architecture through the provided codebase.

During code analysis, I noticed two defined components in the Model class that aren't invoked in the inference pipeline model_trainer_pvcnn_only_demo.py:

1. self.sdf_decoder (line 41).
2. self.grid_coord (line 56).

As I'm attempting to reproduce the full training process, could you please clarify:

• Whether these modules are part of the training-specific implementation?
• If so, what would be the proper way to integrate them into the training pipeline?
• Are there any additional components needed to activate these modules?

Also, I notice there may exist a feature loss, how can I compute it?

Any guidance or code pointers would be greatly appreciated to better understand the complete architecture design.

Best Regards,
Adam

[Colin97]

Hello,

In our original version, we also included an SDF prediction task as an auxiliary objective; sdf_decoder and grid_coord were used for this purpose. However, this task was later discarded, so you may now ignore these two components.

As for the feature loss, please refer to our paper or the following code snippet (without hard negative mining):

` N, M, _ = PA.shape
coord = torch.cat((PA, PB, PC), 1) # N, 3M, 3

    point_feat = sample_triplane_feat(planes, coord) # N, 3M, C
    l1_reg = self.l1_loss(point_feat, torch.zeros_like(point_feat)).mean()

    featA, featB, featC = torch.split(point_feat, M, dim=1)
    
    cosAB = torch.exp(F.cosine_similarity(featA, featB, dim=2) * self.logit_scale)
    cosAC = torch.exp(F.cosine_similarity(featA, featC, dim=2) * self.logit_scale)
    cosBC = torch.exp(F.cosine_similarity(featB, featC, dim=2) * self.logit_scale)

    triplet_loss = -(torch.log(cosAB / (cosAB + cosAC)) + torch.log(cosAB / (cosAB + cosBC))) / 2

    triplet_acc = torch.logical_and(cosAB > cosAC, cosAB > cosBC)
    return triplet_loss.mean(), triplet_acc.float().mean(dim=-1), l1_reg`

[Colin97]

Sorry, after checking with the team, we confirmed that in our final version, we still retained SDF field prediction as an auxiliary task, using the ground-truth SDF of the mesh as supervision. The sdf_decoder and grid_coord were used for this purpose.

[StupidAdam]

@Colin97 ​​Thanks for your detailed explanation! To clarify our concern: In the code snippet l1_reg = self.l1_loss(point_feat, torch.zeros_like(point_feat)).mean(), the L1 loss is computed between point_feat and a zero tensor. Could you elaborate on the motivation behind this design?

[lauvli]

Hello,

In our original version, we also included an SDF prediction task as an auxiliary objective; sdf_decoder and grid_coord were used for this purpose. However, this task was later discarded, so you may now ignore these two components.

As for the feature loss, please refer to our paper or the following code snippet (without hard negative mining):

` N, M, _ = PA.shape coord = torch.cat((PA, PB, PC), 1) # N, 3M, 3

    point_feat = sample_triplane_feat(planes, coord) # N, 3M, C
    l1_reg = self.l1_loss(point_feat, torch.zeros_like(point_feat)).mean()

    featA, featB, featC = torch.split(point_feat, M, dim=1)
    
    cosAB = torch.exp(F.cosine_similarity(featA, featB, dim=2) * self.logit_scale)
    cosAC = torch.exp(F.cosine_similarity(featA, featC, dim=2) * self.logit_scale)
    cosBC = torch.exp(F.cosine_similarity(featB, featC, dim=2) * self.logit_scale)

    triplet_loss = -(torch.log(cosAB / (cosAB + cosAC)) + torch.log(cosAB / (cosAB + cosBC))) / 2

    triplet_acc = torch.logical_and(cosAB > cosAC, cosAB > cosBC)
    return triplet_loss.mean(), triplet_acc.float().mean(dim=-1), l1_reg`

Hello, while reproducing your team's work, I noticed that you used several loss terms including triplet(1.0), sdf(1.0), feat(10.0), and l1 loss(0.0).

From my understanding:

• triplet_loss = -(torch.log(cosAB / (cosAB + cosAC)) + torch.log(cosAB / (cosAB + cosBC))) / 2
• sdf_loss represents the SDF prediction task
• l1_reg = self.l1_loss(point_feat, torch.zeros_like(point_feat)).mean()

However, I'm unclear about what the feat loss specifically represents. Moreover, I noticed its weight is significantly larger compared to other auxiliary losses, which makes me think it might be a crucial loss function. Could you please provide some guidance on this?

[Colin97]

@Colin97 ​​Thanks for your detailed explanation! To clarify our concern: In the code snippet l1_reg = self.l1_loss(point_feat, torch.zeros_like(point_feat)).mean(), the L1 loss is computed between point_feat and a zero tensor. Could you elaborate on the motivation behind this design?

it's just a regularization term.

[Colin97]

Hello,
In our original version, we also included an SDF prediction task as an auxiliary objective; sdf_decoder and grid_coord were used for this purpose. However, this task was later discarded, so you may now ignore these two components.
As for the feature loss, please refer to our paper or the following code snippet (without hard negative mining):
` N, M, _ = PA.shape coord = torch.cat((PA, PB, PC), 1) # N, 3M, 3

    point_feat = sample_triplane_feat(planes, coord) # N, 3M, C
    l1_reg = self.l1_loss(point_feat, torch.zeros_like(point_feat)).mean()

    featA, featB, featC = torch.split(point_feat, M, dim=1)
    
    cosAB = torch.exp(F.cosine_similarity(featA, featB, dim=2) * self.logit_scale)
    cosAC = torch.exp(F.cosine_similarity(featA, featC, dim=2) * self.logit_scale)
    cosBC = torch.exp(F.cosine_similarity(featB, featC, dim=2) * self.logit_scale)

    triplet_loss = -(torch.log(cosAB / (cosAB + cosAC)) + torch.log(cosAB / (cosAB + cosBC))) / 2

    triplet_acc = torch.logical_and(cosAB > cosAC, cosAB > cosBC)
    return triplet_loss.mean(), triplet_acc.float().mean(dim=-1), l1_reg`

Hello, while reproducing your team's work, I noticed that you used several loss terms including triplet(1.0), sdf(1.0), feat(10.0), and l1 loss(0.0).

From my understanding:

• triplet_loss = -(torch.log(cosAB / (cosAB + cosAC)) + torch.log(cosAB / (cosAB + cosBC))) / 2
• sdf_loss represents the SDF prediction task
• l1_reg = self.l1_loss(point_feat, torch.zeros_like(point_feat)).mean()

However, I'm unclear about what the feat loss specifically represents. Moreover, I noticed its weight is significantly larger compared to other auxiliary losses, which makes me think it might be a crucial loss function. Could you please provide some guidance on this?

where did you find triplet(1.0), sdf(1.0), feat(10.0), and l1 loss(0.0)

[lauvli]

Hello,
In our original version, we also included an SDF prediction task as an auxiliary objective; sdf_decoder and grid_coord were used for this purpose. However, this task was later discarded, so you may now ignore these two components.
As for the feature loss, please refer to our paper or the following code snippet (without hard negative mining):
` N, M, _ = PA.shape coord = torch.cat((PA, PB, PC), 1) # N, 3M, 3

    point_feat = sample_triplane_feat(planes, coord) # N, 3M, C
    l1_reg = self.l1_loss(point_feat, torch.zeros_like(point_feat)).mean()

    featA, featB, featC = torch.split(point_feat, M, dim=1)
    
    cosAB = torch.exp(F.cosine_similarity(featA, featB, dim=2) * self.logit_scale)
    cosAC = torch.exp(F.cosine_similarity(featA, featC, dim=2) * self.logit_scale)
    cosBC = torch.exp(F.cosine_similarity(featB, featC, dim=2) * self.logit_scale)

    triplet_loss = -(torch.log(cosAB / (cosAB + cosAC)) + torch.log(cosAB / (cosAB + cosBC))) / 2

    triplet_acc = torch.logical_and(cosAB > cosAC, cosAB > cosBC)
    return triplet_loss.mean(), triplet_acc.float().mean(dim=-1), l1_reg`

Hello, while reproducing your team's work, I noticed that you used several loss terms including triplet(1.0), sdf(1.0), feat(10.0), and l1 loss(0.0).
From my understanding:

• triplet_loss = -(torch.log(cosAB / (cosAB + cosAC)) + torch.log(cosAB / (cosAB + cosBC))) / 2
• sdf_loss represents the SDF prediction task
• l1_reg = self.l1_loss(point_feat, torch.zeros_like(point_feat)).mean()

However, I'm unclear about what the feat loss specifically represents. Moreover, I noticed its weight is significantly larger compared to other auxiliary losses, which makes me think it might be a crucial loss function. Could you please provide some guidance on this?

where did you find triplet(1.0), sdf(1.0), feat(10.0), and l1 loss(0.0)

I was examining the model configuration via model = torch.load('./model/model_objaverse.ckpt') and noticed the following loss settings in model['hyper_parameters']:

'loss': CfgNode({'triplet': 1.0, 'sdf': 1.0, 'feat': 10.0, 'l1': `0.0})`

Additionally, I observed the sampling parameters:

'num_pos': 64, 'num_neg_random': 256, 'num_neg_hard_pc': 256, 'num_neg_hard_emb': 0

I’m curious about two points:

​​feat loss term​​: Could you clarify what this term represents? Given its high weight (10.0), it seems to play a critical role, but I couldn’t find documentation on its specifics.
​​Hard negative sampling​​: The setting num_neg_hard_emb=0 suggests that feature-based hard negatives might have been disabled in the second stage. Was this an intentional design choice? If so, could you share the rationale behind it?
Thank you for your time and insights!

[Colin97]

Training with triplet loss and SDF loss should work fine.

Training with only distance-based hard negative mining should yield sufficiently good results.