Could a new type of loss be introduced for classes?

[harsanyidani]

In EPro-PnP-Det_v2 if we want to improve the classification performance, could theoretically a new type of loss be introduced with the help of the deformable correspondance head?

I was thinking about how the yaw angle distribution corresponds to different classes. During the AMIS algorithm we could use the generated rotation distribution, evaluate it from 0 to 2pi with some density. Then feed this distribution to a simple network which classifies based on yaw angle. Maybe this isn't suitable for all classes but it might be useful to train a binary classsifier for pedestrians and cones (which can be mixed for classifiers that are based on purely image inputs) and add the scores to the corresponding ones in the FCOS detection head with some weighting.

Or we could just use these orient logprobs for this purpose?:

EPro-PnP-v2/EPro-PnP-Det_v2/epropnp_det/models/dense_heads/deform_pnp_head.py

Lines 563 to 574 in 85215de

	if 'orient' in debug:
	orient_bins = getattr(self.test_cfg, 'orient_bins', 128)
	orient_grid = torch.linspace(
	0, 2 * np.pi * (orient_bins - 1) / orient_bins,
	steps=orient_bins, device=x3d.device)
	# (orient_bins, num_obj, 4)
	pose_grid = pose_opt[None].expand(orient_bins, -1, -1).clone()
	pose_grid[..., 3] = orient_grid[None, :, None]
	cost = evaluate_pnp(
	x3d, x2d, w2d, pose_grid, self.camera, self.cost_fun, out_cost=True)[1]
	orient_logprob = cost.neg().log_softmax(dim=0) + np.log(orient_bins / (2 * np.pi))
	orient_logprob = orient_logprob.transpose(1, 0).cpu().numpy()

This is just an idea and my question is, could this theoretically work? Can this be backpropagated at all?

Thanks in advance for the answer, and for the previous ones too, they've been very useful.

[harsanyidani]

I was experimenting with this and I have a question in connection with this.

EPro-PnP-v2/EPro-PnP-Det_v2/epropnp_det/models/dense_heads/deform_pnp_head.py

Line 573 in 85215de

orient_logprob = cost.neg().log_softmax(dim=0) + np.log(orient_bins / (2 * np.pi))

Here, we don't exactly get logprobs, but a constant is added to each probability based on the density of the distribution (+ np.log(orient_bins / (2 * np.pi)), so the probabilities won't sum to one.
What is the purpose of this? Better visualization/numerical stability?
Thanks in advance!

[Lakonik]

Theoretically these logprobs can be backpropagated. But I don't think information extracted from the pose distribution can help improve the classification.

+ np.log(orient_bins / (2 * np.pi)) is meant to convert the bin probabilities into a continuous density function on [0, 2pi), such that the integral equals 1. This is mainly for visualization purpose.

[harsanyidani]

Theoretically these logprobs can be backpropagated. But I don't think information extracted from the pose distribution can help improve the classification.

I understand. But theoretically, the yaw angle distribution evaluated with some density from 0 to 2pi would roughly look like the logprobs here?

EPro-PnP-v2/EPro-PnP-Det_v2/epropnp_det/models/dense_heads/deform_pnp_head.py

Line 573 in 85215de

orient_logprob = cost.neg().log_softmax(dim=0) + np.log(orient_bins / (2 * np.pi))

np.log(orient_bins / (2 * np.pi)) is meant to convert the bin probabilities into a continuous density function on [0, 2pi), such that the integral equals 1. This is mainly for visualization purpose.

I don't understand this. Without np.log(orient_bins / (2 * np.pi)), the probabililities sum to one, with it they don't.