application about it

I have a problem is that I get the 3d-2d matching pairs through the network, then I want to get the transformation matrix through Epro-PnP , and then in the second step project the 3d points onto the image plane with the resulting matrix, and then do the subsequent operations through the neural network , I hope the entire network is end-to-end. Is this idea feasible?