The most impressive all-in-one 3D reconstruction and generation, with a single-image input (till 2023-10-10), but ...

[yuedajiong]

2023-10-10:

1. dataset is too small ... (objvarse/3d, imagenet-l/2d, ...)
2. the shape generation network is not deep enough, it is diffucult to generate better shape from src_mesh. such as deeper iteration in alphafold, we can image that the shaping is a dynamic processing, step by step.
3. we need better priori stroage, not parameters in functional netowrk, but priori memory network(!), even vecter-db(?), ... such as MPCN.
4. the pose network is too simple, more fowerful network such as robustlock.
5. ......
   Anyway, the is the best design on this 3d recon/gen task, so far.

2023-10-12:
after try, modify, try, modify, try, modify, ...
I think:
if we want to fit a better mesh, these aspects in model we need to optimize from random initialized:
<notice: SINGLE image in, only, here>
a1. input-image: segmentation/mask(SAM) or not #segment background in model or not
a2. input-priori: where? what? #network-parameter, special memeory-priori-netowrk
a*. ......
b1. mesh - morphism #IMPORTANT ball/ring/...; pick up object template; even smplx for human #open/unbound/...
b2. mesh - scale, center, ...
b3. mesh - pose, #IMPORT if we can not predict pose correctly, can not fit a better mesh. (use pose-network? direct fit? or pose related loss)
b*. ......
c1. render-image: background vs foreground/object
c2. render-image: silhouette / shape
c3. render-image: texture
c4. render-light: no light modeling in this paper.
c*. ......

for these fiting aspects, there are different priority to optimzed, there are different composions to optimized.

in this paper, I found that, the pose prediction is not very accurate， so, the final mesh is not good enough.
i think how to do pose prediciton even representation(important, especial for single image in, what is the front-side/start-point of azimuth and eleation), is still an open problem.