-
Notifications
You must be signed in to change notification settings - Fork 45
Description
Training Resnet34 (pretrained) for hand and object pose and it's training and validation loss is -4, -6, 5, 7, .. or sometime 150+, 190+...,
100%|██████████| 12328/12328 [1:19:17<00:00, 2.59it/s]
WARNING:root:NaN or Inf found in input tensor.
WARNING:root:NaN or Inf found in input tensor.
100%|██████████| 12328/12328 [49:31<00:00, 4.15it/s]
WARNING:root:NaN or Inf found in input tensor.
WARNING:root:NaN or Inf found in input tensor.
Epoch : 0 finished. Training Loss: nan. Validation Loss: nan
100%|██████████| 12328/12328 [1:20:41<00:00, 2.55it/s]
WARNING:root:NaN or Inf found in input tensor.
WARNING:root:NaN or Inf found in input tensor.
100%|██████████| 12328/12328 [49:48<00:00, 4.13it/s]
WARNING:root:NaN or Inf found in input tensor.
WARNING:root:NaN or Inf found in input tensor.
Epoch : 1 finished. Training Loss: nan. Validation Loss: nan
100%|██████████| 12328/12328 [1:20:53<00:00, 2.54it/s]
WARNING:root:NaN or Inf found in input tensor.
WARNING:root:NaN or Inf found in input tensor.
100%|██████████| 12328/12328 [56:47<00:00, 3.62it/s]
WARNING:root:NaN or Inf found in input tensor.
WARNING:root:NaN or Inf found in input tensor.
Epoch : 2 finished. Training Loss: nan. Validation Loss: nan
100%|██████████| 12328/12328 [1:19:47<00:00, 2.58it/s]
WARNING:root:NaN or Inf found in input tensor.
WARNING:root:NaN or Inf found in input tensor.
90%|█████████ | 11156/12328 [44:38<04:41, 4.16it/s]