[Question] Is it a must to substitute non_slot_dict around calling apply_gradient on tensorflow optimizer?

[Nov11]

This template is for generic questions that a user may have in using HugeCTR.

Note: Before filing an issue, you may want to check out our compiled Q&A list first.

code link:

HugeCTR/sparse_operation_kit/sparse_operation_kit/optimizer.py

Line 205 in d4fecdb

# 2. Switch non_slot_dict

I'm reading code of optimizer wrapper for tensorflow graph mode.
It seems for an optimize like Adam as self._optimizer, its beta1_power and beta2_power are never updated. Instead the Variable copies in OptimizerWrapper are updated.
I suppose it is ok to update non slot variables in self._optimizers and just skip the substitution of non_slot_dict.

Please correct me if I'm wrong.