Fix IncrementalFixedLagSmoother with factor removal #4612

Job	Run time
ubuntu-clang-quaternions Release	25m 19s
ubuntu-clang-cayleymap Release	24m 39s
ubuntu-build-examples Release	25m 20s
ubuntu-no-unstable Release	17m 6s
ubuntu-clang-system-libs Release	24m 52s
ubuntu-clang-tbb Release	23m 21s
ubuntu-clang-deprecated Release	24m 32s
	2h 45m 9s

Provide feedback