Description
A while ago we removed the index pointer from ComponentData back to the owning component in order to save memory. This caused significant trouble for certain operations that walked up the storage tree (like getname, index(), and when generating ComponentUIDs). At the time, those were not frequent operations, so we didn't think that it was too much of a performance hit.
However, I am increasingly running into situations where not having a "cheap" index pointer has made code more cumbersome or complex and I believe that we should re-add the pointer.
If we re-add the pointer, there is an opportunity to significantly reduce the memory footprint for VarData by moving the _lb, _ub, _domain, _fixed, and _stale values out of VarData slots and into dicts held on the owning Var. The idea is that for the vast majority of models, those values are uniform across all the VarData objects in any given Var. In that case, we could only store the scalar value. When someone overrides the "default", then we create the dict and store the exceptions. This has a break-even memory profile at N=2 (for sparse Vars). At N=1000, the sparse form reduces memory by 44%. The only downside is that if the Var does end up with each of the 5 slots getting specialized values for each _VarData, memory usage goes up by 60-70% (worst case for N=1000; for N=2 it is 250% for tuple indices and 300% for scalar indices). Access times are roughly equivalent.
Attached is a script I used to play around & profile: centralized_vardata.py.txt
I suspect that we can play a special game with SimpleVars so that they don't incur either memory or access overhead.