Description
TL;DR
Top-level memory cached when using dace.compiletime and Quantity and State are both using it as a way to allow type hinting.
DaCe requires a __descriptor__ to describe the memory to it. When using a type hint, the descriptor is called on the class rather than the object. This requires the size to be static - not our use case. To alleviate this, we used dace.compiletime to delay the description to runtime BUT what we discovered is that during SDFG caching it feeds into constant_args, which feeds into the closure, which doesn't get recomputed. Therefore, if the parameters changes, we still call to the first called/binded Quantity/State.
Impact
There's a limited impact to our usage because we follow the "allocate once, never reallocate" paradigm for all of our interfaces (GEOS or Pace), which means we don't see parameters swap at the top-level.
BUT, if we did, it would lead to a very difficult to debug situation and must be addressed urgently.
Mitigation
We can use the orchestrate.LazyMethod override (where we override dace.compiletime) to banish any Quantity/Local type hinting.
We can't do this for State as it is already thoroughly used in the wild.
For state, we can look at orchestration._call_sdfg, which handled caching and marshalling of the arguments. At this time (runtime) we can inspect the closure for States and make sure there's no swap.
Proper fix
The real fix is to figure how to describe to DaCe our memory.
np.ndarray and FloatField do not have this issue, therefore there's a code path for Quantity that is reproducible.
State is a bigger fish, being a complex structure. Can we describe it in the __descriptor__ as dace.structure while making sure the __descriptor__ is called on object rather than the class ?
Reproducer
The unit tests in tests.dsl.orchestration.test_call have been updated to cover all the above, with an xfail on the DSL type hint that would have to be removed and made to work OR error cleanly.
Description
TL;DR
Top-level memory cached when using
dace.compiletimeandQuantityandStateare both using it as a way to allow type hinting.DaCe requires a
__descriptor__to describe the memory to it. When using a type hint, the descriptor is called on theclassrather than theobject. This requires the size to be static - not our use case. To alleviate this, we useddace.compiletimeto delay the description to runtime BUT what we discovered is that during SDFG caching it feeds intoconstant_args, which feeds into theclosure, which doesn't get recomputed. Therefore, if the parameters changes, we still call to the first called/bindedQuantity/State.Impact
There's a limited impact to our usage because we follow the "allocate once, never reallocate" paradigm for all of our interfaces (GEOS or Pace), which means we don't see parameters swap at the top-level.
BUT, if we did, it would lead to a very difficult to debug situation and must be addressed urgently.
Mitigation
We can use the
orchestrate.LazyMethodoverride (where we overridedace.compiletime) to banish anyQuantity/Localtype hinting.We can't do this for
Stateas it is already thoroughly used in the wild.For state, we can look at
orchestration._call_sdfg, which handled caching and marshalling of the arguments. At this time (runtime) we can inspect the closure forStates and make sure there's no swap.Proper fix
The real fix is to figure how to describe to DaCe our memory.
np.ndarrayandFloatFielddo not have this issue, therefore there's a code path forQuantitythat is reproducible.Stateis a bigger fish, being a complex structure. Can we describe it in the__descriptor__asdace.structurewhile making sure the__descriptor__is called onobjectrather than theclass?Reproducer
The unit tests in
tests.dsl.orchestration.test_callhave been updated to cover all the above, with anxfailon the DSL type hint that would have to be removed and made to work OR error cleanly.