1.20 new consistency checks issue, for changing shadow variables, due to fact changes rather that genuine variable changes

[christian8096]

Upgrading to 1.20 we ran into an issue: The new checks in 1.20 flags a shadow variable (start time) changed without any of its genuine planning variables was changed.

The error states “ shadow variable (meeting.startIndexInGrains)'s corrupted value (108) changed to uncorrupted value (144) after all variable listeners were triggered without changes to the genuine variables.”

This is correct and as intended! As we use a sort of chained pattern where an entity also depends on a mix of previous entities shadow variable ( start time ) and facts ( unavailable time periods) that can both affect the shadow variable of a later meeting as well as its genuine variables.

The entity does not reference all these previous entities and facts itself, it is kept in a parallel data model which is updated by a custom variable listener that is called when any of it genuine variables are changed, but also called whenever a fact of the “unavailable time” is added or removed. Hence it works as we would expect, but not as Timefold would expect.

The cascading changes of shadow variables from meeting entities does not seem to trigger the error, it is when a fact is changed, that cause the meetings to change by triggering the custom variable listener, which again moves the meetings, without any genuine variables was changes.

Can this be remedied without moving all the affecting facts into the Meeting.class or can the shadow variable annotation have source variables that point to the ProblemFactCollections of the PlanningSolution in someway or is there a another solution.

Timefold already detects the fact change and trigger all listeners, So it seems that the result is consistent even though we get the error ?

 

[triceo]

Hello @christian8096,

without having access to your codebase, it is difficult for me to point to any particular problem. Instead, I'll just list some basic guidelines:

• If the value of a field changes while the solver is running, it is a variable of an entity, not a fact; it needs to be declared as such.
• If a shadow variable depends on any other variable, that dependency has to be declared. If it depends on multiple variables, all dependencies need to be declared. Transitive dependencies (depending on variables which themselves depend on other variables) do not need to be declared.
• Facts can only be changed using ProblemChange, therefore while the solver is stopped. Such facts do not need to be variables, and any code can depend on them freely.

I believe that, if you follow all of these rules, the exception will go away. If you experience the contrary, please provide code that reproduces this behavior - we have not yet seen any instances where the assertions would be misfiring.

You can of course also get rid of the assertion by using the NO_ASSERT environment mode, but that leaves you open to future score corruptions and as such is not recommended.

[christian8096]

We do use problemchanges and call the problemChangeDirector.addProblemFact(..). Our meetings are chained, not using Timefolds chain mode. But any previous meeting can affect any subsequent meeting, and will trigger a cascade of shadowvariablechanges, which works. But new facts of 'blocked time', can also be added and they can also move subsequent meetings forward, hence the fact change will also trigger a cascading shadowvariable change. the fact are part of the planningsolution but not reference by the meetings itself and there can be many of them. its not the actual value of these fact that determine the shadow variable directly but the combined set that is used. Its not that Timefold misfires, its more that these fact changes should actually trigger shadowvariable changes by design, and it seems that Timefold now prohibits this.

[triceo]

It's a bit hard for me to visualize, but any problem change needs to update a variable on an entity. Any such update will trigger updates to all dependent shadow variables, and then also transitively.

Therefore the only way how something could change without changing shadow variables is if that something is a problem fact, and the shadow variables depends on that problem fact. The shadow variable in this situation depends on a problem constant, but the constant changes, and nobody told the variable.

This would certainly be a situation that the solver doesn't expect - nor can it detect this situation. The only way to support this situation would be to never trigger these assertions at all. The solution on your side would be to trigger a variable change synthetically and, through that, get all the correct shadow variable updates.

[triceo]

Thinking about this more: if anything on an entity changes during a problem change, an event should be triggered by the user. So even if you're changing just some constant somewhere, the entity still needs to be updated, and that's how you get consistency.