Re-running counterexamples ignores changes to generators

[evnu]

#10 added re-running failing counterexamples. If the fix to a failing property is to change the generators, the counterexample will keep on failing until the persisted file is deleted.

Consider this obviously failing example:

property "failing" do
  forall x <- nat() do
    is_float(x)
  end
end

Now, we fix this example by changing the generator:

property "failing" do
  forall x <- float() do
    is_float(x)
  end
end

But due to how counterexamples currently work, we still receive an error:

  1) property failing (PropTest)
     test/prop_test.exs:5
     Property Elixir.PropTest.property failing() failed. Counter-Example is:
     [0]
     
     code: nil
     stacktrace:
       (propcheck) lib/properties.ex:107: PropCheck.Properties.handle_check_results/3
       test/prop_test.exs:5: (test)

[evnu]

Would it be possible to peek into the block passed into PropCheck and determine if the generators changed since the counter example was stored? I don't know how stable the internal property definitions within PropEr are. Maybe :erlang.phash2(p) is enough?

[evnu]

@alfert Do you have an idea how this could be fixed?

[alfert]

Hmm, do you mean that we store a hash of the generator definition (= of the AST) together with the counter example? So, we could check whether this has changed and in that case rerun the property from scratch (i.e. without using the counter example)? I assume that this should work.

[evnu]

Yes, that's what I meant. I was not yet able to come up with a way to retrieve such a hash yet, though.

[alfert]

It depends on what the hash is about. If the generator is "only" the part of forall up to do, this should do-able, since the macro gets the AST for the generator and the do-part as argument. Putting this into an hash-function is easy. After that we only have to use it properly. i.e. inject it into a function that checks whether the hash differs etc. What will not work is to detect whether the definition of nat() has changed (according to your example above).

[evnu]

What will not work is to detect whether the definition of nat() has changed (according to your example above).

For "simple" generators, it might be possible to remember the seed value (if there is such a thing) when a test failed, use that to produce an extra value and store that and the seed along with the counter example. Now, before running the counter example, we could try to regenerate this stored value. If that succeeds, we know that the generator did not change. If we fail to do that, either the generator changed, or it was not deterministic enough. In either case, we should try to run the property without the counter example. @alfert Do you know if PropEr uses a seed when evaluating generators? If so, is it possible to influence this?

[evnu]

@alfert I think this can be closed. I can't think of a usable solution to solve this beside the additional informational output from #52.

[evnu]

For future reference, this pull request in PropEr seems related (setting a seed for reproducibility).

[alfert]

As proposed in May, i.e. 6 month ago, I close this item.