Skip to content

Implement systematic perturber #123

@MattWindsor91

Description

@MattWindsor91

The perturber (part of the test that takes a template plan and produces a plan with a sampled is currently entirely random.

John and Ally suggested that it would be nice to have a systematic mode whereby the tester progressively enumerates every possible perturbance of the plan.

If Go had generators, it would look something like this:

// on one machine
for _, s := range samplesOf(subjects) {
  // probably *in parallel* for each compiler, making sure to handle different compiler configuration surfaces
  for _, c := range compilers {
    for _, m := range c.marches {
      for _, o := range c.opts {
        yield()
      }
    }
  }
} 

The way I see this working is something like the following:

  • Abstract the current randomised sampler out into an interface, and create a 'window-based' sampler that returns subjects 0-n, n+1-2n, ..., and wraps over at the end ensuring the sample size is the same. We might be able to ship this separately from the below.
  • Abstract the main implementations of the perturber's functionality into a 'policy' interface, with the current randomised policy becoming the default.
  • Make a systematic policy that slots into the same interface, but holds a state machine that implements for each machine the loop suggested above.
  • (Bonus) Make a mechanism for the perturber and/or sampler to signal when it has run out of permutations, and make it possible for the tester to finish when that occurs.

If we play our cards right, we might be able to make it possible to replace setc with perturb, by making the perturber able to admit a policy that just sets everything to user-specified levels.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestmoonshotVery long-term suggestion

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions