Pipeline

A Pipeline class should help users (direct or via Workload) to run passes and schedules on on or more payload modules.

Single Pipeline vs Separate

Transform schedules don't need a Pass Manager to run on a payload module (they can apply directly), but when running passes (through apply_registered_pass) they create one Pass Manager per pass. While a single pass inside a schedule would be low cost and simpler than exiting the scheduler to a whole new pass manager and back into the schedule, a long streak of passes (for example, lowering to LLVM) would be much simpler to exit into a traditional pass manager. Furthermore, by that point, the IR would be so different that the original match may not even exist anymore, so any continuity argument would be hard to defend.

Separate Schedules from Passes

Here we separate the two, but now it's up to the user to know how to apply them in the right order and actually propagate that code down their own compilers.

Temporal interpolation:

  sched = Schedule(ir.context)
  sched.add_schedule(...)
  sched.apply(module)

  pipeline = Pipeline(ir.context)
  pipeline.add_passes(...)
  pipeline.apply(module)

  sched2 = Schedule(ir.context)
  sched2.add_schedule(...)
  sched2.apply(module)

  pipeline2 = Pipeline(ir.context)
  pipeline2.add_passes(...)
  pipeline2.apply(module)

Composition:

  sched = Schedule(ir.context)
  sched.add_schedule(...)

  pipeline = Pipeline(ir.context)
  pipeline.add_passes(...)

  sched2 = Schedule(ir.context)
  sched2.add_schedule(...)

  pipeline2 = Pipeline(ir.context)
  pipeline2.add_passes(...)

  # Identical implementation details
  sched.apply(pipeline.apply(sched2.apply(pipeline2.apply(module))))

  # If we have something like ComposedPipeline which has an `add` method
  # we could create new pipelines by adding the pieces
  bottom = pipeline + sched2 + pipeline2
  top = sched
  full_pipeline = top + bottom
  full_pipeline.apply(module)

  # ComposedPipeline could support other interfaces as well
  full_pipeline = ComposedPipeline()
  full_pipeline.add_pipeline(pipeline2)
  ...
  full_pipeline = ComposedPipeline([pipeline2, ...])

Can we do this?

  sched = Schedule(ir.context)
  pipeline = Pipeline(ir.context)

  sched1 = sched.add_schedule(...)
  bundle1 = pipeline.add_passes(...)
  # This is a different module altogether
  sched2 = sched.add_schedule(...)
  # Unless we use different PMs for the pipeline this is not possible
  bundle2 = pipeline.add_passes(...)

  sched1.apply(bundle1.apply(sched2.apply(bundle.apply(module))))

Single Pipeline

Here we create a single pipeline object that stores the information on the application order (first-come basis) and then a single apply method runs the whole pipeline on a particular payload. This moves the ordering problem to pipeline creation time versus pipeline application time in the above case.

  pipeline = Pipeline(ir.context)

  pipeline.add_schedule(...) # No PM needed so far
  pipeline.add_passes(...)   # First use, PM created
  pipeline.add_schedule(...) # PM exists, but unused here
  pipeline.add_passes(...)   # Can we reuse the PM created above?

  # Run the first schedule, then the first bundle of passes,
  # then the second schedule, then the second bundle, etc.
  pipeline.run(module)

Combination of Passes and Schedules

Passes in Schedules

We should allow passes in schedules (via apply_registered_pass) but we should discourage users from building a whole pass pipeline inside schedules. Given the scheduler creates a new PM per pass application, this is wasteful and can easily be run as a pass bundle.

Schedules in Passes

Similarly, the transform interpreter pass has impedance mismatches with the pass manager. For example, a schedule inside the payload module will remain there after application, and the pass manager needs to remove them with a further cleanup pass. With straight application of schedules into payload modules, this is not an issue.