Make the pipeline compatible with shots decoupling

[JerryChen97]

Context:
We want to:

1. qml.set_shots or partial(set_shots, shots=...) works (the latter is the original requirement); ideally we wish it works for all interfaces and all shots/shot vec
2. previous shots setting should still work as was, until someday we decide to deprecate (ideally we wish they don't even talk with each other)

Description of the Change:

1. transforms realm is further divided (tbd): anything applied to the qnode is a user transform; (placeholder for outer); (placeholder for inner)
2. the logic relationship between the three layers of transformers:

user_transform(
  `_setup_transform_program` would decide 
  outer_transform(
    inner_transform -> presented to `run` and do whatever defined there
  ) // outer_post_processing
) // user_post_processing

Benefits:
Further clearance and decoupling for pipeline!

Possible Drawbacks:

Related GitHub Issues:
[SC-90145]

[JerryChen97]

Some technical details about what this will look like.

Right now, our execution pipeline has two transform program components. The "outer" and the "inner". The outer is run before we hit the ML boundary, while the inner is run after. The outer always contains the user transforms. If we are using a device derivative, the outer also contains device preprocessing. If we are not using device derivatives, the inner contains device preprocessing. The inner transform program also contains caching and conversion to numpy.

We now need to break this into three transform program stages. The user, the outer device, and the inner.

Right now we resolve the execution config and setup the transform programs based on the initial tape. We need to instead run the user transforms, then resolve the execution pipeline, then setup the inner and outer transform programs.

This change will cause some minor changes in tests, but I've double checked, and they are are all acceptable differences. This will also solve some long standing bugs, and allow us to get rid of this patch:

pennylane/pennylane/workflow/resolution.py

Line 283 in 3229b05

if (

Once we update the workflow, we will also need to update construct_batch and get_transform_program. We will now always be applying final transforms at the end of the user transform program, and before device preprocessing.

[albi3ro]

Either in test_set_shots to test_qnode, can we add some tests where we use the device tracker to make sure we are choosing the best diff method appropriately?

For example, starting with a device with shots, the overriding to shots=None, and double check we only have one execution when taking the gradient?

Or starting with shots=None, overriding with finite shots, and checking we have 1+2*num_parameters executions when taking the derivative.

[JerryChen97]

Either in test_set_shots to test_qnode, can we add some tests where we use the device tracker to make sure we are choosing the best diff method appropriately?

For example, starting with a device with shots, the overriding to shots=None, and double check we only have one execution when taking the gradient?

Or starting with shots=None, overriding with finite shots, and checking we have 1+2*num_parameters executions when taking the derivative.

Done. Added in test_set_shots

[albi3ro]

Thanks for your patience 👍 Looks good :)

[andrijapau]

Very satisfying, nice job! 🚀