Tests should be able to use multiple versions of the provider

[corymhall]

Hello!

• Vote on this issue by adding a 👍 reaction
• If you want to implement this feature, comment to let us know (we'll work with you on design, scheduling, etc.)

Issue details

This request is specifically relevant for upgrade tests where you want to deploy with an old version of the provider and then deploy with the local version of the provider. Currently this does not actually test the upgrade scenario from a users perspective. For example if a user were to perform an upgrade it might look something like this:

v6 => v7 provider upgrade

1. pulumi up --refresh
2. Read call using v6
3. Update call using v7

Because our tests use the local provider by overriding the PATH what we end up getting in the test is something like this:

1. pulumi up --refresh
2. Read call using v7
3. Update call using v7

This can lead us to miss issues that occur.

Affected area/feature