Re-evaluate need for caching examples

[guineveresaenger]

We cache examples both locally and in GH Actions. This cache is meant to persist across builds to speed up subsequent builds.

Currently the cache key invalidation is not sufficient for all use cases; specifically an update to the converter version will not invalidate the examples cache in CI.

Additionally, local caches sometimes do not get invalidated either: #2323

It would be useful to determine whether we still need to cache generated examples to speed up our builds.

[blampe]

I did a performance spike around the examples cache. Some findings below.

We spawn a CLI process in order to generate examples, the rationale being that this reduces dependencies.

pulumi-terraform-bridge/pkg/tfgen/convert_cli.go

Lines 57 to 60 in 961efe5

	// Pulumi CLI now supprts a handy `pulumi convert` command. This file implements integrating with
	// this command for the purposes of initial conversion of Terraform examples into PCL language. This
	// integration is preferable to linking the functionality in as it allows bridged providers to not
	// build-depend on the TF converter.

However I believe this is incorrect and unnecessary. Our provider binaries already depend on tfgen. The impact of this is probably overstated. Refactoring DocEdits into a separate package shaved only about 10MB off the AWS binary, which tells me Go's tree shaking is already doing a pretty good job removing unnecessary dependencies. I also say this is unnecessary because our tfgen binaries should be free to take on additional dependencies without impacting our providers.

Similarly, when I replaced the logic to exec the CLI with logic to instead run a long-lived server in-memory, the binary size only increased by a few megabytes. Performance-wise that sped up example generation by ~6x according to my benchmark:

Before: BenchmarkConvertViaPulumiCLI-10    	       6	 177623444 ns/op	   38805 B/op	     255 allocs/op
After:  BenchmarkConvertViaPulumiCLI-10    	      37	  31241613 ns/op	  401332 B/op	    2368 allocs/op

With a cold cache AWS examples currently take ~4m20s to generate, and with this change to use an in-memory server that takes it down to ~3m50s. So 6x faster in theory but in practice I think we're still very slow because we wait to gather all examples until we start processing them -- a side effect of consuming the CLI like an API. This could probably get faster by generating these examples on the fly.

So with my changes (to drop the CLI usage) make tfgen takes ~6m20s on a cold examples cache, ~3m50s of which is examples. When the cache is warm make tfgen takes ~2m42s, so the cache saves us ~3m38s in subsequent builds. That's not nothing but it's also not a big deal when we're talking about workflows that take hours to run.

At minimum, we should:

• Try to fix the cache key, since that's trivial. If it continues to give us trouble I agree it would be worth removing the caching functionality for the sake of having one less thing to maintain.
• If we keep this cache and would like to optimize it further we could continue my branch to replace the CLI in favor of a proper server, but I think there are bigger contributors to build time to look into first.

[corymhall]

@blampe I would be curious to see the performance numbers when running in CI. It doesn't look like a big deal when running locally because it only takes a couple of minutes, but it aws takes 30m in CI to run make schema.

[blampe]

@blampe I would be curious to see the performance numbers when running in CI. It doesn't look like a big deal when running locally because it only takes a couple of minutes, but it aws takes 30m in CI to run make schema.

I suspect most of this is actually time spent linking, but I need to test that theory. Locally, ldflags are static so we still leverage the build cache but on CI we trigger a full re-link on every PR which is expensive.