Allow to specify servers / user in resource instead of provider

[jvanhees]

What motivated this proposal?

First off, I just started working with Terraform, so I'm unsure if this is even feasible.

We are building a single-tenant architecture where we are running separate a separate NATS instance for each client. This NATS instance has a JetStream stream which is configured using Terraform.

We've added modules for each of our components in our system, including a NATS module that sets up a GCP Compute Engine with the NATS Docker image, adds networking, firewalls, etc etc. Ideally, this module also configures the JetStream Stream, but because of the JetStream provider we cannot use depends_on to only configure JetStream once the Compute Engine is running. We also don't know the instance IP address because (when we create a new environment using the module) there is no instance created yet.

What is the proposed change?

I would like to be able to specify the user and servers within the resource, which in turn overrides the values configured in the provider. This is also how Google Cloud Platform handles it with the project value (you can configure it in the provider, or set it for each resource individually which overrides the value inherited from the provider).

Who benefits from this change?

It would be easier to configure JetStream in Terraform modules, especially when Terraform also configures the instances on which JetStream runs.

What alternatives have you evaluated?

When using the provider, Terraform considers the module a "legacy module", with this error:

Error: Module is incompatible with count, for_each, and depends_on
│ ...
│ 
│ The module at module.nats is a legacy module which contains its own local provider configurations, and so calls to it may not use the count, for_each, or depends_on
│ arguments.
│ 
│ If you also control the module "../../components/nats", consider updating this module to instead expect provider configurations to be passed by its caller.

I cannot pass a provider because the instance IP is still unkown.

[ripienaar]

I have not seen any terraform providers that would work in this way with per resource accounts. All the ones I saw is you configure the provider and all from there uses that.

Perhaps if you find some example of such a provider I can see what the patterns are.

As for legacy, I guess we need to rewrite on their latest API. Given recent Hashicorp moves I am disinclined to do that though.

[jvanhees]

One example from Google is the project argument on their resources: https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_instance#project

I could use the google provider to set my project for all GCP related resources like this (https://registry.terraform.io/providers/hashicorp/google/latest/docs/guides/provider_reference - note the optional project argument):

provider "google" {
  project = "my-google-project-id"
}

But I can also use the project argument on the google_compute_instance resource to override it, or as a substitute:

resource "google_compute_instance" "default" {
  ...
  project = "my-google-project-id"
}

As I mentioned, I'm by no means an expert on Terraform so I cannot say if this is a standard pattern within the Terraform ecosystem. As of now, I see now way of creating our entire infrastructure including JetStream within one Terraform workspace.

I understand your hesitation regarding the developments around Terraform, however I would like to mention that something like OpenTofu is based on Terraform 1.5 which uses (correct me if I'm wrong) the new module structure.

Ps. I am not bound to using Terraform for JetStream, but we didn't see any way of setting up a JetStream Stream in the NATS configuration file. I initially used a nats-box container with script that would configure the stream at deploy time, but I believe something like Terraform or a config file is much neater. But perhaps I don't have a full understanding of NATS best practices :) .

[ripienaar]

Yeah but isn’t it projects in the same general account? What you are describing would be different accounts. So us that’s like an entire different isolation

not sure, will think about it, but doesn’t sound like a pattern that align with our use

[jvanhees]

There is one more isolation level above projects in Google Cloud, which would be organisations, but I don't think that any of the resources really care about that. The resources just have to know in what project they need to be placed, just like the JetStream config has to know on which server it needs to be applied.

What other options would I have if I want to configure multiple JetStream instances with different configs? Am I limited to different Terraform working directories?

[ripienaar]

Seems to me the accepted terraform way is to have many providers and then pick the provider per resource

https://developer.hashicorp.com/terraform/language/meta-arguments/resource-provider