Only call terraform init at runtime when remote backend is configured

[carolynvs]

I noticed that we call terraform init at buildtime which is great. I added #79 to make that perform even better.
But since we have that, is there a reason that we call init again when when run the bundle? I'm noticing that it's causing terraform to re-download and install providers that were already installed (or at least that's what the output implies).

Initializing Terraform...
/usr/bin/terraform terraform init

Initializing the backend...

Initializing provider plugins...
- Reusing previous version of hashicorp/azurerm from the dependency lock file
- Reusing previous version of hashicorp/random from the dependency lock file
- Installing hashicorp/azurerm v2.72.0...
- Installed hashicorp/azurerm v2.72.0 (signed by HashiCorp)
- Installing hashicorp/random v3.1.0...
- Installed hashicorp/random v3.1.0 (signed by HashiCorp)

Terraform has made some changes to the provider dependency selections recorded
in the .terraform.lock.hcl file. Review those changes and commit them to your
version control system if they represent changes you intended to make.

Terraform has been successfully initialized!

Since we only need the terraform init call at runtime when a remote backend is configured (as opposed to the bundle uses porter's state functionality), we should add an if statement around that init call so that we only do it when remote backend configuration is present.

[vdice]

Maybe so, definitely worth trying for the reasons stated above. I do notice the runtime init consumes the optional backend config (e.g. I want my terraform state stored in an Azure Keyvault backend, etc.) But as long as we can support that same thing during the build-time init call, we should be good to remove the runtime call... I htink.

[carolynvs]

Oh, okay I wasn't thinking that init updated the tfstate and so should be persisted remotely. 🤔

If anyone is interested in playing around with this and figuring out if it's possible, that would be great!

[vdice]

I think my wording may have been unclear; currently at runtime the backend configuration (if supplied) is used by terraform init to set things up for storing tfstate in said backend (whereas, the init call at buildtime runs with -backend=false.)

Here's the current code that does so: https://github.com/getporter/terraform-mixin/blob/main/pkg/terraform/init.go#L12-L20.

And here's an example bundle that makes use of a backend config for remote tfstate storage: https://github.com/getporter/terraform-mixin/blob/main/examples/azure-keyvault/porter.yaml

This was done b/c Porter's de facto way of sending in creds (e.g. in this case, backend config creds) is just-in-time/at runtime -- but if we can figure out how to support sending in creds at buildtime for the init call, then we'd be looking close! Buildkit? Or new ways of doing so w/ Docker? Haven't looked in a bit.

[carolynvs]

Since the person building the bundle is a different person than who runs the bundle, passing in creds at buildtime for anything other than retrieving private resources to store in the invocation image for all consumers of the bundle to use, such as a private git repo or private terraform module, wouldn't be correct.

If the person who is running the bundle wants to use remote state, they will need to run porter init with their credentials.

So is the only reason we call init at runtime is to connect to the remote backend? Could we skip that call if they aren't using a remote backend?

[vdice]

Could we skip that call if they aren't using a remote backend?

Yup, I believe so!

[carolynvs]

I'll update the issue with our new solution.

[jk-f5]

I'm currently working around a related issue when using a private Terraform module registry.

Terraform requires a ~/.terraform.d/credentials.tfrc.json to authenticate so in order to provide this I'm doing a multistage build to avoid exposing that file in a layer.

This is how I'm doing it currently:

FROM debian:stretch-slim as init

ARG BUNDLE_DIR

COPY .credentials.tfrc.json /root/.terraform.d/credentials.tfrc.json

RUN apt-get update \
 && apt-get install -y ca-certificates

COPY . $BUNDLE_DIR

# PORTER_MIXINS

RUN rm -f $BUNDLE_DIR/.credentials.tfrc.json /root/.terraform.d/credentials.tfrc.json

# -----------------------------------------------------------------------------
FROM debian:stretch-slim

ARG BUNDLE_DIR

RUN apt-get update \
 && apt-get install -y ca-certificates wget unzip \
 && wget https://releases.hashicorp.com/terraform/1.0.4/terraform_1.0.4_linux_amd64.zip \
 && unzip terraform_1.0.4_linux_amd64.zip -d /usr/local/bin \
 && rm terraform_1.0.4_linux_amd64.zip

COPY --from=init $BUNDLE_DIR/ $BUNDLE_DIR/

And of course the # PORTER_MIXINS line above is replaced with the dockerlines from the terraform mixin:

RUN apt-get update && apt-get install -y wget unzip && \
 wget https://releases.hashicorp.com/terraform/1.0.4/terraform_1.0.4_linux_amd64.zip && \
 unzip terraform_1.0.4_linux_amd64.zip -d /usr/bin && \
 rm terraform_1.0.4_linux_amd64.zip
COPY terraform/ $BUNDLE_DIR/terraform/
RUN cd $BUNDLE_DIR/terraform && terraform init -backend=false

I'm not sure what the best solution is to this but some thoughts are: It might be nice to be able to override the dockerfile entry for a given mixin. Or maybe be able to prevent mixins from replacing or appending to the dockerfile entirely and allowing users have complete control over their Dockerfile.tmpl

[carolynvs]

@jk-f5 The most recent release of Porter, https://github.com/getporter/porter/releases/tag/v1.0.0-alpha.18, we support the docker --build-args and --secret flags during porter build. So now you can use that to pass in extra or secret information and use it in your custom dockerfile. I think that should give you everything you need but if you run into any further gaps, just let me know!