Support passing env vars into docker_environments from user's shell

[psontag]

Is your feature request related to a problem? Please describe.

When using docker_environments for testing I have some dynamic configuration that needs to be passed into the docker_environment via env vars. Currently that is not possible.

[test]
extra_env_vars = [
  "ENV_VAR",
]

The env var ENV_VAR will be read from the docker image and not from my shell.
See the comment here for more details.

Describe the solution you'd like
It should be possible to pass env vars from the user's shell into docker_environments.
This could either be done via a new field on the docker_environment target or by adapting the current fields to do an additional lookup of the env vars in the user's shell.

[andreaimprovised]

Also ran into this recently with python_test() not picking up shell variables. Scratched my head for awhile trying to figure out what was going on. [docker].run_args with --env also had no effect if I recall correctly.

We use--surprise--docker-compose to stand up a bunch of databases, localstack, etc... for our development and testing environment. Then we define a bunch shell variables with URLs pointing to those services. Then we have our tests look for those shell variables at runtime. There are other variables that are set, like in CI, that would be important to pass through as well beyond the docker-compose related variables.

I think this is all fairly common practice. There might be other ways to communicate this information. I can think of two:

1. A physical file (e.g. .env) that could be generated and then be loaded at runtime by the application manually. But I can't think of a way to have pants dynamically mount or cp the file in there at runtime. Maybe a shell_command() to generate the file, a file()/resource()on the file, and then a dependency on that file across all test targets.
2. Using %(env.VAR) / env() to pull the variables dynamically as this persons did on python_tests() targets. But perhaps it would work on docker_environment.test_extra_env_vars or docker_environment.subprocess_environment_env_vars.

   docker_environment(
       test_extra_env_vars=[
           "VAR1=%(env.VAR1)",
           "VAR2=%(env.VAR2)",
           "VAR3=%(env.VAR3)",
       ]
   )

   So, essentially duplicating what would be in [test].extra_env_vars.

Maybe a recommendation on how to use docker_environment() in a docker-compose setting is warranted if the maintainers have a strategy for this already. Would be an excellent doc page. It would be really slick to have everything all hooked up with pants.backend.shell and shell_command() for running docker-compose.

This issue, combined with --test-debug not working with docker_environment() yet (#17182), is why I dropped docker_environment() from our test and build strategy. Hoping to come back to docker_environment() someday.

[gauthamnair]

#20122 demonstrates a way to get the shell args through for the narrow case of test.extra_env_vars.

It does so by:

• grab the environment present locally
• merge it with the environment vars in the target environment (say docker). Prefer those in the target if they are present.

There seem to be a few dimensions to discuss, but at least:

• Should we generally allow the local environment to be combined with whatever is in the running environment? test.extra_env_vars is just one way that we read from the environment, but it could be needed in other places too. For example, within an individual python_test target via its extra_env_vars moved field.

[andreaimprovised]

thanks gautham!

the preference order makes sense to me.

i would agree that consistency is probably important in how docker_environment() allows local environment shell variables to be injected.

i think, in addition to pants test, we would also expect to be able to inject the same local environment variables to pants run <python_source_target> for adhoc code execution against the same docker-compose resources my tests run against. and potentially even against arbitrary resources in my cloud environment. my alternative is to package the code in a docker image and then run the docker image, as pants allows for injecting local environment variables into pants run <docker_target>.

so, perhaps this is an opportunity for a broader change?

[gauthamnair]

Updated #20122 to be a blunt change where the local environment variable values are available to pants for setting up sandboxes in non-local environments

[cedric-fauth]

Hi, any updates on this one?