Secret values are exposed in tasks

[stoney95]

Describe the problem

I want to install dependencies when creating a project from a copier template. The dependencies are hosted on a private pypi mirror. Credentials are required to install dependencies from there. I run the installation via a python script. But we could also assume that we want to set an environment variable etc.

So, in the template I ask for the username and password. I mark the password as secret. I did not find any proper documentation of how to use this password securely.

In the documentation for tasks the template variables are inserted via {{ var_name }}. Using this exposes the secret variable.

Template

username:
  type: str

password:
  type: str
  secret: true
  default: "1234"

_tasks:
  - "python my_script.py --username={{ username }} --password={{ password }}"

To Reproduce

1. Run copier copy --trust with the template from above
2. Answer
   1. username="user"
   2. password="secret1234"
3. Output will be

> Running task 1 of 1: python my_script.py --username=user --password=secret1234

With this the password is exposed in the terminal

Logs

No response

Expected behavior

There a multiple options:

1. I can surpress the output of tasks, e.g only show Running task 1 of 1.
2. The output of the tasks detects the password to be a secret and uses asteriks instead, e.g. Running task 1 of 1: python my_script.py --username=user --password=***
3. There is another way to access information provided via a secret template variable, e.g. an environment variable.

I prefer option 2, but am also happy with any other option

Screenshots/screencasts/logs

No response

Operating system

Windows

Operating system distribution and version

Windows 11

Copier version

9.4.1

Python version

3.11

Installation method

pip+pypi

Additional context

No response

[sisp]

Thanks for reporting this problem, @stoney95! 🙏 This is an interesting observation, and I like your suggested solution 2. This would be similar to how, e.g., GitLab CI redacts the values of masked CI variables in the CI job log. The main challenge seems to be distinguishing a secret value from another substring in the task's stdout/stderr that coincides with the the secret value.

[stoney95]

@sisp, thanks for reacting so quickly! Does this also mean that there is currently no way built into copier to handle secrets?

[sisp]

If you mean "redact secrets in stdout/stderr", then Copier has no built-in way to handle it right now. In contrast to a regular question, the answer of a secret question is not recorded in the answers file (typically .copier-answers.yml), so when you update a project using copier update, you need to provide the secret again.

[stoney95]

Thanks for pointing out current behavior of a secret question. This is also clearly documented 👍

My question is rather how answers to secret questions are currently supposed to be accessed? The only way I can currently imagine is to use {{ secret_name }}, where the secret value is required. This will always expose the password. So, I was wondering if there is a mechanism to access the value of a secret question in a safe way

[sisp]

That's correct, secret values are accessed like any other values. Why do you think this is suboptimal? When you render a Git-ignored file with a secret value, it is not printed or versioned. When you use it as a CLI flag value in a task, then indeed it is not redacted, which I think we should try to fix.

Another approach to "handling" secret values might be assigning them to environment variables passed to tasks such that either a task program uses the environment variable internally or it is used to pass a CLI flag value:

_tasks:
  - command: "python my_script.py --password $PASSWORD"
    env: # or `environment` since we tend to avoid abbreviations
      PASSWORD: "{{ password }}"

This syntax is inspired by Task's environment variables. A downside of this approach is that the use of environment variables has different syntax on Unix and Windows.

Is this a feature you'd like Copier to offer?

[yajo]

Have you considered rendering the secrets into a git-ignored file and then using that file in your script? Example:

1. Create a .gitignore entry to ignore the secrets file:

   secrets.json
   

2. Render the secrets into the secrets.json.jinja file using a Jinja template:

   {
     "username": "{{ username }}",
     "password": "{{ password }}"
   }

3. Use the secrets in your script:

   import json
   
   with open('secrets.json') as f:
       secrets = json.load(f)
   
   username = secrets['username']
   password = secrets['password']
   # Use the username and password as needed

4. Call that script from tasks:

   _tasks:
     - "python read_secrets.py"

This way, your secrets are stored securely in a file that is not tracked by git, and your script can safely read them without exposing them in the terminal.

[stoney95]

@sisp, I also like your suggestion. This could be an alternative to redacting secrets in the tasks output.

@yajo, thanks for sharing your approach 👍

[sisp]

@yajo's suggestion is now available as of v9.5.0 via the new external data feature (#1880). Is this a solution to your problem, @stoney95?

[yajo]

IMHO it should be, so closing optimistically.

[pauloneves]

The proposed solution is a nice hack, but has some security problems, like the possibility that a password lingers around in a .secrets.yml file when the tasks fail. Even if I put a "rm .secrets.yml" as a task, it can fail.

There were other very nice solutions like the -env or the not displaying of secrets.