Conditional exclude fails on update for new question

[mschoettle]

Describe the problem

I added a new question to our template and added a conditional (templated) _exclude to exclude a file if the question is answered with no.

When calling copier update I get the following error:

Traceback (most recent call last):
  File "/home/<user>/.cache/uv/archive-v0/LPweG3ZFVAPurgjpe8gG_/lib/python3.13/site-packages/copier/_main.py", line 1088, in run_copy
    self._render_template()
    ~~~~~~~~~~~~~~~~~~~~~^^
  File "/home/<user>/.cache/uv/archive-v0/LPweG3ZFVAPurgjpe8gG_/lib/python3.13/site-packages/copier/_main.py", line 757, in _render_template
    if self.match_exclude(dst_relpath):
       ^^^^^^^^^^^^^^^^^^
  File "/home/<user>/.local/share/uv/python/cpython-3.13.2-linux-x86_64-gnu/lib/python3.13/functools.py", line 1042, in __get__
    val = self.func(instance)
  File "/home/<user>/.cache/uv/archive-v0/LPweG3ZFVAPurgjpe8gG_/lib/python3.13/site-packages/copier/_main.py", line 710, in match_exclude
    return self._path_matcher(
           ~~~~~~~~~~~~~~~~~~^
        self._render_string(exclusion, extra_context=extra_context)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        for exclusion in self.all_exclusions
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/<user>/.cache/uv/archive-v0/LPweG3ZFVAPurgjpe8gG_/lib/python3.13/site-packages/copier/_main.py", line 447, in _path_matcher
    spec = PathSpec.from_lines(_pathspec_pattern, normalized_patterns)
  File "/home/<user>/.cache/uv/archive-v0/LPweG3ZFVAPurgjpe8gG_/lib/python3.13/site-packages/pathspec/pathspec.py", line 260, in from_lines
    patterns = [pattern_factory(line) for line in lines if line]
                                                  ^^^^^
  File "/home/<user>/.cache/uv/archive-v0/LPweG3ZFVAPurgjpe8gG_/lib/python3.13/site-packages/copier/_main.py", line 446, in <genexpr>
    normalized_patterns = (normalize("NFD", pattern) for pattern in patterns)
                                                                    ^^^^^^^^
  File "/home/<user>/.cache/uv/archive-v0/LPweG3ZFVAPurgjpe8gG_/lib/python3.13/site-packages/copier/_main.py", line 711, in <genexpr>
    self._render_string(exclusion, extra_context=extra_context)
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/<user>/.cache/uv/archive-v0/LPweG3ZFVAPurgjpe8gG_/lib/python3.13/site-packages/copier/_main.py", line 995, in _render_string
    return tpl.render(**self._render_context(), **(extra_context or {}))
           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/<user>/.cache/uv/archive-v0/LPweG3ZFVAPurgjpe8gG_/lib/python3.13/site-packages/jinja2/environment.py", line 1295, in render
    self.environment.handle_exception()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/home/<user>/.cache/uv/archive-v0/LPweG3ZFVAPurgjpe8gG_/lib/python3.13/site-packages/jinja2/environment.py", line 942, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "<template>", line 1, in top-level template code
jinja2.exceptions.UndefinedError: 'use_orms' is undefined

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/<user>/.cache/uv/archive-v0/LPweG3ZFVAPurgjpe8gG_/bin/copier", line 12, in <module>
    sys.exit(CopierApp.run())
             ~~~~~~~~~~~~~^^
  File "/home/<user>/.cache/uv/archive-v0/LPweG3ZFVAPurgjpe8gG_/lib/python3.13/site-packages/plumbum/cli/application.py", line 631, in run
    inst, retcode = subapp.run(argv, exit=False)
                    ~~~~~~~~~~^^^^^^^^^^^^^^^^^^
  File "/home/<user>/.cache/uv/archive-v0/LPweG3ZFVAPurgjpe8gG_/lib/python3.13/site-packages/plumbum/cli/application.py", line 626, in run
    retcode = inst.main(*tailargs)
  File "/home/<user>/.cache/uv/archive-v0/LPweG3ZFVAPurgjpe8gG_/lib/python3.13/site-packages/copier/_cli.py", line 428, in main
    return _handle_exceptions(inner)
  File "/home/<user>/.cache/uv/archive-v0/LPweG3ZFVAPurgjpe8gG_/lib/python3.13/site-packages/copier/_cli.py", line 71, in _handle_exceptions
    method()
    ~~~~~~^^
  File "/home/<user>/.cache/uv/archive-v0/LPweG3ZFVAPurgjpe8gG_/lib/python3.13/site-packages/copier/_cli.py", line 418, in inner
    with self._worker(
         ~~~~~~~~~~~~^
        dst_path=destination_path,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<4 lines>...
        overwrite=True,
        ^^^^^^^^^^^^^^^
    ) as worker:
    ^
  File "/home/<user>/.cache/uv/archive-v0/LPweG3ZFVAPurgjpe8gG_/lib/python3.13/site-packages/copier/_main.py", line 278, in __exit__
    raise value
  File "/home/<user>/.cache/uv/archive-v0/LPweG3ZFVAPurgjpe8gG_/lib/python3.13/site-packages/copier/_cli.py", line 426, in inner
    worker.run_update()
    ~~~~~~~~~~~~~~~~~^^
  File "/home/<user>/.cache/uv/archive-v0/LPweG3ZFVAPurgjpe8gG_/lib/python3.13/site-packages/copier/_main.py", line 102, in _wrapper
    return func(*args, **kwargs)
  File "/home/<user>/.cache/uv/archive-v0/LPweG3ZFVAPurgjpe8gG_/lib/python3.13/site-packages/copier/_main.py", line 1173, in run_update
    self._apply_update()
    ~~~~~~~~~~~~~~~~~~^^
  File "/home/<user>/.cache/uv/archive-v0/LPweG3ZFVAPurgjpe8gG_/lib/python3.13/site-packages/copier/_main.py", line 1197, in _apply_update
    with replace(
         ~~~~~~~^
        self,
        ^^^^^
    ...<9 lines>...
        exclude=[*self.template.exclude, *self.exclude],
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ) as old_worker:
    ^
  File "/home/<user>/.cache/uv/archive-v0/LPweG3ZFVAPurgjpe8gG_/lib/python3.13/site-packages/copier/_main.py", line 278, in __exit__
    raise value
  File "/home/<user>/.cache/uv/archive-v0/LPweG3ZFVAPurgjpe8gG_/lib/python3.13/site-packages/copier/_main.py", line 1210, in _apply_update
    old_worker.run_copy()
    ~~~~~~~~~~~~~~~~~~~^^
  File "/home/<user>/.cache/uv/archive-v0/LPweG3ZFVAPurgjpe8gG_/lib/python3.13/site-packages/copier/_main.py", line 102, in _wrapper
    return func(*args, **kwargs)
  File "/home/<user>/.cache/uv/archive-v0/LPweG3ZFVAPurgjpe8gG_/lib/python3.13/site-packages/copier/_main.py", line 1097, in run_copy
    rmtree(self.subproject.local_abspath)
    ~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/<user>/.local/share/uv/python/cpython-3.13.2-linux-x86_64-gnu/lib/python3.13/shutil.py", line 763, in rmtree
    _rmtree_safe_fd(stack, onexc)
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
  File "/home/<user>/.local/share/uv/python/cpython-3.13.2-linux-x86_64-gnu/lib/python3.13/shutil.py", line 704, in _rmtree_safe_fd
    onexc(func, path, err)
    ~~~~~^^^^^^^^^^^^^^^^^
  File "/home/<user>/.local/share/uv/python/cpython-3.13.2-linux-x86_64-gnu/lib/python3.13/shutil.py", line 665, in _rmtree_safe_fd
    orig_st = os.lstat(name, dir_fd=dirfd)
FileNotFoundError: [Errno 2] No such file or directory: PosixPath('/tmp/copier._main.old_copy.nyycf1xx/dev')

Template

https://github.com/opalmedapps/deploy-pie/blob/main/copier.yaml

Relevant line: https://github.com/opalmedapps/deploy-pie/blob/e423eb7f4e06b5db7defa705db6759308563617e/copier.yaml#L425

_exclude:
  - "{% if not use_orms %}compose.orms.yaml{% endif %}"

To Reproduce

1. Run copier copy with commit 2f4b65597c3689648d8d1a011d69d9122c41cc87
2. Run copier update

Logs

Expected behavior

No exception :)

It seems that _exclude is evaluated too early. I wonder if it is possible to do it later, i.e., first ask questions, and then evaluate. I guess this happens because the new template is copied first?

Screenshots/screencasts/logs

No response

Operating system

Linux

Operating system distribution and version

Rocky Linux

Copier version

9.11.3

Python version

CPython 3.13.2

Installation method

uvx+pypi

Additional context

No response

[mschoettle]

It's actually not limited to new questions. It also happens to variables present in .copier_answers.

For now, I am working around it with:

- "{% if use_orms is not defined or not use_orms %}compose.orms.yaml{% endif %}"

[sisp]

Thanks for reporting this bug, @mschoettle! 👍

I've eventually been able to confirm it, but it's subtle and the bug only surfaces in your case because of your strict undefined extension.

Your assessment of the root cause seems correct. I've also discovered that the new update algorithm suffers from this problem, too. And I think a similar problem exists for the _skip_if_exists setting as well. 🙈 But I'm not 100% sure yet how to fix this easily. 🙁 Architecturally, it might be useful to decouple question-answering from project generation, so we could load the current project's answers, run pre-migrations (potentially modifying the answers), ask the new template's questions, and then generate the fresh copies of old and new templates using also the new template's _exclude patterns which would be rendered using the new template's answers. Alternatively – possibly simpler but more hacky –, we could generate copies without the new template's _exclude patterns and remove excluded paths in the temporary locations as a post-processing step. There seems to be (at least) one problem though: The current update algorithm also overwrites the existing subproject with a fresh copy of the new template, so post-generation removal of excluded paths is tricky. The new update algorithm is simpler and cleaner in this regard, so we won't be facing this problem anymore. Still, post-processing only feels like the second best option (although also the current draft of the new update algorithm relies on some post-processing, too). I'll give this some more thought.

At least you have a workaround for now. 😌