feat: Add _operation variable

[lkubb]

• Makes exclude configuration templatable.
• Adds a _copier_operation variable to the rendering contexts for exclude and tasks, representing the current operation - either copy, recopy or update.
• Adds the (semi-)undocumented _copier_python, _copier_conf.os and _folder_name variables to the docs in a second commit (since I restructured them anyways) extracted into docs: Restructure context vars, add undocumented ones #1856, will rebase once it's merged rebased

Notes:
This was proposed here: #1718 (comment)

I hope the way it's implemented and tested is as intended by @yajo. The tests are quite synthetic, I would expect the usual application to be _copier_operation (!/=)= "update".

Fixes: #1725
Fixes: #1625
(mostly) Fixes: #240

Alternative to #1732 (regarding update exclusions only).

[yajo]

The docs would need to reflect this change too.

[yajo]

No need to pass ˋmodeˋ if it is in ˋself.operationˋ, right?

[lkubb]

Initially I did it like that, but noticed that this would cause a behavior change (at least in theory):

During _apply_update(), self.operation is update, but it calls on run_copy() several times, which would pass copy to _check_unsafe() before this patch.

[yajo]

Yes. However, that is correct. You will notice that there are calls to replace. In those calls, you can replace some configuration for the sub-worker that is created. Could you please try doing it that way?

[lkubb]

Sorry, I'm not sure I'm following. First, let me sum up:

• Introducing _copier_conf.operation means we have an attribute on the worker representing the current high-level (user-requested) operation.
• You're proposing to use this reference for _check_unsafe instead of the parameter.
• I noted that doing this will change how _check_unsafe behaves during the individual copy operations that run during an update, where the high-level operation is update, but the low-level one is copy, advocating for keeping the parameter.

I'm already using replace for overriding the operation during update. Are you saying the high-level operation during the individual copy operations should be copy? Because that would mean _copier_conf.operation is always copy during template rendering, i.e. defeat the purpose of this feature.

[pawamoy]

During an update, the current template version is used to re-generate a fresh project. It makes sense to me that this project generation would use the copy phase, to make sure tasks run accordingly. But I 100% guarantee that many users will find this un-intuitive: they want copy tasks to only apply when they generate a project, not when Copier does so during an update. They don't care what Copier is doing behind the scene.

So, in my opinion, we need a third operation, like "copy-during-update". It's hard to find a name that conveys what happens. tmp-copy? Maybe update-diff, since that's what the re-generated project is used for, to compute a diff?

With these three operations, users can make sure things only run when they generate a project (not when Copier does so during an update), and when they update a project. If they want a task to be repeated in temporary generated projects, they can accept update-diff too:

if "{{ _copier_operation }}" in ("copy", "update-diff"):
    # do something

Typical examples of tasks that shouldn't be run during update-diff:

• git init (there's no point in doing so)
• printing messages to the user (you don't want the message to appear in the middle of an update's output)
• any task with side-effects, like HTTP POSTing stuff, or creating/removing files on the file-system, DB queries, and so many more things

[sisp]

The current implementation, which uses a context variable and decorator, exposes _operation = "update" to the copy tasks during copier update, so Copier's internals of re-generating a fresh project aren't leaked to the template developer. So, a task can be limited to running only on copier copy with:

_tasks:
  - command: ./run-task.sh
    when: "{{ _operation == 'copy' }}"

Does this address your concern?

[pawamoy]

What about update tasks that will run in temporary copies then? 🤔 My point is:

• copier copy -> copy op -> all good
• copier update -> update op (in temp copy) then update op again (in actual project) -> update tasks run in temp copy too, might cause issues

But I checked #240 again and I only see people with git init-like use-cases, i.e. they generally want to run something at copy and not at update, not the opposite. So for now the current changes might be enough 🙂

[pawamoy]

Only example I found, that is implicit, would be template writers running a "resolve and install" update task. It can take some time and is useless in the temporary copy.

[sisp]

Your overview of the situation is very helpful, thanks! Then, I see two scenarios and solutions:

1. Run tasks only after copier copy (e.g., git init – i.e., no task run after internal temporary copy):

   _tasks:
     - command: git init
       when: "{{ _operation == 'copy' }}"

2. Run tasks only after copier update (e.g., "resolve and install"):

   _migrations:
     - ./resolve-and-install.sh

I must admit that this situation is not entirely obvious. We might need to add documentation to clarify it better.

[pawamoy]

Yep it's a bit counter-intuitive to have to use a migration to avoid running update tasks in the temp copy. Docs would be a good first step!

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 98.02%. Comparing base (ecbe5fa) to head (dced5eb).
Report is 121 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1733      +/-   ##
==========================================
+ Coverage   98.00%   98.02%   +0.01%     
==========================================
  Files          53       54       +1     
  Lines        5720     5775      +55     
==========================================
+ Hits         5606     5661      +55     
  Misses        114      114              

Flag	Coverage Δ
unittests	98.02% <100.00%> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[lkubb]

@yajo Is there anything I still need to do here? Just making sure you noticed the changes. :)

[yajo]

Sorry for taking so long to review! I don't have a lot of time and this one required deep thinking.

[yajo]

Yes. However, that is correct. You will notice that there are calls to replace. In those calls, you can replace some configuration for the sub-worker that is created. Could you please try doing it that way?

[sisp]

Sorry for taking so long to give feedback. 🫣 This PR really needs some deep thinking.

Adding the attribute operation to the Worker class seems conceptually wrong to me because we'd make the operation mode part of its state while it also has methods to execute either operation (run_copy() and run_update()). Thus, it would be possible to set the operation attribute to "update" and call run_copy() (or set the operation attribute to "copy" and call run_update()) which would lead to incorrect behavior. This indicates a bad design to me.

How about using a context variable to manage the operation context? This is a simplified example of what I mean:

from __future__ import annotations

import sys
from contextvars import ContextVar
from dataclasses import dataclass, replace
from functools import wraps
from typing import Callable, Literal, TypeVar

if sys.version_info >= (3, 10):
    from typing import ParamSpec
else:
    from typing_extensions import ParamSpec


Operation = Literal["copy", "update"]
P = ParamSpec("P")
R = TypeVar("R")

_operation: ContextVar[Operation] = ContextVar("_operation")


def as_operation(value: Operation) -> Callable[[Callable[P, R]], Callable[P, R]]:
    def _decorator(func: Callable[P, R]) -> Callable[P, R]:
        @wraps(func)
        def _wrapper(*args: P.args, **kwargs: P.kwargs) -> R:
            token = _operation.set(_operation.get(value))
            try:
                return func(*args, **kwargs)
            finally:
                _operation.reset(token)

        return _wrapper

    return _decorator


@dataclass
class Worker:
    template: str

    @as_operation("copy")
    def run_copy(self) -> None:
        print(f"Worker({self.template}).run_copy(): {_operation.get()}")

    @as_operation("update")
    def run_update(self) -> None:
        print(f"Worker({self.template}).run_update(): {_operation.get()}")
        replace(self, template="old").run_copy()
        replace(self, template="new").run_copy()


worker = Worker("new")

print("$ copier copy ...")
worker.run_copy()
# -> Worker(new).run_copy(): copy

print("$ copier update ...")
worker.run_update()
# -> Worker(new).run_update(): update
# -> Worker(old).run_copy(): update
# -> Worker(new).run_copy(): update

The Worker.run_copy() and Worker.run_update() methods both set their operation context via the decorator as_operation(). The decorator wraps the decorated method and sets the operation context to the provided value only when the operation context has not been set yet; if the operation context has been set, it remains as is. Thus, a call of Worker.run_update() sets the operation context to "update", and when Worker.run_copy() gets called within this context, the operation context will remain "update".

Using ContextVar is thread-safe. Also, the following requirement stated in the Python docs

Important: Context Variables should be created at the top module level and never in closures. Context objects hold strong references to context variables which prevents context variables from being properly garbage collected.

is met.

I agree that the Jinja variable _copier_conf.operation only seems to make sense when rendering template expressions in copier.yaml. And although the update seems to produce correct results, the fact that the _copier_conf.operation value is unstable during an update (while rendering files etc.), it is asking for trouble IMO. So, how about omitting the _copier_conf.operation variable in the Jinja context while rendering files, directories, and path names? We'd need to document this behavior of course.

WDYT, @lkubb @yajo?

[lkubb]

Sorry for taking so long to give feedback. 🫣 This PR really needs some deep thinking.

@sisp Thank you for your thoughtful feedback and sorry for taking my time to reply as well.

Adding the attribute operation to the Worker class seems conceptually wrong to me because we'd make the operation mode part of its state while it also has methods to execute either operation (run_copy() and run_update()).

Agreed, I wasn't really happy with it myself, also because it meant the forced inclusion of the operation in inappropriate contexts. Part of this design was caused by the tight coupling of the Copier worker with the renderer imho, I wasn't sure how to solve it properly without a major refactoring.

Thus, it would be possible to set the operation attribute to "update" and call run_copy() (or set the operation attribute to "copy" and call run_update()) which would lead to incorrect behavior. This indicates a bad design to me.

Note that it's only the former situation that leads to incorrect behavior, the latter is the expected path since the run_update method replaces the operation value itself.

How about using a context variable to manage the operation context? This is a simplified example of what I mean:

Thanks a lot for your well thought-out proposal. I agree this design is superior and am currently implementing it. I'm still contemplating a detail since it aggravates a flaw I just noticed in the exclude rendering logic by decoupling the context switch implementation from the worker:

Since match_excludes is a cached property whose value now depends on the _operation, a changed _operation is not reflected in its value if an instance has already been used in a different context. This is not an issue in practice currently since a context switch always coincides with the creation of a new worker instance via replace, but it's definitely a gotcha in case this changes at some point.

[Note: An analog of this is currently present in match_skip, especially if externally mutated instance attributes need to be considered in worker methods].

I might need to address this by deleting the property's cached value explicitly in run_(re)copy/run_update to force regeneration, which is still a bit smelly, or turning it into a cached function instead (less smelly, but a breaking change).

So, how about omitting the _copier_conf.operation variable in the Jinja context while rendering files, directories, and path names? We'd need to document this behavior of course.

👍 Your proposal allows to restrict it to the scopes where it makes sense, namely exclude and task templating.

[sisp]

Thanks for updating this PR, @lkubb! 👍 This looks very good, just a couple of quite minor requests and ideas from my side.

[sisp]

Great! Just some absolutely minor remarks. Once #1856 has been merged and this one rebased, I think we can merge. But I'd like to give also @yajo the opportunity to give feedback, as he performed the first review iteration.

[sisp]

Fantastic work, @lkubb! 👌 LGTM! 🎉

[pawamoy]

Do we agree that this PR also fixes the long-standing #240?

[sisp]

@pawamoy I think it addresses part of #240, but pre/post copy/update granularity is not addressed. There's #1511, but it seems stale.

@lkubb @copier-org/maintainers A thought that originates from #1948 (review) (and subsequent comments): I think we should rename _operation to _copier_operation.

[pawamoy]

If I read #240 correctly, pre/post were specific to migrations, and are now handled with a _stage variable being "before" or "after". Tasks get the same _stage variable with value "task".

I'm not sure what pre-copy, post-copy, pre-update, or post-update would mean exactly. IMO this PR solves #240.

Let me also link again to my comment above about a third update-diff operation, which I think is super important to have.

[sisp]

You're right, I forgot about the availability of _stage now. 🤦 Yes, this PR solves #240 then. 👍

(I've also replied to your comment above.)

[pawamoy]

Actually after seeing a few duplicates of #240 like #635, I think this PR only partially solves it, as you pointed @sisp. Users want to run things before and after a copy, or before and after an update. The operation won't allow that. Currently tasks only run after. Using context hooks to run things before would be a hack (earliest is when the first Jinja template string is rendered, which is definitely not before anything). But anyway we could close #240 and re-open a fresh issue to get rid of all the legacy discussions about it.

[sisp]

Ah, you're right, @pawamoy. I got confused. 😕 But looking at https://copier.readthedocs.io/en/stable/updating/#how-the-update-works (and thinking about the detailed implementation), I think pre-/post-migrations are equivalent to pre-/post-update tasks for copier update – although implementing pre-/post-update tasks that run always (independent of template versions) is a bit hacky. So the only thing we seem to be missing is a pre-copy task. Right? 🤔

[pawamoy]

So the only thing we seem to be missing is a pre-copy task. Right?

Seems like it, yes! I feel like this deserves a proper entry in the docs, something like "Pre/post hooks", which is how Cookiecutter named them, and will be familiar to most users. This entry would document that you can use migration as pre/post-update hooks (with examples), tasks as post-copy or post-update hooks, and... pre-copy hooks are not supported yet. The docs entry should emphasize that for simple message display, users can set message_after_copy and related settings. Let me copy this in a new docs issue.

[lkubb]

I think we should rename _operation to _copier_operation.

@sisp Done and rebased.

Edit: Having read the discussion in #1948, I want to emphasize that (because of #1733 (comment)) this patch does NOT make _operation/_copier_operation available in all rendering contexts, it's limited to tasks and exclude. Maybe this aspect should be reconsidered in light of extensions.

Given the many comments this PR has accumulated over the half year it has been open, I'll try to give a final summary of what it provides and where it fits in the Copier hook system.

This PR

1. Adds templating functionality to _exclude items
2. Adds a _copier_operation context variable for the _tasks and _exclude rendering contexts, which indicates whether copier (re)copy or copier update was invoked by the user. It's absent in regular templates.

(1+2) allows to provide dumb boilerplate files which will be excluded during updates
(2) additionally relates to hooks, see after a copy below

If you need your logic to run...

before/after any `update`

Use a `migration` without specifying a version constraint and with appropriate `when` (or even `when: 'true'` to run both before and after). Example:

_migrations:
  - this_script_runs_after_all_updates.sh
  - command: this_script_runs_before_all_updates.sh
    when: "{{ _stage == 'before' }}"
  - command: this_script_runs_before_and_after_all_updates.sh
    when: "true"

You can rely on the provided env vars (STAGE/VERSION_{,PEP440_}{FROM,TO,CURRENT} to target specific commands.

before a `copy`

Come up with an ugly hack possibly involving a custom extension or provide a patch to core, not officially supported atm.

after a `copy`

Use a `task` with `when: '{{ _copier_operation == "copy" }}'`. Example:

_tasks:
  - command: this_script_runs_after_copy_operations_only.sh
    when: "{{ _copier_operation == 'copy' }}"

Note

All of the hook suggestions apply to the generated project only, not the two intermediate copies involved in update operations. If you need a task to run on these intermediate copies as well, specify a plain task. Off the top of my head, I can't come up with any use case for this, it seems fragile.

Is there anything I can do to help carry this PR over the line? It has been stalled for a while now and the release of Copier 9.5.0 broke the ugly hack I was using to simulate _copier_operation:

_tasks:
  # Don't run this when updating.
  # This breaks when recopying.
  - >-
    {%- if
      not _copier_conf.answers.last
      and '.new_copy.' not in _copier_conf.dst_path.name
      and '.old_copy.' not in _copier_conf.dst_path.name -%}
        "{{ _copier_python }}" "{{ "{src}{sep}tasks{sep}initialize.py".format(src=_copier_conf.src_path, sep=_copier_conf.sep) }}" init
    {%- endif -%}

Specifically #1880, which added this line and thereby removed _copier_conf.answers.last from the context (whose value being empty was used as a proxy for a copy operation):

copier/copier/main.py

Line 382 in e444514

conf.pop("answers")

[sisp]

[...] this patch does NOT make _operation/_copier_operation available in all rendering contexts, it's limited to tasks and exclude. Maybe this aspect should be reconsidered in light of extensions.

Thanks for pointing this out. I think we can keep _copier_operation limited to only those two contexts and expand if needed when there is a real use case.

I'm sorry that this PR has been open for so long. I took the liberty to do a (hopefully) final rebase to resolve merge conflicts. Everything looks good to me. 🎉 Thanks for your patience with the reviews/discussions and keeping this PR up to date! 🙇 👏

As @yajo has requested changes some time ago, I'm afraid I need to wait for his approval to merge.

[yajo]

I think there's a needed small extra explanation in the docs. Other than that, if it looks good to you, it's good to me :D