Mixed file management (migrations cmds vs jinja)

[OverkillGuy]

I can't seem to find out how to mix _migrations cmds (used during major upgrade), while keeping the regular upgrade/reconciliation loop of other non-major updates.
I worry I'd have to _exclude the entire file to migrate, loosing the latter, in order for the former to work.

Example

I'm using copier to manage the lifecycle of my pyproject.toml file, at the
root of a python project. In v1, this file was a poetry file. Template's v2
"upgrades" this to use uv instead. Same file different syntax/tool.

The nicest way to upgrade from v1 to v2 (poetry -> uv) is to run uvx migrate-to-uv, so what I want to have is:

• when upgrading everywhere except v2, use the normal expansion
• when passing through v2, run the migration task instead of the usual process.

So I wrote it as a migration task:

_migrations:
  - version: v2.0.0
    command: uvx migrate-to-uv && uv lock
    when: "{{ _stage == 'before' }}"

But this poses a few problems to me (may be because I misunderstand the tool?):

cmd-result conflicts with jinja-expand

The result of v2 template expansion is clashing with the result of v1->v2 migration tasks, making a merge conflict where the "before" is result of migration task, and "after" is the result of from-scratch v2 template expansion of the file.

The result is this pointless merge-conflict exercise where I have to teach the user to blindly "take ours" to fix this conflict but ONLY in case of v2, ONLY for this one file. It's a bad user experience.

Any chance I am just misunderstanding the tool/a flag?

And the second issue is a consequence of me trying to work around this:

Disabling the update altogether via _exclude means missing on all minor updates

From what I see of the docs, it seems like the solution to avoid this merge conflict, is to make the file _excluded, like this:

_exclude:
  - "{% if _copier_operation == 'update' -%}pyproject.toml{% endif %}"

But then it means that updates like v1.3 -> v1.4, or v2.1 -> v2.2 would completely ignore the minor, non-cmd, templated pyproject.toml changes.
This feels wrong, like I'm giving up a major part of copier's power just to get migrations to work?

The current track I was exploring while reporting here was maybe update the Jinja filter to say something like "if operation is update and major before = v1 and major after = v2: exclude pyproject.toml"?

_exclude:
  - "{% if _copier_operation == 'update' and _version_pep440_from.major == '1' and _version_pep440_to.major = '2' -%}pyproject.toml{% endif %}"

But this still feels hacky?

So, in summary:

How is mixed (migration cmds vs jinja) file lifecycle management supposed to work?

I'm sure others have bumped into it, and figured this out, so how do y'all manage the lifecycle of files that mostly work from regular jinja upgrades, but major bumps require broader commands that need to ignore jinja for just this one round?

I tried going over the docs all over again, and this kind of workflow wasn't clearly outlined (beyond _migrations and _exclude being listed individually).

If it's unclear to others too, I'd absolutely volunteer to document this workflow once I understand it better, if it helps anyone else.

[sisp]

I think your expectation is valid, and your implementation using a pre-update migration looks right, too. I've created a test case to debug this scenario:

def test_update_migrate_poetry_group_to_dependency_groups(
    tmp_path_factory: pytest.TempPathFactory,
) -> None:
    src, dst = map(tmp_path_factory.mktemp, ("src", "dst"))

    build_file_tree(
        {
            src / "copier.yml": "_subdirectory: template/",
            src / "template" / "{{ _copier_conf.answers_file }}.jinja": (
                "{{ _copier_answers|to_yaml }}"
            ),
            src / "template" / "pyproject.toml.jinja": (
                """\
                [tool.poetry.group.dev.dependencies]
                pytest = "*"
                """
            ),
        }
    )
    git_save(src, tag="v1")

    build_file_tree(
        {
            src / "copier.yml": (
                """\
                _subdirectory: template/

                _migrations:
                -   version: v2
                    when: "{{ _stage == 'before' }}"
                    command: "{{ _copier_python }} {{ _copier_conf.src_path / 'migrate_pep735.py' }}"
                """
            ),
            src / "template" / "pyproject.toml.jinja": (
                """\
                [dependency-groups]
                dev = ["pytest"]
                """
            ),
            src / "migrate_pep735.py": (
                """\
                from pathlib import Path
                import tomlkit

                pyproject = Path("pyproject.toml")
                doc = tomlkit.parse(pyproject.read_bytes())

                deps = sorted(doc["tool"]["poetry"]["group"]["dev"]["dependencies"])
                del doc["tool"]["poetry"]["group"]["dev"]

                dep_groups = tomlkit.table()
                dep_groups.add("dev", deps)
                doc.add("dependency-groups", dep_groups)

                pyproject.write_text(tomlkit.dumps(doc).lstrip())
                """
            ),
        }
    )
    git_save(src, tag="v2")

    run_copy(str(src), dst, vcs_ref="v1")

    git_save(dst, "init")
    pyproject = dst / "pyproject.toml"
    pyproject.write_text(pyproject.read_text() + 'mypy = "*"\n')
    git_save(dst, "add mypy in poetry group")

    run_update(dst, overwrite=True, unsafe=True)
    assert pyproject.read_text() == snapshot(
        """\
[dependency-groups]
dev = ["mypy", "pytest"]
"""
    )

On the current master branch, I get the following snapshot:

+ <<<<<<< before updating
+ [tool.poetry.group.dev.dependencies]
+ pytest = "*"
+ mypy = "*"
+ =======
  [dependency-groups]
- dev = ["mypy", "pytest"]
?        --------
+ dev = ["pytest"]
+ >>>>>>> after updating

This is indeed more than I'd expect. Adding && git add pyproject.toml to the migration commands helps, resulting in the following snapshot:

  [dependency-groups]
+ <<<<<<< before updating
  dev = ["mypy", "pytest"]
+ =======
+ dev = ["pytest"]
+ >>>>>>> after updating

I think this conflict cannot be avoided because the same line in pyproject.toml has changed in both the template and the project (after applying the pre-update migration), so Git's merge algorithm cannot know how these changes should be combined. Excluding this file from the Git-based update algorithm isn't generally valid because there could be other parts of the file that require updating as well which aren't covered by the migration.

That said, I think it shouldn't be necessary to run git add pyproject.toml after the actual migration command because the result of the pre-update migration should be treated as the current project state without additional actions. The current behavior is caused by the admittedly hacky, manual implementation of essentially a 3-way merge. I have a draft PR (#2376) waiting to be finalized for the next major release which simplifies the update algorithm implementation. When I run the test case with this implementation, I get the (essentially) same snapshot without the additional git add pyproject.toml command:

  [dependency-groups]
+ <<<<<<< HEAD
  dev = ["mypy", "pytest"]
+ =======
+ dev = ["pytest"]
+ >>>>>>> copier/after-updating

Does this analysis make sense to you, @OverkillGuy?

[sisp]

On second thought, I wonder: Should pre-update migrations also run on the fresh copy of the old template version? 🤔 This way, the imperative updating via the migration task would also be applied to the fresh copy of the old template version, so if the migration performs the changes from the old to the new template version programmatically, there is no diff between the migrated old copy and the new copy, so the Git update algorithm has nothing to do. I've prototyped this quickly by modifying #2376 a bit, and then the snapshot is empty, i.e. there is no merge conflict. But I'll need to double-check that applying pre-update migrations to the fresh copy of the old template version is generally correct.

[OverkillGuy]

Thank you for looking into it! I felt crazy, as the behaviour I was seeing wasn't what I expected, and now you confirm this is surprising to you too.

Adding a git add in migration cmd is an interesting trick, but as you describe, it's only a symptom of the real issue.

On the 3 way merge, I do remember trying to understand the lifecycle graph, to understand the order of operations, puzzling if something was missing that was making my conflict not happen as I wanted, but I never managed to wrap my head around the system enough to see if either my code was wrong, or the described workflow was.

I'm happy to hear that there's a possibly smarter solution on the way that would likely avoid the issue altogether.

As for the possibility of pre-update migration on fresh copy of old template: Yes, that felt like the right thing to do, back when I tried to understand the issue, but I'm out of my depth on this issue, I'd need to dig much deeper into copier to be of useful advice, happy to defer to you.

I'm just happy that whichever way you choose, I've given you useful information that I couldn't make sense of, but you can improve things for everyone with. =)