refactor!: replace jinja2-ansible-filters by jinja2-copier-extension

[sisp]

I've replace the jinja2-ansible-filters dependency with our own reimplementation (jinja2-copier-extension) of most of the Jinja2 filters by relying on the Ansible filters documentation as much as possible to avoid copying/rewriting GPL-licensed code. The tests are based on a combination of Ansible filters documentation and some thinking of my own.

According to copier-org/jinja2-copier-extension#1, which I've released as v0.1.0:

This is the set of re-implemented filters from jinja2-ansible-filters:

• ans_groupby
• ans_random
• b64decode
• b64encode
• basename
• bool
• checksum
• comment – This filter seems a bit involved, and I'm not sure how relevant it is for Copier.
• dirname
• expanduser
• expandvars
• extract
• fileglob
• flatten
• from_json
• from_yaml
• from_yaml_all
• hash
• mandatory
• md5
• quote
• random_mac
• realpath
• regex_escape
• regex_findall
• regex_replace
• regex_search
• relpath
• sha1
• shuffle
• splitext
• strftime
• subelements – This filter seems a bit involved, and I'm not sure how relevant it is for Copier.
• ternary
• to_datetime
• to_json
• to_nice_json
• to_nice_yaml
• to_uuid
• to_yaml
• type_debug
• win_basename
• win_dirname
• win_splitdrive

At this point, 2 filters are missing.

I'm marking this PR as a draft for now because it's a breaking change at least because two filters from jinja2-ansible-filters are missing, potentially also because the reimplementation might not behave 100% identically (although I tried to be thorough). Thus, after merging this PR, we'd need to make a major release. But to reduce the number of major releases (and breaking changes for our users), I suggest collecting a few more PRs with breaking changes (e.g., revising Copier's public API, possibly modeling copier.yaml content with Pydantic, perhaps more). I'll be happy to keep them up-to-date until we're ready for batch merging and making the major release.

The way I see it, jinja2-copier-extension (actually, no different to jinja2-ansible-filters) is essentially factored out code from Copier, as the registered Jinja2 filters (and their exact behavior) are directly exposed as template features of Copier. Hence, I believe we should pin the version of jinja2-copier-extension in our dependency manifest to have control over the exposed Jinja2 filters and their exact behavior. When a new release of jinja2-copier-extension becomes available, its level of change may affect also Copier's level of change, e.g. a major release may require a major Copier release if a breaking change in jinja2-copier-extension is also breaking for Copier's users (e.g., when it cannot or should not be compensated by Copier). In addition, I suggest to include updates of jinja2-copier-extension version pins (and perhaps even links to its release logs) in our changelog. For this reason, I've added a package rule to our Renovate config that overrides the semantic commit type for jinja2-copier-extension updates to fix. I've drawn some inspiration for this approach from Pydantic's dependency on pydantic-core.

(That said, I'm wondering whether we should also pin the jinja2 package (in a different PR, of course), as its behavior is directly exposed to Copier template authors. 🤔)

Sorry for the long delay since submitting #1809.

WDYT, @copier-org/maintainers?

Supersedes #1809.
Fixes #1398.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 97.97%. Comparing base (1b4bbb7) to head (847fdb5).
Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2039      +/-   ##
==========================================
- Coverage   98.01%   97.97%   -0.05%     
==========================================
  Files          54       54              
  Lines        5807     5822      +15     
==========================================
+ Hits         5692     5704      +12     
- Misses        115      118       +3     

Flag	Coverage Δ
unittests	97.97% <100.00%> (-0.05%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[pawamoy]

Regarding version pinning, I generally find it too costly in terms of maintenance, but that's because I have lots of inter-dependencies between the numerous projects I maintain. Here it's just one dependency (copier -> jinja2-copier-extension), so if you and @yajo are fine maintaining that, that's fine 👍

Regarding a major bump in Copier, that's probably the safest thing to do, yes, even though I allowed myself in the past not to consider a change to be breaking after code searches on GitHub yielded zero (or almost zero) use of the changed API. Got to make one's life easier sometimes.

So to conclude: sounds good to me!

[yajo]

IMHO we should put lower bounds only.

This way we leave downstream packagers to do their work in a more easy way. On our side, we still supply the nix packages for whoever wants officially supported pinnings. Or they can reuse our poetry.lock in any other way.

[sisp]

Hm, but we don't test the behavior of Jinja filters from jinja2-copier-extension in Copier, so if we change a filter in an incompatible way without pinning in our manifest, the identical Copier version will behave differently when installed before and after that release of jinja2-copier-extension.

Concretely, I'm planning to revise some API inconsistencies, that I retained from jinja2-ansible-filters for backwards compatibility, in jinja2-copier-extension in a future release. For example:

 def do_regex_findall(
     string: str,
-    # TODO(sisp): Rename argument to `pattern` for consistency with other filters.
-    regex: str,
+    pattern: str,
-    # TODO(sisp): Require flags to be keyword-only arguments.
-    multiline: bool = False,  # noqa: FBT001 FBT002
-    ignorecase: bool = False,  # noqa: FBT001 FBT002
+    *,
+    multiline: bool = False,
+    ignorecase: bool = False,
 ) -> list[str] | list[tuple[str, ...]]:

Renaming regex to pattern may be a breaking change in case somebody passes the pattern as a keyword argument. And making multiline and ignorecase keyword-only arguments may be breaking in case somebody passes values as positional arguments. If we release these changes as v0.2.0 and use a lower bound in our pyproject.toml

jinja2-copier-extension = ">=0.1.0"

then next time somebody re-installs Copier (e.g., even just pipx run copier with an expired package cache suffices) a previously working Copier template may suddenly fail to work.

Isn't the difference in this case that the Jinja filters registered by jinja2-copier-extensions.CopierExtension are part of Copier's public (Jinja) API, whereas the public API of other dependencies is used merely internally in Copier?

[yajo]

Apart from that comment, I think the improvement is cool. I don't mind doing 1 or 10 major releases. They're free 🤷🏼‍♂️ . We can become the next google chrome, no problem 😆