Merge pull request #116 from plone/feature/address-scaffolding-evals

Fix scaffolding evaluation failures

Author: MrTango

.github/workflows/python-package.yml

@@ -23,7 +23,7 @@ jobs:
               uses: actions/checkout@v2
               with:
                   repository: plone/copier-templates
-                  path: copier-templates
+                  path: develop/plone/src/copier-templates
             - name: Set up Python ${{ matrix.python-version }}
               uses: actions/setup-python@v2
               with:
@@ -44,6 +44,31 @@ jobs:
                   flake8 . --exclude=copier-templates --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
             - name: Test with pytest
               env:
-                  PLONECLI_TEMPLATES_DIR: ${{ github.workspace }}/copier-templates
+                  PLONECLI_TEMPLATES_DIR: ${{ github.workspace }}/develop/plone/src/copier-templates
               run: |
                   pytest
+
+    scaffolding:
+        runs-on: ubuntu-latest
+        timeout-minutes: 20
+        steps:
+            - uses: actions/checkout@v4
+            - name: Checkout copier-templates
+              uses: actions/checkout@v4
+              with:
+                  repository: plone/copier-templates
+                  path: develop/plone/src/copier-templates
+            - name: Set up uv
+              uses: astral-sh/setup-uv@v6
+              with:
+                  python-version: "3.12"
+            - name: Install dependencies
+              run: uv sync --extra test
+            - name: Validate generated scaffolding
+              run: uv run python evals/scaffolding/run_evals.py --ci-validation
+            - name: Upload scaffolding report
+              if: always()
+              uses: actions/upload-artifact@v4
+              with:
+                  name: scaffolding-evaluation
+                  path: evals/scaffolding/results/

CHANGES.md

@@ -3,7 +3,12 @@
 ## 7.0.0b15 (unreleased)
 
 
-- Nothing changed yet.
+- Allow the plonecli skill to start the development server when the user
+  explicitly requests it.
+  [MrTango]
+
+- Harden scaffolding evaluation and remove false template Git warnings.
+  [MrTango]
 
 
 ## 7.0.0b14 (2026-08-13)

evals/scaffolding/.gitignore

@@ -0,0 +1,4 @@
+workspaces/
+results/
+__pycache__/
+*.py[cod]

evals/scaffolding/EVALUATION.md

@@ -0,0 +1,89 @@
+# Scaffolding evaluation findings
+
+> Historical baseline from before the fixes. A full verification run passed
+> 245/245 cases with no warnings on 2026-08-13. The ignored
+> `results/report.md` is mutable and may instead contain the latest quick or
+> CI-validation run.
+
+## Scope
+
+The full run executed 245 cases against the development templates checkout:
+
+- all 3 project templates;
+- all 20 feature templates individually;
+- explicit finite high-interaction matrices for backend/Svelte booleans, behavior booleans, REST booleans crossed with target mode, reachable content-type states, view choices/booleans, vocabulary types, all viewlet managers and template states, and Zope distribution/storage choices;
+- combined, reversed-order, repeated-application, hostile-input, and command-chain cases;
+- harmless root commands and the CLI command unit suite.
+
+Open-ended strings, integers, and environment-discovered choices cannot have a literal exhaustive Cartesian product. They are covered with defaults, non-default valid values, manual-choice paths, and hostile quote/newline/backslash partitions. Finite domains are exhaustive only in the explicitly named high-interaction matrices; other templates receive individual non-default cases plus their focused unit tests.
+
+Result: **237 passed, 8 failed**. See the ignored runtime report at `results/report.md` and per-case logs under `results/logs/`.
+
+## Problems
+
+### High: free text can generate invalid TOML
+
+`backend_addon` and `zope-setup` interpolate title, description, and author values directly into quoted TOML. Quotes, newlines, and backslashes can produce an invalid `pyproject.toml`.
+
+Evidence:
+
+- `hostile-backend-toml-strings`: generation returned success, but TOML validation failed.
+- `hostile-zope-toml-strings`: the generated TOML was invalid and the post-copy hook aborted while parsing it.
+
+Use a TOML-safe Jinja filter or generate these values through `tomlkit` instead of interpolating raw strings.
+
+### High: `zope_instance` cannot be added through plonecli
+
+All four `zope_instance` CLI cases failed because `plonecli add` did not list `zope_instance` in a standalone `zope-setup` project. The generated project contains both `[tool.plone.project.settings]` and `[tool.plone.backend_addon.settings]`. Project detection checks backend settings first, classifies the project as `backend_addon`, and exposes the wrong subtemplate set.
+
+Either avoid writing backend-addon settings for standalone Zope projects or make project detection prefer the substantive project settings in this mixed layout.
+
+### High: chained `create` then `setup` fails
+
+The CLI declares `chain=True`, but `chain-create-then-setup` failed after successfully creating the backend add-on. The group retains the project context detected before `create`, so `setup` still reports that it is outside a package.
+
+Refresh project context after creation, or remove command chaining if cross-context chains are not supported.
+
+### Medium: theme variants conflict without non-interactive resolution
+
+The all-feature sequence failed when `theme_barceloneta` followed `theme`: both own `profiles/default/theme.xml` and related theme paths. Copier requested an interactive overwrite despite `--defaults`, then aborted in the non-TTY evaluation.
+
+Treat theme templates as explicit alternatives and reject a second theme with a clear message, or add a documented overwrite/replacement flow.
+
+### Medium: Barceloneta integration test uses a stale path
+
+The root integration suite generated the test at `src/collective/mythemetest/tests/test_theme_my_test_theme.py`, but `tests/test_theme_barceloneta_integration.py` expects it under top-level `tests/`. Result: 23 integration cases passed and 1 failed.
+
+Update the assertion and pytest target to the generated `src/<package>/tests/` layout.
+
+## Optimization opportunities
+
+- Copier template extensions emitted hundreds of deprecation warnings because `ContextHook.update` is deprecated. Migrate hooks to modify context in `hook`.
+- `click_aliases` reads deprecated `click.__version__`; update or replace the dependency before Click 9.1.
+- Feature generation inside these nested, `--no-git` workspaces reports the outer plonecli repository as dirty. Git cleanliness checks should be scoped to the detected generated project rather than walking into an unrelated parent repository.
+- Keep the generated TOML/XML/Python validators as CI checks. They found failures that successful Copier exit codes did not detect.
+
+## Resolution
+
+All findings above have been addressed:
+
+- free-text TOML values use serialization filters;
+- standalone Zope projects are detected correctly;
+- chained `create` → `setup` refreshes project context;
+- theme variants reject conflicting overlays;
+- the Barceloneta integration test uses the generated package test path;
+- context hooks use the current in-place API;
+- the deprecated command-alias dependency was removed;
+- Git checks are scoped to the generated project;
+- subtemplate validation tasks use Copier's `_copier_operation` value and no
+  longer report files generated earlier in the same copy as pre-existing
+  changes;
+- generated TOML/XML/Python validation runs in CI.
+
+## Baseline test receipts
+
+- Root unit suite: **209 passed, 16 skipped**.
+- Copier-template unit suite: **386 passed, 2 integration tests deselected**.
+- Copier-template integration suite: **2 passed**.
+- Root integration suite: **23 passed, 1 failed** (stale Barceloneta test path above).
+- Full scaffolding matrix: **199 passed, 8 failed** (the eight cases map to four product problems above).

evals/scaffolding/README.md

@@ -0,0 +1,92 @@
+# plonecli scaffolding evaluations
+
+This directory contains a reusable, real-CLI evaluation of plonecli commands and
+all copier templates in the development checkout. It is intentionally separate
+from product and template source.
+
+## Run
+
+From the repository root:
+
+```sh
+uv run python evals/scaffolding/run_evals.py --quick
+uv run python evals/scaffolding/run_evals.py --ci-validation
+uv run python evals/scaffolding/run_evals.py
+```
+
+`--quick` runs a reduced smoke subset. `--ci-validation` runs every template
+once plus hostile-input and command checks. With no flag, the runner executes
+the explicit finite high-interaction matrices described below. It audits the
+repository template inventory and fails when a new template has no lane. The runner itself invokes plonecli only as
+`uv run --project /workspaces/plonecli plonecli`, so copied generated projects cannot shadow the checkout with their own environment. It sets:
+
+```text
+PLONECLI_TEMPLATES_DIR=/workspaces/plonecli/develop/plone/src/copier-templates
+```
+
+Generated trees are disposable and always live beneath `workspaces/`. Reports
+and per-case command logs are written beneath `results/`:
+
+- `results/report.json` — machine-readable case inventory, commands, coverage,
+  validation results, counts, and failures.
+- `results/report.md` — human-readable coverage table and problem summary.
+- `results/logs/*.log` — captured stdout/stderr for every case.
+
+Both output directories are ignored by git and replaced at the start of a run.
+
+## Coverage
+
+The full run covers:
+
+- harmless root commands: help, template list, versions, and bash/zsh/fish
+  completion output;
+- real non-default creation of `backend_addon`, `zope-setup`, and the `addon`
+  composite, plus a real standalone `setup` application;
+- every currently shipped backend subtemplate individually against a copied
+  clean parent, plus `zope_instance` against a copied Zope parent;
+- both backend headless states and both Svelte custom-element states;
+- the complete behavior boolean matrix (4 cases);
+- all REST boolean states crossed with normal/manual registration targets
+  (64 cases);
+- all reachable content-type gated boolean/choice states;
+- every view base class × template × marker × normal/manual target state;
+- both vocabulary implementation choices;
+- all 26 viewlet managers with both template values (52 cases);
+- both Zope distributions × all three storage modes, and all three
+  `zope_instance` storage modes;
+- one all-backend-subtemplates project, reversed-order pairs, and representative
+  repeated-application/idempotency cases;
+- TOML-hostile quote/newline/backslash partitions and a real chained
+  `create` → `setup` command.
+
+Names are unique per isolated project to make collisions deterministic. The
+report records the full planned and actually executed counts by category, and
+each matrix case records its parameter values.
+
+## Validation and safety
+
+Every generated project receives deterministic syntax and duplicate-registration
+checks without installing Plone. The template unit suite supplies
+feature-specific semantic assertions:
+
+- every TOML file is parsed with `tomllib`;
+- every XML and ZCML file is parsed;
+- every Python file is compiled;
+- exact duplicate direct-child XML registrations are reported where practical;
+- every subprocess exit code is checked;
+- stdin is disabled and every command has a configurable timeout.
+
+The harness does **not** run `serve` or `debug`, and does not run a generated
+project's `test` task because those branches can start services or resolve a
+full Plone environment. Instead it runs the repository's root CLI command unit
+test suite, which covers `serve`, `debug`, and `test` dispatch and error paths
+with mocked subprocesses. Template hooks may still ask native `uv` to resolve
+small hook-only tools (`tomlkit`, Copier extensions); use a warmed uv cache for
+the most network-independent run.
+
+## Reading failures
+
+A nonzero runner exit means at least one case failed or was blocked. Start with
+`results/report.md`, then inspect the referenced log. Failures are retained as
+evaluation findings rather than hidden or retried with defaults. Reports include
+repository commits, dirty state, Python, and uv provenance.