chore: POC for backward compatible checks of models

[williambdean]

Description

The model_version in the ModelBuilder is a bit brittle.

This puts some checks that a PRs changes would be able to be loaded in again.

Current checks might also be brittle as well. Open to suggestions.

Related Issue

• Closes #
• Related to #

Checklist

• Checked that the pre-commit linting/style checks pass. Feel free to comment pre-commit.ci autofix to auto-fix.
• Included tests that prove the fix is effective or that the new feature works
• Added necessary documentation (docstrings and/or example notebooks) using numpydoc format.
• If you are a pro: each commit corresponds to a relevant logical change

---

📚 Documentation preview 📚: https://pymc-marketing--2273.org.readthedocs.build/en/2273/

[cursor]

PR Summary

Medium Risk
Adds new CI workflow logic (worktrees, environment setup, patching files) and model load/fit execution paths, which may be brittle across dependency/version changes despite being isolated to CI/scripts.

Overview
Introduces a backwards-compatibility CI check that captures baseline model artifacts from main, captures the same models from the PR head, and fails if versions or posterior variable/dim structures differ (.github/workflows/backcompat.yml).

Adds a new scripts/backcompat toolkit (capture/compare, mock sampling, auto-discovered model definitions, and local runner scripts) to quickly build+fit models with mocked sampling, persist *.nc + manifest.json, and compare against baselines; also patches pymc_marketing/pytensor_utils.py to import ancestors compatibly across PyTensor versions used in CI.

^{Written by Cursor Bugbot for commit 8b58b56. This will update automatically on new commits. Configure here.}

[codecov]

Codecov Report

❌ Patch coverage is 50.00000% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 93.04%. Comparing base (43f39c7) to head (8b58b56).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
pymc_marketing/pytensor_utils.py	50.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2273      +/-   ##
==========================================
- Coverage   93.05%   93.04%   -0.02%     
==========================================
  Files          78       78              
  Lines       12228    12231       +3     
==========================================
+ Hits        11379    11380       +1     
- Misses        849      851       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[cursor]

Inconsistent fit_data_fn fallback between capture and compare

Medium Severity

The fit_data_fn fallback differs between capture.py and compare.py. When fit_data_fn is None, capture.py falls back to build_kwargs (e.g., {"data": df}) while compare.py falls back to {}. This affects 6 of 7 active models (all CLV models). The capture step passes unexpected build_kwargs like data=df through to pm.sample(), while the compare step passes nothing — meaning baseline and candidate models are fitted under different conditions.

Additional Locations (1)

• scripts/backcompat/compare.py#L99-L101

 

[williambdean]

@copilot Please add tqdm to the conda enviroment so that it will be installed for these backward compatible checks in the CI/CD

[Copilot]

@williambdean I've opened a new pull request, #2295, to work on those changes. Once the pull request is ready, I'll request review from you.

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

[cursor]

Inconsistent fit_data_kwargs fallback between capture and compare

Medium Severity

When fit_data_fn is None, capture.py falls back to build_kwargs (e.g., {"data": df}) while compare.py falls back to {}. This affects 6 of 8 models (all CLV models without an explicit fit_data_fn). The capture step passes extra kwargs like data=df through to fit() and ultimately to pm.sample(), but the compare step does not. The two steps are fitting models with different arguments, undermining the validity of the backward-compatibility comparison. It only works silently because mock_sample accepts and ignores unknown **kwargs.

Additional Locations (1)

• scripts/backcompat/capture.py#L29-L32

 

[cursor]

Local script missing worktree script copy unlike CI

Medium Severity

The local script creates the origin/main worktree and immediately runs python -m scripts.backcompat.capture from it, but unlike the CI workflow, it never copies the current branch's backcompat scripts into the worktree. The CI workflow explicitly does cp -r scripts/backcompat/*.py into the main worktree because these scripts don't exist on main yet. The local script will fail with a ModuleNotFoundError before this PR merges, and after merging, it will use potentially stale scripts from main whenever the backcompat harness is modified on a branch — diverging from CI behavior.

Additional Locations (1)

• .github/workflows/backcompat.yml#L53-L59

 

[ricardoV94]

What about building a hash of the generated model (more precisely fgraph_from_model), and having that hard-coded in the CI. Then there is no need to try and run in older versions of the library.

Want something dumb? mmm.model.logp().dprint(print_type=True) If the logp hasn't changed then the model likely neither? Maybe add model.named_vars_to_dims for good measure

[williambdean]

Would you recommend changing this:

pymc-marketing/pymc_marketing/model_builder.py

Lines 175 to 223 in beb6224

	@property
	def id(self) -> str:
	"""Generate a unique hash value for the model.
	The hash value is created using the last 16 characters of the SHA256 hash encoding,
	based on the model configuration, version, and model type.
	Returns
	-------
	str
	A string of length 16 characters containing a unique hash of the model.
	Examples
	--------
	.. code-block:: python
	model = MyModel()
	model.id
	"0123456789abcdef"
	"""
	def _serialize_for_hash(obj):
	"""Serialize objects for deterministic hashing."""
	if hasattr(obj, "to_dict"):
	return obj.to_dict()
	if hasattr(obj, "model_dump"):
	# Handle Pydantic models (e.g., HSGPKwargs)
	return obj.model_dump(mode="json")
	# For other objects, try to use their __dict__ but filter out non-serializable items
	if hasattr(obj, "__dict__"):
	# Filter out methods, functions, and other non-serializable attributes
	return {
	k: v
	for k, v in obj.__dict__.items()
	if not callable(v) and not k.startswith("_")
	}
	# Last resort: convert to string representation
	return str(obj)
	hasher = hashlib.sha256()
	# Use JSON serialization with custom default for deterministic model IDs
	config_json = json.dumps(
	self._serializable_model_config, sort_keys=True, default=_serialize_for_hash
	)
	hasher.update(config_json.encode())
	hasher.update(self.version.encode())
	hasher.update(self._model_type.encode())
	return hasher.hexdigest()[:16]

Currently, it doesn't need data or build for an id.

[ricardoV94]

I don't have a good picture tbh. Caching is not a trivial problem and MMM models have so many moving parts that my confidence in it is pretty low.