update docs and use extension prefix for app extensions (#3088)

This mostly contains doc updates for creating app extensions with one
fix so app extensions use the extension prefix config instead of
hardcoding `extensions`.

Co-authored-by: jasquat <jasquat@users.noreply.github.com>

Author: jasquat

docs/explanation/dev/app_extensions.md

@@ -7,9 +7,21 @@ repository at application boot:
 SPIFFWORKFLOW_BACKEND_APP_EXTENSIONS='["example"]'
 ```
 
-Arena loads `extensions/<name>/__init__.py` beneath
-`SPIFFWORKFLOW_BACKEND_BPMN_SPEC_ABSOLUTE_DIR` and calls `init_app(app)` after
-initializing the database integration. Packages support relative imports.
+Arena loads `<prefix>/<name>/__init__.py` beneath
+`SPIFFWORKFLOW_BACKEND_BPMN_SPEC_ABSOLUTE_DIR`, where `<prefix>` is
+`SPIFFWORKFLOW_BACKEND_EXTENSIONS_PROCESS_MODEL_PREFIX` (default: `extensions`).
+It calls `init_app(app)` after initializing the database integration. Packages
+support relative imports. Install any additional dependencies in the deployment
+environment.
+
+Python app extensions share this directory prefix with BPMN/UI extensions, but
+have their own explicit enable list above; they do not require
+`SPIFFWORKFLOW_BACKEND_EXTENSIONS_API_ENABLED`. For example, setting the prefix to
+`custom/extensions` loads `custom/extensions/example/__init__.py`, not
+`extensions/example/__init__.py`. There is no fallback to the default directory.
+The prefix must be a nonempty repository-relative directory, not the repository
+root. Absolute paths, `..` traversal, and symlinks escaping the repository are
+rejected when Python app extensions are enabled.
 
 ```python
 # extensions/example/__init__.py
@@ -28,6 +40,32 @@ identifiers, and package paths must stay within the configured repository.
 Packages receive private module identities to isolate relative imports between
 app instances.
 
+## Initialization lifecycle and ordering
+
+The app factory calls `setup_config`, `db.init_app`, `migrate.init_app`, then
+`init_app_extensions`, in that order. The database engine is available at this
+point, but no connection is opened or schema migration run by those initialization
+calls. Access Flask context-dependent APIs inside `with app.app_context():`.
+Database URI and engine options have already been consumed; an initializer cannot
+reconfigure the existing engine by changing those settings.
+
+Initializers run in configured list order, with no automatic dependency sorting.
+If one initializer depends on another's registrations, list its dependency first.
+Keep module imports free of side effects and use `init_app(app)` to register the
+extension's hooks and providers.
+
+This is an early registration hook, not an "application fully ready" hook.
+Celery, the background-job publisher, scheduler, and some logging/error handling
+services are initialized afterward. Do not assume they are available, or start
+background work that depends on unfinished setup. Initializers can also run
+before worker processes fork; take care with process-local resources.
+
+API processes, workers, and Flask CLI commands all create apps. Initializers must
+therefore account for multiple app instances and command-line use, including
+schema migration commands. Document each extension's dependencies, configuration,
+setup/migration requirements, and operational behavior alongside its package in
+the process-model repository.
+
 ## Model source providers
 
 An extension can register a provider at `app.extensions["model_sources"]` to

docs/how_to_guides/deployment/deploy.md

@@ -42,6 +42,13 @@ You do not need container-specific Celery env var differences for this split; `.
 
 ## Configuration
 
-The app uses sqlalchemy and supports mysql, postgres, or sqlite.
-You must set `SPIFFWORKFLOW_BACKEND_DATABASE_TYPE` to one of these three values as well as setting `SPIFFWORKFLOW_BACKEND_DATABASE_URI`.
+The app uses SQLAlchemy and supports MySQL, PostgreSQL, or SQLite.
+Set `SPIFFWORKFLOW_BACKEND_DATABASE_URI` to the SQLAlchemy connection URL; Arena
+infers the database type from that URL. `SPIFFWORKFLOW_BACKEND_DATABASE_TYPE`
+selects the default connection configuration when no URI is supplied.
 Check the `default.py` config file in the `spiffworkflow-backend/src/spiffworkflow_backend/config` directory for more env vars.
+
+For trusted Python initialization code, see
+[Python app extensions](/explanation/dev/app_extensions). Configuration and
+operational instructions for individual extensions belong with their packages
+in the process-model repository.

docs/how_to_guides/extend_the_system/extensions.md

@@ -23,6 +23,11 @@ This variable activates the extensions feature in the SpiffWorkflow backend:
 
 By default, Spiff Arena will look for extensions in `[SPIFFWORKFLOW_BACKEND_BPMN_SPEC_ABSOLUTE_DIR]/extensions`, but that can be configured using `SPIFFWORKFLOW_BACKEND_EXTENSIONS_PROCESS_MODEL_PREFIX`.
 
+[Python app extensions](/explanation/dev/app_extensions) use the same configured
+directory prefix, but are enabled separately with
+`SPIFFWORKFLOW_BACKEND_APP_EXTENSIONS`. They run trusted Python initialization
+code at application boot, rather than BPMN workflows through the extensions API.
+
 ### Creating an Extension
 
 After enabling extensions from the backend, you can create extensions in the Spiff Arena frontend.

spiffworkflow-backend/src/spiffworkflow_backend/services/app_extensions.py

@@ -24,10 +24,16 @@ def init_app_extensions(app: Flask) -> None:
     if not names:
         return
     root = Path(app.config["SPIFFWORKFLOW_BACKEND_BPMN_SPEC_ABSOLUTE_DIR"]).resolve()
+    prefix = app.config.get("SPIFFWORKFLOW_BACKEND_EXTENSIONS_PROCESS_MODEL_PREFIX", "extensions")
+    if not isinstance(prefix, str) or not prefix.strip() or Path(prefix).is_absolute() or ".." in Path(prefix).parts:
+        raise ValueError("SPIFFWORKFLOW_BACKEND_EXTENSIONS_PROCESS_MODEL_PREFIX must be a repository-relative directory")
+    extensions_root = (root / prefix).resolve()
+    if extensions_root == root or not extensions_root.is_relative_to(root):
+        raise ValueError("Application extension directory escapes its repository or is the repository root")
     for name in names:
-        package = (root / "extensions" / name).resolve()
+        package = (extensions_root / name).resolve()
         source = (package / "__init__.py").resolve()
-        if not package.is_relative_to(root / "extensions") or not source.is_relative_to(package):
+        if not package.is_relative_to(extensions_root) or not source.is_relative_to(package):
             raise ValueError(f"Application extension escapes its repository package: {name!r}")
         if not source.is_file():
             raise FileNotFoundError(f"Application extension initializer not found: {source}")

spiffworkflow-backend/tests/spiffworkflow_backend/unit/test_app_extensions.py

@@ -7,17 +7,18 @@
 from spiffworkflow_backend.services.app_extensions import init_app_extensions
 
 
-def make_extension(root: Path, name: str, source: str) -> Path:
-    package = root / "extensions" / name
+def make_extension(root: Path, name: str, source: str, *, prefix: str = "extensions") -> Path:
+    package = root / prefix / name
     package.mkdir(parents=True)
     (package / "__init__.py").write_text(source)
     return package
 
 
-def configured_app(root: Path, names: object) -> Flask:
+def configured_app(root: Path, names: object, *, prefix: object = "extensions") -> Flask:
     app = Flask(__name__)
     app.config["SPIFFWORKFLOW_BACKEND_BPMN_SPEC_ABSOLUTE_DIR"] = str(root)
     app.config["SPIFFWORKFLOW_BACKEND_APP_EXTENSIONS"] = names
+    app.config["SPIFFWORKFLOW_BACKEND_EXTENSIONS_PROCESS_MODEL_PREFIX"] = prefix
     return app
 
 
@@ -55,21 +56,67 @@ def test_missing_package_fails(tmp_path: Path) -> None:
         init_app_extensions(configured_app(tmp_path, ["missing"]))
 
 
+@pytest.mark.parametrize("prefix", ["extensions", "plugins", "company/plugins"])
 @pytest.mark.parametrize("escape", ["directory", "initializer"])
-def test_symlink_escape_fails(tmp_path: Path, escape: str) -> None:
+def test_symlink_escape_fails(tmp_path: Path, escape: str, prefix: str) -> None:
     outside = tmp_path / "outside"
     outside.mkdir()
     (outside / "__init__.py").write_text("raise AssertionError('must not run')\n")
     root = tmp_path / "models"
-    (root / "extensions").mkdir(parents=True)
-    package = root / "extensions" / "escape"
+    (root / prefix).mkdir(parents=True)
+    package = root / prefix / "escape"
     if escape == "directory":
         package.symlink_to(outside, target_is_directory=True)
     else:
         package.mkdir()
         (package / "__init__.py").symlink_to(outside / "__init__.py")
     with pytest.raises(ValueError, match="escapes"):
-        init_app_extensions(configured_app(root, ["escape"]))
+        init_app_extensions(configured_app(root, ["escape"], prefix=prefix))
+
+
+@pytest.mark.parametrize("prefix", ["plugins", "company/plugins"])
+def test_custom_prefix_loads_only_configured_directory(tmp_path: Path, prefix: str) -> None:
+    make_extension(tmp_path, "example", "raise AssertionError('must not load default directory')\n")
+    package = make_extension(
+        tmp_path, "example", "from .helper import VALUE\ndef init_app(app):\n    app.extensions['value'] = VALUE\n", prefix=prefix
+    )
+    (package / "helper.py").write_text("VALUE = 'custom'\n")
+    app = configured_app(tmp_path, ["example"], prefix=prefix)
+    init_app_extensions(app)
+    assert app.extensions["value"] == "custom"
+
+
+def test_default_prefix_when_setting_absent(tmp_path: Path) -> None:
+    make_extension(tmp_path, "example", "def init_app(app):\n    app.extensions['loaded'] = True\n")
+    app = configured_app(tmp_path, ["example"])
+    app.config.pop("SPIFFWORKFLOW_BACKEND_EXTENSIONS_PROCESS_MODEL_PREFIX")
+    init_app_extensions(app)
+    assert app.extensions["loaded"] is True
+
+
+def test_missing_custom_package_does_not_fall_back(tmp_path: Path) -> None:
+    make_extension(tmp_path, "example", "raise AssertionError('must not fall back')\n")
+    with pytest.raises(FileNotFoundError, match="plugins"):
+        init_app_extensions(configured_app(tmp_path, ["example"], prefix="plugins"))
+
+
+@pytest.mark.parametrize("prefix", ["", "   ", None, 42, "/outside", "../outside", "plugins/../extensions", ".", "./"])
+def test_invalid_prefix_fails(tmp_path: Path, prefix: object) -> None:
+    make_extension(tmp_path, "example", "raise AssertionError('must not run')\n")
+    with pytest.raises(ValueError):
+        init_app_extensions(configured_app(tmp_path, ["example"], prefix=prefix))
+
+
+@pytest.mark.parametrize("prefix", ["extensions", "plugins", "company/plugins"])
+def test_prefix_symlink_cannot_escape_repository(tmp_path: Path, prefix: str) -> None:
+    root = tmp_path / "models"
+    outside = tmp_path / "outside"
+    make_extension(outside, "example", "raise AssertionError('must not run')\n", prefix="packages")
+    directory = root / prefix
+    directory.parent.mkdir(parents=True)
+    directory.symlink_to(outside / "packages", target_is_directory=True)
+    with pytest.raises(ValueError, match="escapes"):
+        init_app_extensions(configured_app(root, ["example"], prefix=prefix))
 
 
 def test_repositories_with_same_name_are_isolated(tmp_path: Path) -> None: