Merge branch 'main' into copilot/fix-browserslist-vulnerability

Author: yyyyyyyan

.github/instructions/python-docstrings.instructions.md

@@ -83,7 +83,7 @@ The reverse of overstating: when a docstring uses applicability language (`Use w
 - **Describe intent, not a code-controlled enumeration** — a docstring listing the members of a set the code controls (a `parametrize` list, a module constant, registry entries) goes stale the moment the set changes. Name the intent and the controlling symbol ("every app in `BESPOKE_BASE_APP_PLUGINS`"), not the members. Applies in `tests/` too.
 - **Framework/core symbols stay app-agnostic** — a shared framework/core symbol's docstring describes its behaviour in generic terms; naming one downstream app's domain concept (`backup_type`, `snippet_filename`) couples the abstraction's contract to one consumer. Restate as the generic role.
 - **State the role; don't pin what the reader can grep.** Two shapes rot on the next change and are never edited at the change site: **a count** ("read by seven non-alerts apps", "the only two callers") and **an enumeration of callers or consumers**. Write the *property* instead — not "read by seven non-alerts apps" but "read by every app offering `alert_on_fail`"; not "used by `a.py`, `b.py`, `c.py`" but "shared across the subtree". When the enumeration is genuinely load-bearing, the enforcement belongs in code (a registry, an `__all__`, a guard), with the docstring pointing at it — prose is not a mechanism.
-- **A parity claim is scoped to the members it covers.** "The expected row orders are the same literals `TestListQueryPaginatedPostgres` asserts" reads as a guarantee over the whole class; when it holds for only some members it is false for the rest and nothing marks which. Scope the assertion and name the exceptions ("…the all-NULL and `select_related` cases are MySQL-only"). Cross-dialect test classes are the recurring shape.
+- **A parity claim is scoped to the members it covers.** "The expected row orders are the same literals `TestListQueryPaginatedPostgres` asserts" reads as a guarantee over the whole class; when it holds for only some members it is false for the rest and nothing marks which. Scope the assertion and name the exceptions ("…the all-NULL case is PostgreSQL-only"). Cross-dialect test classes are the recurring shape.
 
 ## Say it once, in the surface that owns it
 

.github/instructions/python-duplication.instructions.md

@@ -42,7 +42,7 @@ Concrete cases: `3306`/`5432` ports → `DEFAULT_MYSQL_PORT`/`DEFAULT_POSTGRESQL
 | `len(v) > 0` `field_validator` | `NonEmptyStr` |
 | `.strip().lower()` `field_validator` | `Annotated[str, StringConstraints(strip_whitespace=True, to_lower=True)]` or `LowercaseStr` |
 | `field_validator` doing a string-*shape* check (split on a separator, reject empty halves, reject stray whitespace) | `Annotated[str, StringConstraints(pattern=...)]` field type |
-| `.nulls_last()` on an `ORDER BY` term | `app/core/db/utils.py::NullsLastOrdering(column, *, descending=False)` — `.nulls_last()` emits SQL MySQL cannot parse. Pass the bare column plus `descending=`, never a pre-`desc()`-ed expression |
+| `.nulls_last()` on an `ORDER BY` term | `app/core/db/utils.py::NullsLastOrdering(column, *, descending=False)` — one shared cache-keyed construct; pass the bare column plus `descending=`, never a pre-`desc()`-ed expression (which would render `<expr> DESC ASC NULLS LAST`) |
 
 **Rule of thumb.** If a new decorator or class has 15+ lines of state management (timestamps, eviction, key hashing, TTL math), ask "why isn't this `@alru_cache` or `@ttl_cache`?" Flag as **Important**.
 

.github/workflows/python.yaml

@@ -80,7 +80,7 @@ jobs:
       - name: Install test dependencies
         run: poetry sync --no-root
       - name: Run tests
-        run: make test PYTEST_MARKERS='not postgres and not mysql' COV=${{ matrix.python == '3.11.9' && '1' || '0' }}
+        run: make test PYTEST_MARKERS='not postgres' COV=${{ matrix.python == '3.11.9' && '1' || '0' }}
       - name: Generate coverage comment data
         id: coverage_comment
         if: matrix.python == '3.11.9' && github.event_name == 'pull_request'
@@ -125,34 +125,6 @@ jobs:
       - name: Run PostgreSQL tests
         run: make test PYTEST_MARKERS='postgres' PYTEST_WORKERS=0 COV=0
 
-  test_mysql:
-    name: test (mysql)
-    runs-on: ubuntu-latest
-    services:
-      mysql:
-        image: mysql:8.0.46
-        env:
-          MYSQL_ROOT_PASSWORD: sep
-          MYSQL_DATABASE: sep_test
-        ports: ["3306:3306"]
-        options: >-
-          --health-cmd "mysqladmin ping -h 127.0.0.1 -psep --silent"
-          --health-interval 10s --health-timeout 5s --health-retries 10
-    env:
-      SEP_TEST_MYSQL_DSN: mysql+aiomysql://root:sep@127.0.0.1:3306/sep_test
-    steps:
-      - name: Checkout codebase
-        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
-      - name: Setup Python
-        uses: ./.github/actions/setup-python-job
-        with:
-          python-version: "3.11.9"
-          cache: poetry
-      - name: Install test dependencies
-        run: poetry sync --no-root --with mysql
-      - name: Run MySQL tests
-        run: make test PYTEST_MARKERS='mysql' PYTEST_WORKERS=0 COV=0
-
   build:
     strategy:
       matrix:

README.md

@@ -403,7 +403,7 @@ SEP supports multiple database engines for different components. Each component
 ```yaml
 SEP:
   DATABASE:
-    ENGINE: sqlite  # Database engine: sqlite, mysql, postgresql
+    ENGINE: sqlite  # Database engine: sqlite, postgresql
     USER: null
     PASSWORD: null
     HOST: ""  # Database host (empty string for SQLite to avoid URL construction issues)
@@ -429,18 +429,6 @@ TASKS:
     NAME: tasks.db
 ```
 
-#### MySQL/MariaDB Configuration
-```yaml
-SEP:
-  DATABASE:
-    ENGINE: mysql
-    USER: sep_user
-    PASSWORD: your_secure_password
-    HOST: localhost
-    PORT: 3306
-    NAME: sep_database
-```
-
 #### PostgreSQL Configuration
 ```yaml
 SEP:
@@ -455,7 +443,6 @@ SEP:
 
 Supported database engines:
 - `sqlite`: SQLite database (default for development)
-- `mysql`: MySQL/MariaDB database
 - `postgresql`: PostgreSQL database
 
 > [!NOTE]

app/core/db/config.py

@@ -30,18 +30,17 @@
 from app.core.utils.fields import AsyncDatabaseEngine
 
 #: The driver kwarg each async dialect uses for its connect timeout. asyncpg
-#: takes ``timeout``, aiomysql takes ``connect_timeout``; aiosqlite's
-#: ``timeout`` means lock wait, not connect, so SQLite is absent.
+#: takes ``timeout``; aiosqlite's ``timeout`` means lock wait, not connect, so
+#: SQLite is absent.
 _CONNECT_TIMEOUT_KEYS: dict[AsyncDatabaseEngine, str] = {
     AsyncDatabaseEngine.POSTGRESQL: "timeout",
-    AsyncDatabaseEngine.MYSQL: "connect_timeout",
 }
 
 
 class DatabaseOptions(BaseModel):
     """Define configuration options for a database connection.
 
-    :param ENGINE: The database engine to use (e.g., SQLite, MySQL, PostgreSQL).
+    :param ENGINE: The database engine to use (e.g., SQLite, PostgreSQL).
         Defaults to SQLite.
     :param USER: The username for the database connection.
     :param PASSWORD: The password for the database connection.
@@ -58,9 +57,8 @@ class DatabaseOptions(BaseModel):
         SQLAlchemy's default. Must be ``> 0``.
     :param CONNECT_TIMEOUT: Seconds to wait for a TCP connect. Unset passes no
         ``connect_args``, leaving the driver's own default. Forwarded as
-        ``timeout`` for asyncpg and ``connect_timeout`` for aiomysql; omitted
-        for SQLite, where that key means lock wait rather than connect. Must
-        be ``> 0``.
+        ``timeout`` for asyncpg; omitted for SQLite, where that key means lock
+        wait rather than connect. Must be ``> 0``.
     :param POOL_PRE_PING: Whether to test each pooled connection for liveness
         before handing it out. Defaults to ``True`` so a dead connection is
         discarded and replaced transparently. Unlike the sizing fields, this

app/core/db/crud.py

@@ -51,7 +51,6 @@
     PaginatedResponse,
     Pagination,
 )
-from app.core.utils.fields import DatabaseDialect
 
 logger = logging.getLogger(__name__)
 
@@ -296,40 +295,6 @@ async def _mutate_where(
         await session.commit()
         return result
 
-    @classmethod
-    async def _mutate_where_returning_with_for_update(
-        cls,
-        session: AsyncSession,
-        builder: _QueryBuilder,
-        *whereclause: ColumnExpressionArgument[bool],
-        returning: Iterable[str] | bool,
-        **equal_filters: Any,
-    ) -> list[Any]:
-        """Execute a DML statement with `FOR UPDATE`.
-
-        This method is a workaround for MySQL, which does not support `RETURNING` in
-        UPDATE/DELETE statements. It first selects the rows with `FOR UPDATE`, then
-        executes the DML statement, and finally returns the selected rows.
-        """
-        query = cls._build_query(
-            *whereclause, builder=_select_builder(col(cls.Model.id)), **equal_filters
-        ).with_for_update()
-        result = await cls._exec(session, query)
-
-        if row_ids := result.all():
-            ids_filter = col(cls.Model.id).in_(row_ids)
-            await cls._mutate_where(session, builder, ids_filter, returning=False)
-        else:
-            return []
-
-        if returning is True:
-            return await cls.list(session, ids_filter)
-
-        if set(returning) == {"id"}:
-            return row_ids
-
-        return await cls.values_list(session, returning, ids_filter)
-
     @classmethod
     async def _dml_where(
         cls,
@@ -342,23 +307,24 @@ async def _dml_where(
         """Execute a DML statement (UPDATE or DELETE) with the specified filters.
 
         This method ensures that at least one filter is provided to avoid unintentional
-        mass updates or deletions, and checks for database dialect-specific handling of
-        the `RETURNING` clause.
+        mass updates or deletions.
+
+        :param session: The SQLAlchemy asynchronous session to use for database
+            operations.
+        :param builder: The builder producing the UPDATE or DELETE statement.
+        :param whereclause: The filter expressions applied to the statement.
+        :param returning: The column names to return, ``True`` for whole rows, or
+            ``False`` for none.
+        :param equal_filters: Additional equality filters applied to the statement.
+        :return: The raw cursor result when nothing is returned, a list of rows when
+            more than one column is requested, or a list of scalars for a single one.
+        :raises ValueError: If neither ``whereclause`` nor ``equal_filters`` is given.
         """
         if not whereclause and not equal_filters:
             raise ValueError(
                 "You must specify at least one filter in *whereclause or **equal_filters"
             )
 
-        if returning and session.get_bind().name == DatabaseDialect.MYSQL:
-            return await cls._mutate_where_returning_with_for_update(
-                session,
-                builder,
-                *whereclause,
-                returning=returning,
-                **equal_filters,
-            )
-
         result = await cls._mutate_where(
             session,
             builder,
@@ -841,26 +807,21 @@ async def get_or_create(
         instance exists, it returns it. Otherwise, it creates and saves a new one.
 
         The creation step is conflict-tolerant: it uses a dialect-aware idempotent
-        insert (``INSERT ... ON CONFLICT DO NOTHING`` / ``INSERT IGNORE``) so that two
-        calls racing to create the same row do not surface a duplicate-key error. The
-        losing call no-ops on the insert and refetches the winning row with
-        ``created=False``. ``created`` is ``True`` only for the call whose insert
+        insert (``INSERT ... ON CONFLICT DO NOTHING`` on PostgreSQL and SQLite) so
+        that two calls racing to create the same row do not surface a duplicate-key
+        error. The losing call no-ops on the insert and refetches the winning row
+        with ``created=False``. ``created`` is ``True`` only for the call whose insert
         actually landed.
 
         :param session: The SQLAlchemy asynchronous session to use for database
             operations.
-        :type session: AsyncSession
         :param instance_create: The data used to filter and possibly create the
             instance.
-        :type instance_create: B
         :param filter_include: The set of fields of `instance_create` to be included in
             the search filter. Use None (default) for all fields.
-        :type filter_include: set[str] | None
         :param extra_fields: Additional fields to be set on the created instance.
-        :type extra_fields: Any
         :return: The existing or newly created instance of `cls.Model`, and a bool
             specifying whether a new instance was created.
-        :rtype: tuple[T, bool]
         :raises HTTPBadRequestException: If a ``DatabaseError`` occurs during the
             insert commit.
         :raises RuntimeError: If the post-conflict refetch matches no row, meaning

app/core/db/utils.py

@@ -33,7 +33,7 @@
     text,
     TypeDecorator,
 )
-from sqlalchemy.dialects import mysql, postgresql, sqlite
+from sqlalchemy.dialects import postgresql, sqlite
 from sqlalchemy.dialects.postgresql import JSONB
 from sqlalchemy.engine import Connection
 from sqlalchemy.ext.asyncio import async_sessionmaker, AsyncEngine, create_async_engine
@@ -153,9 +153,6 @@ def func_json_extract(
       expression indexes keep matching.
     - SQLite: ``json_extract(col, '$.a.b')``. SQLite auto-unquotes scalars, so
       the result is directly comparable to a string.
-    - MySQL: ``json_extract(col, '$.a.b')``. No functional index is created on
-      MySQL because a width-limited ``CAST`` would introduce comparison
-      truncation; MySQL dev environments fall back to non-indexed filtering.
 
     :param db_engine: The database engine type (e.g., ``"postgresql"``).
     :type db_engine: str
@@ -183,39 +180,29 @@ def func_json_extract(
 def idempotent_insert(engine_name: str, table: Any) -> GenericInsert:
     """Return a dialect-specific INSERT that ignores duplicate-key conflicts.
 
-    PostgreSQL and SQLite use ``INSERT ... ON CONFLICT DO NOTHING``; MySQL uses
-    ``INSERT IGNORE ...``. The caller chains ``.values(...)`` and passes the
-    result to ``session.execute``.
+    PostgreSQL and SQLite use ``INSERT ... ON CONFLICT DO NOTHING``. The caller
+    chains ``.values(...)`` and passes the result to ``session.execute``.
 
-    :param engine_name: SQLAlchemy engine ``name`` (``"postgresql"``, ``"sqlite"``,
-        or ``"mysql"``).
-    :type engine_name: str
+    :param engine_name: SQLAlchemy engine ``name`` (``"postgresql"`` or
+        ``"sqlite"``).
     :param table: The target table or ORM model class.
-    :type table: Any
     :return: A dialect-specific insert construct.
-    :rtype: GenericInsert
     :raises NotImplementedError: If the dialect is not supported.
     """
     if engine_name == DatabaseDialect.POSTGRESQL:
         return postgresql.insert(table).on_conflict_do_nothing()
     if engine_name == DatabaseDialect.SQLITE:
         return sqlite.insert(table).on_conflict_do_nothing()
-    if engine_name == DatabaseDialect.MYSQL:
-        return mysql.insert(table).prefix_with("IGNORE")
     raise NotImplementedError(f"idempotent_insert: unsupported dialect {engine_name!r}")
 
 
 class NullsLastOrdering(ColumnElement):
     """Render an ``ORDER BY`` term that places NULLs last on every supported dialect.
 
-    PostgreSQL and SQLite render the standard ``NULLS LAST`` clause. MySQL has no
-    such syntax, so its hook prepends ``ISNULL(<expr>) ASC`` -- ``ISNULL`` yields
-    ``1`` for NULL and ``0`` otherwise, pinning NULLs last independently of the
-    primary direction.
+    PostgreSQL and SQLite render the standard ``NULLS LAST`` clause.
 
     Takes the direction as a flag rather than a pre-directed expression: wrapping an
-    already-``desc()``-ed expression would make the MySQL hook emit the invalid
-    ``ISNULL(<expr> DESC)``.
+    already-``desc()``-ed expression would render ``<expr> DESC ASC NULLS LAST``.
 
     Participates in SQLAlchemy's compiled-statement cache, with a key that
     discriminates both column and direction.
@@ -253,30 +240,6 @@ def _compile_nulls_last_ordering(
     return f"{compiler.process(element.column, **kw)} {direction} NULLS LAST"
 
 
-@compiles(NullsLastOrdering, DatabaseDialect.MYSQL)
-def _compile_nulls_last_ordering_mysql(
-    element: NullsLastOrdering, compiler: SQLCompiler, **kw: Any
-) -> str:
-    """Render MySQL's ``ISNULL(<expr>) ASC, <expr> <direction>`` equivalent.
-
-    The interpolated text is the compiler's own rendering of the wrapped
-    expression, never a client-supplied value: sort keys are allowlisted by
-    :attr:`~app.core.db.list_query.ListQuerySpec.sortable` before they reach the
-    construct, and :class:`NullsLastOrdering` coerces a raw string argument into a
-    bound parameter rather than SQL text. Path literals carried by
-    :func:`func_json_extract` use ``literal_execute``, so the dialect's literal
-    processor inlines them at execution -- the same rendering that function
-    documents, unchanged by the wrapper.
-
-    :param element: The ordering construct being compiled.
-    :param compiler: The active SQL compiler.
-    :return: The rendered pair of ``ORDER BY`` terms.
-    """
-    rendered = compiler.process(element.column, **kw)
-    direction = "DESC" if element.descending else "ASC"
-    return f"ISNULL({rendered}) ASC, {rendered} {direction}"
-
-
 def prepare_unsafe_value_for_json_comparison(db_engine: str, value: Any) -> Any:
     """Prepare a value for JSON comparison based on the database engine.
 

app/core/settings_override/registry.py

@@ -1148,10 +1148,9 @@ def _stored_key_matches_override_key(
     """Return whether a stored override key resolves to the same field as ``key``.
 
     Nested keys go through :func:`canonical_override_key`. Top-level keys also
-    match case-insensitively: the previous SQL ``WHERE key = ...`` lookup
-    inherited MySQL's ``utf8mb4_0900_ai_ci`` collation, so moving the filter
-    into Python must not drop a mixed-case top-level row that DELETE/PATCH
-    used to find. Snapshot application still ignores unknown casing via
+    match case-insensitively so mixed-case stored keys remain visible to
+    DELETE/PATCH after the filter moved into Python. Snapshot application still
+    ignores unknown casing via
     :func:`app.core.settings_override.cache._apply_top_level_row`; DELETE
     removes those inert rows, and PATCH heals their stored key to the
     canonical spelling so the next snapshot can read them.
@@ -1182,7 +1181,7 @@ async def override_rows_for_key(
     non-canonically-cased nested or top-level row visible to DELETE and PATCH,
     which previously matched the stored column with dialect-dependent SQL
     equality and, after the filter moved into Python, missed mixed-case
-    top-level rows on MySQL.
+    top-level rows.
 
     Inactive rows are included: both write paths currently match on
     ``(setting_class, key)`` alone, so an inactive row stays deletable and