test: add oversized test split plan

[alteixeira20]

Summary

Adds a reproducible oversized test-file split plan.

This PR adds:

• tests/tools/build_oversized_test_split_plan.py
• tests/OVERSIZED_TEST_SPLIT_PLAN.md

The plan is generated from current repo data and is intended to guide future split PRs safely. It does not move tests, rewrite assertions, extract helpers, change CI, or change production behavior.

Target branch

• This PR targets dev, not main.

Linked Issue

Fixes #3983

Part of #2523

Type of Change

• Bug fix
• New feature
• Breaking change
• Test tooling
• Documentation
• CI / configuration

Checklist

• I searched open issues and open PRs - this is not a duplicate.
• This PR targets dev.
• My changes are limited to the scope described above - no unrelated refactors or whitespace changes mixed in.
• I actually ran the app (docker compose up or uvicorn app:app) and verified the change works end-to-end.

For this planning-only test-tooling PR, "end-to-end" means the audit builder was run against the live test tree, regenerated the plan deterministically, and produced current metrics from pytest collection, taxonomy classification, AST scanning, and static setup-signal scans. This PR does not change app runtime behavior, routes, UI, or user-facing behavior.

How to Test

Regenerate the plan:

.venv/bin/python tests/tools/build_oversized_test_split_plan.py

Validate formatting and syntax:

git diff --check
.venv/bin/python -m py_compile tests/tools/build_oversized_test_split_plan.py

Confirm the generated plan is fresh after regeneration:

git diff --exit-code -- tests/OVERSIZED_TEST_SPLIT_PLAN.md

Confirm pytest collection count:

.venv/bin/python -m pytest --collect-only -q tests

Local validation:

• git diff --check: passed
• py_compile: passed
• builder regeneration: passed
• deterministic regeneration check: passed
• taxonomy sanity: passed, zero all-unknown failure
• test files scanned: 583
• collected pytest items counted: 3586

Methodology

The builder generates the plan using:

• recursive test_*.py discovery under tests/;
• physical line counts;
• AST counts for test_* functions and Test* classes;
• pytest --collect-only -q tests for collected item counts;
• tests._taxonomy.classify_test_path for area/sub-area classification;
• static setup-signal scans for route/API, DB/session, import-state, security, filesystem, subprocess/script, async/threading, and UI/static indicators.

Static signals are treated only as review prompts. They are not proof that a file is unsafe, and the generated plan explicitly states that future split PRs still need manual review.

Taxonomy Coverage Note

The generated plan intentionally surfaces uncategorized split candidates in a dedicated section instead of reclassifying them in this PR.

This PR does not change taxonomy behavior. Any taxonomy refinement for oversized split candidates should happen in a separate focused issue/PR before those files are used as split targets.

Behavior Added

The new builder:

• writes tests/OVERSIZED_TEST_SPLIT_PLAN.md;
• writes raw metrics to /tmp/oversized-test-file-metrics.json;
• fails closed if pytest collection fails;
• fails closed if taxonomy classification returns unknown for every file;
• records thresholds used for candidate selection;
• separates taxonomy coverage gaps from first manual-review candidates;
• separates first manual-review candidates from high-risk/deferred candidates;
• documents safe rules for later split PRs.

Safety Details

This PR does not:

• move any test files;
• rewrite assertions;
• extract helpers;
• change production code;
• change CI workflows;
• change root tests/conftest.py;
• change taxonomy behavior;
• change normal pytest behavior.

Future split PRs should be separate, scoped, and validated independently.

Visual / UI changes - REQUIRED if you touched anything that renders

Anything that changes what the UI looks like - buttons, icons, padding, colors, fonts, spacing, layout, CSS, HTML, SVG, or any static/js/ module that draws to the DOM - needs all of the following. PRs that change rendering without these WILL be closed.

• Screenshot or short clip of the change in the running app, attached below. Mobile screenshot too if the change affects mobile.
• Style match: the change uses Odysseus's existing visual language.
• No new component patterns.
• I am not an LLM agent submitting a bulk PR.

N/A: planning/test-tooling-only PR. No UI, CSS, HTML, SVG, or rendering files changed.

Screenshots / clips

N/A - no visual or UI changes.

[RaresKeY]

I rechecked this against the current PR head and refreshed the linked roadmap state. #3982 has merged and #3973 is closed now, so the old sequencing note is no longer the blocker. The remaining issue is that the generated plan is stale on this branch.

Findings

issue (test): Regenerate the checked-in split plan on the current head

• Problem: Running venv/bin/python tests/tools/build_oversized_test_split_plan.py on the exact PR head rewrites the checked-in plan. The committed document reports 541 scanned test files and 3263 collected pytest items, but the generator now reports 543 and 3267. It also updates the area/sub-area distributions and the tests/test_platform_compat.py line count, so the PR's own freshness check fails after regeneration.

• Impact: This PR is meant to add a reproducible, current-data planning document. As submitted, it would land stale generated data and leave the next split-planning work based on counts that the branch itself no longer reproduces.

• Ask: Update the branch, rerun the builder, and commit the regenerated tests/OVERSIZED_TEST_SPLIT_PLAN.md. Since the branch is also behind current dev, rerun it after the final branch update so the landed plan reflects the actual merge candidate.

• Location: tests/OVERSIZED_TEST_SPLIT_PLAN.md:29

Validation

• git diff --check origin/dev...HEAD passed.
• venv/bin/python -m py_compile tests/tools/build_oversized_test_split_plan.py passed.
• venv/bin/python tests/tools/build_oversized_test_split_plan.py completed and rewrote the generated plan.
• git diff --exit-code -- tests/OVERSIZED_TEST_SPLIT_PLAN.md failed with the generated-plan drift above.
• venv/bin/python -m pytest --collect-only -q tests collected 3267 tests.

PR Hygiene

• #3982 is merged and #3973 is closed, while #3983 and #2523 remain open. The branch still needs the generated plan refreshed and then regenerated once more after the final update/rebase onto current dev.

[alteixeira20]

Thanks for the review.

I updated the branch onto current dev and regenerated tests/OVERSIZED_TEST_SPLIT_PLAN.md from the current test tree, so the checked-in plan now reflects the current builder output.

I also re-ran the freshness validation after regeneration. The remaining failing CI check is unrelated to this PR: the same two tests/test_api_chat_security.py failures reproduce on the exact current dev base with _WebhookManager missing fire_and_forget.

This PR still only changes the generated split-plan markdown and the plan builder script.

[RaresKeY]

LGTM